While interning at Datastax, I mainly work on DevCenter (preview version without Xtext integrated), a GUI for Cassandra database. It is built as an Eclipse RCP and I implement the editor part on Xtext. Xtext is powerful and extremely customizable, the drawback is steep learning curve. Therefore I decide to write down some problems I encountered during implementing it.
This post may contain misunderstanding or errors.
### means your project name.
Write .xtext grammar file
First you need to write the grammar for your language. Go through 5 mins tutorials then you should be good for simple languae structure. In my case, I have to build the editor part for Cassandra CQL language, which is a SQL-like language. If you want to build an editor for an existing language, please check if the grammar file of that language is available. In the beginning I write .xtext grammar file based on CQL doc by myself, which is error-prone and time consuming. Thankfully, Cassandra has an existing Antlr grammar, using it also save you some time for testing, since it has been tested by Cassandra database team. Since xtext uses Antlr internally, their grammar are not too different. The following are a series of simple naive hueristic human translating steps:
Translating from Antlr .g grammar to .xtext grammar
There are a lot of java source code in .g grammar file enclosed by {} with optional prefix @init, @after or something else. These are useless in xtext grammar.
For rules name, here comes a table (I don't know there is a fragment terminal rule in xtext until I see someone post that at stackoverflow.com...) Xtext wil generate a EObject Class for each parser rule, you may want to rewrite part of your grammar to let you do rest of the job more easily.
Rules
antlr
xtext
parser rule
rules starting with lowercase
rules starting with uppercase (each will be a Java Class)
terminal rule
rules starting with uppercase
rules using all uppercase with terminal prefix
fragment terminal rule
fragmentID
terminal fragmentID
In antlr a list of arguments may be defined like this:
After above steps, you may have things like this (Cassandra grammar has a lot of them)
// unreserved keywords is a datatype
TableName: c=ID | q=QUOTED_NAME | u=UnreservedKeywords;
It will translated to a Class called TableName which has getC()getQ()getU() to return three different fields. Later I found that since all of them return String (Terminal rule and DataType rule return String, Parser rule return a Class), I could simply write that rule as
TableName: name = (ID | QUOTED_NAME | UnreservedKeywords);
Simple, clean and easy to use.
In Cassandra, numbers and time all stored as string while pasing. I waste a lot of time writing terminal rules for them, even if it could generate a parser, the parsing results are wrong. So be sure to check if there is an existing grammar file!
You will get syntax highlight and content assist of keywords for free. The problem is their token id and constant name are generated by xtext (or antlr), which is not human readable and the name may change if you add more keywords in your grammar file. Besides of that your keywords also become case sensitive. You may find some case insensitive keywords tutorials on line, but those never work for me.
There is a workaround in Cassandra grammar.
xtext grammar
Rule:
K_SELECT selectClause=SelectClause K_FROM table=TableName T_SEMICOLON
;
terminal fragment A: 'a'|'A';
...
terminal fragment Z: 'z'|'Z';
terminal K_SELECT: S E L E C T;
terminal K_FROM: F R O M;
terminal T_SEMICOLON: ';';
...
Then you could have all your keywords be case insensitive and they all have similiar constant name in ###.parser.antlr.internal.Internal###Parser.java. The missing content assist for keywords could be added by adding a lot of complete__(...) in your content assist Class. Syntax coloring needs more work, I check text of each leaf node of AST, if its toLowerCase matches any of keywords, then color it with keyword color scheme. I also need to create a collection of all keywords, not very smart, but you only need to do it once. Using regular express to manipulate the terminal K_KEYWORDS part would save you a lot of time.
Prefer Java over Xtext
If you prefer using Java instead of Xtend, you could open workflow file Generate###.mwe2, which is in the same package as .xtext file is, and edit following lines:
In first few line, change the boolean value of var generateXtendStub = true to false.
Change validation.ValidatorFragment to validation.JavaValidatorFragment
Change contentAssist.ContentAssistFragment to contentAssist.JavaBasedContentAssistFragment
Doing so will keep only generator in .xtend, the rest files all become .java. If you would like to use xtend, I suggest you study xtend for a while if you never use it before.
Xtend
If you successfully find a way to develope only in Java, then you may skip this section. Otherwise I suggest you go through Xtend doc or a shorter post colletcing few code snippets. Besides of above, here are few issues I had corrsponding to xtend syntax
There is no break to break loop, you have to use return
Although you could omit ; in xtend, I still add them in the end. It makes me easier if I need to copy that to pure Java files.
Don't forget to put val or var to declare a variable. If you think a code looks legit but error message says This expression is not allowed in this context, since it doesn't cause any side effects., you may forget to put val or var.
Basically just follow this post from mo-seph then you are good. You register two class in ###UiModule.java at ui package in ui projet, Configuration.java is color palette. Calculator.java will traverse your AST, check if current node is particular type, if so then color it by a color you define in Configuration.java.
I define Keyspace/Table/Column name as parser rules in grammar file (which means xtext will create classes for them), then I only need to check current node type to color them for a specil color scheme. For those Terminal rules, I only have a string, and check if string.equals(any of my keyword string) and color them for keyword color scheme.
Previously I define table name as a field of a Statement instead of a parser rule. The problem is the table name could also be an unreserved keyword, and I can't distinguish which is which from leaf node.
Label Provider, Outline
The default behavior is it will create node for every field for each EObject. You could read xtext doc for this part since it's not too hard to read.
A problem I encounterd is the default root node text is filename without extension, and I want to also put extension on root node. Simply return super.sameMethodName() + ".suffix" will not work. An Itemis employee teach to how to solve this.
and then you reutrn the file name in def _xtext(Root Node Of Your Model)
Validator
Similar to JUnit, put a @Check in from of any method, everytime that kind of node changed, it will trigger that method. I put it before root node of my model (which means everytime the file changed (so does the root node), that function is triggered) to calculate something automatically.
Assume you have this kind of syntax SELECT a, b, c FROM foo; in your language, and you want to check if all columns are in table foo. If any column doesn't exist, you want to put an error underline under that column.
While I was trying, in each checking methods, you can only underline those fields belong to current node. So assume a doesn't exist in table foo, I may unerline a columns in node SelectClause, or underline field name in node ColumnName. If you want to underline in other node's field, you'll get an error.
@Check
def checkExist(SelectClause sc) {
val stmt = sc.eContainer; // get parent EObject ndoe
val tableName = stmt.table?.name; // if table field is optional, you need to check nullity
var index = -1;
for (col: columns) {
if (func-table-doesn't-contain-col) {
index = index + 1
error('error message',
###Package.Literals.SELECT_STATEMENT__COLUMNS, // a constant specifiy a field type
index, // underline only one element in a list
A_STRING_CONSTANT_YOU_NEED_FOR_QUICKFIX);
}
}
}
If you want to underline a field instead of an element in a list, you just need to omit the index argument.
Parsing Error Message Provider
I learn this from this post. You get a recongnitionException instance and use it to get the information you want.
Two main types of exceptions I encountered are:
MismatchedTokenException: parser is in a production rule and it expecting some else than the unexpecting token. I guess the error situation by the token I have and expecting token. I think there are more clever way to do thing, instead of enumerating all combinations.
NoViableAltException: parser is about to apply a new production rule, but the token it had can't fit any one of viable production rules. I could have a DFA state number and a decision number, but I have no idea how to use them.
Actually above two exceptions are throwed by Antlr. For further informaiton you may need to refer to The Definitive ANTLR Reference.
Same package with .xtext file, there is a ###RuntimeModule.java, edit it
add one more file in the same packge with Validator
parsing error message provider code snippet
class SyntaxErrorMessageProviderCustom extends SyntaxErrorMessageProvider {
@Inject IGrammarAccess grammarAccess
override getSyntaxErrorMessage(ISyntaxErrorMessageProvider.IParserErrorContext context) {
val re = context.getRecognitionException();
val unexpectedTokenType = re?.token?.type;
val unexpectedTokenText = re?.token?.text;
val expectingTokenType = re?.expecting;
// token types are some integer generated by xtext
// check ###.parser.antlr.internal.Internal###Parser.java for those constants
// what you want to do, check sample at above post
super.getSyntaxErrorMessage(context)
}
}
Template
The fatest way to do this:
Run the Xtext editor as an Eclipse application, find Preferences.###.Templates. Add templates right here.
Export templates as templates.xml. Add an attribute id for each node.
<template ..... name="Select All">SELECT * FROM ${table}</template>
to
<template ..... name="Select All" id="SelectAll">SELECT * FROM ${table}</template>
In ui project, create a templates folder, copy templates.xml to it. Then you are done.
Do not copy templates.xml into your workspace and open that .xml to add id attribute. Eclipse will reformat your .xml to something like this
<template ...
...
...>
SELECT * FROM ${table}
</template>
which cause your template insert a lot of space in front of each line.
Formatter
I just follow offical document and do some simple linewraps only.
Content Assist
Go to your ui project contentassist package. Check the parent Abstract###ProposalProvider Class to see what methods you could override. Method names tell you all. Scroll to the bottom and you will find lots of content assist methods for terminal rules. For each method, write acceptor.accept(...) once for one row of content assist.
Quick Fix
Didn't dig too must for this. Just treat it as text editing. A frustrating part is you can't retrieve the index you passe in if your error occurred in a list.
If your quick fix doesn't make any change not even an exception, you may use errStartPos and errEndPos as arguments instead of errStartPos and errLength.
A code snippet to quote a string, use NodeModelUtils.getNode(o) to get the INode of an EObjecto. Which means you could get the position of a node in original text file. (basically based on this post)
@Fix(CASE_INSENSITIVE_COLUMN_NAME)
def caseInsensitiveColumnName(Issue issue, IssueResolutionAcceptor acceptor) {
acceptor.accept(issue, 'Add double quote', 'Add double quote', null) [
context | // this line for IModification() interface
element, context | // thie line for ISemanticModification() interface, element is EObject
// issue.getOffset(); // get error starting position
// issue.getLength(); // get error length
// val inode = NodeModelUtils.getNode(element); // get INode of EObject
// inode.getOffset(); // get node starting position
// inode.getLength(); // get node length
val doc = context.xtextDocument
val str = doc.get(issue.offset, issue.length);
doc.replace(issue.offset, issue.length, '"' + str + '"')
]
}
Integrated into an Eclipse RCP
Haven't come to this step.
Some places you may find solutions for your xtext problems: