While interning at Datastax, I mainly work on DevCenter (preview version without Xtext integrated), a GUI for Cassandra database. It is built as an Eclipse RCP and I implement the editor part on Xtext. Xtext is powerful and extremely customizable, the drawback is steep learning curve. Therefore I decide to write down some problems I encountered during implementing it.
- This post may contain misunderstanding or errors.
###means your project name.
Write .xtext grammar file
First you need to write the grammar for your language. Go through 5 mins tutorials then you should be good for simple languae structure. In my case, I have to build the editor part for Cassandra CQL language, which is a SQL-like language. If you want to build an editor for an existing language, please check if the grammar file of that language is available. In the beginning I write .xtext grammar file based on CQL doc by myself, which is error-prone and time consuming. Thankfully, Cassandra has an existing Antlr grammar, using it also save you some time for testing, since it has been tested by Cassandra database team. Since xtext uses Antlr internally, their grammar are not too different. The following are a series of simple naive hueristic human translating steps:
Translating from Antlr .g grammar to .xtext grammar
There are a lot of java source code in
.g grammar file enclosed by
} with optional prefix
@after or something else. These are useless in xtext grammar.
For rules name, here comes a table (I don't know there is a fragment terminal rule in xtext until I see someone post that at stackoverflow.com...) Xtext wil generate a
EObject Class for each parser rule, you may want to rewrite part of your grammar to let you do rest of the job more easily.
|parser rule||rules starting with lowercase||rules starting with uppercase (each will be a Java Class)|
|terminal rule||rules starting with uppercase||rules using all uppercase with
|fragment terminal rule||
In antlr a list of arguments may be defined like this:
The corresponding xtext grammar is:
For those parser rules with an argument enclosed by
Most of the time they could be moved to the left hand side
The original antlr grammar define CollectionLiteral in recursive sense,
I rewrite it to iterative sense (purpose for =>)
After above steps, you may have things like this (Cassandra grammar has a lot of them)
It will translated to a Class called
TableName which has
getU() to return three different fields. Later I found that since all of them return String (Terminal rule and DataType rule return String, Parser rule return a Class), I could simply write that rule as
Simple, clean and easy to use.
In Cassandra, numbers and time all stored as string while pasing. I waste a lot of time writing terminal rules for them, even if it could generate a parser, the parsing results are wrong. So be sure to check if there is an existing grammar file!
Defining terminal rules for keywords
If you write your grammar like this
You will get syntax highlight and content assist of keywords for free. The problem is their token id and constant name are generated by xtext (or antlr), which is not human readable and the name may change if you add more keywords in your grammar file. Besides of that your keywords also become case sensitive. You may find some case insensitive keywords tutorials on line, but those never work for me.
There is a workaround in Cassandra grammar.
Then you could have all your keywords be case insensitive and they all have similiar constant name in
###.parser.antlr.internal.Internal###Parser.java. The missing content assist for keywords could be added by adding a lot of
complete__(...) in your content assist Class. Syntax coloring needs more work, I check text of each leaf node of AST, if its toLowerCase matches any of keywords, then color it with keyword color scheme. I also need to create a collection of all keywords, not very smart, but you only need to do it once. Using regular express to manipulate the
terminal K_KEYWORDS part would save you a lot of time.
Prefer Java over Xtext
If you prefer using Java instead of Xtend, you could open workflow file
Generate###.mwe2, which is in the same package as .xtext file is, and edit following lines:
- In first few line, change the boolean value of
var generateXtendStub = trueto
Doing so will keep only
generator in .xtend, the rest files all become .java. If you would like to use xtend, I suggest you study xtend for a while if you never use it before.
If you successfully find a way to develope only in Java, then you may skip this section. Otherwise I suggest you go through Xtend doc or a shorter post colletcing few code snippets. Besides of above, here are few issues I had corrsponding to xtend syntax
- There is no
breakto break loop, you have to use
- Although you could omit
;in xtend, I still add them in the end. It makes me easier if I need to copy that to pure Java files.
- Don't forget to put
varto declare a variable. If you think a code looks legit but error message says
This expression is not allowed in this context, since it doesn't cause any side effects., you may forget to put
- The way to do tyepcast:
- The way to do in xtend is
- There is no
a++, you have to use
a = a + 1.
Basically just follow this post from mo-seph then you are good. You register two class in
ui package in
Configuration.java is color palette.
Calculator.java will traverse your AST, check if current node is particular type, if so then color it by a color you define in
I define Keyspace/Table/Column name as parser rules in grammar file (which means xtext will create classes for them), then I only need to check current node type to color them for a specil color scheme. For those Terminal rules, I only have a string, and check if string.equals(any of my keyword string) and color them for keyword color scheme.
Previously I define table name as a field of a
Statement instead of a parser rule. The problem is the table name could also be an unreserved keyword, and I can't distinguish which is which from leaf node.
Label Provider, Outline
The default behavior is it will create node for every field for each EObject. You could read xtext doc for this part since it's not too hard to read.
A problem I encounterd is the default root node text is filename without extension, and I want to also put extension on root node. Simply
return super.sameMethodName() + ".suffix" will not work. An Itemis employee teach to how to solve this.
and then you reutrn the file name in
def _xtext(Root Node Of Your Model)
Similar to JUnit, put a
@Check in from of any method, everytime that kind of node changed, it will trigger that method. I put it before root node of my model (which means everytime the file changed (so does the root node), that function is triggered) to calculate something automatically.
Assume you have this kind of syntax
SELECT a, b, c FROM foo; in your language, and you want to check if all columns are in table
foo. If any column doesn't exist, you want to put an error underline under that column.
You may write your grammar like this.
While I was trying, in each checking methods, you can only underline those fields belong to current node. So assume
a doesn't exist in table
foo, I may unerline a
columns in node
SelectClause, or underline field
name in node
ColumnName. If you want to underline in other node's field, you'll get an error.
If you want to underline a field instead of an element in a list, you just need to omit the index argument.
Parsing Error Message Provider
I learn this from this post. You get a
recongnitionException instance and use it to get the information you want.
Two main types of exceptions I encountered are:
MismatchedTokenException: parser is in a production rule and it expecting some else than the unexpecting token. I guess the error situation by the token I have and expecting token. I think there are more clever way to do thing, instead of enumerating all combinations.
NoViableAltException: parser is about to apply a new production rule, but the token it had can't fit any one of viable production rules. I could have a DFA state number and a decision number, but I have no idea how to use them.
Actually above two exceptions are throwed by Antlr. For further informaiton you may need to refer to The Definitive ANTLR Reference.
Same package with
.xtext file, there is a
###RuntimeModule.java, edit it
add one more file in the same packge with Validator
The fatest way to do this:
- Run the Xtext editor as an Eclipse application, find
Templates. Add templates right here.
- Export templates as
templates.xml. Add an attribute
idfor each node. to
uiproject, create a
templates.xmlto it. Then you are done.
Do not copy
templates.xml into your workspace and open that
.xml to add
id attribute. Eclipse will reformat your
.xml to something like this
which cause your template insert a lot of space in front of each line.
I just follow offical document and do some simple linewraps only.
Go to your
contentassist package. Check the parent
Abstract###ProposalProvider Class to see what methods you could override. Method names tell you all. Scroll to the bottom and you will find lots of content assist methods for terminal rules. For each method, write
acceptor.accept(...) once for one row of content assist.
Didn't dig too must for this. Just treat it as text editing. A frustrating part is you can't retrieve the
index you passe in if your error occurred in a list.
If your quick fix doesn't make any change not even an exception, you may use
errEndPos as arguments instead of
A code snippet to quote a string, use
NodeModelUtils.getNode(o) to get the
INode of an
o. Which means you could get the position of a node in original text file. (basically based on this post)
Integrated into an Eclipse RCP
Haven't come to this step.