Tag : java « Ray's repository

Published on: August 30, 2013

Tags: java xtext eclipse xtend

While interning at Datastax, I mainly work on DevCenter (preview version without Xtext integrated), a GUI for Cassandra database. It is built as an Eclipse RCP and I implement the editor part on Xtext. Xtext is powerful and extremely customizable, the drawback is steep learning curve. Therefore I decide to write down some problems I encountered during implementing it.

This post may contain misunderstanding or errors.
### means your project name.

Write .xtext grammar file

First you need to write the grammar for your language. Go through 5 mins tutorials then you should be good for simple languae structure. In my case, I have to build the editor part for Cassandra CQL language, which is a SQL-like language. If you want to build an editor for an existing language, please check if the grammar file of that language is available. In the beginning I write .xtext grammar file based on CQL doc by myself, which is error-prone and time consuming. Thankfully, Cassandra has an existing Antlr grammar, using it also save you some time for testing, since it has been tested by Cassandra database team. Since xtext uses Antlr internally, their grammar are not too different. The following are a series of simple naive hueristic human translating steps:

Translating from Antlr .g grammar to .xtext grammar

There are a lot of java source code in .g grammar file enclosed by { } with optional prefix @init, @after or something else. These are useless in xtext grammar.

For rules name, here comes a table (I don't know there is a fragment terminal rule in xtext until I see someone post that at stackoverflow.com...) Xtext wil generate a EObject Class for each parser rule, you may want to rewrite part of your grammar to let you do rest of the job more easily.

Rules	antlr	xtext
parser rule	rules starting with lowercase	rules starting with uppercase (each will be a Java Class)
terminal rule	rules starting with uppercase	rules using all uppercase with `terminal` prefix
fragment terminal rule	`fragment` `ID`	`terminal fragment` `ID`

In antlr a list of arguments may be defined like this:

functionArgs
    : '(' ')'
    | '(' t1=term ( ',' tn=term )* ')'
    ;

The corresponding xtext grammar is:

FunctionArgs
   : '(' ')'
   | '(' ts+=Term  (',' ts+=Term )* ')'
   ;

For those parser rules with an argument enclosed by [ ]

properties[PropertyDefinitions props]
    : property[props] (K_AND property[props])*
    ;

Most of the time they could be moved to the left hand side

Properties
    : props+=Property (K_AND props+=Property)*
    ;

The original antlr grammar define CollectionLiteral in recursive sense,

set_tail
    : '}'
    | ',' t=term set_tail
    ;

map_tail
    : '}'
    | ',' k=term ':' v=term map_tail
    ;

map_literal
    : '{' '}'
    | '{'map_tail
    ;

set_or_map
    : ':' v=term 
    | set_tail
    ;

collection_literal
    : '[' ( t1=term ( ',' tn=term )* )? ']'
    | '{' t=term v=set_or_map
    | '{' '}'
    ;

I rewrite it to iterative sense (purpose for =>)

CollectionLiteral:
  => MapLiteral | SetLiteral | ListLiteral
;

MapLiteral:
    '{' {MapLiteral} (entries+=MapEntry (',' entries+=MapEntry)* )? '}'
;

MapEntry:
    key=Term ':' value=Term
;

SetLiteral:
    '{' {SetLiteral} (values+=Term (',' values+=Term)* )+ '}'
;

ListLiteral:
    '[' {ListLiteral} ( values+=Term (',' values+=Term)* )? ']'
;

After above steps, you may have things like this (Cassandra grammar has a lot of them)

// unreserved keywords is a datatype
TableName: c=ID | q=QUOTED_NAME | u=UnreservedKeywords;

It will translated to a Class called TableName which has getC() getQ() getU() to return three different fields. Later I found that since all of them return String (Terminal rule and DataType rule return String, Parser rule return a Class), I could simply write that rule as

TableName: name = (ID | QUOTED_NAME | UnreservedKeywords);

Simple, clean and easy to use.

In Cassandra, numbers and time all stored as string while pasing. I waste a lot of time writing terminal rules for them, even if it could generate a parser, the parsing results are wrong. So be sure to check if there is an existing grammar file!

Defining terminal rules for keywords

If you write your grammar like this

Rule:
    'SELECT' selectClause=SelectClause 'FROM' table=TableName ';'
;

You will get syntax highlight and content assist of keywords for free. The problem is their token id and constant name are generated by xtext (or antlr), which is not human readable and the name may change if you add more keywords in your grammar file. Besides of that your keywords also become case sensitive. You may find some case insensitive keywords tutorials on line, but those never work for me.

There is a workaround in Cassandra grammar.

xtext grammar

Rule:
    K_SELECT selectClause=SelectClause K_FROM table=TableName T_SEMICOLON
;

terminal fragment A: 'a'|'A';
    ...
terminal fragment Z: 'z'|'Z';

terminal K_SELECT: S E L E C T;
terminal K_FROM: F R O M;
terminal T_SEMICOLON: ';';
    ...

Then you could have all your keywords be case insensitive and they all have similiar constant name in ###.parser.antlr.internal.Internal###Parser.java. The missing content assist for keywords could be added by adding a lot of complete__(...) in your content assist Class. Syntax coloring needs more work, I check text of each leaf node of AST, if its toLowerCase matches any of keywords, then color it with keyword color scheme. I also need to create a collection of all keywords, not very smart, but you only need to do it once. Using regular express to manipulate the terminal K_KEYWORDS part would save you a lot of time.

Prefer Java over Xtext

If you prefer using Java instead of Xtend, you could open workflow file Generate###.mwe2, which is in the same package as .xtext file is, and edit following lines:

In first few line, change the boolean value of var generateXtendStub = true to false.
Change validation.ValidatorFragment to validation.JavaValidatorFragment
Change contentAssist.ContentAssistFragment to contentAssist.JavaBasedContentAssistFragment

Doing so will keep only generator in .xtend, the rest files all become .java. If you would like to use xtend, I suggest you study xtend for a while if you never use it before.

Xtend

If you successfully find a way to develope only in Java, then you may skip this section. Otherwise I suggest you go through Xtend doc or a shorter post colletcing few code snippets. Besides of above, here are few issues I had corrsponding to xtend syntax

There is no break to break loop, you have to use return
Although you could omit ; in xtend, I still add them in the end. It makes me easier if I need to copy that to pure Java files.
Don't forget to put val or var to declare a variable. If you think a code looks legit but error message says This expression is not allowed in this context, since it doesn't cause any side effects., you may forget to put val or var.
The way to do tyepcast:
```
(instance1 as Class2).doSomething();
```

The way to do

if (obj1 instanceof Class1) {
    ((Class1)obj1).doSomething();
} else {
    ...
}

in xtend is

switch (obj1) {
    Class1: {
            obj1.doSomething();
    }
    default: {
            ...
    }
}

There is no a++, you have to use a = a + 1.

Syntax coloring

Basically just follow this post from mo-seph then you are good. You register two class in ###UiModule.java at ui package in ui projet, Configuration.java is color palette. Calculator.java will traverse your AST, check if current node is particular type, if so then color it by a color you define in Configuration.java.

I define Keyspace/Table/Column name as parser rules in grammar file (which means xtext will create classes for them), then I only need to check current node type to color them for a specil color scheme. For those Terminal rules, I only have a string, and check if string.equals(any of my keyword string) and color them for keyword color scheme.

Previously I define table name as a field of a Statement instead of a parser rule. The problem is the table name could also be an unreserved keyword, and I can't distinguish which is which from leaf node.

Label Provider, Outline

The default behavior is it will create node for every field for each EObject. You could read xtext doc for this part since it's not too hard to read.

A problem I encounterd is the default root node text is filename without extension, and I want to also put extension on root node. Simply return super.sameMethodName() + ".suffix" will not work. An Itemis employee teach to how to solve this.

java code to retrieve filename

myXtextDocument.readOnly(new IUnitOfWork<String, XtextResource>() {
    public String exec(XtextResource resource) {
        return resource.getURI().lastSegment();
    }
});

xtend code

override createRoot(IXtextDocument doc) {
    filename = doc.readOnly([res|
        return res.URI.lastSegment
    ]);               
    super.createRoot(doc)     
}

and then you reutrn the file name in def _xtext(Root Node Of Your Model)

Validator

Similar to JUnit, put a @Check in from of any method, everytime that kind of node changed, it will trigger that method. I put it before root node of my model (which means everytime the file changed (so does the root node), that function is triggered) to calculate something automatically.

Assume you have this kind of syntax SELECT a, b, c FROM foo; in your language, and you want to check if all columns are in table foo. If any column doesn't exist, you want to put an error underline under that column.

You may write your grammar like this.

SelectStatement:
    K_SELECT selectClause=SelectClause K_FROM table=TableName T_SEMICOLON
;

SelectClause:
    columns+=ColumnName (T_COMMA columns+=ColumnName)*
;

TableName: name=STRING;
ColumnName: name=STRING;

While I was trying, in each checking methods, you can only underline those fields belong to current node. So assume a doesn't exist in table foo, I may unerline a columns in node SelectClause, or underline field name in node ColumnName. If you want to underline in other node's field, you'll get an error.

@Check
def checkExist(SelectClause sc) {
    val stmt = sc.eContainer;  // get parent EObject ndoe
  val tableName = stmt.table?.name; // if table field is optional, you need to check nullity
  
  var index = -1;
  for (col: columns) {
        if (func-table-doesn't-contain-col) {
        index = index + 1
            error('error message', 
                ###Package.Literals.SELECT_STATEMENT__COLUMNS, // a constant specifiy a field type
                index,  // underline only one element in a list
                A_STRING_CONSTANT_YOU_NEED_FOR_QUICKFIX);
        }
  }
}

If you want to underline a field instead of an element in a list, you just need to omit the index argument.

Parsing Error Message Provider

I learn this from this post. You get a recongnitionException instance and use it to get the information you want.

Two main types of exceptions I encountered are:

MismatchedTokenException: parser is in a production rule and it expecting some else than the unexpecting token. I guess the error situation by the token I have and expecting token. I think there are more clever way to do thing, instead of enumerating all combinations.
NoViableAltException: parser is about to apply a new production rule, but the token it had can't fit any one of viable production rules. I could have a DFA state number and a decision number, but I have no idea how to use them.

Actually above two exceptions are throwed by Antlr. For further informaiton you may need to refer to The Definitive ANTLR Reference.

Same package with .xtext file, there is a ###RuntimeModule.java, edit it

public class ###RuntimeModule extends ....Abstract###RuntimeModule {
    public Class <? extends ISyntaxErrorMessageProvider> bindISyntaxErrorMessageProvider() {
        return SyntaxErrorMessageProviderCustom.class;
    }
}

add one more file in the same packge with Validator

parsing error message provider code snippet

class SyntaxErrorMessageProviderCustom extends SyntaxErrorMessageProvider {
    @Inject IGrammarAccess grammarAccess
    override getSyntaxErrorMessage(ISyntaxErrorMessageProvider.IParserErrorContext context) {
    val re = context.getRecognitionException();
    val unexpectedTokenType = re?.token?.type;
    val unexpectedTokenText = re?.token?.text;
    val expectingTokenType = re?.expecting;
    // token types are some integer generated by xtext 
    // check ###.parser.antlr.internal.Internal###Parser.java for those constants
    
    // what you want to do, check sample at above post
    
    super.getSyntaxErrorMessage(context)
  }
}

Template

The fatest way to do this:

Run the Xtext editor as an Eclipse application, find Preferences.###.Templates. Add templates right here.

Export templates as templates.xml. Add an attribute id for each node.

<template ..... name="Select All">SELECT * FROM ${table}</template>

<template ..... name="Select All" id="SelectAll">SELECT * FROM ${table}</template>

In ui project, create a templates folder, copy templates.xml to it. Then you are done.

Do not copy templates.xml into your workspace and open that .xml to add id attribute. Eclipse will reformat your .xml to something like this

<template ...
     ...
     ...>
     SELECT * FROM ${table}
     </template>

which cause your template insert a lot of space in front of each line.

Formatter

I just follow offical document and do some simple linewraps only.

Content Assist

Go to your ui project contentassist package. Check the parent Abstract###ProposalProvider Class to see what methods you could override. Method names tell you all. Scroll to the bottom and you will find lots of content assist methods for terminal rules. For each method, write acceptor.accept(...) once for one row of content assist.

Quick Fix

Didn't dig too must for this. Just treat it as text editing. A frustrating part is you can't retrieve the index you passe in if your error occurred in a list.

If your quick fix doesn't make any change not even an exception, you may use errStartPos and errEndPos as arguments instead of errStartPos and errLength.

A code snippet to quote a string, use NodeModelUtils.getNode(o) to get the INode of an EObject o. Which means you could get the position of a node in original text file. (basically based on this post)

@Fix(CASE_INSENSITIVE_COLUMN_NAME)
def caseInsensitiveColumnName(Issue issue, IssueResolutionAcceptor acceptor) {
    acceptor.accept(issue, 'Add double quote', 'Add double quote', null) [
        context |   // this line for IModification() interface
        element, context |  // thie line for ISemanticModification() interface, element is EObject
            // issue.getOffset();   // get error starting position
            // issue.getLength();   // get error length
      // val inode = NodeModelUtils.getNode(element);   // get INode of EObject
      // inode.getOffset(); // get node starting position
      // inode.getLength(); // get node length
      
      val doc = context.xtextDocument
            val str = doc.get(issue.offset, issue.length);
            doc.replace(issue.offset, issue.length, '"' + str + '"')
    ]
}

Integrated into an Eclipse RCP

Haven't come to this step.

Ray's repository

xtext 2.4.2 notes