Lookahead in parsing rule conditions?

[rempAut]

I'm working on a grammar for structured text language (PLC programming).
There is a CASE statement, which looks like this.

CASE example OF

foo_state:
    RETURN;
    dummyAction;

10:
    RETURN;
    dummyAction;

20,30:
    RETURN;
    dummyAction;

40..50, 60..80:
    RETURN;
    dummyAction;

bar_state:    // failing here at bar_state, because it expects action call statement, please continue reading
    RETURN;
    dummyAction;

ELSE
    RETURN;
    dummyAction;

END_CASE;

Each case can be defined by:

• integer:5
• Integer list: 20,30
• Integer range: 20..30
• Enumerated value: bar_state

Each case can have 1 or many statements.

For simplicity, a statement can be a RETURN or action call ActionCall;

I'm following a specification of EBNF rules describing this language. My grammar looks like this:

grammar StructuredText

entry Document:
    (stStatements+=Statement_list);

/* Numerics declarations */
Signed_Integer returns number:
('+'|'-')? Integer;

/* Structured Text */
Statement_list:
statements=Statement ';' (Statement ';')*;

Statement:
Selection_statement | Action_call_statement ;

Action_call_statement:
action=Identifier | 'RETURN';

/* Selection statements */
//selection_statement = if_statement | case_statement;
Selection_statement:
Case_statement;

//case_statement = "CASE" expression "OF" case_element {case_element} ["ELSE" statement_list] "END_CASE";
Case_statement:
'CASE' Identifier 'OF' Case_element (Case_element)* ('ELSE' Statement_list)? 'END_CASE';
//case_element = case_list ":" statement_list;
Case_element:
Case_list ':' statements=Statement_list;
//case_list = case_list_element {"," case_list_element};
Case_list:
caseList+=Case_list_element (',' caseList+=Case_list_element)*;
//case_list_element = subrange | signed_integer | enumerated_value;
Case_list_element:
(numCase=Signed_Integer ('..' numericCaseEnd+=Signed_Integer)?) | (enumCase=Identifier);

// identifier characters
terminal fragment Underscore: /_/;
terminal fragment Letters: /[a-zA-Z]+/;
terminal fragment Digits: /[0-9]+/;
terminal Integer: Digits ((Underscore)? Digits)*;
terminal Identifier: ((Underscore (Digits | Letters))| Letters) ((Underscore)? (Digits | Letters))*;

// hidden
hidden terminal WS: /\s+/;
hidden terminal ML_COMMENT: /\/\*[\s\S]*?\*\//;
hidden terminal ML_COMMENT_ST: /\(\*[\s\S]*?\*\)/;
hidden terminal SL_COMMENT: /\/\/[^\n\r]*/;

I understand that the problem is when I enter the Statement_list from Case_element rule, the lexer find the terminal Identifier and parser expects Action_call_statement, but I need to parse it as the beginning of a new Case_list_element

I'm stuck and don't know how to get it running. I'm looking for any kind of advice. I was able to get it running in chevrotain using the lookahead condition in a rule gate, but I don't know how to do it with langium and was not able to find anything.

[msujew]

Hey @rempAut,

Langium barely exposes any internals of it using Chevrotain, so that makes the GATE approach quite difficult to use. In theory it's still possible, but you would have to basically copy all of the generic internal parser code (see here) to create your own parser and modify parts in order to support the gate statements.

Another way, which would be way easier, is to create a special identifier token which is only able to match identifiers within case elements:

Case_list_element:
(numCase=Signed_Integer ('..' numericCaseEnd+=Signed_Integer)?) | (enumCase=CaseElementIdentifier);

terminal CaseElementIdentifier: ((Underscore (Digits | Letters))| Letters) ((Underscore)? (Digits | Letters))* 
    // Lengthy positive lookahead check to find whether we are within a case list (note the colon at the end)
    /(?=(,\s*([_A-Za-z]\w*|([+\-]?[0-9][0-9_]*\s*(\.\.\s*[+\-]?[0-9][0-9_]*)?))\s*)*\:)/;
terminal Identifier: ((Underscore (Digits | Letters))| Letters) ((Underscore)? (Digits | Letters))*;

Note that your current grammar generates quite a malformed AST. You should use more assignments, as multiple unassigned subrule calls like you're currently doing (such as in Case_statement) will make it basically impossible to work with the resulting AST.

[rempAut]

Thank you, this seems to solve my problem. It is the lookahead solution, but on a terminal level (shame I didnt think of it...)
I'm aware of the AST problem, just not sure how and when to use the assignments, need to learn by trying.

[msujew]

To be honest, I also wouldn't have thought of that. The GATE based solution is usually better, but unfortunately it's not easily implemented in Langium. The terminal level solution - while it works well - is difficult to read/understand and might lead to unexpected parsing/lexer errors, as slight changes in the input might lead to the lexer returning the Identifier token instead.

[msujew]

I'm aware of the AST problem, just not sure how and when to use the assignments, need to learn by trying.

A bit of advice there: You probably want to assign everything which has some kind of information stored inside of it. Most keywords are irrelevant to the AST (such as :, ;, CASE or OF), but subrules and terminals (Identifier for example) should always be assigned, as you lose information without them.