Does lookahead for LL(k) need to be configured?

[drhagen]

This grammar

grammar HelloWorld

entry Model:
    variable=ID ':' Unit '*' '=';

Unit:
    Multiplicative;

Multiplicative infers Unit:
    Atom ({infer Binary.left=current} operator=('*' | '/') right=Atom)?;

Atom:
    name=ID;

hidden terminal WS: /\s+/;
terminal ID: /[_a-zA-Z][\w_]*/;

cannot parse this file

x: nmol * =

The error suggests that the LL(k) parser is only looking one token ahead and choosing the Multiplicative branch at * rather than the Model branch. I admit that I am not an expert in LL(k) parsers, but I would think that it could look two tokens ahead and conclude that only the Model branch is valid. Do I need to configure the amount of lookahead available to the parser?

[msujew]

Hey @drhagen,

the issue isn't really that the lookahead doesn't look 2 tokens into the input stack. Instead, it simply disregards outer context. I.e. when performing lookahead inside of the Multiplicative rule, it never takes into account that the rule stack contains Model. See also the issue over at TypeFox/chevrotain-allstar#1

[drhagen]

Gotcha. Is there a way to force lookahead when classifying a token. I tried all of these and they didn't work:

terminal STAR: '*'(!'=');
terminal STAR: /\*(?!\s*\=)/;

I was also surprised that even this grammar would not parse x: nmol *=.

grammar HelloWorld

terminal STAR_EQUALS: '*=';
terminal STAR: '*';

entry Model:
    variable=ID ':' Unit STAR_EQUALS;

Unit:
    Multiplicative;

Multiplicative infers Unit:
    Atom ({infer Binary.left=current} operator=(STAR | '/') right=Atom)?;

Atom infers Unit:
    {infer Atom} name=ID;

hidden terminal WS: /\s+/;
terminal ID: /[_a-zA-Z][\w_]*/;

I guess I don't understand overlapping token definitions are disambiguated.

[msujew]

Your second grammar works perfectly fine for me:

That is pretty in line with what I would expect.

[drhagen]

To prove I'm not a crazy person, here is that grammar not working:

But when I try it in a new tab it works. I must have gotten the playground into an unrecoverable state.

[drhagen]

And with the functioning playground, I found that both forms of lookahead work. I found it interesting and convenient that lookahead of the form terminal STAR: '*'(!'=') operates at the token level instead of the character level.

[drhagen]

I am turning my comment into an answer because it is actually the solution to the original problem.

Langium configures lookahead based on local alternatives only, not global ones. For this particular case, it is possible to apply lookahead manually at the tokenization phase using regex lookahead (i.e. (?=...)). All mentions of the token at which lookahead failed need to be turned into terminals and put into the appropriate order.

grammar HelloWorld

terminal ASSIGNMENT_STAR: /\*(?=\s*=)/;
terminal STAR: '*';

entry Model:
    variable=ID ':' Unit ASSIGNMENT_STAR '=';

Unit:
    Multiplicative;

Multiplicative infers Unit:
    Atom ({infer Binary.left=current} operator=(STAR | '/') right=Atom)?;

Atom infers Unit:
    {infer Atom} name=ID;

hidden terminal WS: /\s+/;
terminal ID: /[_a-zA-Z][\w_]*/;