Trouble with empty array

[ymc9]

Hi there,

I'm running into a parsing problem with my array syntax. Here's a simplified repro:

grammar HelloWorld

entry Model:
    (persons+=Person | greetings+=Greeting)*;

Person:
    'person' name=ID (array?='[]')?;

Array: '[' (items+=ID)* ']';

Greeting:
    'Hello' person=[Person:ID] items=Array '!';

hidden terminal WS: /\s+/;
terminal ID: /[_a-zA-Z][\w_]*/;

Input:

person John
Hello John []!

Error (reproduced on langium playground):

Expecting keyword '[' but found `[]`.HelloWorld(parsing-error)
Expecting keyword '!' but found `[]`.HelloWorld(parsing-error)

The empty array on the second line wouldn't parse, and it seems the conflict is introduced by the (array?='[]') in Person rule. Does such a rule generate a token implicitly?

[msujew]

Hey @ymc9,

Does such a rule generate a token implicitly?

Yes, every unique usage of keywords (anything wrapped inside of a string) generates a token type for the lexer. This is expected behavior for basically all parser generators (such as ANTLR). The obvious fix is to modify the Person rule slightly:

Person:
    'person' name=ID (array?='[' ']')?;

The issue is that since [] is a longer token type pattern, it has precedence over shorter ones, such as [. So during the lexing phase, encountering the string [] will always yield the [] token instead of separate [ and ] tokens.

[ymc9]

Got it. Thanks for the confirmation and solution!