Data type using one terminal behaves differently from that terminal

[drhagen]

If I have this grammar

grammar HelloWorld

entry Model:
    ((persons+=Person) EOL)*;

Person:
    'person' name=ID;

hidden terminal WS: /[ \t]+/;
terminal ID: /[_a-zA-Z][\w_]*/;
terminal NEWLINE: '\n';
EOL returns string:
    NEWLINE;

and try to parse this file

person John
person Jane

I get an error "Expecting end of file but found 'person'".

If I replace EOL with NEWLINE and delete EOL entirely

grammar HelloWorld

entry Model:
    ((persons+=Person) NEWLINE)*;

Person:
    'person' name=ID;

hidden terminal WS: /[ \t]+/;
terminal ID: /[_a-zA-Z][\w_]*/;
terminal NEWLINE: '\n';

the file parses successfully. What am I missing something about Langium because I would have expected those to be equivalent?

[msujew]

@drhagen I've created issue #1358 from this, see there for more info on this.

As for the behavior difference, its a bug, since we didn't expect people to use data type rule without property assignment. That breaks a few things in the parser. Using this grammar instead works as expected:

grammar HelloWorld

entry Model:
    ((persons+=Person) eol=EOL)*;

Person:
    'person' name=ID;

hidden terminal WS: /[ \t]+/;
terminal ID: /[_a-zA-Z][\w_]*/;
terminal NEWLINE: '\n';
EOL returns string:
    NEWLINE;

[drhagen]

since we didn't expect people to use data type rule without property assignment

This makes me curious. What is the standard way to make a rule for which you do not care about the result? For example, how would you fix a EOL: NEWLINE | EOF rule? This is invalid, as is. I never care about the output of this rule. I had thought returns string did the minimum amount of processing, but your comment makes me think that isn't the typical solution. I did not see a returns null option for throwing away the output entirely. :-)

[msujew]

This makes me curious. What is the standard way to make a rule for which you do not care about the result?

I'm pretty sure the bug has been in there since the first few releases of Langium, so in 2 years+ no one has encountered that yet (or at least bothered to report it). Note that the EOF token is a fairly new addition to the framework.

Most languages probably just aren't case sensitive, so there was never a need to create a parser rule just to discard its result.