how to match EOF

[Yokozuna59]

Is it possible to match end of file (EOF)? I tried using:

terminal EOF: /\z/;

But it didn't work out for me.

---

Originally, I wanted to create a language in which newlines were important. For example, an assignment statement:

x=1
y=2

If I used:

hidden terminal WS: /\s+/;

It well make the following code valid:

x=1 y=2

So now I've got to add a newline after that assignment statement to make sure that each statement is on a separate line, but if the assignment was at the EOF, it creates an exception asking for a newline (playground).

[msujew]

I think EOF matching can be accomplished with Langium but is rather complicated. Instead I would recommend the following approach:

Model: (variables+=Assignment (NEWLINE+ variables+=Assignment)*)?

And remove NEWLINE+ from the Assignment rule.

[msujew]

You could create a dummy token and then return EOF from the chevrotain import in its stead. You'll need to override the TokenBuilder service for that.

[spoenemann]

Sounds like we could add EOF as a built-in keyword to the grammar language?

[Yokozuna59]

You could create a dummy token and then return EOF from the chevrotain import in its stead. You'll need to override the TokenBuilder service for that.

It looks like it's really complicated, so I need to modify the source code rather than a complex regex in Langium.

Sounds like we could add EOF as a built-in keyword to the grammar language?

I guess it would be great to do so! Since it might be a common use case for languages/files where indentation/newlines are important, e.g., Python, Yaml.

[Yokozuna59]

You could create a dummy token and then return EOF from the chevrotain import in its stead. You'll need to override the TokenBuilder service for that.

Am I missing something?

Added a dummy terminal rule:

terminal EOF: /\z/;

Override TokenBuilder:

import { TokenType, EOF } from 'chevrotain';
import { DefaultTokenBuilder } from 'langium';
import { TerminalRule } from 'langium/lib/grammar/generated/ast';

class CustomTokenBuilder extends DefaultTokenBuilder {
    override buildTerminalToken(terminal: TerminalRule): TokenType {
        let tokenType = super.buildTerminalToken(terminal);
        if (tokenType.name === 'EOF') {
            tokenType = EOF;
        }
        return tokenType;
    }
}

I've also set the TokenBuilder as CustomTokenBuilder as shown here:

const services = await createServicesForGrammar({
    grammar: grammer,
    module: {
        parser: {
            TokenBuilder: () => new CustomTokenBuilder(),
        },
    },
});

[msujew]

Are you building your language services purely through the createServicesForGrammar function? That one is meant for internal framework tests and more advanced stuff like the Langium playground, but it shouldn't really be used by devs. Do you have a reproducible setup to look into?

Otherwise this seems to look fine. Do you experience any issues with this approach?

[Yokozuna59]

Are you building your language services purely through the createServicesForGrammar function? That one is meant for internal framework tests and more advanced stuff like the Langium playground, but it shouldn't really be used by devs.

I'm using it for unit testing; it works just fine, but I'm not sure if it's the best approach.

Do you have a reproducible setup to look into?

Otherwise this seems to look fine. Do you experience any issues with this approach?

Here's the actual code link.

Note: I modified the sixth line here to be:

"pie" (showData?="showData")? (NEWLINE+ | EOF)

---

I'm currently creating a wrapper grammar that has subgrammars (currently one, though).

If there is a newline, it will show $type as the subgrammar (PieChart).

pie\n

But if there's EOF it will show the parent one (Mermaid), which isn't as excepted here.

pie

[Yokozuna59]

The thing is that it's working fine for matching stuff, but I keep getting parser and laxer errors, which is why I commented them out. Here is an example.

https://github.com/Yokozuna59/mermaid-lint/blob/2e5a085cfda71fe60dfa37cc66ff7a1315c459c6/packages/mermaid-parser/tests/common/accessible.test.ts#L285_L286

[Yokozuna59]

After #1162 has been merged, how to actually use it as a rule instead of a value?

I'm trying to create a fragment rule so that could have either NEWLINE+ or EOF, but it keep throwing errors. I tried different ways to implement it but non of them worked.

[Yokozuna59]

I tried using the suggested code above, but some test cases were failing. Anyway, I used this approach and it's working fine:

fragment EOL returns string: NEWLINE+ | EOF;

[Yokozuna59]

@msujew Sorry to disturb you again, but after removing the overridden lexer, errors were thrown everywhere; it appears that any rule ending with EOF would return undefined instead of the value of the rule.

This is the grammar: https://github.com/mermaid-js/mermaid/blob/0c574335676b2ba7d5af3d3ac4c6dd1975c1056b/packages/parser/src/language/common/common.langium#L7-L9
And this is code example: https://github.com/mermaid-js/mermaid/blob/0c574335676b2ba7d5af3d3ac4c6dd1975c1056b/packages/parser/tests/pie.test.ts#L37

This also occurred if the type is array. It would return the variable as undefined not each elements in the list.

Should I file an issue?

[msujew]

It looks like the EOF token generates an invalid CST node, which then prevents the property assignment from the fragment rule:

I'll probably have time to write a fix on Monday and release a bugfix version.

[msujew]

I've created an issue for this: #1262

[msujew]

@Yokozuna59 This has been fixed in release 2.1.1 of Langium 👍

[Yokozuna59]

Another issue came across:

The issue only occurred when trying to run the language server; it's generates AST correctly without any issue. Test cases that are run in node (vitest) are passing.

But any grammar that has pattern regex that matches everything break the language server, e.g., here is JOURNEY_TASK_TITLE pattern:

/([^\n\r:]+?(?=%%)|[^\n\r:]+)/

EDIT: the pattern is overridden in TokenBuilder.

[msujew]

I see, we have a minor issue in our default token builder. See #1276. You can easily work around it by manually removing the EOF token from the array returned by the token builder.

[msujew]

A fix for this has been released in [email protected]