How is Chevrotain used in this project?

[bd82]

Hello.

I am the author of Chevrotain and I was wondering
if you could provide details on how Chevrotain is used in this project?
and If you have any feedback on Chevrotain?

It seems this is quite a complex use case:

• I understand you are generating a Chevrotain Parser from an XText/Antlr like grammar.
• I also see quite a-lot of overrides/changes to the built-in Chevrotain logic, for example: the CST creation logic.
• and even a wrapper that is used to "expose" protected Chevrotain.Parser methods.
  • Which is a little strange as protected is a design time only construct and a runtime based workaround
    may affect runtime performance.

Cheers.
Shahar.

[msujew]

Hi @bd82,

first off, thank you for your work on Chevrotain. It has made building the parser generator and debugging the generated parsers a breeze. While Langium is quite a complex use case, Chevrotain is able to handle everything really well (while still being really fast).

Regarding your observations:

• Right, we are generating a parser that is based on our own internal LangiumParser which in turn uses the Chevrotain EmbeddedActionsParser. We also use the default Chevrotain lexer implementation currently, but we plan to add additional API to easily create custom lexers, e.g. with different lexer states. I don't think that we use anything else. I experimented with the auto completion feature, but it didn't fit our use case, so I rolled my own. You planned to deprecate/rewrite that either way, right?
• Our CST needed a bit more information regarding which grammar feature was used to create it, among other things. This is used almost everywhere in the default implementations of the different LSP services. Especially the completion stuff makes heavy use of that.
• Regarding the exposed protected parser methods: We used to inherit the LangiumParser directly from the EmbeddedActionsParser but we didn't want to expose everything from the EmbeddedActionsParser. That's why we came up with the wrapper. So far we haven't been concerned with runtime performance, but we will keep that in mind.

Feedback:
I think there are a few things which we would like to see improved. While the error recovery itself is working really well, the parser errors produced by that often end up at unexpected places. Currently we simply take the token property and display the parser error at that exact position. In our hello world example this leads to the following parser error:

However, one would expect the parser error to be displayed at the end of the line, where the actual ! token is missing. Do you have a recommendation how to fix such issues from our side?

Another thing which I would like to see is some sort of parser mode which we could use to identify any parser rule that could be invoked at a certain place. For code completion, we just run the parser on the whole file and find out which CST node is at the specified completion offset. Then we invoke some sort of rule interpreter to heuristically determine which rule could also have been invoked at that place. It works quite well, but Chevrotain support for that use case would be superb. The main issue is that it needs to run on broken input.

@spoenemann any other feedback from your side?

[bd82]

Content Assist

The original implementation built-in Chevrotain ran a different and interpreted parser engine.
But that meant it was not fully aligned with way the "real" parser engine behaved.
That is why I am thinking about removing that feature.

The last time I needed to implement content assist logic I've done so using a CST visitor with additional context.
More details here: https://github.com/SAP/xml-tools/tree/master/packages/content-assist

[msujew]

Interesting, that's the same approach Xtext took with ANTLR. We also thought about it, but that would have required to generate an additional content-assist parser. Instead we took a different route, mixing the parser engine of Chevrotain with an interpreted approach that tries interpreting the last few encountered CST nodes.

On a slightly different node: Do you think it makes sense to create the parsers completely in memory? Generating the parser code was just the fastest approach to everything. However, with one of our grammars in memory, we should be able to create an entire parser that works just like the generated one, right?

[bd82]

Thanks for the in-depth overview.

Lexer States

Do you mean something like this?

• https://chevrotain.io/docs/features/lexer_modes.html

Error Positions

I am unable to reproduce the issue you are describing on the JSON example on the playground.

• https://chevrotain.io/playground/

Nor when I remove the semicolon from the tinyC example test:

• https://github.com/Chevrotain/chevrotain/blob/master/examples/grammars/tinyc/tinyc_spec.js#L9

Perhaps this is due to a combination with error recovery logic?
Or an off by one offset calculation error?

Can you reproduce it in one of the examples sub-packages in chevrotain repo, e.g via a test with failing assertions?

Normally an error would appear one token after when something is missing.
The parser normally not whitespace aware, but you could traverse "backwards" from the error token position to
the "true" error position, e.g:

• Missing comma in naïve Chevrotain Playground --> error on next token.
  

• Missing comma in smarter Jetbrains IDEA --> error on whitespace before the error position.
  

[msujew]

Alright, the your lexer modes sound exactly what we are looking for. That makes things a lot easier, now we just need to think about how to integrate that nicely into our framework.

Interestingly enough, I'm also unable to reproduce the issue with the playground. I used this gist and the following as input:

Person x
Hello x
Person y

The error marker appears at Person y as expected. Looks like I need to dive deeper into why that happens. I also think it's an issue of our own error recovery.

Either way thanks for your input 👍

[msujew]

On second thought, our error marker is also working correctly if we find something after the missing !:

However, using just the following input:

Person x
Hello x

Will lead to no error marker in the playground. There seems to be something fishy going on with EOLs.

[msujew]

Even more interesting, if I add a new line at the end of the file, the error marker is showing up as expected as well: