How to recover from parser error?

[durianwaffle]

In our grammar, if there is parser error such as a random string without semicolon, the validator will fail for the current document.
Any solution on that?

Can I define a rule to parse the random strings in the grammar? If so, how should define this rule?

[msujew]

Hey @durianwaffle,

without diving to deep into parsing theory: There are some kinds of parser errors that are hard to recover from, due to the way how recursive descent parsers work. You're encountering one because the parser fails to perform its lookahead function, which in turn prevents the parser from identifying which rule to parse.

Usually, the parser then attempts to skip tokens to recover from this error, but this doesn't always work.

Can I define a rule to parse the random strings in the grammar? If so, how should define this rule?

I don't think that's possible, at least not to my knowledge. I would assume that even if you were able to define such a rule, you would likely run into unexpected side effects.

[durianwaffle]

Hi Mark, why the parser fails to perform its lookahead function?

And do you think it's possible that I parse the documents before parser error and after parser error individually to validator?

[durianwaffle]

I found that after this error, which is hard to recover from, Langium fails to parse the code below.
And the validator will not execute at all on that document.

Do you know what're the functions that I can use to validate the document that doesn't contain those errors?

[durianwaffle]

For example, in Xtext, if there are errors that can't recover from, the validator will show red squiggle lines for all the code below that error by default. But the tokenizer and parser will fail to parse the code after that error same as in Langium.

Can langium also implement that?

[msujew]

why the parser fails to perform its lookahead function?

Because the lookahead is instantly aborted once a token is identified that doesn't fit any of the given alternatives. See also here.

And do you think it's possible that I parse the documents before parser error and after parser error individually to validator?

No, because the entry point of the grammar already appeared, and so it will just result in more errors for all non-trivial grammars.

Do you know what're the functions that I can use to validate the document that doesn't contain those errors?

I guess you would need to look at the errors in the document? I'm not entirely sure.

Can langium also implement that?

Maybe? I need to take a look at what Chevrotain provides us as error messages to figure out whether parsing succeeded until the end of the document. If something like that isn't available, maybe we figure it out by looking at the last parsed token in relation to the input text.

[durianwaffle]

Do you think it's possible to cache AST and those diagnostics? I found that with this error. langium can't generate AST.

[msujew]

Well theoretically yes, but you will start to run into offset errors very quickly - every change modifies the offsets in the CST, which then cascades into all LSP services (including diagnostics). If we cache the previous AST/CST, the editor would start desyncing from what we have stored in memory.

I'm well aware that this results in essentially a complete parser failure, but we're at a technical limit here, as error recovery in LL parsers simply have some limits that we cannot work around within the framework.

[durianwaffle]

Okay, @msujew do you why I can see all the errors in message 'textDocument/publishDiagnostics' from worker but there is no validation error on screen?

[msujew]

Without further information, I have no clue. Do you have a reproducible example?