[Feature request] Surface in CST if _any_ nodes are recovered nodes

[matthew-dean]

Use case: when recoveryEnabled is set to true, it's hard to tell from the returned CST if any of the nodes are recovered nodes, without iterating through nodes. In other words, were any tokens "dropped" or skipped.

The presence of errors is not sufficient, because I'm still consuming (error) tokens that will still result in a complete CST (and resulting AST), and I (incorrectly) assumed that recoveredNode would be equal to true if any of the nodes within were recovered nodes, but apparently that's not the case?

[matthew-dean]

Let me phrase this a different way, since I may not fully understand recoveredNode. I want to know if the CST has essentially saved all tokens, such that, if I needed to, I could re-create the entire source. Some Chevrotain errors, and some errors I'm adding to the stack, are valid errors, errors that should be surfaced to the user, but are recoverable. That is, an AST can still be built that produces the same text output, as all nodes are saved in the CST.

Some errors, however, indicate that parsing simply failed entirely. For example: NotAllInputParsedException. An AST cannot be formed from that CST, therefore I need a hard stop.

So I'm looking for an indicator of the following: was all input saved to the CST? Or were some tokens not saved?

[bd82]

So I'm looking for an indicator of the following: was all input saved to the CST? Or were some tokens not saved?

Hmm, recoveredNode indicates there was re-sync recovery in this node.
But there may be other types of recovery, e.g single token insertion or single token deletion.
And as you mentioned there may be other edge cases, such as "NotAllInputParsed"

[bd82]

Perhaps you should try to examine this condition in a more "direct" manner by overloading consumeInternal or consumeToken and flag every token index which has been consumed and then inspect that all tokens have been consumed?

You will have to to inspect the source code to be certain this condition is indeed correct, meaning that consumeToken is only called when a token is actually consumed.

• A quick check found that SKIP_TOKEN also uses `consumeToken
• So more advanced logic may be needed to figure this out.