Lexer Grammar vs. Lexer States vs. Catch All Rules vs Keywords

[cdietrich]

hi, is there a way to achive something similar like an terminal ANY_OTHER: .; token would do in Xtext (en- and disabled via lexer modes).
defining that in the grammar seems no to work as chevrotain seems to prefer it over other terminals and keywords.

Error: Errors detected in definition of Lexer:
Token: ->.<- can never be matched.
Because it appears AFTER the Token Type ->TEMPLATE_LITERAL_FULL<-in the lexer's definition.
See https://chevrotain.io/docs/guide/resolving_lexer_errors.html#UNREACHABLE

the only workaround i have found is:

• use fake terminal in grammar https://github.com/cdietrich/langium-template-experiments/blob/2e57b47d56d0b72a501a05adbb45b0d8b329a2b1/src/language/hello-world.langium#L53
• exchange pattern at runtime https://github.com/cdietrich/langium-template-experiments/blob/2e57b47d56d0b72a501a05adbb45b0d8b329a2b1/src/language/CustomTokenBuilder.ts#L64C14-L64C14,
• disable longer alt runtime https://github.com/cdietrich/langium-template-experiments/blob/2e57b47d56d0b72a501a05adbb45b0d8b329a2b1/src/language/CustomTokenBuilder.ts#L52

[msujew]

@cdietrich The workaround you've outlined is probably the way to go. Unfortunately, the Chevrotain lexer <-> Langium Lexer/TokenBuilder interaction experience isn't perfect yet, as Langium tries to infer a lot of information from the grammar (such as LONGER_ALT). This might not work as expected when building more complicated lexers.

Any contribution to make this easier is appreciated :)

[cdietrich]

@msujew have a follow up question: why are whitespace tokens put to the beginning of the token list?

[msujew]

It's a performance optimization, see also here.