How to use Lexer Modes in langium service?

[yuyi919]

It can be used to tokenize some markup language

references:
https://chevrotain.io/docs/features/lexer_modes.html

[msujew]

Hi @yuyiRai thank you for your question.

I just wrote up a long-winded answer, until I noticed that our API currently does not allow to construct a multi mode lexer definition...

You can theoretically assign push/pop modes to different token types like this by overriding the DefaultTokenBuilder class. In particular, you can override the buildKeywordToken method.

However, there's no way to create a lexer definition like required for Chevrotains lexer because the return type of the buildTokens method is an array of tokens. I created a new issue for this #320

[luiz290788]

Hi @msujew, I saw that the issue you create is already solved, but the solution is still not clear to me. Would we still need to override the DefaultTokenBuilder?

[msujew]

Yes, you need to override the DefaultTokenBuilder and return a IMultiModeLexerDefinition. We make use of this feature in a few closed source projects.

[msujew]

We're currently looking into making it easier to create multi mode lexer, though it's fairly low on our priority list. See also this discussion for more information.

[luiz290788]

Thanks, it is good to know that this has been validated internally.