I could not find anything about `Context-dependent lexing`

[mingodad]

Reading the documentation it says that Context-dependent lexing is provided by rex but I could not find any description about it or how to use it.

Is it somehow like LEX/FLEX states ?

[GuntherRademacher]

Well, the documentation has this, sorry if it is not clear enough:

Context dependent tokenization

REx automatically does context dependent lexing. This means that for each state of the parser, a specific lexer is generated that recognizes only the tokens that are valid continuations expected by the parser in this particlar state.

This eliminates the need for manually setting up rules for different lexical states, as it is sometimes supported in other parser generators. Rather the approach taken by REx exploits the parser's context information to pre-select the token set allowed in a given situation.

At runtime, lexer operation is driven by the parser. The parser sends a request to the lexer for the next token and passes the information about allowable token set. The lexer then responds with an expected token out out this set, or with an error, if none can be found.

Yes, it somehow is like lexer states, but it is all done automatically. Each invocation of the lexer only recognizes the tokens that can subsequently be processed by the parser.

There is a command line option to suppress this feature: -contextindependentlexer.

[mingodad]

Thank you !
I didn't found/remember seem it and did a search for Context-dependent lexing and found nothing maybe would be a good idea to have the same description in both places to be able to search and find it.