parser-state aware lexer #27
Replies: 2 comments 6 replies
-
|
I recommend reading https://www-users.cse.umn.edu/~evw/pubs/vanwyk07gpce/vanwyk07gpce.pdf. While this paper discusses the problem of integrating two languages together, the problem of parser-state aware lexers occur in some languages we already try to parse (e.g., Python3). For a list of candidates that fit this description, look for grammars that have Antlr lexer modes. This paper was selected from tree-sitter web page https://tree-sitter.github.io/tree-sitter/#underlying-research. Presumably, tree-sitter implements parser-state aware lexers. I would like to see an option added to allow the lexing strategy to be switched to parser-aware lexing. We can keep the default to the standard Antlr algorithm: (1) Antlr picks the rule with the maximum-matching input; (2) If two or more rules match, Antlr picks the first rule. |
Beta Was this translation helpful? Give feedback.
-
|
Very interesting paper, thanks for that. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
@kaby76 can you elaborate ?
Beta Was this translation helpful? Give feedback.
All reactions