How do I match any token in the grammar?

[kaby76]

A lexer rule allows one to match any character in the input stream, e.g., VALUE: /./+. How do I match any token type in a parser rule of the grammar? It seems the only way to match anything is to create a parser rule any : TOKEN1 | TOKEN2 | TOKEN3 | ... | LASTTOKENINGRAMMAR that laboriously enumerates every token in the grammar.

And, while I'm at it, how do I specify a token set with operations such as minus in a parser rule? E.g., ". - TOKEN1" (all token types except TOKEN1, used in the Java Language Specification), or "~(TOKEN1 | TOKEN2)" (all token types except TOKEN1 and TOKEN2)?

[erezsh]

It seems the only way to match anything is to create a parser rule any

Yes, I don't think there's another way. Can you explain why you need such a thing?

all token types except TOKEN1

Lark doesn't support it. Unless you enumerate every token except the ones you want to omit.

[kaby76]

Token set operators and the "catch-all '.' token set" come in handy for translation from other parser generators. Some people define "error rules", which use the token type catch-all ".". I use "start: .*;" in Antlr to just hand me a list of tokens for debugging. And, sometimes I need to explore moving a parser rule to the lexer or vice versa (and as I said, lexer rules have the catch-all "." to denote any character). It's not really that much of a problem to work around the issue because I can express the token set by enumeration. During a port of the Antlr C grammar to Lark syntax, there was a rule or two that required a token set operator, which I fudged just so I can see if the grammar would work in Lark--it does, more or less!

[erezsh]

There is one other possible approach, which is to define an ANY token:

ANY.-1000: /.+/

We give it very low priority so it will always be picked last.

You can also use negative regex prediction to exclude certain matches.

YMMV.