Lexing unicode characters

[bendrissou]

Hi,

I am trying to lex unicode characters.

Let's say I have this lexer rule:

\\u[0-9|a-f|A-F][0-9|a-f|A-F][0-9|a-f|A-F][0-9|a-f|A-F] "UNICODE-4"

This rule would only match an encoded unicodes e.g. \u2081, and not a decoded/raw unicode, such as: ₁.

Does grmtools have any support for unicode?

[ltratt]

In this regard, we (probably) just do whatever the regex crate does. It has various mechanisms for matching unicode characters: hopefully one of those works!

[ratmice]

I happened to have something that matches subscripts (not exactly as given in your example, but pretty close)...

It is the familiar [a-zA-Z][a-zA-Z0-9] var decl thing, but allows the first character to be a greek letter and the trailing characters include subscripted numbers...

So it should match input like: λ λ1 λ₁ ℴ ℴ1 ℴ₁ ℬ₁, etc.

%%
[a-zA-Z\p{Greek}][a-zA-Z0-9]* "var"
[a-zA-Z\p{Greek}][a-zA-Z0-9\u207f-\u2089\u2090-\u209c\u1d62-\u1d6a]* "subscripted"
[a-zA-Z\p{Greek}\u1d49c-\u1d59f\u2100-\u214f][a-zA-Z0-9\u207f-\u2089\u2090-\u209c\u1d62-\u1d6a]* "fancy_subscripted"
[ \n\t] ;

I note that the fancy_subscripted one doesn't seem to be matching all the characters I would expect, I'm not exactly sure what is going on with it though 🤦

[ratmice]

apologies, I wanted to say one more thing, which is that because the lrlex format is read as utf8, you can also just use unicode characters directly like so...

%%
∀ "forall"

As such you can modify the comment above with the characters inlined even, something like:
[a-zA-Z\p{Greek}][a-zA-Z0-9ⁿ-₉ₐ-ₜᵢ-ᵪ]* "subscripted2"

[bendrissou]

Thank you for the examples. Also good to know that lrlex is read as utf8. So the regex notation works as in antlr.

[\000-\077]  "OCT-CODE"
[\x00-\xff] "HEX-CODE"
[\u0000-\uffff] "UNICODE-4"
[\u00000000-\uffffffff] "UNICODE-8"

[ltratt]

Sounds like this one has reached a happy resolution -- thanks @ratmice for your input :)

[ratmice]

@ltratt I guess I was curious if this was something you would want to see a grmtools book PR documenting.
Either with the ".lex files are utf8" in the "differences between posix lex" section, or just adding some small examples but for the most part directing people to the regex docs?

Edit: I should mention i'm a bit torn on the idea, since I don't want to just regurgitate a bunch of regex documentation in a way that is bound to become stale, but perhaps it'd be good to have something?

[ltratt]

Documenting this seems like a good idea! There is at least some sort of relationship to the "dialect" of lexing we do w.r.t. #483 too.