Can token ids be passed through regex?

[g-eoj]

I can get token ids to be recognized by guided grammar - I have not been able to with guided regex so far. Is it just not possible?

I've tried the byte marker approach from here, in Python:

llguidance/docs/special_tokens.md

Line 15 in b5b8b64

Byte FF is chosen as a marker because it is not a valid UTF-8 byte, so it should not normally

[mmoskal]

It's not possible to just use a regex. Instead use a lark grammar like this:

start: ( <[1234]> | /myregex/ ) <[38383]>

more docs