Rust-style hashtag strings

[lylythechosenone]

Is there a way to parse r#"..."# strings in pure LALRPOP without a custom lexer? (Of course, parsing this where they always have exactly one # is easy, but parsing a variable but identical number of #s is the real issue.) If not, is there a way to inject a custom lexer just for that, while using the default lexer for everything else?

[dburgener]

What's wrong with using a backreference?

Untested, but something like: r"r(#+)\".*\"(\1)"?

[Pat-Lafon]

I may be completely mistaken, but this reminds me a lot of regular languages versus context-free languages. Regex libraries are generally regular languages with some extra bells and whistles. For counting problems like palindromes and I think this (having equal numbers of # on each side and ignoring inner parts of a different amount), you usually need a context-free grammar.

[Pat-Lafon]

But you can also look how it was originally implemented in Rust. They have a special procedure in the lexer to count the number of hashtags in a row: https://github.com/rust-lang/rust/pull/9674/files#diff-b334b7cea4a1582e38c0544f215bf1e54f29fcce526265be588fed34a4c5912f

[Pat-Lafon]

As I'm thinking about this, if you are willing to limit the max number of # to a small number for practicality, you can just have #, ##, and ### as unique tokens created by the built-in lexer and then have your grammar look for those three cases. A huge hack, but doesn't require too much extra work on your part.

[dburgener]

There's probably some other subtlety I'm missing, but does just making the inner .* non-greedy with .*? fix it? r(#+)\".*?\"(\1) gives the right answer for me to your regex101 image.

[lylythechosenone]

That might actually be it!

[yannham]

@Pat-Lafon back-references usually make you much more powerful than regular languages, you can recognize languages that aren't even context-free . Some modern flavour of regexp are even Turing-complete 🙃