Add LexKind

[ratmice]

This is a follow up to PR #483 we now have a number of ways in which RegexOptions can be used to alter the lrlex format primarily by modifying the interpretation of regular expressions.

In that PR it had been mentioned doing follow up work adding a LexKind similar to YaccKind

One thing I have been thinking about which is kind of tangential but related to this is issue #349 which delays the time that YaccKind is known until parser reading time. What seems like it should be a trivial refactor, but also a breaking change. I think it ends up a little bit less trivial than I think it first appears since it introduces some new kinds of propositions for directives that current directives doesn't require and the format parsing code has no current way to enforce.

1. YaccKind affects other directives in the file format so presumably it needs to be first before other directives
2. It should also probably be unique in that we don't want multiple *Kinds.

Thus my thoughts have been that it would be good to try and come to some kind of conclusion on #349 as we introduce a LexKind, just to either avoid introducing any new code that could require this same delicate refactoring, or close #349 abandoning the idea of having self describing formats.

Anyhow I'll post some more new thoughts related to that other issue where they belong.

[ltratt]

I think a couple of things are in our favour here:

1. %yacckind would only make sense for Yacc inputs. So we already know that we're dealing with a Yacc-ish format. If we ever support non-Yacc formats, they'll go down a different codepath.
2. We could easily say "%yacckind, if it exists, must be the first declaration in the file. If it exists and is not the first declaration, an error is raised".

Would those be enough to enable progress?

[ratmice]

I put a bunch more comments in #349 some of it overlaps the big issue is backwards compatibility, we currently need to specify the YaccKind not through the builder, but for YaccGrammar::new so there is kind of a chicken-egg problem in my comment over there I tried to get around that by introducing a new variant YaccKind::SelfDescribing. I think that is the biggest impediment to progress.

[ratmice]

So, implementing the grmtools section I have noticed that regex::RegexBuilder is not uniform about whether an option defaults to true or false.

In #403 I had originally proposed appending a boolean allow_wholeline_comments true
In #403 (comment) we wanted to dropped the need for the bool trailer.

Unfortunately this makes it difficult to say unicode false. Because it is a value you would want to disable rather than enable.

My inclination is to prefix false flags with ! so !unicode basically the same way as https://github.com/softdevteam/grmtools/blob/master/lrpar/cttests/src/regex_opt.test it at least seems like a slightly simpler implementation than having an optional trailing boolean value?

In my initial pass through implementing it I've taken that approach anyways, but it isn't too difficult to try another approach if we would rather do something else.

[ratmice]

So I am wondering if we should close this now,
What I failed to notice is that there is a LexerKind it is reexported from CTLexerBuilder::LexerKind to top-level.

I looked into adding lexerkind field to the %grmtools section, but the circumstances between YaccParser,
and LRNonStreamingLexerDef are significantly different.

In particular, CTParserBuilder doesn't need to know yacckind it just passes it along.
Whereas CTLexerBuilder looks at the lexerkind to figure out the concrete LRNonStreamingLexerDef in order to build a dyn LexerDef. Thus we actually need to know the lexerkind before we get to the current %grmtools section parsing code.

In theory it would be possible to bring the grmtools section parsing code outside of LRNonStreamingLexerDef.
But it would require quite a bit of reorganization where adding yacckind kind of fit within the current construction process.
Thoughts?

[ratmice]

So, I think I'm going to take one more look at making the %grmtools{lexerkind LRNonStreamingLexer} a valid option.

My plan is to essentially parse the lexer's %grmtools section twice, the first time from within CTLexerBuilder,
just pulls out lexerkind. Subsequently it passes the complete file including the %grmtools section into the LRNonStreamingLexer.

The first time is so we can figure out the concrete type, and the second case is to pass options to that specific lexer kind.

There is some mild concern that the grmtools section is currently very context sensitive in that if you don't know all the keys/values it may be difficult to differentiate a key from a value, but since CTLexerBuilder, and all the concrete LexerKind implementations are not like a plugin system taking arbitrary lexer kinds, but concrete types coming from the lrlex crate perhaps that is not an issue.

[ltratt]

There is some mild concern that the grmtools section is currently very context sensitive in that if you don't know all the keys/values it may be difficult to differentiate a key from a value, but since CTLexerBuilder, and all the concrete LexerKind implementations are not like a plugin system taking arbitrary lexer kinds, but concrete types coming from the lrlex crate perhaps that is not an issue.

I tend to agree.

[ratmice]

Okay, well I think the best argument for changing it is for making it possible to write a grammar that includes parsing that section. I would gravitate I guess to just including a comma separator between key-val sequences.

Here is something largely in the spirit of what we accept, but requiring comma between pairs of flags and key/value pairs.
With optional trailing comma (I have no preference regarding). While still being pretty simple.

%grmtools{case_insensitive}
%%
%grmtools 'MAGIC'
! '!'
[A-Z][A-Z_]* 'IDENT'
[0-9]+ 'NUM'
, ','
\{ '{'
\} '}'
\( '('
\) ')'
[\n\t\ ] ;

Edit: now including '::' separated paths in value position.

%grmtools{yacckind Original(NoAction)}
%token MAGIC IDENT NUM
%%
start: MAGIC '{' contents '}';
contents: %empty | val_seq comma_opt;
val_seq: valbind | val_seq ',' valbind;
path: IDENT | IDENT '::' IDENT;
valbind: IDENT val | '!' IDENT;
val: %empty | path | NUM | path '(' path ')';
comma_opt: %empty | ',';

that allows all these %grmtools{x, !y, a b, c d(e),}, %grmtools{}, %grmtools{x}
but avoids things we don't currently accept like %grmtools{!x y, !x y(z), a(b) c,}, %grmtools{,}
while allowing for simple key comparison, and differentiating keys from values...

Anyhow, curious whether that seems acceptable?

[ratmice]

This one was fixed by initially by #551 completed in #556