Streamline long term maintenance of lexer/parser code generators

[alecthomas]

I have some concerns with the long term maintenance of both the lexer and parser code generators which we should solve (or at least, not make worse) before getting these changes into master.

Problem 1: divergence

My first concern is that as the runtime code evolves, it diverges from the code generators. This has already occurred with the lexer, where the generated code does not handle lookahead. There will need to be some tests to detect this somehow. For the parser, enumerating the nodes and ensuring they're all handled by the generator (somehow) might work.

• Conformance tests for lexer
• Conformance tests for parser

Problem 2: ergonomics

There are also some ergonomic issues with generating the code. Specifically, having to have an implementation of the runtime code lying around as the "source of truth".

Proposal: serialisable form for both lexers and parse trees

• Serialisable lexer
• Create participle gen lexer
• Serialisable parser
• Create participle gen parser

One solution to this is for the code generators to be decoupled completely from the code. The lexer and parser would be extended to be JSON marshallable and the code generators would become standalone binaries that could ingest this serialised form and output code. This might be non-trivial for the parser because it is tightly coupled to reflection - TBD.

Another option would be to make standalone binaries that parse the Go code directly, making the Go AST and compile-time type system the intermediate form. This would be much more complicated though.

[alecthomas]

This has been implement for the lexer in bfe7c69.

[klondikedragon]

@alecthomas - This is a nice separation to have!

I've tested the update in bfe7c69, it looks like there is one more issue in the code generated from the serialized JSON definition of the lexer. Imagine that the lexer codegen was invoked using something like:

~/go/bin/participle gen lexer --name GeneratedLexer lexer < internal/lexer/lexer.json | gofmt > internal/lexer/lexer_generated.go

And imagine that the lexer definition had included some rules that looked like:

	"AllStrings": {
		{Name: `DoubleQuoteStringBegin`, Pattern: `"`, Action: lexer.Push(`DoubleQuoteString`)},
		{Name: `SingleQuoteStringBegin`, Pattern: `'`, Action: lexer.Push(`SingleQuoteString`)},
		{Name: `BacktickStringBegin`, Pattern: "`", Action: lexer.Push(`BacktickString`)},
	},

The codegen template that generates code to push a new lexer state intends to refer to the state struct name that includes the name specified on the command line (e.g., GeneratedLexer in the above example)
bfe7c69#diff-5119a11cbfb79273b05b2942dfb351ab4bf736ca7cf91d773f360e75bcb07eb4R173

However, what the template generates instead is to put the state struct name using the name of the rule instead of the top level struct name. For example, it generates

	switch state.name {
	case "AllStrings":
		if match := matchDoubleQuoteStringBegin(l.s, l.p); match[1] != 0 {
			sym = -22
			groups = match[:]
			l.states = append(l.states, lexerDoubleQuoteStringBeginState{name: "DoubleQuoteString"})
		} else if match := matchSingleQuoteStringBegin(l.s, l.p); match[1] != 0 {
			sym = -23
			groups = match[:]
			l.states = append(l.states, lexerSingleQuoteStringBeginState{name: "SingleQuoteString"})
		} else if match := matchBacktickStringBegin(l.s, l.p); match[1] != 0 {
			sym = -24
			groups = match[:]
			l.states = append(l.states, lexerBacktickStringBeginState{name: "BacktickString"})
		}

In all cases in this example, the proper name of the struct would have been lexerGeneratedLexerState.

Manually changing this the generated code compiles as before.

I've also been debugging why the generated lexer isn't producing 100% accurate results vs the reflection lexer. It seems it may not be handling the special lookahead word boundary \b quite the same, and in another case it may not be greedily matching the same in a regexp with a complex set of nested non-matching groups. I have to have some simple rules that highlight the gaps in the codegen lexer shortly.

[klondikedragon]

@alecthomas - here's a one-liner fix so the generated lexer code with multiple states will now compile: #268

I'm working on some tests to add that will highlight some of the issues I've found in the generated lexer and maybe fixes for them if not too complicated.

[alecthomas]

Magic, thanks!

[alecthomas]

re. your last point, yes the generated lexer has diverged in terms of functionality already. I need to come up with a way for tests to pick this divergence up in future.

[klondikedragon]

@alecthomas - Check out #269 which is my attempt to have the current lexer test suite also cover generated lexers, and to make it easy to add new test cases that test both the reflection lexer and generated lexer without much duplicate code. The generated lexer is so much faster than the reflection lexer, I'm hoping this will make it easier to maintain at least a subset of features in the generated lexer. Participle has unparalleled developer ergonomics, and if we can get generated lexer & parser production-ready, it will have leading performance as well!

[alecthomas]

While I very much appreciate the effort, and I definitely don't want to discourage you from further contributions, I would prefer something a bit simpler to maintain, and a bit more targeted. I'm imagining something like a single "specification" lexer that checks against a spec, and cross-checks a generated lexer against the runtime lexer. I'm happy to flesh this out further if you are interested in contributing it.

[klondikedragon]

If I understand what you're envisioning correctly, it would be some single set of lexer rules that would try to exercise all of the different features of the lexer, and then different unit tests that would use that one generated lexer to try and cover all parts of the lexer? It seems possible but to me harder to maintain than the proposed approach, as it would require a different set of unit tests than what is currently there (current unit tests start from different lexers with one specific element its testing).

Perhaps I'm missing how a single generated test lexer might be easier to maintain? Right now running the Makefile will immediately regenerate all of the JSON files and all of the generated test lexers, and the generated lexers also match the current unit tests so they can stay in sync as the unit tests change together. If there were to be one specification lexer that tried to cover all parts of the lexer it seems that would duplicate a lot of the work of the current test suite and would have to be maintained as well as the lexer evolves.

Having the goal of easy maintenance makes a lot of sense to keep time burden of any changes low. I'm interested in helping and open to ideas how to help in that direction!

[alecthomas]

Right now the lexer tests are pretty ad-hoc - it's basically arbitrary if a particular feature is exercised or not. The new lexer would be a more systematic approach to this, and once complete, we could remove overlapping tests from the existing suite. I would envisage that there would be only one test input that tests every feature of the lexer, though that might be a bit hard to debug, so that might not be a great idea. It would need some experimentation.

I think it would also be good if the proposed test system generated the codegenned lexer on the fly for each test run, probably to a temporary directory, so that no manual steps are required when updating the lexer/tests. This would also function as a kind of integration test for the code generation itself.

[alecthomas]

I'll have a bit of a poke around tonight.

[alecthomas]

This is basically what I had in mind. Interestingly, these tests shouldn't attempt to capture validation errors (ie. malformed stateful lexers), those should remain in the existing test suite.