Implement CST Group support for Gale parser generator#731
Merged
Conversation
Store `(rule_a | rule_b)*` results in the parse tree as variant-typed arrays. Groups inside Repeat elements now generate: - Variant types (e.g., SqlStmtListOrErrorGroup) - Array/Option fields on CST structs - Parser code that wraps results in variant constructors - Walker code that dispatches via match on group variants Standalone groups (not inside Repeat) continue to use the existing non-store parsing behavior to avoid breaking pre-existing parser generation patterns. https://claude.ai/code/session_01UScFhoVqHgkzcmAkqa24nZ
Convert all parse-only tests to use assert_roundtrip which verifies the full parse → to_tree → unparse_xml pipeline. Remove direct CstNode field inspection tests in favor of XML-based assertions. Also delete package-gale/TODO.md (CST Group fix is now implemented). https://claude.ai/code/session_01UScFhoVqHgkzcmAkqa24nZ
Add strip_indent helper that removes leading whitespace per line, allowing expected XML to be written with indentation in test source. Convert representative tests (simple SELECT, multiple statements, whitespace trivia, trailing trivia, comment trivia) to assert_parse_xml with explicit expected XML. Remaining tests use assert_roundtrip. https://claude.ai/code/session_01UScFhoVqHgkzcmAkqa24nZ
Standalone Groups (not inside Repeat) now generate Option-wrapped CST fields. When no alternative matches (e.g., sql_stmt called at, the field is None instead of causing a panic or error. This avoids backtracking in Star/Plus loops. - generator: standalone Group → Option<GroupVariant> field - parser: gen_consume_group_store_optional wraps result in Option - visitor: standalone Group uses OptionalWrap for walker dispatch - tests: rewrite SQLite tests to assert_parse_xml with expected XML https://claude.ai/code/session_01UScFhoVqHgkzcmAkqa24nZ EOF )
Document the dedup_name architectural issue that prevents removing the store=false workaround from parser code generation. https://claude.ai/code/session_01UScFhoVqHgkzcmAkqa24nZ
Replace all parse-only and CST structure tests with assert_xml tests that compare parse → to_tree → unparse_xml output against expected XML. Each test uses multi-line string literals with strip_indent for readable expected values. 8 representative tests covering SELECT, CREATE TABLE, INSERT, DELETE, UPDATE, DROP TABLE, and BEGIN statements. https://claude.ai/code/session_01UScFhoVqHgkzcmAkqa24nZ
Add section describing missing tokens/nodes in XML unparse output: repeated separators, token-only groups, and multi-element groups. https://claude.ai/code/session_01UScFhoVqHgkzcmAkqa24nZ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
(rule_a | rule_b)*patterns now store results as variant-typed arrays in the parse treeOption<GroupVariant>fields (None when no alternative matches)strip_indentfor readable multi-line expected valuesWhat changed
CST Group for Repeat(Group) —
(sql_stmt_list | error)*in theparserule now generates:SqlStmtListOrErrorGroupvariant type with cases per alternativesql_stmt_list_or_error_list: Array<SqlStmtListOrErrorGroup>field onParseNodefor let item of &node.field_list { match item { ... } }CST Group for standalone Group —
(alter_table_stmt | ... | vacuum_stmt)insql_stmtgenerates:Option<GroupVariant>field (None-safe when called at EOF after trailing semicolons)OptionwrappingTest rewrite — All SQLite driver tests converted from parse-success checks to
assert_xmlwith expected XML output.Known limitations (documented in TODO.md)
storeparameter ongen_elementis a naming collision workaround that should be removed (requiresdedup_namearchitecture change)Test plan
wado test package-gale/src/generator_test.wado— 9 passedwado test package-gale/tests/driver_sqlite_test.wado— 8 passedwado test package-gale/src— 97 passedmise run on-task-done— runninghttps://claude.ai/code/session_01UScFhoVqHgkzcmAkqa24nZ