Skip to content

Commit 9f5553c

Browse files
authored
Merge pull request #1982 from wado-lang/claude/package-gale-import-s-design-tg9l2z
feat(gale): compose `import S;` against the supplied inputs
2 parents e47d83d + dbfb925 commit 9f5553c

751 files changed

Lines changed: 35055 additions & 2396 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

package-gale/AGENTS.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ Dev-cycle essentials for working on Gale, a Wado-native ANTLR4-compatible parser
44

55
- [`antlr4-compatibility.md`](./antlr4-compatibility.md) — the compatibility contract, prediction / codegen design, soundness invariants, descriptor pipeline, and triage.
66
- [`resilient-parser.md`](./resilient-parser.md) — error-resilient parsing and the flat CST.
7+
- [`import.md`](./import.md) — grammar composition: how `import S;` resolves and what a delegate contributes.
78
- [`perf.md`](./perf.md) — performance notes: budget, levers, and measured perf dead-ends.
89
- [`TODO.md`](./TODO.md) — open work.
910
- [`README.md`](./README.md) and [WEP: Gale](../docs/wep-2026-03-02-gale.md) — overall design context.
@@ -22,7 +23,7 @@ Files headed `// Do not edit by hand` are generated. To change one, edit its sou
2223

2324
## Compatibility principle
2425

25-
Gale targets full compatibility with the ANTLR4 `.g4` syntax. The g4 parser must accept any well-formed grammar upstream `antlr4` accepts; a real-world `.g4` that ANTLR4 accepts but Gale rejects is a Gale bug.
26+
Gale targets full compatibility with the ANTLR4 `.g4` syntax. The g4 parser must accept any well-formed grammar upstream `antlr4` accepts; a real-world `.g4` that ANTLR4 accepts but Gale rejects is a Gale bug. The one exception is `import Foo = Bar;`; claim (a) in [`antlr4-compatibility.md`](./antlr4-compatibility.md) carves it out.
2627

2728
- Compatibility is a capability contract, not byte-for-byte output. Parse trees, tokens, and semantics must match; incidental rendering differences that carry no structure may diverge (e.g. the `<EOF>` marker in `toStringTree()`).
2829
- Gale is a superset: it may accept grammars ANTLR4 rejects only when the meaning is uniquely determined by Gale's language model — never an invented behavior. When accepting would require guessing, reject loudly. Examples: `.`/`~X`-led left-recursive suffixes, and a lexer `mode` inside a combined `grammar` (ANTLR4 allows modes only in a `lexer grammar`; a combined grammar already bundles a lexer, so it desugars unambiguously — still rejected in a `parser grammar`).

package-gale/README.md

Lines changed: 81 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,8 @@ with no runtime to install and no version to keep in sync.
77

88
The `.g4` format is ANTLR4's; for the full grammar language, see ANTLR4's
99
[documentation](https://github.com/antlr/antlr4/tree/master/doc). Gale accepts
10-
every grammar ANTLR4 accepts — and a few it rejects, where the meaning is
11-
unambiguous (see [Design](#design)).
10+
every grammar ANTLR4 accepts bar one — and a few it rejects, where the meaning
11+
is unambiguous (see [Design](#design)).
1212

1313
## Design
1414

@@ -22,6 +22,8 @@ with no remaining choice — e.g. a `.`- or `~X`-led left-recursive suffix like
2222
`mode` inside a combined `grammar`, which ANTLR4 restricts to a `lexer grammar`
2323
but which is unambiguous since a combined grammar already bundles a lexer. Where
2424
the meaning is not uniquely determined, Gale rejects loudly rather than guessing.
25+
That costs it one grammar ANTLR4 accepts, `import Foo = Bar;`, carved out under
26+
claim (a) in [`antlr4-compatibility.md`](./antlr4-compatibility.md).
2527

2628
Self-contained output, no version drift. Gale inlines its entire runtime into
2729
every generated parser. There is no `gale-runtime` package to keep aligned with
@@ -436,6 +438,83 @@ complete real examples, and
436438
[WEP: Gale Highlight Query](../docs/wep-2026-07-12-gale-highlight-query.md) for
437439
the design.
438440

441+
### What the context tier buys
442+
443+
The override form is the reason to highlight from a parse rather than a token
444+
stream. [`example/`](./example) carries a three-grammar demo of it.
445+
[`MiniHtml.g4`](./example/MiniHtml.g4) imports
446+
[`MiniCss.g4`](./example/MiniCss.g4) and [`MiniJs.g4`](./example/MiniJs.g4), so
447+
one `use` builds a single recognizer: one lexer with three modes, one parser,
448+
one tree. The three `.scm` queries ride in beside the grammars and are
449+
concatenated, so each language keeps its own.
450+
451+
Embedding is the vehicle. The point is one line of
452+
[`MiniJs.highlights.scm`](./example/MiniJs.highlights.scm):
453+
454+
```scheme
455+
(params (JS_IDENT) @variable.parameter)
456+
```
457+
458+
`arrow` and `group` both open with `(`, so whether an identifier inside the
459+
parentheses is a parameter is settled by the `=>` after the closing paren:
460+
461+
```js
462+
let add = (a, b = (1 + 2)) => a + b; // a, b are parameters
463+
let one = ((a)); // a is a variable
464+
```
465+
466+
Two things stand between a highlighter and that answer. A lexer classifies the
467+
identifier when it reads it, before the deciding token exists, and no
468+
mode-stack state brings it closer. A regex highlighter does look ahead, and
469+
`\(([^)]*)\)\s*=>` would settle a flat parameter list. But a default value nests
470+
parentheses, so `[^)]*` stops at the inner `)` and misses the `=>`. Finding that
471+
closing paren means matching brackets, which no regular expression does.
472+
473+
The parser matches them, and the query reads the answer off the rule stack.
474+
`MiniCss.highlights.scm` does the same in the small, though a stateful lexer
475+
could keep up there: one `CSS_IDENT` token becomes a selector, a property, or a
476+
value by where the parse put it.
477+
478+
What you pay for it is that a context capture fires only where the enclosing
479+
rule parsed. Half-write a declaration as `a { b: }` and `b` falls back to no
480+
class, because nothing placed it under `declaration`. Defaults still apply and
481+
the text is preserved, so the file stays readable while you type; only the
482+
contextual half is lost.
483+
484+
That also makes a grammar bug look like this limitation. If input the language
485+
allows loses its classes, the grammar rejected it. Check there first.
486+
487+
The whole page's highlighted HTML is pinned in
488+
[`example/highlight_test.wado`](./example/highlight_test.wado):
489+
490+
```sh
491+
wado test package-gale/example/highlight_test.wado
492+
```
493+
494+
### Composing a grammar for an embedded language
495+
496+
The example is also what `import` looks like for an embedded language. It needs
497+
no feature beyond composition itself; two conventions carry it:
498+
499+
- **The embedded grammar's lexer rules live in a mode of its own** (`mode CSS;`).
500+
A composite has one lexer, so without a mode `MiniHtml`'s `TEXT` would
501+
swallow a stylesheet whole.
502+
- **Its token names are prefixed** (`CSS_IDENT`, `JS_IDENT`). A composite has
503+
one token space, where the first rule of a given name wins and the rest are
504+
dropped.
505+
- **Its parser rules name their tokens rather than spelling them**
506+
(`CSS_COLON`, not `':'`). A parser literal aliases only to a `DEFAULT_MODE`
507+
rule, so spelling one mints a token the delegate's mode never produces.
508+
509+
The host owns the boundaries and nothing else: `MiniHtml.g4` declares
510+
`mode CSS` itself, holding just the `</style>` that leaves it, and composition
511+
unifies the two `mode CSS` declarations by name. So the host says where each
512+
language begins and ends, and neither delegate names its host.
513+
514+
The trade is that an embedded grammar no longer works on its own, since its
515+
rules sit in a mode nothing enters. The full contract, and what a grammar
516+
usable both ways would need, are in [`import.md`](./import.md).
517+
439518
## Compatibility and further reading
440519

441520
Gale targets the full ANTLR4 `.g4` grammar syntax, plus the small superset

0 commit comments

Comments
 (0)