Skip to content
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
a266548
gale: compose `import S;` against the supplied inputs
claude Sep 6, 2026
1db617b
gale: distill the import composition
claude Sep 6, 2026
f7427aa
gale: make a claim's `use` answer for the descriptor's slave grammars
claude Sep 6, 2026
7b980cd
gale: queue the aliased import and per-delegate action language
claude Sep 6, 2026
741c78a
gale: an example that highlights from the parse, not the token stream
claude Sep 6, 2026
9358ddc
gale: compose the highlight example instead of stitching it
claude Sep 6, 2026
dec0250
gale: a grammar name must identify one supplied input
claude Sep 6, 2026
c1ee104
gale: distill the composition and its example
claude Sep 6, 2026
62ebe90
gale: make the example demonstrate what it claims
claude Sep 6, 2026
711bcb8
gale: accept the CSS the example rejected, and stop calling it a limi…
claude Sep 6, 2026
a8ed2bd
gale: every supplied grammar reaches the composite, and the third emb…
claude Sep 6, 2026
cb0d9ab
gale: distill the review response
claude Sep 6, 2026
e52575f
gale: a delegate token no longer hides the host's terminator
claude Sep 6, 2026
8df91a1
fix(compiler): type a re-minted payload binding by the slot read
claude Sep 6, 2026
cd8d9a2
docs(gale): name the action-body compatibility exception
claude Sep 6, 2026
9b79a8d
chore: tidy
wado-bot[bot] Sep 6, 2026
81ee6c9
fix(compiler): a reference binding takes the payload as it stands
claude Sep 6, 2026
84f57f4
chore: tidy
wado-bot[bot] Sep 6, 2026
dbfb925
gale: distill the review response
claude Sep 6, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
3 changes: 2 additions & 1 deletion package-gale/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ Dev-cycle essentials for working on Gale, a Wado-native ANTLR4-compatible parser

- [`antlr4-compatibility.md`](./antlr4-compatibility.md) — the compatibility contract, prediction / codegen design, soundness invariants, descriptor pipeline, and triage.
- [`resilient-parser.md`](./resilient-parser.md) — error-resilient parsing and the flat CST.
- [`import.md`](./import.md) — grammar composition: how `import S;` resolves and what a delegate contributes.
- [`perf.md`](./perf.md) — performance notes: budget, levers, and measured perf dead-ends.
- [`TODO.md`](./TODO.md) — open work.
- [`README.md`](./README.md) and [WEP: Gale](../docs/wep-2026-03-02-gale.md) — overall design context.
Expand All @@ -22,7 +23,7 @@ Files headed `// Do not edit by hand` are generated. To change one, edit its sou

## Compatibility principle

Gale targets full compatibility with the ANTLR4 `.g4` syntax. The g4 parser must accept any well-formed grammar upstream `antlr4` accepts; a real-world `.g4` that ANTLR4 accepts but Gale rejects is a Gale bug.
Gale targets full compatibility with the ANTLR4 `.g4` syntax. The g4 parser must accept any well-formed grammar upstream `antlr4` accepts; a real-world `.g4` that ANTLR4 accepts but Gale rejects is a Gale bug. The exceptions are enumerated under claim (a) in [`antlr4-compatibility.md`](./antlr4-compatibility.md) — today the action-body one below and `import Foo = Bar;`.

- Compatibility is a capability contract, not byte-for-byte output. Parse trees, tokens, and semantics must match; incidental rendering differences that carry no structure may diverge (e.g. the `<EOF>` marker in `toStringTree()`).
- Gale is a superset: it may accept grammars ANTLR4 rejects only when the meaning is uniquely determined by Gale's language model — never an invented behavior. When accepting would require guessing, reject loudly. Examples: `.`/`~X`-led left-recursive suffixes, and a lexer `mode` inside a combined `grammar` (ANTLR4 allows modes only in a `lexer grammar`; a combined grammar already bundles a lexer, so it desugars unambiguously — still rejected in a `parser grammar`).
Expand Down
85 changes: 82 additions & 3 deletions package-gale/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ with no runtime to install and no version to keep in sync.

The `.g4` format is ANTLR4's; for the full grammar language, see ANTLR4's
[documentation](https://github.com/antlr/antlr4/tree/master/doc). Gale accepts
every grammar ANTLR4 accepts — and a few it rejects, where the meaning is
unambiguous (see [Design](#design)).
every grammar ANTLR4 accepts bar a short list — and a few it rejects, where the
meaning is unambiguous (see [Design](#design)).

## Design

Expand All @@ -21,7 +21,9 @@ with no remaining choice — e.g. a `.`- or `~X`-led left-recursive suffix like
`e ~';' e`, which ANTLR4 errors on (no operator token to climb); or a lexer
`mode` inside a combined `grammar`, which ANTLR4 restricts to a `lexer grammar`
but which is unambiguous since a combined grammar already bundles a lexer. Where
the meaning is not uniquely determined, Gale rejects loudly rather than guessing.
the meaning is not uniquely determined, Gale rejects loudly rather than guessing
— including a handful of grammars ANTLR4 accepts, listed under claim (a) in
[`antlr4-compatibility.md`](./antlr4-compatibility.md).

Self-contained output, no version drift. Gale inlines its entire runtime into
every generated parser. There is no `gale-runtime` package to keep aligned with
Expand Down Expand Up @@ -436,6 +438,83 @@ complete real examples, and
[WEP: Gale Highlight Query](../docs/wep-2026-07-12-gale-highlight-query.md) for
the design.

### What the context tier buys

The override form is the reason to highlight from a parse rather than a token
stream. [`example/`](./example) carries a three-grammar demo of it.
[`MiniHtml.g4`](./example/MiniHtml.g4) imports
[`MiniCss.g4`](./example/MiniCss.g4) and [`MiniJs.g4`](./example/MiniJs.g4), so
one `use` builds a single recognizer: one lexer with three modes, one parser,
one tree. The three `.scm` queries ride in beside the grammars and are
concatenated, so each language keeps its own.

Embedding is the vehicle. The point is one line of
[`MiniJs.highlights.scm`](./example/MiniJs.highlights.scm):

```scheme
(params (JS_IDENT) @variable.parameter)
```

`arrow` and `group` both open with `(`, so whether an identifier inside the
parentheses is a parameter is settled by the `=>` after the closing paren:

```js
let add = (a, b = (1 + 2)) => a + b; // a, b are parameters
let one = ((a)); // a is a variable
```

Two things stand between a highlighter and that answer. A lexer classifies the
identifier when it reads it, before the deciding token exists, and no
mode-stack state brings it closer. A regex highlighter does look ahead, and
`\(([^)]*)\)\s*=>` would settle a flat parameter list. But a default value nests
parentheses, so `[^)]*` stops at the inner `)` and misses the `=>`. Finding that
closing paren means matching brackets, which no regular expression does.

The parser matches them, and the query reads the answer off the rule stack.
`MiniCss.highlights.scm` does the same in the small, though a stateful lexer
could keep up there: one `CSS_IDENT` token becomes a selector, a property, or a
value by where the parse put it.

What you pay for it is that a context capture fires only where the enclosing
rule parsed. Half-write a declaration as `a { b: }` and `b` falls back to no
class, because nothing placed it under `declaration`. Defaults still apply and
the text is preserved, so the file stays readable while you type; only the
contextual half is lost.

That also makes a grammar bug look like this limitation. If input the language
allows loses its classes, the grammar rejected it. Check there first.

The whole page's highlighted HTML is pinned in
[`example/highlight_test.wado`](./example/highlight_test.wado):

```sh
wado test package-gale/example/highlight_test.wado
```

### Composing a grammar for an embedded language

The example is also what `import` looks like for an embedded language. It needs
no feature beyond composition itself; two conventions carry it:

- **The embedded grammar's lexer rules live in a mode of its own** (`mode CSS;`).
A composite has one lexer, so without a mode `MiniHtml`'s `TEXT` would
swallow a stylesheet whole.
- **Its token names are prefixed** (`CSS_IDENT`, `JS_IDENT`). A composite has
one token space, where the first rule of a given name wins and the rest are
dropped.
- **Its parser rules name their tokens rather than spelling them**
(`CSS_COLON`, not `':'`). A parser literal aliases only to a `DEFAULT_MODE`
rule, so spelling one mints a token the delegate's mode never produces.

The host owns the boundaries and nothing else: `MiniHtml.g4` declares
`mode CSS` itself, holding just the `</style>` that leaves it, and composition
unifies the two `mode CSS` declarations by name. So the host says where each
language begins and ends, and neither delegate names its host.

The trade is that an embedded grammar no longer works on its own, since its
rules sit in a mode nothing enters. The full contract, and what a grammar
usable both ways would need, are in [`import.md`](./import.md).

## Compatibility and further reading

Gale targets the full ANTLR4 `.g4` grammar syntax, plus the small superset
Expand Down
Loading
Loading