Mago 2.0 - Tracking Issue

[azjezz]

Mago 2.0 - Tracking Issue

Meta issue for the 2.0 release. 2.0 is a ground-up rework of the analysis pipeline plus the tooling and stability commitments that ride on top of it. This issue tracks the whole effort; each box links to (or will spawn) a dedicated issue/PR.

Why 2.0

PHP is effectively two languages: the surface syntax, and the type language embedded in docblocks (@var, @param, @template, @method, @psalm-type, ...). They have different grammars and, in 1.x, are parsed by different code at different times, the syntax tree as one artifact, docblocks re-parsed on demand across a multiple scattered call sites.

2.0 fuses them. A single lowering pass turns the syntax tree into one typed IR that already has docblocks parsed, names resolved, and sugar normalized — and analysis is split into honest phases (scan → inference → check) that hand a typed program forward instead of re-deriving it. As a side effect we stop lying about the tree: what 1.x calls the "AST" is really a lossless CST (it keeps trivia and spans, which is why the formatter round-trips), so 2.0 names it a CST and reserves "the real, lowered tree" for the IR.

The new pipeline

    source
      │ parse
      ▼
    CST            (mago_syntax::cst — lossless: trivia, spans, round-trips)
      │ lower      (per-file, NO cross-file info: name resolution + docblock parse + desugar)
      ▼
    IR<(), ()>     (typed-in-node tree, generic over context C and type T)
      │ scan
      ├─▶ linter
      ▼
    CodebaseMetadata
      │            IR + CodebaseMetadata
      ▼ inference
    IR<(), T>      (type info filled in the nodes)
      │ check
      ▼
    IR<C, T>       (context / "Env" filled: cross-references resolved, etc.)
      │
      ├─▶ security / taint analysis
      └─▶ reporting

Key decisions already settled in design:

• The IR is arena-allocated, generic over C (context) and T (type), with annotationion** (no interior mutability), the generics encode the analysis phase in the type system.
• Lowering is fully local, no codebase access. Linking a name to another file is the checker's on-demand job, not lowering's.
• The IR must be retained and movable across phase boundaries (inference output is the check input), so it lives in a cross-phase arena (bumpalo-herd) or as owned-per-file values, not borrowed from a transient per-thread bump.

Workstreams

1. Compiler core (critical path — roughly in order)

• Rename the syntax tree AST → CST (mago_syntax::ast → mago_syntax::cst). Mechanical, unblocks the naming for everything below. refactor: rename ast to cst everywhere #1933
• New docblock parser (the second language; becomes a lowering input). feat(phpdoc-syntax): introduce a new phpdoc parser #1932
• Define the new IR crate (mago_ir/mago_hir — TBD): node defs generic over S/D/E
  • Introduce high-level intermediate representation feat(hir): introduce high-level intermediate representation #1935
  • Add lowering
    • Add PHP CST lowering feat(hir): native CST lowering with name resolution #1936 chore(hir): some clean up #1937
    • Add PHPDOC CST lowering feat(hir): lower phpdoc to IR #1943
  • Add fold and walker trait feat(hir): add fold and walker traversal over the IR #1944
• New lowering crate: CST -> IR, including name resolution (absorbs mago_names) and docblock parsing/merge + desugaring. feat(hir): native CST lowering with name resolution #1936 chore(hir): some clean up #1937 feat(hir): lower phpdoc to IR #1943
• New codex scanner that builds CodebaseMetadata from the IR (instead of the CST) feat: IR-based codebase scanner #1952
• Introduce mago-oracle, a mago-codex replacement:
  • suffete as a base for oracle::ty - done in fix(oracle): close lattice soundness gaps #1979
  • Introduce a definition scanner, done in Mago Next #1934
  • Introduce a symbol table
  • Introduce a symbol table linker
• New inference crate: IR + CodebaseMetadata -> IR with T filled. feat(inference): introduce mago-inference crate #2036
• New type checker crate: reports issues.
• EOL mago-analyzer, superseded by inference + checker; it does not get a 2.0.

2. Type-system features

• Recursive types: recursive-array<K, V> and recursive-list<T> (Support recursive types #1139).
• Negated types (Negation type #678).
• Pure closure type inference (Support pure closure type inference #453).

3. New commands

• mago check — runs linter → inference → checker as one pass (faster than mago analyze + mago lint separately).
• Taint / security analysis on the typed + context-filled IR, mago security-analy-taint-analysis, TBD) (Support SQL static analysis and type inference for the database access layer #454).
• mago bc-check, detect backward-incompatible changes between the working tree and a given tag (BC Checking #675).

4. Linter & formatter

• (Maybe) move the linter onto the IR rather than the CST — TBD.
• Format/Lint PHPDocs ([formatter] Format Doxygen comments #906, formatting phpdoc #467, New Rule: Disallow legacy shorthand array syntax (T[]) #1016).

5. Editor & LSP

• Stabilize the language server ([Language Server] Ability to @mago-expect, @mago-ignore from Quick Fix #1908, feat: language server #1926).
• Zed editor plugin.

6. Extensibility & stability

• Plugin system — let users write plugins (host language TBD: PHP / JS / Rust / WAS / Something else).
• Stabilize the Rust API. From 2.0.0, mago crates follow semver - no more frequent breaking changes to the Rust API.

Open questions

• Name of the context generic: Ctx vs Env?
• Plugin host language: PHP / JS / Rust / WASM / Something else? (feature: allow creating custom linting rules using PHP. #177)
• Linter on the IR, or keep it on the CST?
• Taint analysis: dedicated mago security-analysis subcommand vs --with-taint-analysis
• Do mago analyze / mago lint get deprecated in favor of mago check, or kept as alias
• IR retention across phases: central bumpalo-herd vs owned-per-file (self_cell)?