Skip to content

parser: fix persistent cache staleness and semanticdb_index order dependence - #162

Merged
pcj merged 1 commit into
stackb:masterfrom
zackerydev:cache-correctness
Aug 24, 2026
Merged

parser: fix persistent cache staleness and semanticdb_index order dependence#162
pcj merged 1 commit into
stackb:masterfrom
zackerydev:cache-correctness

Conversation

@zackerydev

@zackerydev zackerydev commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Two related correctness fixes for the persistent cache (-scala_gazelle_cache_file), found while adopting it in a large private monorepo (~3,100 scala rules) where CI restores the cache across runs while developers regenerate with -scala_fileset_file.

1. Deleted rules never leave the cache and keep resolving symbols

readScalaRuleCacheFile loaded every cached rule's symbols into the resolution scope before the walk. Two consequences:

  • A rule deleted (or renamed) in the tree still provides its symbols from the cache, so it can win symbol-name races against real rules and it keeps its semanticdb_index membership. In our setup this made cache-mode runs disagree with fileset-mode runs: a PR deleting a scala rule regenerates cleanly for the author, then CI (with a restored cache) re-adds the dead rule's _semanticdb entry. Reproduced with a 4-day-old cache: 9 phantom semanticdb_index entries.
  • writeScalaRuleCacheFile writes ScalaRules() = the whole memo, so deleted rules persist in the cache file forever and it grows without bound.

Fix: the cache now seeds the memo only (new MemoParser.SeedScalaRule, no symbol loading). A rule's symbols load on its first ParseScalaRule sha256 hit — in walk order, exactly like a fresh parse — and ScalaRules() returns only rules seen this run, so deleted rules age out of the cache on the next write. Memo hits still skip parsing entirely.

2. semanticdb_index membership depends on symbol insertion order

TrieScope.Put is first-wins per symbol name; a duplicate from another label becomes a Conflict on the incumbent and never enters the trie. semanticdbIndexRule.Resolve computed membership from scope winners only, so which labels appear in deps depends on insertion order — package-walk order (fileset / fresh parse) and sorted cache preload produced different winners for duplicated symbol names, i.e. different generated BUILD files from the same tree.

Fix: membership traversal includes Symbol.Conflicts, making the generated deps the order-independent union. (With fix 1, cache hits load symbols in walk order anyway, but membership should not depend on provider order at all.)

Validation

  • New unit tests: pkg/parser/memo_parser_test.go (seed loads no symbols; hit loads symbols exactly once and never re-parses; sha mismatch re-parses; seeded-but-unvisited rules are dropped from ScalaRules()) and pkg/semanticdb/semanticdb_index_test.go (conflict losers are members; kind filter and external-label exclusion still apply).
  • In our monorepo: with a deliberately stale cache, cache-mode output is now byte-identical to fileset-mode output (previously 9 phantom entries), a rule-deletion simulation drops the rule's _semanticdb entry exactly like a fileset run, and steady-state runs still report 0 runtime parses.
  • go test ./pkg/parser/ ./pkg/semanticdb/ ./pkg/resolver/ passes; the language/scala diff-test failures under plain go test are identical before and after this change (they appear to want bazel-provided assets).

Happy to split this into two PRs or reshape the API (e.g. keep ScalaRules() returning everything behind an option) if you'd prefer — for our use the pruning behavior is the point, since CI restores the cache indefinitely.

…endence

Two related correctness fixes for -scala_gazelle_cache_file:

1. Cached rules no longer enter the symbol scope at startup. Previously
   readScalaRuleCacheFile loaded every cached rule's symbols before the
   walk, so rules deleted from the tree kept providing symbols, kept
   their semanticdb_index membership, and never aged out of the cache
   file. The cache now seeds the memo only (SeedScalaRule); a rule's
   symbols load on its first ParseScalaRule hit, in walk order like a
   fresh parse, and ScalaRules() returns only rules seen this run so
   deletions age out of a persistent cache.

2. semanticdb_index membership now traverses Symbol.Conflicts.
   TrieScope.Put is first-wins per symbol name, so which label holds a
   duplicated name depends on symbol insertion order; membership
   computed from scope winners differed between walk-order runs and
   sorted cache preloads. Including conflict losers makes the generated
   deps order-independent.
@pcj
pcj merged commit 9e88752 into stackb:master Aug 24, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants