Motivation
Surfaced during smoke testing in #20: the TFIDF fallback embedder rebuilds its IDF table against the live (post-retraction) corpus via `db.ListLeaves()` (which now filters retracted nodes by default after #20). When a memory was embedded under one corpus and the embedder is later rebuilt against a different corpus, cosine similarity between freshly-embedded candidates and stored vectors becomes incoherent — they're effectively in different vector spaces.
The visible effect: `engine.findRetractedMatches` may have reduced recall on TFIDF users. A new write semantically identical to a retracted memory might not hit the dedup-against-retracted gate, defeating the protection that catches PII re-introduction.
This is not silently broken — it's degraded. The store-layer filter still correctly excludes retracted from search/find/tree (the read-side invariant holds). The write-side guard is just less reliable on the TFIDF path.
Ollama / nomic-embed users are unaffected — their embeddings are pre-trained against a static model, no corpus drift.
Repro (from #20 smoke)
Against operator's real ~147MB DB:
- Pick a high-relevance leaf as victim
- Retract it
- Build a new TFIDFEmbedder via `engine.NewTFIDFEmbedder(db, 512)` — this loads only live leaves now
- `engine.Remember` with the victim's exact L0 as the new candidate's summary
- Expected: `RetractedMatchError` fires
- Observed: gate did not fire; the write succeeded
The smoke test logged this rather than failed, with the `embedder recall on real data is fuzzy` caveat.
Possible directions
- Build TFIDF corpus from `ListLeavesIncludingRetracted` — keeps retracted-node vocabulary in the IDF table so vector space stays coherent. Costs: retracted memories' tokens influence IDF for live memories too. Probably small effect at typical corpus sizes.
- Re-embed retracted nodes against the new corpus — explicit migration step. Heavyweight; adds storage churn.
- Document and accept — TFIDF is the fallback; recommend Ollama for users who care about retraction recall. Update README to call this out.
- Some hybrid — build a separate TFIDF model just for the retraction-match path that includes retracted nodes in its corpus.
Direction 1 feels lightest and probably right. Direction 3 is the lowest-effort if direction 1 has unintended effects.
Tracking
This was flagged in #20 as a separate concern. Filing for follow-up so it doesn't get lost. Not blocking #20 — the substrate-level invariants (read-side filter, no-resurrection) are correct on TFIDF; only the dedup-gate recall is degraded.
Motivation
Surfaced during smoke testing in #20: the TFIDF fallback embedder rebuilds its IDF table against the live (post-retraction) corpus via `db.ListLeaves()` (which now filters retracted nodes by default after #20). When a memory was embedded under one corpus and the embedder is later rebuilt against a different corpus, cosine similarity between freshly-embedded candidates and stored vectors becomes incoherent — they're effectively in different vector spaces.
The visible effect: `engine.findRetractedMatches` may have reduced recall on TFIDF users. A new write semantically identical to a retracted memory might not hit the dedup-against-retracted gate, defeating the protection that catches PII re-introduction.
This is not silently broken — it's degraded. The store-layer filter still correctly excludes retracted from search/find/tree (the read-side invariant holds). The write-side guard is just less reliable on the TFIDF path.
Ollama / nomic-embed users are unaffected — their embeddings are pre-trained against a static model, no corpus drift.
Repro (from #20 smoke)
Against operator's real ~147MB DB:
The smoke test logged this rather than failed, with the `embedder recall on real data is fuzzy` caveat.
Possible directions
Direction 1 feels lightest and probably right. Direction 3 is the lowest-effort if direction 1 has unintended effects.
Tracking
This was flagged in #20 as a separate concern. Filing for follow-up so it doesn't get lost. Not blocking #20 — the substrate-level invariants (read-side filter, no-resurrection) are correct on TFIDF; only the dedup-gate recall is degraded.