Skip to content

TFIDF embedder corpus shift degrades dedup-against-retracted recall #22

Description

@lazypower

Motivation

Surfaced during smoke testing in #20: the TFIDF fallback embedder rebuilds its IDF table against the live (post-retraction) corpus via `db.ListLeaves()` (which now filters retracted nodes by default after #20). When a memory was embedded under one corpus and the embedder is later rebuilt against a different corpus, cosine similarity between freshly-embedded candidates and stored vectors becomes incoherent — they're effectively in different vector spaces.

The visible effect: `engine.findRetractedMatches` may have reduced recall on TFIDF users. A new write semantically identical to a retracted memory might not hit the dedup-against-retracted gate, defeating the protection that catches PII re-introduction.

This is not silently broken — it's degraded. The store-layer filter still correctly excludes retracted from search/find/tree (the read-side invariant holds). The write-side guard is just less reliable on the TFIDF path.

Ollama / nomic-embed users are unaffected — their embeddings are pre-trained against a static model, no corpus drift.

Repro (from #20 smoke)

Against operator's real ~147MB DB:

  1. Pick a high-relevance leaf as victim
  2. Retract it
  3. Build a new TFIDFEmbedder via `engine.NewTFIDFEmbedder(db, 512)` — this loads only live leaves now
  4. `engine.Remember` with the victim's exact L0 as the new candidate's summary
  5. Expected: `RetractedMatchError` fires
  6. Observed: gate did not fire; the write succeeded

The smoke test logged this rather than failed, with the `embedder recall on real data is fuzzy` caveat.

Possible directions

  1. Build TFIDF corpus from `ListLeavesIncludingRetracted` — keeps retracted-node vocabulary in the IDF table so vector space stays coherent. Costs: retracted memories' tokens influence IDF for live memories too. Probably small effect at typical corpus sizes.
  2. Re-embed retracted nodes against the new corpus — explicit migration step. Heavyweight; adds storage churn.
  3. Document and accept — TFIDF is the fallback; recommend Ollama for users who care about retraction recall. Update README to call this out.
  4. Some hybrid — build a separate TFIDF model just for the retraction-match path that includes retracted nodes in its corpus.

Direction 1 feels lightest and probably right. Direction 3 is the lowest-effort if direction 1 has unintended effects.

Tracking

This was flagged in #20 as a separate concern. Filing for follow-up so it doesn't get lost. Not blocking #20 — the substrate-level invariants (read-side filter, no-resurrection) are correct on TFIDF; only the dedup-gate recall is degraded.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions