Skip to content

fix(schema): anchor tier0 confidence proxy to tier thresholds (user-approval tier was unreachable)#63

Open
danielhertz1999-bit wants to merge 2 commits into
CodeAbra:mainfrom
danielhertz1999-bit:fix/schema-tier0-approval-tier-unreachable
Open

fix(schema): anchor tier0 confidence proxy to tier thresholds (user-approval tier was unreachable)#63
danielhertz1999-bit wants to merge 2 commits into
CodeAbra:mainfrom
danielhertz1999-bit:fix/schema-tier0-approval-tier-unreachable

Conversation

@danielhertz1999-bit

Copy link
Copy Markdown
Contributor

Problem

induce_schemas_tier0 scores each tag co-occurrence with
confidence = min(1.0, count / 10.0), then gates two tiers on it. That proxy
shares the intended slope (0.1) with the threshold constants but drops the
intercept, so the documented tier boundaries can never be satisfied:

co-occurrence count intended tier actual result
3–4 pending_user_approval (USER_APPROVAL_COOCCURRENCE=3, …CONFIDENCE=0.65) discarded — needs count/10 ≥ 0.65count ≥ 6.5, contradicting count < 5
5–8 auto (AUTO_INDUCT_COOCCURRENCE=5) discarded — needs count/10 ≥ 0.85count ≥ 8.5
≥9 auto

So the entire pending_user_approval tier is dead code, and auto only fires
at count ≥ 9 rather than the documented ≥ 5.

Corroborating signals that this is unintended:

  • core/_query_dispatch.py already handles a "pending_user_approval" status
    that is never produced.
  • test_induce_schemas_tier0_threshold_lowered_requires_approval is named for
    this tier, builds a 4-cooccurrence pair, and computes
    match = [c for c in candidates if c.evidence_count == 4] — but never
    asserts on match, so the dead branch slipped through.

Fix

Anchor the proxy to the threshold constants so both boundaries hold exactly:

count == USER_APPROVAL_COOCCURRENCE (3) -> USER_APPROVAL_CONFIDENCE (0.65)
count == AUTO_INDUCT_COOCCURRENCE   (5) -> AUTO_INDUCT_CONFIDENCE   (0.85)
=> confidence = min(1.0, 0.35 + 0.1 * count)

This is the minimal change consistent with the existing constants (same slope,
restores the dropped 0.35 intercept). Resulting behavior: count 3–4 → pending_user_approval, count ≥ 5 → auto, count < 3 → discarded.

Tests

  • Regression: added the missing assertion to
    test_induce_schemas_tier0_threshold_lowered_requires_approval — a
    4-cooccurrence pair must surface as pending_user_approval. Verified it
    fails on the old proxy and passes with this fix.
  • Updated the test_..._byte_identical_to_pre_d26_implementation reference impl
    to mirror the corrected proxy (that test locks the streaming refactor against
    a hand-derived reference; the reference had to move with the behavior).
  • Full schema/insight/sleep-consolidation suites green
    (test_schema_induction, test_schema_induction_streaming,
    test_mcp_schema_list, test_insight, test_sleep_consolidation_streaming).

🤖 Generated with Claude Code

danielhertz1999-bit and others added 2 commits July 8, 2026 17:16
induce_schemas_tier0 computed `confidence = min(1.0, count / 10.0)`, which
shares the intended slope (0.1) with the tier thresholds but drops the
intercept. That made the documented tier boundaries unsatisfiable:

- pending_user_approval (USER_APPROVAL_COOCCURRENCE=3 <= count < 5, with
  confidence >= USER_APPROVAL_CONFIDENCE=0.65) was unreachable: count/10 >= 0.65
  needs count >= 6.5, contradicting count < 5. The whole user-approval tier was
  dead code -- though _query_dispatch already handles the status and a test was
  named for it (with its assertion accidentally omitted).
- auto fired only at count >= 9 instead of the documented
  AUTO_INDUCT_COOCCURRENCE=5, since count/10 >= 0.85 needs count >= 8.5.
  Co-occurrence counts 5-8 were silently discarded.

Anchor the proxy to the constants so both boundaries hold exactly:
  count == 3 -> 0.65 (USER_APPROVAL_CONFIDENCE)
  count == 5 -> 0.85 (AUTO_INDUCT_CONFIDENCE)
i.e. confidence = min(1.0, 0.35 + 0.1 * count).

Tests:
- Add the missing assertion to
  test_induce_schemas_tier0_threshold_lowered_requires_approval (it computed
  `match` but never asserted), pinning the user-approval tier. Fails on the old
  proxy, passes now.
- Update the pre-D26 byte-identical reference impl to mirror the corrected proxy.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… corpus

The tier0 confidence-proxy fix makes schema induction fire for the 5 pinned
seeds -- they share the {seed, lossless-gate} tag pair, count == K_SEEDS == 5 ==
AUTO_INDUCT_COOCCURRENCE -- so the post-kill consolidation now persists one
semantic schema record. test_sigkill_mid_write_leaves_store_lossless compared
the rebuilt HNSW count against an `active` snapshot taken *before* consolidation,
so the +1 schema record tripped `raw=46 != active=45`.

Consolidation legitimately mutates the corpus; the reconciliation invariant is
"HNSW raw count == SQLite active count" on the post-consolidation state. Flush
buffered writes and re-read the active count after consolidation before
comparing. Verified in-process (no SIGKILL, faked embedder) that consolidation
adds exactly the one induced schema and raw == active == 46 afterwards.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@danielhertz1999-bit

Copy link
Copy Markdown
Contributor Author

Follow-up commit 5e1afc4: the confidence fix has an intended side effect the full gate surfaced — test_sigkill_mid_write_leaves_store_lossless seeds 5 pinned records sharing the {seed, lossless-gate} tag pair (count == K_SEEDS == 5 == AUTO_INDUCT_COOCCURRENCE), so the post-kill consolidation now correctly auto-induces one schema record. The test compared the rebuilt HNSW count against an active snapshot taken before consolidation, so the +1 tripped raw=46 != active=45.

That invariant only held because the old proxy never induced anything. Reconciliation is really "HNSW raw count == SQLite active count" on the post-consolidation corpus, so the fix flushes buffered writes and re-reads the active count after consolidation before comparing (strictly more correct — also robust to erasure/other consolidation mutations).

Verified in-process (no SIGKILL, faked embedder) that consolidation adds exactly the one induced schema: boot (active=45, raw=45) → after (active=46, raw=46). This was the only failure in the full not perf and not slow and not live gate (3668 passed / 1 failed pre-fix).

@danielhertz1999-bit

Copy link
Copy Markdown
Contributor Author

FYI, unrelated to this PR — surfaced while running the full not perf and not slow and not live gate on Linux to verify #63, sharing for awareness (no action requested here).

test_hnsw_reuse_recall_parity::test_rebuild_recall_parity_many_cycles fails deterministically in my environment: recall@10 = 0.597 vs the 0.99 floor, after the 100 _rebuild_index_from_sqlite() reuse cycles. The 1-cycle variant (…parity_one_cycle) passes, and it's not embedder-related — the fixture writes raw 32-dim vectors (IAI_MCP_EMBED_DIM=32).

Notably this is the same pinned hnswlib (0.8.0) CI uses, so it isn't a version bump — it looks like a build/platform sensitivity: hnswlib built from source here (Python 3.12.13, numpy 2.2.6, gcc 15 / glibc 2.43, Linux/WSL2 x86_64) loses recall across the repeated mark_deleted-all + add_items(replace_deleted=True) cycles, where CI's build holds ≥ 0.99.

Flagging because the many-cycle reuse path's recall looks fragile across hnswlib builds within the >=0.8.0,<1.0 pin — a future runner/toolchain change could make CI see this too. Happy to open a separate issue with a minimal repro if that's useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant