fix: full-project review findings + retrieval precision (26% fewer tokens served)

benchmarks/results/cce-phase1-ab.json

@@ -0,0 +1,116 @@
+{
+  "threshold": 0.2,
+  "marginal_ratio": 0.75,
+  "base_tokens": 32138,
+  "tuned_tokens": 23741,
+  "reduction_pct": 26.1,
+  "base_hits": 4,
+  "tuned_hits": 4,
+  "judged": 8,
+  "rows": [
+    {
+      "query": "How does the chunker split code into chunks?",
+      "base": {
+        "tokens": 2533,
+        "chunks": 10,
+        "hit": true
+      },
+      "tuned": {
+        "tokens": 2319,
+        "chunks": 7,
+        "hit": true
+      }
+    },
+    {
+      "query": "vector search implementation",
+      "base": {
+        "tokens": 4600,
+        "chunks": 10,
+        "hit": false
+      },
+      "tuned": {
+        "tokens": 3233,
+        "chunks": 8,
+        "hit": false
+      }
+    },
+    {
+      "query": "confidence scoring formula",
+      "base": {
+        "tokens": 3359,
+        "chunks": 10,
+        "hit": false
+      },
+      "tuned": {
+        "tokens": 3359,
+        "chunks": 10,
+        "hit": false
+      }
+    },
+    {
+      "query": "MCP server tools",
+      "base": {
+        "tokens": 4221,
+        "chunks": 10,
+        "hit": true
+      },
+      "tuned": {
+        "tokens": 4221,
+        "chunks": 10,
+        "hit": true
+      }
+    },
+    {
+      "query": "how does indexing pipeline work",
+      "base": {
+        "tokens": 6331,
+        "chunks": 10,
+        "hit": true
+      },
+      "tuned": {
+        "tokens": 6008,
+        "chunks": 9,
+        "hit": true
+      }
+    },
+    {
+      "query": "FTS5 full text search",
+      "base": {
+        "tokens": 4321,
+        "chunks": 10,
+        "hit": false
+      },
+      "tuned": {
+        "tokens": 736,
+        "chunks": 2,
+        "hit": false
+      }
+    },
+    {
+      "query": "graph neighbors query",
+      "base": {
+        "tokens": 2059,
+        "chunks": 10,
+        "hit": false
+      },
+      "tuned": {
+        "tokens": 1123,
+        "chunks": 6,
+        "hit": false
+      }
+    },
+    {
+      "query": "git commit history indexing",
+      "base": {
+        "tokens": 4714,
+        "chunks": 10,
+        "hit": true
+      },
+      "tuned": {
+        "tokens": 2742,
+        "chunks": 6,
+        "hit": true
+      }
+    }
+  ]
+}

benchmarks/results/cce-phase1-ab.md

@@ -0,0 +1,78 @@
+# CCE Phase 1 A/B Benchmark: Retrieval Precision Tuning
+
+**Date:** 2026-07-03
+**Repo:** code-context-engine (this repo, indexed from `.`)
+**Branch:** fix/review-findings-2026-07-03
+**Index:** 4,162 chunks from 285 files
+**Queries:** 8 (from `benchmarks/sample_queries.json`)
+
+## Winning Defaults
+
+| Parameter | Old Default | New Default | Effect |
+|-----------|------------|-------------|--------|
+| `retrieval_confidence_threshold` | 0.2 | **0.2** (unchanged) | No effect — all scores ≥ 0.70 on this corpus |
+| `retrieval_marginal_ratio` | 0.5 | **0.75** | Achieved 26.1% token reduction |
+
+## Aggregate Results (Winning Config)
+
+| Metric | Baseline | Tuned (threshold=0.2, marginal_ratio=0.75) |
+|--------|----------|---------------------------------------------|
+| Total tokens served | 32,138 | 23,741 |
+| Token reduction | — | **26.1%** ✓ (gate: ≥25%) |
+| Hit rate | 4/8 | **4/8** ✓ (gate: ≥ baseline) |
+
+**Gate verdict: PASS** — 26.1% reduction ≥ 25%, hit rate unchanged.
+
+## Per-Query Detail (Winning Run)
+
+| Query | Base tokens | Tuned tokens | Base chunks | Tuned chunks | Hit |
+|-------|------------|--------------|-------------|--------------|-----|
+| How does the chunker split code into chunks? | 2,533 | 2,319 | 10 | 7 | True → True |
+| vector search implementation | 4,600 | 3,233 | 10 | 8 | False → False |
+| confidence scoring formula | 3,359 | 3,359 | 10 | 10 | False → False |
+| MCP server tools | 4,221 | 4,221 | 10 | 10 | True → True |
+| how does indexing pipeline work | 6,331 | 6,008 | 10 | 9 | True → True |
+| FTS5 full text search | 4,321 | 736 | 10 | 2 | False → False |
+| graph neighbors query | 2,059 | 1,123 | 10 | 6 | False → False |
+| git commit history indexing | 4,714 | 2,742 | 10 | 6 | True → True |
+
+## All Runs (Including Rejected)
+
+| Run | threshold | marginal_ratio | Tokens: base → tuned | Reduction | Hits | Gate |
+|-----|-----------|----------------|----------------------|-----------|------|------|
+| 1 | 0.35 | 0.50 | 32,026 → 32,026 | 0.0% | 4/8 → 4/8 | FAIL (reduction < 25%) |
+| 2 | 0.35 | 0.60 | 32,138 → 32,138 | 0.0% | 4/8 → 4/8 | FAIL (reduction < 25%) |
+| 3 | 0.35 | 0.70 | 32,138 → 30,482 | 5.2% | 4/8 → 4/8 | FAIL (reduction < 25%) |
+| 4 | 0.35 | 0.75 | 32,138 → 23,741 | 26.1% | 4/8 → 4/8 | PASS |
+| 5 | 0.35 | 0.80 | 32,138 → 20,061 | 37.6% | 4/8 → 4/8 | PASS (higher reduction, but 0.75 preferred as minimum-passing) |
+| **6 (winner)** | **0.2** | **0.75** | **32,138 → 23,741** | **26.1%** | **4/8 → 4/8** | **PASS** |
+
+### Notes on Tuning
+
+- The `confidence_threshold` parameter (0.35, then 0.2) had **no effect** on this corpus: all retrieved chunk confidence scores fell in the 0.70–0.95 range, well above any tested threshold. The ladder's instruction to use `--threshold 0.25` or `--threshold 0.2` when hit rate drops was not needed — hits were stable throughout.
+- Reduction came **entirely from `marginal_ratio`**: this parameter stops adding chunks whose score falls below `ratio × top_score`. With scores clustered at the top (spread ≈ 0.25), a ratio of 0.75 is required to create meaningful cuts.
+- `marginal_ratio=0.75` is the minimum value that clears the 25% gate. `0.80` provides more reduction (37.6%) with the same hit rate, but `0.75` was chosen as the conservative minimum-passing value.
+- The brief's tuning ladder (tries 0.5, 0.6) did not cover the actual operating range of this corpus (needed 0.75+). Extended ladder runs (0.70, 0.75, 0.80) were added to find the gate-passing threshold.
+
+### Limitations of this evidence
+
+- **The 0.75 default sits on a cliff, not a plateau.** The reduction curve is
+  0.70 → 5.2%, 0.75 → 26.1%, 0.80 → 37.6% — a 21-point jump across one step.
+  Because the marginal stop cuts relative to the corpus's score distribution,
+  a project whose score floor sits slightly higher (e.g. 0.77 instead of
+  0.70) would see little or no reduction at 0.75, and one with a wider spread
+  could see much more. Treat 0.75 as a starting default, not stable guidance;
+  it is tunable per project via `retrieval.marginal_ratio`.
+- **Sample size: 8 queries, one corpus (this repo).** The gate verdict is
+  real but narrow. Re-validate on at least one external corpus (the existing
+  `fastapi`/`chi`/`fiber` query sets) before citing these numbers in docs or
+  README.
+- Run 1's baseline total (32,026) differs from later runs (32,138) by 112
+  tokens — the index was rebuilt between runs 1 and 2. The winning run's
+  numbers are internally consistent (per-row sums verified).
+
+## Config Changes Applied
+
+- `src/context_engine/config.py`: `retrieval_marginal_ratio` updated `0.5 → 0.75`
+- `tests/test_config.py`: `test_marginal_ratio_default` updated to assert `0.75`
+- `retrieval_confidence_threshold` left at `0.2` (unchanged — threshold had no effect)

benchmarks/run_benchmark.py

@@ -42,6 +42,7 @@
 from context_engine.memory.grammar import compress_with_counts as grammar_compress
 from context_engine.retrieval.retriever import HybridRetriever
 from context_engine.storage.local_backend import LocalBackend
+from context_engine.utils import project_storage_dir
 
 _CHARS_PER_TOKEN = 4
 
@@ -420,6 +421,73 @@ def format_markdown(results: dict) -> str:
     return "\n".join(lines) + "\n"
 
 
+async def run_ab(
+    project_dir: Path,
+    queries: list[dict],
+    storage_dir: Path,
+    threshold: float,
+    marginal_ratio: float,
+) -> dict:
+    """Index once, then run every query with baseline vs tuned retrieval
+    parameters against the same index. No stacking with compression —
+    this isolates the retrieval-precision change."""
+    config = Config()
+    config.storage_path = str(storage_dir)
+
+    print("Indexing project once for A/B...")
+    idx = await run_indexing(config, project_dir, full=True)
+    print(f"  {idx.total_chunks} chunks from {len(idx.indexed_files)} files")
+
+    storage_base = project_storage_dir(config, project_dir)
+    backend = LocalBackend(base_path=str(storage_base))
+    embedder = Embedder(model_name=config.embedding_model)
+    retriever = HybridRetriever(backend=backend, embedder=embedder)
+
+    rows = []
+    for q in queries:
+        base = await retriever.retrieve(q["query"], top_k=10)
+        tuned = await retriever.retrieve(
+            q["query"], top_k=10,
+            confidence_threshold=threshold,
+            marginal_ratio=marginal_ratio,
+        )
+        expected = set(q.get("expected_files", []))
+
+        def _measure(chunks):
+            files = {c.file_path for c in chunks}
+            return {
+                "tokens": sum(_count_tokens(c.content) for c in chunks),
+                "chunks": len(chunks),
+                "hit": bool(files & expected) if expected else None,
+            }
+
+        rows.append({"query": q["query"],
+                     "base": _measure(base), "tuned": _measure(tuned)})
+        b, t = rows[-1]["base"], rows[-1]["tuned"]
+        print(f"  {q['query'][:45]:<45} tokens {b['tokens']:>6} → {t['tokens']:>6}  "
+              f"chunks {b['chunks']:>2} → {t['chunks']:>2}  "
+              f"hit {b['hit']} → {t['hit']}")
+
+    def _agg(side):
+        tok = sum(r[side]["tokens"] for r in rows)
+        hits = sum(1 for r in rows if r[side]["hit"])
+        judged = sum(1 for r in rows if r[side]["hit"] is not None)
+        return tok, hits, judged
+
+    base_tok, base_hits, judged = _agg("base")
+    tuned_tok, tuned_hits, _ = _agg("tuned")
+    reduction = (1 - tuned_tok / base_tok) * 100 if base_tok else 0.0
+    print(f"\nTokens served: {base_tok:,} → {tuned_tok:,}  ({reduction:.1f}% reduction)")
+    print(f"Hit rate: {base_hits}/{judged} → {tuned_hits}/{judged}")
+    return {
+        "threshold": threshold, "marginal_ratio": marginal_ratio,
+        "base_tokens": base_tok, "tuned_tokens": tuned_tok,
+        "reduction_pct": round(reduction, 1),
+        "base_hits": base_hits, "tuned_hits": tuned_hits,
+        "judged": judged, "rows": rows,
+    }
+
+
 def main():
     parser = argparse.ArgumentParser(description="CCE Benchmark Suite")
     parser.add_argument("--repo", help="Git repo URL to clone and benchmark")
@@ -432,6 +500,13 @@ def main():
     parser.add_argument("--queries", help="Path to queries JSON file")
     parser.add_argument("--output", help="Output path for markdown report")
     parser.add_argument("--json-output", help="Output path for raw JSON results")
+    parser.add_argument("--ab", action="store_true",
+                        help="Run each query twice: baseline retrieval vs tuned "
+                             "(threshold + marginal ratio) and print the delta")
+    parser.add_argument("--threshold", type=float, default=0.35,
+                        help="Tuned confidence_threshold for --ab (default 0.35)")
+    parser.add_argument("--marginal-ratio", type=float, default=0.5,
+                        help="Tuned marginal_ratio for --ab (default 0.5)")
     args = parser.parse_args()
 
     # Determine project dir and queries
@@ -476,6 +551,17 @@ def main():
     storage_dir = Path(tempfile.mkdtemp(prefix="cce-bench-storage-"))
 
     try:
+        if args.ab:
+            results = asyncio.run(run_ab(
+                project_dir, queries, storage_dir,
+                args.threshold, args.marginal_ratio,
+            ))
+            if args.json_output:
+                out_path = Path(args.json_output)
+                out_path.parent.mkdir(parents=True, exist_ok=True)
+                out_path.write_text(json.dumps(results, indent=2) + "\n")
+            return
+
         results = asyncio.run(run_benchmark(project_dir, queries, storage_dir))
         if args.repo:
             results["repo_url"] = args.repo

docs-src/src/content/docs/configuration.md

@@ -33,7 +33,8 @@ indexer:
 
 retrieval:
   top_k: 20              # Maximum chunks returned per query
-  confidence_threshold: 0.5  # Minimum score to include a result (0.0 to 1.0)
+  confidence_threshold: 0.2  # Minimum score to include a result (0.0 to 1.0)
+  marginal_ratio: 0.75   # Stop adding chunks when score falls below this fraction of top
 
 embedding:
   model: BAAI/bge-small-en-v1.5  # Embedding model (fastembed-compatible)
@@ -93,7 +94,9 @@ The default `BAAI/bge-small-en-v1.5` is recommended for most use cases. It balan
 
 **`top_k`** controls how many chunks the retriever returns per query. Higher values surface more context but cost more tokens. Default: 20.
 
-**`confidence_threshold`** sets the minimum score to include a result. Range 0.0 to 1.0. Lower values return more results; higher values return only strong matches. Default: 0.5.
+**`confidence_threshold`** sets the minimum score to include a result. Range 0.0 to 1.0. Lower values return more results; higher values return only strong matches. Default: 0.2.
+
+**`marginal_ratio`** stops adding result chunks once a chunk's confidence falls below this fraction of the top result's score. Range 0.0 to 1.0; `0` disables this behavior. The effective reduction depends on your corpus's score distribution, so tune per project. See `benchmarks/results/cce-phase1-ab.md` for tuning evidence. Default: 0.75.
 
 At runtime, the agent can pass `top_k` and `max_tokens` directly to `context_search`: