Skip to content

Phase D follow-up: formal shadow evaluation + relevance acceptance for hybrid retrieval #319

Description

@SweetSophia

Split from #261.

Context

Hybrid retrieval is merged (#261 capability), off by default (ADR), and live on the reference deployment since 2026-09-03 (llama.cpp local embeddings, coverage 99.3%, recall serving with zero lexical fallback). The rollout went coverage-gate → serve directly.

What did not run

Phase D specified two acceptance gates that were skipped, not passed:

  1. Shadow evaluation — run hybrid and keyword rankings side-by-side on real query traffic (or a sampled set), log both, compare while keyword remains the user-facing path.
  2. Relevance/quality acceptance — a scored query set (gold answers or graded relevance judgments) with an agreed threshold that hybrid must meet or beat keyword.

Proposed acceptance

  • Shadow harness: dual-rank a query corpus without serving hybrid results; persist per-query rankings + metrics
  • Scored query set: ≥ N queries with graded relevance, versioned in-repo
  • Metric report: recall@k / nDCG or equivalent, hybrid vs keyword, published as evidence
  • Threshold decision documented (accept / tune / reject), covering the keyword-only fallback path for over-context articles
  • Results recorded for the reference deployment profile (nomic-embed-text-v1.5 Q8_0, 768-d)

Non-goals: changing the default-off posture; remote embedding providers; Ollama (excluded by policy).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:retrievalRecall and retrieval pipelineepicLarge cross-cutting initiativepriority:mediumMedium priority

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions