Merge pull request #326 from SweetSophia/fix/325-shadow-measurement

fix(shadow-eval): expose denominators and preflight fixture freshness

Author: SweetSophia

.github/workflows/ci.yml

@@ -165,6 +165,11 @@ jobs:
       - name: Run hybrid shadow harness tests (no services required)
         run: npm run test:hybrid-shadow
 
+      - name: Run shadow freshness tests against isolated temporary corpus
+        run: npm run test:hybrid-shadow-db
+        env:
+          SHADOW_TEST_DATABASE_URL: ${{ env.DATABASE_URL }}
+
       - name: Run memory tests
         run: npm run test:memory
 

docs/HYBRID-SHADOW-EVALUATION.md

@@ -0,0 +1,98 @@
+# Shadow evaluation: denominators and fixture freshness
+
+The local operator CLI `npm run hybrid:shadow-eval` runs keyword then hybrid
+search without serving results. This is measurement tooling, not a serving
+acceptance gate. Reports remain private; admin runs can include restricted
+rankings. Search may write caches, authorize dispatch, and call embeddings.
+
+## Freshness before evaluation
+
+Every dual-path run performs an exact article lookup **before searching** and
+persists `freshness-<uuid>.json`, even if subsequent retrieval fails. It also
+includes freshness scope, timestamps, outcome and counts in aggregate JSON and
+Markdown. A database lookup failure aborts the run; it is not evidence of absence.
+To inspect only freshness, without search or embedding configuration:
+
+```sh
+npm run hybrid:shadow-eval -- --preflight-only --scopes unscoped --out <private-dir>
+```
+
+Supply `DATABASE_URL` for the authorized evaluation corpus (the extension-less
+application identity). Do not use production access without separate approval.
+The standalone preflight emits only private JSON, not an empty metric report.
+Files are created exclusively with owner-only permissions; directories created
+by the harness use mode 0700. Do not publish these files.
+
+Scope is a trusted local-operator choice, **not authentication**. Like the
+existing `--scopes admin`, the following option asserts that the operator is
+already authorized to inspect all scopes. Never expose these flags through an
+untrusted API or hand an unauthorized user an all-corpus database credential.
+
+```sh
+# Only an authorized administrator may request broader freshness evidence.
+# Retrieval remains unscoped; the preflight may distinguish hidden from missing.
+npm run hybrid:shadow-eval -- --preflight-only --scopes unscoped --freshness-admin --out <private-dir>
+```
+
+Without `--freshness-admin`, preflight evidence is limited to the evaluation
+scope. `--scopes admin` already explicitly selects all scopes for both phases.
+There is no automatic privilege escalation after a lookup/search miss.
+The exported freshness helper likewise supports only unscoped (`undefined` or
+`[]`) and admin (`["*"]`) scopes; named or mixed scopes are rejected.
+
+Judgments include grade-zero entries and are reported per query/slug:
+
+- `visible`: exactly one match in the authorized evidence, visible to evaluation.
+- `excluded-by-scope`: one active match, outside evaluation scope; only emitted
+  with explicit admin evidence.
+- `corpus-absent`: no active match; only emitted with admin evidence.
+- `unknown/not-visible`: no authorized visible match. Missing and restricted
+  slugs are indistinguishable; this is **not** a claim of global absence.
+- `ambiguous`: more than one match in the authorized evidence, since article
+  slugs are unique only within a topic. No article is silently selected.
+
+The corpus here is recall-eligible: soft-deleted and recall-quarantined articles
+are excluded before aggregation, as are articles whose every provenance source
+group contains a revoked lineage or a generation/snapshot mismatch. This matches
+the final eligibility checks in both recall paths: no provenance is eligible,
+and one entirely valid source group is sufficient. A blocked duplicate cannot
+create ambiguity. `corpus-absent` means absent from this eligible corpus, not
+physical absence; no quarantine or lineage metadata/reasons are reported.
+Unscoped evidence keeps missing, restricted and privacy-ineligible rows
+indistinguishable. The single SQL statement observes a snapshot, without recall's
+serialization locks; a later revocation can invalidate this preflight.
+Draft, reviewed and published statuses are all included, matching the provider;
+this is not published-only mode. Hidden duplicates cannot be detected by unscoped evidence: `visible` and
+`clear` describe only the inspected scope. The lookup projects only fixture
+slugs and aggregate counts, not titles, tags, topic names or article IDs.
+
+`needs-review` warns about any non-visible judgment but does not stop exploratory
+measurement or alter the fixture. Before using a report for a decision, review
+these outcomes explicitly. A `clear` preflight alone is not decision-grade
+retrieval evidence. It checks presence/ambiguity, not semantic relevance, and is
+a point-in-time observation: corpus or permission changes after it can invalidate
+it. Search misses never establish stale judgments. No scores, judgments or
+metric denominators are automatically changed; fixture revisions require review.
+
+## Metric denominators
+
+For each path, `metrics.<path>.denominators` records `evaluated` and `excluded`
+query counts separately for `recall`, `ndcg`, and `mrr`. Markdown prints the same
+counts. Recall@k and nDCG@k exclude queries with no positive judgments; if every
+query is excluded, their values are null (`n/a`). MRR@limit includes every query,
+including no-positive queries and misses as zero, so its excluded count is zero.
+The MRR cutoff remains the returned limit, not k. Existing metric formulas and
+single-credit duplicate-slug ranking semantics are unchanged.
+
+## Isolated verification
+
+```sh
+npm run test:hybrid-shadow
+SHADOW_TEST_DATABASE_URL=<disposable-app-role-url> npm run test:hybrid-shadow-db
+```
+
+The database tests create connection-local temporary Article and provenance tables,
+include restricted/deleted/quarantined/duplicate and lineage-revocation fixtures,
+and never modify the application corpus.
+CI runs it against its disposable database. Production retrieval quality and
+serving acceptance remain outside these tests.

package.json

@@ -10,6 +10,7 @@
     "test:hybrid-provider": "node --test scripts/hybrid-provider.test.mjs",
     "test:hybrid-stack": "bash scripts/activate-hybrid-retrieval-stack.test.sh",
     "hybrid:shadow-eval": "tsx scripts/hybrid-shadow-eval.ts",
+    "test:hybrid-shadow-db": "node --import tsx --test scripts/hybrid-shadow-freshness.test.ts",
     "test:hybrid-shadow": "node --import tsx --test scripts/hybrid-shadow-eval.test.ts",
     "test:memory": "node --env-file=.env.test --import tsx --test 'src/__tests__/memory/**/*.test.ts'",
     "test:wiki": "node --import tsx --test 'src/__tests__/wiki/**/*.test.ts'",

scripts/hybrid-shadow-eval.test.ts

@@ -191,6 +191,43 @@ test("rapid report writes preserve aggregate JSON, JSONL and honest secret-free
   } finally { await rm(dir, { recursive: true, force: true }); }
 });
 
+test("explicit per-path denominators cover mixed, all-excluded and normal judgments", async () => {
+  const dir = await mkdtemp(path.join(tmpdir(), "denominator-test-"));
+  try {
+    for (const positives of [0, 1, 3]) {
+      const set: QuerySet = { version: 1, queries: Array.from({ length: 3 }, (_, i): QuerySet["queries"][number] => ({
+        id: `query-${i}`, query: "test query", relevance: i < positives ? { relevant: 3 } : i === 1 ? {} : { relevant: 0 },
+      })) };
+      const keyword = await rankings(set, [[row("relevant")], [], [row("zero"), row("relevant")]], "keyword");
+      const hybrid = await rankings(set, [[], [], []]);
+      const report = buildReport(set, keyword, hybrid, parseArgs(["--k", "1"]), {});
+      for (const p of ["keyword", "hybrid"]) assert.deepEqual(report.metrics[p].denominators, {
+        recall: { evaluated: positives, excluded: 3 - positives },
+        ndcg: { evaluated: positives, excluded: 3 - positives },
+        mrr: { evaluated: 3, excluded: 0 },
+      });
+      assert.equal(report.metrics.keyword.recall, positives ? 1 / positives : null);
+      assert.equal(report.metrics.keyword.ndcg, positives ? 1 / positives : null);
+      assert.equal(report.metrics.keyword.mrr, positives === 3 ? 0.5 : positives ? 1 / 3 : 0);
+      assert.equal(report.metrics.hybrid.recall, positives ? 0 : null);
+      assert.equal(report.metrics.hybrid.mrr, 0);
+      const files = await writeReport(report, dir);
+      assert.deepEqual(JSON.parse(await readFile(files.aggregatePath, "utf8")).metrics, report.metrics);
+      const md = await readFile(files.summaryPath, "utf8");
+      for (const p of ["keyword", "hybrid"]) assert.ok(md.includes(`| ${p} | ${positives} | ${3 - positives} | ${positives} | ${3 - positives} | 3 | 0 |`));
+      assert.match(md, /MRR all-query denominator/);
+      assert.match(md, /Not checked; not decision-grade/);
+    }
+  } finally { await rm(dir, { recursive: true, force: true }); }
+});
+
+test("preflight flags require explicit admin evidence opt-in", () => {
+  assert.equal(parseArgs([]).freshnessAdmin, undefined);
+  assert.equal(parseArgs(["--preflight-only"]).preflightOnly, true);
+  assert.equal(parseArgs(["--freshness-admin"]).freshnessAdmin, true);
+  assert.equal(parseArgs(["--freshness-admin"]).scopes, undefined);
+});
+
 test("pure import has no provider/Prisma initialization; CLI rejects arguments before runtime setup", () => {
   const script = path.resolve(import.meta.dirname, "hybrid-shadow-eval.ts");
   const env = { ...process.env, DATABASE_URL: "", NOOSPHERE_HYBRID_QUERY_PROFILE_ID: "" };

scripts/hybrid-shadow-eval.ts

@@ -14,8 +14,10 @@
  * No article edits or corpus-vector writes are requested by this harness.
  * Reports can contain restricted article titles; keep them private.
  *
- * Usage (always dual-path; unreachable hybrid services may cause fallback):
+ * Usage (dual-path evaluation by default; --preflight-only performs only the
+ * freshness lookup and writes no metric reports — see docs/HYBRID-SHADOW-EVALUATION.md):
  *   npm run hybrid:shadow-eval -- --limit 5 --k 5 --out <dir>
+ *   npm run hybrid:shadow-eval -- --preflight-only --out <dir>
  *
  * Full dual-path run: must execute inside the compose network so the pinned
  * provider endpoint (host.docker.internal:8741) resolves, with the app-role
@@ -46,6 +48,7 @@ import { pathToFileURL } from "node:url";
 import type { MemoryProvider } from "@/lib/memory/provider";
 import type { MemoryResult } from "@/lib/memory/types";
 import { HYBRID_MAX_WINDOW } from "@/lib/memory/hybrid-ranking";
+import { checkFixtureFreshness, type FixtureFreshness } from "./hybrid-shadow-freshness";
 
 interface GradedQuery {
   id: string;
@@ -76,9 +79,11 @@ function requireEnv(name: string): string {
   return value;
 }
 
-export function parseArgs(argv: string[]): { limit: number; k: number; outDir: string; scopes: string[] | undefined } {
-  const opts = { limit: 10, k: 5, outDir: "hybrid-shadow-reports", scopes: undefined as string[] | undefined };
+export function parseArgs(argv: string[]): { limit: number; k: number; outDir: string; scopes: string[] | undefined; preflightOnly?: boolean; freshnessAdmin?: boolean } {
+  const opts: ReturnType<typeof parseArgs> = { limit: 10, k: 5, outDir: "hybrid-shadow-reports", scopes: undefined as string[] | undefined };
   for (let i = 0; i < argv.length; i += 1) {
+    if (argv[i] === "--preflight-only") { opts.preflightOnly = true; continue; }
+    if (argv[i] === "--freshness-admin") { opts.freshnessAdmin = true; continue; }
     if (!["--limit", "--k", "--out", "--scopes"].includes(argv[i])) throw new Error(`unknown argument: ${argv[i]}`);
     if (!argv[i + 1]?.trim() || argv[i + 1].startsWith("--")) throw new Error(`missing value for ${argv[i]}`);
     if (argv[i] === "--limit") opts.limit = Number(argv[++i]);
@@ -207,8 +212,9 @@ export function buildReport(
   hybridRankings: Ranking[],
   opts: ReturnType<typeof parseArgs>,
   environment: Readonly<Record<string, string | undefined>>,
+  freshness: FixtureFreshness | null = null,
 ) {
-  const metrics: Record<string, { path: string; recall: number | null; ndcg: number | null; mrr: number | null; latencyP50: number; fallbacks: number; fallbackUnknown: number }> = {};
+  const metrics: Record<string, { path: string; recall: number | null; ndcg: number | null; mrr: number | null; latencyP50: number; fallbacks: number; fallbackUnknown: number; denominators: { recall: { evaluated: number; excluded: number }; ndcg: { evaluated: number; excluded: number }; mrr: { evaluated: number; excluded: number } } }> = {};
   for (const [path, rankings] of [["keyword", keywordRankings], ["hybrid", hybridRankings]] as const) {
     let recallSum = 0, recallN = 0, ndcgSum = 0, ndcgN = 0, mrrSum = 0, fallbacks = 0, fallbackUnknown = 0;
     const latencies: number[] = [];
@@ -235,6 +241,11 @@ export function buildReport(
       latencyP50: latencies[Math.floor(latencies.length / 2)] ?? 0,
       fallbacks,
       fallbackUnknown,
+      denominators: {
+        recall: { evaluated: recallN, excluded: rankings.length - recallN },
+        ndcg: { evaluated: ndcgN, excluded: rankings.length - ndcgN },
+        mrr: { evaluated: rankings.length, excluded: 0 },
+      },
     };
   }
 
@@ -245,6 +256,7 @@ export function buildReport(
     limit: opts.limit,
     k: opts.k,
     relevanceTiers: RELEVANCE_TIERS,
+    freshness,
     observation: {
       scope: opts.scopes?.includes("*") ? "admin (all scopes, all statuses)" : "unscoped (unrestricted articles, all statuses)",
       redis: environment.REDIS_URL ? "configured" : "not configured",
@@ -282,6 +294,20 @@ export async function writeReport(report: ReturnType<typeof buildReport>, outDir
     const m = report.metrics[p];
     md.push(`| ${p} | ${m.recall?.toFixed(3) ?? "n/a"} | ${m.ndcg?.toFixed(3) ?? "n/a"} | ${m.mrr?.toFixed(3) ?? "n/a"} | ${m.latencyP50} | ${m.fallbacks} | ${m.fallbackUnknown} |`);
   }
+  md.push("", "| path | recall evaluated | recall excluded | nDCG evaluated | nDCG excluded | MRR all-query denominator | MRR excluded |",
+    "| --- | --- | --- | --- | --- | --- | --- |");
+  for (const p of ["keyword", "hybrid"]) {
+    const d = report.metrics[p].denominators;
+    md.push(`| ${p} | ${d.recall.evaluated} | ${d.recall.excluded} | ${d.ndcg.evaluated} | ${d.ndcg.excluded} | ${d.mrr.evaluated} | ${d.mrr.excluded} |`);
+  }
+  md.push("", "## Fixture freshness");
+  if (report.freshness) {
+    const f = report.freshness;
+    md.push("", `Evaluation scope: ${f.evaluationScope}; evidence scope: ${f.evidenceScope}.`,
+      `Started: ${f.startedAt}; checked: ${f.checkedAt}; outcome: ${f.outcome}.`, "", f.limitation,
+      "", ...Object.entries(f.counts).map(([status, count]) => `- ${status}: ${count}`),
+      "", "Per-judgment outcomes are in the private aggregate JSON.");
+  } else md.push("", "Not checked; not decision-grade evidence.");
   md.push(``, `Full per-query rankings: \`${path.basename(jsonl)}\``);
   md.push(`Aggregate report: \`${path.basename(aggregatePath)}\``);
   await writeFile(summaryPath, md.join("\n") + "\n", { flag: "wx", mode: 0o600 });
@@ -293,19 +319,28 @@ async function main(): Promise<void> {
   const fixturePath = path.resolve(import.meta.dirname, "../src/__tests__/fixtures/hybrid-shadow-queries.json");
   const querySet = loadQuerySet(await readFile(fixturePath, "utf8"));
   const databaseUrl = requireEnv("DATABASE_URL");
-  const baseEnv = {
+  const baseEnv = opts.preflightOnly ? process.env : {
     ...process.env,
     NOOSPHERE_HYBRID_QUERY_PROFILE_ID: requireEnv("NOOSPHERE_HYBRID_QUERY_PROFILE_ID"),
     NOOSPHERE_HYBRID_CACHE_HMAC_ACTIVE_VERSION: requireEnv("NOOSPHERE_HYBRID_CACHE_HMAC_ACTIVE_VERSION"),
     NOOSPHERE_HYBRID_CACHE_HMAC_KEYS_B64: requireEnv("NOOSPHERE_HYBRID_CACHE_HMAC_KEYS_B64"),
   };
-  // Pure imports above never initialize Prisma, Redis, or a provider.
-  const [{ PrismaClient }, { PrismaPg }, { Pool }, { createNoosphereProvider }, { closeRedisClient }] = await Promise.all([
-    import("@prisma/client"), import("@prisma/adapter-pg"), import("pg"),
-    import("@/lib/memory/noosphere"), import("@/lib/cache/redis"),
-  ]);
+  // Preflight-only never imports Prisma, Redis, or the search provider.
+  const { Pool } = await import("pg");
   const pool = new Pool({ connectionString: databaseUrl, max: 2 });
   try {
+    const freshness = await checkFixtureFreshness(pool, querySet, opts.scopes, opts.freshnessAdmin);
+    // Persist preflight even if later retrieval fails. No search/cache/embedding
+    // work is requested by preflight-only. Do not treat it as a metric report.
+    await mkdir(opts.outDir, { recursive: true, mode: 0o700 });
+    const freshnessPath = path.join(opts.outDir, `freshness-${randomUUID()}.json`);
+    await writeFile(freshnessPath, JSON.stringify({ querySetVersion: querySet.version, freshness }, null, 2) + "\n", { flag: "wx", mode: 0o600 });
+    process.stdout.write(`freshness: ${freshness.outcome}; evidence: ${freshnessPath}\n`);
+    if (opts.preflightOnly) return;
+    const [{ PrismaClient }, { PrismaPg }, { createNoosphereProvider }, { closeRedisClient }] = await Promise.all([
+      import("@prisma/client"), import("@prisma/adapter-pg"),
+      import("@/lib/memory/noosphere"), import("@/lib/cache/redis"),
+    ]);
     const prisma = new PrismaClient({ adapter: new PrismaPg(pool) });
     try {
       const keywordProvider = createNoosphereProvider({ prisma, allowedScopes: opts.scopes,
@@ -315,13 +350,13 @@ async function main(): Promise<void> {
       process.stdout.write(`shadow eval: ${querySet.queries.length} queries, limit ${opts.limit}, k ${opts.k}, scopes ${opts.scopes ? "admin (all scopes, all statuses)" : "unscoped (unrestricted articles, all statuses)"}\n`);
       const keywordRankings = await runPath("keyword", keywordProvider, querySet, opts.limit);
       const hybridRankings = await runPath("hybrid", hybridProvider, querySet, opts.limit);
-      const files = await writeReport(buildReport(querySet, keywordRankings, hybridRankings, opts, baseEnv), opts.outDir);
+      const files = await writeReport(buildReport(querySet, keywordRankings, hybridRankings, opts, baseEnv, freshness), opts.outDir);
       process.stdout.write(`\nreport: ${files.jsonl}\naggregate: ${files.aggregatePath}\nsummary: ${files.summaryPath}\n`);
     } finally {
-      await prisma.$disconnect();
+      try { await prisma.$disconnect(); } finally { await closeRedisClient(); }
     }
   } finally {
-    try { await pool.end(); } finally { await closeRedisClient(); }
+    await pool.end();
   }
 }