Smarter dedup: trend direction, outcome-based cross-segment, metric aliases

Three dedup improvements to eliminate ~50% of duplicate discoveries:

1. Trend dedup by metric + direction (ignore change_pct): prevents the
   same trend from re-surfacing daily as the lookback window shifts
2. Outcome-based cross-segment dedup (±10pp): catches mirror discoveries
   from definitionally coupled segments (sleep_duration vs time_in_bed)
3. Metric alias groups: hrv/hrv_daily, steps/intraday_steps,
   sleep_duration/time_in_bed treated as equivalent during dedup

Existing discovery dedup also updated with alias + trend awareness.
32 pattern-ranker tests (14 new), 21 pattern-spotter vitest passing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: schwaaamp

docs/planning/insight-engine-v3.md

@@ -1591,14 +1591,208 @@ catch:
 - `_shared/pattern-ranker.ts` — no changes needed
 - All analyzer modules — no changes needed
 
-**Status: DONE (2026-03-19)**
+**Status: DONE (2026-03-19, batched narration live)**
 - Batched AI narration live: single GPT-4o-mini call per pipeline run
 - All discoveries and observations get plain-language titles and summaries
 - OpenAI json_object wrapper format handled (extracts array from `{ "discoveries": [...] }`)
 - Template fallback on AI failure — 21 vitest passing
 - Admin dashboard updated to show v3 narration diagnostics (mode, patterns narrated, AI succeeded)
 - Experiment matching NOT YET IMPLEMENTED — `suggested_experiment_id` still null (deferred to next iteration)
 
+### 8.5 Deduplication Improvements (Step 9: FILTER)
+
+#### 8.5.1 The Problem
+
+Empirical review of user 3597587c's 67 active discoveries (2026-03-23) revealed ~34 are duplicates or near-duplicates that the current dedup fails to catch. Three root causes:
+
+**Problem 1: Trends re-surface daily with different change_pct values**
+
+The same trend (e.g., "workout frequency increasing") appears 4 times across 4 days because the lookback window shifts and the change_pct swings (79% → 2700% → 1300% → 833%). The dedup requires `change_pct` within ±5 percentage points — but these are 10-100x apart.
+
+Examples from real data:
+- "Workout Frequency Has Increased" — 4 copies (March 19-22)
+- "HRV Decreasing" — 2 copies per metric variant
+- "Steps Increasing" — 4 copies across steps/intraday_steps
+- "High Activity Increasing" — 3 copies
+
+**Problem 2: Coupled segment metrics produce mirror discoveries**
+
+`sleep_duration` and `time_in_bed` segment days nearly identically (r≈0.95). Every discovery from one is duplicated by the other — same outcome, same change_pct, different segment name. The dedup requires matching `segment_metric_key`, so it treats them as distinct.
+
+7 mirror pairs found:
+- "Wider Glucose Range on Low Sleep Days" / "...on Low Time in Bed Days" (+37%)
+- "Higher Max Glucose on Low Sleep Days" / "...on Low Time in Bed Days" (+20.8%)
+- "More Sedentary on Low Sleep Days" / "...on Low Time in Bed Days" (+16.3%)
+- etc.
+
+**Problem 3: Metric aliases produce identical discoveries**
+
+`hrv` is literally `hrv_daily` for WHOOP users (the derived metric falls back). `steps` equals `intraday_steps` (daily sum of same data). Each discovery appears twice — once per metric variant.
+
+8 alias duplicates found:
+- "HRV Decreasing" / "Daily HRV Decreasing" (both -7.6%)
+- "Improved HRV After Low Activity" / "Higher Daily HRV After Low Activity" (both +15.3%)
+- "Steps Increasing" / "Intraday Steps Increasing" (both ~30-35%)
+
+#### 8.5.2 The Fixes
+
+All three fixes are changes to `areDuplicates()` and `deduplicatePatterns()` in `pattern-ranker.ts`. No pipeline or analyzer changes needed.
+
+**Fix 1: Trend dedup by metric_key + direction (ignore change_pct)**
+
+For trend candidates, the current rule `metric_key + segment_metric_key + change_pct ±5%` is too narrow. Change_pct for trends varies wildly as the lookback window shifts.
+
+New rule for trends: Two trend candidates are duplicates if:
+- Same `metric_key` (or aliases — see Fix 3)
+- Same `direction` (both increasing or both decreasing)
+- `change_pct` is ignored for trends
+
+This also applies to existing discovery dedup: if an active discovery with the same metric_key and direction already exists, the new trend is a duplicate regardless of change_pct.
+
+**Fix 2: Outcome-based dedup for segment comparisons**
+
+Two segment comparison candidates are duplicates if:
+- Same `metric_key` (outcome)
+- `change_pct` within ±10% (wider tolerance for cross-segment dedup)
+- `segment_metric_key` can differ (this is the key relaxation)
+
+This catches the sleep_duration/time_in_bed mirrors. When "Low Sleep Days → glucose_range +37%" and "Low Time in Bed Days → glucose_range +37%" both survive BH, the second is deduped because the outcome and change match.
+
+The wider ±10% tolerance (vs the current ±5%) accounts for slight differences when two correlated segment metrics don't segment days exactly the same way.
+
+**Fix 3: Metric alias groups**
+
+Define alias sets where metrics produce identical or near-identical values for a given user:
+
+```typescript
+const METRIC_ALIAS_GROUPS: string[][] = [
+  ['hrv', 'hrv_daily'],            // hrv = hrv_daily ?? hrv_sleep (WHOOP users: always hrv_daily)
+  ['steps', 'intraday_steps'],     // daily total = sum of intraday
+  ['sleep_duration', 'time_in_bed'], // definitionally coupled (r≈0.95)
+];
+```
+
+During dedup, two metric keys are considered equivalent if they belong to the same alias group. This means:
+- "HRV Decreasing" and "Daily HRV Decreasing" → same finding
+- "Steps Increasing" and "Intraday Steps Increasing" → same finding
+- Any discovery with outcome `sleep_duration` matches against one with `time_in_bed`
+- Any segment using `sleep_duration` is equivalent to one using `time_in_bed`
+
+The alias check is used in BOTH within-batch dedup AND existing-discovery dedup.
+
+#### 8.5.3 Updated `areDuplicates` Logic
+
+```typescript
+const METRIC_ALIAS_GROUPS: string[][] = [
+  ['hrv', 'hrv_daily'],
+  ['steps', 'intraday_steps'],
+  ['sleep_duration', 'time_in_bed'],
+];
+
+// Pre-computed lookup: metric_key → canonical representative
+const METRIC_CANONICAL: Map<string, string> = new Map();
+for (const group of METRIC_ALIAS_GROUPS) {
+  const canonical = group[0]; // first entry is the canonical
+  for (const key of group) {
+    METRIC_CANONICAL.set(key, canonical);
+  }
+}
+
+function canonicalKey(metricKey: string): string {
+  return METRIC_CANONICAL.get(metricKey) ?? metricKey;
+}
+
+function areDuplicates(a: PatternCandidate, b: PatternCandidate): boolean {
+  const aOutcome = canonicalKey(a.metric_key);
+  const bOutcome = canonicalKey(b.metric_key);
+
+  // Rule 1: Trend dedup — same metric + same direction (ignore change_pct)
+  if (a.type === 'trend' && b.type === 'trend') {
+    return aOutcome === bOutcome && a.direction === b.direction;
+  }
+
+  // Rule 2: Outcome-based dedup — same outcome + similar change (segment can differ)
+  if (aOutcome === bOutcome) {
+    // If segments are also aliases, use tighter threshold
+    const aSegment = canonicalKey(a.segment_metric_key ?? '');
+    const bSegment = canonicalKey(b.segment_metric_key ?? '');
+    const threshold = aSegment === bSegment ? 5 : 10;
+    if (Math.abs(a.change_pct - b.change_pct) <= threshold) {
+      return true;
+    }
+  }
+
+  return false;
+}
+```
+
+#### 8.5.4 Existing Discovery Dedup Enhancement
+
+The existing discovery dedup also needs to understand aliases and trend direction. The `existingForDedup` data currently only carries `metric_key` and `change_pct`. To support trend dedup, it needs the `pattern_type` and `direction` (if trend) from `metrics_impact`.
+
+Update the existing discovery query to include `pattern_type` from metrics_impact:
+
+```typescript
+const existingForDedup = allExisting.map(d => ({
+  metrics_impact: d.metrics_impact as Array<{
+    metric_key: string;
+    change_pct: number;
+    pattern_type?: string;
+  }> | null,
+  discovery_type: d.discovery_type,
+  title: d.title, // title contains direction hint for trends
+}));
+```
+
+For trend matching against existing: if the existing discovery's `pattern_type === 'trend'` and the canonical metric_key matches, treat as duplicate regardless of change_pct.
+
+#### 8.5.5 Impact Estimate
+
+For user 3597587c (67 discoveries):
+- Fix 1 (trend dedup): eliminates ~12 repeat trends
+- Fix 2 (outcome-based dedup): eliminates ~14 mirror segment discoveries
+- Fix 3 (metric aliases): eliminates ~8 alias duplicates
+
+Total: ~34 eliminations → **67 → ~33 unique discoveries**
+
+For user 73f1a17e (7 discoveries):
+- 1 duplicate removed (repeat lagged effect)
+- **7 → 6 unique discoveries**
+
+#### 8.5.6 Data Cleanup
+
+After deploying the fix, existing duplicate discoveries need to be cleaned. Two approaches:
+
+**Option A: Delete all and re-run**
+```sql
+DELETE FROM user_discoveries
+WHERE discovery_type IN ('unenrolled_pattern', 'observation')
+  AND status IN ('new', 'viewed');
+-- Then invoke spot-patterns-cron to regenerate
+```
+
+**Option B: Keep highest-ranked of each duplicate set** (preserves viewed status)
+More complex — requires a script to identify duplicate groups and delete all but the best.
+
+Recommendation: Option A (delete + re-run). The AI narration will regenerate fresh text, and the new dedup logic will prevent duplicates from returning.
+
+#### 8.5.7 Implementation Scope
+
+**Files to modify:**
+- `_shared/pattern-ranker.ts` — rewrite `areDuplicates()`, add alias groups, update existing dedup
+- `_shared/pattern-ranker.test.ts` — new tests for trend dedup, outcome-based dedup, alias groups
+- `ai-engine/engines/pattern-spotter.ts` — update `existingForDedup` to include pattern_type
+
+**Files unchanged:**
+- All analyzers, metric-discovery, blacklist, BH families — no changes needed
+
+**Status: DONE (2026-03-23)**
+- `areDuplicates()` rewritten with three rules: trend direction dedup, outcome-based cross-segment dedup, metric aliases
+- `canonicalKey()` exported for alias resolution (hrv↔hrv_daily, steps↔intraday_steps, sleep_duration↔time_in_bed)
+- `deduplicatePatterns()` updated: existing discovery matching uses aliases + trend direction awareness
+- 32 pattern-ranker tests passing (14 new), 21 pattern-spotter vitest passing
+- Expected reduction: ~67 → ~33 unique discoveries for user 3597587c after re-run
+
 ---
 
 ## 9. Migration Strategy

supabase/functions/_shared/pattern-ranker.test.ts

@@ -1,6 +1,7 @@
 import { assertEquals } from 'https://deno.land/std@0.224.0/assert/mod.ts';
 import {
   areDuplicates,
+  canonicalKey,
   classifyCandidate,
   compositeScore,
   deduplicatePatterns,
@@ -101,6 +102,24 @@ Deno.test('rankCandidates: single candidate → rank 1', () => {
   assertEquals(ranked[0].rank, 1);
 });
 
+// ── canonicalKey ────────────────────────────────────────────────────────────
+
+Deno.test('canonicalKey: hrv_daily → hrv (alias)', () => {
+  assertEquals(canonicalKey('hrv_daily'), 'hrv');
+});
+
+Deno.test('canonicalKey: intraday_steps → steps (alias)', () => {
+  assertEquals(canonicalKey('intraday_steps'), 'steps');
+});
+
+Deno.test('canonicalKey: time_in_bed → sleep_duration (alias)', () => {
+  assertEquals(canonicalKey('time_in_bed'), 'sleep_duration');
+});
+
+Deno.test('canonicalKey: resting_hr → resting_hr (no alias, returns self)', () => {
+  assertEquals(canonicalKey('resting_hr'), 'resting_hr');
+});
+
 // ── areDuplicates ───────────────────────────────────────────────────────────
 
 Deno.test('areDuplicates: same metric_key + segment_metric_key + similar change → true', () => {
@@ -115,12 +134,80 @@ Deno.test('areDuplicates: different metric_key → false', () => {
   assertEquals(areDuplicates(a, b), false);
 });
 
-Deno.test('areDuplicates: same metric_key but change_pct differs by > 5 → false', () => {
+Deno.test('areDuplicates: same metric_key but change_pct differs by > 5 with same segment → false', () => {
   const a = makeCandidate({ change_pct: 10.0 });
-  const b = makeCandidate({ change_pct: 20.0 }); // 10 pp difference
+  const b = makeCandidate({ change_pct: 20.0 }); // 10 pp difference, same segment
+  assertEquals(areDuplicates(a, b), false);
+});
+
+// ── Fix 1: Trend dedup by metric + direction (ignore change_pct) ───────────
+
+Deno.test('areDuplicates: two trends, same metric, same direction, wildly different change_pct → true', () => {
+  const a = makeCandidate({ type: 'trend', metric_key: 'has_workout', direction: 'increasing', change_pct: 79.7 });
+  const b = makeCandidate({ type: 'trend', metric_key: 'has_workout', direction: 'increasing', change_pct: 2700 });
+  assertEquals(areDuplicates(a, b), true);
+});
+
+Deno.test('areDuplicates: two trends, same metric, different direction → false', () => {
+  const a = makeCandidate({ type: 'trend', metric_key: 'hrv', direction: 'increasing', change_pct: 10 });
+  const b = makeCandidate({ type: 'trend', metric_key: 'hrv', direction: 'decreasing', change_pct: -10 });
+  assertEquals(areDuplicates(a, b), false);
+});
+
+Deno.test('areDuplicates: trend vs segment_comparison same metric → false (different types)', () => {
+  const a = makeCandidate({ type: 'trend', metric_key: 'hrv', direction: 'decreasing', change_pct: -14 });
+  const b = makeCandidate({ type: 'segment_comparison', metric_key: 'hrv', change_pct: -14 });
+  assertEquals(areDuplicates(a, b), false);
+});
+
+// ── Fix 2: Outcome-based dedup (different segment, same outcome + change) ──
+
+Deno.test('areDuplicates: same outcome, similar change, different segment → true (±10%)', () => {
+  const a = makeCandidate({ metric_key: 'glucose_range', segment_metric_key: 'sleep_duration', change_pct: 37 });
+  const b = makeCandidate({ metric_key: 'glucose_range', segment_metric_key: 'time_in_bed', change_pct: 37 });
+  assertEquals(areDuplicates(a, b), true);
+});
+
+Deno.test('areDuplicates: same outcome, change differs by 8% (within ±10), different segment → true', () => {
+  const a = makeCandidate({ metric_key: 'sedentary_min', segment_metric_key: 'sleep_duration', change_pct: -20.9 });
+  const b = makeCandidate({ metric_key: 'sedentary_min', segment_metric_key: 'time_in_bed', change_pct: -16.3 });
+  // Difference is 4.6 pp, segments are aliases → uses 5 pp threshold
+  assertEquals(areDuplicates(a, b), true);
+});
+
+Deno.test('areDuplicates: same outcome, change differs by 12%, unrelated segments → false', () => {
+  const a = makeCandidate({ metric_key: 'sedentary_min', segment_metric_key: 'avg_glucose', change_pct: 20 });
+  const b = makeCandidate({ metric_key: 'sedentary_min', segment_metric_key: 'strain', change_pct: 32 });
+  // Difference is 12 pp, segments are NOT aliases → uses 10 pp threshold → 12 > 10 → false
   assertEquals(areDuplicates(a, b), false);
 });
 
+// ── Fix 3: Metric aliases ──────────────────────────────────────────────────
+
+Deno.test('areDuplicates: hrv and hrv_daily trends with same direction → true (aliases)', () => {
+  const a = makeCandidate({ type: 'trend', metric_key: 'hrv', direction: 'decreasing', change_pct: -7.6 });
+  const b = makeCandidate({ type: 'trend', metric_key: 'hrv_daily', direction: 'decreasing', change_pct: -7.6 });
+  assertEquals(areDuplicates(a, b), true);
+});
+
+Deno.test('areDuplicates: steps and intraday_steps trends → true (aliases)', () => {
+  const a = makeCandidate({ type: 'trend', metric_key: 'steps', direction: 'increasing', change_pct: 30.8 });
+  const b = makeCandidate({ type: 'trend', metric_key: 'intraday_steps', direction: 'increasing', change_pct: 33.5 });
+  assertEquals(areDuplicates(a, b), true);
+});
+
+Deno.test('areDuplicates: sleep_duration segment vs time_in_bed segment, same outcome → true (segment aliases)', () => {
+  const a = makeCandidate({ metric_key: 'glucose_max', segment_metric_key: 'sleep_duration', change_pct: 20.8 });
+  const b = makeCandidate({ metric_key: 'glucose_max', segment_metric_key: 'time_in_bed', change_pct: 20.8 });
+  assertEquals(areDuplicates(a, b), true);
+});
+
+Deno.test('areDuplicates: hrv outcome and hrv_daily outcome, same segment → true (outcome aliases)', () => {
+  const a = makeCandidate({ type: 'lagged_effect', metric_key: 'hrv', segment_metric_key: 'wake_hour', change_pct: 14.8 });
+  const b = makeCandidate({ type: 'lagged_effect', metric_key: 'hrv_daily', segment_metric_key: 'wake_hour', change_pct: 14.8 });
+  assertEquals(areDuplicates(a, b), true);
+});
+
 // ── deduplicatePatterns ─────────────────────────────────────────────────────
 
 Deno.test('deduplicatePatterns: removes candidates matching existing discoveries', () => {