Skip to content

unified-experiment-analysis.md

Latest commit

 

History

History
260 lines (202 loc) · 12.1 KB

unified-experiment-analysis.md

File metadata and controls

260 lines (202 loc) · 12.1 KB

Unified Experiment Analysis Strategy

Context

The experiment analysis system had two independent code paths that produced different results in different tables:

  1. Old path (mobile-triggered): analyze-experiment/index.ts → writes to experiment_results table with simple verdict model. No logging to app_logs. Mobile hooks (useCompletionCheck, useExperimentResults) called this directly.

  2. New path (cron-triggered): auto-complete-experiments → experiment-analyst.ts → writes to experiment_outcomes + user_discoveries with Magnitude of Impact model. Logs to app_logs via Logger.

Both could run on the same experiment, producing conflicting analyses in different tables. The old analyze-experiment edge function had zero observability — no app_logs entries, no diagnostics persistence, no AI call tracking. Admins had no dashboard to review experiment analyses.

Goal: Single analysis engine, single results table, full diagnostics for admin observability, and an /admin/experiments dashboard.

What was deleted

Item Reason
supabase/functions/analyze-experiment/ (entire directory) Replaced by ai-engine/engines/experiment-analyst.ts via /analyze route
experiment_results table Redundant — experiment_outcomes has a superset of its data. Migration drops the table after data backfill.

What was kept

Item Role
ai-engine/engines/experiment-analyst.ts Single analysis engine for both paths
experiment_outcomes table Canonical analysis results (with new diagnostics JSONB column)
user_discoveries table User-facing results (no change)
auto-complete-experiments/index.ts Cron trigger (already calls the new engine — no change)

Implementation Steps

Step 1: Add diagnostics persistence to experiment-analyst.ts — COMPLETE

Created _shared/persistExperimentDiagnostics.ts following the persistRunDiagnostics.ts pattern. Writes a comprehensive diagnostics blob to experiment_outcomes.diagnostics (new JSONB column) on every analysis run.

Migration: Added diagnostics JSONB column to experiment_outcomes.

Diagnostics blob structure:

{
  started_at: string;
  elapsed_ms: number;
  baseline: { start: string; end: string; days: number };
  experiment: { start: string; end: string; days: number };
  metrics_analyzed: number;
  metric_details: Array<{
    metric_key: string;
    metric_label: string;
    data_source: string;
    baseline: { count: number; mean: number; stddev: number; min: number; max: number };
    experiment: { count: number; mean: number; stddev: number; min: number; max: number };
    change_pct: number;
    effect_size: number;
    effect_magnitude: string;
    direction: string;
    insufficient_data: boolean;
  }>;
  adherence: { total_checkins: number; adhered: number; rate_pct: number };
  confounders: {
    detected: string[];
    baseline_supplements: string[];
    experiment_supplements: string[];
  };
  concurrent: { count: number; ids: string[]; attribution_confidence: string };
  ai: {
    model: string;
    prompt_version: string;
    system_prompt_hash: string;
    user_prompt_length: number;
    response_length: number;
    latency_ms: number;
    compliance_corrections: string[];
    fallback_used: boolean;
  };
  outcome: {
    overall_magnitude: string;
    confidence: string;
    attribution_confidence: string;
    discovery_created: boolean;
    discovery_id: string | null;
  };
}

Changes to experiment-analyst.ts:

  • Builds diagnostics object throughout the function
  • Persists via the new helper alongside the existing experiment_outcomes upsert
  • computeStats extended to return min/max for per-metric diagnostics
  • SHA-256 hash of system prompt stored for change detection

Files:

  • Created: supabase/functions/_shared/persistExperimentDiagnostics.ts
  • Modified: supabase/functions/ai-engine/engines/experiment-analyst.ts
  • Created: supabase/migrations/20260323000000_unify_experiment_analysis.sql

Step 2: Redirect mobile app to use ai-engine/analyze instead of analyze-experiment — COMPLETE

The mobile hooks previously called the old edge function directly. Redirected them to use analyzeExperiment() from experimentAIClient.ts, which calls the ai-engine /analyze route.

useCompletionCheck.ts:

  • Changed analysis trigger from supabase.functions.invoke('analyze-experiment', ...) to analyzeExperiment(experiment.id) from experimentAIClient.ts

useExperimentResults.ts:

  • Changed query to read from experiment_outcomes instead of experiment_results
  • Fetches associated user_discoveries for the AI narrative (summary_text, detailed_analysis)
  • Changed trigger mutation to use analyzeExperiment() from experimentAIClient.ts

types.ts:

  • Updated ExperimentResult interface to match experiment_outcomes schema + joined discovery fields
  • Old MetricResult interface kept (deprecated) for backward compatibility with remaining test utilities

VerdictBanner.tsx:

  • Updated props from verdict/confidenceLevel to magnitude/confidence
  • Updated display configs: magnitude levels (high/moderate/low/minimal/inconclusive) instead of verdicts (positive/negative/neutral/inconclusive)

[id].tsx (experiment detail screen):

  • Updated VerdictBanner usage to new props
  • Updated metric rendering from metric_results to metric_changes (new field names: baseline_mean, experiment_mean, effect_size_cohens_d, etc.)
  • Updated confounders section from confounders_detected to confounders_present
  • Updated adopt button condition from overall_verdict === "positive" to overall_magnitude === "high" || overall_magnitude === "moderate"

Files:

  • Modified: mobile/src/hooks/useExperimentResults.ts
  • Modified: mobile/src/hooks/useCompletionCheck.ts
  • Modified: mobile/src/utils/experiments/types.ts
  • Modified: mobile/src/components/Experiments/VerdictBanner.tsx
  • Modified: mobile/src/app/experiment/[id].tsx

Step 3: Migration — backfill + drop experiment_results — COMPLETE

Migration: supabase/migrations/20260323000000_unify_experiment_analysis.sql

  1. Adds diagnostics JSONB column to experiment_outcomes
  2. Backfills experiments that have experiment_results but no experiment_outcomes — maps old verdict model to magnitude model (positive→moderate, negative→moderate, neutral→minimal, inconclusive→inconclusive) and confidence (high→strong, moderate→moderate, low→suggestive)
  3. Drops RLS policies on experiment_results
  4. Drops the experiment_results table

Files:

  • Created: supabase/migrations/20260323000000_unify_experiment_analysis.sql

Step 4: Delete old analyze-experiment edge function — COMPLETE

Removed the entire supabase/functions/analyze-experiment/ directory. The ai-engine /analyze route handles all analysis now.

Files:

  • Deleted: supabase/functions/analyze-experiment/ (entire directory)

Step 5: Admin dashboard — /admin/experiments — COMPLETE

Created two new routes following the insights dashboard pattern:

web/src/app/routes/admin/experiments.tsx — Listing page

  • Loader queries experiment_outcomes + experiments (for title/variable)
  • Filter controls: days (7/14/30/90), user, magnitude
  • Stats grid: Total Analyzed, Avg Adherence %, Magnitude breakdown, Attribution Confidence breakdown
  • Table: Date, User, Experiment Title, Duration, Adherence, Metrics count, Magnitude, Confidence, Attribution, Diagnostics available, Detail link

web/src/app/routes/admin/experiments.$outcomeId.tsx — Detail page

  • Loader queries: experiment_outcomes, experiments, user_discoveries, experiment_checkins, concurrent experiments
  • Summary stats: Duration, Adherence, Metrics, Elapsed time
  • 6 tabs:
    1. Overview — experiment metadata, dates, status, AI model/version
    2. Metrics — per-metric table with full diagnostics (n, mean, stddev, min, max for both periods), change%, Cohen's d, magnitude, direction, data sufficiency indicator. Falls back to metric_changes if no diagnostics.
    3. Confounders — detected confounders list, baseline vs experiment supplements/medications
    4. Concurrent — concurrent experiments table, attribution confidence, attribution map
    5. AI Analysis — model, prompt version, system prompt hash, response latency, compliance corrections, fallback indicator
    6. User-Facing — discovery title, summary, detailed analysis, status

Nav link: Added "Experiments" to AdminNav() in web/src/app/root.tsx.

Files:

  • Created: web/src/app/routes/admin/experiments.tsx
  • Created: web/src/app/routes/admin/experiments.$outcomeId.tsx
  • Modified: web/src/app/root.tsx

Step 6: Add Logger to experiment-analyst.ts — COMPLETE

Replaced all console.log/console.error calls in experiment-analyst.ts with structured logging via the Logger class that writes to app_logs.

Key log points:

  • Analysis started (experiment_id, user_id)
  • Data fetch summary (metrics count, baseline/experiment row counts)
  • Confounder detection results
  • AI call start/end with latency
  • AI fallback triggered (with error reason)
  • Compliance corrections applied
  • Outcome upsert success/failure
  • Discovery creation success/failure
  • Analysis complete (magnitude, confidence, elapsed_ms)

Files:

  • Modified: supabase/functions/ai-engine/engines/experiment-analyst.ts

Status Summary

Step What Status
1 Diagnostics persistence + column migration COMPLETE
2 Redirect mobile hooks to ai-engine COMPLETE
3 Migration: backfill + drop experiment_results COMPLETE
4 Delete old analyze-experiment function COMPLETE
5 Admin dashboard (2 pages + nav link) COMPLETE
6 Add Logger to experiment-analyst COMPLETE

Test Results

  • 134/134 test suites pass (2688 tests, 0 failures)
  • Web typecheck clean (0 errors)
  • Updated test files:
    • mobile/src/hooks/__tests__/experimentQueryHooks.test.ts — updated to mock experimentAIClient and assert experiment_outcomes queries
    • mobile/src/hooks/__tests__/experimentMutationHooks.test.ts — updated to mock analyzeExperiment instead of supabase.functions.invoke
    • mobile/src/utils/experiments/__tests__/factories.ts — updated createMockResult to match new ExperimentResult shape

Verification Checklist

  1. Mobile: complete an experiment → verify analysis writes to experiment_outcomes (not experiment_results)
  2. Mobile: view results screen → verify it reads from experiment_outcomes
  3. Cron: verify auto-complete still works (no change to that path)
  4. Admin: load /admin/experiments → verify listing shows all analyzed experiments
  5. Admin: click into detail → verify all 6 tabs render with real data
  6. Admin: verify app_logs has entries for experiment analysis (category='experiment')
  7. Verify experiment_results table no longer exists
  8. Verify analyze-experiment function directory no longer exists

Key Files Changed

File Action
supabase/functions/analyze-experiment/ DELETED
supabase/functions/ai-engine/engines/experiment-analyst.ts MODIFIED (diagnostics, Logger)
supabase/functions/_shared/persistExperimentDiagnostics.ts CREATED
supabase/migrations/20260323000000_unify_experiment_analysis.sql CREATED
mobile/src/hooks/useExperimentResults.ts MODIFIED (read from experiment_outcomes, call ai-engine)
mobile/src/hooks/useCompletionCheck.ts MODIFIED (call ai-engine instead of analyze-experiment)
mobile/src/utils/experiments/types.ts MODIFIED (updated ExperimentResult interface)
mobile/src/components/Experiments/VerdictBanner.tsx MODIFIED (magnitude-based display)
mobile/src/app/experiment/[id].tsx MODIFIED (new field names)
web/src/app/routes/admin/experiments.tsx CREATED
web/src/app/routes/admin/experiments.$outcomeId.tsx CREATED
web/src/app/root.tsx MODIFIED (added nav link)
mobile/src/hooks/__tests__/experimentQueryHooks.test.ts MODIFIED
mobile/src/hooks/__tests__/experimentMutationHooks.test.ts MODIFIED
mobile/src/utils/experiments/__tests__/factories.ts MODIFIED