RFC: Thompson Sampling for observation quality optimization

[alessandropcostabr]

Proposal

Use Thompson Sampling (multi-armed bandit) to dynamically learn which model produces the best observations for each observation type, building on the tier routing and feedback collection infrastructure.

Context

With tier routing (#1569), observations are processed by different models based on queue complexity. But the optimal model per observation type is unknown a priori — it depends on usage patterns that vary per user and project.

The observation_feedback table (#1569) already collects usage signals. This RFC proposes using those signals as reward/penalty for a bandit algorithm.

Design

Arms: {observation_type}:{model} (e.g., discovery:haiku, completion:sonnet, summary:opus)

Reward (alpha += 1):

• Observation accessed via get_observations within 7 days
• Observation injected via semantic search AND Claude referenced it in response

Penalty (beta += 1):

• Observation never accessed in 30 days
• Observation injected but Claude ignored it

Selection: Thompson Sampling — sample from Beta(alpha, beta) for each arm, select the highest.

Schema addition:

CREATE TABLE bandit_arms (
  arm_id TEXT PRIMARY KEY,         -- e.g., 'discovery:haiku'
  alpha REAL DEFAULT 1.0,          -- success count + prior
  beta REAL DEFAULT 1.0,           -- failure count + prior
  updated_at_epoch INTEGER
);

Architecture

Observation arrives
  → Bandit selects model for this obs_type
  → SDK Agent processes with selected model
  → Observation stored

7 days later (cron or lazy evaluation):
  → Feedback signals checked for this observation
  → Was it accessed/used? → reward (alpha += 1)
  → Never accessed? → penalty (beta += 1)
  → Bandit priors updated

Inspiration

Adapted from an internal project using Thompson Sampling for model routing in a Telegram bot. The bandit approach learned optimal model selection over ~2 weeks with user reactions as reward signal. The same principle applies here with observation usage as the reward.

Streak Detection (complementary)

Different failure types need different responses:

Failure type	Pattern	Action
API timeout	ETIMEDOUT	Exponential backoff
Rate limit	429	Pause 60s + model fallback
Content policy	content_policy	Skip observation (don't retry)
Context overflow	context_window	Reduce prompt size
Model error	500, overloaded	Fallback to alternative model

Prerequisites

• Tier routing merged (feat: tier routing by queue complexity + observation feedback table #1569) — provides model-per-type infrastructure
• Feedback collection active — provides usage signals
• Sufficient data: ~2 weeks of feedback signals for meaningful priors

Open Questions

1. Should the bandit also optimize prompt templates, not just models?
2. What's the right exploration rate for ~120 observations/day?
3. Should we warm-start from known priors (e.g., summaries always prefer higher-quality models)?
4. How to handle cold-start for new observation types?

Implementation Order

1. Bandit Engine — core Thompson Sampling logic + bandit_arms table
2. Reward Evaluator — cron/lazy evaluation of feedback signals
3. Integration — replace static tier routing with bandit selection
4. Patrol — monitoring dashboard for arm performance
5. Streak Detection — failure-type-specific responses

[github-actions]

This GitHub issue proposes using Thompson Sampling, a multi-armed bandit algorithm, to dynamically optimize the selection of models for processing observations based on their type, leveraging feedback on observation quality. Currently, observations are routed via tiered infrastructure, but the optimal model for each type depends on user-specific and project-specific patterns. The proposal outlines a design where combinations of observation type and model serve as "arms," and feedback (such as observations being accessed or ignored) is used to update reward and penalty scores for each arm using a Beta distribution. A new database table (bandit_arms) would store these metrics, and the system would iteratively select models based on the algorithm. The architecture relies on feedback signals collected over ~2 weeks, using heuristics like access patterns and ignored observations to adjust the model routing in near real-time. The RFC also considers complementary mechanisms like streak detection for handling systemic errors, as well as open questions on exploration rates, cold-start conditions, and possible extensions to optimize prompts

[Whatsonyourmind]

The Beta-conjugate model in this RFC is the right choice for binary reward signals. A few mathematical notes on the design:

Arm granularity tradeoff: The proposed {observation_type}:{model} keying is correct for avoiding Simpson's paradox. But with 5 observation types and 4 models, you have 20 arms, each needing independent exploration.

Cold start mitigation: With Beta(1,1) priors, Thompson Sampling explores aggressively until each arm has ~20 observations. For 20 arms, that is ~400 observations before stabilization. Two ways to speed this up:

1. Warm start from tier routing. If feat: tier routing by queue complexity + observation feedback table #1569 already has quality signals per model, initialize Beta parameters from historical data: alpha = prior_successes + 1, beta = prior_failures + 1.

2. Hierarchical pooling. Share information across observation types for the same model. If haiku works well for discovery, that is partial evidence for completion. A hierarchical Beta: each arm's prior is Beta(alpha_global * w + 1, beta_global * w + 1) where w controls pooling strength.

Decay for non-stationarity: The exponential decay (alpha *= decay_factor per epoch) works but consider a sliding window of the last N observations instead. This forgets old data completely, which is more appropriate when model capabilities change discontinuously (e.g., model version updates).

Reward signal design: Binary (accessed/not accessed) is the simplest but lossy. If you have richer signals — time spent reading, whether cited in a response, whether user expanded it — a continuous reward in [0,1] works with the same Beta model via moment matching: alpha = mean * kappa, beta = (1-mean) * kappa where kappa controls confidence.

Good RFC — Thompson Sampling is the right algorithm for this exploration problem.

[alessandropcostabr]

Update: Prerequisites met, bug found and fixed, 27 days of production data

All prerequisite PRs are now merged:

• feat: tier routing by queue complexity + observation feedback table #1569 (tier routing + observation_feedback table) — merged manually by @thedotmack after rebasing
• feat: semantic context injection via Chroma on UserPromptSubmit #1568 (semantic context injection via Chroma) — merged
• fix: resolve 3 upstream bugs (summarize, ChromaSync, HealthMonitor) #1566 (bug fixes), feat: drain orphaned pending messages on SIGTERM session completion #1567 (SIGTERM drain), feat: claude-mem-sync for multi-machine observation synchronization #1570 (multi-machine sync) — all merged

Bug found: generated_by_model never wired

While auditing the pipeline for bandit readiness, we found that the generated_by_model column exists in the schema (added in the Phase 0 governance migration) but was never included in any INSERT statement. All 3,657 observations have this field as NULL. Fix submitted in #1602 — threads the model ID from each agent (SDK, Gemini, OpenRouter) through processAgentResponse() into all three store methods.

Production evolution (27 days, core projects only)

Phase	Period	Obs	Sessions	Reuse rate	What changed
W1: plugin installed	09-15 Mar	1,714	15	0%	Observations captured, no feedback wiring yet
W2: injection activated	16-22 Mar	720	13	0.4%	semantic_inject_hit + relevance_count started
W3: multi-project	23-29 Mar	573	19	5.1%	Cross-project context growing
W4: Phase 0 governance	30 Mar-04 Apr	650	20	34.5%	Deterministic ordering, status lifecycle, quality gates
Total	27 days	3,657	62	—	0% → 34.5%

W4's 34.5% means 1 in 3 observations gets reused in future sessions. The remaining 65.5% is the optimization target — Thompson Sampling can learn which {type}:{model} combinations produce observations worth keeping.

Observation type maturation (discovery shrinks as the system matures, decision-making grows):

Type	W1	W2	W3	W4	Total
discovery	1,029 (60%)	406 (56%)	232 (40%)	243 (37%)	1,910
change	335	92	121	156	704
feature	161	98	86	104	449
bugfix	107	79	71	83	340
decision	23 (1.3%)	32 (4.4%)	35 (6.1%)	49 (7.5%)	139
refactor	59	13	28	15	115

Feedback signals: 8,643 total — breakdown:

• semantic_inject_hit: 6,783 (78.5%) — observation injected via Chroma into prompt
• search_accessed: 925 (10.7%) — accessed via explicit search
• semantic_injected: 898 (10.4%) — injected but not necessarily cited
• get_observations_hit: 37 (0.4%) — accessed via get_observations

Responding to @Whatsonyourmind's questions

On the 7-day feedback delay and over-exploration:
Our data shows ~135 obs/day (core). Two mitigations for the ~945 in-flight observations before first rewards:

1. Most reward signals are immediate. semantic_inject_hit (78.5% of feedback) fires in the same session — no 7-day wait needed. Only the "never accessed in 30 days" penalty requires delay.
2. Warm-start from historical data. 8,643 feedback signals already collected. Initialize Beta priors from these instead of Beta(1,1).

On hierarchical pooling:
Strongly agree. With 6 observation types × N models, pure per-arm exploration is slow at ~135 obs/day. Pooling across types for the same model makes sense — if sonnet produces good discovery observations, that's evidence for change too.

On sliding window vs decay:
Sliding window fits better — model capabilities change discontinuously (new releases). A 30-day window covers ~4,050 observations at our rate, plenty for convergence.

Next steps

1. Wire generated_by_model into the write path → fix: wire generated_by_model into observation write path #1602
2. Accumulate 2 weeks of model-tagged data
3. Implement bandit engine with warm-start from existing feedback

[Whatsonyourmind]

@alessandropcostabr 27 days of production data is solid — enough to see real patterns in the Thompson posterior.

With the bug fixed and prerequisites merged, the key validation metric is: does the quality ranking from Thompson Sampling correlate with actual observation utility (measured by downstream task performance when that observation is included vs excluded)? If the correlation is >0.3 after 27 days, the prior is learning something real. If it's flat, the reward signal might be too noisy.

The production data should also let you check whether the exploration rate is reasonable — if some observations have <5 samples after 27 days, the exploration might be too conservative (lower the UCB constant). If all observations have ~equal samples, it's exploring too much (raise it).