bug(core): indeterminate agent process-probe vetoes authoritative runtime-dead — sessions stuck forever in 'detecting/runtime_lost' #2025

Bug

Worker sessions whose tmux runtime is genuinely gone get stuck permanently on the dashboard sidebar in state: detecting / reason: runtime_lost ("Detecting runtime truth (runtime lost)") and never reach a terminal state. The lifecycle probe pipeline lets an indeterminate agent process-probe veto the runtime's authoritative "dead" signal, then writes nothing — freezing the session in limbo.

This is the inverse failure mode of #1838: that issue fixed ps-timeout being conflated with "process not found" (which bulk-terminated live sessions) by making isProcessRunning return INDETERMINATE on any probe failure. That fix overcorrected — a definitively-dead tmux session (tmux list-panes exits non-zero → throws → caught → INDETERMINATE) is now indistinguishable from a transient probe failure, so confirmed-dead runtimes never terminate.

Source: AO dogfooding — codex smoke-test workers (ao-180, ao-183) observed stuck on sidebar
Reported by: @hrishic7
Analyzed against: 5d0b624f (origin/main HEAD)
Confidence: High — traced end-to-end against live on-disk session metadata
Affects: all agents (codex AND claude-code share the pattern), tmux runtime

Reproduction

1. Spawn a codex (or claude) worker in a tmux runtime.
2. Let the tmux session die WITHOUT a manual kill and WITHOUT a PR (e.g. ao stop killed the pane, smoke test ended, tmux server reaped).
3. Observe the sidebar: session sits at "Detecting runtime truth (runtime lost)", Runtime: missing, PR: none — and never leaves.

A session that is manually killed (e.g. ao-180) does leave, because the kill path writes terminated/manually_killed directly, bypassing the probe pipeline. That difference is the tell.

Live evidence

~/.agent-orchestrator/projects/agent-orchestrator_1a434010b7/sessions/ao-183.json (STUCK):

session.state = "detecting"   session.reason = "runtime_lost"
runtime.state = "missing"     runtime.reason = "tmux_missing"
terminatedAt  = null
lifecycleEvidence = "idle_beyond_threshold activity_signal=valid via_native activity=idle at=2026-05-22T11:11:38.624Z"  # STALE/frozen

.../sessions/ao-180.json (LEFT — only because manually killed):

session.state = "terminated"  session.reason = "manually_killed"
runtime.state = "missing"     runtime.reason = "manual_kill_requested"
terminatedAt  = "2026-05-22T11:29:18.624Z"

The idle_beyond_threshold ... activity=idle evidence is frozen from the last poll while tmux was still alive (its at= exactly matches the last entry in the codex rollout JSONL). It is preserved verbatim by skipMetadataWrite: true, NOT recomputed — the runtime is dead, the idle string is a fossil.

Root Cause

When the tmux session is gone, the two probes disagree by design:

• Runtime probe is authoritative-dead. runtime-tmux isAlive uses tmux has-session, catches failure, returns clean false → runtimeProbe.state = "dead", runtime.state = "missing"/"tmux_missing".
  packages/plugins/runtime-tmux/src/index.ts:200-207, packages/core/src/lifecycle-manager.ts:948-958
• Agent probe is indeterminate. agent-codex isProcessRunning runs tmux list-panes -t <id>; with the session gone this throws ("can't find session"), caught by the broad catch → returns PROCESS_PROBE_INDETERMINATE.
  packages/plugins/agent-codex/src/index.ts:782-834 (catch at :831)
  Identical pattern in claude-code: packages/plugins/agent-claude-code/src/activity-detection.ts:208-270 (catch at :267).
• getActivityState returns null (not idle) because it correctly checks process-liveness first and bails on INDETERMINATE.
  packages/plugins/agent-codex/src/index.ts:666-677

The pipeline then short-circuits on the indeterminate agent probe before resolveProbeDecision (which has the authoritative runtimeProbe.state === "dead" in hand) can declare terminal:

packages/core/src/lifecycle-manager.ts:1179-1200

if (processProbe.indeterminate) {
  return {
    status: session.status,                          // stays whatever it was (stuck/detecting)
    evidence: session.metadata["lifecycleEvidence"], // STALE, preserved
    detectingAttempts: currentDetectingAttempts,
    skipMetadataWrite: true,                          // writes NOTHING
  };
}
const probeDecision = resolveProbeDecision({ runtimeProbe, processProbe, ... }); // never reached

Two compounding defects:

1. Indeterminate agent probe outranks authoritative runtime-dead. runtime.isAlive definitively reports the tmux session gone, but the indeterminate agent probe vetoes it.
2. detecting never self-drains for this path. The escalation budget (DETECTING_MAX_ATTEMPTS = 3 / time limit in createDetectingDecision, packages/core/src/lifecycle-status-decisions.ts:146-157) is never even incremented, because :1179 returns with skipMetadataWrite: true before createDetectingDecision runs. And even when it does fire, it escalates detecting → stuck — another non-terminal state — so a confirmed-dead runtime never reaches terminated.

sm.list() writing detecting/runtime_lost is correct and intentional (#1735) — a hot read path must not unilaterally terminate sessions + nuke worktrees on a transient probe. The bug is that the pipeline that is supposed to drain detecting can't, when the agent probe is indeterminate but the runtime is authoritatively dead.

Why the sidebar keeps showing it

deriveLegacyStatus maps runtime_lost → killed (terminal) only when session.state === "terminated" (packages/core/src/lifecycle-state.ts:447-449); detecting maps to non-terminal detecting (:459-460). Stuck at detecting, the session is never categorized terminal, so it never leaves the active sidebar. Summary string: packages/web/src/lib/serialize.ts:87-88.

Fix

Core-first, smallest correct change:

1. Don't let indeterminate veto authoritative-dead. At lifecycle-manager.ts:1179, when processProbe.indeterminate BUT runtimeProbe.state === "dead" (and !runtimeProbe.failed), do not short-circuit — fall through to resolveProbeDecision so the dead runtime drives a terminal decision.
2. Confirmed-dead runtime should escalate to terminated/runtime_lost, not park in stuck. Audit the detecting/createDetectingDecision escalation so a runtime that is authoritatively dead resolves terminal.

Optional defense-in-depth (avoids reopening #1838): in the agent isProcessRunning impls, distinguish tmux list-panes exiting with "can't find session" (→ return false/dead) from genuine transient failures (→ INDETERMINATE), instead of one broad catch.

Impact

• Codex/claude worker sessions accumulate as permanent "Detecting runtime truth (runtime lost)" cards that can only be cleared by manual kill.
• Clutters the sidebar and (per bug(core,web): dashboard request paths run expensive agent enrichment for terminated/runtime-lost sessions #1991) keeps these zombie records on the expensive enrichment path.
• Stuck-detection / cleanup downstream of terminal state never fire for these sessions.

Related

• bug(agent-plugins): ps timeout conflated with "process not found" — bulk-terminates all sessions when ps exceeds 5s #1838 (CLOSED) — the inverse fix that introduced INDETERMINATE; this issue is its overcorrection.
• bug(core,web): dashboard request paths run expensive agent enrichment for terminated/runtime-lost sessions #1991 — enrichment cost for terminated/runtime-lost sessions (same zombie records).
• bug(core): manually running Claude in orphaned tmux revives stuck session on dashboard #1967 / bug(core): stopping Claude from dashboard terminal goes through detecting → stuck instead of terminating cleanly #1966 — stuck-vs-terminated and revival of stuck sessions (adjacent lifecycle gaps).