bench eval continue is limited by replay wire protocol, not model openness — and timed-out runs lose their world before they can be salvaged

[JeremyJC67]

Summary

Two separate gaps make it hard to recover from a timed-out run, and they are being conflated in discussion as "continue only supports open models". The constraint is not open vs closed weights — it is the wire protocol at the replay seam, plus the fact that a timed-out run's world is destroyed before anything can be salvaged from it.

1. bench eval continue is limited by protocol, not by model openness

src/benchflow/continue_run/run_folder.py:283 rejects any run whose agent is not openhands. That guard is truthful, and removing it alone would make things worse, not better:

• orchestrator.py:366 hardcodes agent="openhands" in the continuation's rollout config, so relaxing the gate launches OpenHands against another agent's recording rather than resuming that agent.
• orchestrator.py:336-342 emits only LLM_BASE_URL / LLM_API_KEY / LLM_MODEL, which only OpenHands consumes (agents/registry.py:939-941). Other agents read ANTHROPIC_BASE_URL, OPENAI_BASE_URL, GOOGLE_GEMINI_BASE_URL, or BENCHFLOW_PROVIDER_*.
• replay_proxy.py:498-501 serves only POST /v1/chat/completions; completion_to_sse (:340-425) emits OpenAI chat.completion.chunk frames only. claude-agent-acp posts Anthropic Messages to /v1/messages, codex-acp posts Responses to /v1/responses (litellm_runtime.py:1525-1568) — both 404 at the proxy.
• With no proxy reached, resolve_agent_env falls back to host credentials, so the "replay" would burn a full live run that the artifacts then label as a replay.

Recording is not the blocker for API-key runs: the LiteLLM gateway writes llm_trajectory.jsonl agent-agnostically (rollout/__init__.py:1351, litellm_runtime.py:178-201), and litellm_config.py:477-483 already forces the chat-completions bridge so streaming claude-agent-acp runs record at all (#833). The genuinely unrecoverable case is subscription-auth runs (uses_native_subscription_auth, litellm_runtime.py:1650): they bypass the gateway entirely and produce no recording, so no replay mechanism can help them.

Feasibility, if we want to generalize:

Agent	Ingress needed	Note
opencode, mimo, pi-acp, deepagents, harvey-lab-harness	none — already OpenAI chat wire	cheapest wins; needs per-agent env wiring + a placeholder model for ACP set_model
claude-agent-acp (API-key runs)	Anthropic Messages ingress + SSE framing	highest value for the current paper runs
codex-acp	/v1/responses ingress	recordings carry input, not messages
gemini	Google native wire	suggest scoping out
subscription-auth runs (any agent)	—	no recording exists; out of reach by construction

An alternative worth considering before building three ingresses: keep the LiteLLM gateway in front of the agent as in a normal run and point its upstream at the replay server. The gateway already does all the per-agent protocol translation; the replay side would only need to serve recorded responses.

2. A timed-out run's world is destroyed, so nothing can be salvaged afterwards

Independently of replay: the stage-snapshot machinery (RolloutConfig.snapshot_stages) can capture pre-verify on the timeout path, and --keep-snapshots exports a sha256-verified tar before teardown — but --snapshot-stages is only reachable via the SDK or a task declaration, not from bench eval run. So an operator running a large sweep cannot ask for the capture that would let them recover, and the container is gone by the time they discover the timeout.

3. Not every timeout is resumable, and treating them alike wastes effort

Runs that time out with zero tool calls and zero events have no progress to resume; they belong to the idle-watchdog / timeout-budget work (#1062, #1063, #1066), not to continue. Triage before choosing an instrument: continue is for runs that made real progress and then ran out of budget.

Proposed sequence

1. Small, no replay code touched — --snapshot-stages on bench eval run (threaded exactly like --keep-snapshots), a typed error at the continue gate that names the supported agents and the reason, and batch continue skipping unsupported runs instead of aborting the batch. PR to follow.
2. claude-agent-acp continue support (Anthropic ingress, or the gateway-in-front redesign above).
3. Optionally codex-acp; gemini scoped out.

Happy to take 1 and 2 — 1 is ready now.