fix(conductor): holistic-review fixes + CI + committed example artifacts (v0.3.3)

A whole-codebase review (6 dimensions, each finding independently verified)
confirmed 17 issues; verdict "solid". This addresses the correctness ones plus
the highest-leverage packaging gaps.

Correctness:
- Coordinator.run_all() is now safe to call repeatedly: a fresh per-batch token
  (not a per-Coordinator one) so trace files never collide/truncate, and a fresh
  owned ledger per batch so cost isn't double-counted across runs. (was HIGH)
- Orchestrator owns its ledger PER run() (created in run(), like the Tracer), so
  re-running the same Orchestrator no longer writes to a closed ledger while
  stale in-memory rows leak into the next run's budget/summary. (was MEDIUM)
- replay_trace now compares terminal status: a run that ended budget_exceeded
  can no longer replay as max_steps and still report match:true. (was MEDIUM)
- CascadeBackend returns a new turn (dataclasses.replace) instead of mutating the
  strong backend's turn, so a reused turn object can't accumulate extra_usages.
- replay.py docstring corrected (it never diffed per-provider cost).

Packaging / positioning (the buried-lede gaps):
- GitHub Actions CI (pytest on py3.9 + py3.12) + CI/python/license badges.
- examples/ now COMMITS real outputs (a run trace, a per-provider ledger, a
  sandbox snapshot/rollback trace) so the signature artifact is visible on
  GitHub without cloning. README links them up top.
- README "Known limitations (tracked)" section acknowledging the deferred
  low-severity items rather than hiding them.

Regression tests for every correctness fix (second run_all, Orchestrator re-run,
replay budget-status divergence, coordinator mid-job budget cutoff). 98 tests
pass; wheel builds. Version 0.3.3.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: akagifreeez

.github/workflows/ci.yml

@@ -0,0 +1,28 @@
+name: CI
+
+on:
+  push:
+    branches: [master, main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.9", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      # token-router is the in-process accounting dependency (public repo).
+      - name: Install token-router
+        run: pip install "git+https://github.com/akagifreeez/token-router"
+      - name: Install conductor (no deps; token-router already installed)
+        run: pip install -e . --no-deps
+      - name: Install pytest
+        run: pip install pytest
+      - name: Run tests
+        run: pytest -q

.gitignore

@@ -7,13 +7,17 @@ __pycache__/
 *.py[cod]
 *.egg-info/
 .pytest_cache/
+.coverage
+.coverage.*
+htmlcov/
 build/
 dist/
 
 # runtime artifacts
 traces/
 *.jsonl
 !tests/**/*.jsonl
+!examples/**/*.jsonl
 
 # secrets — never commit keys
 *.key

README.md

@@ -1,7 +1,16 @@
 # 🎛️ Conductor
 
+[![CI](https://github.com/akagifreeez/conductor/actions/workflows/ci.yml/badge.svg)](https://github.com/akagifreeez/conductor/actions/workflows/ci.yml)
+[![Python](https://img.shields.io/badge/python-3.9%2B-blue.svg)](pyproject.toml)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
+
 **A vendor-neutral, self-hosted control plane for LLM agents.**
 
+> 👀 **See it without running anything:** [`examples/`](examples/) holds real
+> committed outputs — a [run trace](examples/example-trace.jsonl), a
+> [per-provider cost ledger](examples/example-ledger.jsonl), and a
+> [sandbox snapshot/rollback trace](examples/example-sandbox-trace.jsonl).
+
 One self-built tool-use loop drives *any* provider — Claude (official Anthropic
 SDK), any OpenAI-compatible API (OpenAI / OpenRouter / Groq / Mistral / Together
 / Fireworks), or a local model (Ollama / LM Studio / vLLM) — behind a single
@@ -318,6 +327,26 @@ conductor run --provider local --model qwen2.5:3b-instruct --task "..."
 - **Later (one of):** microVM (KVM/Firecracker) comparison · a lightweight web
   dashboard over traces/ledger.
 
+## Known limitations (tracked, not hidden)
+
+A whole-codebase review surfaced these; the correctness ones are fixed, the rest
+are acknowledged rather than papered over:
+
+- **Replay does not re-apply a budget.** A run that ended `budget_exceeded`
+  replays without the ceiling, so its status differs — the comparison now
+  *reports this as a non-match* (`status_match: false`) instead of hiding it.
+- **The real-backend adapters' failure branches are lightly covered** by tests.
+  The offline doubles exercise the happy path + injection/leak guards; live error
+  paths (provider 5xx, a container that won't start) lean on the resilience layer
+  rather than dedicated tests.
+- **`ProxmoxSandbox` (API) auto-adds unknown SSH host keys** (`AutoAddPolicy`) for
+  the `pct exec` channel — fine on a trusted LAN/Tailscale, not for hostile nets.
+- **`DockerSandbox` containers are not resource/network-constrained by default**
+  (no `--network none`/`--memory`/`--cap-drop`). Add them for untrusted workloads.
+- **Container isolation tier:** Docker and Proxmox LXC share the host kernel
+  (contained + revertible, but not a VM-grade boundary). A microVM backend is the
+  future option for that.
+
 ## Tech
 
 Python · `requests` for the OpenAI-compatible path · official `anthropic` SDK

conductor/__init__.py

@@ -51,7 +51,7 @@
 )
 from .tracer import Tracer
 
-__version__ = "0.3.2"
+__version__ = "0.3.3"
 
 __all__ = [
     "AgentBackend",

conductor/backends/cascade.py

@@ -27,6 +27,7 @@
 """
 from __future__ import annotations
 
+from dataclasses import replace
 from typing import Callable, List, Optional
 
 from .base import AgentBackend, AssistantTurn, Message, ToolSpec
@@ -100,5 +101,9 @@ def step(
             system=system, messages=messages, tools=tools,
             max_tokens=max_tokens, temperature=temperature,
         )
-        strong_turn.extra_usages = list(strong_turn.extra_usages) + [cheap_turn.usage]
-        return strong_turn
+        # Return a NEW turn rather than mutating the strong backend's object: a
+        # backend that reuses/returns a shared AssistantTurn would otherwise
+        # accumulate extra_usages across steps and double-count the cheap leg.
+        return replace(
+            strong_turn, extra_usages=list(strong_turn.extra_usages) + [cheap_turn.usage]
+        )

conductor/coordinator.py

@@ -91,34 +91,43 @@ def __init__(
         self.budget_usd = budget_usd
         self.trace_dir = trace_dir
         self.run_id_prefix = run_id_prefix
-        # Unique per-coordinator token so repeated run_all() calls (or two
-        # coordinators sharing a prefix) never collide on trace/ledger filenames
-        # (a Tracer opens its file truncating - the CLI guards this the same way).
-        self._uniq = f"{int(time.time() * 1000)}-{uuid.uuid4().hex[:6]}"
-        self._owns_ledger = ledger is None
-        if ledger is not None:
-            self.ledger = ledger
-        else:
-            # Mirror the Orchestrator's owned-ledger crash-durability: stream to JSONL.
-            path = os.path.join(trace_dir, f"ledger-{run_id_prefix}-{self._uniq}.jsonl")
-            self.ledger = Ledger(pricing=make_pricing(), jsonl_path=path)
+        # A caller-owned (shared) ledger persists across batches; an owned ledger
+        # is created FRESH per run_all() (see below), never once in __init__ - so
+        # repeated run_all() calls don't collide on filenames or double-count cost.
+        self._injected_ledger = ledger
+        self._batch = 0
+        # Exposed after run_all so callers can inspect the most recent batch ledger.
+        self.ledger = ledger if ledger is not None else Ledger(pricing=make_pricing())
 
     def run_all(self, jobs: List[Job]) -> CoordinatorResult:
         result = CoordinatorResult(budget_usd=self.budget_usd)
+        self._batch += 1
+        # Fresh per-CALL token so repeated run_all() invocations never collide on
+        # trace/ledger filenames (Tracer truncates) - a per-Coordinator token did.
+        batch = f"{int(time.time() * 1000)}-{uuid.uuid4().hex[:6]}-b{self._batch}"
+        owns_ledger = self._injected_ledger is None
+        if owns_ledger:
+            # Fresh owned ledger per batch: crash-durable JSONL, and no stale rows
+            # from a prior batch leaking into this batch's budget/summary.
+            path = os.path.join(self.trace_dir, f"ledger-{self.run_id_prefix}-{batch}.jsonl")
+            ledger = Ledger(pricing=make_pricing(), jsonl_path=path)
+        else:
+            ledger = self._injected_ledger
+        self.ledger = ledger
         try:
             for i, job in enumerate(jobs):
                 # Skip before starting if the global budget is already spent.
-                if self.budget_usd is not None and ledger_cost_usd(self.ledger) >= self.budget_usd:
+                if self.budget_usd is not None and ledger_cost_usd(ledger) >= self.budget_usd:
                     result.outcomes.append(JobOutcome(label=job.label, status="skipped_budget"))
                     continue
                 orch = Orchestrator(
                     job.backend,
                     job.registry,
-                    run_id=f"{self.run_id_prefix}-{self._uniq}-{i}-{job.label}",
+                    run_id=f"{self.run_id_prefix}-{batch}-{i}-{job.label}",
                     system=job.system or DEFAULT_SYSTEM,
                     max_steps=job.max_steps,
                     trace_dir=self.trace_dir,
-                    ledger=self.ledger,          # shared -> global budget
+                    ledger=ledger,               # shared -> global budget for this batch
                     budget_usd=self.budget_usd,  # each step also checks the global spend
                 )
                 # Isolate per-job failures: one erroring agent must not abort the
@@ -133,9 +142,9 @@ def run_all(self, jobs: List[Job]) -> CoordinatorResult:
                                    error=f"{type(exc).__name__}: {exc}")
                     )
         finally:
-            if self._owns_ledger:
-                self.ledger.close()
+            if owns_ledger:
+                ledger.close()
         # 6 dp to match ledger_summary["est_cost_usd"] (avoid two disagreeing totals).
-        result.total_cost_usd = round(ledger_cost_usd(self.ledger), 6)
-        result.ledger_summary = conductor_summary(self.ledger)
+        result.total_cost_usd = round(ledger_cost_usd(ledger), 6)
+        result.ledger_summary = conductor_summary(ledger)
         return result

conductor/orchestrator.py

@@ -116,17 +116,15 @@ def __init__(
         # run start, teardown in finally) and wires the registry's dangerous-tool
         # gate to dispatch through it. Without one, dangerous tools stay blocked.
         self.sandbox = sandbox
-        # A fresh ledger per run unless the caller shares one across runs (e.g.
-        # to aggregate the per-provider cost split of a two-provider demo). An
-        # owned ledger is streamed to its own JSONL (crash-durable, mirroring the
-        # trace) and closed by us; a shared/injected ledger is the caller's to
-        # configure and close.
-        self._owns_ledger = ledger is None
-        if ledger is not None:
-            self.ledger = ledger
-        else:
-            ledger_path = os.path.join(trace_dir, f"ledger-{run_id}.jsonl")
-            self.ledger = Ledger(pricing=make_pricing(), jsonl_path=ledger_path)
+        # Ledger policy: a shared/injected ledger is the caller's to configure and
+        # close; an OWNED ledger is created FRESH inside run() (not here) - exactly
+        # like the per-run Tracer - so that re-running the same Orchestrator doesn't
+        # write to a closed ledger (silently dropping disk rows while in-memory
+        # rows from the prior run leak into the next run's budget/summary).
+        self._injected_ledger = ledger
+        # self.ledger is (re)assigned at the start of each run(); seed it so the
+        # attribute exists even before the first run.
+        self.ledger = ledger if ledger is not None else Ledger(pricing=make_pricing())
 
     def _trace_sandbox(self, tracer: Tracer, event: str, detail: str = "") -> None:
         """Emit a sandbox lifecycle event, never letting a trace-write failure
@@ -141,6 +139,19 @@ def _trace_sandbox(self, tracer: Tracer, event: str, detail: str = "") -> None:
             pass
 
     def run(self, task: str) -> RunResult:
+        # Set up the ledger FRESH per run when we own it (single-use, like the
+        # Tracer): this makes re-running the same Orchestrator safe and keeps the
+        # on-disk JSONL and the in-memory rows in agreement.
+        if self._injected_ledger is not None:
+            self.ledger = self._injected_ledger
+            owns_ledger = False
+        else:
+            self.ledger = Ledger(
+                pricing=make_pricing(),
+                jsonl_path=os.path.join(self.trace_dir, f"ledger-{self.run_id}.jsonl"),
+            )
+            owns_ledger = True
+
         tools = self.registry.specs()
         messages: List[Message] = [Message(role="user", text=task)]
         final_text = ""
@@ -252,7 +263,7 @@ def run(self, task: str) -> RunResult:
             try:
                 tracer.close()
             finally:
-                if self._owns_ledger:
+                if owns_ledger:
                     self.ledger.close()
 
         return RunResult(

conductor/replay.py

@@ -10,9 +10,13 @@
   events) instead of re-executing tools.
 
 So a replay produces a fresh trace that should match the original's tool I/O,
-final answer, and per-provider cost. This proves the trace is **complete and
+final answer, and terminal status. This proves the trace is **complete and
 sufficient** to reconstruct the run deterministically - the foundation for audit
 and (later) what-if replay. No provider key, no tool side effects, no sandbox.
+
+(The replay records the same per-call ``Usage`` it read from the trace, so the
+replayed ledger reproduces the original cost too; the comparison below checks
+tool I/O, final answer, and status - it does not separately diff cost.)
 """
 from __future__ import annotations
 
@@ -145,16 +149,24 @@ def replay_trace(path: str, *, run_id: str, trace_dir: str = "traces") -> Tuple[
     res = orch.run(task)
 
     original_results = [e.get("content", "") for e in events if e.get("kind") == "tool_result"]
-    original_final = next((e.get("final_text", "") for e in events if e.get("kind") == "run_end"), "")
+    original_run_end = next((e for e in events if e.get("kind") == "run_end"), {})
+    original_final = original_run_end.get("final_text", "")
+    original_status = original_run_end.get("status", "")
     new_events = load_trace(res.trace_path)
     new_results = [e.get("content", "") for e in new_events if e.get("kind") == "tool_result"]
 
     tool_results_match = new_results == original_results
     final_match = res.final_text == original_final
+    # Compare terminal status too: a run that ORIGINALLY ended e.g. "budget_exceeded"
+    # must not silently replay as "max_steps" and still be reported as a match.
+    status_match = res.status == original_status
     comparison = {
-        "match": tool_results_match and final_match,
+        "match": tool_results_match and final_match and status_match,
         "tool_results_match": tool_results_match,
         "final_match": final_match,
+        "status_match": status_match,
+        "original_status": original_status,
+        "replayed_status": res.status,
         "original_final": original_final,
         "replayed_final": res.final_text,
         "n_tool_results": len(original_results),

examples/README.md

@@ -0,0 +1,25 @@
+# Example artifacts
+
+These are **real outputs** committed so you can see what Conductor produces
+without cloning and running it. All were generated key-free by the offline demos
+(`conductor demo` and `conductor sandbox-demo`).
+
+| File | What it is |
+|---|---|
+| [`example-trace.jsonl`](example-trace.jsonl) | A run trace: every LLM request/response and tool call/result, one JSON line each (`run_start` → `llm_request` → `llm_response` → `tool_call` → `tool_result` → … → `run_end`). This is the substrate for observability and deterministic replay. |
+| [`example-ledger.jsonl`](example-ledger.jsonl) | The per-provider cost ledger for a two-provider `demo` run — one row per LLM call, keyed by `backend`, so cost splits per provider. |
+| [`example-sandbox-trace.jsonl`](example-sandbox-trace.jsonl) | A `sandbox-demo` trace showing the OS-isolation story: `sandbox` setup → a destructive `run_shell` (snapshotted first) → `sandbox_rollback` → teardown, with the host untouched. |
+
+Regenerate them yourself (no API key needed):
+
+```bash
+conductor demo --trace-dir examples
+conductor sandbox-demo --trace-dir examples
+```
+
+Replay a trace deterministically (reproduces the recorded tool I/O + final answer
++ terminal status, with no provider calls and no tool side effects):
+
+```bash
+conductor replay --trace examples/example-trace.jsonl
+```

examples/example-ledger.jsonl

@@ -0,0 +1,4 @@
+{"task_id": "1782697895239-anthropic-69bf76", "stage": "llm", "backend": "anthropic", "model": "claude-opus-4-8", "prompt_tokens": 50, "completion_tokens": 0, "total_tokens": 50, "cost_usd": 0.00025, "latency_ms": 0.0, "estimated": true}
+{"task_id": "1782697895239-anthropic-69bf76", "stage": "llm", "backend": "anthropic", "model": "claude-opus-4-8", "prompt_tokens": 57, "completion_tokens": 17, "total_tokens": 74, "cost_usd": 0.00071, "latency_ms": 0.0, "estimated": true}
+{"task_id": "1782697895239-local-201d09", "stage": "llm", "backend": "local", "model": "qwen2.5:3b-instruct", "prompt_tokens": 50, "completion_tokens": 0, "total_tokens": 50, "cost_usd": 0.0, "latency_ms": 0.0, "estimated": true}
+{"task_id": "1782697895239-local-201d09", "stage": "llm", "backend": "local", "model": "qwen2.5:3b-instruct", "prompt_tokens": 57, "completion_tokens": 17, "total_tokens": 74, "cost_usd": 0.0, "latency_ms": 0.0, "estimated": true}