|
1 | | -# Loop Runs (Goal Runs) |
| 1 | +# Loops |
2 | 2 |
|
3 | | -A loop run wraps a coding session in a durable **act → verify → continue/stop** |
4 | | -loop: an agent (the *maker*) works toward a goal in an isolated worktree, a |
5 | | -deterministic verifier runs shell commands after every turn, and the loop |
6 | | -continues — feeding failures back to the maker — until the verifier passes, |
7 | | -a budget is exhausted, or a human needs to decide. Every step is recorded in an |
8 | | -evidence ledger, so a run is auditable after the fact. |
| 3 | +Loops are Codekin's durable control plane for outcome-driven agent work: you |
| 4 | +state an outcome and acceptance criteria (a **recipe**), and Codekin runs a |
| 5 | +coding agent in checkpointed stages until the criteria pass, a human decision |
| 6 | +is needed, or a budget boundary is reached. The full design rationale lives in |
| 7 | +[LOOPS-REWRITE-SPEC.md](./LOOPS-REWRITE-SPEC.md); this page documents what is |
| 8 | +implemented today (Phase 1: durable engine core). |
9 | 9 |
|
10 | | -Loop runs differ from [AI Workflows](WORKFLOWS.md) in shape: a workflow is a |
11 | | -scheduled one-shot session that produces a report; a loop run is goal-driven |
12 | | -and iterates until a machine-checkable condition holds. |
| 10 | +## Core loop |
13 | 11 |
|
14 | | -## Lifecycle |
15 | | - |
16 | | -``` |
17 | | -queued → running ⇄ verifying ⇄ checking → succeeded |
18 | | - ⇅ failed |
19 | | - blocked aborted |
20 | | - awaiting_human |
| 12 | +```text |
| 13 | +preflight → act → evaluate → (review) → decide → … → finalize |
21 | 14 | ``` |
22 | 15 |
|
23 | | -| Status | Meaning | |
24 | | -|---|---| |
25 | | -| `queued` | Created; maker session not yet started. | |
26 | | -| `running` | The maker is working a turn. | |
27 | | -| `verifying` | The verify commands are executing against the worktree. | |
28 | | -| `checking` | The checker (second provider) is reviewing the diff. | |
29 | | -| `blocked` | A maker/checker tool call is waiting on human approval or a question. Non-terminal: answer the prompt (open the session from the sidebar) and the loop resumes; unanswered prompts are denied by the router timeout and the loop continues on the denial. | |
30 | | -| `awaiting_human` | Escalated to a human checkpoint — repeated readonly violations, a checker `escalate` verdict, or an unparseable verdict. Terminal. | |
31 | | -| `succeeded` | Verifier green (and checker approval, if configured); changes landed per the completion policy. | |
32 | | -| `failed` | Turn/cost budget exhausted, unrecoverable error, or the run was interrupted by a server restart. | |
33 | | -| `aborted` | Cancelled by the user. | |
34 | | - |
35 | | -**Restarts.** In-flight runs do not survive a server restart: at boot, any run |
36 | | -persisted in a non-terminal status is marked `failed` with a |
37 | | -"interrupted by a server restart" row in its ledger. (Reattaching to a live |
38 | | -maker session after a restart is future work — until then the ledger is honest |
39 | | -rather than optimistic.) |
40 | | - |
41 | | -## Turn mechanics |
42 | | - |
43 | | -Each maker turn ends with a session result, which triggers: |
44 | | - |
45 | | -1. **Budgets** — the run fails once `maxTurns` or `maxCostUsd` is reached. |
46 | | -2. **Readonly enforcement** — files matching `readonly` globs must not change; |
47 | | - a violation re-prompts the maker, and repeated violations escalate. |
48 | | -3. **No-change nudge** — a clean tree is not success; the maker is re-prompted. |
49 | | -4. **Verify (debounced)** — the `verify` commands run in order in the worktree |
50 | | - (10 min per command); the run skips re-verifying when the diff is unchanged. |
51 | | - Failures are fed back to the maker as the next turn's input. |
52 | | -5. **Checker review (optional)** — when the spec names a `checker`, a second |
53 | | - provider reviews the diff read-only and must end its reply with |
54 | | - `VERDICT: approve | request_changes | escalate`. |
55 | | -6. **Finalization** — on success Codekin (not the agent) commits the verified |
56 | | - tree and, per `completionPolicy`, pushes and opens a PR. Auto-merge is never |
57 | | - performed. |
58 | | - |
59 | | -## Tool allowlists |
60 | | - |
61 | | -Maker sessions are created with the shared headless-agent allowlist |
62 | | -(`server/agent-allowlist.ts`) — git, gh, package managers, build/test tools, |
63 | | -non-destructive file operations. Destructive commands (`rm`, `sudo`, |
64 | | -`git push --force`, …) still require approval; a run waiting on one shows as |
65 | | -`blocked`. Checker sessions get a read-only subset (no Write/Edit) — a reviewer |
66 | | -that needs to write has left its mandate. |
67 | | - |
68 | | -## Templates |
69 | | - |
70 | | -A loop template is a markdown file with YAML frontmatter (spec) and a body |
71 | | -(default goal text): |
72 | | - |
73 | | -```markdown |
| 16 | +- **act** — a maker session (claude / codex / opencode) works in an isolated |
| 17 | + git worktree on the run's branch. |
| 18 | +- **evaluate** — command evaluators (your own build/test/lint commands, judged |
| 19 | + by exit code) run in order after every maker turn. Failures are fed back to |
| 20 | + the maker; transient environment errors (timeout, spawn failure) retry per |
| 21 | + the recipe's `retry` policy. |
| 22 | +- **review** — rubric evaluators put an independent model — always a |
| 23 | + *different provider* than the maker — over the diff. It answers with |
| 24 | + `approve` / `request_changes` / `escalate`; an unparseable verdict escalates |
| 25 | + rather than silently passing. |
| 26 | +- **decide** — deterministic code, not the model. The maker never decides |
| 27 | + whether its own acceptance criteria passed. The decision order is: user |
| 28 | + cancel/pause → budgets → protected paths → no-change nudge → evaluation → |
| 29 | + no-progress detection → review → completion. |
| 30 | +- **finalize** — Codekin itself commits the verified tree and, per the |
| 31 | + completion action, pushes and opens a PR. Auto-merge does not exist. |
| 32 | + |
| 33 | +## Durability |
| 34 | + |
| 35 | +Every transition is an append-only row in `loop_events` (monotonic sequence |
| 36 | +per run) and orchestration counters are checkpointed after every decided |
| 37 | +turn. On restart Codekin reconciles instead of failing runs: |
| 38 | + |
| 39 | +- `paused` and `awaiting_approval` runs are left waiting (they hold no |
| 40 | + process); |
| 41 | +- in-flight runs resume at a stage boundary — a fresh session in the |
| 42 | + surviving worktree with a regenerated context prompt (provider sessions are |
| 43 | + not resumed in-provider); |
| 44 | +- a run whose worktree is gone fails honestly with a reason. |
| 45 | + |
| 46 | +Execution **state** and terminal **outcome** are separate fields: a run ends |
| 47 | +`done` + `completed` / `completed_with_warnings` / `failed` / `canceled`. |
| 48 | +Waived or failed-optional evaluators qualify the outcome — a run never shows |
| 49 | +an unqualified green with a skipped check. |
| 50 | + |
| 51 | +## Controls |
| 52 | + |
| 53 | +Every control appends an auditable event: |
| 54 | + |
| 55 | +- **Pause** — parks the run durably at the next safe boundary; **Resume** |
| 56 | + continues in the same worktree with a fresh session. |
| 57 | +- **Stop** — cancels now; the worktree is kept for inspection. |
| 58 | +- **Steer** — queue an operator instruction; it reaches the maker at the next |
| 59 | + safe boundary (mid-turn injection is not attempted). |
| 60 | +- **Interventions** — when the run cannot decide safely it parks in |
| 61 | + `awaiting_approval` with a pending intervention card: completion approval |
| 62 | + (guided mode), budget extension (extend adds 50% of the original budget), |
| 63 | + or escalation (repeated protected-path violations, no-progress, reviewer |
| 64 | + escalation). Resolving the card continues or ends the run. |
| 65 | + |
| 66 | +## Budgets and no-progress |
| 67 | + |
| 68 | +`budgets.turns` and `budgets.costUsd` are hard caps; `budgets.wallTime` is |
| 69 | +optional. At a boundary the run *asks* for a bounded extension (guided / |
| 70 | +guarded modes) or stops with a partial result (autonomous mode). The |
| 71 | +no-progress detector compares diff summaries and normalized failure |
| 72 | +fingerprints across evaluate cycles — producing more text is not progress — |
| 73 | +and escalates after `budgets.noProgressAttempts` identical failures. |
| 74 | + |
| 75 | +## Recipes |
| 76 | + |
| 77 | +A recipe is Markdown + YAML frontmatter, reviewable in git: |
| 78 | + |
| 79 | +```yaml |
74 | 80 | --- |
75 | | -kind: flaky-e2e |
76 | | -name: Flaky E2E Quarantine |
77 | | -maker: |
78 | | - provider: claude |
79 | | -checker: # optional — omit for a single-provider loop |
80 | | - provider: opencode |
81 | | -verify: |
82 | | - - npm test |
83 | | - - npm run lint |
84 | | -readonly: # optional |
85 | | - - .github/workflows/** |
86 | | -maxTurns: 12 |
87 | | -maxCostUsd: 5 |
88 | | -completionPolicy: pr # pr | merge | commit-only (defaults to pr) |
| 81 | +apiVersion: codekin.dev/v2 |
| 82 | +kind: LoopRecipe |
| 83 | +metadata: |
| 84 | + id: ci-autorepair |
| 85 | + name: CI Autorepair |
| 86 | +agent: |
| 87 | + provider: auto # resolves at run start; recorded on the run |
| 88 | +workspace: |
| 89 | + strategy: worktree |
| 90 | + protectedPaths: [".github/workflows/**"] |
| 91 | +evaluators: |
| 92 | + - id: tests |
| 93 | + type: command |
| 94 | + command: npm test # shell string (trusted repo code) or argv array |
| 95 | + timeout: 15m |
| 96 | + retry: { maxAttempts: 2 } |
| 97 | + - id: review |
| 98 | + type: rubric |
| 99 | + provider: different-from-maker |
| 100 | +budgets: |
| 101 | + turns: 12 |
| 102 | + costUsd: 5 |
| 103 | + wallTime: 90m |
| 104 | +policy: |
| 105 | + mode: guarded # guided | guarded | autonomous |
| 106 | +completion: |
| 107 | + action: pull-request # or commit-only; auto-merge does not exist |
89 | 108 | --- |
90 | | -Find the flaky e2e test on this branch, fix the root cause... |
| 109 | +The outcome prompt (markdown body) goes here. |
91 | 110 | ``` |
92 | 111 |
|
93 | | -Templates are read from two places: |
| 112 | +Validation is strict — unknown fields fail. The parsed recipe is normalized, |
| 113 | +content-hashed, and frozen into every run, so editing the file never changes |
| 114 | +what a past run claims it executed. Recipes load from: |
94 | 115 |
|
95 | | -- **Built-ins** shipped with the package (`server/loops/*.md`): |
96 | | - `ci-autorepair`, `coverage-increase`, `dependency-upgrade`. |
97 | | -- **Per-repo templates** in `{repo}/.codekin/loops/*.md`. A repo template with |
98 | | - the same `kind` overrides the built-in; a repo template with a **new kind is |
99 | | - a first-class loop** — kinds are an open set, validated only as lowercase |
100 | | - slugs (letters, digits, `.`, `_`, `-`, max 64 chars). |
| 116 | +- built-ins shipped with the package: `server/loops/*.md` |
| 117 | + (`ci-autorepair`, `coverage-increase`, `dependency-upgrade`); |
| 118 | +- per-repo overrides: `{repo}/.codekin/loops/*.md` (same id wins). |
101 | 119 |
|
102 | | -## API |
| 120 | +Evaluator types beyond `command` and `rubric` (test-report, diff-policy, |
| 121 | +artifact, ci, human, composite) arrive with the Phase 3 evaluator platform |
| 122 | +and are rejected at validation until then. |
103 | 123 |
|
104 | | -All endpoints require the master Bearer token. |
| 124 | +## Evidence |
| 125 | + |
| 126 | +Full evaluator output is retained as content-addressed artifacts |
| 127 | +(`~/.codekin/loop-artifacts/`), referenced from structured `loop_evaluations` |
| 128 | +rows; the maker sees only a tail as feedback. Reviews are artifacts too. |
| 129 | + |
| 130 | +## API |
105 | 131 |
|
106 | | -| Endpoint | Description | |
107 | | -|---|---| |
108 | | -| `GET /api/goal-runs/templates?repoPath=` | Available templates (built-ins + repo). | |
109 | | -| `GET /api/goal-runs/runs?kind=&status=&limit=` | List runs, newest first. | |
110 | | -| `GET /api/goal-runs/runs/:id` | One run plus its turn-by-turn evidence ledger. | |
111 | | -| `POST /api/goal-runs/runs` | Start a run: `{ kind, repo, branch, goal? }`. `goal` overrides the template's default goal text. | |
112 | | -| `POST /api/goal-runs/runs/:id/abort` | Abort an in-flight (or restart-orphaned) run. | |
| 132 | +Mounted at `/api/loops` (master Bearer token). See |
| 133 | +[API-REFERENCE.md](./API-REFERENCE.md#loops) for the endpoint list. Live |
| 134 | +updates ride the shared `workflow_event` WS channel (`engine: 'loop'`) as |
| 135 | +pings; clients reconcile against `GET /runs/:id/events?after=<sequence>`. |
113 | 136 |
|
114 | 137 | ## UI |
115 | 138 |
|
116 | | -The **Loop Runs** sidebar entry (`/loops`) lists runs with live status, spend |
117 | | -vs budget, and turn count; a run's detail view shows the evidence ledger. The |
118 | | -maker and checker are ordinary sessions (`source: agent`) and appear in the |
119 | | -sidebar — open one to answer a `blocked` prompt or watch the agent work. |
| 139 | +The Loops tab in Automations is an interim run list + control surface (start, |
| 140 | +pause/resume/stop, steer, intervention cards, evaluator scorecard). The full |
| 141 | +control plane — wizard with repo/branch pickers, four-tab run workspace, |
| 142 | +timeline — is Phase 2 of the spec. |
120 | 143 |
|
121 | 144 | ## Storage |
122 | 145 |
|
123 | | -SQLite at `~/.codekin/runs.db` (WAL, `0600`), shared with the workflow engine — |
124 | | -one runs database for all background automation. Rows from the pre-unification |
125 | | -`~/.codekin/goal-runs.db` are copied over automatically on first boot (the |
126 | | -legacy file is left in place). Tables: `goal_runs` (one row per |
127 | | -run) and `goal_run_turns` (the evidence ledger — diff stat, verify command, |
128 | | -exit code, output tail, checker verdict, cost per action). |
| 146 | +Tables in the shared `~/.codekin/runs.db`: `loop_runs`, `loop_stages`, |
| 147 | +`loop_attempts`, `loop_events`, `loop_checkpoints`, `loop_evaluations`, |
| 148 | +`loop_artifacts` (metadata), `loop_interventions`. The v1 `goal_runs` / |
| 149 | +`goal_run_turns` tables are dropped on first open — v1 had no users and no |
| 150 | +history worth preserving (spec §12). |
0 commit comments