Harden Bellwether watch guidance (#42)

Author: roderik

CHANGELOG.md

@@ -7,6 +7,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed
+
+- Hardened the Bellwether skill and hook instructions so agents must keep the watch loop running while CI is pending or in progress, and must react immediately when the first CI failure or review comment appears.
+- Clarified that CI failures must be reproduced and rerun locally before pushing, and that watch timeouts may exit when the only remaining blockers are external, such as missing review approval.
+
 ## [0.0.7] - 2026-03-30
 
 ### Added

README.md

@@ -33,7 +33,7 @@ bellwether skills add
 
 ## Hooks
 
-Bellwether can install `PostToolUse` and `Stop` hooks into Claude Code (`~/.claude/settings.json`) and Codex (`~/.codex/hooks.json`). After a `git push` or `gh pr create/ready`, the hook reminds the agent to monitor CI. When the agent tries to stop on a branch with an open PR, the stop hook runs `bellwether check` for that PR and continues the turn if the PR is still not merge-ready.
+Bellwether can install `PostToolUse` and `Stop` hooks into Claude Code (`~/.claude/settings.json`) and Codex (`~/.codex/hooks.json`). After a `git push` or `gh pr create/ready`, the hook tells the agent to resume the Bellwether loop immediately. When the agent tries to stop on a branch with an open PR, the stop hook runs `bellwether check` for that PR and blocks stopping unless the PR is actually merge-ready. Pending or in-progress CI is not treated as success.
 
 ```bash
 # Install hooks into Claude Code and Codex
@@ -102,10 +102,13 @@ check --watch
   → CI failing?        → show filtered error logs → fix → push → repeat
   → Unresolved review? → show comment with context → address → reply → repeat
   → Merge conflict?    → sync branch → push → repeat
+  → CI pending only?   → keep watching, do not stop, do not assume success
   → Still blocked?     → keep watching until ready/timeout → repeat
 ```
 
-It queries GitHub's Check Runs API directly, filters job logs down to signal (compiler errors, test failures — not noise), and surfaces review threads with file and line context. `--watch` returns immediately if actionable work already exists; otherwise it establishes a baseline and waits for new actionable work, readiness, terminal PR state, or an inactivity timeout — no manual polling required.
+It queries GitHub's Check Runs API directly, filters job logs down to signal (compiler errors, test failures — not noise), and surfaces review threads with file and line context. `--watch` returns immediately if actionable work already exists; otherwise it establishes a baseline and waits for new actionable work, readiness, terminal PR state, or an inactivity timeout — no manual polling required. A pending or in-progress job is never a success condition; it means Bellwether should keep waiting until the first actionable signal appears or the PR becomes ready.
+
+When a CI job fails, the agent workflow is: reproduce the failing command locally, fix the root cause, rerun that same failing command locally until it passes, then push and restart the watch. A timeout with green CI and zero unresolved reviews can still mean "waiting for review approval" or another external blocker; that case should not be turned into an infinite watch loop.
 
 ## Agent usage
 

SKILL.md

@@ -11,13 +11,14 @@ allowed-tools: Bash(npx bellwether *) Bash(bunx bellwether *) Bash(bellwether *)
 
 # Bellwether — Drive PR to Merge-Ready
 
-Self-contained cycle: watch CI -> fix failures -> address reviews -> watch again -> until merge-ready.
+Self-contained cycle: watch CI -> the instant work appears, fix it -> push -> watch again -> until merge-ready.
 
 IMPORTANT: Execute ALL work in the main thread. Do NOT use the Agent tool, Task tool, or spawn sub-agents. Track all state in your working memory (context window). This skill runs as a sequential loop — you fetch, evaluate, fix, commit, reply, then loop.
 
 ## Critical: Use the bellwether CLI
 
 - The ONLY way to check CI status, PR state, and reviews is `bellwether check --watch`. This command returns only when the PR is actually merge-ready (`pr.ready=true`), when actionable work appears (CI failures, unresolved reviews, merge conflicts), when the PR reaches a terminal state (`merged` or `closed`), or on timeout. It keeps polling while the PR is still not ready but there is nothing local to fix yet, including pending external checks and `mergeable=blocked|unstable|unknown`. Do NOT add sleep or polling.
+- **`pending` or `in_progress` CI is NOT success, NOT "probably fine", and NOT a reason to stop.** It means the loop is still running. Keep `bellwether check --watch` in control until it returns actionable work or `pr.ready=true`.
 - NEVER use `gh api`, `gh pr checks`, `gh pr view --json`, `gh api repos/*/check-runs`, or any manual GitHub API calls to check CI or review status.
 - NEVER use `sleep` to wait for CI. The `--watch` flag handles waiting internally.
 - NEVER parse review comments manually via `gh api`. The bellwether CLI returns them in structured format.
@@ -33,7 +34,7 @@ When `bellwether check --watch` returns, apply the first matching rule:
 | 1 | Any CI check is failing | Fix CI (Phase 1). Push. Restart watch. |
 | 2 | Any unresolved review comment is actionable | Fix comments (Phase 2). Push. Restart watch. |
 | 3 | `pr.mergeable=dirty\|behind` | Sync branch. Push. Restart watch. |
-| 4 | All checks pending, no actionable reviews | Wait/watch (restart watch). |
+| 4 | Any CI checks are pending or in progress, and there are no actionable reviews yet | Immediately restart `bellwether check --watch`. This is waiting, not success; do not report completion, do not stop, and do not say it will probably pass. |
 | 5 | `pr.ready=true` | Done. Report "merge-ready". |
 | 6 | `pr.state=merged\|closed` | Done. Report status. |
 | 7 | CI green, 0 unresolved reviews, only missing PR review approval | Done. Report "waiting for review approval". |
@@ -42,6 +43,8 @@ When `bellwether check --watch` returns, apply the first matching rule:
 
 **Pending CI does not block review work.** If CI is pending but unresolved actionable review comments exist, fix the reviews (rule 2) immediately. Do not wait for CI to finish first.
 
+**The first moment new work appears, act immediately.** The instant `bellwether check --watch` returns with a CI failure, unresolved actionable review, merge conflict, or behind branch, start fixing it right away. Do not defer it to a later pass.
+
 **Unresolved review comments on touched or related code are always in scope.** Do not wait for a new user prompt. The only exceptions:
 - Remaining blockers are purely external/non-code and you have explicitly said so.
 - You are blocked by missing information that cannot be discovered locally.
@@ -51,9 +54,14 @@ When `bellwether check --watch` returns, apply the first matching rule:
 ```
 1. bellwether check --watch        (returns only on pr.ready=true, actionable work, merged/closed PR state, or timeout; DO NOT substitute with gh/GitHub API calls — use this exact command)
 2. Apply the Decision Table above — the first matching rule determines your action
-3. If timed out -> go to 1 (restart watch)
+3. If timed out:
+   - If any CI job is still pending or in progress -> go to 1 (restart watch)
+   - If CI is green, unresolved reviews are 0, and the only remaining blocker is review approval or another external dependency you cannot clear yourself -> done, report that exact external blocker
+   - Otherwise -> go to 1 (restart watch)
 ```
 
+Never interpret a pending check or in-progress job as "close enough". Those states are never terminal. A timeout only becomes terminal when it confirms there is no actionable work left and the remaining blocker is purely external.
+
 ## What `bellwether check --watch` returns
 
 Three sections:
@@ -69,12 +77,13 @@ Important: `allPassing=true` is not sufficient on its own. The only success cond
 For each `FAIL` key in the CI section:
 
 1. **Read the error log** — it contains actual compiler/test output with file paths and line numbers.
-2. **Fix the code** — minimal change that resolves the root cause.
-3. **Verify locally** — run the same check that failed.
-4. **Stage, commit, push** — stage files by name (never `git add -A`).
+2. **Reproduce the failure locally first** — identify the exact failing command or the closest local equivalent and run it before changing code. If the exact CI command cannot run locally, use the nearest faithful reproduction and explicitly note why.
+3. **Fix the code** — minimal change that resolves the root cause.
+4. **Re-run the failing command locally until it passes** — do not treat the fix as done just because the code looks right.
+5. **Stage, commit, push** — stage files by name (never `git add -A`).
 5. **Go to step 1 of the loop** — restart the watch. New CI runs, new bot comments may arrive.
 
-DO NOT proceed to Phase 2 until CI is green. Fix CI first, push, restart the watch.
+DO NOT proceed to Phase 2 until CI is green. When CI fails, the required sequence is: reproduce locally -> fix -> rerun the failing command locally -> push -> restart the watch.
 
 ## Phase 2: Address review comments
 
@@ -131,6 +140,9 @@ DO NOT restart the watch until ALL replies are posted AND all threads are resolv
 
 - **Fix everything, don't ask** — your job is to resolve all issues autonomously. Fix CI failures, address reviews, resolve conflicts. Do NOT ask the user "should I fix this?", "want me to keep watching?", "should I come back later?", or any variation — the answer is always yes, keep going. Only escalate if a fix requires a product decision you genuinely cannot make (e.g. choosing between two valid business rules).
 - **NEVER pause or prompt the user for continuation** — CI runs can take a long time (10+ minutes). This is normal. Always continue the loop until `pr.ready=true`, `pr.state=merged|closed`, or you have exhausted all possible fixes. Do NOT ask the user if they want to wait, come back later, or stop watching. The `--watch` flag handles waiting — trust it and keep looping.
+- **`in_progress` means "stay in the loop", not "we're probably fine"** — an in-progress job is unfinished work. Stay alive, keep Bellwether watching, and be ready to fix the first failure or review that appears.
+- **React immediately to the first actionable signal** — if Bellwether returns because one job failed, one review thread opened, or the branch became dirty/behind, start fixing it right then. Do not stop, summarize, or assume a later pass will take care of it.
+- **A CI fix is not complete until the failing command passes locally** — when Bellwether surfaces a red check, reproduce that failure locally first, then rerun the same failing command locally after the fix. Only push once the local repro passes, unless the CI environment cannot be reproduced and you explicitly state the closest equivalent you verified instead.
 - **A non-mergeable PR with unresolved comments is work, not status** — 25 unresolved review comments is not an occasion to report status. It is a mandatory task queue. If the Decision Table says fix, fix. Never misclassify a review backlog as informational.
 - **One fix per watch cycle** — fix CI OR reviews, not both. Push and restart watch.
 - **Minimal changes** — don't refactor unrelated code.

skills/bellwether/SKILL.md

@@ -11,13 +11,14 @@ allowed-tools: Bash(npx bellwether *) Bash(bunx bellwether *) Bash(bellwether *)
 
 # Bellwether — Drive PR to Merge-Ready
 
-Self-contained cycle: watch CI -> fix failures -> address reviews -> watch again -> until merge-ready.
+Self-contained cycle: watch CI -> the instant work appears, fix it -> push -> watch again -> until merge-ready.
 
 IMPORTANT: Execute ALL work in the main thread. Do NOT use the Agent tool, Task tool, or spawn sub-agents. Track all state in your working memory (context window). This skill runs as a sequential loop — you fetch, evaluate, fix, commit, reply, then loop.
 
 ## Critical: Use the bellwether CLI
 
 - The ONLY way to check CI status, PR state, and reviews is `bellwether check --watch`. This command returns only when the PR is actually merge-ready (`pr.ready=true`), when actionable work appears (CI failures, unresolved reviews, merge conflicts), when the PR reaches a terminal state (`merged` or `closed`), or on timeout. It keeps polling while the PR is still not ready but there is nothing local to fix yet, including pending external checks and `mergeable=blocked|unstable|unknown`. Do NOT add sleep or polling.
+- **`pending` or `in_progress` CI is NOT success, NOT "probably fine", and NOT a reason to stop.** It means the loop is still running. Keep `bellwether check --watch` in control until it returns actionable work or `pr.ready=true`.
 - NEVER use `gh api`, `gh pr checks`, `gh pr view --json`, `gh api repos/*/check-runs`, or any manual GitHub API calls to check CI or review status.
 - NEVER use `sleep` to wait for CI. The `--watch` flag handles waiting internally.
 - NEVER parse review comments manually via `gh api`. The bellwether CLI returns them in structured format.
@@ -33,7 +34,7 @@ When `bellwether check --watch` returns, apply the first matching rule:
 | 1 | Any CI check is failing | Fix CI (Phase 1). Push. Restart watch. |
 | 2 | Any unresolved review comment is actionable | Fix comments (Phase 2). Push. Restart watch. |
 | 3 | `pr.mergeable=dirty\|behind` | Sync branch. Push. Restart watch. |
-| 4 | All checks pending, no actionable reviews | Wait/watch (restart watch). |
+| 4 | Any CI checks are pending or in progress, and there are no actionable reviews yet | Immediately restart `bellwether check --watch`. This is waiting, not success; do not report completion, do not stop, and do not say it will probably pass. |
 | 5 | `pr.ready=true` | Done. Report "merge-ready". |
 | 6 | `pr.state=merged\|closed` | Done. Report status. |
 | 7 | CI green, 0 unresolved reviews, only missing PR review approval | Done. Report "waiting for review approval". |
@@ -42,6 +43,8 @@ When `bellwether check --watch` returns, apply the first matching rule:
 
 **Pending CI does not block review work.** If CI is pending but unresolved actionable review comments exist, fix the reviews (rule 2) immediately. Do not wait for CI to finish first.
 
+**The first moment new work appears, act immediately.** The instant `bellwether check --watch` returns with a CI failure, unresolved actionable review, merge conflict, or behind branch, start fixing it right away. Do not defer it to a later pass.
+
 **Unresolved review comments on touched or related code are always in scope.** Do not wait for a new user prompt. The only exceptions:
 - Remaining blockers are purely external/non-code and you have explicitly said so.
 - You are blocked by missing information that cannot be discovered locally.
@@ -51,9 +54,14 @@ When `bellwether check --watch` returns, apply the first matching rule:
 ```
 1. bellwether check --watch        (returns only on pr.ready=true, actionable work, merged/closed PR state, or timeout; DO NOT substitute with gh/GitHub API calls — use this exact command)
 2. Apply the Decision Table above — the first matching rule determines your action
-3. If timed out -> go to 1 (restart watch)
+3. If timed out:
+   - If any CI job is still pending or in progress -> go to 1 (restart watch)
+   - If CI is green, unresolved reviews are 0, and the only remaining blocker is review approval or another external dependency you cannot clear yourself -> done, report that exact external blocker
+   - Otherwise -> go to 1 (restart watch)
 ```
 
+Never interpret a pending check or in-progress job as "close enough". Those states are never terminal. A timeout only becomes terminal when it confirms there is no actionable work left and the remaining blocker is purely external.
+
 ## What `bellwether check --watch` returns
 
 Three sections:
@@ -69,12 +77,13 @@ Important: `allPassing=true` is not sufficient on its own. The only success cond
 For each `FAIL` key in the CI section:
 
 1. **Read the error log** — it contains actual compiler/test output with file paths and line numbers.
-2. **Fix the code** — minimal change that resolves the root cause.
-3. **Verify locally** — run the same check that failed.
-4. **Stage, commit, push** — stage files by name (never `git add -A`).
+2. **Reproduce the failure locally first** — identify the exact failing command or the closest local equivalent and run it before changing code. If the exact CI command cannot run locally, use the nearest faithful reproduction and explicitly note why.
+3. **Fix the code** — minimal change that resolves the root cause.
+4. **Re-run the failing command locally until it passes** — do not treat the fix as done just because the code looks right.
+5. **Stage, commit, push** — stage files by name (never `git add -A`).
 5. **Go to step 1 of the loop** — restart the watch. New CI runs, new bot comments may arrive.
 
-DO NOT proceed to Phase 2 until CI is green. Fix CI first, push, restart the watch.
+DO NOT proceed to Phase 2 until CI is green. When CI fails, the required sequence is: reproduce locally -> fix -> rerun the failing command locally -> push -> restart the watch.
 
 ## Phase 2: Address review comments
 
@@ -131,6 +140,9 @@ DO NOT restart the watch until ALL replies are posted AND all threads are resolv
 
 - **Fix everything, don't ask** — your job is to resolve all issues autonomously. Fix CI failures, address reviews, resolve conflicts. Do NOT ask the user "should I fix this?", "want me to keep watching?", "should I come back later?", or any variation — the answer is always yes, keep going. Only escalate if a fix requires a product decision you genuinely cannot make (e.g. choosing between two valid business rules).
 - **NEVER pause or prompt the user for continuation** — CI runs can take a long time (10+ minutes). This is normal. Always continue the loop until `pr.ready=true`, `pr.state=merged|closed`, or you have exhausted all possible fixes. Do NOT ask the user if they want to wait, come back later, or stop watching. The `--watch` flag handles waiting — trust it and keep looping.
+- **`in_progress` means "stay in the loop", not "we're probably fine"** — an in-progress job is unfinished work. Stay alive, keep Bellwether watching, and be ready to fix the first failure or review that appears.
+- **React immediately to the first actionable signal** — if Bellwether returns because one job failed, one review thread opened, or the branch became dirty/behind, start fixing it right then. Do not stop, summarize, or assume a later pass will take care of it.
+- **A CI fix is not complete until the failing command passes locally** — when Bellwether surfaces a red check, reproduce that failure locally first, then rerun the same failing command locally after the fix. Only push once the local repro passes, unless the CI environment cannot be reproduced and you explicitly state the closest equivalent you verified instead.
 - **A non-mergeable PR with unresolved comments is work, not status** — 25 unresolved review comments is not an occasion to report status. It is a mandatory task queue. If the Decision Table says fix, fix. Never misclassify a review backlog as informational.
 - **One fix per watch cycle** — fix CI OR reviews, not both. Push and restart watch.
 - **Minimal changes** — don't refactor unrelated code.