Merge pull request #943 from sortie-ai/feat/936-budget-stop-visibility

feat(orchestrator): surface per-issue budget holds to operators

Author: sergeyklay

CHANGELOG.md

@@ -15,6 +15,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `sortie validate` now reports a warning when an agent block sets `mcp_config` for an agent kind that never receives the generated MCP configuration file. `claude-code`, `codex`, `copilot-cli` and `opencode` receive it; `kiro` does not, so an `mcp_config` value in a `kiro` block had no effect and nothing said so. The reference documentation now states which kinds consume the file. It is a warning and not an error: such a configuration stays valid, the run proceeds, and the exit code is unchanged.
   ([#928](https://github.com/sortie-ai/sortie/issues/928))
 
+- An issue that Sortie has stopped dispatching because it reached its `agent.max_sessions` or `agent.max_tokens` ceiling now says so. Previously such an issue stayed in an active tracker state and was simply never picked up again, with no log line at any level, no marker on the tracker, and `GET /api/v1/{identifier}` reporting it as unknown; the only way to explain the stall was to count the issue's rows in the run history against the ceiling configured in `WORKFLOW.md`. The orchestrator now logs a warning naming the issue, which of the two ceilings stopped it, and its usage against that ceiling, once when the hold begins rather than on every poll, and the tick summary reports how many issues are held. `GET /api/v1/state` lists them with the same numbers, the per-issue endpoint answers for a held issue instead of reporting it unknown, and the dashboard shows a card and a table. Two metrics, `sortie_budget_exhaustions_total` and `sortie_budget_exhausted_issues`, report how often a hold begins and how many issues are held right now. What the ceilings count, and when they stop an issue, is unchanged.
+  ([#936](https://github.com/sortie-ai/sortie/issues/936))
+
 ### Fixed
 
 - A malformed end-of-turn notification from the `codex app-server` no longer leaves the turn outcome reported as the bare word `turn` followed by a trailing space. A payload that fails to parse carries no status word, so the turn now reports the shared failure message instead: the status API's `last_message` field and the recorded run history both read `turn failed`.

docs/architecture/05-workflow-specification.md

@@ -221,15 +221,20 @@ Fields:
   - Default: `0` (unlimited; no effort budget enforced).
   - Maximum number of completed worker sessions for a single issue before the orchestrator
     stops re-dispatching it. Counted from `run_history` entries.
-  - When the count reaches `max_sessions`, the claim is released and a warning is logged.
+  - When the count reaches `max_sessions`, a warning is logged. The retry handler releases the
+    claim it holds; the poll tick's own rebuild of the exhausted-issue set writes the candidate
+    into that set instead, because a held candidate never held a claim to release.
   - `0` disables the budget (unlimited retries).
   - Changes are re-applied at runtime and affect future retry timer evaluations.
 - `max_tokens` (integer)
   - Default: `0` (unlimited; no token budget enforced).
   - Cumulative per-issue token ceiling. The orchestrator sums `total_tokens` across the
     issue's `run_history` entries and stops re-dispatching once the sum reaches `max_tokens`.
-  - When the sum reaches `max_tokens`, the claim is released and a warning is logged, exactly
-    as `max_sessions` does. A failed token query fails open: dispatch proceeds.
+  - When the sum reaches `max_tokens`, a warning is logged, on the same two lanes and for the
+    same reason `max_sessions` states above. A failed token query fails open, but not the same
+    way on both lanes: the retry handler's check is skipped and dispatch proceeds for that
+    issue, while the rebuild folds the prior set's entries for the failing axis forward
+    unchanged and keeps the other axis fresh.
   - A run whose coding agent reported no token usage is recorded unmeasured and contributes
     nothing to the sum. A sum below `max_tokens` that includes at least one unmeasured run
     allows the dispatch and logs a warning naming the issue, the sum, the ceiling, and the

docs/architecture/06-configuration-specification.md

@@ -118,10 +118,11 @@ Validation checks:
 Effort-budget and notification config are validated outside this preflight, by design:
 
 - The per-issue token ceiling (`agent.max_tokens`) is not a scheduler preflight check. It is a
-  re-dispatch gate evaluated on the retry path alongside `agent.max_sessions` (Section 8.4), so
-  the ceiling stops a blocked issue from being re-dispatched rather than failing startup or a
-  poll tick. Config-level validation rejects a negative `agent.max_tokens` when the config is
-  parsed, which both startup validation and the live-reload fail-safe path consume.
+  re-dispatch gate evaluated on both the retry path and the poll tick's own rebuild, alongside
+  `agent.max_sessions` (Section 8.4), so the ceiling stops a blocked issue from being
+  re-dispatched rather than failing startup or a poll tick. Config-level validation rejects a
+  negative `agent.max_tokens` when the config is parsed, which both startup validation and the
+  live-reload fail-safe path consume.
 - The `notifications` backend list (Section 5.3.11) is structurally validated when the config is
   parsed: the value must be a sequence, every entry must carry a non-empty string `kind`, and
   `max_per_session`, when present, must be a non-negative integer. `max_per_session` is optional:

docs/architecture/08-polling-scheduling-and-reconciliation.md

@@ -130,37 +130,55 @@ Retry handling behavior:
 
 Per-issue effort budget (defense-in-depth):
 
-- When `agent.max_sessions > 0`, the retry handler counts completed sessions for the issue
-  from `run_history` before fetching candidates.
-- If the count reaches `max_sessions`, the claim is released and a warning is logged instead
-  of re-dispatching.
-- If the count query fails, the budget check is skipped (fail-open) and dispatch proceeds
-  normally.
+- When `agent.max_sessions > 0`, two lanes evaluate it: the retry handler counts completed
+  sessions for the issue from `run_history` before fetching candidates, once per retry; the
+  poll tick's rebuild runs the same count as one batch query over the whole candidate set,
+  once per tick, before dispatch (see the rebuild bullet below).
+- If the count reaches `max_sessions`, a warning is logged on both lanes. The retry handler
+  releases the claim it holds and does not re-dispatch; the rebuild writes the candidate into
+  the exhausted-issue set instead, because a candidate the rebuild holds never held a claim to
+  release.
+- If the retry handler's count query fails, its check is skipped (fail-open) and dispatch
+  proceeds normally for that issue. The rebuild's own fail-open, which differs because it
+  rebuilds a whole set rather than deciding one dispatch, is described in the rebuild bullet
+  below.
 - `max_sessions = 0` (default) disables the budget entirely.
 
 Per-issue token budget (cost ceiling):
 
-- When `agent.max_tokens > 0`, the retry handler sums `total_tokens` across the issue's
-  `run_history` entries on the same pre-dispatch path, after the session check.
-- If the sum reaches `max_tokens`, the claim is released and a warning is logged instead of
-  re-dispatching, identical in mechanism to the session ceiling.
+- When `agent.max_tokens > 0`, two lanes evaluate it: the retry handler sums `total_tokens`
+  across the issue's `run_history` entries on the same pre-dispatch path, after the session
+  check, once per retry; the poll tick's rebuild runs the same sum as one batch query over the
+  whole candidate set, once per tick, after the session-count query (see the rebuild bullet
+  below).
+- If the sum reaches `max_tokens`, a warning is logged on both lanes. The retry handler
+  releases the claim it holds and does not re-dispatch; the rebuild writes the candidate into
+  the exhausted-issue set instead, for the reason the effort-budget bullet above states.
 - The session and token ceilings are independent hard ceilings. A re-dispatch is blocked when
   either is reached, so whichever fills first across polling cycles is the one that fires. When
   a single evaluation finds both exhausted, the reported and logged reason names the token
   budget; the block itself is identical regardless of which ceiling triggered it.
 - The per-tick rebuild of the exhausted-issue set accounts for these two ceilings only: it runs
   the session-count batch query when `max_sessions > 0` and the token-sum batch query when
-  `max_tokens > 0`, and unions the results. Each blocked issue carries a machine-readable reason
-  (`token_budget` or `session_budget`, token taking precedence over session) surfaced in the
-  runtime snapshot beside the exhausted set. The rebuild is skipped entirely when both budgets
-  are disabled.
+  `max_tokens > 0`, and unions the results, token taking precedence over session when one issue
+  reaches both. On the tick an issue enters the set, the rebuild emits one warning record naming
+  the issue, the reason, and the used and budgeted numbers, and increments a counter for that
+  reason; a tick that re-observes an issue already in the set under the same reason emits
+  neither, so the record and the counter fire once per hold rather than once per tick. Each
+  blocked issue's reason and numbers travel with its own entry, keyed by issue, rather than in a
+  map alongside the set. On a query error for one axis, the rebuild folds the prior set's
+  entries for that axis forward unchanged and keeps the other axis fresh, which is a different
+  fail-open mechanism from the retry lane's own skip-and-proceed; dispatch still proceeds for
+  every candidate not in the set. The rebuild is skipped entirely when both budgets are
+  disabled.
 - The consecutive handoff-absence ceiling (§14.2) is evaluated separately, by the same
   mechanism that parks a `blocked` soft stop, after the release-evaluation step below so a
   park released this tick is not immediately re-parked. It shares the `parked` reason
   vocabulary with the `blocked` park rather than the exhausted-issue set's own reasons, and is
   skipped entirely under `tracker.handoff_evidence: off`, which records no absence.
-- If the token query fails, the check fails open and dispatch proceeds, matching the session
-  check. A token sum recorded before the token columns were added reads as zero.
+- If the retry handler's token query fails, its check fails open and dispatch proceeds for that
+  issue, the same fail-open shape the session check above uses. A token sum recorded before the
+  token columns were added reads as zero.
 - A sum below the ceiling that includes at least one unmeasured run allows the dispatch and
   logs a warning naming the issue, the sum, the ceiling, and the unmeasured count, matching the
   visibility of the query-failure fail-open case above. The retry path warns on every occurrence

docs/architecture/18-logging-status-and-observability.md

@@ -42,6 +42,14 @@ message `"park label write failed"`, at `Warn`, and does not suppress the parkin
 a park emits one `Info` record, message `"issue unparked"`, carrying `trigger`
 (`state_changed`, `label_removed`, or `evidence_observed`) and `reason`.
 
+An issue entering the per-issue budget-exhausted set, on either the poll tick's rebuild or the
+retry lane, emits exactly one `Warn` record, message `"candidate held by budget ceiling"`, carrying
+the standard issue context fields plus `reason` (`token_budget` or `session_budget`),
+`used_sessions`, `budget_sessions`, and, when the token ceiling was evaluated for that issue,
+`used_tokens` and `budget_tokens`. The record fires once per hold: a tick that re-observes an
+already-announced hold under the same reason emits nothing further, and whichever lane discovers a
+hold is the only one that announces it.
+
 The periodic workspace sweep emits exactly one summary record per pass, at `Info` level, message
 `"sweep: pass complete"`, on every pass that produced a candidate set, including a pass over zero
 keys, a pass whose tracker read failed, and a pass that removed nothing. This is deliberate: a
@@ -66,7 +74,9 @@ removed expired workspace"`, carrying `workspace_key`, `last_activity` (RFC3339)
 
 The `"tick completed"` `Info` record carries four integer attributes for the blocker gate in
 addition to its existing ones: `held_by_blockers`, `blockers_unresolved`, `blockers_not_read`, and
-`blockers_incomplete`, each counting the candidates the tick held for that reason.
+`blockers_incomplete`, each counting the candidates the tick held for that reason. It also carries
+`budget_exhausted`, the number of this tick's candidates in the per-issue budget-exhausted set
+after the rebuild.
 
 A candidate the dispatch gate holds for a blocker reason produces exactly one of five per-issue
 records, and a pass whose reads were refused by the forge produces exactly one pass-level record in
@@ -122,12 +132,12 @@ should return:
   - `seconds_running` (aggregate runtime seconds as of snapshot time, including active sessions)
 - `rate_limits` (latest coding-agent rate limit payload, if available)
 - `budget_exhausted_count` (number of issues currently blocked by a re-dispatch budget; always present)
-- `budget_exhausted` (sorted list of blocked issue IDs; omitted when the set is empty)
-- `budget_exhausted_reason` (map from blocked issue ID to the gate that fired, `token_budget` or
-  `session_budget`; `token_budget` takes precedence over `session_budget` when one issue reaches
-  both gates; omitted when the set is empty). The exhausted set and its reasons are rebuilt per
-  tick from those two gates (Section 8.4); an issue's reason entry exists exactly when that issue
-  is in `budget_exhausted`.
+- `budget_exhausted` (list of blocked-issue records, sorted by identifier; always present, empty
+  when the set is empty). Each record carries the issue's ID and identifier, the reason
+  (`token_budget` or `session_budget`; `token_budget` takes precedence over `session_budget` when
+  one issue reaches both gates), the used and budgeted session and token counts, and the time the
+  hold began. The set is rebuilt per tick from those two gates (Section 8.4) and also updated by
+  the retry lane when it discovers a hold between ticks.
 - `parked_count` (number of issues currently held out of primary dispatch; always present)
 - `parked` (sorted list of parked issue IDs; omitted when the set is empty)
 - `parked_reason` (map from parked issue ID to the park's reason, `agent_blocked` or
@@ -281,7 +291,8 @@ Minimum endpoints:
       "generated_at": "2026-02-24T20:15:30Z",
       "counts": {
         "running": 2,
-        "retrying": 1
+        "retrying": 1,
+        "budget_exhausted": 1
       },
       "running": [
         {
@@ -317,6 +328,19 @@ Minimum endpoints:
           "error": "no available orchestrator slots"
         }
       ],
+      "budget_exhausted": [
+        {
+          "issue_id": "ghi789",
+          "issue_identifier": "MT-651",
+          "reason": "session_budget",
+          "used_sessions": 3,
+          "budget_sessions": 3,
+          "used_tokens": null,
+          "budget_tokens": 0,
+          "unmeasured_sessions": null,
+          "exhausted_at": "2026-02-24T20:12:00Z"
+        }
+      ],
       "agent_totals": {
         "input_tokens": 5000,
         "output_tokens": 2400,
@@ -360,6 +384,7 @@ Minimum endpoints:
         }
       },
       "retry": null,
+      "budget_exhausted": null,
       "logs": {
         "agent_session_logs": [
           {
@@ -381,8 +406,13 @@ Minimum endpoints:
     }
     ```
 
+  - `status` is `"running"` when the issue has a running session, otherwise `"retrying"` when it has
+    a pending retry, otherwise `"budget_exhausted"` when it is held out of dispatch by a per-issue
+    budget ceiling; `budget_exhausted` in the response body carries the same record shape as the
+    `budget_exhausted` array on `GET /api/v1/state` and is `null` when the issue is not in that set.
   - If the issue is unknown to the current in-memory state, return `404` with an error response
-    (for example `{"error":{"code":"issue_not_found","message":"..."}}`).
+    (for example `{"error":{"code":"issue_not_found","message":"..."}}`); an issue held by a budget
+    ceiling is not unknown to it, so this case is distinct from that one.
 
 - `POST /api/v1/refresh`
   - Queues an immediate tracker poll + reconciliation cycle (best-effort trigger; implementations
@@ -451,6 +481,8 @@ Defined metrics (label sets and buckets are specified here; see ADR-0008 for his
 | `sortie_issue_parks_total{reason}` | Counter | Issue park events, partitioned by reason (`agent_blocked`, `handoff_absence`). Incremented once per park, whichever trigger produced it. |
 | `sortie_dispatch_rule_match_total{layer,rule}` | Counter | Dispatch rule match outcomes, partitioned by resolution layer (`rule`, `default`, `fallback`) and matched rule name. Empty rule names report as `<none>` to bound label cardinality. |
 | `sortie_candidate_holds_total{reason}` | Counter | `IncCandidateHolds`. Candidates the dispatch loop held, partitioned by reason (`blocked_by`, `blockers_unresolved`, `blockers_not_read`, `blockers_incomplete`). Incremented once per held candidate; never incremented for a candidate rejected by an eligibility or capacity gate, and never incremented a second time for the pass-level `blocker reads halted for this tick` ERROR that accompanies a run of `blockers_unresolved` holds. |
+| `sortie_budget_exhaustions_total{reason}` | Counter | `IncBudgetExhaustions`. Issue entries into the per-issue budget-exhausted set, partitioned by reason (`token_budget`, `session_budget`, open to a later value). Incremented once per hold, from whichever lane discovered it; never incremented on a tick that merely re-observes an already-announced hold. |
+| `sortie_budget_exhausted_issues{reason}` | Gauge | `SetBudgetExhaustedIssues`. Issues currently held out of dispatch, partitioned by reason. Recomputed from the full budget-exhausted set on every gauge update, including every declared reason at zero when nothing is held, so a reason that clears reports zero rather than freezing at its last value. |
 | `sortie_tool_calls_total{tool,result}` | Counter | Agent tool call completions, partitioned by tool name and result (`success`, `error`). |
 | `sortie_ci_status_checks_total{result}` | Counter | CI status check outcomes, partitioned by result (`passing`, `pending`, `failing`, `error`). |
 | `sortie_ci_escalations_total{action}` | Counter | CI escalation actions when fix retries are exhausted, partitioned by action (`label`, `comment`, `error`). |

docs/architecture/22-test-and-validation-matrix.md

@@ -162,6 +162,21 @@ Unless otherwise noted, Sections 17.1 through 17.7 are `Core Conformance`. Bulle
 - The worker-exit absence park records the same tracker state the run's own terminal observation
   resolved, not an unrecorded state, and is releasable by a later state change without an
   intervening backfill tick
+- An issue entering the per-issue budget-exhausted set, on either the poll tick's rebuild or the
+  retry lane, produces exactly one log record naming the issue, the reason, and the used and
+  budgeted numbers
+- The record and its counter increment fire once per hold: repeated ticks over the same held
+  issue produce neither, and an issue that leaves the candidate set and returns still held under
+  the same reason produces neither either
+- Whichever lane, poll tick or retry timer, discovers a hold is the only one that announces it;
+  the other lane's rebuild or block leaves the announcement memory alone
+- The budget-exhausted gauge reports a per-reason level derived from the current set, seeded to
+  zero for every declared reason on each recompute, so a reason that clears reports zero rather
+  than freezing at its last published value
+- Both `GET /api/v1/state` and `GET /api/v1/{identifier}` carry the budget-exhausted record's
+  identifier, reason, and numbers, and the per-issue endpoint answers for a budget-blocked issue
+  instead of reporting it unknown
+- The dashboard renders a budget-blocked card and table only when the exhausted set is non-empty
 - A dispatch that does not drive issue state performs neither the dispatch-time transition nor the
   handoff transition, and enqueues no reaction on its own exit
 - Abnormal worker exit increments retries with 10s-based exponential backoff