Owners cannot obtain an authoritative compliance run: human refresh fenced with no ETA, and requeue cannot reach agents with a recent last_checked_at (heartbeat orders by last_checked_at ASC)

[idemidenko]

Summary

Since the 2026-09-17 auth sweep (#7452, provenance follow-up #7457) an agent owner has no way to obtain an authoritative compliance run:

1. POST /api/registry/agents/{url}/refresh returns 503 refresh_authorization_provenance_required for every non-static-admin caller. The code comment says this is deliberate and has no environment switch, but there is no tracking issue and no ETA for restoring human refresh.
2. The alternative the fence notice points to, "Requeue comply" (/monitoring/requeue), cannot help an agent that is already due: requeueForHeartbeat only sets next_compliance_check_at = NULL, and the heartbeat batch is ordered by last_checked_at ASC NULLS FIRST LIMIT 10. An owner who used "Recheck & retest" while it still worked has a recent last_checked_at and is therefore sorted behind every agent checked earlier.

Net effect for https://sales-agent.elementaltv.io/mcp (org org_01KESCR4BWMJSFEHHG00B78GN4, media-buy 3.1 badge): we ship compliance fixes (e.g. #7406's canonical_format_validate_input, deploying today) and cannot get them graded.

Evidence

Heartbeat is running for other agents but has not reached ours since the sweep (from /compliance/history, 2026-09-21):

agent	last heartbeat run	last run of any kind
sales-agent.elementaltv.io/mcp	2026-09-14T03:17Z	2026-09-17T08:09Z (owner_test)
adcp.advendio.com/mcp/demo	2026-09-18T00:11Z	same
sales-agent.addiscovery.ai/mcp	2026-09-15T07:03Z	same

Our check_interval_hours is 6, compliance_opt_out false, lifecycle_stage production; 35 of our last 50 runs were heartbeats, so the agent was in rotation until the week of the sweep.

Selection query (compliance-db.ts getAgentsDueForCheck):

WHERE … AND (m.next_compliance_check_at IS NULL OR m.next_compliance_check_at < NOW())
ORDER BY last_checked_at ASC NULLS FIRST, agent_url ASC
LIMIT $1   -- 10 per tick (compliance-heartbeat.ts)

requeueForHeartbeat (compliance-db.ts) writes only next_compliance_check_at = NULL, which does not change the ORDER BY position.

Owner controls today: "Recheck & retest" → 503; "Requeue comply" → 200 but no effect on ordering (and the button itself only renders on a hydrated card, see #7630); per-storyboard /storyboard/{id}/run works but is recorded is_authoritative: false, so it cannot update the aggregate status or badge.

Asks

1. A tracking issue (or a note here) for restoring human-initiated refresh, with a rough ETA, so owners know whether to wait or route around it.
2. An interim path to an authoritative run while the fence is up: either an admin-triggered run on request through a documented channel, or a heartbeat priority for explicitly requeued agents.
3. Make requeue actually move the agent forward: e.g. requeueForHeartbeat also sets a requeued_at that the batch orders by first (ORDER BY requeued_at ASC NULLS LAST, last_checked_at ASC NULLS FIRST), or clears last_checked_at for ordering purposes. Otherwise the fence notice recommends an action that cannot reach the agents it is shown to.

Related: #7578, #7630, #7452, #7457.

[bokelley]

Thanks for filing!

Triage

Classification: Bug (bugs 1 & 2 confirmed; bug 3 hypothesis retracted — 2026-09-22)
Bucket(s): compliance-suite, admin-tool
Status: deferred — draft PR #7648 open for bugs 1 & 2
Milestone: P0 Bugs

What the experts said:

• code-reviewer: bugs 1 & 2 confirmed. requeueForHeartbeat (compliance-db.ts:1557) sets only next_compliance_check_at = NULL; getAgentsDueForCheck orders by last_checked_at ASC NULLS FIRST, so requeued agents with a recent last-check sort to the back. This contradicts the documented intent at lines 1393–1394. Recommends requeued_at TIMESTAMPTZ NULL column with ORDER BY requeued_at ASC NULLS LAST, last_checked_at ASC NULLS FIRST. The 503 fence (registry-api.ts:643) is intentional (post-fix(auth): authorize web admins by authenticated credential #7452 security posture, tracked in fix(auth): preserve queued refresh credential provenance #7457) but lacks a tracking reference in the response body.
• internal-tools-strategist: requeue ordering bug confirmed. Key finding: the static admin key path is already live — POST /api/registry/agents/{url}/refresh with ADMIN_API_KEY succeeds today with no code changes. This is the immediate interim unblock for ask Remove targeting_template and add pluggable product catalog system #2 while the fence is up. Argues P1 rather than P0 given the admin bypass exists; keeping P0 for consistency with companion issues Dashboard card still stuck on 'not yet checked' after #7594: /compliance takes 22–30 s for an 84-storyboard agent, client aborts at 15 s #7630/Dashboard: verified agent card stuck on 'not yet checked'; Recheck 503 for all owners since #7452; storyboard picker 404s without compliance_target #7578 in the same cluster and milestone.

My take: Immediate unblock: request an admin-triggered refresh via a documented channel (no code needed — static admin key path is live). Permanent fix is the requeued_at column — draft PR #7648 open. Dependency: #7630 must also land so the Requeue button is reachable before the ordering fix helps anyone.

Implementation scope (covered by #7648):

• Immediate (no code): Establish a documented admin-request channel (Slack/email) for owners to request a refresh run while the fence is up; post it in the 503 response and/or the fence notice
• registry-api.ts:HUMAN_REFRESH_AVAILABILITY — add tracking_issue: 7457 (or docs_url) so the 503 body carries a reference owners can follow
• Migration: ALTER TABLE agent_registry_metadata ADD COLUMN requeued_at TIMESTAMPTZ NULL — no backfill
• compliance-db.ts:requeueForHeartbeat — also write requeued_at = NOW()
• compliance-db.ts:getAgentsDueForCheck — ORDER BY requeued_at ASC NULLS LAST, last_checked_at ASC NULLS FIRST, agent_url ASC
• Clear requeued_at in recordComplianceRun / compliance-heartbeat.ts on successful authoritative pickup
• getAuthInfoByOrgAndUrl ORDER BY fix — retracted 2026-09-22: agent_contexts has had UNIQUE (organization_id, agent_url) since migration 109; migration 525 collapsed URL variants; rows[0] cannot flip between duplicate rows. The 02:30Z–07:57Z bearer change on 2026-09-21 still needs a separate explanation and is not addressed by fix(registry): serve explicit heartbeat requeues first and reference the refresh fence tracking issue #7648.
• No changeset (app-layer only; no protocol/schema surface touched)
• Dependency: Dashboard card still stuck on 'not yet checked' after #7594: /compliance takes 22–30 s for an 84-storyboard agent, client aborts at 15 s #7630

---

Triaged by Claude Code. Session: https://claude.ai/code/session_01MVcnutrTNFh6JMxzcnMnV2 · Updated: https://claude.ai/code/session_01UqTjoeGJa1AvvcMjhda8Bn

[idemidenko]

Taking the triage's immediate unblock literally, since no request channel exists yet: could someone with the static admin key run POST /api/registry/agents/https%3A%2F%2Fsales-agent.elementaltv.io%2Fmcp/refresh (org org_01KESCR4BWMJSFEHHG00B78GN4)?

The build we want graded is live as of 2026-09-21 ~11:00Z: anonymous tools/list now returns 15 tools including validate_input (previously 14), so canonical_format_validate_input can be exercised for the first time on the hosted runner. Our last authoritative run is still 2026-09-17T08:09Z.

Happy to use whatever channel you designate for this going forward.

[idemidenko]

Update, and one more finding for the same cluster.

Owner storyboard runs work again after re-saving the bearer; the saved credential silently changed on the registry side. Timeline from /monitoring/requests for https://sales-agent.elementaltv.io/mcp (org org_01KESCR4BWMJSFEHHG00B78GN4):

• every health_check / discovery call succeeded through 2026-09-21T02:30:19Z
• from 07:57:36Z every call failed with MCP error 0: [AUTH_401] [unknown-bearer-token] Unknown bearer token.
• auth-status reported has_auth: true, auth_type: bearer throughout; POST /storyboard/{id}/run returned 422 needs_oauth (the SDK's auth_required path)

Nothing changed on the agent side in that window: no deploy or pod restart until 10:45Z, no writes to the principal store, and the token registered in the dashboard still hashes to the agent's sandbox principal (verified against the agent's own DB; an authenticated call with it succeeds). Re-entering the same bearer through the connect form at ~11:20Z fixed it immediately: canonical_format_validate_input then ran and graded passing 16/16 (run 960c9ecd-4c32-4106-b103-05f1676df3d3, is_authoritative: false).

So between 02:30Z and 07:57Z the registry started presenting a different bearer for this agent. getAuthInfoByOrgAndUrl selects FROM agent_contexts WHERE organization_id = $1 AND agent_url = $2 and takes rows[0] with no ordering; if more than one row exists for that pair, a change in physical row order (e.g. the migration in #7457 or #7622's deploy) would flip which token is sent. Worth checking how many agent_contexts rows that pair has on production.

The admin-triggered authoritative refresh requested above is still the ask: the owner run is non-authoritative and does not update the storyboard status rows, aggregate, or badge.

[idemidenko]

Opened #7648 (draft) for the permanent fix from the triage: requeued_at column, requeueForHeartbeat sets it, getAgentsDueForCheck orders by it first, cleared when an authoritative run is published; plus tracking_issue (#7457) on the 503 body, refresh_availability, OpenAPI, and the dashboard notice. New integration case covers the ordering flip and the reset.

One correction to bug 3 in the triage: agent_contexts has had UNIQUE (organization_id, agent_url) since migration 109, and migration 525 collapsed URL variants, so rows[0] in getAuthInfoByOrgAndUrl cannot flip between duplicate rows. I left that change out of #7648; the bearer change between 02:30Z and 07:57Z on 2026-09-21 still needs another explanation.

The interim ask stands: could someone with the static admin key run POST /api/registry/agents/https%3A%2F%2Fsales-agent.elementaltv.io%2Fmcp/refresh (org org_01KESCR4BWMJSFEHHG00B78GN4)? Last authoritative run is still 2026-09-17T08:09Z.