fix(proxy): broaden rate-limit reset detection (more headers + body fields, IETF/Discord/Anthropic/OpenAI)

Author: nnemirovsky

CLAUDE.md

@@ -202,7 +202,7 @@ Auto-failover on 429/401 is primary; `pool rotate` is an override.
 
 - **Classification** (`classifyFailover`, `internal/proxy/pool_failover.go`, from `SluiceAddon.Response` for pooled destinations): `429`/`403 + insufficient_quota` -> rate-limited; `401`/token-body `invalid_grant`/`invalid_token` -> auth-failure; `5xx`/other -> no-op. Token-endpoint body trusted only when the request URL matched the OAuth index.
 - **Pool attribution** (`poolForResponse`): a response is pool-attributed either (a) the flow's CONNECT host has a pooled binding (API-host 429/403), or (b) the request URL matches the OAuth token-URL index for a member (token-endpoint 401/`invalid_grant`). (b) is essential — an OAuth refresh hits `auth.openai.com` (no pool binding; only `api.openai.com` has one), so without it the token-endpoint classification is dead code for Codex. Member recovery + fail-closed are the R1 mechanism above (`OAuthIndex.MatchAll` + the refresh-token join key, never `OAuthIndex.Match`).
-- **Synchronous in-memory failover (I1):** health is updated in-process before the response returns (`MarkCooldown` write lock, `ResolveActive` read lock) so the switch never waits on the 2s watcher (which only reconciles); a detached `onFailover` also writes `SetCredentialHealth(member,'cooldown',now+ttl,reason)` for durability. **Cooldown window (B1):** `cooldownFromResponse(class, f.Response.Header, f.Response.Body)` (`internal/proxy/pool_failover.go`) derives the TTL from the upstream recovery hints — `Retry-After` (delta-seconds or HTTP-date), then `x-ratelimit-reset` / `x-ratelimit-reset-requests` / `x-ratelimit-reset-tokens` (delta or epoch) — clamped to `[floor(class), vault.MaxCooldown=6h]`. When NO such header is present it parses the 429 JSON body (`parseBodyRecoveryHint`) for OpenAI Codex `usage_limit_reached`'s `resets_in_seconds` (delta-from-now, preferred) / `resets_at` (unix epoch); the body reset is authoritative and can be hours/days out, so a body-derived window is clamped to the higher `[floor(class), vault.MaxUsageLimitCooldown=24h]`. Codex usage-limit 429s carry no recovery header, so without the body fallback the pool re-probes an account that won't reset for hours every 60s (the degrade-flap root cause). Neither hint falls back to the class default (`vault.RateLimitCooldown`=60s / `vault.AuthFailCooldown`=300s). Floors: rate-limit `vault.MinRateLimitFloor`=10s (a short parsed window is honored, not floored up to 60s), auth-failure `AuthFailCooldown` (a revoked/expired token is never re-probed in seconds). **Cooldown extension is monotonic on both layers:** a member parked 300s for auth that then trips a 60s rate-limit keeps the LATER expiry — `MarkCooldown` and `SetCredentialHealth`'s `cooldown` upsert (CASE-compared against the stored future `cooldown_until`) both keep `max(existing-future, new)`. Only extend is monotonic: an explicit clear (zero/past `until`) and any transition to `healthy` still shorten/clear, and lazy expiry still wins over an expired stored cooldown. No in-flight retry — next request uses the new member.
+- **Synchronous in-memory failover (I1):** health is updated in-process before the response returns (`MarkCooldown` write lock, `ResolveActive` read lock) so the switch never waits on the 2s watcher (which only reconciles); a detached `onFailover` also writes `SetCredentialHealth(member,'cooldown',now+ttl,reason)` for durability. **Cooldown window (B1):** `cooldownFromResponse(class, f.Response.Header, f.Response.Body)` (`internal/proxy/pool_failover.go`) derives the TTL from the upstream recovery hints, spanning the conventions across AI providers and general rate limiters. Headers are tried in precedence order (first usable wins; `Retry-After` MUST be first per IETF draft-ietf-httpapi-ratelimit-headers): `Retry-After` (RFC 9110 delta-seconds or HTTP-date), `RateLimit-Reset` (IETF draft, delta-seconds), `X-RateLimit-Reset` (generic; GitHub/Twitter unix epoch, others delta), `X-RateLimit-Reset-After` (Discord, delta float), OpenAI `x-ratelimit-reset-requests` / `x-ratelimit-reset-tokens` (unit-suffixed durations), Anthropic `anthropic-ratelimit-requests-reset` / `anthropic-ratelimit-tokens-reset` (RFC3339/ISO-8601 absolute timestamp). Each value parses as delta-seconds, a unix epoch (magnitude-disambiguated at `epochThreshold` ~2001), an HTTP-date (Retry-After only), an RFC3339/ISO-8601 timestamp, or a unit-suffixed duration; clamped to `[floor(class), vault.MaxCooldown=6h]`. When NO usable header is present it parses the JSON body (`parseBodyRecoveryHint`, skipped if empty or >64 KiB) for `resets_in_seconds`, `resets_at`, `retry_after`, or `reset_after` — probed top-level FIRST then nested under a top-level `error` object — covering OpenAI Codex `usage_limit_reached` (`resets_in_seconds`/`resets_at`) and the Discord/generic `retry_after`/`reset_after` conventions. Each body value uses the same epoch-vs-delta disambiguation as the headers; the body reset is authoritative and can be hours/days out, so a body-derived window is clamped to the higher `[floor(class), vault.MaxUsageLimitCooldown=24h]`. Codex usage-limit 429s carry no recovery header, so without the body fallback the pool re-probes an account that won't reset for hours every 60s (the degrade-flap root cause). Neither hint falls back to the class default (`vault.RateLimitCooldown`=60s / `vault.AuthFailCooldown`=300s). Floors: rate-limit `vault.MinRateLimitFloor`=10s (a short parsed window is honored, not floored up to 60s), auth-failure `AuthFailCooldown` (a revoked/expired token is never re-probed in seconds). **Cooldown extension is monotonic on both layers:** a member parked 300s for auth that then trips a 60s rate-limit keeps the LATER expiry — `MarkCooldown` and `SetCredentialHealth`'s `cooldown` upsert (CASE-compared against the stored future `cooldown_until`) both keep `max(existing-future, new)`. Only extend is monotonic: an explicit clear (zero/past `until`) and any transition to `healthy` still shorten/clear, and lazy expiry still wins over an expired stored cooldown. No in-flight retry — next request uses the new member.
 - **Reload doesn't resurrect a cooled member:** the durable write is detached/best-effort, so any reload (SIGHUP or the 2s watcher on any unrelated DB write) rebuilds the resolver from store rows via `NewPoolResolver`; `Server.StorePool` calls `PoolResolver.MergeLiveCooldowns(prev)` to carry forward still-active in-memory cooldowns before the atomic swap (monotonic; drops cooldowns for credentials no longer in any pool).
 - **Exhaustion + edge-triggered notices (A1/A2):** a pool is exhausted iff `PoolResolver.HasHealthyMember(pool)` (RLock, single `now`, mirrors `cooling()` lazy-expiry) is false — NOT `to == from`. Per-pool exhaustion state lives on the long-lived `Server` (`poolExhausted` map, NOT `PoolHealth`, so it survives resolver swaps and is not pruned on membership change). `handlePoolFailover` emits one "pool exhausted" notice on the `false->true` edge and wakes a dedicated recovery monitor goroutine (cap-1 `recoveryWake`). The monitor (`internal/proxy/server.go`, started in `New`, stopped idempotently from both `Close` and `GracefulShutdown`) sleeps until `SoonestCooldown(pool)` (clamped to a ~1s floor), `Load()`s the current resolver each wake, and on `HasHealthyMember -> true` flips `true->false`, emits one "pool recovered" notice (`FormatPoolRecoveredNotice`), and invokes `onPoolRecovered`. This replaces the old per-cooldown-window flap that respammed `cred_failover` + a Telegram notice every ~30/60s.
 - **Recovery auto-reset (opt-in, per pool):** if the recovered pool has a non-empty `auth_reset_target`, `onPoolRecovered` (wired in `cmd/sluice/main.go` via `wirePoolRecovery`) calls `containerMgr.ResetAuth(ctx, target)` in a detached goroutine with a fresh bounded context and emits an `agent_auth_reset` audit event (`Verdict "recover"`, `Credential` = pool, `Reason` = target). Empty target = no reset (opt-out default); a `ResetAuth` error is logged, not fatal. This un-latches an agent (hermes) that latched "usage limit reached" so it resumes without a manual `auth reset`.