Skip to content

Commit 6d5e680

Browse files
authored
feat(pools): channel parity (REST + Telegram) and refresh-grant token-host scoping (#47)
* feat(poolops): extract channel-agnostic pool operations Move pool create/list/status/rotate/remove logic into internal/poolops so CLI, REST, and Telegram share one implementation. Adds structured errors (already-pooled member, live-member removal, delete conflict) returned channel-neutrally. * feat(cli): wire pool subcommand through poolops sluice pool create|list|status|rotate|remove now delegate to the shared poolops package; rotate-race hint points to 'sluice pool status'. * feat(api): rest endpoints for credential pools Add /api/pools CRUD plus status/rotate via poolops; pool-create honors the 409 conflict contract and returns structured pool-delete conflict payloads. Token-host grant parse restricted to form bodies. * feat(telegram): /pool create|list|status|rotate|remove commands Telegram bot reaches pool management through poolops, matching CLI and REST channel parity. * fix(proxy): scope pool token-host phantom expansion to refresh_token grants Restrict the pool token-host phantom split-host expansion to refresh_token grant requests so non-refresh traffic is unaffected; parse only form bodies. * feat(telegram): friendlier pool failover notification text Reword the pool failover Telegram notice into human-readable text (task 4b). * docs: document /pool and /api/pools surfaces; channel-parity plan Document the pool management surfaces across CLI/REST/Telegram in CLAUDE.md and README, and record the channel-feature-parity plan (incl. task 4b) as completed. * fix(api): dedicated pool-referenced 409 schema; build create 201 from request data split the pool-delete 409 body into PoolReferencedErrorResponse so the generic ErrorResponse envelope is no longer coupled to one endpoint (regenerated api.gen.go via oapi-codegen). PostApiPools now builds the 201 body from the request members + failover default instead of gating on a store read-back, so a read-back error can no longer report a successful create as a 500. poolStatusError no longer checks ErrCredentialInUseByPool, which is unreachable from the pool handlers (raised only by the credential-removal path). * fix(proxy): gate grant_type probe to token-host POSTs; raise+observe probe cap requestFlowGrantType now skips the string(body)+ParseQuery grant_type probe unless the request is an HTTP POST whose scheme+host matches a known OAuth token endpoint (new OAuthIndex.MatchesHost), removing the per-request body parse on the hot path for non-OAuth traffic. The form/JSON parsers in extractRequestRefreshToken and requestGrantType no longer double-stringify the body for an explicit form Content-Type. The probe cap is raised 8KiB->64KiB so a large RFC 7523 refresh payload at a pool token host is still expanded, with a rate-limited WARNING when the cap truncates a probe. StreamRequestModifier's buffered-path-only limitation is documented explicitly. * fix(proxy): drop awkward parenthetical from empty-reason failover notice FormatFailoverNotice rendered '(unknown reason)' / 'after unknown reason' when the reason tag was empty. Drop the reason clause entirely in that case for both the exhausted and normal messages. The cred_failover / pool_exhausted audit Reason format is unchanged. * fix(telegram): escape pool name in rotate-race hint; assert pool removal the /pool status hint in the rotate-race message wrote the raw pool name into an HTML-parsed reply, so a name with <,>,& would break rendering; htmlCode it like the other occurrence. The pool-remove test guarded on 'err == nil', which GetPool returns even for a missing pool; assert the pool row is actually gone instead. * docs(readme): note bearer auth required for REST API examples the /api/* curl examples omit the Authorization header though the endpoints are behind BearerAuth; add a single leading note covering the credential, pool, and rule examples. * fix(telegram): drop dead ErrCredentialInUseByPool branch in poolRemove * fix(api): map internal pool-create failures to 500 not 400 Invert poolCreateError: conflict sentinels stay 409, genuine client-input validation sentinels are 400, and everything else (tx/DB/INSERT failures inside store.CreatePoolWithMembers, which are wrapped fmt.Errorf strings with no sentinel) now defaults to 500 instead of being misclassified as a client 400. Add typed, errors.Is-able sentinels in internal/store for the client-validation cases that previously had only wrapped messages (ErrPoolNoMembers, ErrPoolStrategyInvalid, ErrPoolMemberDuplicate, ErrPoolMemberNotFound, ErrPoolMemberNotOAuth), wrapped at origin so the existing human-readable text is preserved verbatim. Document the new 500 response on POST /api/pools in the OpenAPI spec and regenerate. * fix(proxy): make unknown failover reason read naturally; drop dead empty case humanizeFailoverReason now renders an unknown tag as 'unknown reason (<tag>)' so the surrounding 'failed over ... after %s.' and exhausted '... to (%s).' clauses read naturally instead of the redundant 'failed over ... after failover (<tag>).'. Remove the unreachable case "": branch: FormatFailoverNotice is the sole caller and short-circuits an empty reason before ever calling here, so the empty-tag wording has a single source of truth there. Update the notice/humanize tests accordingly. * fix(telegram): HTML-escape pool member LastFailureReason in status poolStatus sends with HTML parse mode (htmlCode emits <code>), but appended m.LastFailureReason (upstream error text, may contain < > &) raw, which breaks rendering or is rejected by the Bot API. Escape it with htmlEscape (prose, not an identifier, so not htmlCode) consistent with every other user-facing value on that line. Add a test asserting a reason with < > & is rendered escaped. * fix(telegram): html-escape bind hint in pool create reply
1 parent 2e54339 commit 6d5e680

25 files changed

Lines changed: 3276 additions & 228 deletions

CLAUDE.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -181,6 +181,8 @@ sluice pool rotate <name> # operator override: advance active member
181181
sluice pool remove <name>
182182
```
183183

184+
Pools are reachable from all channels — CLI `sluice pool`, REST `/api/pools` (`GET`/`POST`, `GET`/`DELETE /api/pools/{name}`, `POST /api/pools/{name}/rotate`), and Telegram `/pool` — all via the channel-agnostic `internal/poolops`.
185+
184186
Auto-failover on 429/401 is primary; `pool rotate` is an override.
185187

186188
**Data model (migration `000006_credential_pools`):** `credential_pools` (name, strategy reserved `failover`), `credential_pool_members` (ordered, pool->credential FK), `credential_health` (`healthy|cooldown`, `cooldown_until`, `last_failure_reason`), all CHECK-constrained. Store API in `internal/store/pools.go`. `reloadAll` loads pool+health into an atomic-pointer-swapped `PoolResolver` (`internal/vault/pool.go`), rewired via `srv.StorePool`/`SetPoolResolver` on SIGHUP and the 2s data-version watcher.

README.md

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -329,6 +329,11 @@ Manage sluice from your phone. Approve connections and tool calls, add credentia
329329
| `/mcp list` | List registered MCP upstreams |
330330
| `/mcp add <name> --command <cmd> [flags]` | Register a new MCP upstream (stdio/http/websocket, see `/help`; chat message auto-deleted because `--env` may carry secrets) |
331331
| `/mcp remove <name>` | Remove an MCP upstream |
332+
| `/pool create <name> <a,b[,c]>` | Create a credential pool (ordered OAuth members, failover order) |
333+
| `/pool list` | List credential pools |
334+
| `/pool status <name>` | Active member and per-member health |
335+
| `/pool rotate <name>` | Operator override: advance the active member |
336+
| `/pool remove <name>` | Remove a credential pool |
332337
| `/status` | Proxy stats and pending approvals |
333338
| `/audit recent [N]` | Last N audit entries |
334339

@@ -338,6 +343,11 @@ Manage sluice from your phone. Approve connections and tool calls, add credentia
338343

339344
REST API on port 3000 for programmatic approval integration. `GET /api/approvals` lists pending requests, `POST /api/approvals/{id}/resolve` resolves them. Use this to build custom approval UIs or integrate with existing workflows.
340345

346+
All `/api/*` endpoints below are protected by bearer auth. Every request must
347+
send `Authorization: Bearer $SLUICE_API_TOKEN` (the token sluice prints at
348+
startup). The curl examples omit the header for brevity, but it is required
349+
for the credential, pool, and rule calls shown here.
350+
341351
Credential management endpoints support both static and OAuth types:
342352

343353
```bash
@@ -350,6 +360,26 @@ curl -X POST http://localhost:3000/api/credentials \
350360
-d '{"name":"openai_oauth","type":"oauth","token_url":"https://auth.example.com/token","access_token":"at-xxx","refresh_token":"rt-xxx","destination":"api.openai.com","env_var":"OPENAI_API_KEY"}'
351361
```
352362

363+
Credential pools are managed over the same REST surface as the CLI `sluice pool` and Telegram `/pool` commands:
364+
365+
```bash
366+
# List pools
367+
curl http://localhost:3000/api/pools
368+
369+
# Create a pool (members are ordered OAuth credential names; strategy defaults to "failover")
370+
curl -X POST http://localhost:3000/api/pools \
371+
-d '{"name":"openai","members":["codex_a","codex_b"]}'
372+
373+
# Pool status (active member + per-member health)
374+
curl http://localhost:3000/api/pools/openai
375+
376+
# Operator override: advance the active member
377+
curl -X POST http://localhost:3000/api/pools/openai/rotate
378+
379+
# Remove a pool
380+
curl -X DELETE http://localhost:3000/api/pools/openai
381+
```
382+
353383
## Data Loss Prevention
354384

355385
Two complementary inspection layers protect against credential leakage and dangerous tool use:

api/openapi.yaml

Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -519,6 +519,139 @@ paths:
519519
schema:
520520
$ref: "#/components/schemas/ErrorResponse"
521521

522+
/api/pools:
523+
get:
524+
operationId: getApiPools
525+
summary: List credential pools
526+
tags: [pools]
527+
responses:
528+
"200":
529+
description: Credential pools
530+
content:
531+
application/json:
532+
schema:
533+
type: array
534+
items:
535+
$ref: "#/components/schemas/Pool"
536+
post:
537+
operationId: postApiPools
538+
summary: Create a credential pool
539+
tags: [pools]
540+
requestBody:
541+
required: true
542+
content:
543+
application/json:
544+
schema:
545+
$ref: "#/components/schemas/CreatePoolRequest"
546+
responses:
547+
"201":
548+
description: Pool created
549+
content:
550+
application/json:
551+
schema:
552+
$ref: "#/components/schemas/Pool"
553+
"400":
554+
description: Invalid request
555+
content:
556+
application/json:
557+
schema:
558+
$ref: "#/components/schemas/ErrorResponse"
559+
"409":
560+
description: Pool name collides or a member is already pooled
561+
content:
562+
application/json:
563+
schema:
564+
$ref: "#/components/schemas/ErrorResponse"
565+
"500":
566+
description: Internal error creating the pool (transaction/DB failure)
567+
content:
568+
application/json:
569+
schema:
570+
$ref: "#/components/schemas/ErrorResponse"
571+
572+
/api/pools/{name}:
573+
get:
574+
operationId: getApiPoolsName
575+
summary: Pool status (active member + per-member health)
576+
tags: [pools]
577+
parameters:
578+
- name: name
579+
in: path
580+
required: true
581+
schema:
582+
type: string
583+
responses:
584+
"200":
585+
description: Pool status
586+
content:
587+
application/json:
588+
schema:
589+
$ref: "#/components/schemas/PoolStatus"
590+
"404":
591+
description: Pool not found
592+
content:
593+
application/json:
594+
schema:
595+
$ref: "#/components/schemas/ErrorResponse"
596+
delete:
597+
operationId: deleteApiPoolsName
598+
summary: Remove a credential pool
599+
tags: [pools]
600+
parameters:
601+
- name: name
602+
in: path
603+
required: true
604+
schema:
605+
type: string
606+
responses:
607+
"204":
608+
description: Pool removed
609+
"404":
610+
description: Pool not found
611+
content:
612+
application/json:
613+
schema:
614+
$ref: "#/components/schemas/ErrorResponse"
615+
"409":
616+
description: >-
617+
Pool still referenced by one or more bindings. The response body's
618+
`bindings` field lists the blocking bindings (id + destination).
619+
content:
620+
application/json:
621+
schema:
622+
$ref: "#/components/schemas/PoolReferencedErrorResponse"
623+
624+
/api/pools/{name}/rotate:
625+
post:
626+
operationId: postApiPoolsNameRotate
627+
summary: Operator override — advance the active pool member
628+
tags: [pools]
629+
parameters:
630+
- name: name
631+
in: path
632+
required: true
633+
schema:
634+
type: string
635+
responses:
636+
"200":
637+
description: Rotation result
638+
content:
639+
application/json:
640+
schema:
641+
$ref: "#/components/schemas/PoolRotateResult"
642+
"404":
643+
description: Pool not found
644+
content:
645+
application/json:
646+
schema:
647+
$ref: "#/components/schemas/ErrorResponse"
648+
"409":
649+
description: Rotate raced a concurrent membership change
650+
content:
651+
application/json:
652+
schema:
653+
$ref: "#/components/schemas/ErrorResponse"
654+
522655
/api/audit/recent:
523656
get:
524657
operationId: getApiAuditRecent
@@ -619,6 +752,36 @@ components:
619752
code:
620753
type: string
621754

755+
PoolReferencedErrorResponse:
756+
description: >-
757+
409 body for DELETE /api/pools/{name} when the pool is still
758+
referenced by one or more bindings. Carries the generic error/code
759+
plus the structured list of blocking bindings (id + destination); the
760+
CLI and Telegram surfaces render the same list (channel parity). This
761+
is a dedicated schema so the generic ErrorResponse envelope is not
762+
coupled to one endpoint.
763+
type: object
764+
required: [error, bindings]
765+
properties:
766+
error:
767+
type: string
768+
code:
769+
type: string
770+
bindings:
771+
type: array
772+
items:
773+
$ref: "#/components/schemas/PoolReferencingBinding"
774+
775+
PoolReferencingBinding:
776+
type: object
777+
required: [id, destination]
778+
properties:
779+
id:
780+
type: integer
781+
format: int64
782+
destination:
783+
type: string
784+
622785
HealthResponse:
623786
type: object
624787
required: [status]
@@ -1009,6 +1172,98 @@ components:
10091172
timeout_sec:
10101173
type: integer
10111174

1175+
Pool:
1176+
type: object
1177+
required: [name, strategy, members]
1178+
properties:
1179+
name:
1180+
type: string
1181+
strategy:
1182+
type: string
1183+
description: "Pool strategy (only 'failover' is supported)"
1184+
created_at:
1185+
type: string
1186+
format: date-time
1187+
members:
1188+
type: array
1189+
items:
1190+
$ref: "#/components/schemas/PoolMember"
1191+
1192+
PoolMember:
1193+
type: object
1194+
required: [credential, position]
1195+
properties:
1196+
credential:
1197+
type: string
1198+
position:
1199+
type: integer
1200+
1201+
CreatePoolRequest:
1202+
type: object
1203+
required: [name, members]
1204+
properties:
1205+
name:
1206+
type: string
1207+
strategy:
1208+
type: string
1209+
description: "Pool strategy; defaults to 'failover' when omitted"
1210+
members:
1211+
type: array
1212+
description: "Ordered member credential names (failover order)"
1213+
items:
1214+
type: string
1215+
1216+
PoolStatus:
1217+
type: object
1218+
required: [name, strategy, active, members]
1219+
properties:
1220+
name:
1221+
type: string
1222+
strategy:
1223+
type: string
1224+
active:
1225+
type: string
1226+
description: "Currently active member credential name"
1227+
members:
1228+
type: array
1229+
items:
1230+
$ref: "#/components/schemas/PoolMemberStatus"
1231+
1232+
PoolMemberStatus:
1233+
type: object
1234+
required: [credential, position, active, state]
1235+
properties:
1236+
credential:
1237+
type: string
1238+
position:
1239+
type: integer
1240+
active:
1241+
type: boolean
1242+
state:
1243+
type: string
1244+
description: "healthy, cooldown, or healthy (cooldown expired)"
1245+
cooldown_until:
1246+
type: string
1247+
format: date-time
1248+
last_failure_reason:
1249+
type: string
1250+
1251+
PoolRotateResult:
1252+
type: object
1253+
required: [pool, from, to]
1254+
properties:
1255+
pool:
1256+
type: string
1257+
from:
1258+
type: string
1259+
description: "Member that was active and is now parked"
1260+
to:
1261+
type: string
1262+
description: "New active member after the rotation"
1263+
parked_until:
1264+
type: string
1265+
format: date-time
1266+
10121267
AuditEntry:
10131268
type: object
10141269
required: [timestamp, verdict]

cmd/sluice/main.go

Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -516,12 +516,7 @@ func main() {
516516
// Exhausted: no distinct member to fail over to (every
517517
// member cooling) — report it as pool exhaustion, NOT a
518518
// self-referential "X -> X" transition.
519-
msg := fmt.Sprintf("pool %s failed over %s -> %s (%s)",
520-
ev.Pool, ev.From, ev.To, ev.Reason)
521-
if ev.Exhausted {
522-
msg = fmt.Sprintf("pool %s exhausted: all members cooling down (%s); no healthy account to fail over to",
523-
ev.Pool, ev.Reason)
524-
}
519+
msg := proxy.FormatFailoverNotice(ev)
525520
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
526521
defer cancel()
527522
for _, ch := range failoverBroker.Channels() {

0 commit comments

Comments
 (0)