feat(phase-12): implement TA-P12-004 federation SLO automation

Author: benjinus

.env.example

@@ -39,6 +39,10 @@ TELAGENT_FEDERATION_SELF_DOMAIN=node-a.tel
 TELAGENT_FEDERATION_ENVELOPE_RATE_LIMIT_PER_MIN=600
 TELAGENT_FEDERATION_SYNC_RATE_LIMIT_PER_MIN=300
 TELAGENT_FEDERATION_RECEIPT_RATE_LIMIT_PER_MIN=600
+# Phase 12: federation DLQ auto replay scheduler
+TELAGENT_FEDERATION_DLQ_REPLAY_INTERVAL_SEC=60
+TELAGENT_FEDERATION_DLQ_REPLAY_BATCH_SIZE=100
+TELAGENT_FEDERATION_DLQ_REPLAY_STOP_ON_ERROR=false
 # Optional: federation source key pinning mode (disabled|enforced|report-only)
 # TELAGENT_FEDERATION_PINNING_MODE=disabled
 # Optional: current key fingerprints by domain (domain=key1|key2,domain2=key3)
@@ -61,3 +65,7 @@ TELAGENT_MONITOR_REQ_P95_WARN_MS=250
 TELAGENT_MONITOR_REQ_P95_CRITICAL_MS=500
 TELAGENT_MONITOR_MAINT_STALE_WARN_SEC=180
 TELAGENT_MONITOR_MAINT_STALE_CRITICAL_SEC=300
+# Phase 12: federation DLQ burn-rate alert thresholds
+TELAGENT_MONITOR_FED_DLQ_ERROR_BUDGET_RATIO=0.01
+TELAGENT_MONITOR_FED_DLQ_BURN_RATE_WARN=2
+TELAGENT_MONITOR_FED_DLQ_BURN_RATE_CRITICAL=5

docs/implementation/phase-12/README.md

@@ -1,7 +1,7 @@
 # TelAgent v1 Phase 12 执行产出（v1.2 候选能力冻结）
 
 - 文档版本：v1.0
-- 状态：Phase 12 执行中（`TA-P12-001` ~ `TA-P12-003` 已完成，`TA-P12-004` ~ `TA-P12-008` 待执行）
+- 状态：Phase 12 执行中（`TA-P12-001` ~ `TA-P12-004` 已完成，`TA-P12-005` ~ `TA-P12-008` 待执行）
 - 最后更新：2026-03-03
 
 ## 1. 产出目录
@@ -11,31 +11,35 @@
 | TA-P12-001 | `ta-p12-001-phase12-candidate-pool-freeze-2026-03-03.md` | Phase 12 候选池冻结 |
 | TA-P12-002 | `ta-p12-002-audit-snapshot-export-2026-03-03.md` | 链上/链下审计快照导出（脱敏） |
 | TA-P12-003 | `ta-p12-003-revoked-did-realtime-session-isolation-2026-03-03.md` | revoked DID 实时会话隔离（订阅+驱逐） |
+| TA-P12-004 | `ta-p12-004-federation-slo-automation-2026-03-03.md` | 联邦 SLO 自动化（DLQ 自动重放 + burn-rate 告警） |
 
 ## 2. 当前证据目录
 
 - 启动文档：
   - `ta-p12-001-phase12-candidate-pool-freeze-2026-03-03.md`
   - `ta-p12-002-audit-snapshot-export-2026-03-03.md`
   - `ta-p12-003-revoked-did-realtime-session-isolation-2026-03-03.md`
+  - `ta-p12-004-federation-slo-automation-2026-03-03.md`
 - 机读清单：
   - `manifests/2026-03-03-p12-candidate-pool-freeze.json`
   - `manifests/2026-03-03-p12-audit-snapshot-check.json`
   - `manifests/2026-03-03-p12-revoked-did-isolation-check.json`
+  - `manifests/2026-03-03-p12-federation-slo-automation-check.json`
 - 日志：
   - `logs/2026-03-03-p12-node-build.txt`
   - `logs/2026-03-03-p12-node-test.txt`
   - `logs/2026-03-03-p12-audit-snapshot-check-run.txt`
   - `logs/2026-03-03-p12-revoked-did-isolation-check-run.txt`
+  - `logs/2026-03-03-p12-federation-slo-automation-check-run.txt`
 
 ## 3. 当前进展
 
 - `TA-P12-001`：DONE
 - `TA-P12-002`：DONE
 - `TA-P12-003`：DONE
-- `TA-P12-004`：TODO
+- `TA-P12-004`：DONE
 - `TA-P12-005`：TODO
 - `TA-P12-006`：TODO
 - `TA-P12-007`：TODO
 - `TA-P12-008`：TODO
-- 下一步：进入 `TA-P12-004`（联邦 SLO 自动化：DLQ 自动重放 + burn-rate 告警）。
+- 下一步：进入 `TA-P12-005`（Agent SDK Python Beta）。

docs/implementation/phase-12/logs/2026-03-03-p12-federation-slo-automation-check-run.txt

@@ -0,0 +1,5 @@
+[TA-P12-004] autoReplayPass=true pendingBefore=9 pendingAfter=8
+[TA-P12-004] burnRateWarnPass=true burnRateCriticalPass=true
+[TA-P12-004] schedulerTickPass=true schedulerRunsDelta=1
+[TA-P12-004] decision=PASS
+[TA-P12-004] output=/Users/xiasenhai/Workspace/OpenClaw/telagent/docs/implementation/phase-12/manifests/2026-03-03-p12-federation-slo-automation-check.json

docs/implementation/phase-12/logs/2026-03-03-p12-node-test.txt

@@ -10,81 +10,88 @@
 > @telagent/node@0.1.0 test /Users/xiasenhai/Workspace/OpenClaw/telagent/packages/node
 > node --test dist/*.test.js dist/**/*.test.js
 
-✔ created response returns data envelope and Location header (201.988721ms)
-✔ list response returns paginated envelope shape (27.751717ms)
-✔ validation errors use RFC7807 shape and problem+json content type (14.101667ms)
-✔ node audit snapshot exports de-sensitized envelope and links.self (101.777339ms)
-✔ node audit snapshot rejects invalid query with RFC7807 response (7.444877ms)
-✔ TA-P12-003 revoked DID event isolates session and rejects message send with RFC7807 (24.044224ms)
-✔ not found uses RFC7807 shape (3.760304ms)
-✔ identities and groups endpoints are accessible with expected status codes (27.138716ms)
-✔ messages, attachments and federation endpoints are accessible (94.28094ms)
-✔ routes only serve /api/v1/* prefix (226.560098ms)
-✔ identity endpoint responds with data envelope (38.693577ms)
-✔ mailbox store defaults to sqlite backend (4.980825ms)
-✔ mailbox store parses postgres backend config (0.693923ms)
-✔ postgres backend requires connection url (0.629182ms)
-✔ mailbox backend rejects unsupported value (0.344071ms)
-✔ federation protocol defaults to v1 and supports self version (0.983959ms)
-✔ federation supported protocols auto-include self version (0.423962ms)
-✔ domain proof config defaults to enforced mode (0.34525ms)
-✔ domain proof config accepts report-only mode and custom values (4.038343ms)
-✔ domain proof mode rejects unsupported value (1.352386ms)
-✔ domain proof numeric settings require positive integers (1.878416ms)
-✔ federation pinning defaults to disabled mode (1.161413ms)
-✔ federation pinning parses current/next keys and cutover timestamp (0.519115ms)
-✔ federation pinning rejects invalid mode (0.325709ms)
-✔ federation pinning enabled requires key mappings (0.400825ms)
-✔ federation pinning map requires domain=keys format (1.494653ms)
-✔ finalityDepth only materializes finalized blocks (41.850422ms)
-✔ reorg rollback replays canonical events and restores deterministic view (16.65258ms)
-✔ TA-P4-009 E2E main path: create -> invite -> accept -> group chat (text/image/file) (273.019079ms)
-✔ TA-P4-010 E2E offline 24h pull keeps dedupe and per-conversation order (47.117487ms)
-✔ TA-P4-006 init-upload sanitizes filename and emits attachment objectKey (1.425367ms)
-✔ TA-P4-006 complete-upload enforces manifest and checksum integrity (0.706507ms)
-✔ TA-P4-006 complete-upload is idempotent and rejects checksum divergence (0.392383ms)
-✔ TA-P4-006 expired upload sessions are cleaned and cannot be completed (0.243144ms)
-✔ TA-P11-003 accepts valid domain proof challenge and canonical hash (27.116966ms)
-✔ TA-P11-003 rejects illegal domain challenge on malformed domain (0.722989ms)
-✔ TA-P11-003 rejects when canonical domainProofHash mismatches payload (0.715704ms)
-✔ TA-P11-003 rotates challenge nonce near expiry and accepts renewed domain proof (2.871016ms)
-✔ TA-P11-003 report-only mode returns warning without blocking create flow (0.911579ms)
-✔ TA-P4-007 federation envelopes support idempotent retries (2.030286ms)
-✔ TA-P4-007 federation auth token is enforced when configured (2.950585ms)
-✔ TA-P4-007 federation rate limit rejects burst traffic (7.381569ms)
-✔ TA-P4-008 group-state sync enforces domain consistency (0.414258ms)
-✔ TA-P8-002 group-state sync rejects stale stateVersion and records resilience counters (1.162181ms)
-✔ TA-P8-002 group-state sync detects split-brain on same stateVersion with different state (0.368307ms)
-✔ TA-P9-002 federation accepts compatible protocol versions and tracks usage stats (17.272465ms)
-✔ TA-P9-002 federation rejects unsupported protocol versions (0.265048ms)
-✔ TA-P11-004 federation pinning enforces sourceKeyId with current/next rotation (0.641915ms)
-✔ TA-P11-004 federation pinning report-only mode allows traffic but records warnings (0.3097ms)
-✔ TA-P11-005 federation DLQ captures failures and replays in sequence order (0.816906ms)
-✔ TA-P4-008 node-info publishes domain and federation security policy (0.196557ms)
-✔ assertSufficient throws INSUFFICIENT_GAS_TOKEN_BALANCE when native balance is not enough (2.46961ms)
-✔ TA-P11-006 rotate key keeps old key usable in grace window then expires (2.56483ms)
-✔ TA-P11-006 revoke and recover lifecycle is verifiable (0.479821ms)
-✔ TA-P11-006 rejects invalid did and malformed key id (0.467458ms)
-✔ TA-P4-002 sequence allocator keeps per-conversation monotonic order (3.892848ms)
-✔ TA-P4-003 dedupe keeps idempotent writes for same envelopeId (0.758626ms)
-✔ TA-P4-003 duplicate envelopeId with different payload is rejected (0.895827ms)
-✔ TA-P4-004 cleanupExpired removes expired envelopes and releases dedupe key (0.608604ms)
-✔ TA-P4-005 provisional envelopes are retracted when group is reorged back (0.858395ms)
-✔ TA-P4-005 send is rejected when group chain state is REORGED_BACK (0.921126ms)
-✔ TA-P12-002 buildAuditSnapshot exports hashed retraction samples (1.107947ms)
-✔ TA-P12-002 buildAuditSnapshot normalizes sample and scan bounds (0.199506ms)
-✔ TA-P12-003 revoked DID event isolates related sessions and evicts active sessions (2.119033ms)
-✔ TA-P12-003 buildAuditSnapshot includes revocation isolation evidence (0.583113ms)
-✔ TA-P6-001 mailbox persists messages and seq after service restart (23.514943ms)
-✔ TA-P11-006 message send validates signal/mls key lifecycle status (1.754109ms)
-✔ TA-P11-007 revoked DID cannot continue sending new messages (0.465936ms)
-✔ TA-P5-002 monitoring snapshot normalizes dynamic route segments and records counters (22.521397ms)
-✔ TA-P5-002 monitoring emits warning/critical alerts when thresholds are exceeded (0.508388ms)
-ℹ tests 70
+✔ created response returns data envelope and Location header (142.978537ms)
+✔ list response returns paginated envelope shape (9.281913ms)
+✔ validation errors use RFC7807 shape and problem+json content type (101.586937ms)
+✔ node audit snapshot exports de-sensitized envelope and links.self (21.198809ms)
+✔ node audit snapshot rejects invalid query with RFC7807 response (3.576696ms)
+✔ TA-P12-003 revoked DID event isolates session and rejects message send with RFC7807 (214.699747ms)
+✔ TA-P12-004 node metrics exposes federation DLQ replay burn-rate section (4.371435ms)
+✔ not found uses RFC7807 shape (5.272078ms)
+✔ identities and groups endpoints are accessible with expected status codes (32.918991ms)
+✔ messages, attachments and federation endpoints are accessible (69.476523ms)
+✔ routes only serve /api/v1/* prefix (343.033357ms)
+✔ identity endpoint responds with data envelope (10.371252ms)
+✔ mailbox store defaults to sqlite backend (6.798975ms)
+✔ mailbox store parses postgres backend config (0.552916ms)
+✔ postgres backend requires connection url (0.652736ms)
+✔ mailbox backend rejects unsupported value (0.352623ms)
+✔ federation protocol defaults to v1 and supports self version (1.430739ms)
+✔ federation supported protocols auto-include self version (0.510169ms)
+✔ domain proof config defaults to enforced mode (5.823333ms)
+✔ domain proof config accepts report-only mode and custom values (0.628877ms)
+✔ domain proof mode rejects unsupported value (0.638306ms)
+✔ domain proof numeric settings require positive integers (0.535536ms)
+✔ federation pinning defaults to disabled mode (0.461271ms)
+✔ federation pinning parses current/next keys and cutover timestamp (0.625499ms)
+✔ federation pinning rejects invalid mode (0.322217ms)
+✔ federation pinning enabled requires key mappings (0.301392ms)
+✔ federation pinning map requires domain=keys format (0.297021ms)
+✔ federation SLO automation config defaults are applied (0.414898ms)
+✔ federation SLO automation config accepts custom values (0.614628ms)
+✔ federation SLO burn-rate thresholds require positive values (0.367184ms)
+✔ finalityDepth only materializes finalized blocks (50.109869ms)
+✔ reorg rollback replays canonical events and restores deterministic view (20.18709ms)
+✔ TA-P4-009 E2E main path: create -> invite -> accept -> group chat (text/image/file) (410.714016ms)
+✔ TA-P4-010 E2E offline 24h pull keeps dedupe and per-conversation order (41.684085ms)
+✔ TA-P4-006 init-upload sanitizes filename and emits attachment objectKey (1.715338ms)
+✔ TA-P4-006 complete-upload enforces manifest and checksum integrity (29.778586ms)
+✔ TA-P4-006 complete-upload is idempotent and rejects checksum divergence (1.512655ms)
+✔ TA-P4-006 expired upload sessions are cleaned and cannot be completed (0.426007ms)
+✔ TA-P11-003 accepts valid domain proof challenge and canonical hash (107.917819ms)
+✔ TA-P11-003 rejects illegal domain challenge on malformed domain (0.87869ms)
+✔ TA-P11-003 rejects when canonical domainProofHash mismatches payload (0.95517ms)
+✔ TA-P11-003 rotates challenge nonce near expiry and accepts renewed domain proof (2.894795ms)
+✔ TA-P11-003 report-only mode returns warning without blocking create flow (0.59398ms)
+✔ TA-P4-007 federation envelopes support idempotent retries (11.038269ms)
+✔ TA-P4-007 federation auth token is enforced when configured (0.591878ms)
+✔ TA-P4-007 federation rate limit rejects burst traffic (0.325528ms)
+✔ TA-P4-008 group-state sync enforces domain consistency (0.430939ms)
+✔ TA-P8-002 group-state sync rejects stale stateVersion and records resilience counters (0.797773ms)
+✔ TA-P8-002 group-state sync detects split-brain on same stateVersion with different state (0.766815ms)
+✔ TA-P9-002 federation accepts compatible protocol versions and tracks usage stats (53.843521ms)
+✔ TA-P9-002 federation rejects unsupported protocol versions (0.310898ms)
+✔ TA-P11-004 federation pinning enforces sourceKeyId with current/next rotation (0.671058ms)
+✔ TA-P11-004 federation pinning report-only mode allows traffic but records warnings (0.372314ms)
+✔ TA-P11-005 federation DLQ captures failures and replays in sequence order (11.166528ms)
+✔ TA-P4-008 node-info publishes domain and federation security policy (0.299215ms)
+✔ TA-P12-004 federation SLO runOnce auto-replays DLQ and records burn-rate metrics (7.966754ms)
+✔ TA-P12-004 federation SLO scheduler periodically replays DLQ (1102.711982ms)
+✔ assertSufficient throws INSUFFICIENT_GAS_TOKEN_BALANCE when native balance is not enough (2.572739ms)
+✔ TA-P11-006 rotate key keeps old key usable in grace window then expires (2.551018ms)
+✔ TA-P11-006 revoke and recover lifecycle is verifiable (0.534262ms)
+✔ TA-P11-006 rejects invalid did and malformed key id (0.334605ms)
+✔ TA-P4-002 sequence allocator keeps per-conversation monotonic order (3.675683ms)
+✔ TA-P4-003 dedupe keeps idempotent writes for same envelopeId (0.653546ms)
+✔ TA-P4-003 duplicate envelopeId with different payload is rejected (0.995472ms)
+✔ TA-P4-004 cleanupExpired removes expired envelopes and releases dedupe key (0.617718ms)
+✔ TA-P4-005 provisional envelopes are retracted when group is reorged back (0.791942ms)
+✔ TA-P4-005 send is rejected when group chain state is REORGED_BACK (0.540054ms)
+✔ TA-P12-002 buildAuditSnapshot exports hashed retraction samples (0.95024ms)
+✔ TA-P12-002 buildAuditSnapshot normalizes sample and scan bounds (0.154561ms)
+✔ TA-P12-003 revoked DID event isolates related sessions and evicts active sessions (1.906471ms)
+✔ TA-P12-003 buildAuditSnapshot includes revocation isolation evidence (0.670945ms)
+✔ TA-P6-001 mailbox persists messages and seq after service restart (22.976621ms)
+✔ TA-P11-006 message send validates signal/mls key lifecycle status (1.579605ms)
+✔ TA-P11-007 revoked DID cannot continue sending new messages (0.467104ms)
+✔ TA-P5-002 monitoring snapshot normalizes dynamic route segments and records counters (3.120293ms)
+✔ TA-P5-002 monitoring emits warning/critical alerts when thresholds are exceeded (0.505947ms)
+✔ TA-P12-004 federation DLQ burn-rate alert is emitted and tracked (1.16978ms)
+ℹ tests 77
 ℹ suites 0
-ℹ pass 70
+ℹ pass 77
 ℹ fail 0
 ℹ cancelled 0
 ℹ skipped 0
 ℹ todo 0
-ℹ duration_ms 1451.377978
+ℹ duration_ms 2458.799264