Skip to content

Commit bc2466d

Browse files
alari76claude
andcommitted
feat(deployments): incident response — auto-diagnosed breaches, security probes, deployment-aware audits
Phase D of the Joe value-expansion plan: detect → diagnose → propose fix. - autoDiagnose per deployment (operator opt-in, requires linked repo): a probe-breach signal spawns a diagnostic child with the breach evidence and recent sample history inline; the child investigates logs/recent merges, writes .codekin/reports/incidents/<date>_<id>.md, and lands it as a PR (implementing only clear low-risk fixes, else proposing) - Hard constraints in the child task: diagnose, never operate — no restarts, no host changes, no monitoring-config edits - 6h per-probe cooldown against flapping; spawn failures notify Joe instead of rejecting the signal (no duplicate alerts via redelivery); Joe is told whenever an auto-diagnosis starts - Child manager lifted out of the router into ws-server and injected, so the breach handler and the REST API share one instance - http probe checkHeaders opt-in: missing HSTS/CSP on https = breach - security-audit workflows become deployment-aware: prompt receives a <deployment-status> block with live probe state when the repo has a linked monitored deployment - Joe template v7: incident-response guidance Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent a4dee99 commit bc2466d

10 files changed

Lines changed: 280 additions & 7 deletions

docs/DEPLOYMENTS.md

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ Deployments live in `~/.codekin/deployments.json`:
3434

3535
| Type | Samples | Breach conditions |
3636
| --- | --- | --- |
37-
| `http` | status, latency, TLS days-remaining (`checkTls`) | non-expected status (default: ≥400), unreachable/timeout, certificate < 14 days |
37+
| `http` | status, latency, TLS days-remaining (`checkTls`), security headers (`checkHeaders`) | non-expected status (default: ≥400), unreachable/timeout, certificate < 14 days, missing HSTS/CSP headers |
3838
| `pm2` | status, restart count, memory | process missing, status ≠ `online`, memory > `memoryLimitMb`; a restart-count increase publishes a one-off event |
3939
| `disk` | free % | free % < `minFreePct` (default 10) |
4040

@@ -46,6 +46,19 @@ Sampling rides the trigger engine's tick (`registerTickTask`, every 5 minutes)
4646

4747
Breach detection fires on **transitions**, not on every breached sample: `ok → breached` publishes a `probe-breach` signal (once), `breached → ok` publishes `probe-recovered`. Both flow through the durable signal queue (at-least-once, deduped while pending) and land in the orchestrator's notifications. The orchestrator can then inspect current state (`list_deployments`) and history (`get_deployment_samples`) before deciding whether to act.
4848

49+
## Incident response
50+
51+
A breach always notifies the orchestrator, which can inspect state and spawn a diagnostic child under its normal trust rules. Additionally, a deployment with **`"autoDiagnose": true`** (operator opt-in, requires `repoPath`) spawns the diagnostic child automatically the moment a breach signal is processed:
52+
53+
- The child receives the breach evidence (probe, breaches, metrics, recent sample history) inline in its task.
54+
- It investigates — logs, recent commits/merges, deploy correlation — and writes an incident report to `.codekin/reports/incidents/<date>_<deployment>.md`, landing it as a PR. A clear low-risk fix may be implemented on the same branch; anything else becomes a "Proposed remediation" section.
55+
- **Hard constraints**: the child diagnoses, it never operates — no service restarts, no host changes, no edits to monitoring config. Those remain operator-approved actions.
56+
- A 6-hour per-probe cooldown prevents a flapping probe from spawning children repeatedly; the concurrent-children cap (5) applies as everywhere.
57+
58+
The orchestrator is told when an auto-diagnosis starts (and when it can't — e.g. at the children cap), so there is never a silent parallel investigation.
59+
60+
Security audits are deployment-aware: when a repo with a linked, monitored deployment runs its `security-audit` workflow, the prompt receives current probe state (`<deployment-status>` block), so the audit covers the live surface — TLS posture, headers, resource pressure — alongside the code.
61+
4962
## API
5063

5164
- `GET /api/deployments` — registry with each probe's latest sample

server/deployment-config.ts

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,8 @@ export interface HttpProbeConfig {
2424
timeoutMs?: number
2525
/** Also check TLS certificate days-remaining (https URLs). */
2626
checkTls?: boolean
27+
/** Also check security headers (HSTS, CSP) on https responses. */
28+
checkHeaders?: boolean
2729
}
2830

2931
export interface Pm2ProbeConfig {
@@ -48,6 +50,13 @@ export interface DeploymentConfig {
4850
/** Optional link back to the source repo (connects incidents to recent merges). */
4951
repoPath?: string
5052
enabled: boolean
53+
/**
54+
* Operator opt-in: spawn a diagnostic child session automatically when a
55+
* probe breaches (requires `repoPath`). The child investigates and writes an
56+
* incident report; it never touches the running system. Default false —
57+
* without it, breaches only notify the orchestrator.
58+
*/
59+
autoDiagnose?: boolean
5160
probes: ProbeConfig[]
5261
}
5362

server/deployment-monitor.ts

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,8 @@ function certDaysRemaining(url: string, timeoutMs: number): Promise<number | nul
100100
})
101101
}
102102

103-
async function runHttpProbe(probe: HttpProbeConfig): Promise<ProbeResult> {
103+
/** Exported for tests (stubbed global fetch). */
104+
export async function runHttpProbe(probe: HttpProbeConfig): Promise<ProbeResult> {
104105
const timeoutMs = probe.timeoutMs ?? DEFAULT_HTTP_TIMEOUT_MS
105106
const breaches: string[] = []
106107
const metrics: ProbeMetrics = { status: null, latencyMs: null, certDays: null }
@@ -117,6 +118,15 @@ async function runHttpProbe(probe: HttpProbeConfig): Promise<ProbeResult> {
117118
if (!statusOk) {
118119
breaches.push(`http ${res.status}${probe.expectStatus !== undefined ? ` (expected ${probe.expectStatus})` : ''}`)
119120
}
121+
// Security-header posture of the live surface (opt-in, https only).
122+
if (probe.checkHeaders && probe.url.startsWith('https:')) {
123+
const hsts = res.headers.get('strict-transport-security') !== null
124+
const csp = res.headers.get('content-security-policy') !== null
125+
metrics.hsts = hsts ? 1 : 0
126+
metrics.csp = csp ? 1 : 0
127+
if (!hsts) breaches.push('missing Strict-Transport-Security header')
128+
if (!csp) breaches.push('missing Content-Security-Policy header')
129+
}
120130
} catch (err) {
121131
metrics.latencyMs = Date.now() - started
122132
const msg = err instanceof Error && err.name === 'AbortError' ? `timeout after ${timeoutMs}ms` : err instanceof Error ? err.message : String(err)

server/deployment-routes.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,7 @@ export function createDeploymentRouter(verifyToken: VerifyFn, extractToken: Extr
7676
name: String(body.name),
7777
repoPath: body.repoPath,
7878
enabled: body.enabled !== false,
79+
autoDiagnose: body.autoDiagnose === true,
7980
probes: body.probes ?? [],
8081
})
8182
res.json({ config })

server/incident-response.test.ts

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
/** Tests for incident-response helpers — branch naming (must satisfy the spawn route's pattern) and the diagnostic task content. */
2+
import { describe, it, expect, vi, afterEach } from 'vitest'
3+
import { buildIncidentTask, incidentBranchName, type BreachPayload } from './incident-response.js'
4+
import { runHttpProbe } from './deployment-monitor.js'
5+
import type { DeploymentSample } from './deployment-monitor.js'
6+
7+
/** The branchName validation pattern from orchestrator-session-router. */
8+
const BRANCH_PATTERN = /^[a-zA-Z0-9][a-zA-Z0-9/_.-]*$/
9+
10+
const NOW = new Date('2026-08-30T14:30:00.000Z')
11+
12+
const payload: BreachPayload = {
13+
deploymentId: 'codekin-prod',
14+
deploymentName: 'Codekin Production',
15+
repoPath: '/srv/repos/codekin',
16+
probeKey: 'codekin-prod::http:https://app.example.com/health',
17+
probeType: 'http',
18+
breaches: ['http 502'],
19+
metrics: { status: 502, latencyMs: 120 },
20+
}
21+
22+
describe('incidentBranchName', () => {
23+
it('produces a branch that passes the spawn route validation', () => {
24+
expect(incidentBranchName('codekin-prod', NOW)).toMatch(BRANCH_PATTERN)
25+
expect(incidentBranchName('My Weird/ID!!', NOW)).toMatch(BRANCH_PATTERN)
26+
expect(incidentBranchName('---', NOW)).toMatch(BRANCH_PATTERN)
27+
})
28+
29+
it('embeds the deployment and timestamp for uniqueness', () => {
30+
expect(incidentBranchName('codekin-prod', NOW)).toBe('incident/codekin-prod-202608301430')
31+
})
32+
})
33+
34+
describe('buildIncidentTask', () => {
35+
const sample: DeploymentSample = {
36+
id: 1, deploymentId: 'codekin-prod', probeKey: payload.probeKey, probeType: 'http',
37+
ok: false, breaches: ['http 502'], metrics: { status: 502 }, createdAt: NOW.toISOString(),
38+
}
39+
40+
it('carries the breach evidence, report path, and operational hard constraints', () => {
41+
const task = buildIncidentTask(payload, [sample], NOW)
42+
expect(task).toContain('Codekin Production')
43+
expect(task).toContain('http 502')
44+
expect(task).toContain('.codekin/reports/incidents/2026-08-30_codekin-prod.md')
45+
expect(task).toContain('Do not restart services')
46+
expect(task).toContain('BREACHED (http 502)')
47+
})
48+
49+
it('handles missing history gracefully', () => {
50+
const task = buildIncidentTask(payload, [], NOW)
51+
expect(task).toContain('(no history available)')
52+
})
53+
})
54+
55+
describe('runHttpProbe security headers', () => {
56+
afterEach(() => {
57+
vi.unstubAllGlobals()
58+
})
59+
60+
function stubFetch(headers: Record<string, string>) {
61+
vi.stubGlobal('fetch', vi.fn(async () => ({
62+
status: 200,
63+
headers: { get: (name: string) => headers[name.toLowerCase()] ?? null },
64+
})))
65+
}
66+
67+
it('breaches on missing HSTS/CSP when checkHeaders is set', async () => {
68+
stubFetch({})
69+
const result = await runHttpProbe({ type: 'http', url: 'https://x.test/', checkHeaders: true })
70+
expect(result.ok).toBe(false)
71+
expect(result.breaches).toEqual([
72+
'missing Strict-Transport-Security header',
73+
'missing Content-Security-Policy header',
74+
])
75+
})
76+
77+
it('passes when both headers are present, recording them as metrics', async () => {
78+
stubFetch({ 'strict-transport-security': 'max-age=63072000', 'content-security-policy': "default-src 'self'" })
79+
const result = await runHttpProbe({ type: 'http', url: 'https://x.test/', checkHeaders: true })
80+
expect(result.ok).toBe(true)
81+
expect(result.metrics).toMatchObject({ hsts: 1, csp: 1 })
82+
})
83+
84+
it('does not check headers without the opt-in', async () => {
85+
stubFetch({})
86+
const result = await runHttpProbe({ type: 'http', url: 'https://x.test/' })
87+
expect(result.ok).toBe(true)
88+
})
89+
})

server/incident-response.ts

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
/**
2+
* Incident response — turns a probe breach into a diagnostic child task.
3+
*
4+
* The autonomy model: auto-diagnosis is a per-deployment operator opt-in
5+
* (`autoDiagnose: true` + a linked repo), never a default. The spawned child
6+
* investigates and reports; it is explicitly forbidden from touching the
7+
* running system — restarts and host changes stay propose-only, per the
8+
* sudo-free / hard-floor policy.
9+
*/
10+
11+
import type { DeploymentSample } from './deployment-monitor.js'
12+
13+
/** Payload of a `probe-breach` signal (shape published by DeploymentMonitor). */
14+
export interface BreachPayload {
15+
deploymentId: string
16+
deploymentName: string
17+
repoPath: string | null
18+
probeKey: string
19+
probeType: string
20+
breaches?: string[]
21+
metrics?: Record<string, unknown>
22+
}
23+
24+
/** Minimum time between auto-spawned diagnostic children for one probe. */
25+
export const DIAGNOSE_COOLDOWN_MS = 6 * 60 * 60 * 1000
26+
27+
/** Branch-safe slug from a deployment id (spawn route validates the pattern). */
28+
export function incidentBranchName(deploymentId: string, now: Date): string {
29+
const slug = deploymentId.toLowerCase().replace(/[^a-z0-9_.-]+/g, '-').replace(/^[^a-z0-9]+/, '') || 'deployment'
30+
const stamp = now.toISOString().slice(0, 16).replace(/[-:T]/g, '')
31+
return `incident/${slug}-${stamp}`
32+
}
33+
34+
function formatSample(s: DeploymentSample): string {
35+
const state = s.ok ? 'ok' : `BREACHED (${s.breaches.join('; ')})`
36+
return `- ${s.createdAt}${state}${JSON.stringify(s.metrics)}`
37+
}
38+
39+
/** The diagnostic child's task text, carrying the breach evidence inline. */
40+
export function buildIncidentTask(payload: BreachPayload, recentSamples: DeploymentSample[], now: Date): string {
41+
const date = now.toISOString().slice(0, 10)
42+
return [
43+
`Diagnose a production probe breach for the deployment "${payload.deploymentName}".`,
44+
'',
45+
`Probe: ${payload.probeKey} (type: ${payload.probeType})`,
46+
`Breaches: ${(payload.breaches ?? []).join('; ') || 'unknown'}`,
47+
`Metrics at breach: ${JSON.stringify(payload.metrics ?? {})}`,
48+
'',
49+
'Recent probe samples (newest first):',
50+
...(recentSamples.length ? recentSamples.map(formatSample) : ['- (no history available)']),
51+
'',
52+
'Do the following:',
53+
'1. Investigate the likely cause. Useful angles: application/process logs readable from this repo\'s deployment, recent commits and merged PRs in this repository, and whether the breach correlates with a deploy.',
54+
`2. Write an incident report to .codekin/reports/incidents/${date}_${payload.deploymentId}.md — symptoms, evidence, root cause (or best-supported hypothesis), impact, and remediation. The file must contain only the finished report.`,
55+
'3. If the root cause is a clear, low-risk fix in this repository\'s code or config, implement it on this branch. Otherwise end the report with a "Proposed remediation" section describing exactly what should be done and by whom.',
56+
'',
57+
'Hard constraints: you are diagnosing, not operating. Do not restart services, kill processes, or modify anything on the host outside this repository — those actions are operator-approved only. Do not modify the deployment registry or monitoring configuration.',
58+
].join('\n')
59+
}

server/orchestrator-manager.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ Agent ${AGENT_DISPLAY_NAME} tracks repositories you work with in Codekin.
5050
* forever. CLAUDE.md is system-managed; user memory lives in PROFILE.md,
5151
* REPOS.md and journal/, which are never overwritten.
5252
*/
53-
export const CLAUDE_MD_TEMPLATE_VERSION = 6
53+
export const CLAUDE_MD_TEMPLATE_VERSION = 7
5454

5555
const CLAUDE_MD_TEMPLATE = `<!-- codekin-template-version: ${CLAUDE_MD_TEMPLATE_VERSION} -->
5656
# Agent ${AGENT_DISPLAY_NAME} — Codekin Orchestrator
@@ -95,7 +95,7 @@ You have first-class \`codekin\` MCP tools — **always prefer them over curl**:
9595
- \`spawn_child\` / \`list_children\` / \`get_child\` / \`get_child_transcript\` — create and monitor coding sessions
9696
- \`pending_prompts\` / \`respond_to_prompt\` — see and unblock sessions waiting on an approval or question
9797
- \`get_repo_activity\` — activity tier per managed repo (active / cooling / dormant) and the signals behind it; dormant repos have their scheduled workflows held automatically, cooling repos run at most weekly
98-
- \`list_deployments\` / \`get_deployment_samples\` — monitored deployed apps and their probe state (http health/latency/TLS, pm2 status/restarts/memory, disk). Probe breaches and recoveries reach you as notifications; when one arrives, check current state and recent samples before reacting — and remember host actions requiring elevated privileges are propose-only, never run yourself
98+
- \`list_deployments\` / \`get_deployment_samples\` — monitored deployed apps and their probe state (http health/latency/TLS, pm2 status/restarts/memory, disk). Probe breaches and recoveries reach you as notifications; when one arrives, check current state and recent samples before reacting — and remember host actions requiring elevated privileges are propose-only, never run yourself. For a real breach on a deployment with a linked repo, spawn a diagnostic child into that repo (unless a notification says one was auto-spawned): its task is to investigate logs and recent merges and write an incident report to \`.codekin/reports/incidents/\`. The child diagnoses — it never restarts or operates the system
9999
- \`list_runs\` — every background run (workflows + loops) in one feed; watch for \`blocked\` and \`awaiting_human\`
100100
- \`start_loop\` / \`abort_run\` — launch a goal run (e.g. \`ci-autorepair\`) that iterates until its verify commands pass
101101
- \`trigger_workflow\` — run a workflow (e.g. \`repo-health.weekly\`) now instead of waiting for its schedule

server/workflow-loader.test.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ const mockRealpathSync = vi.hoisted(() => vi.fn((p: string) => p))
1515
vi.mock('child_process', () => ({
1616
execSync: (...args: any[]) => mockExecSync(...args),
1717
execFileSync: (...args: any[]) => mockExecFileSync(...args),
18+
// Imported (via deployment-monitor) at module load; not exercised here.
19+
execFile: vi.fn(),
1820
}))
1921

2022
vi.mock('fs', async (importOriginal) => {

server/workflow-loader.ts

Lines changed: 40 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,8 @@ import type { WorkflowEngine, WorkflowRun } from './workflow-engine.js'
4040
import { SessionGoneError } from './workflow-engine.js'
4141
import type { SessionManager } from './session-manager.js'
4242
import type { WsServerMessage } from './types.js'
43+
import { loadDeployments } from './deployment-config.js'
44+
import { tryGetDeploymentMonitor } from './deployment-monitor.js'
4345

4446
// ---------------------------------------------------------------------------
4547
// Types
@@ -254,6 +256,35 @@ export function isWorkflowReportsBranch(branch: string): boolean {
254256
// Workflow registration
255257
// ---------------------------------------------------------------------------
256258

259+
/**
260+
* XML-delimited live-deployment context for security audits. Empty string when
261+
* the repo has no linked, monitored deployment (or the monitor isn't up).
262+
*/
263+
function buildDeploymentContext(repoPath: string): string {
264+
try {
265+
const deployments = loadDeployments().deployments.filter(d => d.enabled && d.repoPath === repoPath)
266+
if (deployments.length === 0) return ''
267+
const monitor = tryGetDeploymentMonitor()
268+
const samples = monitor?.latestSamples() ?? []
269+
270+
const lines: string[] = ['<deployment-status>']
271+
for (const d of deployments) {
272+
lines.push(`Deployment "${d.name}" (${d.id}):`)
273+
const own = samples.filter(s => s.deploymentId === d.id)
274+
if (own.length === 0) lines.push('- no probe samples yet')
275+
for (const s of own) {
276+
const state = s.ok ? 'ok' : `BREACHED: ${s.breaches.join('; ')}`
277+
lines.push(`- ${s.probeKey}${state}${JSON.stringify(s.metrics)} (${s.createdAt})`)
278+
}
279+
}
280+
lines.push('</deployment-status>', '', 'This repository has live monitored deployment(s); the audit should cover the deployed surface (TLS posture, security headers, exposed endpoints, resource pressure) alongside the code. Current probe state above.', '', '')
281+
return lines.join('\n')
282+
} catch (err) {
283+
console.error('[workflow] Failed to build deployment context:', err)
284+
return ''
285+
}
286+
}
287+
257288
function registerWorkflow(engine: WorkflowEngine, sessions: SessionManager, def: WorkflowDef) {
258289
engine.registerWorkflow({
259290
kind: def.kind,
@@ -370,9 +401,17 @@ function registerWorkflow(engine: WorkflowEngine, sessions: SessionManager, def:
370401
].join('\n')
371402
}
372403

404+
// Security audits become deployment-aware when the repo has a
405+
// linked, monitored deployment: the prompt receives current probe
406+
// state so the audit covers the live surface, not just the code.
407+
let deploymentContext = ''
408+
if (def.kind.startsWith('security-audit')) {
409+
deploymentContext = buildDeploymentContext(repoPath)
410+
}
411+
373412
// Prepend the guard to suppress Claude's CLAUDE.md-driven file-write
374413
// behavior, which otherwise creates a duplicate report file.
375-
const prompt = `${WORKFLOW_PROMPT_GUARD}\n\n${commitContext}${userPrompt}`
414+
const prompt = `${WORKFLOW_PROMPT_GUARD}\n\n${commitContext}${deploymentContext}${userPrompt}`
376415

377416
if (repoOverride) {
378417
console.log(`[workflow:${def.kind}] Using per-repo prompt override from ${repoPath}`)

0 commit comments

Comments
 (0)