feat(deployments): registry, deterministic probes, breach signals to Joe (#606)

Phase C2 of the Joe value-expansion plan: Joe watches production.

- ~/.codekin/deployments.json registry (0600): deployments with http /
  pm2 / disk probes, optional repoPath link back to the source repo
- DeploymentMonitor: probes are plain code (no LLM in the hot path),
  sampled every 5 min via the engine's new registerTickTask — periodic
  work rides the dispatch tick instead of adding interval loops
- deployment_samples table in runs.db (30-day retention, pruned at boot)
- Breach detection on transitions: ok→breached publishes one durable
  probe-breach signal (not one per sample), breached→ok publishes
  probe-recovered; pm2 restart-count increases publish one-off
  probe-event signals; all deduped while pending and delivered to Joe
  as notifications through the at-least-once queue
- Probe failures (pm2 absent, df unparseable) are breaches themselves —
  a broken probe is visible, never silent
- Sudo-free by policy: fetch, pm2 jlist, df as the unprivileged user
- Discovery proposes pm2 processes, never auto-enrolls
- REST: /api/deployments CRUD + /discover + /samples
- MCP: list_deployments + get_deployment_samples (template v6)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: alari76

docs/DEPLOYMENTS.md

@@ -0,0 +1,54 @@
+# Deployment Monitoring
+
+Codekin can watch the apps you deploy — not just the repos that build them. The design is two-layered: **deterministic probes** sample cheaply on the trigger engine's tick with no LLM anywhere in the hot path, and **the agent enters only on a breach**, delivered as a durable signal into the orchestrator's notification stream.
+
+Everything runs sudo-free as the Codekin server user: HTTP requests, `pm2 jlist`, and `df`. Anything that would need elevated privileges is out of scope for probes by policy.
+
+## Registry
+
+Deployments live in `~/.codekin/deployments.json`:
+
+```json
+{
+  "deployments": [
+    {
+      "id": "codekin-prod",
+      "name": "Codekin Production",
+      "repoPath": "/srv/repos/codekin",
+      "enabled": true,
+      "probes": [
+        { "type": "http", "url": "https://app.example.com/api/health", "checkTls": true },
+        { "type": "pm2", "processName": "codekin", "memoryLimitMb": 1024 },
+        { "type": "disk", "path": "/", "minFreePct": 10 }
+      ]
+    }
+  ]
+}
+```
+
+`repoPath` links a deployment back to its source repo, connecting incidents to recent merges.
+
+`GET /api/deployments/discover` proposes monitorable pm2 processes (marking ones already configured). Discovery never auto-enrolls — the operator (or the agent, under trust) confirms via `POST /api/deployments`.
+
+## Probes
+
+| Type | Samples | Breach conditions |
+| --- | --- | --- |
+| `http` | status, latency, TLS days-remaining (`checkTls`) | non-expected status (default: ≥400), unreachable/timeout, certificate < 14 days |
+| `pm2` | status, restart count, memory | process missing, status ≠ `online`, memory > `memoryLimitMb`; a restart-count increase publishes a one-off event |
+| `disk` | free % | free % < `minFreePct` (default 10) |
+
+Probe *failures* (pm2 absent, `df` unparseable) are breaches too — a broken probe is visible, never silent.
+
+## Sampling & signals
+
+Sampling rides the trigger engine's tick (`registerTickTask`, every 5 minutes) — no dedicated interval loop. Samples persist to the `deployment_samples` table in `runs.db` (30-day retention, pruned at boot).
+
+Breach detection fires on **transitions**, not on every breached sample: `ok → breached` publishes a `probe-breach` signal (once), `breached → ok` publishes `probe-recovered`. Both flow through the durable signal queue (at-least-once, deduped while pending) and land in the orchestrator's notifications. The orchestrator can then inspect current state (`list_deployments`) and history (`get_deployment_samples`) before deciding whether to act.
+
+## API
+
+- `GET /api/deployments` — registry with each probe's latest sample
+- `POST /api/deployments` / `PATCH /api/deployments/:id` / `DELETE /api/deployments/:id`
+- `GET /api/deployments/discover` — pm2 process proposals
+- `GET /api/deployments/samples?probeKey=&limit=` — sample history, newest first

server/codekin-mcp-api.ts

@@ -119,6 +119,18 @@ export class CodekinApi {
     return this.request('GET', '/api/workflows/repo-activity')
   }
 
+  // --- deployments ----------------------------------------------------------
+
+  listDeployments(): Promise<unknown> {
+    return this.request('GET', '/api/deployments')
+  }
+
+  getDeploymentSamples(probeKey: string, limit?: number): Promise<unknown> {
+    const params = new URLSearchParams({ probeKey })
+    if (limit) params.set('limit', String(limit))
+    return this.request('GET', `/api/deployments/samples?${params.toString()}`)
+  }
+
   // --- trust ----------------------------------------------------------------
 
   getTrustLevel(opts: { action: string; category: string; severity?: string; repo?: string }): Promise<unknown> {

server/codekin-mcp-server.ts

@@ -98,6 +98,25 @@ export function buildCodekinMcpServer(api: CodekinApi): McpServer {
     () => run(() => api.getRepoActivity()),
   )
 
+  server.registerTool(
+    'list_deployments',
+    {
+      description:
+        'Monitored deployments with each probe\'s latest sample (http status/latency/TLS, pm2 status/restarts/memory, disk free). Breaches arrive as notifications; use this for current state.',
+      inputSchema: {},
+    },
+    () => run(() => api.listDeployments()),
+  )
+
+  server.registerTool(
+    'get_deployment_samples',
+    {
+      description: 'Sample history for one probe (newest first) — use to see when a breach started or whether a metric is trending.',
+      inputSchema: { probeKey: z.string().describe('Probe key from list_deployments'), limit: z.number().int().positive().optional() },
+    },
+    ({ probeKey, limit }) => run(() => api.getDeploymentSamples(probeKey, limit)),
+  )
+
   server.registerTool(
     'list_runs',
     {

server/deployment-config.ts

@@ -0,0 +1,117 @@
+/**
+ * Deployment registry persistence.
+ *
+ * The list of deployed apps Codekin monitors, stored as JSON at
+ * ~/.codekin/deployments.json. Each deployment carries a set of deterministic
+ * probes (http / pm2 / disk) sampled by the DeploymentMonitor. The registry is
+ * meant to be bootstrapped by discovery (pm2 process list) and confirmed by
+ * the operator — never silently auto-enrolled.
+ */
+
+import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'fs'
+import { homedir } from 'os'
+import { join } from 'path'
+
+// ---------------------------------------------------------------------------
+// Types
+// ---------------------------------------------------------------------------
+
+export interface HttpProbeConfig {
+  type: 'http'
+  url: string
+  /** Expected status code. Default: any 2xx/3xx. */
+  expectStatus?: number
+  timeoutMs?: number
+  /** Also check TLS certificate days-remaining (https URLs). */
+  checkTls?: boolean
+}
+
+export interface Pm2ProbeConfig {
+  type: 'pm2'
+  processName: string
+  /** Breach when resident memory exceeds this (MB). Unset = no memory check. */
+  memoryLimitMb?: number
+}
+
+export interface DiskProbeConfig {
+  type: 'disk'
+  path: string
+  /** Breach when free space drops below this percentage. Default 10. */
+  minFreePct?: number
+}
+
+export type ProbeConfig = HttpProbeConfig | Pm2ProbeConfig | DiskProbeConfig
+
+export interface DeploymentConfig {
+  id: string
+  name: string
+  /** Optional link back to the source repo (connects incidents to recent merges). */
+  repoPath?: string
+  enabled: boolean
+  probes: ProbeConfig[]
+}
+
+export interface DeploymentsFile {
+  deployments: DeploymentConfig[]
+}
+
+/** Stable identity of one probe within a deployment — the sample/breach-state key. */
+export function probeKey(deployment: DeploymentConfig, probe: ProbeConfig): string {
+  const target = probe.type === 'http' ? probe.url : probe.type === 'pm2' ? probe.processName : probe.path
+  return `${deployment.id}::${probe.type}:${target}`
+}
+
+// ---------------------------------------------------------------------------
+// Load / Save
+// ---------------------------------------------------------------------------
+
+const CONFIG_DIR = join(homedir(), '.codekin')
+const CONFIG_PATH = join(CONFIG_DIR, 'deployments.json')
+
+export function loadDeployments(): DeploymentsFile {
+  try {
+    if (existsSync(CONFIG_PATH)) {
+      const raw = readFileSync(CONFIG_PATH, 'utf-8')
+      return JSON.parse(raw) as DeploymentsFile
+    }
+  } catch (err) {
+    console.error('[deployments] Failed to load config:', err)
+  }
+  return { deployments: [] }
+}
+
+export function saveDeployments(config: DeploymentsFile): void {
+  if (!existsSync(CONFIG_DIR)) {
+    mkdirSync(CONFIG_DIR, { recursive: true })
+  }
+  writeFileSync(CONFIG_PATH, JSON.stringify(config, null, 2), { encoding: 'utf-8', mode: 0o600 })
+}
+
+// ---------------------------------------------------------------------------
+// CRUD helpers
+// ---------------------------------------------------------------------------
+
+export function upsertDeployment(deployment: DeploymentConfig): DeploymentsFile {
+  const config = loadDeployments()
+  const idx = config.deployments.findIndex(d => d.id === deployment.id)
+  if (idx >= 0) config.deployments[idx] = deployment
+  else config.deployments.push(deployment)
+  saveDeployments(config)
+  return config
+}
+
+export function removeDeployment(id: string): DeploymentsFile {
+  const config = loadDeployments()
+  config.deployments = config.deployments.filter(d => d.id !== id)
+  saveDeployments(config)
+  return config
+}
+
+export function updateDeployment(id: string, patch: Partial<DeploymentConfig>): DeploymentsFile {
+  const config = loadDeployments()
+  const idx = config.deployments.findIndex(d => d.id === id)
+  if (idx < 0) throw new Error(`Deployment not found: ${id}`)
+  config.deployments[idx] = { ...config.deployments[idx], ...patch, id }
+  saveDeployments(config)
+  return config
+}

server/deployment-monitor.test.ts

@@ -0,0 +1,147 @@
+/**
+ * Tests for the DeploymentMonitor — sampling, breach/recovery transition
+ * signals (alert once, not per sample), one-off events, and sample queries.
+ * Uses a real in-memory SQLite database with injected probe runners and a
+ * captured signal publisher.
+ */
+import { describe, it, expect, afterEach } from 'vitest'
+import { DeploymentMonitor, type ProbeResult, type ProbeMetrics } from './deployment-monitor.js'
+import type { DeploymentsFile } from './deployment-config.js'
+
+const NOW = new Date('2026-08-30T12:00:00.000Z')
+
+const okResult = (metrics: ProbeMetrics = {}): ProbeResult => ({ ok: true, breaches: [], events: [], metrics })
+const breachedResult = (breach: string, metrics: ProbeMetrics = {}): ProbeResult =>
+  ({ ok: false, breaches: [breach], events: [], metrics })
+
+function makeConfig(): DeploymentsFile {
+  return {
+    deployments: [{
+      id: 'app-1',
+      name: 'Codekin Prod',
+      enabled: true,
+      probes: [{ type: 'http', url: 'https://example.test/health' }],
+    }],
+  }
+}
+
+describe('DeploymentMonitor', () => {
+  let monitor: DeploymentMonitor
+  let published: Array<{ kind: string; payload?: Record<string, unknown>; dedupeKey?: string }>
+
+  function makeMonitor(httpResults: ProbeResult[], config: DeploymentsFile = makeConfig()) {
+    published = []
+    let call = 0
+    monitor = new DeploymentMonitor({
+      dbPath: ':memory:',
+      publish: (input) => published.push(input),
+      loadConfig: () => config,
+      runners: {
+        http: async () => httpResults[Math.min(call++, httpResults.length - 1)],
+      },
+    })
+    return monitor
+  }
+
+  afterEach(() => {
+    monitor.close()
+  })
+
+  it('samples every enabled probe and records metrics', async () => {
+    makeMonitor([okResult({ status: 200, latencyMs: 45 })])
+    await monitor.sampleAll(NOW)
+
+    const samples = monitor.latestSamples()
+    expect(samples).toHaveLength(1)
+    expect(samples[0]).toMatchObject({
+      deploymentId: 'app-1',
+      probeKey: 'app-1::http:https://example.test/health',
+      ok: true,
+      metrics: { status: 200, latencyMs: 45 },
+    })
+    expect(published).toHaveLength(0)
+  })
+
+  it('skips disabled deployments', async () => {
+    const config = makeConfig()
+    config.deployments[0].enabled = false
+    makeMonitor([okResult()], config)
+    await monitor.sampleAll(NOW)
+    expect(monitor.latestSamples()).toHaveLength(0)
+  })
+
+  it('signals a breach only on the ok → breached transition, and recovery on the way back', async () => {
+    makeMonitor([
+      okResult(),
+      breachedResult('http 502'),
+      breachedResult('http 502'),
+      okResult(),
+    ])
+
+    await monitor.sampleAll(NOW)                                      // ok — nothing
+    await monitor.sampleAll(new Date(NOW.getTime() + 5 * 60_000))     // breach → signal
+    await monitor.sampleAll(new Date(NOW.getTime() + 10 * 60_000))    // still breached → silent
+    await monitor.sampleAll(new Date(NOW.getTime() + 15 * 60_000))    // recovered → signal
+
+    expect(published.map(p => p.kind)).toEqual(['probe-breach', 'probe-recovered'])
+    expect(published[0].payload).toMatchObject({
+      deploymentId: 'app-1',
+      deploymentName: 'Codekin Prod',
+      breaches: ['http 502'],
+    })
+    expect(published[0].dedupeKey).toBe('probe-breach::app-1::http:https://example.test/health')
+  })
+
+  it('signals a breach immediately when the first-ever sample is breached', async () => {
+    makeMonitor([breachedResult('unreachable: ECONNREFUSED')])
+    await monitor.sampleAll(NOW)
+    expect(published.map(p => p.kind)).toEqual(['probe-breach'])
+  })
+
+  it('publishes one-off events regardless of breach state', async () => {
+    published = []
+    const config: DeploymentsFile = {
+      deployments: [{
+        id: 'app-1', name: 'Codekin Prod', enabled: true,
+        probes: [{ type: 'pm2', processName: 'codekin' }],
+      }],
+    }
+    let call = 0
+    monitor = new DeploymentMonitor({
+      dbPath: ':memory:',
+      publish: (input) => published.push(input),
+      loadConfig: () => config,
+      runners: {
+        pm2: async (_probe, previous) => {
+          call++
+          const restarts = call === 1 ? 3 : 4
+          const events: string[] = []
+          const prev = typeof previous?.restarts === 'number' ? previous.restarts : null
+          if (prev !== null && restarts > prev) events.push(`restarted (${prev} → ${restarts})`)
+          return { ok: true, breaches: [], events, metrics: { status: 'online', restarts, memoryMb: 120 } }
+        },
+      },
+    })
+
+    await monitor.sampleAll(NOW)
+    await monitor.sampleAll(new Date(NOW.getTime() + 5 * 60_000))
+
+    expect(published.map(p => p.kind)).toEqual(['probe-event'])
+    expect(published[0].payload).toMatchObject({ event: 'restarted (3 → 4)' })
+  })
+
+  it('returns sample history newest-first and prunes by retention', async () => {
+    makeMonitor([okResult({ status: 200 })])
+    await monitor.sampleAll(NOW)
+    await monitor.sampleAll(new Date(NOW.getTime() + 5 * 60_000))
+
+    const key = 'app-1::http:https://example.test/health'
+    const history = monitor.listSamples({ probeKey: key })
+    expect(history).toHaveLength(2)
+    expect(history[0].id).toBeGreaterThan(history[1].id)
+
+    // Both samples are in the past relative to a far-future cutoff → pruned.
+    monitor.pruneSamples(-24 * 60 * 60 * 1000)
+    expect(monitor.listSamples({ probeKey: key })).toHaveLength(0)
+  })
+})