Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions docs/DEPLOYMENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ Deployments live in `~/.codekin/deployments.json`:
| `http` | status, latency, TLS days-remaining (`checkTls`), security headers (`checkHeaders`) | non-expected status (default: ≥400), unreachable/timeout, certificate < 14 days, missing HSTS/CSP headers |
| `pm2` | status, restart count, memory | process missing, status ≠ `online`, memory > `memoryLimitMb`; a restart-count increase publishes a one-off event |
| `disk` | free % | free % < `minFreePct` (default 10) |
| `host` | memory available %, load per core, apt upgradable/security counts (cached 6h), reboot-required | memory < `minMemAvailablePct` (10), load/core > `maxLoadPerCore` (3), pending security updates, reboot required |

Probe *failures* (pm2 absent, `df` unparseable) are breaches too — a broken probe is visible, never silent.

Expand All @@ -46,6 +47,22 @@ Sampling rides the trigger engine's tick (`registerTickTask`, every 5 minutes)

Breach detection fires on **transitions**, not on every breached sample: `ok → breached` publishes a `probe-breach` signal (once), `breached → ok` publishes `probe-recovered`. Both flow through the durable signal queue (at-least-once, deduped while pending) and land in the orchestrator's notifications. The orchestrator can then inspect current state (`list_deployments`) and history (`get_deployment_samples`) before deciding whether to act.

## Host monitoring & the maintenance ladder

The `host` probe treats the machine itself as a monitored asset. Register it on any deployment (conventionally a dedicated `"host"` entry):

```json
{ "id": "host", "name": "This machine", "enabled": true, "probes": [{ "type": "host" }, { "type": "disk", "path": "/" }] }
```

Maintenance autonomy follows the trust ladder from the expansion plan, and this phase implements the first two rungs:

- **Observe** — breaches and metrics flow to the orchestrator like any probe.
- **Propose** — breaches that need privileges to fix carry the exact operator-run command in their text (e.g. `sudo apt-get update && sudo apt-get upgrade`, or a reboot window). The orchestrator relays and tracks; it never executes.
- **Routine-execute** is deliberately not implemented: the pre-approved action-class list starts empty, and the hard floor (restarts, reboots, anything touching live sessions) always requires a human regardless of trust.

A **weekly host digest** (memory, load, pending updates, reboot state, deployment-probe health) is delivered to the orchestrator as a notification; the last-sent timestamp persists across restarts. No digest is sent when no host probe is configured.

## Incident response

A breach always notifies the orchestrator, which can inspect state and spawn a diagnostic child under its normal trust rules. Additionally, a deployment with **`"autoDiagnose": true`** (operator opt-in, requires `repoPath`) spawns the diagnostic child automatically the moment a breach signal is processed:
Expand Down
20 changes: 18 additions & 2 deletions server/deployment-config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,20 @@ export interface DiskProbeConfig {
minFreePct?: number
}

export type ProbeConfig = HttpProbeConfig | Pm2ProbeConfig | DiskProbeConfig
/**
* Host probe config lives in host-probe.ts; re-declared here structurally to
* keep this module dependency-free. `type: 'host'` monitors the machine
* itself: memory, load, pending updates, reboot-required — all sudo-free.
*/
export interface HostProbeConfigRef {
type: 'host'
minMemAvailablePct?: number
maxLoadPerCore?: number
alertOnSecurityUpdates?: boolean
alertOnRebootRequired?: boolean
}

export type ProbeConfig = HttpProbeConfig | Pm2ProbeConfig | DiskProbeConfig | HostProbeConfigRef

export interface DeploymentConfig {
id: string
Expand All @@ -66,7 +79,10 @@ export interface DeploymentsFile {

/** Stable identity of one probe within a deployment — the sample/breach-state key. */
export function probeKey(deployment: DeploymentConfig, probe: ProbeConfig): string {
const target = probe.type === 'http' ? probe.url : probe.type === 'pm2' ? probe.processName : probe.path
const target = probe.type === 'http' ? probe.url
: probe.type === 'pm2' ? probe.processName
: probe.type === 'disk' ? probe.path
: 'system'
return `${deployment.id}::${probe.type}:${target}`
}

Expand Down
9 changes: 7 additions & 2 deletions server/deployment-monitor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,9 @@ import {
type HttpProbeConfig,
type Pm2ProbeConfig,
type DiskProbeConfig,
type HostProbeConfigRef,
} from './deployment-config.js'
import { runHostProbe } from './host-probe.js'

const execFileAsync = promisify(execFile)

Expand Down Expand Up @@ -64,6 +66,7 @@ export interface ProbeRunners {
http: (probe: HttpProbeConfig) => Promise<ProbeResult>
pm2: (probe: Pm2ProbeConfig, previous: ProbeMetrics | null) => Promise<ProbeResult>
disk: (probe: DiskProbeConfig) => Promise<ProbeResult>
host: (probe: HostProbeConfigRef) => Promise<ProbeResult>
}

/** Durable-queue publisher — the trigger engine's enqueueSignal, injected. */
Expand Down Expand Up @@ -208,7 +211,7 @@ async function runDiskProbe(probe: DiskProbeConfig): Promise<ProbeResult> {
return { ok: breaches.length === 0, breaches, events: [], metrics }
}

const DEFAULT_RUNNERS: ProbeRunners = { http: runHttpProbe, pm2: runPm2Probe, disk: runDiskProbe }
const DEFAULT_RUNNERS: ProbeRunners = { http: runHttpProbe, pm2: runPm2Probe, disk: runDiskProbe, host: runHostProbe }

// ---------------------------------------------------------------------------
// Monitor
Expand Down Expand Up @@ -280,7 +283,9 @@ export class DeploymentMonitor {
? await this.runners.http(probe)
: probe.type === 'pm2'
? await this.runners.pm2(probe, previous?.metrics ?? null)
: await this.runners.disk(probe)
: probe.type === 'disk'
? await this.runners.disk(probe)
: await this.runners.host(probe)

this.db.prepare(`
INSERT INTO deployment_samples (deployment_id, probe_key, probe_type, ok, breaches, metrics, created_at)
Expand Down
2 changes: 1 addition & 1 deletion server/deployment-routes.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ import { tryGetDeploymentMonitor, discoverPm2Processes } from './deployment-moni
type VerifyFn = (token: string | undefined) => boolean
type ExtractFn = (req: Request) => string | undefined

const PROBE_TYPES = new Set(['http', 'pm2', 'disk'])
const PROBE_TYPES = new Set(['http', 'pm2', 'disk', 'host'])

/** Structural validation of a probe entry; returns an error string or null. */
function validateProbe(probe: unknown): string | null {
Expand Down
101 changes: 101 additions & 0 deletions server/host-probe.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
/** Tests for the host probe — pure evaluation over injected raw readings, and the weekly digest builder. */
import { describe, it, expect } from 'vitest'
import { evaluateHostProbe, buildHostDigest, type HostRaw } from './host-probe.js'
import type { DeploymentSample } from './deployment-monitor.js'

const HEALTHY: HostRaw = {
memTotalKb: 8 * 1024 * 1024,
memAvailableKb: 4 * 1024 * 1024,
load1: 0.5,
cores: 4,
upgradable: 0,
securityUpgradable: 0,
rebootRequired: false,
}

describe('evaluateHostProbe', () => {
it('is ok on a healthy host and reports rounded metrics', () => {
const result = evaluateHostProbe({ type: 'host' }, HEALTHY)
expect(result.ok).toBe(true)
expect(result.metrics).toMatchObject({
memAvailablePct: 50,
memTotalMb: 8192,
load1: 0.5,
cores: 4,
upgradable: 0,
securityUpgradable: 0,
rebootRequired: 0,
})
})

it('breaches on low memory and high load per core', () => {
const result = evaluateHostProbe({ type: 'host' }, {
...HEALTHY,
memAvailableKb: 400 * 1024, // ~5%
load1: 14, // 3.5/core
})
expect(result.ok).toBe(false)
expect(result.breaches).toEqual([
'memory low: 5% available',
'load high: 14.00 (3.50 per core)',
])
})

it('proposes operator-run remediation for pending security updates and reboot', () => {
const result = evaluateHostProbe({ type: 'host' }, {
...HEALTHY,
upgradable: 12,
securityUpgradable: 3,
rebootRequired: true,
})
expect(result.breaches).toHaveLength(2)
expect(result.breaches[0]).toContain('3 security update(s) pending')
expect(result.breaches[0]).toContain('operator-run')
expect(result.breaches[1]).toContain('reboot required')
})

it('respects opt-outs and custom thresholds', () => {
const relaxed = evaluateHostProbe(
{ type: 'host', alertOnSecurityUpdates: false, alertOnRebootRequired: false, maxLoadPerCore: 10 },
{ ...HEALTHY, securityUpgradable: 5, rebootRequired: true, load1: 20 },
)
expect(relaxed.ok).toBe(true)
})

it('treats unreadable sources as unknown, not breached', () => {
const result = evaluateHostProbe({ type: 'host' }, {
...HEALTHY,
memTotalKb: null,
memAvailableKb: null,
upgradable: null,
securityUpgradable: null,
})
expect(result.ok).toBe(true)
expect(result.metrics.memAvailablePct).toBeNull()
expect(result.metrics.upgradable).toBeNull()
})
})

describe('buildHostDigest', () => {
const hostSample: DeploymentSample = {
id: 10, deploymentId: 'host', probeKey: 'host::host:system', probeType: 'host',
ok: false,
breaches: ['2 security update(s) pending (proposed, operator-run: sudo apt-get update && sudo apt-get upgrade)'],
metrics: { memAvailablePct: 42, memTotalMb: 8192, load1: 0.7, cores: 4, upgradable: 9, securityUpgradable: 2, rebootRequired: 1 },
createdAt: '2026-08-30T12:00:00.000Z',
}
const httpSample: DeploymentSample = {
id: 11, deploymentId: 'app', probeKey: 'app::http:https://x/health', probeType: 'http',
ok: true, breaches: [], metrics: { status: 200 }, createdAt: '2026-08-30T12:00:00.000Z',
}

it('summarizes host state, proposals, and deployment probe health', () => {
const digest = buildHostDigest(hostSample, [hostSample, httpSample])
expect(digest).toContain('Memory: 42% available of 8192MB')
expect(digest).toContain('Updates: 9 upgradable, 2 security')
expect(digest).toContain('operator-run: sudo apt-get update')
expect(digest).toContain('Reboot required: YES')
expect(digest).toContain('Deployment probes: 1/1 healthy')
expect(digest).toContain('Active host breaches:')
})
})
Loading
Loading