fix: dashboard terminal endpoints die without supervision causing recurring /ao outages

[AgentWrapper]

Problem

AO dashboard and terminal endpoints are frequently unavailable in real usage because web + websocket processes are started as ad-hoc dev commands and are not supervised. Users repeatedly hit:

• AO dashboard upstream unavailable ... 127.0.0.1:3000 connection refused
• terminal stuck at Connecting… XDA when 14800/14801 are down

Why this matters

This breaks AO day-to-day operation and undermines confidence in the orchestration UX.

Scope

Implement a permanent production-grade runtime model so dashboard and terminal sockets are resilient.

Requirements

1. Provide a supervised startup path (systemd user service or equivalent) for:
   • dashboard web server
   • terminal websocket server
   • direct terminal websocket server
2. Auto-restart on crash and boot/login where appropriate.
3. Health checks + clear status command output for 3000/14800/14801 readiness.
4. Remove dependency on fragile ad-hoc dev shell processes for normal operation.
5. Document operator setup/recovery in repo docs.

Acceptance Criteria

• Killing one process auto-recovers without manual restarts.
• /ao/ and /sessions/<id> remain reachable under routine restarts.
• Terminal connects reliably (no perpetual Connecting… XDA from missing ws backends).
• Tests added for health/status behavior where feasible.

Nice to have

• CLI command for ao services install|start|stop|status
• watchdog that self-heals missing terminal ws processes.

[AgentWrapper]

Progress update (Mar 10, 2026):\n\n1) Immediate host hotfix is in place to restore operator usability now:\n- Added temporary user-systemd units on this host for dashboard (3000), terminal-ws (14800), and direct-terminal-ws (14801).\n- All units configured with auto-restart and enabled for user session startup.\n- Verified health endpoints return 200 on all three ports.\n- Verified crash recovery by killing terminal-ws main PID; systemd restarted it automatically.\n\n2) Next in progress:\n- Implementing permanent repo-wide supervised runtime model (systemd + portable fallback), health/status checks, docs, and tests for issue #417.

[AgentWrapper]

Progress update (March 10, 2026)

1. Immediate host hotfix is active to restore operator usability now.

• Temporary user systemd units are running for dashboard (3000), terminal-ws (14800), and direct-terminal-ws (14801).
• All three are configured with Restart=always and enabled in the user session.
• Health checks are green on all three ports (200 on /, /health, /health).
• Crash recovery verified by killing terminal-ws main PID and confirming automatic restart.

2. Permanent fix for this repo is in progress.

• Implementing first-class ao services install|start|stop|status with supervised runtime model.
• Adding health/status output for 3000/14800/14801 readiness.
• Updating docs for setup/recovery + migration from ad-hoc dev commands.
• Adding tests for health/status behavior.