Implement Dashboard Job Submission with Relay Server

[alicup29]

Summary

Replaces the direct WebRTC data channel approach for dashboard↔worker communication with an HTTP + SSE relay architecture. The dashboard can now browse worker filesystems, submit training jobs, and monitor live epoch-level progress — all without establishing WebRTC peer connections.

Motivation

The original dashboard job submission used WebRTC data channels between the browser and HPC workers. This worked on local networks but consistently failed in production HPC environments due to:

• Symmetric NAT on HPC clusters blocking STUN-based hole punching
• Firewall rules dropping UDP traffic and blocking TURN relay
• ICE negotiation timeouts causing jobs to silently fail to submit

After 8+ commits debugging ICE/STUN issues on the earlier add-dashboard-job-submission branch, we redesigned the architecture around outbound-only connections from workers, routing all communication through the signaling server and a lightweight SSE relay on the EC2 instance. This eliminates NAT traversal entirely — workers connect outbound via WebSocket, and browsers connect outbound via HTTP/SSE.

See the companion server-side PR: webRTC-connect#29

Architecture

Dashboard (browser)
    │
    ├── HTTP POST ──→ Signaling Server ──→ Worker (via WebSocket)
    │                      │
    └── SSE ←── Relay ←────┘ (worker sends status/progress back)

• File browsing: Dashboard sends fs_list HTTP request → signaling pushes fs_list_req to worker WS → worker responds with fs_list_res → signaling forwards to relay → dashboard receives via SSE
• Job submission: Dashboard POSTs config to /api/jobs/submit → signaling pushes job_assigned to worker → worker runs job, sends job_status/job_progress over WS → relay → dashboard SSE
• Training progress: Worker's RelayChannel shim filters PROGRESS_REPORT:: messages, forwarding only epoch-level events (train_begin, epoch_end, train_end) to avoid flooding the relay

Changes

Dashboard (dashboard/)

• app.js: Replaced ~200 lines of WebRTC connection code (connectToWorker, disconnectFromWorker, sendFsMessage, onDataChannelMessage) with SSE helpers (sseConnect) and HTTP API wrappers. Added full Step 3 validation flow (browse .slp → validate path → check videos → resolve missing → submit). Added job_progress SSE handler with training log terminal display.
• index.html: Added #sj-training-log container for live epoch output
• styles.css: Added training log terminal styles and validation/video resolution UI styles
• config.js: Updated endpoints for relay server and signaling.sleap.ai

Worker (sleap_rtc/worker/)

• mesh_coordinator.py: Added relay-forwarded message handlers (fs_list_req, use_worker_path, fs_check_videos, job_assigned) that translate between JSON relay format and the worker's existing :: protocol. Added RelayChannel shim class that mimics RTCDataChannel.send() but routes protocol messages through WebSocket as JSON, with epoch-level progress filtering.
• worker_class.py: Added worker mounts to registration properties for dashboard metadata enrichment

Key fixes along the way

• Worker status mismatch (available vs idle) preventing worker selection
• SSE offset appended to filesystem path breaking mount validation
• Stale SSE responses resetting file browser columns
• Relay job_id collision from spread operator letting internal per-model IDs overwrite the signaling server's job_id
• Infinite scroll re-rendering wiping deeper navigation columns

Test plan

• File browser navigates worker filesystem via relay
• Path validation and video resolution work end-to-end
• Job submission succeeds and status updates flow to dashboard
• Training log terminal shows epoch summaries with loss metrics
• WandB link appears when enabled in training config
• Job completion and failure states display correctly
• Relay is not flooded (only ~1 message per epoch forwarded)

---

🤖 Generated with Claude Code