Dashboard Multi-config Training, Stop Early, and Cancel fixes

[alicup29]

Summary

Enables uploading multiple training config YAMLs (e.g., centroid + centered_instance) in a single dashboard job submission, with per-model progress tracking, Stop Early, and Cancel Training support.

Replaces #74 (which had merge conflicts from squash-merged #73).

Motivation

The SLEAP top-down pipeline requires two sequential models (centroid → centered_instance). Previously, the dashboard only supported single-config submissions, forcing users to submit and monitor two separate jobs. This PR enables the full top-down workflow in one submission, matching what the SLEAP GUI and sleap-app already support.

Key Changes

Dashboard — Multi-config upload

• Config upload UI now shows config cards with auto-detected model type, filename, and per-card hyperparameters (batch size, learning rate, max epochs, run name, WandB project)
• "+ Add Another Config" mini dropzone appears after first config is added
• submitJob() sends config_contents (array) and model_types (array) instead of singular config_content
• Queue label shows "Model 1 / 2", "Model 2 / 2" during training

Dashboard — Stop Early button

• Yellow "Stop Early" button alongside "Cancel Training" and "Close"
• Sends mode: "stop" via generic relay — stops current model at checkpoint, continues to next model and inference
• Idempotent (ignores clicks if already stopping)

Dashboard — Model switch detection

• MODEL_TYPE:: messages from the worker are relayed as SSE events with event: "model_type_switch"
• Dashboard detects model switch and updates: status label ("Starting centered_instance..."), epoch counter (reset to 0), metrics (reset), max epochs (from new config), queue label
• Stop Early and Cancel buttons re-enable for each new model
• Worker logs include model name: "Training started (centroid)", "Training complete (centroid)"

Worker — Stop Early vs Cancel Training

• _handle_job_cancel reads mode field: "stop" (stop early) vs "cancel" (cancel everything)
• Both send ZMQ {"command": "stop"} to sleap-nn's TrainingControllerZMQ for graceful DDP shutdown
• Cancel sets _cancel_requested flag → pipeline skips remaining models + inference
• Stop Early only sets _stop_requested → pipeline continues to next model

Worker — Cancel fixes

• cancelled check now runs BEFORE returncode == 0 check in exit analysis — previously ZMQ cancel resulted in clean exit (code 0) which was misidentified as success
• MODEL_TYPE:: relay added to RelayChannel so dashboard receives model switch events
• Stop/cancel routed through generic relay (/api/worker/message) instead of dedicated endpoint — no signaling server changes needed

Files changed

File	Change
dashboard/app.js	Multi-config cards, Stop Early handler, model switch detection, per-model progress
dashboard/index.html	Config list container, Stop Early button
dashboard/styles.css	Config card styles, mini dropzone, .btn-warning
sleap_rtc/worker/mesh_coordinator.py	MODEL_TYPE:: relay, _handle_job_cancel with mode field
sleap_rtc/worker/job_executor.py	_cancel_requested flag, cancelled-before-success exit check

Test plan

• Upload single config — works as before, no changes
• Upload two configs (centroid + centered_instance) — per-card hyperparams shown
• Submit multi-config job — "Model 1 / 2" shown, training starts
• Stop Early on model 1 — stops at checkpoint, model 2 starts, buttons re-enable
• Cancel Training on model 1 — stops training, skips model 2 + inference
• Stop Early on model 2 — stops at checkpoint, inference runs
• Cancel Training on model 2 — stops training, skips inference
• Worker logs show "— centroid stopped, starting centered_instance —"
• Status label updates: "Training centroid..." → "Starting centered_instance..." → "Training centered_instance..."

🤖 Generated with Claude Code