Add checks for invalid resource definitions (#243)

* Change default execution mode to direct

The Slurm execution mode does not work in all HPC environments.
- In one case the sacct command submitted after job completion always
  failed.
- In another case the HPC admin stated a strong preference to use direct
  mode in order to reduce the load on the Slurm servers.

* Validate job runtime does not exceed slurm scheduler walltime (#238)

Add validation during workflow creation that checks resource_requirements
runtime against slurm scheduler walltime. For jobs with an explicit
scheduler, runtime must not exceed that scheduler's walltime. For jobs
without an explicit scheduler, at least one scheduler must have a
sufficient walltime since any scheduler can pick up unassigned jobs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Expand scheduler resource validation to memory and GPUs with interactive prompts

Replaces the hard-error runtime-vs-walltime check with a broader
validate_scheduler_resources that checks runtime, memory, and GPUs
against slurm scheduler allocations. Resource warnings are now shown
interactively with a y/N prompt in CLI commands (create, run, submit).
Non-interactive contexts (MCP server, piped stdin) hard-fail with a
message suggesting --skip-checks. Scheduler fields that are not set
(mem, gres) are skipped for that dimension. Unassigned jobs must have
at least one scheduler suitable across all dimensions simultaneously.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix Slurm compute nodes created with is_active=NULL causing false orphan detection

The Slurm job runner's create_compute_node() was not setting is_active=true,
unlike the local job runner in run_jobs_cmd.rs. This caused the watcher's
orphan detection to miss active Slurm compute nodes (SQL NULL != 1), falsely
marking running jobs as orphaned with return code -128.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix direct mode OOM-killing jobs with default resource requirements

In srun mode, jobs with the "default" resource requirement (no explicit
resource_requirements) skip --mem to avoid artificially constraining them.
Direct mode was not doing the same — it enforced the default RR's 1 MiB
memory limit, causing the resource monitor to SIGKILL any job that
allocates real memory. Skip memory limit enforcement for default RRs in
direct mode to match srun behavior.

Also re-applies job_parallelism test to mode: direct now that this works.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add grace period before orphan detection in torc watch

Require 3 consecutive polls with no valid Slurm allocation before running
orphan cleanup. This prevents a race where a Slurm job is just starting up
and squeue hasn't reflected the new job yet, causing the watcher to
falsely mark running jobs as orphaned with return code -128.

With a 10-second poll interval, this gives ~30 seconds for new allocations
to become visible.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix example walltime mismatch and remove bogus srun_termination_signal warning

- slurm_staged_pipeline: increase work_scheduler walltime from 04:00:00
  to 08:00:00 to match work_resources runtime PT8H (YAML, JSON5, KDL)
- Remove false warning comparing srun_termination_signal time to
  sigkill_headroom_seconds — these are independent (signal fires relative
  to the step's --time, not the allocation end)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add interactive recovery

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Author: daniel-thom

docs/src/core/reference/cli-cheatsheet.md

@@ -58,7 +58,8 @@
 | `torc status <id>`                          | Workflow status and job summary                |
 | `torc workflows check-resources <id>`       | Check memory/CPU/time usage                    |
 | `torc results list <id> --include-logs`     | Job results with log paths                     |
-| `torc recover <id>`                         | One-shot recovery (diagnose + fix + resubmit)  |
+| `torc recover <id>`                         | Interactive recovery wizard (default)          |
+| `torc recover <id> --no-prompts`            | Automatic recovery (no prompts, for scripting) |
 | `torc watch <id> --recover --auto-schedule` | Full production recovery mode                  |
 | `torc workflows sync-status <id>`           | Fix orphaned jobs (stuck in "running")         |
 | `torc workflows correct-resources <id>`     | Upscale violated + downsize over-allocated RRs |

docs/src/core/reference/cli.md

@@ -279,6 +279,11 @@ Diagnoses job failures (OOM, timeout), adjusts resource requirements, and resubm
 a workflow has completed with failures. For continuous monitoring, use `torc watch --recover`
 instead.
 
+By default, runs an interactive wizard that displays failed jobs, lets you choose per-category
+actions (retry with adjusted resources or skip), select a Slurm scheduler, and confirm before
+executing. Use `--no-prompts` to skip the wizard and apply heuristics automatically. When stdin is
+not a terminal (e.g., piped or scripted), non-interactive mode is used automatically.
+
 **Usage:** `torc recover [OPTIONS] <WORKFLOW_ID>`
 
 ### Arguments
@@ -294,6 +299,7 @@ instead.
 - `--retry-unknown` — Also retry jobs with unknown failure causes
 - `--recovery-hook <RECOVERY_HOOK>` — Custom recovery script for unknown failures
 - `--dry-run` — Show what would be done without making any changes
+- `--no-prompts` — Skip interactive wizard and apply heuristics automatically
 
 ### When to Use
 
@@ -312,12 +318,15 @@ Use `torc watch --recover` instead for:
 ### Examples
 
 ```bash
-# Basic recovery
+# Interactive recovery (default)
 torc recover 123
 
 # Dry run to preview changes without modifying anything
 torc recover 123 --dry-run
 
+# Skip interactive prompts (for scripting)
+torc recover 123 --no-prompts
+
 # Custom resource multipliers
 torc recover 123 --memory-multiplier 2.0 --runtime-multiplier 1.5
 

docs/src/specialized/fault-tolerance/automatic-recovery.md

@@ -92,14 +92,92 @@ This data is analyzed to determine failure causes:
 For one-shot recovery when a workflow has failed:
 
 ```bash
-# Preview what would be done (recommended first step)
+# Interactive recovery (default when running in a terminal)
+torc recover 42
+
+# Preview what would be done without making changes
 torc recover 42 --dry-run
 
-# Execute the recovery
-torc recover 42
+# Skip interactive prompts (for scripting)
+torc recover 42 --no-prompts
+```
+
+### Interactive Recovery Wizard
+
+By default, `torc recover` runs an interactive wizard that guides you through the recovery process
+step by step:
+
+1. **Diagnose failures** — Categorizes failed jobs into OOM, timeout, and unknown failures and
+   displays a summary table
+2. **Per-category decisions** — For each failure category, choose to retry with adjusted resources,
+   customize the multiplier, or skip
+3. **Scheduler selection** — Choose to auto-generate new Slurm schedulers or reuse an existing one
+   (with optional walltime override and allocation count)
+4. **Review and confirm** — Shows the full recovery plan and asks for confirmation before executing
+
+The wizard runs automatically when stdin is a terminal. When piped or scripted (non-TTY), the
+command falls back to automatic mode. Use `--no-prompts` to explicitly skip the wizard.
+
+#### Example Session
+
 ```
+=== Recovery Wizard ===
+
+Diagnosing failures for workflow 42...
+
+OOM Failures (3 jobs):
+  ID       Name                           RC     Memory     Peak Memory    Reason
+  ---      ----                           ---    ------     -----------    ------
+  107      train_model_7                  137    8g         10.2 GB        sigkill_137
+  112      train_model_12                 137    8g         9.8 GB         memory_exceeded
+  123      train_model_23                 137    8g         11.1 GB        sigkill_137
+
+Timeout Failures (1 job):
+  ID       Name                           RC     Runtime      Exec (min)   Reason
+  ---      ----                           ---    -------      ----------   ------
+  145      postprocess                    152    PT30M        29.8         sigxcpu_152
+
+OOM failures (3 jobs): [R]etry with 1.5x memory / [A]djust multiplier / [S]kip (default: R): r
+Timeout failures (1 job): [R]etry with 1.4x runtime / [A]djust multiplier / [S]kip (default: R): a
+  Enter runtime multiplier [default: 1.4]: 2.0
+
+--- Recovery Plan ---
+
+  Memory: 8g -> 12g (1.5x) for 3 jobs: train_model_7, train_model_12, train_model_23
+  Runtime: PT30M -> PT1H (2x) for 1 job: postprocess
+
+  Total: 4 jobs to retry
+
+--- Slurm Scheduler ---
 
-This command:
+Existing schedulers for this workflow:
+
+  ID     Name                      Account        Partition      Walltime     Nodes
+  ---    ----                      -------        ---------      --------     -----
+  5      gpu_scheduler             myproject      gpu            04:00:00     1
+
+Scheduler: [A]uto-generate new / [E]xisting (enter ID) (default: A): e
+  Enter scheduler ID: 5
+  Walltime [default: 04:00:00] (press Enter to keep): 06:00:00
+  Creating new scheduler with walltime 06:00:00...
+  Created scheduler 'gpu_scheduler_recovery' (ID 8) with walltime 06:00:00
+  Number of allocations [default: 1]: 2
+
+  Scheduler: gpu_scheduler_recovery (ID 8), 2 allocation(s)
+
+Proceed with recovery? (y/N): y
+```
+
+### Non-Interactive Mode
+
+Use `--no-prompts` to skip the wizard and apply recovery heuristics automatically. This is useful
+for scripting or when you want the default behavior without interaction:
+
+```bash
+torc recover 42 --no-prompts
+```
+
+In non-interactive mode, the command:
 
 1. Detects and cleans up orphaned jobs from terminated Slurm allocations
 2. Checks that the workflow is complete and no workers are active
@@ -109,9 +187,8 @@ This command:
 6. Resets failed jobs and regenerates Slurm schedulers
 7. Submits new allocations
 
-> **Note:** Step 1 (orphan cleanup) handles the case where Slurm terminated an allocation
-> unexpectedly, leaving jobs stuck in "running" status. This is done automatically before checking
-> preconditions.
+> **Note:** Orphan cleanup handles the case where Slurm terminated an allocation unexpectedly,
+> leaving jobs stuck in "running" status. This is done automatically before checking preconditions.
 
 ### Options
 
@@ -121,25 +198,8 @@ torc recover <workflow_id> \
   --runtime-multiplier 1.4 \    # Runtime increase factor for timeout (default: 1.4)
   --retry-unknown \             # Also retry jobs with unknown failure causes
   --recovery-hook "bash fix.sh" \  # Custom script for unknown failures
-  --dry-run                     # Preview without making changes
-```
-
-### Example Output
-
-```
-Diagnosing failures...
-Applying recovery heuristics...
-  Job 107 (train_model): OOM detected, increasing memory 8g -> 12g
-  Applied fixes: 1 OOM, 0 timeout
-Resetting 1 job(s) for retry...
-  Reset 1 job(s)
-Reinitializing workflow...
-Regenerating Slurm schedulers...
-  Submitted Slurm allocation with 1 job
-
-Recovery complete for workflow 42
-  - 1 job(s) had memory increased
-Reset 1 job(s). Slurm schedulers regenerated and submitted.
+  --dry-run \                   # Preview without making changes
+  --no-prompts                  # Skip interactive wizard
 ```
 
 ## The `torc watch --recover` Command
@@ -263,13 +323,13 @@ With default settings:
 
 ## Choosing the Right Command
 
-| Use Case                          | Command                  |
-| --------------------------------- | ------------------------ |
-| One-shot recovery after failure   | `torc recover`           |
-| Continuous monitoring             | `torc watch -r`          |
-| Preview what recovery would do    | `torc recover --dry-run` |
-| Production long-running workflows | `torc watch -r`          |
-| Manual investigation, then retry  | `torc recover`           |
+| Use Case                           | Command                     |
+| ---------------------------------- | --------------------------- |
+| Interactive recovery after failure | `torc recover`              |
+| Automatic recovery (scripting)     | `torc recover --no-prompts` |
+| Continuous monitoring              | `torc watch -r`             |
+| Preview what recovery would do     | `torc recover --dry-run`    |
+| Production long-running workflows  | `torc watch -r`             |
 
 ## Complete Workflow Example
 
@@ -530,16 +590,16 @@ If jobs are requesting more resources than partitions allow:
 2. Use smaller multipliers
 3. Consider splitting jobs into smaller pieces
 
-## Comparison: Automatic vs Manual Recovery
+## Comparison: Interactive vs Automatic vs AI-Assisted Recovery
 
-| Feature                | Automatic            | Manual/AI-Assisted      |
-| ---------------------- | -------------------- | ----------------------- |
-| Human involvement      | None                 | Interactive             |
-| Speed                  | Fast                 | Depends on human        |
-| Handles OOM/timeout    | Yes                  | Yes                     |
-| Handles unknown errors | Retry only           | Full investigation      |
-| Cost optimization      | Basic                | Can be sophisticated    |
-| Use case               | Production workflows | Debugging, optimization |
+| Feature                | Interactive (`torc recover`) | Automatic (`--no-prompts`) | AI-Assisted   |
+| ---------------------- | ---------------------------- | -------------------------- | ------------- |
+| Human involvement      | Guided wizard                | None                       | AI + human    |
+| Speed                  | Minutes                      | Fast                       | Varies        |
+| Handles OOM/timeout    | Yes                          | Yes                        | Yes           |
+| Handles unknown errors | User chooses                 | Retry only                 | Investigation |
+| Scheduler control      | Choose or auto-generate      | Auto-generate              | Manual        |
+| Use case               | Most recovery scenarios      | Scripting, `torc watch`    | Complex bugs  |
 
 ## Implementation Details
 

examples/json/slurm_staged_pipeline.json5

@@ -336,7 +336,7 @@
     {
       "name": "work_scheduler",
       "account": "my_account",
-      "walltime": "04:00:00",
+      "walltime": "08:00:00",
       "nodes": 1
     },
     {

examples/kdl/slurm_staged_pipeline.kdl

@@ -14,7 +14,7 @@ slurm_scheduler "setup_scheduler" {
 
 slurm_scheduler "work_scheduler" {
     account "my_account"
-    walltime "04:00:00"
+    walltime "08:00:00"
     nodes 1
 }
 

examples/yaml/slurm_staged_pipeline.yaml

@@ -16,7 +16,7 @@ slurm_schedulers:
     nodes: 1
   - name: "work_scheduler"
     account: "my_account"
-    walltime: "04:00:00"
+    walltime: "08:00:00"
     nodes: 1
   - name: "postprocess_scheduler"
     account: "my_account"

slurm-tests/workflows/cancel_workflow.yaml

@@ -8,7 +8,7 @@ name: cancel_workflow
 description: Workflow cancellation test — cancel while jobs are running
 project: slurm-tests
 execution_config:
-  mode: slurm
+  mode: direct
 
 resource_requirements:
   - name: sleep_resources

slurm-tests/workflows/failure_recovery.yaml

@@ -10,7 +10,7 @@ description: Test workflow for Slurm job retry with failure handlers
 project: slurm-tests
 metadata: '{"test_type": "failure_recovery", "stages": 3}'
 execution_config:
-  mode: slurm
+  mode: direct
 
 failure_handlers:
   - name: retry_on_exit_42

slurm-tests/workflows/job_parallelism.yaml

@@ -1,8 +1,8 @@
 # Test: Job-Based Parallelism
 #
-# 1-node allocation with NO resource_requirements on jobs.
-# Jobs get the auto-assigned "default" RR, so srun skips resource limit flags
-# and each job can use the full allocation's resources.
+# 1-node allocation with NO resource_requirements on jobs (direct mode).
+# Jobs get the auto-assigned "default" RR and run directly (no srun wrapper),
+# so each job can use the full allocation's resources.
 #
 # Concurrency is controlled by --max-parallel-jobs (passed via torc submit).
 # The test submits with --max-parallel-jobs 2, so 2 of the 4 jobs run at a time.
@@ -15,7 +15,7 @@ name: job_parallelism
 description: Job-based parallelism — no resource requirements, controlled by --max-parallel-jobs
 project: slurm-tests
 execution_config:
-  mode: slurm
+  mode: direct
 
 resource_monitor:
   enabled: true

slurm-tests/workflows/resource_monitoring.yaml

@@ -8,7 +8,7 @@ name: resource_monitoring
 description: Resource monitoring validation — CPU and memory usage captured
 project: slurm-tests
 execution_config:
-  mode: slurm
+  mode: direct
 
 resource_monitor:
   enabled: true