Prototype: AI-assisted error recovery (#96)

* Prototype: AI-assisted error recovery

Author: daniel-thom

CLAUDE.md

@@ -251,7 +251,7 @@ The Rust client provides a **unified CLI and library interface** with these key
 
 ### Job Status as Integer
 
-Job status values are stored as INTEGER (0-9) in the database, not strings:
+Job status values are stored as INTEGER (0-10) in the database, not strings:
 
 - 0 = uninitialized
 - 1 = blocked
@@ -263,6 +263,7 @@ Job status values are stored as INTEGER (0-9) in the database, not strings:
 - 7 = canceled
 - 8 = terminated
 - 9 = disabled
+- 10 = pending_failed (awaiting AI classification)
 
 ### Resource Formats
 

api/openapi.yaml

@@ -6030,6 +6030,16 @@ components:
           type: integer
         jobs_sort_method:
           $ref: "#/components/schemas/jobs_sort_method"
+        resource_monitor_config:
+          description: Resource monitoring configuration as JSON string
+          type: string
+        slurm_defaults:
+          description: Default Slurm parameters to apply to all schedulers as JSON string
+          type: string
+        use_pending_failed:
+          default: false
+          description: Use PendingFailed status for failed jobs (enables AI-assisted recovery)
+          type: boolean
         status_id:
           type: integer
       required:

docs/src/SUMMARY.md

@@ -7,6 +7,8 @@
 - [Overview](./getting-started/getting-started.md)
 - [Installation](./getting-started/installation.md)
 - [Quick Start (Local)](./getting-started/quick-start-local.md)
+- [Quick Start (HPC/Slurm)](./getting-started/quick-start-hpc.md)
+- [Quick Start (Remote Workers)](./getting-started/quick-start-remote.md)
 
 # Core Documentation
 
@@ -25,7 +27,6 @@
   - [Exporting and Importing Workflows](./core/workflows/export-import-workflows.md)
   - [Archiving Workflows](./core/workflows/archiving.md)
 - [How-Tos](./core/how-to/index.md)
-  - [Submit a Workflow to Slurm](./core/how-to/submit-slurm-workflow.md)
   - [Track Workflow Status](./core/how-to/track-workflow-status.md)
   - [Cancel a Workflow](./core/how-to/cancel-workflow.md)
   - [View Job Logs](./core/how-to/view-job-logs.md)
@@ -58,10 +59,10 @@
 
 ---
 
-# Specialized Topics
+# Execution Modes
 
 - [HPC & Slurm](./specialized/hpc/index.md)
-  - [Quick Start (HPC)](./specialized/hpc/quick-start-hpc.md)
+  - [Submit a Workflow to Slurm](./specialized/hpc/submit-slurm-workflow.md)
   - [Slurm Workflows](./specialized/hpc/slurm-workflows.md)
   - [Debugging Slurm Workflows](./specialized/hpc/debugging-slurm.md)
   - [Working with Slurm](./specialized/hpc/slurm.md)
@@ -70,11 +71,16 @@
   - [HPC Deployment](./specialized/hpc/hpc-deployment.md)
   - [Custom HPC Profile](./specialized/hpc/custom-hpc-profile.md)
 - [Remote Workers](./specialized/remote/index.md)
-  - [Quick Start (Remote Workers)](./specialized/remote/quick-start-remote.md)
   - [Setting Up Remote Workers](./specialized/remote/remote-workers.md)
+
+---
+
+# Advanced Topics
+
 - [Fault Tolerance & Recovery](./specialized/fault-tolerance/index.md)
   - [Automatic Failure Recovery](./specialized/fault-tolerance/automatic-recovery.md)
   - [Configurable Failure Handlers](./specialized/fault-tolerance/failure-handlers.md)
+  - [AI-Assisted Recovery](./specialized/fault-tolerance/ai-assisted-recovery.md)
   - [Job Checkpointing](./specialized/fault-tolerance/checkpointing.md)
 - [Administration & Security](./specialized/admin/index.md)
   - [Server Deployment](./specialized/admin/server-deployment.md)
@@ -100,6 +106,7 @@
   - [Central Database](./specialized/design/database.md)
   - [Workflow Recovery Design](./specialized/design/recovery.md)
   - [Failure Handler Design](./specialized/design/failure-handlers.md)
+  - [AI-Assisted Recovery Design](./specialized/design/ai-assisted-recovery.md)
   - [Workflow Graph](./specialized/design/workflow-graph.md)
   - [Interface Architecture](./specialized/design/interfaces.md)
 

docs/src/core/concepts/job-states.md

@@ -7,34 +7,44 @@ stateDiagram-v2
     [*] --> uninitialized
     uninitialized --> ready: initialize_jobs
     uninitialized --> blocked: has dependencies
+    uninitialized --> disabled: job disabled
 
     blocked --> ready: dependencies met
     ready --> pending: runner claims
     pending --> running: execution starts
 
     running --> completed: exit 0
-    running --> failed: exit != 0
+    running --> failed: exit != 0 (handler match + max retries)
+    running --> pending_failed: exit != 0 (no handler match)
+    running --> ready: exit != 0 (failure handler retry)
     running --> canceled: user cancels
     running --> terminated: system terminates
 
+    pending_failed --> failed: AI classifies as permanent
+    pending_failed --> ready: AI classifies as transient
+    pending_failed --> uninitialized: reset-status
+
     completed --> [*]
     failed --> [*]
     canceled --> [*]
     terminated --> [*]
+    disabled --> [*]
 
     classDef waiting fill:#6c757d,color:#fff
     classDef ready fill:#17a2b8,color:#fff
     classDef active fill:#ffc107,color:#000
     classDef success fill:#28a745,color:#fff
     classDef error fill:#dc3545,color:#fff
     classDef stopped fill:#6f42c1,color:#fff
+    classDef classification fill:#fd7e14,color:#fff
 
     class uninitialized,blocked waiting
     class ready ready
     class pending,running active
     class completed success
     class failed error
-    class canceled,terminated stopped
+    class canceled,terminated,disabled stopped
+    class pending_failed classification
 ```
 
 ## State Descriptions
@@ -48,3 +58,7 @@ stateDiagram-v2
 - **failed** (6) - Finished with error (exit code != 0)
 - **canceled** (7) - Explicitly canceled by user or torc. Never executed.
 - **terminated** (8) - Explicitly terminated by system, such as at wall-time timeout
+- **disabled** (9) - Job is disabled and will not run
+- **pending_failed** (10) - Job failed without a matching failure handler. Awaiting AI-assisted
+  classification to determine if the error is transient (retry) or permanent (fail). See
+  [AI-Assisted Recovery](../specialized/fault-tolerance/ai-assisted-recovery.md).

docs/src/core/concepts/workflow-definition.md

@@ -125,6 +125,40 @@ This creates:
 - 10 parallel `process_*` jobs
 - 1 `aggregate` job that waits for all 10 to complete
 
+## Failure Recovery Options
+
+Control how Torc handles job failures:
+
+### Default Behavior
+
+By default, jobs that fail without a matching failure handler use `Failed` status:
+
+```yaml
+name: my_workflow
+jobs:
+  - name: task
+    command: ./run.sh  # If this fails, status = Failed
+```
+
+### AI-Assisted Recovery (Opt-in)
+
+Enable intelligent classification of ambiguous failures:
+
+```yaml
+name: ml_training
+use_pending_failed: true  # Enable AI-assisted recovery
+
+jobs:
+  - name: train_model
+    command: python train.py
+```
+
+With `use_pending_failed: true`:
+
+- Jobs without matching failure handlers get `PendingFailed` status
+- AI agent can analyze stderr and decide whether to retry or fail
+- See [AI-Assisted Recovery](../../specialized/fault-tolerance/ai-assisted-recovery.md) for details
+
 ## See Also
 
 - [Workflow Specification Formats](../workflows/workflow-formats.md) — Complete syntax reference

docs/src/core/how-to/index.md

@@ -2,7 +2,6 @@
 
 Step-by-step guides for common tasks.
 
-- [Submit a Workflow to Slurm](./submit-slurm-workflow.md) - Running workflows on HPC clusters
 - [Track Workflow Status](./track-workflow-status.md) - Monitoring workflow progress
 - [Cancel a Workflow](./cancel-workflow.md) - Stopping running workflows
 - [View Job Logs](./view-job-logs.md) - Accessing job output

docs/src/core/how-to/submit-slurm-workflow.md

@@ -1,57 +1 @@
-# How to Submit a Workflow to Slurm
-
-Submit a workflow specification to a Slurm-based HPC system with automatic scheduler generation.
-
-## Quick Start
-
-```bash
-torc submit-slurm --account <your-account> workflow.yaml
-```
-
-Torc will:
-
-1. Detect your HPC system (e.g., NREL Kestrel, Eagle)
-2. Match job requirements to appropriate partitions
-3. Generate Slurm scheduler configurations
-4. Submit everything for execution
-
-## Preview Before Submitting
-
-Always preview the generated configuration first:
-
-```bash
-torc slurm generate --account <your-account> workflow.yaml
-```
-
-This shows the Slurm schedulers and workflow actions that would be created without submitting.
-
-## Requirements
-
-Your workflow must define resource requirements for jobs:
-
-```yaml
-name: my_workflow
-
-resource_requirements:
-  - name: standard
-    num_cpus: 4
-    memory: 8g
-    runtime: PT1H
-
-jobs:
-  - name: process_data
-    command: python process.py
-    resource_requirements: standard
-```
-
-## Options
-
-```bash
-# See all options
-torc submit-slurm --help
-```
-
-## See Also
-
-- [Slurm Workflows](../../specialized/hpc/slurm-workflows.md) — Full Slurm integration guide
-- [HPC Profiles](../../specialized/hpc/hpc-profiles.md) — Available HPC system configurations
+# Submit a Workflow to Slurm

docs/src/getting-started/getting-started.md

@@ -77,6 +77,45 @@ The repository includes ready-to-run workflow specifications in YAML, JSON5, and
 
 See the [examples README](https://github.com/NREL/torc/tree/main/examples) for the complete list.
 
-## Next: Quick Start
+## Choose Your Execution Mode
+
+Torc supports three fundamentally different execution environments. Choose the one that matches your
+use case:
+
+### Local Execution
+
+**Best for:** Development, testing, small-scale workflows on your workstation or a single server
+
+- Jobs run directly on the machine where you start the job runner
+- No scheduler needed — simple setup with `torc run`
+- Resource management via local CPU/memory/GPU tracking
+- **→ [Quick Start (Local)](quick-start-local.md)**
+
+### HPC/Slurm
+
+**Best for:** Large-scale computations on institutional HPC clusters
+
+- Jobs submitted to Slurm scheduler for compute node allocation
+- Automatic resource matching to partitions/QOS
+- Built-in profiles for common HPC systems
+- **→ [Quick Start (HPC/Slurm)](quick-start-hpc.md)**
+
+### Remote Workers
+
+**Best for:** Distributed execution across multiple machines you control via SSH
+
+- Jobs distributed to remote workers over SSH
+- No HPC scheduler required — you manage the machines
+- Flexible heterogeneous resources (mix of CPU/GPU machines)
+- **→ [Quick Start (Remote Workers)](quick-start-remote.md)**
+
+---
+
+**All three modes:**
+
+- Share the same workflow specification format
+- Use the same server API for coordination
+- Support the same monitoring tools (CLI, TUI, Dashboard)
+- Can be used together (e.g., develop locally, deploy to HPC)
 
 Continue to the [Quick Start](./quick-start.md) guide to run your first workflow.

docs/src/getting-started/installation.md

@@ -2,7 +2,34 @@
 
 ## Precompiled Binaries (Recommended)
 
-Download precompiled binaries from the [releases page](https://github.com/NREL/torc/releases).
+1. Download the appropriate archive for your platform from the
+   [releases page](https://github.com/NREL/torc/releases):
+   - **Linux**: `torc-<version>-x86_64-unknown-linux-gnu.tar.gz`
+   - **macOS (Intel)**: `torc-<version>-x86_64-apple-darwin.tar.gz`
+   - **macOS (Apple Silicon)**: `torc-<version>-aarch64-apple-darwin.tar.gz`
+
+2. Extract the archive:
+
+   ```bash
+   # For .tar.gz files
+   tar -xzf torc-<version>-<platform>.tar.gz
+
+   # For .zip files
+   unzip torc-<version>-<platform>.zip
+   ```
+
+3. Add the binaries to a directory in your system PATH:
+
+   ```bash
+   # Option 1: Copy to an existing PATH directory
+   cp torc* ~/.local/bin/
+
+   # Option 2: Add the extracted directory to your PATH
+   export PATH="/path/to/extracted/torc:$PATH"
+   ```
+
+   To make the PATH change permanent, add the export line to your shell configuration file
+   (`~/.bashrc`, `~/.zshrc`, etc.).
 
 **macOS users**: The precompiled binaries are not signed with an Apple Developer certificate. macOS
 Gatekeeper will block them by default. To allow the binaries to run, remove the quarantine attribute