Merge pull request #53 from NREL/feat/slurm-debugging

Add slurm parse-logs and sacct CLI commands

Author: daniel-thom

docs/src/how-to/debugging.md

@@ -498,12 +498,214 @@ find . -name "job_*.o" -o -name "job_runner_*.log"
 torc reports results <workflow_id> --output-dir <correct_path>
 ```
 
+## Slurm-Specific Debugging Tools
+
+When running workflows on Slurm clusters, Torc provides additional debugging tools specifically designed for Slurm environments.
+
+### Parsing Slurm Log Files for Errors
+
+The `torc slurm parse-logs` command scans Slurm stdout/stderr log files for known error patterns and correlates them with affected Torc jobs:
+
+```bash
+# Parse logs for a specific workflow
+torc slurm parse-logs <workflow_id>
+
+# Specify custom output directory
+torc slurm parse-logs <workflow_id> --output-dir /path/to/output
+
+# Output as JSON for programmatic processing
+torc slurm parse-logs <workflow_id> --format json
+```
+
+The command detects common Slurm failure patterns including:
+
+**Memory Errors:**
+- `out of memory`, `oom-kill`, `cannot allocate memory`
+- `memory cgroup out of memory`, `Exceeded job memory limit`
+- `task/cgroup: .*: Killed`
+- `std::bad_alloc` (C++), `MemoryError` (Python)
+
+**Slurm-Specific Errors:**
+- `slurmstepd: error:`, `srun: error:`
+- `DUE TO TIME LIMIT`, `DUE TO PREEMPTION`
+- `NODE_FAIL`, `FAILED`, `CANCELLED`
+- `Exceeded.*step.*limit`
+
+**GPU/CUDA Errors:**
+- `CUDA out of memory`, `CUDA error`, `GPU memory.*exceeded`
+
+**Signal/Crash Errors:**
+- `Segmentation fault`, `SIGSEGV`
+- `Bus error`, `SIGBUS`
+- `killed by signal`, `core dumped`
+
+**Python Errors:**
+- `Traceback (most recent call last)`
+- `ModuleNotFoundError`, `ImportError`
+
+**File System Errors:**
+- `No space left on device`, `Disk quota exceeded`
+- `Read-only file system`, `Permission denied`
+
+**Network Errors:**
+- `Connection refused`, `Connection timed out`, `Network is unreachable`
+
+**Example output (table format):**
+
+```
+Slurm Log Analysis Results
+==========================
+
+Found 2 error(s) in log files:
+
+╭─────────────────────────────┬──────────────┬──────┬─────────────────────────────┬──────────┬──────────────────────────────╮
+│ File                        │ Slurm Job ID │ Line │ Pattern                     │ Severity │ Affected Torc Jobs           │
+├─────────────────────────────┼──────────────┼──────┼─────────────────────────────┼──────────┼──────────────────────────────┤
+│ slurm_output_12345.e        │ 12345        │ 42   │ Out of Memory (OOM) Kill    │ critical │ process_data (ID: 456)       │
+│ slurm_output_12346.e        │ 12346        │ 15   │ CUDA out of memory          │ error    │ train_model (ID: 789)        │
+╰─────────────────────────────┴──────────────┴──────┴─────────────────────────────┴──────────┴──────────────────────────────╯
+```
+
+### Viewing Slurm Accounting Data
+
+The `torc slurm sacct` command displays a summary of Slurm job accounting data for all scheduled compute nodes in a workflow:
+
+```bash
+# Display sacct summary table for a workflow
+torc slurm sacct <workflow_id>
+
+# Also save full JSON files for detailed analysis
+torc slurm sacct <workflow_id> --save-json --output-dir /path/to/output
+
+# Output as JSON for programmatic processing
+torc slurm sacct <workflow_id> --format json
+```
+
+The command displays a summary table with key metrics:
+- **Slurm Job**: The Slurm job ID
+- **Job Step**: Name of the job step (e.g., "worker_1", "batch")
+- **State**: Job state (COMPLETED, FAILED, TIMEOUT, OUT_OF_MEMORY, etc.)
+- **Exit Code**: Exit code of the job step
+- **Elapsed**: Wall clock time for the job step
+- **Max RSS**: Maximum resident set size (memory usage)
+- **CPU Time**: Total CPU time consumed
+- **Nodes**: Compute nodes used
+
+**Example output:**
+
+```
+Slurm Accounting Summary for Workflow 123
+
+╭────────────┬───────────┬───────────┬───────────┬─────────┬─────────┬──────────┬─────────╮
+│ Slurm Job  │ Job Step  │ State     │ Exit Code │ Elapsed │ Max RSS │ CPU Time │ Nodes   │
+├────────────┼───────────┼───────────┼───────────┼─────────┼─────────┼──────────┼─────────┤
+│ 12345      │ worker_1  │ COMPLETED │ 0         │ 2h 15m  │ 4.5GB   │ 4h 30m   │ node01  │
+│ 12345      │ batch     │ COMPLETED │ 0         │ 2h 16m  │ 128.0MB │ 1m 30s   │ node01  │
+│ 12346      │ worker_1  │ FAILED    │ 1         │ 45m 30s │ 8.2GB   │ 1h 30m   │ node02  │
+╰────────────┴───────────┴───────────┴───────────┴─────────┴─────────┴──────────┴─────────╯
+
+Total: 3 job steps
+```
+
+Use `--save-json` to also save full sacct JSON output to files for detailed analysis:
+```bash
+torc slurm sacct 123 --save-json --output-dir output
+# Creates: output/sacct_12345.json, output/sacct_12346.json, etc.
+```
+
+### Viewing Slurm Logs in torc-dash
+
+The torc-dash web interface provides two ways to view Slurm logs:
+
+#### Debugging Tab - Slurm Log Analysis
+
+The Debugging tab includes a "Slurm Log Analysis" section:
+
+1. Navigate to the **Debugging** tab
+2. Find the **Slurm Log Analysis** section
+3. Enter the output directory path (default: `output`)
+4. Click **Analyze Slurm Logs**
+
+The results show all detected errors with their Slurm job IDs, line numbers, error patterns, severity levels, and affected Torc jobs.
+
+#### Debugging Tab - Slurm Accounting Data
+
+The Debugging tab also includes a "Slurm Accounting Data" section:
+
+1. Navigate to the **Debugging** tab
+2. Find the **Slurm Accounting Data** section
+3. Click **Collect sacct Data**
+
+This displays a summary table showing job state, exit codes, elapsed time, memory usage (Max RSS), CPU time, and nodes for all Slurm job steps. The table helps quickly identify failed jobs and resource usage patterns.
+
+#### Scheduled Nodes Tab - View Slurm Logs
+
+You can view individual Slurm job logs directly from the Details view:
+
+1. Select a workflow
+2. Go to the **Details** tab
+3. Switch to the **Scheduled Nodes** sub-tab
+4. Find a Slurm scheduled node in the table
+5. Click the **View Logs** button in the Logs column
+
+This opens a modal with tabs for viewing the Slurm job's stdout and stderr files.
+
+### Viewing Slurm Logs in the TUI
+
+The `torc tui` terminal interface also supports Slurm log viewing:
+
+1. Launch the TUI: `torc tui`
+2. Select a workflow and press Enter to load details
+3. Press Tab to switch to the **Scheduled Nodes** tab
+4. Navigate to a Slurm scheduled node using arrow keys
+5. Press `l` to view the Slurm job's logs
+
+The log viewer shows:
+- **stdout tab**: Slurm job standard output (`slurm_output_<id>.o`)
+- **stderr tab**: Slurm job standard error (`slurm_output_<id>.e`)
+
+Use Tab to switch between stdout/stderr, arrow keys to scroll, `/` to search, and `q` to close.
+
+### Debugging Slurm Job Failures
+
+When a Slurm job fails, follow this debugging workflow:
+
+1. **Parse logs for known errors:**
+   ```bash
+   torc slurm parse-logs <workflow_id>
+   ```
+
+2. **If OOM or resource issues are detected, collect sacct data:**
+   ```bash
+   torc slurm sacct <workflow_id>
+   cat output/sacct_<slurm_job_id>.json | jq '.jobs[].steps[].tres.requested'
+   ```
+
+3. **View the specific Slurm log files:**
+   - Use torc-dash: Details → Scheduled Nodes → View Logs
+   - Or use TUI: Scheduled Nodes tab → press `l`
+   - Or directly: `cat output/slurm_output_<slurm_job_id>.e`
+
+4. **Check the job's own stderr for application errors:**
+   ```bash
+   torc reports results <workflow_id> > report.json
+   jq -r '.results[] | select(.return_code != 0) | .job_stderr' report.json | xargs cat
+   ```
+
+5. **Review dmesg logs for system-level issues:**
+   ```bash
+   cat output/dmesg_slurm_<slurm_job_id>_*.log
+   ```
+
 ## Related Commands
 
 - **`torc results list`**: View summary of job results in table format
 - **`torc workflows status`**: Check overall workflow status
 - **`torc reports check-resource-utilization`**: Analyze resource usage and find over-utilized jobs
 - **`torc jobs list`**: View all jobs and their current status
+- **`torc slurm parse-logs`**: Parse Slurm logs for known error patterns
+- **`torc slurm sacct`**: Collect Slurm accounting data for workflow jobs
 - **`torc-dash`**: Launch web interface with interactive Debugging tab
+- **`torc tui`**: Launch terminal UI with Slurm log viewing
 
 The `reports results` command and torc-dash Debugging tab complement these by providing complete log file paths and content viewing for in-depth debugging when high-level views aren't sufficient.