Fix the watcher (#63)

* Fixes many behaviors in the watcher regarding recovery logic.

* Only collect dmesg output if failures occurred.

* Fix the slurm regenerate command to create optimal Slurm configurations for remaining work.

* Add optimizations to server database queries involving job completions.

* Make the poll intervals consistent between the job-runner and server.

* Improve the summary report.

* Add linting of code and docs.

Author: daniel-thom

.cargo-husky/hooks/pre-commit

@@ -0,0 +1,16 @@
+#!/bin/sh
+#
+# Pre-commit hook for torc project
+# Runs Rust formatting, linting, and markdown formatting
+#
+
+set -e
+
+echo '+cargo fmt -- --check'
+cargo fmt -- --check
+
+echo '+cargo clippy -- -D warnings'
+cargo clippy -- -D warnings
+
+echo '+dprint check'
+dprint check

.github/CROSS_COMPILATION_TEST_RESULTS.md

@@ -2,14 +2,12 @@
 
 ## Test Summary
 
-**Date**: 2025-11-24
-**Status**: ✅ **LIKELY TO SUCCEED** with recommended changes
+**Date**: 2025-11-24 **Status**: ✅ **LIKELY TO SUCCEED** with recommended changes
 
 ## Binaries Tested
 
-✅ `torc` (client + TUI + plot_resources) - Builds successfully
-✅ `torc-server` - Builds successfully
-✅ `torc-slurm-job-runner` - Builds successfully
+✅ `torc` (client + TUI + plot_resources) - Builds successfully ✅ `torc-server` - Builds
+successfully ✅ `torc-slurm-job-runner` - Builds successfully
 
 ## Dependency Analysis
 
@@ -18,6 +16,7 @@
 **Status**: Should work across all platforms
 
 **Analysis**:
+
 - `plotly_kaleido` downloads pre-built binaries at build time
 - Supports all our target platforms:
   - ✅ Linux x86_64
@@ -33,19 +32,23 @@
 **Status**: Will likely fail on musl builds without changes
 
 **Current situation**:
+
 - `torc-server` depends directly on `openssl` crate
 - Main `Cargo.toml` has `openssl-sys` with `vendored` feature for dev-dependencies only
 - Musl builds require vendored OpenSSL or static linking
 
-**Issue**: Cross-compilation with `cross` tool might fail for musl targets if OpenSSL isn't vendored for release builds.
+**Issue**: Cross-compilation with `cross` tool might fail for musl targets if OpenSSL isn't vendored
+for release builds.
 
 ### 3. Other Dependencies - ✅ GOOD
 
 **SQLite** (via rusqlite):
+
 - Uses `bundled` feature - will compile from source
 - Works perfectly for all targets
 
 **TUI dependencies** (ratatui, crossterm):
+
 - Pure Rust, no native dependencies
 - Should work fine everywhere
 
@@ -84,25 +87,25 @@ Set environment variable in the GitHub workflow:
 
 The `cross` tool includes OpenSSL in its Docker images. The workflow may work as-is.
 
-**Recommendation**: Try the workflow as-is first. If it fails with OpenSSL linking errors, implement Option 1.
+**Recommendation**: Try the workflow as-is first. If it fails with OpenSSL linking errors, implement
+Option 1.
 
 ## Potential Issues to Watch
 
 ### 1. Kaleido Download Failures
 
-**Symptom**: Build fails with "failed to download kaleido"
-**Cause**: GitHub rate limiting or network issues
-**Solution**: Builds run on GitHub Actions should have good connectivity
+**Symptom**: Build fails with "failed to download kaleido" **Cause**: GitHub rate limiting or
+network issues **Solution**: Builds run on GitHub Actions should have good connectivity
 
 ### 2. Windows OpenSSL
 
-**Status**: Already handled correctly
-**Evidence**: Existing test workflow successfully builds on Windows using vcpkg
+**Status**: Already handled correctly **Evidence**: Existing test workflow successfully builds on
+Windows using vcpkg
 
 ### 3. Binary Size
 
-**Observation**: Musl builds with vendored OpenSSL will be larger
-**Mitigation**: Already using `--release` flag. Consider adding strip step:
+**Observation**: Musl builds with vendored OpenSSL will be larger **Mitigation**: Already using
+`--release` flag. Consider adding strip step:
 
 ```yaml
 - name: Strip binaries (Unix)
@@ -116,16 +119,19 @@ The `cross` tool includes OpenSSL in its Docker images. The workflow may work as
 ## Test Plan
 
 ### Phase 1: Test workflow as-is
+
 1. Push a test tag: `git tag v0.7.0-test && git push origin v0.7.0-test`
 2. Monitor GitHub Actions
 3. Check if all builds succeed
 
 ### Phase 2: If musl build fails with OpenSSL errors
+
 1. Implement Option 1 (vendored OpenSSL for musl)
 2. Push another test tag: `git tag v0.7.0-test2 && git push origin v0.7.0-test2`
 3. Verify success
 
 ### Phase 3: Validate binaries
+
 Download and test each binary:
 
 ```bash
@@ -144,7 +150,9 @@ Download and test each binary:
 
 ## Conclusion
 
-The workflow is well-designed and should work with high probability. The main risk is OpenSSL linking on musl, but this has known solutions. All other dependencies are either pure Rust or download pre-built binaries.
+The workflow is well-designed and should work with high probability. The main risk is OpenSSL
+linking on musl, but this has known solutions. All other dependencies are either pure Rust or
+download pre-built binaries.
 
 **Confidence Level**: 85% success without changes, 99% with OpenSSL vendoring
 

.github/RELEASE.md

@@ -5,6 +5,7 @@ This document explains how to use the automated release build system for Torc.
 ## Overview
 
 The release workflow builds binaries for:
+
 - **macOS**: Apple Silicon (aarch64)
 - **Linux**:
   - x86_64-musl (static, works on all distros)
@@ -14,26 +15,32 @@ The release workflow builds binaries for:
 ## Binaries Produced
 
 Each platform build produces three binaries:
+
 1. `torc` - Unified CLI with all features
 2. `torc-server` - Standalone server
 3. `torc-slurm-job-runner` - Slurm job runner
 
 ## Triggering a Release
 
 ### Automatic (Recommended)
+
 Push a version tag to trigger the build:
+
 ```bash
 git tag v0.7.0
 git push origin v0.7.0
 ```
 
 This will:
+
 1. Build binaries for all platforms
 2. Create a draft GitHub release
 3. Upload all binaries to the release
 
 ### Manual
+
 You can also trigger builds manually from the GitHub Actions UI:
+
 1. Go to Actions → "Build Release Binaries"
 2. Click "Run workflow"
 3. Optionally specify a tag name
@@ -45,18 +52,21 @@ Manual builds create artifacts but don't create a GitHub release.
 ### Which Linux binary should users download?
 
 **For maximum compatibility (recommended for most users):**
+
 - Use `torc-x86_64-unknown-linux-musl.tar.gz`
 - This is a fully static binary that works on any Linux distro
 - No external dependencies required
 
 **For better performance on modern systems:**
+
 - Use `torc-x86_64-unknown-linux-gnu.tar.gz`
 - Built on Ubuntu 20.04, compatible with glibc 2.31+
 - Works on Ubuntu 20.04+, Debian 11+, RHEL 8+, etc.
 
 ## Adding More Platforms
 
 ### Additional Linux Versions
+
 To support older distros, add entries to the matrix in `.github/workflows/release.yml`:
 
 ```yaml
@@ -67,6 +77,7 @@ To support older distros, add entries to the matrix in `.github/workflows/releas
 ```
 
 ### Intel macOS
+
 To also build for Intel Macs, add:
 
 ```yaml
@@ -77,6 +88,7 @@ To also build for Intel Macs, add:
 ```
 
 ### ARM64 Linux
+
 For ARM64 servers (like AWS Graviton), add:
 
 ```yaml
@@ -89,17 +101,23 @@ For ARM64 servers (like AWS Graviton), add:
 ## Troubleshooting
 
 ### Build fails with OpenSSL errors on Windows
+
 The workflow installs OpenSSL via vcpkg. If it fails:
+
 1. Check that vcpkg is available on the runner
 2. Verify the OpenSSL environment variables are set correctly
 
 ### Build fails with musl linking errors
+
 If you see linker errors with musl:
+
 1. Ensure `cross` is being used (set `use_cross: true`)
 2. Or ensure musl-tools are installed for native builds
 
 ### Binary size is too large
+
 Release binaries include debug symbols. To reduce size:
+
 1. Add strip step to workflow after build:
    ```yaml
    - name: Strip binaries (Unix)
@@ -135,13 +153,17 @@ Expand-Archive torc-x86_64-pc-windows-msvc.zip
 ## Automation Tips
 
 ### Auto-publish releases
+
 To automatically publish releases (instead of drafts), change in release.yml:
+
 ```yaml
 draft: false  # Change from true
 ```
 
 ### Build on every push
+
 To build binaries on every push (for testing), add to `on:` section:
+
 ```yaml
 on:
   push:
@@ -150,7 +172,9 @@ on:
 ```
 
 ### Notification on failure
+
 Add a notification step at the end of the build job:
+
 ```yaml
 - name: Notify on failure
   if: failure()

.github/TESTING_WORKFLOW.md

@@ -15,8 +15,9 @@ git push origin v0.7.0-test1
 Go to: https://github.com/NREL/torc/actions/workflows/release.yml
 
 You should see 4 build jobs running in parallel:
+
 - ✅ Build aarch64-apple-darwin (macOS Apple Silicon)
-- ⚠️  Build x86_64-unknown-linux-musl (Linux static)
+- ⚠️ Build x86_64-unknown-linux-musl (Linux static)
 - ✅ Build x86_64-unknown-linux-gnu (Linux glibc)
 - ✅ Build x86_64-pc-windows-msvc (Windows)
 
@@ -40,6 +41,7 @@ git push origin v0.7.0-test2
 ```
 
 **If kaleido download fails:**
+
 - This is usually a transient network issue
 - Re-run the failed job from GitHub Actions UI
 
@@ -62,13 +64,15 @@ tar xzf torc-aarch64-apple-darwin.tar.gz
 ### 5. Test on different Linux distributions
 
 **Test musl binary on Alpine:**
+
 ```bash
 docker run -it --rm -v $(pwd):/workspace alpine:latest sh
 cd /workspace
 ./torc --version
 ```
 
 **Test glibc binary on Ubuntu 20.04:**
+
 ```bash
 docker run -it --rm -v $(pwd):/workspace ubuntu:20.04 sh
 cd /workspace
@@ -77,6 +81,7 @@ apt-get update && apt-get install -y ca-certificates
 ```
 
 **Test glibc binary on Ubuntu 24.04:**
+
 ```bash
 docker run -it --rm -v $(pwd):/workspace ubuntu:24.04 sh
 cd /workspace
@@ -106,12 +111,14 @@ git push origin v0.7.0
 GitHub Actions has a 6-hour job limit. Our builds should complete in ~10-30 minutes per platform.
 
 If timing out:
+
 - Check if dependencies are being cached (cache hit logs)
 - Consider removing less important targets
 
 ### Wrong binaries in release
 
 Check the glob patterns in `create-release` job:
+
 ```yaml
 files: |
   artifacts/torc-aarch64-apple-darwin/*.tar.gz
@@ -129,6 +136,7 @@ files: |
 ### Testing without creating a release
 
 Use manual trigger:
+
 1. Go to Actions → "Build Release Binaries" → "Run workflow"
 2. Leave tag name empty or use "test"
 3. Artifacts will be created but no release
@@ -152,15 +160,18 @@ git push origin --delete v0.7.0-test1 v0.7.0-test2
 ### Faster builds with better caching
 
 The workflow already caches:
+
 - `~/.cargo/registry` (downloaded crates)
 - `~/.cargo/git` (git dependencies)
 - `target` (compiled artifacts)
 
 Cache is keyed by:
+
 - OS and target triple
 - Cargo.lock hash
 
 To bust cache (if needed):
+
 - Update Cargo.lock: `cargo update`
 - Or manually delete cache from GitHub UI
 
@@ -180,6 +191,7 @@ matrix:
 ## Next Steps
 
 After successful test:
+
 1. Document installation instructions for users
 2. Add checksums (sha256) to release notes
 3. Consider setting up a release schedule