Prohibit limit_resources: false in Slurm mode and validate execution_config fields (#228)

* Omit --exact from srun when limit_resources is false

When limit_resources=false, srun was still passed --exact, which causes
it to default --cpus-per-task to 1. This silently restricted
multi-threaded jobs to a single CPU core via cgroups, causing significant
slowdowns (45%+ observed) compared to direct execution mode.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* SQLite synchronous and increased cache_size

* Increase max record transfer size from 10,000 to 100,000

Centralizes the limit as MAX_RECORD_TRANSFER_COUNT in lib.rs and updates
all server pagination, client batch creation, and API defaults to use it.
Also increases the default max request body size to 200 MiB to accommodate
larger payloads.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Prohibit limit_resources: false in Slurm mode

Slurm mode always requires resource limits for correct srun behavior.
Setting limit_resources: false with mode: slurm now returns a validation
error directing users to use direct mode instead. Removes the
limit_resources=false code paths from srun argument building and updates
documentation and tests accordingly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Validate execution_config fields against effective mode

Reject mode-incompatible fields at workflow creation time: direct-only
fields (termination_signal, sigterm_lead_seconds, oom_exit_code) error
in slurm mode, and slurm-only fields (srun_termination_signal,
enable_cpu_bind) error in direct mode. Auto mode infers from
slurm_schedulers presence. Restructures docs and struct comments to
group fields by shared/direct/slurm.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Author: daniel-thom

api/openapi.yaml

@@ -3051,7 +3051,7 @@ paths:
         name: limit
         required: false
         schema:
-          default: 10000
+          default: 100000
           type: integer
         style: form
       responses:
@@ -4169,7 +4169,7 @@ paths:
         name: limit
         required: false
         schema:
-          default: 10000
+          default: 100000
           type: integer
         style: form
       responses:
@@ -6388,7 +6388,7 @@ paths:
           name: limit
           required: false
           schema:
-            default: 10000
+            default: 100000
             type: integer
           style: form
       responses:

docs/src/core/concepts/execution-modes.md

@@ -169,24 +169,18 @@ Key flags:
 
 ### Resource Enforcement in Slurm Mode
 
-When `limit_resources: true`:
+Slurm mode always passes `--cpus-per-task`, `--mem`, and `--gpus` to srun. Slurm's cgroups enforce
+these limits: jobs exceeding memory are killed by Slurm with exit code 137.
 
-- `--cpus-per-task`, `--mem`, and `--gpus` are passed to srun
-- Slurm's cgroups enforce these limits
-- Jobs exceeding memory are killed by Slurm with exit code 137
-
-When `limit_resources: false`:
-
-- CPU and memory flags are omitted
-- Jobs can use any available resources in the allocation
-- GPU flags are still passed (required for GPU access)
+> **Note**: `limit_resources: false` is not supported in Slurm mode. If you need to run jobs without
+> resource enforcement inside a Slurm allocation, use `mode: direct` instead. See
+> [Disabling Resource Limits](#disabling-resource-limits) below.
 
 ### Slurm Mode Configuration
 
 ```yaml
 execution_config:
   mode: slurm
-  limit_resources: true            # Pass resource limits to srun
   srun_termination_signal: TERM@120  # Send SIGTERM 120s before step timeout
   sigkill_headroom_seconds: 180    # End steps 3 minutes before allocation ends
   enable_cpu_bind: false           # Set to true to enable Slurm CPU binding
@@ -215,26 +209,28 @@ This ensures:
 
 ## Disabling Resource Limits
 
-Set `limit_resources: false` to disable resource enforcement:
+Set `limit_resources: false` to disable resource enforcement in direct mode:
 
 ```yaml
 execution_config:
-  mode: direct  # or slurm
+  mode: direct
   limit_resources: false
 ```
 
-Effects:
+This is useful when exploring resource requirements for new jobs or during local development. Jobs
+can use any available system resources without being killed for exceeding their declared limits.
 
-| Feature                | limit_resources: true     | limit_resources: false  |
-| ---------------------- | ------------------------- | ----------------------- |
-| Memory limits (direct) | OOM detection and SIGKILL | No enforcement          |
-| Memory limits (slurm)  | srun --mem passed         | --mem omitted           |
-| CPU limits (slurm)     | srun --cpus-per-task      | --cpus-per-task omitted |
-| GPU allocation         | Always passed to srun     | Always passed to srun   |
-| Timeout termination    | Enforced                  | Enforced                |
+Effects in direct mode:
 
-Note: GPU allocation is always requested regardless of `limit_resources` because jobs need explicit
-GPU access in Slurm.
+| Feature             | limit_resources: true     | limit_resources: false |
+| ------------------- | ------------------------- | ---------------------- |
+| Memory limits       | OOM detection and SIGKILL | No enforcement         |
+| Timeout termination | Enforced                  | Enforced               |
+
+> **Important**: `limit_resources: false` is only supported in direct mode. Setting it with
+> `mode: slurm` will produce an error at workflow creation time. Slurm mode relies on `srun` with
+> `--exact`, `--cpus-per-task`, and `--mem` for correct concurrent job execution — omitting these
+> flags causes jobs to run sequentially instead of in parallel.
 
 ## Exit Codes
 
@@ -263,7 +259,6 @@ execution_config:
 ```yaml
 execution_config:
   mode: auto  # Use slurm inside allocations
-  limit_resources: true
   srun_termination_signal: TERM@120
   sigkill_headroom_seconds: 300
 ```

docs/src/core/reference/cli.md

@@ -450,7 +450,7 @@ List workflows
 
 - `-u`, `--user <USER>` — User to filter by (defaults to USER environment variable)
 - `--all-users` — List workflows for all users (overrides --user)
-- `-l`, `--limit <LIMIT>` — Maximum number of workflows to return. Default: `10000`
+- `-l`, `--limit <LIMIT>` — Maximum number of workflows to return. Default: `100000`
 - `--offset <OFFSET>` — Offset for pagination (0-based). Default: `0`
 - `--sort-by <SORT_BY>` — Field to sort by
 - `--reverse-sort` — Reverse sort order
@@ -849,7 +849,7 @@ List compute nodes for a workflow
 
 ###### **Options:**
 
-- `-l`, `--limit <LIMIT>` — Maximum number of compute nodes to return. Default: `10000`
+- `-l`, `--limit <LIMIT>` — Maximum number of compute nodes to return. Default: `100000`
 - `-o`, `--offset <OFFSET>` — Offset for pagination (0-based). Default: `0`
 - `-s`, `--sort-by <SORT_BY>` — Field to sort by
 - `-r`, `--reverse-sort` — Reverse sort order. Default: `false`
@@ -898,7 +898,7 @@ List files
 ###### **Options:**
 
 - `--produced-by-job-id <PRODUCED_BY_JOB_ID>` — Filter by job ID that produced the files
-- `-l`, `--limit <LIMIT>` — Maximum number of files to return. Default: `10000`
+- `-l`, `--limit <LIMIT>` — Maximum number of files to return. Default: `100000`
 - `--offset <OFFSET>` — Offset for pagination (0-based). Default: `0`
 - `--sort-by <SORT_BY>` — Field to sort by
 - `--reverse-sort` — Reverse sort order
@@ -1030,7 +1030,7 @@ List jobs
 
 - `-s`, `--status <STATUS>` — Filter by job status
 - `--upstream-job-id <UPSTREAM_JOB_ID>` — Filter by upstream job ID (jobs that depend on this job)
-- `-l`, `--limit <LIMIT>` — Maximum number of jobs to return. Default: `10000`
+- `-l`, `--limit <LIMIT>` — Maximum number of jobs to return. Default: `100000`
 - `--offset <OFFSET>` — Offset for pagination (0-based). Default: `0`
 - `--sort-by <SORT_BY>` — Field to sort by
 - `--reverse-sort` — Reverse sort order
@@ -1120,7 +1120,7 @@ List job-to-job dependencies for a workflow
 
 ###### **Options:**
 
-- `-l`, `--limit <LIMIT>` — Maximum number of dependencies to return. Default: `10000`
+- `-l`, `--limit <LIMIT>` — Maximum number of dependencies to return. Default: `100000`
 - `--offset <OFFSET>` — Offset for pagination (0-based). Default: `0`
 
 ## `torc job-dependencies job-file`
@@ -1135,7 +1135,7 @@ List job-file relationships for a workflow
 
 ###### **Options:**
 
-- `-l`, `--limit <LIMIT>` — Maximum number of relationships to return. Default: `10000`
+- `-l`, `--limit <LIMIT>` — Maximum number of relationships to return. Default: `100000`
 - `--offset <OFFSET>` — Offset for pagination (0-based). Default: `0`
 
 ## `torc job-dependencies job-user-data`
@@ -1150,7 +1150,7 @@ List job-user_data relationships for a workflow
 
 ###### **Options:**
 
-- `-l`, `--limit <LIMIT>` — Maximum number of relationships to return. Default: `10000`
+- `-l`, `--limit <LIMIT>` — Maximum number of relationships to return. Default: `100000`
 - `--offset <OFFSET>` — Offset for pagination (0-based). Default: `0`
 
 ## `torc resource-requirements`
@@ -1200,7 +1200,7 @@ List resource requirements
 
 ###### **Options:**
 
-- `-l`, `--limit <LIMIT>` — Maximum number of resource requirements to return. Default: `10000`
+- `-l`, `--limit <LIMIT>` — Maximum number of resource requirements to return. Default: `100000`
 - `--offset <OFFSET>` — Offset for pagination (0-based). Default: `0`
 - `--sort-by <SORT_BY>` — Field to sort by
 - `--reverse-sort` — Reverse sort order
@@ -1285,7 +1285,7 @@ List events for a workflow
 ###### **Options:**
 
 - `-c`, `--category <CATEGORY>` — Filter events by category
-- `-l`, `--limit <LIMIT>` — Maximum number of events to return. Default: `10000`
+- `-l`, `--limit <LIMIT>` — Maximum number of events to return. Default: `100000`
 - `-o`, `--offset <OFFSET>` — Offset for pagination (0-based). Default: `0`
 - `-s`, `--sort-by <SORT_BY>` — Field to sort by
 - `-r`, `--reverse-sort` — Reverse sort order. Default: `false`
@@ -1357,7 +1357,7 @@ List results
 - `--failed` — Show only failed jobs (non-zero return code)
 - `-s`, `--status <STATUS>` — Filter by job status (uninitialized, blocked, canceled, terminated,
   done, ready, scheduled, running, pending, disabled)
-- `-l`, `--limit <LIMIT>` — Maximum number of results to return. Default: `10000`
+- `-l`, `--limit <LIMIT>` — Maximum number of results to return. Default: `100000`
 - `--offset <OFFSET>` — Offset for pagination (0-based). Default: `0`
 - `--sort-by <SORT_BY>` — Field to sort by
 - `--reverse-sort` — Reverse sort order
@@ -1574,7 +1574,7 @@ Show the current Slurm configs in the database
 
 ###### **Options:**
 
-- `-l`, `--limit <LIMIT>` — Maximum number of configs to return. Default: `10000`
+- `-l`, `--limit <LIMIT>` — Maximum number of configs to return. Default: `100000`
 - `--offset <OFFSET>` — Offset for pagination (0-based). Default: `0`
 
 ## `torc slurm get`
@@ -1930,7 +1930,7 @@ List scheduled compute nodes for a workflow
 
 ###### **Options:**
 
-- `-l`, `--limit <LIMIT>` — Maximum number of scheduled compute nodes to return. Default: `10000`
+- `-l`, `--limit <LIMIT>` — Maximum number of scheduled compute nodes to return. Default: `100000`
 - `-o`, `--offset <OFFSET>` — Offset for pagination (0-based). Default: `0`
 - `-s`, `--sort-by <SORT_BY>` — Field to sort by
 - `-r`, `--reverse-sort` — Reverse sort order. Default: `false`

docs/src/core/reference/workflow-spec.md

@@ -173,21 +173,41 @@ Defines a Slurm HPC job scheduler configuration.
 
 ## ExecutionConfig
 
-Controls how jobs are executed and terminated. Supports three modes for different execution
-environments.
-
-| Name                       | Type                        | Default     | Description                                                   |
-| -------------------------- | --------------------------- | ----------- | ------------------------------------------------------------- |
-| `mode`                     | string                      | `"auto"`    | Execution mode: `"direct"`, `"slurm"`, or `"auto"`            |
-| `limit_resources`          | boolean                     | `true`      | Enforce memory/CPU limits                                     |
-| `termination_signal`       | string                      | `"SIGTERM"` | Signal to send before SIGKILL (direct mode)                   |
-| `sigterm_lead_seconds`     | integer                     | `30`        | Seconds before SIGKILL to send termination signal             |
-| `sigkill_headroom_seconds` | integer                     | `60`        | Seconds before end_time for SIGKILL or srun --time adjustment |
-| `timeout_exit_code`        | integer                     | `152`       | Exit code for timed-out jobs (matches Slurm TIMEOUT)          |
-| `oom_exit_code`            | integer                     | `137`       | Exit code for OOM-killed jobs (128 + SIGKILL)                 |
-| `srun_termination_signal`  | string                      | none        | Slurm signal spec for `srun --signal=<value>`                 |
-| `enable_cpu_bind`          | boolean                     | `false`     | Allow Slurm CPU binding                                       |
-| `stdio`                    | [StdioConfig](#stdioconfig) | see below   | Workflow-level default for stdout/stderr capture              |
+Controls how jobs are executed and terminated. Fields are grouped by which execution mode they apply
+to. Setting a field that doesn't match the effective mode produces a validation error at workflow
+creation time.
+
+### Shared fields (both modes)
+
+| Name                       | Type                        | Default   | Description                                            |
+| -------------------------- | --------------------------- | --------- | ------------------------------------------------------ |
+| `mode`                     | string                      | `"auto"`  | Execution mode: `"direct"`, `"slurm"`, or `"auto"`     |
+| `sigkill_headroom_seconds` | integer                     | `60`      | Seconds before end_time for SIGKILL or srun --time     |
+| `timeout_exit_code`        | integer                     | `152`     | Exit code for timed-out jobs (matches Slurm TIMEOUT)   |
+| `staggered_start`          | boolean                     | `true`    | Stagger job runner startup to mitigate thundering herd |
+| `stdio`                    | [StdioConfig](#stdioconfig) | see below | Workflow-level default for stdout/stderr capture       |
+
+### Direct mode fields
+
+These fields only apply when the effective mode is `direct`. Setting them with `mode: slurm` (or
+`mode: auto` with `slurm_schedulers`) produces a validation error.
+
+| Name                   | Type    | Default     | Description                                           |
+| ---------------------- | ------- | ----------- | ----------------------------------------------------- |
+| `limit_resources`      | boolean | `true`      | Monitor memory/CPU and kill jobs that exceed limits   |
+| `termination_signal`   | string  | `"SIGTERM"` | Signal to send before SIGKILL for graceful shutdown   |
+| `sigterm_lead_seconds` | integer | `30`        | Seconds before SIGKILL to send the termination signal |
+| `oom_exit_code`        | integer | `137`       | Exit code for OOM-killed jobs (128 + SIGKILL)         |
+
+### Slurm mode fields
+
+These fields only apply when the effective mode is `slurm`. Setting them with `mode: direct` (or
+`mode: auto` without `slurm_schedulers`) produces a validation error.
+
+| Name                      | Type    | Default | Description                             |
+| ------------------------- | ------- | ------- | --------------------------------------- |
+| `srun_termination_signal` | string  | none    | Signal spec for `srun --signal=<value>` |
+| `enable_cpu_bind`         | boolean | `false` | Allow Slurm CPU binding (`--cpu-bind`)  |
 
 ### StdioConfig
 

docs/src/core/workflows/workflow-formats.md

@@ -250,7 +250,7 @@ modes:
 ```yaml
 execution_config:
   mode: direct # Options: direct, slurm, auto
-  limit_resources: true # Enforce memory/CPU limits (default: true)
+  limit_resources: true # Enforce memory limits in direct mode (default: true)
 
   # Direct mode settings
   termination_signal: SIGTERM # Signal before SIGKILL (default: SIGTERM)
@@ -293,7 +293,7 @@ execution_config {
 | Field                      | Type   | Default   | Description                                       |
 | -------------------------- | ------ | --------- | ------------------------------------------------- |
 | `mode`                     | string | `auto`    | Execution mode: `direct`, `slurm`, or `auto`      |
-| `limit_resources`          | bool   | `true`    | Enforce memory/CPU limits                         |
+| `limit_resources`          | bool   | `true`    | Enforce memory limits in direct mode only         |
 | `termination_signal`       | string | `SIGTERM` | Signal to send before SIGKILL (direct mode)       |
 | `sigterm_lead_seconds`     | int    | `30`      | Seconds before SIGKILL to send termination signal |
 | `sigkill_headroom_seconds` | int    | `60`      | Seconds before end_time to send SIGKILL           |
@@ -358,21 +358,24 @@ limit_resources: true
 # New style - use execution_config
 execution_config:
   mode: slurm # Replaces use_srun: true
-  limit_resources: true
   srun_termination_signal: "TERM@120"
   sigkill_headroom_seconds: 180 # New: controls srun --time headroom
 ```
 
 **Migration mapping:**
 
-| Old Field                 | New Field in execution_config |
-| ------------------------- | ----------------------------- |
-| `use_srun: true`          | `mode: slurm`                 |
-| `use_srun: false`         | `mode: direct`                |
-| (not set)                 | `mode: auto` (default)        |
-| `limit_resources`         | `limit_resources`             |
-| `srun_termination_signal` | `srun_termination_signal`     |
-| `enable_cpu_bind`         | `enable_cpu_bind`             |
+| Old Field                 | New Field in execution_config                |
+| ------------------------- | -------------------------------------------- |
+| `use_srun: true`          | `mode: slurm`                                |
+| `use_srun: false`         | `mode: direct`                               |
+| (not set)                 | `mode: auto` (default)                       |
+| `limit_resources: false`  | `mode: direct` with `limit_resources: false` |
+| `srun_termination_signal` | `srun_termination_signal`                    |
+| `enable_cpu_bind`         | `enable_cpu_bind`                            |
+
+> **Note**: `limit_resources: false` is only supported with `mode: direct`. If you previously used
+> `limit_resources: false` with srun, switch to `mode: direct` to get the same behavior (jobs run
+> without resource enforcement).
 
 ## Common Features Across All Formats