refactor: deduplicate hard-coded values in test scripts

Extract configuration values to the top of each script with environment
variable overrides.

run-tests.sh configuration (passed to test-integration.sh):
- RETRY_TIMES (default: 15) - cluster readiness retry attempts
- RETRY_DELAY (default: 2) - delay between retries in seconds
- JOB_RETRY_DELAY (default: 1) - delay for job state checks
- JOB_MAX_WAIT (default: 120) - maximum job wait time in seconds
- JOB_POLL_INTERVAL (default: 3) - job polling interval in seconds
- LOG_TAIL_LINES (default: 100) - lines to show in failure logs

test-integration.sh configuration (container-specific defaults):
- PLUGIN_LIBEXEC_DIR (default: /usr/libexec)
- SLURM_SYSCONFDIR (default: /etc/slurm)
- SLURM_JOB_SPOOL (default: /var/spool/slurm-jobs)
- SLURM_LOG_DIR (default: /var/log/slurm)
- SLURM_PARTITION (default: debug)

All timing parameters from run-tests.sh are passed to test-integration.sh
via docker exec -e flags for consistency.

Author: dennisklein

tests/runtime/README.md

@@ -9,7 +9,7 @@ The runtime tests:
 2. Build and install the slurm-singularity-exec plugin
 3. Verify plugin files are installed (library and configuration)
 4. Verify plugin CLI options appear in `sbatch --help` and `srun --help`
-5. Verify SPANK plugin loads when jobs run (check slurmd logs)
+5. Verify SPANK plugin loads when jobs run (check container logs)
 6. Submit and run a containerized test job (if singularity/apptainer is available)
 
 ## Docker Compose Architecture
@@ -28,7 +28,7 @@ The test infrastructure consists of three services orchestrated by Docker Compos
 
 | Volume | Containers | Access | Purpose |
 |--------|------------|--------|---------|
-| `../..` → `/workspace` | All | Read-only (`:z`) | Source code and build scripts |
+| `../..` → `/workspace` | All | Read-write (`:z`) | Source code and build scripts |
 | `plugin-build` | All | Read-write | Shared build artifacts (plugin binaries) |
 | `slurmctld-state` | slurmctld | Read-write | Controller state persistence |
 | `slurmd-state` | slurmd | Read-write | Daemon state persistence |
@@ -58,6 +58,41 @@ The test infrastructure consists of three services orchestrated by Docker Compos
 
 All services communicate via the `slurm-net` bridge network, allowing hostname-based service discovery.
 
+## Configuration
+
+The test infrastructure uses environment variables for configuration, allowing customization without modifying scripts:
+
+### Timing Configuration (set in run-tests.sh, passed to test-integration.sh)
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `RETRY_TIMES` | 15 | Number of retry attempts for cluster readiness |
+| `RETRY_DELAY` | 2 | Delay in seconds between retry attempts |
+| `JOB_RETRY_DELAY` | 1 | Delay in seconds between job state checks |
+| `JOB_MAX_WAIT` | 120 | Maximum wait time in seconds for job completion |
+| `JOB_POLL_INTERVAL` | 3 | Interval in seconds between job status polls |
+| `LOG_TAIL_LINES` | 100 | Number of log lines to show on failure |
+
+### Container Path Configuration (test-integration.sh only)
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `PLUGIN_LIBEXEC_DIR` | `/usr/libexec` | Plugin library directory |
+| `SLURM_SYSCONFDIR` | `/etc/slurm` | Slurm configuration directory |
+| `SLURM_JOB_SPOOL` | `/var/spool/slurm-jobs` | Job output spool directory |
+| `SLURM_LOG_DIR` | `/var/log/slurm` | Slurm log directory |
+| `SLURM_PARTITION` | `debug` | Default Slurm partition name |
+
+### Example: Custom Timing
+
+```bash
+# Faster retries for local testing
+RETRY_TIMES=5 RETRY_DELAY=1 ./run-tests.sh
+
+# Longer timeouts for slow environments
+JOB_MAX_WAIT=300 ./run-tests.sh
+```
+
 ## Quick Start
 
 ```bash

tests/runtime/run-tests.sh

@@ -7,6 +7,14 @@ set -e
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 cd "$SCRIPT_DIR"
 
+# Configuration - can be overridden via environment variables
+: "${RETRY_TIMES:=15}"
+: "${RETRY_DELAY:=2}"
+: "${JOB_RETRY_DELAY:=1}"
+: "${JOB_MAX_WAIT:=120}"
+: "${JOB_POLL_INTERVAL:=3}"
+: "${LOG_TAIL_LINES:=100}"
+
 echo "::group::Clean up previous containers"
 docker compose down -v 2>/dev/null || true
 echo "::endgroup::"
@@ -38,26 +46,30 @@ echo "::endgroup::"
 
 echo "::group::Wait for services"
 echo "Waiting for slurmctld to be ready..."
-# Give slurmctld up to 30 seconds to start (15 retries * 2 seconds)
-RETRIES=15
-DELAY=2
-for i in $(seq 1 $RETRIES); do
+# Give slurmctld up to RETRY_TIMES * RETRY_DELAY seconds to start
+for i in $(seq 1 $RETRY_TIMES); do
     if docker compose exec -T slurmctld scontrol ping >/dev/null 2>&1; then
-        echo "✓ Slurm cluster is ready (attempt $i/$RETRIES)"
+        echo "✓ Slurm cluster is ready (attempt $i/$RETRY_TIMES)"
         break
     fi
-    if [ $i -eq $RETRIES ]; then
-        echo "ERROR: slurmctld not ready after $((RETRIES * DELAY)) seconds"
+    if [ $i -eq $RETRY_TIMES ]; then
+        echo "ERROR: slurmctld not ready after $((RETRY_TIMES * RETRY_DELAY)) seconds"
         docker compose logs slurmctld
         exit 1
     fi
-    sleep $DELAY
+    sleep $RETRY_DELAY
 done
 echo "::endgroup::"
 
 echo "::group::Run integration tests"
 set +e  # Temporarily disable exit on error
-docker compose exec -T slurmctld /workspace/tests/runtime/test-integration.sh
+docker compose exec -T \
+    -e RETRY_TIMES="$RETRY_TIMES" \
+    -e RETRY_DELAY="$RETRY_DELAY" \
+    -e JOB_RETRY_DELAY="$JOB_RETRY_DELAY" \
+    -e JOB_MAX_WAIT="$JOB_MAX_WAIT" \
+    -e JOB_POLL_INTERVAL="$JOB_POLL_INTERVAL" \
+    slurmctld /workspace/tests/runtime/test-integration.sh
 TEST_EXIT_CODE=$?
 set -e  # Re-enable exit on error
 echo "::endgroup::"
@@ -76,12 +88,12 @@ fi
 
 # Show logs if tests failed
 if [ $TEST_EXIT_CODE -ne 0 ]; then
-    echo "::group::slurmctld logs (last 100 lines)"
-    docker compose logs --tail=100 slurmctld
+    echo "::group::slurmctld logs (last $LOG_TAIL_LINES lines)"
+    docker compose logs --tail="$LOG_TAIL_LINES" slurmctld
     echo "::endgroup::"
 
-    echo "::group::slurmd logs (last 100 lines)"
-    docker compose logs --tail=100 slurmd
+    echo "::group::slurmd logs (last $LOG_TAIL_LINES lines)"
+    docker compose logs --tail="$LOG_TAIL_LINES" slurmd
     echo "::endgroup::"
 
     echo "::group::Container status"

tests/runtime/test-integration.sh

@@ -4,22 +4,37 @@
 
 set -e
 
+# Configuration - can be overridden via environment variables
+: "${PLUGIN_LIBEXEC_DIR:=/usr/libexec}"
+: "${SLURM_SYSCONFDIR:=/etc/slurm}"
+: "${SLURM_JOB_SPOOL:=/var/spool/slurm-jobs}"
+: "${SLURM_LOG_DIR:=/var/log/slurm}"
+: "${SLURM_PARTITION:=debug}"
+: "${RETRY_TIMES:=30}"
+: "${RETRY_DELAY:=2}"
+: "${JOB_RETRY_DELAY:=1}"
+: "${JOB_MAX_WAIT:=120}"
+: "${JOB_POLL_INTERVAL:=3}"
+
+PLUGIN_SO="${PLUGIN_LIBEXEC_DIR}/slurm-singularity-exec.so"
+PLUGSTACK_CONF="${SLURM_SYSCONFDIR}/plugstack.conf.d/singularity-exec.conf"
+
 echo "=== Slurm Singularity Plugin Runtime Tests ==="
 echo
 
 # Test 1: Verify plugin files are installed
 echo "Test 1: Verifying plugin installation..."
-if [ -f "/usr/libexec/slurm-singularity-exec.so" ]; then
-    echo "✓ Found plugin library: /usr/libexec/slurm-singularity-exec.so"
+if [ -f "$PLUGIN_SO" ]; then
+    echo "✓ Found plugin library: $PLUGIN_SO"
 else
-    echo "✗ ERROR: Plugin library not found at /usr/libexec/slurm-singularity-exec.so"
+    echo "✗ ERROR: Plugin library not found at $PLUGIN_SO"
     exit 1
 fi
 
-if [ -f "/etc/slurm/plugstack.conf.d/singularity-exec.conf" ]; then
-    echo "✓ Found plugin config: /etc/slurm/plugstack.conf.d/singularity-exec.conf"
+if [ -f "$PLUGSTACK_CONF" ]; then
+    echo "✓ Found plugin config: $PLUGSTACK_CONF"
 else
-    echo "✗ ERROR: Plugin config not found at /etc/slurm/plugstack.conf.d/singularity-exec.conf"
+    echo "✗ ERROR: Plugin config not found at $PLUGSTACK_CONF"
     exit 1
 fi
 echo
@@ -84,7 +99,7 @@ echo
 if [ "$SKIP_CONTAINER_TEST" != "true" ]; then
     echo "Test 5: Creating a test container image..."
     # Use shared directory so container is accessible from both slurmctld and slurmd
-    TEST_CONTAINER="/var/spool/slurm-jobs/test-debian.sif"
+    TEST_CONTAINER="${SLURM_JOB_SPOOL}/test-debian.sif"
     if [ ! -f "$TEST_CONTAINER" ]; then
         # Create a minimal Debian container
         $SINGULARITY_CMD pull "$TEST_CONTAINER" docker://debian:stable-slim
@@ -102,23 +117,23 @@ fi
 
 # Test 6: Wait for Slurm to be ready
 echo "Test 6: Waiting for Slurm cluster to be ready..."
-if ! retry --times=30 --delay=2 -- scontrol ping >/dev/null 2>&1; then
+if ! retry --times="$RETRY_TIMES" --delay="$RETRY_DELAY" -- scontrol ping >/dev/null 2>&1; then
     echo "✗ ERROR: Slurm controller not responding"
     exit 1
 fi
 echo "✓ Slurm controller is responding"
 
 # Wait for node to be ready
-if ! retry --times=30 --delay=2 -- bash -c 'sinfo -h -o "%T" 2>/dev/null | grep -qE "idle|mixed|alloc"'; then
+if ! retry --times="$RETRY_TIMES" --delay="$RETRY_DELAY" -- bash -c 'sinfo -h -o "%T" 2>/dev/null | grep -qE "idle|mixed|alloc"'; then
     echo "✗ ERROR: No compute nodes are ready"
     echo "Showing sinfo output:"
     sinfo
     echo
     echo "Showing last 50 lines of slurmd logs:"
-    tail -50 /var/log/slurm/slurmd.log 2>/dev/null || echo "Could not read slurmd logs"
+    tail -50 "${SLURM_LOG_DIR}/slurmd.log" 2>/dev/null || echo "Could not read slurmd logs"
     echo
     echo "Showing last 50 lines of slurmctld logs:"
-    tail -50 /var/log/slurm/slurmctld.log 2>/dev/null || echo "Could not read slurmctld logs"
+    tail -50 "${SLURM_LOG_DIR}/slurmctld.log" 2>/dev/null || echo "Could not read slurmctld logs"
     exit 1
 fi
 echo "✓ Compute node is ready"
@@ -140,7 +155,7 @@ fi
 
 # Wait for job to complete
 echo "  Waiting for job $TEST_JOB_ID to complete..."
-retry --times=30 --delay=1 -- bash -c "scontrol show job $TEST_JOB_ID 2>/dev/null | grep -qE 'JobState=(COMPLETED|FAILED|CANCELLED)'" >/dev/null 2>&1
+retry --times="$RETRY_TIMES" --delay="$JOB_RETRY_DELAY" -- bash -c "scontrol show job $TEST_JOB_ID 2>/dev/null | grep -qE 'JobState=(COMPLETED|FAILED|CANCELLED)'" >/dev/null 2>&1
 
 JOB_STATE=$(scontrol show job "$TEST_JOB_ID" 2>/dev/null | grep "JobState" | awk '{print $1}' | cut -d= -f2)
 if [ "$JOB_STATE" = "COMPLETED" ]; then
@@ -158,24 +173,24 @@ echo
 if [ "$SKIP_CONTAINER_TEST" != "true" ]; then
     echo "Test 8: Submitting a containerized test job..."
 JOB_SCRIPT=$(mktemp /tmp/test_job.XXXXXX.sh)
-cat > "$JOB_SCRIPT" <<'JOBEOF'
+cat > "$JOB_SCRIPT" <<JOBEOF
 #!/bin/bash
 #SBATCH --job-name=test-singularity
-#SBATCH --output=/var/spool/slurm-jobs/test_job_%j.out
-#SBATCH --error=/var/spool/slurm-jobs/test_job_%j.err
-#SBATCH --partition=debug
+#SBATCH --output=${SLURM_JOB_SPOOL}/test_job_%j.out
+#SBATCH --error=${SLURM_JOB_SPOOL}/test_job_%j.err
+#SBATCH --partition=${SLURM_PARTITION}
 #SBATCH --time=00:01:00
 #SBATCH --nodes=1
 #SBATCH --ntasks=1
 
-echo "Job started at: $(date)"
-echo "Running on node: $(hostname)"
-echo "Job ID: $SLURM_JOB_ID"
+echo "Job started at: \$(date)"
+echo "Running on node: \$(hostname)"
+echo "Job ID: \$SLURM_JOB_ID"
 
 # Test command inside container
 cat /etc/os-release | grep -i pretty
 
-echo "Job completed at: $(date)"
+echo "Job completed at: \$(date)"
 JOBEOF
 
 chmod +x "$JOB_SCRIPT"
@@ -192,39 +207,38 @@ echo
 
 # Test 9: Wait for job to complete
 echo "Test 9: Waiting for job to complete..."
-max_wait=120
 waited=0
 while true; do
     JOB_STATE=$(scontrol show job "$JOB_ID" 2>/dev/null | grep "JobState=" | sed 's/.*JobState=\([^ ]*\).*/\1/')
-    
+
     if [ "$JOB_STATE" = "COMPLETED" ]; then
         echo "✓ Job completed successfully"
         break
     elif [ "$JOB_STATE" = "FAILED" ] || [ "$JOB_STATE" = "CANCELLED" ] || [ "$JOB_STATE" = "TIMEOUT" ]; then
         echo "✗ ERROR: Job failed with state: $JOB_STATE"
         scontrol show job "$JOB_ID"
         exit 1
-    elif [ $waited -ge $max_wait ]; then
-        echo "✗ ERROR: Job did not complete within ${max_wait}s"
+    elif [ $waited -ge $JOB_MAX_WAIT ]; then
+        echo "✗ ERROR: Job did not complete within ${JOB_MAX_WAIT}s"
         scontrol show job "$JOB_ID"
         scancel "$JOB_ID"
         exit 1
     fi
-    
-    echo "  Job state: $JOB_STATE (${waited}s/${max_wait}s)"
-    sleep 3
-    waited=$((waited + 3))
+
+    echo "  Job state: $JOB_STATE (${waited}s/${JOB_MAX_WAIT}s)"
+    sleep "$JOB_POLL_INTERVAL"
+    waited=$((waited + JOB_POLL_INTERVAL))
 done
 echo
 
 # Test 10: Check job output
 echo "Test 10: Checking job output..."
-JOB_OUTPUT="/var/spool/slurm-jobs/test_job_${JOB_ID}.out"
+JOB_OUTPUT="${SLURM_JOB_SPOOL}/test_job_${JOB_ID}.out"
 if [ -f "$JOB_OUTPUT" ]; then
     echo "Job output:"
     cat "$JOB_OUTPUT"
     echo
-    
+
     if grep -q "PRETTY_NAME" "$JOB_OUTPUT"; then
         echo "✓ Job produced expected output (found PRETTY_NAME)"
     else