|
| 1 | +# TASK-140: Add parallel test runner for faster CI |
| 2 | + |
| 3 | +## Priority: P2 (SECONDARY) |
| 4 | + |
| 5 | +## Summary |
| 6 | + |
| 7 | +The full test suite in `zig/harness/` takes a long time to run sequentially. |
| 8 | +Implement a parallel test runner to speed up CI and local development feedback loops. |
| 9 | + |
| 10 | +## Current State |
| 11 | + |
| 12 | +- ~40 test scripts in `zig/harness/test-*.sh` |
| 13 | +- Each test is independent (uses temp DBs, no shared state) |
| 14 | +- Sequential run takes several minutes |
| 15 | +- Each individual test takes 2-10 seconds |
| 16 | + |
| 17 | +## Files to Modify |
| 18 | + |
| 19 | +- `zig/harness/run-all-tests.sh` (new file) |
| 20 | +- `zig/Makefile` (add `test-parallel` target) |
| 21 | + |
| 22 | +## Acceptance Criteria |
| 23 | + |
| 24 | +1. [ ] Create `run-all-tests.sh` that runs tests in parallel |
| 25 | +2. [ ] Use GNU parallel or xargs -P for parallelism |
| 26 | +3. [ ] Collect and summarize results at end |
| 27 | +4. [ ] Exit non-zero if any test fails |
| 28 | +5. [ ] Support `--jobs N` or `PARALLEL_JOBS` env var |
| 29 | +6. [ ] Default to `nproc` or 4 concurrent jobs |
| 30 | +7. [ ] Show progress (e.g., "12/40 tests completed") |
| 31 | +8. [ ] Preserve individual test output for debugging failures |
| 32 | + |
| 33 | +## Implementation Options |
| 34 | + |
| 35 | +### Option A: GNU Parallel |
| 36 | + |
| 37 | +```bash |
| 38 | +#!/usr/bin/env bash |
| 39 | +find . -name 'test-*.sh' -type f | \ |
| 40 | + parallel --jobs ${PARALLEL_JOBS:-$(nproc)} \ |
| 41 | + --bar \ |
| 42 | + --results .tmp/test-results/{/} \ |
| 43 | + bash {} |
| 44 | +``` |
| 45 | + |
| 46 | +### Option B: xargs |
| 47 | + |
| 48 | +```bash |
| 49 | +#!/usr/bin/env bash |
| 50 | +find . -name 'test-*.sh' -type f | \ |
| 51 | + xargs -P ${PARALLEL_JOBS:-4} -I{} bash -c ' |
| 52 | + echo "Running {}" |
| 53 | + bash {} > .tmp/test-output-$(basename {}).log 2>&1 |
| 54 | + echo "Exit code: $?" |
| 55 | + ' |
| 56 | +``` |
| 57 | + |
| 58 | +### Option C: Makefile with dependencies |
| 59 | + |
| 60 | +```makefile |
| 61 | +TESTS := $(wildcard test-*.sh) |
| 62 | +test-parallel: $(TESTS:.sh=.result) |
| 63 | +%.result: %.sh |
| 64 | + @bash $< > .tmp/$@.log 2>&1 && touch $@ || (cat .tmp/$@.log; exit 1) |
| 65 | +``` |
| 66 | + |
| 67 | +## Considerations |
| 68 | + |
| 69 | +1. **Build contention**: Multiple tests may try to rebuild Zig extension concurrently |
| 70 | + - Solution: Add `zig build` as prerequisite before parallel tests |
| 71 | + |
| 72 | +2. **Temp file collisions**: Tests create temp DBs in `.tmp/` |
| 73 | + - Current: Uses `mktemp` so no collision |
| 74 | + - Verify each test uses unique temp files |
| 75 | + |
| 76 | +3. **Output interleaving**: Parallel output can be confusing |
| 77 | + - Solution: Capture per-test output, show only on failure |
| 78 | + |
| 79 | +4. **Resource limits**: Too many concurrent SQLite processes |
| 80 | + - Solution: Default to 4, allow override |
| 81 | + |
| 82 | +## Expected Speedup |
| 83 | + |
| 84 | +With 40 tests at ~5 seconds each: |
| 85 | +- Sequential: ~200 seconds (3+ minutes) |
| 86 | +- Parallel (4 jobs): ~50 seconds |
| 87 | +- Parallel (8 jobs): ~25 seconds |
| 88 | + |
| 89 | +## Parent Docs / Cross-links |
| 90 | + |
| 91 | +- Gap Analysis: `research/zig-cr/97-test-gap-analysis.md` |
| 92 | +- Current runner: `zig/harness/test-parity.sh` (sequential) |
| 93 | + |
| 94 | +## Progress Log |
| 95 | + |
| 96 | +- 2024-12-20: Created task card |
| 97 | + |
| 98 | +## Completion Notes |
| 99 | + |
| 100 | +(To be filled upon completion) |
0 commit comments