|
| 1 | +# TASK-146 — Fail fast and loud across all harnesses (no SKIP / no KNOWN_FAIL / no "acceptable" errors) |
| 2 | + |
| 3 | +## Goal |
| 4 | +Make every harness/test runner fail fast and loud if any assumption, dependency, expectation, or invariant is unmet or invalid. |
| 5 | + |
| 6 | +This turns the harness suite into an *invalidation engine*, not a best-effort report generator. |
| 7 | + |
| 8 | +## Status |
| 9 | +- State: triage |
| 10 | +- Priority: high |
| 11 | + |
| 12 | +## Context |
| 13 | +We discovered multiple places where gaps or missing dependencies can be silently ignored: |
| 14 | + |
| 15 | +1. Cross-open modification compatibility is currently treated as "known limitation" instead of a failing test: |
| 16 | + - `zig/harness/test-cross-open-parity.sh` reports `KNOWN_FAIL: 3` (XO-003 / XO-004 / XO-006) |
| 17 | + - These represent real compatibility gaps. |
| 18 | + |
| 19 | +2. Some harnesses SKIP entirely when dependencies are missing: |
| 20 | + - `zig/harness/test-cross-platform-compat.sh` can print `SKIP: Rust/C extension not found ...` and exit successfully. |
| 21 | + |
| 22 | +3. Some harnesses treat SQL errors as PASS: |
| 23 | + - `zig/harness/test-schema-evolution.sh` currently logs warnings like `WARN: SQL error (may be expected behavior)` and marks PASS. |
| 24 | + |
| 25 | +The desired posture is: |
| 26 | +- Missing dependency → FAIL with actionable instructions. |
| 27 | +- Known limitation / known fail → FAIL (tracked by task card; not ignored). |
| 28 | +- Error-is-acceptable → replace with explicit expected-reject assertion + post-error invariants. |
| 29 | + |
| 30 | +## Files to Modify |
| 31 | +(Exact scope to confirm during execution; keep tight.) |
| 32 | +- `zig/harness/test-parity.sh` (top-level aggregator: enforce nonzero exit on skipped subtests) |
| 33 | +- `zig/harness/test-cross-open-parity.sh` (remove `KNOWN_FAIL` pathway; make XO gaps real FAIL) |
| 34 | +- `zig/harness/test-cross-platform-compat.sh` (remove silent SKIP; fail with provisioning steps) |
| 35 | +- `zig/harness/test-schema-evolution.sh` (replace "acceptable error" PASS with explicit assertions) |
| 36 | +- Any `zig/harness/test-*.sh` that currently prints `SKIP:` / `SKIPPED:` / `WARN:` and still exits 0 |
| 37 | + |
| 38 | +## Acceptance Criteria |
| 39 | +1. Harness discipline |
| 40 | + - No harness script exits 0 when it printed any of: |
| 41 | + - `SKIP:` / `SKIPPED:` |
| 42 | + - `KNOWN_FAIL:` / `KNOWN LIMITATION:` |
| 43 | + - `WARN:` / "acceptable" errors |
| 44 | + - Instead, it exits nonzero with a clear, actionable message. |
| 45 | + |
| 46 | +2. Dependency discipline |
| 47 | + - If a harness requires an oracle extension, sqlite version feature (DROP COLUMN), or other capability: |
| 48 | + - It either provisions it automatically in a deterministic way, OR |
| 49 | + - It fails with a single canonical instruction (e.g. `./scripts/update-crsqlite-oracle.sh`). |
| 50 | + |
| 51 | +3. Invariant discipline |
| 52 | + - Any path that expects a failure (e.g. schema evolution conflicts) asserts: |
| 53 | + - the error class/message pattern |
| 54 | + - post-error invariants (no partial apply, expected db_version behavior, schema sanity, clock/changes sanity) |
| 55 | + |
| 56 | +4. Aggregator discipline |
| 57 | + - `make -C zig test-parity` (or whichever CI entrypoint is canonical) fails if any sub-suite fails or is skipped. |
| 58 | + |
| 59 | +## Parent Docs / Cross-links |
| 60 | +- `AGENTS.md` (Zig testing policy; no sqlite-cr for Zig) |
| 61 | +- `.tasks/triage/TASK-143-cross-open-modification-compat.md` |
| 62 | +- `.tasks/triage/TASK-144-cross-platform-compat-no-skip.md` |
| 63 | +- `.tasks/triage/TASK-145-schema-evolution-no-acceptable-errors.md` |
| 64 | +- `research/zig-cr/92-gap-backlog.md` |
| 65 | + |
| 66 | +## Progress Log |
| 67 | +- 2025-12-21: Created from "zero failures" requirement; codifies fail-fast harness policy. |
| 68 | + |
| 69 | +## Completion Notes |
| 70 | +(Empty until done.) |
0 commit comments