Skip to content

Commit 45eeb12

Browse files
subtleGradientsubtleGradient
committed
docs: add test gap analysis and 10 task cards for parity coverage
Gap Analysis (97-test-gap-analysis.md): - Maps 157 ideal experiments to existing test infrastructure - Identifies 3 test script bugs (P0 blocking) - Documents 21 missing wire format tests (P1) - Documents 9 missing merge value comparison tests (P1) - Documents 14 missing boundary value tests (P2) - Documents 6 missing cross-open/stress tests (P2/P3) Task Cards Created: - TASK-130: Fix test-trigger-parity.sh column bug (P0) - TASK-131: Fix test-alter-parity.sh column bug (P0) - TASK-132: Wire format edge case parity tests (P1) - TASK-133: PK blob format edge case parity tests (P1) - TASK-134: Merge value comparison parity tests (P1) - TASK-135: Delete/resurrection parity tests (P1) - TASK-136: Cross-open modification parity tests (P2) - TASK-137: Boundary value edge case tests (P2) - TASK-138: Config isolation test (P2) - TASK-139: Stress/performance tests (P3) Recommended execution order: P0 first (unblock tests), then P1 (critical parity)
1 parent 7d65b4b commit 45eeb12

11 files changed

+860
-0
lines changed

‎.tasks/backlog/TASK-130-fix-trigger-parity-test-column-bug.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# TASK-130: Fix test-trigger-parity.sh column name bug
2+
3+
## Priority: P0 (BLOCKING)
4+
5+
## Summary
6+
7+
The `test-trigger-parity.sh` script queries the wrong column name for Zig clock tables,
8+
causing 15 false test failures. Both implementations now use `key` but the test queries `pk`.
9+
10+
## Files to Modify
11+
12+
- `zig/harness/test-trigger-parity.sh`
13+
14+
## Acceptance Criteria
15+
16+
1. [ ] Change `pk` to `key` in `dump_clock_zig` function (line ~98)
17+
2. [ ] Run test-trigger-parity.sh and verify all 15 tests pass
18+
3. [ ] If any tests fail after fix, those are REAL parity gaps (document them)
19+
20+
## Bug Details
21+
22+
**Location:** Lines 97-98
23+
24+
```bash
25+
# CURRENT (BROKEN)
26+
dump_clock_zig() {
27+
$SQLITE "$db" -cmd ".load $ZIG_EXT sqlite3_crsqlite_init" \
28+
"SELECT pk, col_name, col_version, db_version, seq FROM ${table}__crsql_clock..."
29+
}
30+
31+
# FIXED
32+
dump_clock_zig() {
33+
$SQLITE "$db" -cmd ".load $ZIG_EXT sqlite3_crsqlite_init" \
34+
"SELECT key, col_name, col_version, db_version, seq FROM ${table}__crsql_clock..."
35+
}
36+
```
37+
38+
## Evidence
39+
40+
Direct verification shows Zig DOES populate clock tables correctly:
41+
42+
```sql
43+
-- Zig
44+
CREATE TABLE foo (id INTEGER PRIMARY KEY NOT NULL, name TEXT);
45+
SELECT crsql_as_crr('foo');
46+
INSERT INTO foo VALUES (1, 'test');
47+
SELECT key, col_name, col_version, db_version, seq FROM foo__crsql_clock;
48+
-- Returns: 1|name|1|1|0
49+
```
50+
51+
Both implementations use the same schema:
52+
```sql
53+
CREATE TABLE IF NOT EXISTS "foo__crsql_clock" (
54+
"key" INTEGER NOT NULL, -- NOT "pk"
55+
"col_name" TEXT NOT NULL,
56+
...
57+
)
58+
```
59+
60+
## Experiments Unblocked
61+
62+
- TR-001 through TR-030 (all 19 trigger/clock experiments)
63+
64+
## Parent Docs / Cross-links
65+
66+
- Analysis: `research/zig-cr/97-test-gap-analysis.md`
67+
- Experiments: `research/zig-cr/96-ideal-parity-experiments.md`
68+
69+
## Progress Log
70+
71+
- 2024-12-20: Created task card
72+
73+
## Completion Notes
74+
75+
(To be filled upon completion)

‎.tasks/backlog/TASK-131-fix-alter-parity-test-column-bug.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# TASK-131: Fix test-alter-parity.sh column name bug
2+
3+
## Priority: P0 (BLOCKING)
4+
5+
## Summary
6+
7+
The `test-alter-parity.sh` script queries the wrong column name for Zig clock tables,
8+
causing 10 false test failures. Same bug as TASK-130.
9+
10+
## Files to Modify
11+
12+
- `zig/harness/test-alter-parity.sh`
13+
14+
## Acceptance Criteria
15+
16+
1. [ ] Find and fix all `pk` references in clock table queries
17+
2. [ ] Run test-alter-parity.sh and verify tests pass
18+
3. [ ] If any tests fail after fix, those are REAL parity gaps (document them)
19+
20+
## Bug Details
21+
22+
The test uses `pk` when querying Zig clock tables but both implementations use `key`.
23+
24+
## Experiments Unblocked
25+
26+
- AT-001 through AT-004 (ALTER TABLE experiments)
27+
28+
## Parent Docs / Cross-links
29+
30+
- Analysis: `research/zig-cr/97-test-gap-analysis.md`
31+
- Related: TASK-130 (same bug, different file)
32+
33+
## Progress Log
34+
35+
- 2024-12-20: Created task card
36+
37+
## Completion Notes
38+
39+
(To be filled upon completion)

‎.tasks/backlog/TASK-132-wire-format-edge-case-parity.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
# TASK-132: Add wire format edge case parity tests
2+
3+
## Priority: P1 (CRITICAL)
4+
5+
## Summary
6+
7+
Add oracle parity tests for wire format edge cases in `crsql_pack_columns`:
8+
- Empty string
9+
- Zero and negative integers
10+
- MIN/MAX INT64
11+
- MAX FLOAT
12+
- Unicode/emoji
13+
14+
## Files to Modify
15+
16+
- `zig/harness/test-oracle-parity.sh` (add new test section)
17+
OR
18+
- `zig/harness/test-wire-format-edge-cases.sh` (new file)
19+
20+
## Acceptance Criteria
21+
22+
1. [ ] Test WF-007: Empty string encoding matches
23+
2. [ ] Test WF-009: Zero encoding matches
24+
3. [ ] Test WF-010: Negative one encoding matches
25+
4. [ ] Test WF-011: MAX_INT64 (9223372036854775807) encoding matches
26+
5. [ ] Test WF-012: MIN_INT64 (-9223372036854775808) encoding matches
27+
6. [ ] Test WF-013: MAX_FLOAT encoding matches
28+
7. [ ] Test WF-014: Unicode/emoji encoding matches
29+
8. [ ] All tests run as part of CI
30+
31+
## Test Template
32+
33+
```bash
34+
# WF-007: Empty string
35+
RUST_RESULT=$(run_rust "SELECT hex(crsql_pack_columns(''));")
36+
ZIG_RESULT=$(run_zig "SELECT hex(crsql_pack_columns(''));")
37+
compare "$RUST_RESULT" "$ZIG_RESULT" "Empty string encoding"
38+
39+
# WF-011: MAX_INT64
40+
RUST_RESULT=$(run_rust "SELECT hex(crsql_pack_columns(9223372036854775807));")
41+
ZIG_RESULT=$(run_zig "SELECT hex(crsql_pack_columns(9223372036854775807));")
42+
compare "$RUST_RESULT" "$ZIG_RESULT" "MAX_INT64 encoding"
43+
44+
# etc.
45+
```
46+
47+
## Parent Docs / Cross-links
48+
49+
- Experiments: `research/zig-cr/96-ideal-parity-experiments.md` (WF-007 through WF-015)
50+
- Gap Analysis: `research/zig-cr/97-test-gap-analysis.md`
51+
52+
## Progress Log
53+
54+
- 2024-12-20: Created task card
55+
56+
## Completion Notes
57+
58+
(To be filled upon completion)

‎.tasks/backlog/TASK-133-pk-blob-format-edge-case-parity.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# TASK-133: Add PK blob format edge case parity tests
2+
3+
## Priority: P1 (CRITICAL)
4+
5+
## Summary
6+
7+
Add oracle parity tests for PK blob encoding in `crsql_changes` with non-integer PKs:
8+
- Text primary keys
9+
- Blob primary keys
10+
- Compound PKs with mixed types
11+
- Unicode text PKs
12+
13+
## Files to Modify
14+
15+
- `zig/harness/test-oracle-parity.sh` (add new test section)
16+
OR
17+
- `zig/harness/test-pk-blob-parity.sh` (new file)
18+
19+
## Acceptance Criteria
20+
21+
1. [ ] Test WF-021: Single text PK encoding matches
22+
2. [ ] Test WF-022: Single blob PK encoding matches
23+
3. [ ] Test WF-023: Compound PK (int, int) encoding matches
24+
4. [ ] Test WF-024: Compound PK (int, text) encoding matches
25+
5. [ ] Test WF-025: Compound PK (int, text, blob) encoding matches
26+
6. [ ] Test WF-026: Unicode text PK encoding matches
27+
7. [ ] All tests run as part of CI
28+
29+
## Test Template
30+
31+
```bash
32+
# WF-021: Text PK
33+
setup_rust "CREATE TABLE t(id TEXT PRIMARY KEY NOT NULL); SELECT crsql_as_crr('t');"
34+
setup_zig "CREATE TABLE t(id TEXT PRIMARY KEY NOT NULL); SELECT crsql_as_crr('t');"
35+
36+
run_rust "INSERT INTO t VALUES('hello');"
37+
run_zig "INSERT INTO t VALUES('hello');"
38+
39+
RUST_PK=$(run_rust "SELECT hex(pk) FROM crsql_changes WHERE [table]='t';")
40+
ZIG_PK=$(run_zig "SELECT hex(pk) FROM crsql_changes WHERE [table]='t';")
41+
compare "$RUST_PK" "$ZIG_PK" "Text PK encoding"
42+
43+
# WF-026: Unicode text PK
44+
run_rust "INSERT INTO t VALUES('🎉');"
45+
run_zig "INSERT INTO t VALUES('🎉');"
46+
# Compare pk blobs
47+
```
48+
49+
## Parent Docs / Cross-links
50+
51+
- Experiments: `research/zig-cr/96-ideal-parity-experiments.md` (WF-021 through WF-027)
52+
- Gap Analysis: `research/zig-cr/97-test-gap-analysis.md`
53+
54+
## Progress Log
55+
56+
- 2024-12-20: Created task card
57+
58+
## Completion Notes
59+
60+
(To be filled upon completion)

‎.tasks/backlog/TASK-134-merge-value-comparison-parity.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# TASK-134: Add merge value comparison parity tests
2+
3+
## Priority: P1 (CRITICAL)
4+
5+
## Summary
6+
7+
When `col_version` is equal between local and remote, the implementations must use
8+
the same value comparison algorithm to determine the winner. This tests that behavior
9+
for all SQLite types.
10+
11+
## Files to Modify
12+
13+
- `zig/harness/test-merge.sh` (add value comparison section)
14+
OR
15+
- `zig/harness/test-merge-value-parity.sh` (new file)
16+
17+
## Acceptance Criteria
18+
19+
1. [ ] Test MR-020: String comparison (lexicographic) - 'apple' vs 'banana'
20+
2. [ ] Test MR-021: Integer comparison - 100 vs 99
21+
3. [ ] Test MR-022: NULL vs value - document winner
22+
4. [ ] Test MR-023: Value vs NULL - document winner
23+
5. [ ] Test MR-024: Float comparison - 3.14 vs 3.15
24+
6. [ ] Test MR-025: Blob comparison - X'AA' vs X'BB'
25+
7. [ ] All tests verify Zig matches C/Rust winner selection
26+
27+
## Test Template
28+
29+
```bash
30+
# MR-020: String comparison
31+
# Setup: Insert 'apple', sync with 'banana' at same cv
32+
# Both should select 'banana' (lexicographically larger)
33+
34+
# 1. Create local row with 'apple'
35+
run_rust "INSERT INTO foo(id, name) VALUES(1, 'apple');"
36+
run_zig "INSERT INTO foo(id, name) VALUES(1, 'apple');"
37+
38+
# 2. Get local state
39+
LOCAL_CV=$(run_both "SELECT col_version FROM foo__crsql_clock WHERE col_name='name';")
40+
41+
# 3. Merge remote 'banana' with same cv
42+
REMOTE_SITE=$(gen_random_site_id)
43+
run_rust "INSERT INTO crsql_changes VALUES('foo', pk, 'name', 'banana', $LOCAL_CV, 99, $REMOTE_SITE, 1, 0);"
44+
run_zig "INSERT INTO crsql_changes VALUES('foo', pk, 'name', 'banana', $LOCAL_CV, 99, $REMOTE_SITE, 1, 0);"
45+
46+
# 4. Verify both selected 'banana'
47+
RUST_VALUE=$(run_rust "SELECT name FROM foo WHERE id=1;")
48+
ZIG_VALUE=$(run_zig "SELECT name FROM foo WHERE id=1;")
49+
compare "$RUST_VALUE" "$ZIG_VALUE" "String comparison winner"
50+
```
51+
52+
## Parent Docs / Cross-links
53+
54+
- Experiments: `research/zig-cr/96-ideal-parity-experiments.md` (MR-020 through MR-025)
55+
- Gap Analysis: `research/zig-cr/97-test-gap-analysis.md`
56+
57+
## Progress Log
58+
59+
- 2024-12-20: Created task card
60+
61+
## Completion Notes
62+
63+
(To be filled upon completion)

‎.tasks/backlog/TASK-135-delete-resurrection-parity.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# TASK-135: Add delete/resurrection parity tests
2+
3+
## Priority: P1 (CRITICAL)
4+
5+
## Summary
6+
7+
Test that the full row lifecycle (insert -> delete -> resurrect) produces identical
8+
causal length (cl) values in both implementations, and that cl comparison correctly
9+
determines winners.
10+
11+
## Files to Modify
12+
13+
- `zig/harness/test-resurrection.sh` (expand existing)
14+
OR
15+
- `zig/harness/test-cl-parity.sh` (new file)
16+
17+
## Acceptance Criteria
18+
19+
1. [ ] Test MR-041: Deleted row + insert merge (higher cl) -> Row resurrected
20+
2. [ ] Test MR-042: cl=1 (live) vs cl=2 (deleted) -> Deleted wins
21+
3. [ ] Test MR-043: cl=2 (deleted) vs cl=3 (resurrected) -> Resurrected wins
22+
4. [ ] Both implementations produce identical cl values through lifecycle
23+
5. [ ] Both implementations make identical winner decisions
24+
25+
## Test Template
26+
27+
```bash
28+
# MR-042: Live vs Deleted
29+
# Local: cl=1 (row exists), cv=5
30+
# Remote: cl=2 (row deleted), cv=1
31+
# Expected: Remote wins (cl=2 > cl=1), row deleted
32+
33+
# Setup local live row
34+
run_both "INSERT INTO foo VALUES(1, 'test');"
35+
# Verify local cl=1
36+
LOCAL_CL=$(run_both "SELECT cl FROM foo__crsql_clock WHERE col_name='-1' OR ...;")
37+
38+
# Merge delete from remote with cl=2
39+
run_both "INSERT INTO crsql_changes VALUES('foo', pk, '-1', NULL, 1, 99, remote_site, 2, 0);"
40+
41+
# Verify row deleted
42+
ROW_COUNT=$(run_both "SELECT COUNT(*) FROM foo WHERE id=1;")
43+
compare "0" "$ROW_COUNT" "Row should be deleted"
44+
45+
# MR-043: Deleted vs Resurrected
46+
# Continue from above state (local cl=2, deleted)
47+
# Merge resurrection with cl=3
48+
run_both "INSERT INTO crsql_changes VALUES('foo', pk, '-1', NULL, 1, 99, remote_site, 3, 0);"
49+
run_both "INSERT INTO crsql_changes VALUES('foo', pk, 'name', 'resurrected', 1, 99, remote_site, 3, 1);"
50+
51+
# Verify row exists with new value
52+
ROW_COUNT=$(run_both "SELECT COUNT(*) FROM foo WHERE id=1;")
53+
VALUE=$(run_both "SELECT name FROM foo WHERE id=1;")
54+
compare "1" "$ROW_COUNT" "Row should exist"
55+
compare "resurrected" "$VALUE" "Row should have new value"
56+
```
57+
58+
## Parent Docs / Cross-links
59+
60+
- Experiments: `research/zig-cr/96-ideal-parity-experiments.md` (MR-041 through MR-043)
61+
- Gap Analysis: `research/zig-cr/97-test-gap-analysis.md`
62+
- Existing: `zig/harness/test-resurrection.sh`
63+
64+
## Progress Log
65+
66+
- 2024-12-20: Created task card
67+
68+
## Completion Notes
69+
70+
(To be filled upon completion)

0 commit comments

Comments
 (0)