Skip to content

Commit 02f2ccf

Browse files
subtleGradientsubtleGradient
committed
++
1 parent 45ad4c0 commit 02f2ccf

38 files changed

+4953
-334
lines changed

‎.tasks/active/TASK-094-alter-table-history-preservation.md‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
# TASK-094: Oracle Parity — ALTER TABLE preserves clock history
2+
3+
## Status
4+
- [ ] Planned
5+
- [ ] Assigned
6+
- [x] In Progress
7+
- [ ] Blocked (reason: ...)
8+
- [ ] Complete
9+
10+
## Priority
11+
high
12+
13+
## Assigned To
14+
(unassigned)
15+
16+
## Parent Docs / Cross-links
17+
- Rust alter logic: `core/rs/core/src/alter.rs`
18+
- Zig alter logic: `zig/src/schema_alter.zig`
19+
- Existing alter tests: `zig/harness/test-alter.sh`
20+
- Gap backlog: `research/zig-cr/92-gap-backlog.md`
21+
22+
## Description
23+
Verify that `crsql_begin_alter` / `crsql_commit_alter` preserves existing clock history and correctly backfills new columns.
24+
25+
This is an **oracle test**: Schema evolution is critical for long-lived databases. If Zig loses clock history during ALTER or fails to backfill new columns, data will be lost or sync will break.
26+
27+
## Files to Modify
28+
- `zig/harness/test-alter-parity.sh` (new or extend `test-alter.sh`)
29+
- `zig/harness/test-parity.sh` (wire into suite)
30+
- `research/zig-cr/92-gap-backlog.md`
31+
32+
## Acceptance Criteria
33+
- [x] Test creates CRR table, inserts data, records clock state.
34+
- [x] Test performs ALTER operations via `crsql_begin_alter`/`crsql_commit_alter`:
35+
1. ADD COLUMN (nullable)
36+
2. ADD COLUMN with DEFAULT
37+
3. DROP COLUMN
38+
4. ADD INDEX
39+
5. DROP INDEX
40+
- [x] After each ALTER:
41+
- Existing clock entries are preserved (same col_version, db_version for unchanged columns)
42+
- New columns have clock entries backfilled (col_version = 1, current db_version)
43+
- Dropped columns have clock entries removed
44+
- [ ] Clock state matches exactly between implementations. **DIVERGENCE FOUND - see notes**
45+
- [x] Test covers edge cases:
46+
- ALTER on empty table
47+
- ALTER on table with 1000+ rows (batching behavior)
48+
- Multiple ALTERs in sequence
49+
- ALTER that adds column then immediately updates it
50+
- [x] Test fails if clock history diverges.
51+
52+
## Progress Log
53+
### 2025-12-18
54+
- Task created from oracle-based parity test suite.
55+
56+
### 2025-12-20
57+
- Implemented comprehensive test suite in `zig/harness/test-alter-parity.sh`
58+
- Wired test into `zig/harness/test-parity.sh`
59+
- **Key Divergence Found**: Zig backfills clock entries for new columns, Rust does NOT
60+
61+
## Completion Notes
62+
63+
### Test Results: PASSED: 16, FAILED: 3
64+
65+
### Files Modified
66+
1. `zig/harness/test-alter-parity.sh` - Comprehensive ALTER TABLE parity test (rewritten)
67+
2. `zig/harness/test-parity.sh` - Wired in test-alter-parity.sh
68+
69+
### Test Command
70+
```bash
71+
cd /Users/tom/Developer/effect-native/cr-sqlite/zig/harness
72+
bash test-alter-parity.sh
73+
```
74+
75+
### Key Findings
76+
77+
#### Behavioral Divergences (Zig vs Rust/C Oracle)
78+
79+
1. **ADD COLUMN Backfill Behavior** (DIVERGENCE):
80+
- **Rust**: Does NOT create clock entries for new columns after ADD COLUMN
81+
- **Zig**: DOES backfill clock entries (col_version=1) for all existing rows
82+
- This is a fundamental design difference that affects sync behavior
83+
84+
2. **DROP COLUMN Clock Cleanup** (PASS):
85+
- Both implementations correctly remove clock entries for dropped columns
86+
87+
3. **INDEX Operations** (PASS):
88+
- ADD INDEX and DROP INDEX preserve clock state in both implementations
89+
90+
4. **Existing History Preservation** (PASS):
91+
- Both implementations preserve existing col_version/db_version during ALTER
92+
93+
5. **UPDATE After ADD COLUMN** (DIVERGENCE):
94+
- Rust: Creates clock entry only when column is updated (lazy)
95+
- Zig: Already has backfilled entry, UPDATE increments col_version
96+
97+
#### Clock Table Schema Difference
98+
- Rust uses `key` column for primary key
99+
- Zig uses `pk` column for primary key
100+
- Tests normalize this by aliasing `key AS pk` for comparison
101+
102+
### Implications
103+
104+
The backfill divergence means:
105+
- **Zig**: After ADD COLUMN, all rows appear "changed" in crsql_changes for the new column
106+
- **Rust**: After ADD COLUMN, new column only appears in crsql_changes when explicitly updated
107+
108+
This affects sync scenarios where:
109+
- A peer adds a column and syncs to another peer
110+
- Zig will send backfill entries for all rows
111+
- Rust will only send entries for rows that were explicitly updated
112+
113+
### Recommendation
114+
This divergence should be documented and a decision made on which behavior is correct:
115+
1. Zig's eager backfill ensures all peers have consistent NULL/DEFAULT values tracked
116+
2. Rust's lazy approach reduces sync traffic but may cause version inconsistencies
117+
118+
### Note on Local Extension
119+
The `lib/crsqlite.dylib` in the repo appears to have a bug with `crsql_commit_alter` ("failed compacting tables post alteration"). Tests use `nix run github:subtleGradient/sqlite-cr` which has a working cr-sqlite version.

‎.tasks/active/TASK-095-zig-test-pk-update-semantics.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# TASK-095: Zig test for PK UPDATE semantics
2+
3+
## Status
4+
- [ ] Planned
5+
- [ ] Assigned
6+
- [x] In Progress
7+
- [ ] Blocked (reason: ...)
8+
- [ ] Complete
9+
10+
## Priority
11+
medium
12+
13+
## Assigned To
14+
(unassigned)
15+
16+
## Parent Docs / Cross-links
17+
- Created by: `.tasks/active/TASK-073-compare-rust-zig-tests.md`
18+
- Rust reference: `core/rs/integration_check/src/t/pk_only_tables.rs` (modify_pkonly_row, junction_table)
19+
- Gap backlog: `research/zig-cr/92-gap-backlog.md`
20+
21+
## Description
22+
The Rust integration suite tests that UPDATE statements which modify primary key columns are correctly handled as delete + create operations. This is critical for sync correctness:
23+
24+
```sql
25+
-- Original row
26+
INSERT INTO foo (id, value) VALUES (1, 'abc');
27+
-- PK update should generate:
28+
-- 1. DELETE tombstone for pk=1
29+
-- 2. INSERT for pk=2 with all values
30+
UPDATE foo SET id = 2 WHERE id = 1;
31+
```
32+
33+
The Zig harness does not currently test this behavior. This can cause sync divergence if implementations differ.
34+
35+
## Files to Modify
36+
- `zig/harness/test-pk-update.sh` (new file)
37+
- `zig/harness/test-parity.sh` (add test runner call)
38+
39+
## Acceptance Criteria
40+
- [x] New test script `zig/harness/test-pk-update.sh` exists
41+
- [x] Tests cover:
42+
- Single-column PK update (simple table)
43+
- Compound PK update (junction table - one column changed)
44+
- Compound PK update (all columns changed)
45+
- PK update on table with non-PK columns
46+
- [ ] Test verifies both base table state AND clock table entries
47+
- [x] Reproducible command: `bash zig/harness/test-pk-update.sh`
48+
49+
## Reproducible Command
50+
```bash
51+
cd /Users/tom/Developer/effect-native/cr-sqlite
52+
bash zig/harness/test-pk-update.sh
53+
```
54+
55+
## Progress Log
56+
### 2025-12-18
57+
- Task created from TASK-073 coverage analysis
58+
59+
## Completion Notes
60+
### 2025-12-20
61+
- Implemented `zig/harness/test-pk-update.sh` and wired it into `zig/harness/test-parity.sh`.
62+
- Repro command: `bash zig/harness/test-pk-update.sh`
63+
- Result: test currently FAILS against Zig extension because PK UPDATE does not emit tombstones (`cid='-1'`) / clock entries for the old PK.
64+
- Follow-up filed: `.tasks/triage/TASK-103-zig-pk-update-must-emit-tombstone-and-insert.md`

‎.tasks/active/TASK-098-zig-ondisk-db-tests.md‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# TASK-098: Zig on-disk DB persistence tests
2+
3+
## Status
4+
- [ ] Planned
5+
- [ ] Assigned
6+
- [ ] In Progress
7+
- [ ] Blocked (reason: ...)
8+
- [x] Complete
9+
10+
## Priority
11+
high
12+
13+
## Assigned To
14+
(unassigned)
15+
16+
## Parent Docs / Cross-links
17+
- Created by: `.tasks/active/TASK-073-compare-rust-zig-tests.md`
18+
- Gap backlog: `research/zig-cr/92-gap-backlog.md`
19+
20+
## Description
21+
**Critical real-system gap**: All Zig harness tests currently use `:memory:` databases. This means:
22+
23+
1. No persistence testing - data survives only within session
24+
2. No WAL mode testing - in-memory doesn't use WAL
25+
3. No crash recovery testing - can't simulate process restart
26+
4. No file locking testing - in-memory has no file contention
27+
28+
Real applications use on-disk databases. The Zig extension must be validated against file-backed SQLite.
29+
30+
## Files to Modify
31+
- `zig/harness/test-persistence.sh` (new file)
32+
- `zig/harness/test-parity.sh` (add test runner call)
33+
34+
## Acceptance Criteria
35+
- [x] New test script `zig/harness/test-persistence.sh` exists
36+
- [x] Tests use temp directory with actual `.sqlite` files
37+
- [x] Tests cover:
38+
- Create CRR, insert data, close DB, reopen, verify data
39+
- Create CRR, insert data, close DB, reopen, query `crsql_changes`
40+
- Verify `crsql_site_id()` persists across sessions
41+
- Verify `crsql_db_version()` persists across sessions
42+
- WAL mode: insert in one session, read uncommitted in another (optional)
43+
- [x] Cleanup: test removes temp files on exit
44+
- [x] Reproducible command: `bash zig/harness/test-persistence.sh`
45+
46+
## Reproducible Command
47+
```bash
48+
cd /Users/tom/Developer/effect-native/cr-sqlite
49+
bash zig/harness/test-persistence.sh
50+
```
51+
52+
## Progress Log
53+
### 2025-12-18
54+
- Task created from TASK-073 coverage analysis
55+
- Identified as HIGH priority real-system gap
56+
57+
### 2025-12-20
58+
- Implemented `zig/harness/test-persistence.sh` with 7 test suites, 12 assertions
59+
- Wired into `zig/harness/test-parity.sh` test runner
60+
- All tests pass
61+
62+
## Completion Notes
63+
**Date**: 2025-12-20
64+
65+
**Files created/modified**:
66+
- `zig/harness/test-persistence.sh` (new, 297 lines)
67+
- `zig/harness/test-parity.sh` (updated to include persistence tests)
68+
69+
**Test coverage (12 assertions)**:
70+
1. Data persistence across sessions (2 tests)
71+
2. crsql_changes persistence (2 tests)
72+
3. crsql_site_id() persistence across sessions (2 tests)
73+
4. crsql_db_version() persistence across sessions (2 tests)
74+
5. Clock table persistence (1 test)
75+
6. CRR schema persistence (1 test)
76+
7. WAL mode persistence (2 tests)
77+
78+
**Test execution**:
79+
```
80+
bash zig/harness/test-persistence.sh
81+
```
82+
83+
**Results**: 12 PASSED, 0 FAILED, 0 SKIPPED
84+
85+
**Key implementation details**:
86+
- Uses `.tmp/persistence-test-$$` for temp files (per AGENTS.md)
87+
- Proper cleanup with `trap 'rm -rf "$TMPDIR"' EXIT`
88+
- Tests real file-backed databases, not `:memory:`
89+
- Validates site_id and db_version survive DB close/reopen
90+
- Tests CRR triggers remain active after session restart

‎.tasks/backlog/TASK-072-zig-sync-bit-per-connection.md‎

Lines changed: 0 additions & 53 deletions
This file was deleted.

‎.tasks/backlog/TASK-073-compare-rust-zig-tests.md‎

Lines changed: 0 additions & 60 deletions
This file was deleted.

0 commit comments

Comments
 (0)