fix(zig): use lazy materialize semantics for ALTER ADD COLUMN (TASK-101)

subtleGradient · subtleGradient · commit 30877c555eb9 · 2025-12-20T18:08:14.000-05:00
- Removed backfillNewColumns() call from crsqlCommitAlterFunc
- Clock entries are now only created when column is explicitly written
- Matches Rust/C oracle behavior exactly
- test-alter-parity.sh: 19/19 pass
diff --git a/.tasks/DELEGATE_WORK_HANDOFF.md b/.tasks/DELEGATE_WORK_HANDOFF.md
@@ -68,6 +68,105 @@ Artifacts:
 
 ---
 
+## Round 2025-12-20 (50) — Implement lazy ALTER ADD COLUMN semantics (1 task)
+
+**Tasks executed**
+- `.tasks/done/TASK-101-impl-alter-add-column-no-backfill.md`
+
+**Commits**
+- (pending commit)
+
+**Environment**
+- OS: darwin (macOS ARM64)
+- Tooling: nix, zig (via nix), bash
+
+**Commands run (exact)**
+```bash
+bash zig/harness/test-alter-parity.sh
+```
+
+**Outputs (paste)**
+
+<details>
+<summary>TASK-101: test-alter-parity.sh (19/19 pass)</summary>
+
+```text
+╔═══════════════════════════════════════════════════════════════════════╗
+║       ALTER TABLE Parity Test (Zig vs Rust/C Oracle)                  ║
+╚═══════════════════════════════════════════════════════════════════════╝
+
+Rust/C: using sqlite-cr (nix run github:subtleGradient/sqlite-cr)
+Zig extension: /Users/tom/Developer/effect-native/cr-sqlite/zig/zig-out/lib/libcrsqlite.dylib
+
+Test 1: ADD COLUMN (nullable)
+  PASS: Pre-alter state - clock states match
+  PASS: Post-ADD COLUMN (nullable) - clock states match
+  PASS: After UPDATE on new column - clock states match
+
+Test 2: ADD COLUMN with DEFAULT
+  PASS: Pre-alter state - clock states match
+  PASS: Post-ADD COLUMN with DEFAULT - clock states match
+
+Test 3: DROP COLUMN
+  PASS: Pre-DROP state - clock states match
+  PASS: Post-DROP COLUMN - clock states match
+  PASS: Dropped column clock entries removed
+
+Test 4: ADD INDEX / DROP INDEX
+  PASS: Pre-INDEX state - clock states match
+  PASS: Post-ADD INDEX - clock states match
+  PASS: Post-DROP INDEX - clock states match
+
+Test 5: ALTER on empty table
+  PASS: Empty table ALTER handled (both have 0 clock entries)
+
+Test 6: ALTER on table with 1000+ rows
+  Rust clock entries: 1000
+  Zig clock entries: 1000
+  PASS: 1000-row ALTER clock count matches
+
+Test 7: Multiple ALTERs in sequence
+  PASS: Sequential ALTERs - dropped column removed from both
+
+Test 8: ADD COLUMN then immediately UPDATE
+  PASS: UPDATE on new column worked in both
+  Clock entries for 'newcol': Rust=1, Zig=1
+  PASS: Both have clock entries for new column after UPDATE
+
+Test 9: Existing clock history preserved
+  PASS: Rust/C preserved existing clock entries
+  PASS: Zig preserved existing clock entries
+
+Test 10: New column backfill behavior
+  Backfill clock entries: Rust=0, Zig=0
+  INFO: Rust does NOT backfill clock entries for new columns
+  PASS: Backfill behavior documented
+
+╔═══════════════════════════════════════════════════════════════════════╗
+║                    ALTER PARITY TEST SUMMARY                          ║
+╠═══════════════════════════════════════════════════════════════════════╣
+║  PASSED:  19                                                         ║
+║  FAILED:  0                                                          ║
+╚═══════════════════════════════════════════════════════════════════════╝
+
+All ALTER parity tests PASSED
+```
+
+**Change made**: Removed `backfillNewColumns()` call from `crsqlCommitAlterFunc` in `zig/src/schema_alter.zig`.
+
+Zig now uses **LAZY MATERIALIZE** semantics for ALTER ADD COLUMN, matching the Rust/C oracle behavior exactly.
+</details>
+
+**Reproduction steps (clean checkout)**
+1. `git clone <repo> && cd cr-sqlite`
+2. `bash zig/harness/test-alter-parity.sh` — verify 19/19 pass
+
+**Known gaps / unverified claims**
+- No coverage captured
+- CI integration not verified this round (local runs only)
+
+---
+
 ## Round 2025-12-20 (49) — Fix merge pk/rowid confusion + ALTER semantics decision (3 tasks)
 
 **Tasks executed**
diff --git a/.tasks/backlog/TASK-101-impl-alter-add-column-no-backfill.md b/.tasks/backlog/TASK-101-impl-alter-add-column-no-backfill.md
diff --git a/.tasks/done/TASK-101-impl-alter-add-column-no-backfill.md b/.tasks/done/TASK-101-impl-alter-add-column-no-backfill.md
@@ -0,0 +1,71 @@
+# TASK-101: Align Zig ALTER ADD COLUMN clocks with oracle (no eager backfill)
+
+## Status
+- [x] Planned
+- [x] Assigned
+- [x] In Progress
+- [ ] Blocked (reason: ...)
+- [x] Complete
+
+## Priority
+high
+
+## Assigned To
+(completed)
+
+## Parent Docs / Cross-links
+- Oracle parity test: `zig/harness/test-alter-parity.sh`
+- Zig alter implementation: `zig/src/schema_alter.zig`
+- Rust/C alter implementation (reference): `core/rs/core/src/alter.rs`
+- sqlite-cr wrapper: `nix run github:subtleGradient/sqlite-cr -- ...`
+- Origin task: `.tasks/active/TASK-094-alter-table-history-preservation.md`
+- Decision task: `.tasks/backlog/TASK-100-decide-alter-new-column-clock-semantics.md`
+
+## Description
+After TASK-094, Zig diverged from the oracle on `ALTER TABLE ... ADD COLUMN`:
+
+- Zig eagerly backfilled `__crsql_clock` entries for the new column.
+- Oracle (sqlite-cr / Rust/C) does not create clock rows for new columns until the column is explicitly written.
+
+This task implements the **oracle-matching behavior** in Zig (LAZY MATERIALIZE semantics), and the parity test is now fully green.
+
+## Files to Modify
+- [x] `zig/src/schema_alter.zig`
+- [x] `zig/harness/test-alter-parity.sh` (no changes needed - already correct)
+- [ ] `zig/harness/test-parity.sh` (no changes needed)
+
+## Acceptance Criteria
+- [x] `zig/harness/test-alter-parity.sh` passes with **0 failures**.
+- [x] After `ADD COLUMN` (nullable) with existing rows:
+  - [x] Zig does **not** create `__crsql_clock` rows for the new column.
+  - [x] Existing column clock rows are unchanged.
+- [x] After `ADD COLUMN ... DEFAULT ...` with existing rows:
+  - [x] Zig does **not** create `__crsql_clock` rows for the new column.
+  - [x] Later explicit `UPDATE` to the new column creates a clock row with `col_version = 1`.
+- [x] After sequential alters (add column then add column then drop column):
+  - [x] Zig clock column set matches oracle when columns are never written.
+- [x] Large-table case (1000+ rows) does not backfill clock rows.
+
+## Progress Log
+### 2025-12-20
+- Evidence from `zig/harness/test-alter-parity.sh` shows Zig eager-backfill causes:
+  - extra clock rows after `ADD COLUMN`
+  - different `col_version` after first UPDATE to new column
+
+### 2025-12-20 (completion)
+- Removed `backfillNewColumns()` call from `crsqlCommitAlterFunc` in `zig/src/schema_alter.zig`
+- Updated docstring to document LAZY MATERIALIZE semantics
+- Rebuilt Zig extension
+- All 19 parity tests now pass with 0 failures
+
+## Completion Notes
+### 2025-12-20
+
+**Change made**: Removed the call to `backfillNewColumns()` from `crsqlCommitAlterFunc` in `zig/src/schema_alter.zig`.
+
+**Rationale**: Zig now uses LAZY MATERIALIZE semantics for ALTER ADD COLUMN, matching the Rust/C oracle behavior. Clock entries are only created when a column is explicitly written (via UPDATE or INSERT), not eagerly when the column is added.
+
+**Test results**: `bash zig/harness/test-alter-parity.sh` - 19 PASSED, 0 FAILED
+
+**Files changed**:
+- `zig/src/schema_alter.zig` - Removed backfill call, updated docstring
diff --git a/research/zig-cr/92-gap-backlog.md b/research/zig-cr/92-gap-backlog.md
@@ -1,6 +1,6 @@
 # 92-gap-backlog
 
-> Last updated: 2025-12-20 (Round 49 — TASK-119, TASK-100 complete)
+> Last updated: 2025-12-20 (Round 50 — TASK-101 complete)
 
 ## Status
 
@@ -101,7 +101,9 @@ Goal: invalidate the hypothesis that "Zig is done" by expanding cross-implementa
   - [x] **TASK-100** — Decide ADD COLUMN clock semantics ✓ `.tasks/done/TASK-100-decide-alter-new-column-clock-semantics.md`
     - **Decision: LAZY MATERIALIZE** — Zig should NOT backfill clock entries on ADD COLUMN
     - Rationale: Clock entries represent writes, not schema changes; matches oracle; O(0) sync payload
-  - [ ] **TASK-101** — Implement lazy semantics in Zig → `.tasks/backlog/TASK-101-impl-alter-add-column-no-backfill.md`
+  - [x] **TASK-101** — Implement lazy semantics in Zig ✓ `.tasks/done/TASK-101-impl-alter-add-column-no-backfill.md`
+    - Removed `backfillNewColumns()` call from `crsqlCommitAlterFunc`
+    - Test: `zig/harness/test-alter-parity.sh` = 19/19 pass ✓
   - [x] **TASK-102** — Fix/replace local oracle dylib (ALTER) ✓ `.tasks/done/TASK-102-fix-oracle-crsqlite-dylib-alter.md`
 
 ### Realistic Sync Tests
diff --git a/zig/src/schema_alter.zig b/zig/src/schema_alter.zig
@@ -84,9 +84,12 @@ fn crsqlBeginAlterFunc(
 /// 1. Clean up clock entries for removed columns
 /// 2. Re-read table schema via PRAGMA table_info
 /// 3. Recreate triggers with new column list
-/// 4. Backfill clock entries for new columns with col_version=1
-/// 5. Release the savepoint
-/// 6. Return NULL on success, error on failure
+/// 4. Release the savepoint
+/// 5. Return NULL on success, error on failure
+///
+/// NOTE: Zig uses LAZY MATERIALIZE semantics for ADD COLUMN (matches oracle).
+/// Clock entries are NOT created eagerly; they are only created when the
+/// column is explicitly written (UPDATE/INSERT).
 fn crsqlCommitAlterFunc(
     pCtx: ?*api.sqlite3_context,
     argc: c_int,
@@ -147,12 +150,10 @@ fn crsqlCommitAlterFunc(
         return;
     };
 
-    // Step 3: Backfill clock entries for new columns
-    backfillNewColumns(db, table_name_ptr) catch {
-        api.result_error(pCtx, "crsql_commit_alter: failed to backfill new columns", -1);
-        _ = api.exec(db, "ROLLBACK TO alter_crr", null, null, null);
-        return;
-    };
+    // NOTE: Zig uses LAZY MATERIALIZE semantics for ADD COLUMN (matches oracle).
+    // Clock entries are NOT created eagerly on ADD COLUMN.
+    // They are only created when the column is explicitly written (UPDATE/INSERT).
+    // This matches Rust/C oracle behavior.
 
     // Release savepoint on success
     if (api.exec(db, "RELEASE alter_crr", null, null, null) != api.SQLITE_OK) {
