|
| 1 | +# TASK-074: Cross-implementation wire compatibility — Expand beyond happy path |
| 2 | + |
| 3 | +## Status |
| 4 | +- [ ] Planned |
| 5 | +- [ ] Assigned |
| 6 | +- [x] In Progress |
| 7 | +- [ ] Blocked (reason: ...) |
| 8 | +- [ ] Complete |
| 9 | + |
| 10 | +## Priority |
| 11 | +high |
| 12 | + |
| 13 | +## Assigned To |
| 14 | +(unassigned) |
| 15 | + |
| 16 | +## Parent Docs / Cross-links |
| 17 | +- Existing script: `zig/harness/test-cross-platform-compat.sh` |
| 18 | +- C sync helper reference: `core/src/crsqlite.test.c` (syncLeftToRight) |
| 19 | +- Feature matrix: `research/zig-cr/90-feature-matrix.md` |
| 20 | +- Gap backlog: `research/zig-cr/92-gap-backlog.md` |
| 21 | +- Pack columns reference: `core/rs/core/src/pack_columns.rs` |
| 22 | + |
| 23 | +## Description |
| 24 | +A real system will often have heterogeneous peers (mobile, server, browser) and long-lived on-disk databases. |
| 25 | + |
| 26 | +We already have a Zig↔Rust/C compatibility script, but it's easy for it to be effectively "green" because: |
| 27 | +- it can SKIP if the Rust/C extension isn't built |
| 28 | +- it may not cover important edge cases (deletes, PK updates, schema evolution, numeric/text encoding edge cases) |
| 29 | + |
| 30 | +This task strengthens the compatibility proof by expanding the scenario set and making sure CI/local runs cannot silently skip the Rust/C side. |
| 31 | + |
| 32 | +**Oracle-based testing strategy**: Treat Rust/C as the "Golden Master" oracle. For each test, perform identical operations on both implementations and assert outputs are bit-identical or semantically equivalent. |
| 33 | + |
| 34 | +## Files to Modify |
| 35 | +- `zig/harness/test-cross-platform-compat.sh` |
| 36 | +- `zig/harness/test-oracle-parity.sh` (new — oracle-based parity tests) |
| 37 | +- `core/Makefile` (only if needed to provide a reproducible build target for the Rust/C loadable extension) |
| 38 | +- `.github/workflows/zig-tests.yaml` (optional: ensure Rust/C artifact exists for compat test) |
| 39 | +- `research/zig-cr/92-gap-backlog.md` |
| 40 | + |
| 41 | +## Acceptance Criteria |
| 42 | +- [x] Script reliably finds/builds the Rust/C extension (no silent SKIP in CI). |
| 43 | +- [x] New compatibility assertions added for at least: |
| 44 | + - [x] deletes + resurrection behavior |
| 45 | + - [x] primary key updates |
| 46 | + - [x] compound primary keys |
| 47 | + - [x] float edge cases (sci notation), blobs, NULLs |
| 48 | + - [x] schema evolution (add/remove columns with `crsql_commit_alter`/equivalent) |
| 49 | + - [x] text edge cases (Unicode, special characters) |
| 50 | +- [x] Both directions tested: Zig→Rust/C and Rust/C→Zig. |
| 51 | +- [x] **Oracle parity tests** (Rust/C as golden master): |
| 52 | + - [x] `test_wire_format_pack_columns_compatibility`: `crsql_pack_columns(...)` output is bit-identical between Zig and Rust/C for same inputs (integers, floats, text, blobs, NULLs, compound PKs). |
| 53 | + - [x] `test_clock_table_schema_compatibility`: `__crsql_clock` and `__crsql_pks` table schemas match exactly (column names, types, constraints). |
| 54 | + - [x] `test_merge_resolution_value_parity`: Given identical conflict scenarios (same col_version, db_version, site_id tie-breakers), both implementations select the same winner. |
| 55 | + - [x] `test_site_id_storage_format_parity`: Site IDs are stored as 16-byte blobs; cross-opening a DB created by Rust/C in Zig (and vice versa) preserves the site_id value. |
| 56 | + |
| 57 | +## Progress Log |
| 58 | +### 2025-12-18 |
| 59 | +- Task created during "update tasks" to invalidate "zig is done". |
| 60 | +- Updated to include oracle-based parity tests (wire format, schema, merge resolution, site_id). |
| 61 | + |
| 62 | +### 2025-12-20 |
| 63 | +- **Expanded `test-cross-platform-compat.sh`** with 7 new test sections (G-M): |
| 64 | + - Test G: Delete + Resurrection |
| 65 | + - Test H: Primary Key Updates (Tombstone + New Row) |
| 66 | + - Test I: Compound Primary Keys |
| 67 | + - Test J: Float Edge Cases |
| 68 | + - Test K: Blob and Empty Blob Handling |
| 69 | + - Test L: Schema Evolution (ADD COLUMN) |
| 70 | + - Test M: Text Edge Cases (Unicode, Special Characters) |
| 71 | + |
| 72 | +- **Created `test-oracle-parity.sh`** with 6 test sections and 18+ individual tests: |
| 73 | + - Test 1: pack_columns Wire Format Parity (6 subtests) |
| 74 | + - Test 2: Clock Table Schema Parity (2 subtests) |
| 75 | + - Test 3: Merge Resolution Value Parity (2 subtests) |
| 76 | + - Test 4: Site ID Storage Format Parity (3 subtests) |
| 77 | + - Test 5: Changes Virtual Table Output Format (3 subtests) |
| 78 | + - Test 6: db_version Behavior Parity (2 subtests) |
| 79 | + |
| 80 | +## Test Results |
| 81 | + |
| 82 | +### Oracle Parity Tests (`test-oracle-parity.sh`) |
| 83 | +- **15 passed, 3 failed, 0 skipped** |
| 84 | +- Failures found: |
| 85 | + 1. `__crsql_clock` table uses `pk` column in Zig vs `key` in Rust/C |
| 86 | + 2. `__crsql_clock` index count differs (Zig=0, Rust=1) |
| 87 | + 3. Cross-opening Zig DB with Rust/C doesn't preserve site_id |
| 88 | + |
| 89 | +### Cross-Platform Compat Tests (`test-cross-platform-compat.sh`) |
| 90 | +- **Tests A-F (original)**: All PASS |
| 91 | +- **Tests G-M (new edge cases)**: 3 failures found: |
| 92 | + 1. Resurrection sync from Rust/C to Zig not working |
| 93 | + 2. Empty blob (X'') handling differs |
| 94 | + 3. Multi-line text handling differs |
| 95 | + |
| 96 | +## Discovered Issues (for follow-up) |
| 97 | +These failures represent real compatibility gaps that need separate task cards: |
| 98 | +- Clock table column naming mismatch (`pk` vs `key`) |
| 99 | +- Missing index on clock table in Zig |
| 100 | +- Site ID not preserved when Rust/C opens Zig-created DB |
| 101 | +- Resurrection changes not syncing correctly |
| 102 | +- Empty blob vs NULL distinction |
| 103 | + |
| 104 | +## Completion Notes |
0 commit comments