Skip to content

Commit 7d65b4b

Browse files
subtleGradientsubtleGradient
committed
docs: add ideal parity experiments checklist
Defines 157 experiments needed for complete parity verification: - Wire Format: 27 experiments (6 tested, 21 needed) - Clock Table Schema: 7 experiments (all tested) - db_version Timing: 13 experiments (11 tested) - rows_impacted: 10 experiments (8 tested) - Merge Resolution: 33 experiments (12 tested, 21 needed) - Trigger/Clock: 19 experiments (1 tested, 15 blocked by test bug) - ALTER Table: 6 experiments (2 tested, 4 blocked) - Fractional Index: 9 experiments (all tested) - Config API: 7 experiments (6 tested) - Edge Cases: 16 experiments (2 tested, 14 needed) - Cross-Open: 6 experiments (3 tested) - Stress: 4 experiments (1 tested) Current coverage: 43% (68/157) with 19 blocked by test bugs
1 parent d24741c commit 7d65b4b

File tree

1 file changed

+310
-0
lines changed

1 file changed

+310
-0
lines changed

‎research/zig-cr/96-ideal-parity-experiments.md‎

Lines changed: 310 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,310 @@
1+
# Ideal Parity Experiments Checklist
2+
3+
**Date:** 2024-12-20
4+
**Purpose:** Define the complete set of experiments needed to fully validate oracle parity
5+
6+
This document specifies what experiments SHOULD exist regardless of what tests currently exist.
7+
After defining the ideal state, we compare against existing tests to identify gaps.
8+
9+
---
10+
11+
## 1. Wire Format Experiments
12+
13+
### 1.1 crsql_pack_columns Encoding
14+
15+
Each type must produce byte-identical output:
16+
17+
| Experiment ID | Input | Expected Output (hex) | Status |
18+
|--------------|-------|----------------------|--------|
19+
| WF-001 | `crsql_pack_columns(42)` | `01092A` | TESTED |
20+
| WF-002 | `crsql_pack_columns('hello')` | `010B0568656C6C6F` | TESTED |
21+
| WF-003 | `crsql_pack_columns(X'DEADBEEF')` | `010C04DEADBEEF` | TESTED |
22+
| WF-004 | `crsql_pack_columns(NULL)` | `0105` | TESTED |
23+
| WF-005 | `crsql_pack_columns(3.14159)` | `0102400921F9F01B866E` | TESTED |
24+
| WF-006 | `crsql_pack_columns(42, 'hello', X'BEEF')` | `03092A0B0568656C6C6F0C02BEEF` | TESTED |
25+
| WF-007 | `crsql_pack_columns('')` (empty string) | TBD | NEEDS TEST |
26+
| WF-008 | `crsql_pack_columns(X'')` (empty blob) | TBD | TESTED (TASK-127-129) |
27+
| WF-009 | `crsql_pack_columns(0)` | TBD | NEEDS TEST |
28+
| WF-010 | `crsql_pack_columns(-1)` | TBD | NEEDS TEST |
29+
| WF-011 | `crsql_pack_columns(9223372036854775807)` (MAX_INT64) | TBD | NEEDS TEST |
30+
| WF-012 | `crsql_pack_columns(-9223372036854775808)` (MIN_INT64) | TBD | NEEDS TEST |
31+
| WF-013 | `crsql_pack_columns(1.7976931348623157e+308)` (MAX_FLOAT) | TBD | NEEDS TEST |
32+
| WF-014 | Unicode text: `crsql_pack_columns('🎉')` | TBD | NEEDS TEST |
33+
| WF-015 | Very long blob (1MB) | TBD | NEEDS TEST |
34+
35+
### 1.2 PK Blob Format in crsql_changes
36+
37+
| Experiment ID | Setup | Expected | Status |
38+
|--------------|-------|----------|--------|
39+
| WF-020 | Single integer PK | pk blob matches | TESTED |
40+
| WF-021 | Single text PK | pk blob matches | NEEDS TEST |
41+
| WF-022 | Single blob PK | pk blob matches | NEEDS TEST |
42+
| WF-023 | Compound PK (int, int) | pk blob matches | NEEDS TEST |
43+
| WF-024 | Compound PK (int, text) | pk blob matches | NEEDS TEST |
44+
| WF-025 | Compound PK (int, text, blob) | pk blob matches | NEEDS TEST |
45+
| WF-026 | Text PK with unicode | pk blob matches | NEEDS TEST |
46+
| WF-027 | Text PK with NULL bytes | pk blob matches | NEEDS TEST |
47+
48+
---
49+
50+
## 2. Clock Table Schema Experiments
51+
52+
| Experiment ID | Check | Expected | Status |
53+
|--------------|-------|----------|--------|
54+
| CT-001 | Column names | `key, col_name, col_version, db_version, site_id, seq` | TESTED |
55+
| CT-002 | Column types | INTEGER/TEXT per schema | TESTED |
56+
| CT-003 | Primary key | `(key, col_name)` | TESTED |
57+
| CT-004 | WITHOUT ROWID | Present | TESTED |
58+
| CT-005 | STRICT | Present | TESTED |
59+
| CT-006 | Index exists | `foo__crsql_clock_dbv_idx` on `db_version` | TESTED |
60+
| CT-007 | site_id default | `DEFAULT 0` | TESTED |
61+
62+
---
63+
64+
## 3. db_version Timing Experiments
65+
66+
| Experiment ID | Scenario | Expected | Status |
67+
|--------------|----------|----------|--------|
68+
| DV-001 | Initial state | db_version = 0 | TESTED |
69+
| DV-002 | After INSERT | db_version = 1 | TESTED |
70+
| DV-003 | After UPDATE | db_version = 2 | TESTED |
71+
| DV-004 | After DELETE | db_version = 3 | TESTED |
72+
| DV-005 | Multiple INSERTs in transaction | All get same db_version | TESTED |
73+
| DV-006 | No-op UPDATE | db_version advances | TESTED |
74+
| DV-007 | Winning merge | db_version advances | TESTED |
75+
| DV-008 | Losing merge | db_version does NOT advance | TESTED |
76+
| DV-009 | No-op merge (same value) | db_version does NOT advance | TESTED |
77+
| DV-010 | ROLLBACK | db_version unchanged | TESTED |
78+
| DV-011 | crsql_next_db_version() | Returns db_version + 1 | TESTED |
79+
| DV-012 | crsql_next_db_version(X) | Returns max(X+1, db_version+1) | NEEDS TEST |
80+
| DV-013 | Concurrent transactions (WAL) | Each gets correct version | NEEDS TEST |
81+
82+
---
83+
84+
## 4. rows_impacted Counter Experiments
85+
86+
| Experiment ID | Scenario | Expected | Status |
87+
|--------------|----------|----------|--------|
88+
| RI-001 | Single winning INSERT | 1 | TESTED |
89+
| RI-002 | Multiple winning INSERTs | Accumulates | TESTED |
90+
| RI-003 | COMMIT resets | 0 | TESTED |
91+
| RI-004 | ROLLBACK does NOT reset | Preserved | TESTED |
92+
| RI-005 | No-op merge (same value) | 0 | TESTED |
93+
| RI-006 | Losing merge (lower cv) | 0 | TESTED |
94+
| RI-007 | Winning merge (higher cv) | 1 | TESTED |
95+
| RI-008 | Delete via merge | 1 | TESTED |
96+
| RI-009 | Resurrection via merge | 1 | NEEDS TEST |
97+
| RI-010 | Partial batch (some win, some lose) | Count of winners | NEEDS TEST |
98+
99+
---
100+
101+
## 5. Merge Resolution Experiments
102+
103+
### 5.1 Causal Length (cl) Dominates
104+
105+
| Experiment ID | Local State | Remote State | Winner | Status |
106+
|--------------|-------------|--------------|--------|--------|
107+
| MR-001 | cl=1, cv=5 | cl=2, cv=1 | Remote (higher cl) | TESTED |
108+
| MR-002 | cl=2, cv=1 | cl=1, cv=5 | Local (higher cl) | TESTED |
109+
| MR-003 | cl=1, cv=1 | cl=1, cv=1 | Local (tie) | TESTED |
110+
111+
### 5.2 col_version Tiebreaker (when cl equal)
112+
113+
| Experiment ID | Local cv | Remote cv | Winner | Status |
114+
|--------------|----------|-----------|--------|--------|
115+
| MR-010 | 1 | 2 | Remote | TESTED |
116+
| MR-011 | 2 | 1 | Local | TESTED |
117+
| MR-012 | 5 | 5 | Value compare | TESTED |
118+
119+
### 5.3 Value Comparison Tiebreaker (when cv equal)
120+
121+
| Experiment ID | Local Value | Remote Value | Winner | Status |
122+
|--------------|-------------|--------------|--------|--------|
123+
| MR-020 | 'apple' | 'banana' | Remote (banana > apple) | NEEDS TEST |
124+
| MR-021 | 100 | 99 | Local (100 > 99) | NEEDS TEST |
125+
| MR-022 | NULL | 'value' | ? | NEEDS TEST |
126+
| MR-023 | 'value' | NULL | ? | NEEDS TEST |
127+
| MR-024 | 3.14 | 3.15 | Remote | NEEDS TEST |
128+
| MR-025 | X'AA' | X'BB' | Remote | NEEDS TEST |
129+
130+
### 5.4 site_id Tiebreaker (when value equal)
131+
132+
| Experiment ID | Scenario | Expected | Status |
133+
|--------------|----------|----------|--------|
134+
| MR-030 | Same value, lower site_id | Lower site_id wins | TESTED |
135+
| MR-031 | Same value, higher site_id | Lower site_id wins | TESTED |
136+
| MR-032 | merge-equal-values=0 | No-op (local wins) | TESTED |
137+
| MR-033 | merge-equal-values=1 | site_id tiebreaker used | TESTED |
138+
139+
### 5.5 Delete/Resurrection Semantics
140+
141+
| Experiment ID | Scenario | Expected | Status |
142+
|--------------|----------|----------|--------|
143+
| MR-040 | Live row + delete merge (higher cl) | Row deleted | TESTED |
144+
| MR-041 | Deleted row + insert merge (higher cl) | Row resurrected | NEEDS TEST |
145+
| MR-042 | cl=1 (live) vs cl=2 (deleted) | Deleted wins | NEEDS TEST |
146+
| MR-043 | cl=2 (deleted) vs cl=3 (resurrected) | Resurrected wins | NEEDS TEST |
147+
148+
---
149+
150+
## 6. Trigger/Clock Capture Experiments
151+
152+
### 6.1 INSERT Trigger
153+
154+
| Experiment ID | Scenario | Expected Clock State | Status |
155+
|--------------|----------|---------------------|--------|
156+
| TR-001 | Simple INSERT | Entry for each non-PK col | BLOCKED (test bug) |
157+
| TR-002 | INSERT with NULL values | NULL columns get entry | BLOCKED (test bug) |
158+
| TR-003 | INSERT with DEFAULT values | DEFAULT columns get entry | BLOCKED (test bug) |
159+
| TR-004 | INSERT on PK-only table | Sentinel (-1) entry | TESTED |
160+
161+
### 6.2 UPDATE Trigger
162+
163+
| Experiment ID | Scenario | Expected Clock State | Status |
164+
|--------------|----------|---------------------|--------|
165+
| TR-010 | UPDATE single column | col_version increments for that col | BLOCKED (test bug) |
166+
| TR-011 | UPDATE multiple columns | col_version increments for all | BLOCKED (test bug) |
167+
| TR-012 | UPDATE to same value | col_version still increments | BLOCKED (test bug) |
168+
| TR-013 | UPDATE NULL to value | col_version increments | BLOCKED (test bug) |
169+
| TR-014 | UPDATE value to NULL | col_version increments | BLOCKED (test bug) |
170+
171+
### 6.3 DELETE Trigger
172+
173+
| Experiment ID | Scenario | Expected Clock State | Status |
174+
|--------------|----------|---------------------|--------|
175+
| TR-020 | Simple DELETE | Tombstone entry (cid=-1) | BLOCKED (test bug) |
176+
| TR-021 | DELETE non-existent row | No change | NEEDS TEST |
177+
178+
### 6.4 Resurrection
179+
180+
| Experiment ID | Scenario | Expected | Status |
181+
|--------------|----------|----------|--------|
182+
| TR-030 | INSERT after DELETE | cl=3, new column entries | BLOCKED (test bug) |
183+
184+
---
185+
186+
## 7. ALTER TABLE Experiments
187+
188+
| Experiment ID | Scenario | Expected | Status |
189+
|--------------|----------|----------|--------|
190+
| AT-001 | ADD COLUMN (nullable) | No backfill clock entries | BLOCKED (test bug) |
191+
| AT-002 | ADD COLUMN with DEFAULT | No backfill clock entries | BLOCKED (test bug) |
192+
| AT-003 | DROP COLUMN | Clock entries for dropped col removed | BLOCKED (test bug) |
193+
| AT-004 | ADD COLUMN + UPDATE | Clock entry created on UPDATE | BLOCKED (test bug) |
194+
| AT-005 | Existing clock preserved | col_version unchanged | TESTED |
195+
| AT-006 | 1000+ row ALTER | All rows handled | TESTED |
196+
197+
---
198+
199+
## 8. Fractional Index Experiments
200+
201+
| Experiment ID | Input | Expected Output | Status |
202+
|--------------|-------|-----------------|--------|
203+
| FI-001 | (NULL, NULL) | 'a ' | TESTED |
204+
| FI-002 | ('a ', NULL) | 'a!' | TESTED |
205+
| FI-003 | (NULL, 'a ') | 'Z~' | TESTED |
206+
| FI-004 | ('a0', 'a1') | 'a0P' | TESTED |
207+
| FI-005 | ('aaa', 'aab') | 'aaaP' | TESTED |
208+
| FI-006 | Very long left key | Correct midpoint | TESTED |
209+
| FI-007 | Empty string error | Error returned | TESTED |
210+
| FI-008 | Invalid order (a > b) | Error returned | TESTED |
211+
| FI-009 | Sequential generation | Maintains ordering | TESTED |
212+
213+
---
214+
215+
## 9. Config API Experiments
216+
217+
| Experiment ID | Scenario | Expected | Status |
218+
|--------------|----------|----------|--------|
219+
| CF-001 | get merge-equal-values default | 1 | TESTED |
220+
| CF-002 | set merge-equal-values=0 | Returns 0 | TESTED |
221+
| CF-003 | set merge-equal-values=1 | Returns 1 | TESTED |
222+
| CF-004 | get unknown setting | Error | TESTED |
223+
| CF-005 | set unknown setting | Error | TESTED |
224+
| CF-006 | Config persists in connection | Yes | TESTED |
225+
| CF-007 | Config resets on new connection | Yes | NEEDS TEST |
226+
227+
---
228+
229+
## 10. Edge Case Experiments
230+
231+
### 10.1 Empty Values
232+
233+
| Experiment ID | Scenario | Expected | Status |
234+
|--------------|----------|----------|--------|
235+
| EC-001 | Empty string in pack_columns | Correct encoding | NEEDS TEST |
236+
| EC-002 | Empty blob in pack_columns | Correct encoding | TESTED |
237+
| EC-003 | Empty string as PK | Works | NEEDS TEST |
238+
| EC-004 | Empty blob as PK | Works | NEEDS TEST |
239+
| EC-005 | Empty string in merge | Merges correctly | NEEDS TEST |
240+
| EC-006 | Empty blob in merge | Merges correctly | TESTED |
241+
242+
### 10.2 Boundary Values
243+
244+
| Experiment ID | Scenario | Expected | Status |
245+
|--------------|----------|----------|--------|
246+
| EC-010 | MAX_INT64 as value | Roundtrips correctly | NEEDS TEST |
247+
| EC-011 | MIN_INT64 as value | Roundtrips correctly | NEEDS TEST |
248+
| EC-012 | MAX_FLOAT as value | Roundtrips correctly | NEEDS TEST |
249+
| EC-013 | Very large text (1MB) | Roundtrips correctly | NEEDS TEST |
250+
| EC-014 | Very large blob (1MB) | Roundtrips correctly | NEEDS TEST |
251+
252+
### 10.3 Unicode and Special Characters
253+
254+
| Experiment ID | Scenario | Expected | Status |
255+
|--------------|----------|----------|--------|
256+
| EC-020 | Emoji in text | Roundtrips correctly | NEEDS TEST |
257+
| EC-021 | NULL bytes in text | Handled correctly | NEEDS TEST |
258+
| EC-022 | SQL injection attempt | Escaped correctly | NEEDS TEST |
259+
260+
---
261+
262+
## 11. Cross-Open Interoperability Experiments
263+
264+
| Experiment ID | Scenario | Expected | Status |
265+
|--------------|----------|----------|--------|
266+
| XO-001 | Zig creates, Rust reads | Data intact | TESTED |
267+
| XO-002 | Rust creates, Zig reads | Data intact | TESTED |
268+
| XO-003 | Zig creates, Rust modifies, Zig reads | All changes visible | NEEDS TEST |
269+
| XO-004 | Rust creates, Zig modifies, Rust reads | All changes visible | NEEDS TEST |
270+
| XO-005 | site_id preserved across opens | Yes | TESTED |
271+
| XO-006 | Multiple alternating opens | State consistent | NEEDS TEST |
272+
273+
---
274+
275+
## 12. Stress/Performance Experiments
276+
277+
| Experiment ID | Scenario | Expected | Status |
278+
|--------------|----------|----------|--------|
279+
| ST-001 | 10,000 row sync | Completes without error | PARTIAL |
280+
| ST-002 | 100,000 changes batch | Memory bounded | NEEDS TEST |
281+
| ST-003 | 1000 concurrent table rows | No deadlock | NEEDS TEST |
282+
| ST-004 | Rapid INSERT/DELETE cycles | Clock consistent | NEEDS TEST |
283+
284+
---
285+
286+
## Summary
287+
288+
| Category | Total | Tested | Needs Test | Blocked |
289+
|----------|-------|--------|------------|---------|
290+
| Wire Format | 27 | 6 | 21 | 0 |
291+
| Clock Table Schema | 7 | 7 | 0 | 0 |
292+
| db_version Timing | 13 | 11 | 2 | 0 |
293+
| rows_impacted | 10 | 8 | 2 | 0 |
294+
| Merge Resolution | 33 | 12 | 21 | 0 |
295+
| Trigger/Clock | 19 | 1 | 3 | 15 |
296+
| ALTER TABLE | 6 | 2 | 0 | 4 |
297+
| Fractional Index | 9 | 9 | 0 | 0 |
298+
| Config API | 7 | 6 | 1 | 0 |
299+
| Edge Cases | 16 | 2 | 14 | 0 |
300+
| Cross-Open | 6 | 3 | 3 | 0 |
301+
| Stress | 4 | 1 | 3 | 0 |
302+
| **TOTAL** | **157** | **68** | **70** | **19** |
303+
304+
**Coverage:** 68/157 = 43% directly tested, 19 blocked by test script bugs
305+
306+
**Next Steps:**
307+
1. Fix test script bugs to unblock 19 experiments
308+
2. Add wire format edge case tests
309+
3. Add merge resolution value comparison tests
310+
4. Add edge case boundary value tests

0 commit comments

Comments
 (0)