Commit dd42d7e
authored
fix(avm): Add cancellation token to prevent C++ simulation race condition on timeout (#19219)
# fix(avm): Add cancellation token to prevent C++ simulation race
condition on timeout
## Summary
Fix race condition where C++ AVM simulation corrupts WorldState after
TypeScript timeout. This was done as follows:
- Add `CancellationToken` mechanism to safely stop C++ simulation before
reverting checkpoints
- **Primary Fix**: PublicProcessor cancels public simulation and
**waits** for it to die before proceeding and reverting checkpoints
- C++ simulation polls this cancellation token regularly so that we know
it will die quickly after cancellation
- **Secondary Fix**: Add mutex locking to checkpoint operations for
thread safety
- Add paired tests proving both the bug exists (without fix) and the fix
works (with cancellation)
## The Bug
When a C++ AVM simulation times out via `Promise.race()` in
`PublicProcessor`, a race condition occurs:
1. TypeScript's timeout fires and calls `revertCheckpoint()` on
WorldState
2. **But C++ simulation continues running** on a libuv worker thread
3. C++ holds a native handle to WorldState obtained before the timeout
4. C++ makes writes to WorldState **after** the checkpoint was reverted
5. This corrupts WorldState, causing `checkWorldStateUnchanged()` to
fail
The root causes are:
1. `GuardedMerkleTreeOperations` only guards TS merkle operations. The
C++ code interacts with the same WorldState without guards.
2. Nothing stops C++ from finishing simulation after PublicProcessor
reaches deadline.
```
Timeline of the race (BEFORE fix):
TypeScript C++ (libuv worker thread)
---------- -------------------------
Start simulation -----------------> Begins executing opcodes
| |
v v
Promise.race timeout fires Still running (e.g., in pad_trees())
| |
v |
| |
v |
IMMEDIATELY revertCheckpoint() |
| v
| Finishes pad_trees()
| Makes write AFTER revert! (CORRUPTION)
v |
checkWorldStateUnchanged() FAILS!
```
## The Fix
The key insight: **signaling cancellation is not enough** - we must
**wait** for C++ to actually stop.
```
Timeline (AFTER fix):
TypeScript C++ (libuv worker thread)
---------- -------------------------
Start simulation -----------------> Begins executing opcodes
| |
v v
Promise.race timeout fires Still running (e.g., in pad_trees())
| |
v |
cancel(100) sets atomic flag | (doesn't check flag yet)
| |
v v
WAIT for simulation promise Finishes pad_trees(), checks flag
| Throws CancelledException
|<-----------------------------Promise rejects
v
NOW revertCheckpoint() (C++ is done)
|
v
checkWorldStateUnchanged() ✓ clean state
```
### C++ Side
1. **`CancellationToken` class** (`cancellation_token.hpp`):
- Thread-safe `std::atomic<bool>` flag
- `cancel()` - signals cancellation (called from TS thread)
- `check()` - throws `CancelledException` if cancelled (called from
worker thread)
3. **Check at each opcode** (`execution.cpp`, `hybrid_execution.cpp`):
```cpp
while (!external_call_stack.empty()) {
if (cancellation_token_) {
cancellation_token_->check();
}
// ... execute opcode
}
```
4. **Check before every WorldState write** (`raw_data_dbs.cpp`):
```cpp
void PureRawMerkleDB::pad_tree(...) {
check_cancellation(); // Throws if cancelled
// ... proceed with write
}
```
5. **Mutex locking for checkpoint operations**
(`cached_content_addressed_tree_store.hpp`):
```cpp
void ContentAddressedCachedTreeStore::checkpoint() {
std::unique_lock lock(mtx_); // Prevent races with C++ writes
cache_.checkpoint();
}
```
### TypeScript Side
1. **`CppPublicTxSimulator.cancel(waitTimeoutMs)`** - Signal AND wait:
```typescript
public async cancel(waitTimeoutMs?: number): Promise<void> {
if (this.cancellationToken) {
cancelSimulation(this.cancellationToken);
}
// Wait for simulation to actually complete
if (waitTimeoutMs !== undefined && this.simulationPromise) {
await Promise.race([
this.simulationPromise.catch(() => {}),
sleep(waitTimeoutMs),
]);
}
}
```
> Note: ideally we'd like to have no timeout here since we really want
to wait for C++ to recognize cancellation. The timeout is really just a
safeguard against some cancellation bug.
2. **`PublicProcessor`** timeout handler:
```typescript
if (err?.name === 'PublicProcessorTimeoutError') {
// Signal cancellation AND WAIT for C++ to stop (up to 100ms)
await this.publicTxSimulator.cancel?.();
// NOW safe to stop the guarded fork
await this.guardedMerkleTree.stop();
}
```
## Test Plan
Four paired tests at two levels verify the bug and fix:
### Replace bug and prove fix at TxSimulator level
Tests using `CppPublicTxSimulator` directly - **identical code**, only
difference is whether `cancel()` is called:
```typescript
async function runRaceConditionTest(useCancellation: boolean): Promise<number> {
// ... setup ...
const simulationPromise = simulator.simulate(tx);
if (useCancellation) {
await simulator.cancel(100); // FIX: Signal AND wait
}
// BUG: No cancel, C++ continues during reverts
await merkleTrees.revertCheckpoint();
// Check for corruption...
}
it('BUG PROOF: race condition exists WITHOUT cancellation'); // Expects >0 corruptions
it('FIX PROOF: no race condition WITH cancellation'); // Expects 0 corruptions
```
### Replicate bug and prove fix at PublicProcessor level
Tests using full `PublicProcessor.process()` with deadline timeout:
```typescript
it('PublicProcessor BUG PROOF: state corruption occurs WITHOUT cancellation');
it('PublicProcessor FIX PROOF: no state corruption WITH cancellation');
```
### Running the tests
```bash
cd yarn-project/simulator
yarn test src/public/public_processor/apps_tests/timeout_race.test.ts
```
## Files Changed
### C++ (barretenberg)
| File | Change |
|------|--------|
| `vm2/simulation/lib/cancellation_token.hpp` | **New**: Thread-safe
cancellation token class |
| `vm2/simulation/lib/raw_data_dbs.hpp` | Add token to `PureRawMerkleDB`
|
| `vm2/simulation/lib/raw_data_dbs.cpp` | Check cancellation before all
writes |
| `vm2/simulation/gadgets/execution.hpp` | Add token to `Execution` |
| `vm2/simulation/gadgets/execution.cpp` | Check cancellation at each
opcode |
| `vm2/simulation/standalone/hybrid_execution.cpp` | Check cancellation
at each opcode |
| `vm2/avm_sim_api.hpp` | Thread token through API |
| `vm2/avm_sim_api.cpp` | Thread token through API |
| `vm2/simulation_helper.hpp` | Thread token to execution |
| `vm2/simulation_helper.cpp` | Thread token to execution |
| `nodejs_module/avm_simulate/avm_simulate_napi.hpp` | NAPI bindings for
token |
| `nodejs_module/avm_simulate/avm_simulate_napi.cpp` | NAPI bindings for
token |
| `nodejs_module/init_module.cpp` | Register NAPI functions |
| `crypto/merkle_tree/.../cached_content_addressed_tree_store.hpp` | Add
mutex to checkpoint ops |
### TypeScript (yarn-project)
| File | Change |
|------|--------|
| `native/src/native_module.ts` | Export `createCancellationToken`,
`cancelSimulation` |
| `simulator/.../public_tx_simulator_interface.ts` | Add
`cancel?(waitTimeoutMs?)` method |
| `simulator/.../cpp_public_tx_simulator.ts` | Track promise, implement
`cancel()` with wait |
| `simulator/.../public_processor.ts` | `await cancel(100)` before
reverts |
| `simulator/.../public_tx_simulation_tester.ts` | Add `cancel()` and
`getSimulator()` |
| `simulator/.../timeout_race.test.ts` | **New**: Paired proof tests |
## Why This Approach
1. **Wait, don't just signal**: The key fix is awaiting the simulation
promise after signaling
2. **Bounded wait**: 100ms timeout prevents indefinite blocking if C++
is stuck
3. **Check before writes**: Prevents partial/corrupted state from
cancelled operations
4. **Mutex on checkpoints**: Prevents races between C++ writes and TS
checkpoint ops
6. **Minimal overhead**: `std::atomic<bool>` check is very cheap (single
memory read)
7. **Backward compatible**: Token is optional, existing code works
unchanged
8. **Testable**: Paired tests definitively prove both the bug and the
fix
## Future Work
### **Move merkle tree guarding into C++** (out of scope for this PR)
Currently, `GuardedMerkleTreeOperations` in TypeScript wraps merkle tree
operations to prevent access after `stop()` is called. However, this
guard is ineffective for C++ because:
1. C++ obtains the native WorldState and bypasses TS guarding
2. C++ can still make writes after TS calls `stop()`
### Guard getRevision
I think this is lower-impact, but ideally getRevision should fail if
called on a stopped instance.
---
🤖 Description generated with [Claude
Code](https://claude.com/claude-code)1 parent 4a06547 commit dd42d7e
File tree
20 files changed
+792
-65
lines changed- barretenberg/cpp/src/barretenberg
- crypto/merkle_tree/node_store
- nodejs_module
- avm_simulate
- vm2
- simulation
- gadgets
- lib
- standalone
- yarn-project
- native/src
- simulator/src/public
- fixtures
- public_processor
- apps_tests
- public_tx_simulator
20 files changed
+792
-65
lines changedLines changed: 8 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
273 | 273 | | |
274 | 274 | | |
275 | 275 | | |
276 | | - | |
277 | | - | |
278 | | - | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
279 | 279 | | |
280 | 280 | | |
| 281 | + | |
281 | 282 | | |
282 | 283 | | |
283 | 284 | | |
284 | 285 | | |
285 | 286 | | |
| 287 | + | |
286 | 288 | | |
287 | 289 | | |
288 | 290 | | |
289 | 291 | | |
290 | 292 | | |
| 293 | + | |
291 | 294 | | |
292 | 295 | | |
293 | 296 | | |
294 | 297 | | |
295 | 298 | | |
| 299 | + | |
296 | 300 | | |
297 | 301 | | |
298 | 302 | | |
299 | 303 | | |
300 | 304 | | |
| 305 | + | |
301 | 306 | | |
302 | 307 | | |
303 | 308 | | |
| |||
Lines changed: 88 additions & 39 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
| 13 | + | |
13 | 14 | | |
14 | 15 | | |
15 | 16 | | |
| |||
116 | 117 | | |
117 | 118 | | |
118 | 119 | | |
119 | | - | |
| 120 | + | |
120 | 121 | | |
121 | 122 | | |
122 | 123 | | |
123 | 124 | | |
| 125 | + | |
124 | 126 | | |
125 | 127 | | |
126 | | - | |
127 | | - | |
| 128 | + | |
| 129 | + | |
128 | 130 | | |
129 | 131 | | |
130 | 132 | | |
| |||
144 | 146 | | |
145 | 147 | | |
146 | 148 | | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
147 | 162 | | |
148 | 163 | | |
149 | 164 | | |
| |||
189 | 204 | | |
190 | 205 | | |
191 | 206 | | |
192 | | - | |
193 | | - | |
194 | | - | |
195 | | - | |
196 | | - | |
197 | | - | |
198 | | - | |
199 | | - | |
200 | | - | |
201 | | - | |
202 | | - | |
203 | | - | |
204 | | - | |
205 | | - | |
206 | | - | |
207 | | - | |
208 | | - | |
209 | | - | |
210 | | - | |
211 | | - | |
212 | | - | |
213 | | - | |
214 | | - | |
215 | | - | |
216 | | - | |
217 | | - | |
218 | | - | |
219 | | - | |
220 | | - | |
221 | | - | |
222 | | - | |
223 | | - | |
224 | | - | |
225 | | - | |
226 | | - | |
227 | | - | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
228 | 247 | | |
229 | 248 | | |
230 | 249 | | |
| |||
299 | 318 | | |
300 | 319 | | |
301 | 320 | | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
302 | 351 | | |
Lines changed: 24 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| 29 | + | |
| 30 | + | |
29 | 31 | | |
30 | 32 | | |
31 | 33 | | |
32 | 34 | | |
33 | 35 | | |
34 | 36 | | |
35 | 37 | | |
| 38 | + | |
36 | 39 | | |
37 | 40 | | |
38 | 41 | | |
| |||
43 | 46 | | |
44 | 47 | | |
45 | 48 | | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
46 | 70 | | |
47 | 71 | | |
48 | 72 | | |
Lines changed: 4 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
19 | 23 | | |
20 | 24 | | |
21 | 25 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
13 | | - | |
| 13 | + | |
| 14 | + | |
14 | 15 | | |
15 | 16 | | |
16 | 17 | | |
| |||
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
26 | | - | |
| 27 | + | |
| 28 | + | |
27 | 29 | | |
28 | 30 | | |
29 | 31 | | |
| |||
32 | 34 | | |
33 | 35 | | |
34 | 36 | | |
35 | | - | |
| 37 | + | |
| 38 | + | |
36 | 39 | | |
37 | 40 | | |
38 | 41 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
5 | 6 | | |
6 | 7 | | |
7 | 8 | | |
| |||
14 | 15 | | |
15 | 16 | | |
16 | 17 | | |
17 | | - | |
| 18 | + | |
| 19 | + | |
18 | 20 | | |
19 | 21 | | |
20 | 22 | | |
| |||
Lines changed: 5 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1724 | 1724 | | |
1725 | 1725 | | |
1726 | 1726 | | |
| 1727 | + | |
| 1728 | + | |
| 1729 | + | |
| 1730 | + | |
| 1731 | + | |
1727 | 1732 | | |
1728 | 1733 | | |
1729 | 1734 | | |
| |||
Lines changed: 8 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
| 33 | + | |
33 | 34 | | |
34 | 35 | | |
35 | 36 | | |
| |||
79 | 80 | | |
80 | 81 | | |
81 | 82 | | |
82 | | - | |
| 83 | + | |
| 84 | + | |
83 | 85 | | |
84 | 86 | | |
85 | 87 | | |
| |||
100 | 102 | | |
101 | 103 | | |
102 | 104 | | |
| 105 | + | |
103 | 106 | | |
104 | 107 | | |
105 | 108 | | |
| |||
265 | 268 | | |
266 | 269 | | |
267 | 270 | | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
268 | 275 | | |
269 | 276 | | |
270 | 277 | | |
0 commit comments