VM Snapshot Runtime — Replace Event Replay with WASM Snapshot/Restore

[TooTallNate]

Summary

This RFC proposes an alternative runtime execution model for Workflow DevKit that replaces event-replay with WASM VM snapshotting. Instead of re-executing the workflow from the beginning and replaying the entire event log on every resumption, we:

1. Run the workflow code inside a QuickJS WASM VM (via quickjs-wasi)
2. Snapshot the VM state when the workflow suspends (waiting for a step, hook, or sleep)
3. Restore the VM from the snapshot when the workflow is re-invoked
4. Only fetch events since the last snapshot and resolve the pending promises

Motivation

The current event-replay model has scaling limitations:

• As the event log grows, re-fetching it becomes expensive (all events must be loaded on every invocation)
• Replaying the full log takes increasingly longer — every step/hook/sleep in the workflow history must be replayed
• There is an effective upper bound on how much work a workflow can do before replay becomes prohibitively slow
• Running "forever" workflows is impractical

The snapshot approach eliminates these scaling issues:

	Event Replay (current)	VM Snapshot (proposed)
Resumption cost	O(n) — replay full event log	O(1) — restore snapshot + fetch delta
Event log growth	Unbounded, all events needed every time	Only events since last snapshot needed
Long-running workflows	Degrades with event count	No degradation over time
Determinism requirement	Yes (seeded PRNG, frozen time, deterministic replay)	Still required for correctness

Prior Art: quickjs-wasi

The quickjs-wasi package was built specifically to explore this technique. It compiles QuickJS-NG to WASM as a WASI reactor and provides:

• A TypeScript API for running sandboxed JS code in QuickJS via WebAssembly
• VM state snapshotting — captures the entire WASM linear memory (including pending promises)
• VM state restoration — restores a snapshotted VM in a fresh WASM instance
• Host function callbacks that survive snapshot/restore (via ID-based dispatch)
• serializeSnapshot() / deserializeSnapshot() for persistent storage
• Configurable WASI clock (wasi.now) for deterministic Date.now() and Math.random() PRNG seeding
• Memory limits and interrupt handler for safety

The core snapshot/restore mechanism has been validated:

• Pending promises survive across snapshot/restore and can be resolved in the restored VM
• Host callback functions can be re-registered after restore
• The serialized snapshot format is versioned for forward compatibility
• 95 tests passing, published as quickjs-wasi@0.1.0
• Already used in production: proxy-agents replaced quickjs-emscripten with quickjs-wasi (PR #394)

Proposed Changes

What Changes vs What Stays the Same

Stays the same:

• User workflow code ("use workflow" / "use step" directives, function syntax)
• The SWC compiler and build pipeline
• Step handler / step execution / retry logic
• Event types and event log structure
• Suspension handler (creates events, queues steps)
• Hook, sleep, webhook APIs
• The World interface for everything except snapshots
• Step input/output serialization (dehydration/hydration)
• Encryption

Changes:

• The workflow VM switches from Node.js vm.Context to QuickJS WASM (via quickjs-wasi)
• The event replay mechanism (EventsConsumer) is replaced by snapshot/restore
• On suspension: snapshot the VM instead of discarding it
• On resumption: restore from snapshot + fetch only delta events
• New world.snapshots storage interface
• The deterministic context is provided via quickjs-wasi's WASI options + Math.random override

New World Interface: snapshots

interface SnapshotMetadata {
  /** The last event ID that was processed before this snapshot was taken */
  lastEventId: string | null;
  /** Timestamp when the snapshot was created */
  createdAt: Date;
}

interface Storage {
  // ... existing runs, steps, events, hooks ...

  snapshots: {
    save(runId: string, data: Uint8Array, metadata: SnapshotMetadata): Promise<void>;
    load(runId: string): Promise<{ data: Uint8Array; metadata: SnapshotMetadata } | null>;
    delete(runId: string): Promise<void>;
  };
}

The lastEventId is stored alongside the snapshot data (not inside the VM) — it is storage-layer metadata. In world-local, this is a .json sidecar file next to the .bin snapshot. In world-vercel (S3), it would use S3's x-amz-meta- metadata headers.

Execution Flow

First Invocation (no snapshot)

1. Workflow message arrives → load run → transition to running
2. world.snapshots.load(runId) → null (no snapshot yet)
3. Load ALL events (same as today — needed for first run)
4. Create QuickJS VM with deterministic options:
   • wasi.now(): fixed timestamp from run.startedAt
   • memoryLimit, interruptHandler for safety
   • Override Math.random with seeded PRNG (seed: {runId}:{workflowName}:{startedAt})
5. Set up WORKFLOW_USE_STEP, sleep, createHook as host functions on VM globals
6. Execute the compiled workflow bundle in QuickJS
7. Workflow calls a step → host function creates a Deferred promise, stores metadata → returns promise handle to QuickJS
8. executePendingJobs() → QuickJS suspends on the deferred promise
9. Gather pending operations (same structure as WorkflowSuspension)
10. vm.snapshot() → QuickJS.serializeSnapshot() → world.snapshots.save(runId, bytes, { lastEventId })
11. Create step_created / wait_created / hook_created events, queue step messages (same as today)

Subsequent Invocations (snapshot exists)

1. Workflow message arrives → load run
2. world.snapshots.load(runId) → { data, metadata }
3. Fetch only events since metadata.lastEventId (using existing events.list() with cursor)
4. QuickJS.deserializeSnapshot(data) → QuickJS.restore(snapshot, options)
5. Re-register host callbacks via registerHostCallback() for step/hook/sleep functions
6. Process the delta events:
   • step_completed → find pending Deferred by correlationId → deferred.resolve(result)
   • hook_received → find pending Deferred → deferred.resolve(payload)
   • wait_completed → find pending Deferred → deferred.resolve()
   • step_failed → find pending Deferred → deferred.reject(error)
7. vm.executePendingJobs() → workflow code continues from where it left off
8. Next suspension or completion:
   • Suspension → snapshot again, save with new lastEventId, create events, queue steps
   • Completion → run_completed event, world.snapshots.delete(runId)
   • Error → run_failed event, world.snapshots.delete(runId)

Pending Operation Tracking

The host maintains a Map<correlationId, Deferred> to track which promises correspond to which steps/hooks/waits. The resolve/reject functions are stored on a QuickJS global object (e.g. globalThis.__resolvers[correlationId]) so they survive in the snapshot. After restore, the host reads __resolvers to rebuild the pending operations map.

Determinism

Determinism is still required even in the snapshot model because:

1. Concurrent executions — the same workflow may be executed more than once simultaneously; they must produce the same result
2. Recovery path — if a snapshot is corrupted or lost, the workflow must be recoverable by replaying from the event log
3. The event log is the source of truth — the snapshot is an optimization, not a replacement for the event log

The deterministic context is provided by:

• quickjs-wasi's wasi.now() option for Date.now() / new Date()
• A host-provided seeded Math.random replacement (seed: {runId}:{workflowName}:{startedAt})
• Deterministic ULID generation for correlation IDs (same seed)

Step Execution Model

Step execution does not change. Only the workflow orchestration code (the "use workflow" function) runs in QuickJS. Steps continue to run with full Node.js access in the step handler, exactly as they do today.

In the workflow VM bundle, step function calls are replaced with globalThis[Symbol.for("WORKFLOW_USE_STEP")]("stepId") proxies (same SWC compiler output as today). The only difference is that these proxies are now host functions backed by vm.newFunction() in quickjs-wasi instead of being JS functions in a Node.js vm.Context.

world-local Implementation

Filesystem storage under {dataDir}/snapshots/:

{dataDir}/snapshots/
  {runId}.bin       # Serialized snapshot (QuickJS.serializeSnapshot output)
  {runId}.json      # Metadata: { lastEventId, createdAt }

Implementation Phases

Phase 1: World Interface + world-local

• Add snapshots to Storage interface in packages/world/src/interfaces.ts
• Add SnapshotMetadata type
• Implement in packages/world-local/src/storage/snapshots-storage.ts
• Wire into createStorage()
• Tests

Phase 2: Snapshot Runtime

• New packages/core/src/runtime/snapshot-runtime.ts
• QuickJS VM setup with deterministic WASI options
• Host function implementations for useStep, sleep, createHook
• Snapshot/restore flow with pending operation tracking
• Integration with existing suspension handler

Phase 3: Wiring

• Feature flag to select snapshot vs replay runtime
• Update workflowEntrypoint() to delegate to appropriate runtime
• Ensure compiled workflow bundle works in both runtimes

Phase 4: Testing

• Unit tests for snapshot runtime
• E2E tests: simple workflow, multi-step workflow, sleep, hooks
• Regression: ensure existing event-replay tests still pass

Phase 5: world-vercel + world-postgres

• Implement snapshots storage in world-vercel (S3 with metadata headers)
• Implement snapshots storage in world-postgres

Open Questions

1. Feature gating: Should this be opt-in per-workflow, per-project, or a global runtime mode?

2. Snapshot size: A fresh QuickJS VM snapshot is ~256 KB. With user data it grows with the heap. Users should apply compression (gzip/zstd) for storage — the WASM memory compresses very well (~60-70% reduction). Should the World implementation handle compression transparently?

3. QuickJS compatibility: The compiled workflow bundle currently targets Node.js vm.runInContext(). For QuickJS, we need to verify that Symbol.for() works correctly and that the compiled output is compatible. May need minor SWC plugin adjustments.

4. Snapshot lifecycle: When should snapshots be cleaned up? On run_completed, run_failed, run_cancelled — same as hook cleanup. Should there be a TTL for orphaned snapshots?

5. Recovery from snapshot corruption: If a snapshot fails to restore, the runtime should fall back to full event replay. Should this be automatic or explicit?

6. Stack size limits: QuickJS-NG disables JS_SetMaxStackSize on WASI, so deep recursion causes a WASM trap (not a catchable exception). Is this a concern for real workflow patterns?