Quantized save: defer wasm + song swap to next beat divisible by N#128
Draft
petersalomonsen wants to merge 18 commits into
Draft
Quantized save: defer wasm + song swap to next beat divisible by N#128petersalomonsen wants to merge 18 commits into
petersalomonsen wants to merge 18 commits into
Conversation
…ble by N
A dropdown next to the save button picks N. N=0 ("Now", the default)
preserves the existing suspend/instantiate/resume behavior — a brief
audible pause but the swap is guaranteed live before save returns.
N>0 instead stashes the freshly compiled wasm (and any new sequencedata
/ toggleSongPlay) in a pending slot on the worklet processor, acks
back to the caller without suspending, and performs an atomic swap
inside process() at the next beat where currentBeat % N == 0. Beats
are derived from the sequencer's currentFrame and the song's BPM
(read live from midisequencer/pattern.js's exported bpm binding).
Last-save-wins: saving again before the trigger fires replaces the
pending entries. On swap, allNotesOff() is called on the outgoing
instance to silence any tails before the receiver pointer is moved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Deploying webassemblymusic with
|
| Latest commit: |
6ab3073
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://0111c42d.webassemblymusic.pages.dev |
| Branch Preview URL: | https://quantized-save.webassemblymusic.pages.dev |
Previously the worklet received raw bytes and called WebAssembly.instantiate(bytes, imports) inside process()'s thread. In audio worklets the bytes overload's compile step falls back to synchronous on the render thread, blowing the ~2.9 ms render-quantum deadline and producing an audible glitch on every save (which was the whole reason the legacy Now path brackets the call with suspend/resume). Move the compile to the main thread via WebAssembly.compile(bytes) and ship the resulting Module across postMessage (Modules are structured cloneable). The worklet now calls `new WebAssembly.Instance(module, …)` which is synchronous but only links imports + sets up memory — small enough to fit inside a render quantum, so saves no longer glitch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related bugs were making the swap land on unpredictable beats: 1. lastBeat starts at -1, so the first process() call after a save sees `currentBeat !== -1` and treats it as a beat *crossing*. If the save happened to land inside a beat where `beat % N === 0` already, the modulo check passed and the swap fired ~3 ms after save instead of waiting for the next N-aligned beat. 2. lastBeat was never reset after a swap finished, so the second save in a session re-entered with stale state and could misfire the same way. Fix (1): when lastBeat is -1 (first call after pending state is set), anchor it to the current beat and return without swapping. We only swap once we've crossed *out of* that beat into a new one that satisfies the quantize. Fix (2): when there's no pending state, reset lastBeat to -1 so the next save anchors fresh. With N=16 a save now reliably fires at the next absolute beat divisible by 16 (16, 32, 48, 64, …) regardless of where in the bar the save landed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the user switches to a song with a different BPM and clicks save: - midiBpm (live binding from pattern.js) is now the new song's BPM, because compileSong just ran setBPM(...) for the new song. - But the audible music is still the old song at the old BPM until the swap fires. - Beat indicator in the UI uses the BPM captured at attachSeek time (the song that's actually playing), so it stays correct from the listener's perspective. - The worklet was using pendingBpm (= midiBpm at save time = new song's BPM) to decide when `currentBeat % N == 0`, so its beats diverged from what the listener was hearing — the swap landed on the wrong audible beat, or never seemed to land at all if the worklet thought it had already passed. Track playingBpm separately in the worklet: - Initial `wasm` message and N=0 immediate replacements set playingBpm to the bpm that just took over. - A quantized swap, once it fires, promotes pendingBpm to playingBpm. - tryQuantizedSwap uses playingBpm for beat math (falls back to pendingBpm if playingBpm hasn't been seeded yet). All three call sites (initial connect, N=0 path in updateSynth, and the existing pending path) now pass bpm along with the wasm message. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The "have to press save twice" symptom was a race between two separate worklet messages: - updateSynth posted `pendingWasmModule` and awaited `pendingWasmReady`. WebAssembly.compile(bytes) on the main thread plus the postMessage round-trip takes ~50–300 ms. - During that window, `process()` was already running with `pendingInstance` set and `lastBeat` anchored. If a beat boundary divisible by N happened to fall inside the window, tryQuantizedSwap fired the swap with only the new wasm — the new sequencedata hadn't arrived from updateSong yet. The listener kept hearing the old song's notes through the new synth, which looked like "the song didn't replace". The second save caught up because by then nothing was pending and both halves landed together. Fix: when quantizeN > 0 and both new wasm and new sequencedata are available, bundle them into a single `pendingWasmModule` message that also carries `pendingSequencedata` and `pendingToggleSongPlay`. The worklet's onmessage handler sets all three pending fields atomically before acking, so any subsequent process() call sees either all pending state or none — never a half-applied save. The legacy quantizeN === 0 path and the WAM (synthsource) path are unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a WTR spec that traces noteons emitted by the worklet's wasm and asserts they line up with the swap event the worklet announces. The spec exposed three distinct bugs that explained the user-reported "wrong beat" and "had to press save twice" symptoms: 1. WebAssembly.Module wasn't being delivered to the worklet on Chromium. The previous commit pre-compiled the bytes on the main thread and posted the resulting Module. Firefox's AudioWorklet structured-clone handles Modules; Chromium's silently drops them, so the bundled save message never arrived and the swap never fired. Switch to posting raw bytes and `await WebAssembly.instantiate(bytes, ...)` in the worklet — the async instantiate path is off-thread on both engines and doesn't audibly glitch the way a synchronous `new WebAssembly.Module` would. 2. The sequencer's `setSequenceData` "replace while playing" branch used `evt.time >= currentTime` to pick where to resume. The swap fires on the first render quantum that crosses a beat boundary, so currentTime is ~2.9 ms past the boundary's exact ms. The new song's event sitting on that boundary (e.g. at 4800 ms when the swap fires at 4801 ms) failed the `>=` check and was silently skipped — the listener heard up to one new-tempo beat of silence before the next note. Allow events from one render quantum back. 3. When a quantized swap fires with new sequencedata, reset the sequencer's `currentFrame` to 0 so the new song plays from its beat 0 at the swap moment. Otherwise a swap at currentTime T leaves the new song waiting for its next event at time > T — at 120 BPM that's up to 500 ms of silence after the swap fires. The worklet now also posts a structured `quantizedSwapApplied` message back to the main thread when each swap fires, replacing the ad-hoc `_debug` string the test was parsing during diagnosis. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two regressions in the previous commit:
1. Audible glitch on every save. Switching from the pre-compiled-Module
path to the send-bytes-and-compile-in-worklet path put the
WebAssembly compile back on the audio thread (Chromium runs
WebAssembly.compile inside an AudioWorkletGlobalScope synchronously
from the render thread's perspective, no matter that the API is
async). Firefox had been glitch-free under the Module path.
2. Beat indicator in the bottom-right drifts after a song switch.
attachSeek captures bpm in a closure at startup, so the indicator
always uses the *first* song's tempo. Multiple swaps compound the
drift.
Fix (1): probe at startup whether the engine can structure-clone a
WebAssembly.Module across AudioWorkletNode.port. Firefox passes; the
worklet posts back an ack, the main thread records that and posts
Module-only on subsequent saves. Chromium silently drops the probe
message, the ack never comes, the 500-ms timeout in the probe trips,
and the main thread falls back to sending raw bytes (with the
worklet awaiting `WebAssembly.instantiate(bytes, …)`). On engines
where Module transfer works we're back to glitch-free saves.
Fix (2): attachSeek now accepts either a static bpm or a getter. The
worklet announces each quantized swap with `{quantizedSwapApplied,
bpm}`; main thread maintains `activePlayingBpm` from those events and
from N=0 immediate replacements, and feeds attachSeek a getter that
reads it. The beat indicator's tempo now follows the audible song.
Test: quantized-save.spec.js now also asserts the beat indicator
reflects the swapped-in tempo after two swaps (would previously have
read at the original first-song bpm). Existing midisynthaudioworklet
spec still passes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both the sequencer and the synth processor hardcoded 128 in several
places (sample-buffer slicing, currentFrame increment, the
quantum-tolerance check in setSequenceData). Pull it into a single
named constant per module — RENDER_QUANTUM_FRAMES — and add a comment
flagging that the wasm synth also assumes this size, so a future
non-default `AudioContext({renderQuantumSize})` would need a
coordinated change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Module-vs-bytes branching plus a startup probe to pick between them was carrying its own complexity (probe handler, timeout, state, two code paths in updateSynth, two code paths in the worklet's onmessage). Cross-browser support for WebAssembly.Module over AudioWorkletNode.port is unreliable enough that the optimization isn't worth the maintenance cost. Keep one path: main thread posts raw bytes, the worklet does `await WebAssembly.instantiate(bytes, …)` in its onmessage handler. The bpm-following beat-indicator changes stay; this only removes the Module-transfer machinery. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Measures each step of the worklet's pendingWasmBytes handler: - WebAssembly.instantiate (await — mostly off-thread compile, but the start function and Promise-resume code run on the audio thread) - loadAudioIntoWasm (synchronous Float32Array copy into wasm memory) - stashing pending state Also counts process() calls before and after the instantiate await so we can tell whether the audio thread actually missed any render quanta (if actual < expected, we underran → audible glitch). Emits a `_swapTiming` message per save. updateSynth's main-thread listener logs it as `[swap-timing] …`. Temporary — to be removed once we've identified where the time actually goes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The first instrumentation pass only measured the window inside the pendingWasmBytes onmessage handler and showed 0 missed quanta — but the user reports the audible glitch is at click time and lasts ~250 ms, far longer than the 5.8 ms instantiate window. The glitch must be caused by something OUTSIDE that window. Add: - Continuous gap monitor in process() that fires a message whenever the inter-quantum gap exceeds 1.5x the expected ~2.9 ms, reporting the actual gap, missed quanta, and audio-context time of occurrence. - Pre-post and post-ack logs in updateSynth so we can see when the main thread sent the bundled message and when the ack came back, in both perf.now() and audio-context-time units. Lets us correlate the audio-thread gap timestamp with the main-thread sequence of events. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 89 ms wall-clock vs 5.8 ms audio-context-time during post→ack shows the audio thread was starved for ~83 ms, but the 5.8 ms instantiate alone can't account for that. The next suspect is structured-clone deserialization of the bundled message (which includes 7347 event objects) — that work runs on the audio thread BEFORE onmessage fires. Add an "onmessage entered" ping the worklet posts immediately at handler entry. updateSynth's listener captures performance.now() when that ping arrives, and the ack log now reports "clone+delivery=Xms" alongside total wallElapsed. If clone is the bottleneck the two numbers will be nearly equal. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The temporary timing/process-gap probes added across the last several commits served their purpose: they pinned the audible save-glitch on the in-worklet `await WebAssembly.instantiate(bytes, …)` blocking the audio thread for ~85 ms on Chromium, and ruled out structured-clone of the sequencedata as the cause. Strip them now that the diagnosis is documented in issue #129. What stays: - The `quantizedSwapApplied` swap-event message (used by the beat indicator to follow the audible tempo across swaps). - `_installTracingReceiver` / `_trace_noteon` (used by quantized-save.spec to verify which notes actually played). - The single-path bytes implementation, with a comment pointing at issue #129 for the trade-off and possible future fixes. What goes: - `_swapTiming` posts and the corresponding console.log. - `_processGap` continuous render-quantum monitor + log. - `_onMessageEntered` ping and the clone-time measurement on main. - `_processCallsTotal`, `_lastProcessTime` counters. - Pre-/post-postMessage `performance.now()` capture in updateSynth. Net: -108 LoC across the two worklet files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ions
Workaround for chromium#40855462 ("Chrome fails to send WebAssembly.Module
over AudioWorkletNode.port"): pass the pre-compiled Module via
processorOptions on AudioWorkletNode construction instead. Module
structured-clones via processorOptions in current Chromium (and Firefox);
only port.postMessage drops it.
This eliminates the audible save glitch (~85 ms on Chrome) by moving the
wasm compile off the audio rendering thread:
- Main thread: WebAssembly.compile(bytes) — V8's off-thread compile
worker pool, no audio impact.
- New AudioWorkletNode constructed with processorOptions: { wasmModule,
audio, sequencedata, toggleSongPlay, bpm, swapAtCtxTime }.
- Processor constructor receives the Module and synchronously creates
an Instance — microseconds of linking, fits inside one render quantum.
- New node is connected through its own GainNode (gain=0) so it renders
silently until the quantize boundary.
- Sample-accurate handoff: at swapAtCtxTime, oldGain.setValueAtTime(0)
and newGain.setValueAtTime(1). The old node is then told to freeze
its sequencer (stopAtCtxTime), and main thread cleans it up 200 ms
later.
Architecture changes:
- audioworkletprocessorsequencer.js exposes MidiSequencer as a class
on AudioWorkletGlobalScope (MidiSequencerClass) in addition to the
legacy singleton. The synth processor now creates a per-instance
sequencer so multiple AudioWorkletNodes can coexist briefly during
a swap without sharing state.
- midisynthaudioworkletprocessor.js: per-instance sequencer; supports
both legacy `wasm`-message construction (for the initial load and
N=0 path) and the new processorOptions-based construction;
swapAtCtxTime/stopAtCtxTime gating in process().
- midisynthaudioworklet.js updateSynth N>0 path completely rewritten
to spawn a new AudioWorkletNode + GainNode per save and crossfade.
The legacy N=0 path (suspend/resume) is unchanged.
- quantizedSwapApplied + _trace_noteon are now re-broadcast as
CustomEvents on `window` so listeners (beat indicator, tests) don't
have to rebind on every save. activePlayingBpm is updated and
audioStartCtxTime is reset to the new song's swap time so the next
quantize boundary is computed relative to the current playback.
- quantized-save.spec.js rewritten to subscribe to the window
CustomEvents and assert in audio-context-time rather than the
worklet's currentFrame-based time (which now resets per node).
Tested: 51 passes / 51 passes on Chromium 148 + Firefox 150 in WTR.
Manual verification (no audible glitch on real Chrome) still pending.
Closes the workaround path tracked in #129.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…essorOptions" This reverts commit e69ba1d.
The three shapes — AS midi synth, WAM midi synth, and pattern-based — share one save handler but route to completely different audio worklet processors. The branching has grown enough that it's not obvious which shape goes where, especially since quantize support only exists for the AS midi synth path. Add a header comment listing the shapes and a sharper inline comment on `useBundledSave` calling out that quantize is a no-op for shapes (2) and (3). No code changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous wording referred to the "original 'had to press save twice' symptom" — but that symptom only existed during early iterations of this branch's quantize work, never in master. Once this branch merges, the comment would mislead a future reader into thinking master ever had that bug. Reword to describe the concrete race the bundling prevents, without implying any historical context. No code changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
WebAssembly.instantiate blocks the audio thread in AudioWorkletGlobalScope regardless of bytes-vs-Module input (no off-thread compile dispatch from the worklet). Compiling on save click made the *currently playing* wasm glitch; moving the compile inside tryQuantizedSwap puts the glitch at the swap moment instead — musically expected, since the listener already knows something is changing. Save click now only stashes bytes in pendingWasmBytesDeferred. At the next beat divisible by quantizeN, tryQuantizedSwap kicks off instantiate via .then (await isn't available inside process) and sets swapInstantiateInFlight; the next process() tick sees pendingInstance and calls _applySwap. An instance LRU cache was considered to avoid recompiling on switch-back, but a switch back to a cached instance produced dangling notes from stale voice state — reverted. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Status: not merging
Final shape works but isn't fully suited for live concerts — the audible glitch from
WebAssembly.instantiateblocking the audio thread can't be hidden, only relocated. Likely follow-up is a multi-window app where another window with its own AudioContext signals takeover.Summary
Now(N=0) preserves the existing suspend/instantiate/resume behavior — short audible pause, guaranteed live before save returns.currentBeat % N == 0, the worklet callsWebAssembly.instantiate(bytes, …)from insidetryQuantizedSwapvia.then, and once the instance is ready the nextprocess()tick applies the atomic swap.playingBpm(whatever's currently audible), notpendingBpm— so swapping into a song with a different BPM still lands on the correct audible beat.allNotesOff()on the outgoing instance, sequencercurrentFramereset to 0 so the new song plays from its own beat 0, then the receiver pointer is moved.The unavoidable trade-off
WebAssembly.instantiateblocks the audio thread synchronously insideAudioWorkletGlobalScopeon every browser we tested — V8/SpiderMonkey don't dispatch the compile to off-thread workers from a worklet. So we get a glitch somewhere. The choice was:This PR picks the second. The save click is glitch-free; the audible pause moves to the swap moment.
Other approaches we tried and dropped:
WebAssembly.Moduleto the worklet: blocked by Chromium silently dropping Module-bearing messages toAudioWorkletNode.port. (Issue Audible glitch on quantized save (synth change): in-worklet wasm compile blocks audio thread; Module postMessage to AudioWorklet broken on Chromium #129, now closed — the bytes-only path here makes that workaround unnecessary.)AudioWorkletNodewithprocessorOptions: glitched in a different place (sequencedata clone on the audio thread at startup), beat indicator froze, replacement landed at the wrong time. Reverted.Test plan
synth1/audioworklet/quantized-save.spec.jsexercises the deferred swap end-to-end (post-save click → expected swap time → tracing receiver confirms note events come from the new instance). Passes on Chromium and Firefox.midisynthaudioworklet.spec.js(Now/legacy path) — no regression in the immediate suspend/resume swap.🤖 Generated with Claude Code