-
Notifications
You must be signed in to change notification settings - Fork 10k
Changelog for workers-rs panic recovery #25285
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
mikenomitch
merged 5 commits into
cloudflare:production
from
guybedford:workers-rs-panics
Sep 19, 2025
Merged
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
dcb8db3
changelog for workers-rs panic recovery
guybedford 0f897d9
feedback
guybedford fd85278
Small wording tweaks to workers-rs panic changelog
mikenomitch 8f9b3e9
Further tweaks
mikenomitch 30d122b
Fix typo in workers-rs panic recovery changelog
mikenomitch File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
69 changes: 69 additions & 0 deletions
69
src/content/changelog/workers/2025-09-19-workers-rs-panic-recovery.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| --- | ||
| title: Panic Recovery for Rust Workers | ||
| description: Making workers-rs deployments more reliable with panic recovery | ||
guybedford marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| date: 2025-09-19 | ||
| --- | ||
|
|
||
| import { WranglerConfig, Aside } from "~/components"; | ||
|
|
||
| A long-standing problem we've had with [workers-rs](https://github.com/cloudflare/workers-rs) is that its Rust panics are non-recoverable, and can put the worker into an invalid state. | ||
|
|
||
| Working on the [wasm-bindgen](https://github.com/wasm-bindgen/wasm-bindgen) internals, we've been able to implement a solution for panic recovery to ensure more reliable deployments. | ||
|
|
||
| ## Fixing Rust Panics with Wasm Bindgen | ||
|
|
||
| We use [wasm-bindgen](https://github.com/wasm-bindgen/wasm-bindgen) to build and deploy Rust workers, but Wasm Bindgen is not designed to support recoverable panics. | ||
|
|
||
| When a panic happens in Wasm Bindgen, the entire Wasm application is considered to be in an invalid state, and any further function calls could result in overflows or memory exceptions. | ||
|
|
||
| But Wasm is a virtual machine - if we can reset all the Wasm internal state back to its initial state then we can continue to take new requests. | ||
|
|
||
| It's possible to add a panic handler in Rust to detect immediately when a panic happens: | ||
guybedford marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ```rust | ||
| std::panic::set_hook(Box::new(move |panic_info| { | ||
| hook_impl(panic_info); | ||
| })); | ||
| ``` | ||
|
|
||
| With some extra wiring we can then export a registration function to JavaScript to allow a custom panic hook to be attached: | ||
|
|
||
| ```js | ||
| import { setPanicHook } from "./index.js"; | ||
guybedford marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| setPanicHook(function (err) { | ||
| console.error("Panic handler!", err); | ||
| }); | ||
| ``` | ||
|
|
||
| We then call a new reset state function when a panic occurs, to reinitialize the Wasm application to back to as it was when the application first started. | ||
|
|
||
| ## Resetting VM State in Wasm Bindgen | ||
|
|
||
| We worked upstream on the Wasm Bindgen project to implement a new [`--experimental-reset-state-function` compilation option](https://github.com/wasm-bindgen/wasm-bindgen/pull/4644) which outputs a new `__wbg_reset_state` function. | ||
|
|
||
| This function clears all internal state related to the Wasm VM, and also ensures object references are uniquely associated with the Wasm instance identity. Wasm bindgen exports stateless JS wrapper functions which call into Wasm. Updating their internal | ||
| Wasm instance binding to the new instance allows exposing the new Wasm instance without having to rebind the exported functions. | ||
mikenomitch marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| One other necessary change here was associating Wasm-created JS objects with an instance identity. If a JS object created by an earlier instance is then passed into a new instance later on, a new "stale object" error is specially thrown when using this feature. | ||
|
|
||
| ## Layered Solution | ||
|
|
||
| By implementing only the minimal primitive upstream in Wasm Bindgen necessary, we could then comprehensively solve the remaining JS state concerns for workers-rs specifically. | ||
|
|
||
| For this a proxy wrapper was needed to ensure all top-level exported class instantiations (such as for Rust Durable Objects) are fully reinitialized when resetting the Wasm instance. This is because | ||
| the workerd runtime will instantiate exported classes, which would then be associated with the Wasm instance. So tracking and reinitializing these exported classes was necessary. | ||
|
|
||
| This approach is then all that is needed to provide full panic recovery for Rust Workers. Requests in progress during a panic will provide 500 errors, but the worker will then instantly recover for future requests with an instant reset. | ||
|
|
||
| Of course we never want panics, but when they do happen they are isolated and can be investigated further from the error logs - avoiding broader service disruption. | ||
|
|
||
| ## WebAssembly Exception Handling | ||
|
|
||
| In future, we would like to see full support for recoverable panics without even needing reinitialization at all. | ||
|
|
||
| With the [WebAssembly Exception Handling](https://github.com/WebAssembly/exception-handling/blob/main/proposals/exception-handling/Exceptions.md) proposal part of the newly announced [WebAssembly 3.0](https://webassembly.org/news/2025-09-17-wasm-3.0/) specification, | ||
| it would actually be possible to fully unwind panics as normal JS errors. Concurrent requests would no longer fail at all, and all state would remain just fine. | ||
|
|
||
| This would be a larger Wasm Bindgen initiative, but a very useful one to explore further. | ||
|
|
||
| **We're making significant improvements to the reliability of [Rust Workers](https://github.com/cloudflare/workers-rs). Join us in `#rust-on-workers` on the [Cloudflare Developers Discord](https://discord.gg/cloudflaredev) to stay updated.** | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.