Add docs and example for debugging with core dumps (bytecodealliance#7087)

* Add docs for debugging with core dumps

* Fix reference to old style CLI flag

* Add `no_run` to example that is only there to trap

Author: fitzgen

.gitignore

@@ -21,3 +21,4 @@ foo
 publish
 vendor
 examples/build
+*.coredump

docs/SUMMARY.md

@@ -3,6 +3,8 @@
 - [Introduction](./introduction.md)
 - [Examples](./examples.md)
   - [Debugging WebAssembly](./examples-debugging.md)
+    - [Debugging with `gdb` and `lldb`](./examples-debugging-native-debugger.md)
+    - [Debugging with Core Dumps](./examples-debugging-core-dumps.md)
   - [Profiling WebAssembly](./examples-profiling.md)
     - [Profiling with Perf](./examples-profiling-perf.md)
     - [Profiling with VTune](./examples-profiling-vtune.md)
@@ -15,6 +17,7 @@
     - [WASI](./examples-rust-wasi.md)
     - [Linking Modules](./examples-rust-linking.md)
     - [Debugging](./examples-rust-debugging.md)
+    - [Core Dumps](./examples-rust-core-dumps.md)
     - [Using Multi-Value](./examples-rust-multi-value.md)
   - [Embedding in C](./examples-c-embed.md)
     - [Hello, World!](./examples-c-hello-world.md)

docs/examples-debugging-core-dumps.md

@@ -0,0 +1,90 @@
+# Debugging WebAssembly with Core Dumps
+
+Wasmtime can be configured to generate [the standard Wasm core dump
+format][spec] whenever guest Wasm programs trap. These core dumps can then be
+consumed by external tooling (such as [`wasmgdb`][wasmgdb]) for post-mortem analysis.
+
+This page focuses on generating and inspecting core dumps via the Wasmtime
+command-line interface. For details on how to generate core dumps via the
+`wasmtime` embedding API, see [Core Dumps in a Rust
+Embedding](./examples-rust-core-dumps.md).
+
+First, we need to compile some code to Wasm that can trap. Consider the
+following Rust code:
+
+```rust,no_run
+// trap.rs
+
+fn main() {
+    foo(42);
+}
+
+fn foo(x: u32) {
+    bar(x);
+}
+
+fn bar(x: u32) {
+    baz(x);
+}
+
+fn baz(x: u32) {
+    assert!(x != 42);
+}
+```
+
+We can compile it to Wasm with the following command:
+
+```shell-session
+$ rustc --target wasm32-wasi -o ./trap.wasm ./trap.rs
+```
+
+Next, we can run it in Wasmtime and capture a core dump when it traps:
+
+```shell-session
+$ wasmtime -D coredump=./trap.coredump ./trap.wasm
+thread 'main' panicked at /home/nick/scratch/trap.rs:14:5:
+assertion failed: x != 42
+note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
+Error: failed to run main module `/home/nick/scratch/trap.wasm`
+
+Caused by:
+    0: core dumped at /home/nick/scratch/trap.coredump
+    1: failed to invoke command default
+    2: wasm coredump generated while executing store_name:
+       modules:
+         <module>
+       instances:
+         Instance(store=1, index=1)
+       memories:
+         Memory(store=1, index=1)
+       globals:
+         Global(store=1, index=0)
+       backtrace:
+       error while executing at wasm backtrace:
+           0: 0x5961 - <unknown>!__rust_start_panic
+           1: 0x562a - <unknown>!rust_panic
+           2: 0x555d - <unknown>!std::panicking::rust_panic_with_hook::h58e7d0b3d70e95b6
+           3: 0x485d - <unknown>!std::panicking::begin_panic_handler::{{closure}}::h1853004619879cfd
+           4: 0x47bd - <unknown>!std::sys_common::backtrace::__rust_end_short_backtrace::hed32bc5557405634
+           5: 0x4f02 - <unknown>!rust_begin_unwind
+           6: 0xac01 - <unknown>!core::panicking::panic_fmt::h53ca5bf48b428895
+           7: 0xb1c5 - <unknown>!core::panicking::panic::h62c2c2bb054da7e1
+           8:  0x661 - <unknown>!trap::baz::h859f39b65389c077
+           9:  0x616 - <unknown>!trap::bar::h7ad12f9c5b730d17
+          10:  0x60a - <unknown>!trap::foo::ha69c95723611c1a0
+          11:  0x5fe - <unknown>!trap::main::hdfcd9f2d150fc3dc
+          12:  0x434 - <unknown>!core::ops::function::FnOnce::call_once::h24336e950fb97d1e
+          13:  0x40b - <unknown>!std::sys_common::backtrace::__rust_begin_short_backtrace::h2b37384d2b1a57ff
+          14:  0x4ec - <unknown>!std::rt::lang_start::{{closure}}::he86eb1b6ac6d7501
+          15: 0x24f7 - <unknown>!std::rt::lang_start_internal::h21f6a1d8f3633b54
+          16:  0x497 - <unknown>!std::rt::lang_start::h7d256f21902ff32b
+          17:  0x687 - <unknown>!__main_void
+          18:  0x3e6 - <unknown>!_start
+       note: using the `WASMTIME_BACKTRACE_DETAILS=1` environment variable may show more debugging information
+```
+
+You now have a core dump at `./trap.coredump` that can be consumed by external
+tooling to do post-mortem analysis of the failure.
+
+[spec]: https://github.com/WebAssembly/tool-conventions/blob/main/Coredump.md
+[wasmgdb]: https://github.com/xtuc/wasm-coredump/blob/main/bin/wasmgdb/README.md

docs/examples-debugging-native-debugger.md

@@ -0,0 +1,45 @@
+# Debugging with `gdb` and `lldb`
+
+The following steps describe how to use `gdb` or `lldb` to debug both the Wasm
+guest and the host (i.e. the Wasmtime CLI or your Wasmtime-embedding program) at
+the same time:
+
+1. Compile your WebAssembly with debug info enabled, usually `-g`; for
+   example:
+
+    ```sh
+    clang foo.c -g -o foo.wasm
+    ```
+
+2. Run Wasmtime with the debug info enabled; this is `-D debug-info` from the
+   CLI and `Config::debug_info(true)` in an embedding (e.g. see [debugging in a
+   Rust embedding](./examples-rust-debugging.md))
+
+3. Use a supported debugger:
+
+    ```sh
+    lldb -- wasmtime run -D debug-info foo.wasm
+    ```
+    ```sh
+    gdb --args wasmtime run -D debug-info foo.wasm
+    ```
+
+If you run into trouble, the following discussions might help:
+
+- On MacOS with LLDB you may need to run: `settings set
+  plugin.jit-loader.gdb.enable on`
+  ([#1953](https://github.com/bytecodealliance/wasmtime/issues/1953))
+
+- With LLDB, call `__vmctx.set()` to set the current context before calling any
+  dereference operators
+  ([#1482](https://github.com/bytecodealliance/wasmtime/issues/1482)):
+  ```sh
+  (lldb) p __vmctx->set()
+  (lldb) p *foo
+  ```
+
+- The address of the start of instance memory can be found in `__vmctx->memory`
+
+- On Windows you may experience degraded WASM compilation throughput due to the
+  enablement of additional native heap checks when under the debugger by default.
+  You can set the environment variable `_NO_DEBUG_HEAP` to `1` to disable them.

docs/examples-debugging.md

@@ -1,41 +1,11 @@
 # Debugging WebAssembly
 
-The following steps describe a common way to debug a WebAssembly module in
-Wasmtime:
+Wasmtime currently provides the following support for debugging misbehaving
+WebAssembly:
 
-1. Compile your WebAssembly with debug info enabled, usually `-g`; for
-   example:
+* We can [live debug and step through the guest Wasm and the host at the same
+  time with `gdb` or `lldb`.](./examples-debugging-native-debugger.md)
 
-    ```sh
-    clang foo.c -g -o foo.wasm
-    ```
-
-2. Run Wasmtime with the debug info enabled; this is `-g` from the CLI and
-   `Config::debug_info(true)` in an embedding (e.g. see [debugging in a Rust
-   embedding](./examples-rust-debugging.md))
-
-3. Use a supported debugger:
-
-    ```sh
-    lldb -- wasmtime run -D debug-info foo.wasm
-    ```
-    ```sh
-    gdb --args wasmtime run -D debug-info foo.wasm
-    ```
-
-If you run into trouble, the following discussions might help:
-
-- On MacOS with LLDB you may need to run: `settings set
-  plugin.jit-loader.gdb.enable on`
-  ([#1953](https://github.com/bytecodealliance/wasmtime/issues/1953))
-- With LLDB, call `__vmctx.set()` to set the current context before calling any
-  dereference operators
-  ([#1482](https://github.com/bytecodealliance/wasmtime/issues/1482)):
-  ```sh
-  (lldb) p __vmctx->set()
-  (lldb) p *foo
-  ```
-- The address of the start of instance memory can be found in `__vmctx->memory`
-- On Windows you may experience degraded WASM compilation throughput due to the
-  enablement of additional native heap checks when under the debugger by default.
-  You can set the environment variable `_NO_DEBUG_HEAP` to `1` to disable them.
+* When a Wasm guest traps, we can [generate Wasm core
+  dumps](./examples-debugging-core-dumps.md), that can be consumed by other
+  tools for post-mortem analysis.

docs/examples-rust-core-dumps.md

@@ -0,0 +1,19 @@
+# Core Dumps
+
+You can also [browse this source code online][code] and clone the wasmtime
+repository to run the example locally.
+
+[code]: https://github.com/bytecodealliance/wasmtime/blob/main/examples/fib-debug/main.rs
+
+This examples shows how to configure capturing [core dumps] when a Wasm guest
+traps that can then be passed to external tools (like [`wasmgdb`]) for
+post-mortem analysis.
+
+[core dumps]: https://github.com/WebAssembly/tool-conventions/blob/main/Coredump.md
+[`wasmgdb`]: https://github.com/xtuc/wasm-coredump/blob/main/bin/wasmgdb/README.md
+
+## `main.rs`
+
+```rust,ignore
+{{#include ../examples/coredump.rs}}
+```

examples/coredump.rs

@@ -0,0 +1,96 @@
+//! An example of how to configure capturing core dumps when the guest Wasm
+//! traps that can then be passed to external tools for post-mortem analysis.
+
+// You can execute this example with `cargo run --example coredump`.
+
+use anyhow::Result;
+use wasmtime::*;
+
+fn main() -> Result<()> {
+    println!("Configure core dumps to be captured on trap.");
+    let mut config = Config::new();
+    config.coredump_on_trap(true);
+    let engine = Engine::new(&config)?;
+    let mut store = Store::new(&engine, ());
+
+    println!("Define a Wasm module that will mutate local state and then trap.");
+    let module = Module::new(
+        store.engine(),
+        r#"
+            (module $trapper
+                (memory 10)
+                (global $g (mut i32) (i32.const 0))
+
+                (func (export "run")
+                    call $a
+                )
+
+                (func $a
+                    i32.const 0x1234
+                    i64.const 42
+                    i64.store
+                    call $b
+                )
+
+                (func $b
+                    i32.const 36
+                    global.set $g
+                    call $c
+                )
+
+                (func $c
+                    unreachable
+                )
+            )
+        "#,
+    )?;
+
+    println!("Instantiate the module.");
+    let instance = Instance::new(&mut store, &module, &[])?;
+
+    println!("Invoke its 'run' function.");
+    let run = instance
+        .get_func(&mut store, "run")
+        .expect("should have 'run' export");
+    let args = &[];
+    let results = &mut [];
+    let ok = run.call(&mut store, args, results);
+
+    println!("Calling that function trapped.");
+    assert!(ok.is_err());
+    let err = ok.unwrap_err();
+    assert!(err.is::<Trap>());
+
+    println!("Extract the captured core dump.");
+    let dump = err
+        .downcast_ref::<WasmCoreDump>()
+        .expect("should have an attached core dump, since we configured core dumps on");
+
+    println!(
+        "Number of memories in the core dump: {}",
+        dump.memories().len()
+    );
+    for (i, mem) in dump.memories().iter().enumerate() {
+        if let Some(addr) = mem.data(&store).iter().position(|byte| *byte != 0) {
+            let val = mem.data(&store)[addr];
+            println!("  First nonzero byte for memory {i}: {val} @ {addr:#x}");
+        } else {
+            println!("  Memory {i} is all zeroes.");
+        }
+    }
+
+    println!(
+        "Number of globals in the core dump: {}",
+        dump.globals().len()
+    );
+    for (i, global) in dump.globals().iter().enumerate() {
+        let val = global.get(&mut store);
+        println!("  Global {i} = {val:?}");
+    }
+
+    println!("Serialize the core dump and write it to ./example.coredump");
+    let serialized = dump.serialize(&mut store, "trapper.wasm");
+    std::fs::write("./example.coredump", serialized)?;
+
+    Ok(())
+}