Debugger: add top-level crate and async wrapper allowing for a "debugger environment". (#12183)

* Debugger: add top-level crate and async wrapper allowing for a "debug environment".

The debug support in Wasmtime so far is structured around async
callbacks that occur at certain kinds of events, like breakpoints. This
is a suitable foundation but makes for an awkward implementation of a
top-level debugger implementation, which likely has an event loop
dealing with user commands (via a UI or a protocol connection) and
expects to perform actions such as "run until next breakpoint".

This PR introduces a new crate that wraps a `Store` in a `Debugger`.
This wrapper embodies an inner async body that can perform whatever
actions it likes on the `Store` that is passed back in. This inner body
is spawned as an async task. The debugger wrapper registers its own
`DebugHandler` callback that communicates with the outside world via
bidirectional command/response queues.

On the "outside", the `Debugger` presents an interface suitable for
inserting into a debug protocol server or UI: an async method that runs
until next event and returns that event, and a method that permits
querying or modifying the store whenever the `run` method is not
executing. The latter operates by sending a closure over the queue,
because the `Store` must continue to be owned by the async task that is
(still) running and suspended in async callbacks.

Right now, this is exercised only via a few unit tests, but the intent
is to next build up the "top half" of the debugger using this
abstraction, e.g. by running a gdbstub protocol server (likely as a Wasm
component in a "debug-main WIT world" -- RFC needed for this).

Also, when we eventually move debugging over to native use of
`run_concurrent`, this paradigm should remain mostly unchanged at this
level of API: there can still be an object that has an async method that
runs and yields the next event, and there can still be a method that
takes a closure that can operate (within its scope only) on the `Store`.

A few warts that I could use feedback on:

- Cancelation safety is weird. Fibers panic when dropped before
  execution of their body completes, and this seems to mean that we
  can't allow a `Debugger` to drop early (or at least, the `tokio::test`
  unit test that owns the runtime that runs the async task to finish
  before the debugged body completes!). If there is a better way to
  handle cancelation safety here, I'm all ears.

- It's not clear to me if the boxed-closure-and-`Any` approach to
  providing access to the `Store` is the best we can do, but I suspect
  it is.

* Cancel safety!

* Add new crate to publish.rs script.

* Review feedback.

* Review feedback: state diagram.

* Update after merge from main: new DebugEvent for epoch yields.

* Make Debugger drop-safe by making debug event callbacks compatible with fiber teardown.

* CI fix on `debugger` crate manifest.

- Do not link to non-existent README in new crate.
- Remove a few other attributes that our internal crates don't have (now
  the set of attributes at the top level is the same as for the new
  error crate).

Author: cfallin

Cargo.lock

@@ -164,6 +164,7 @@ members = [
   "crates/wasi-preview1-component-adapter",
   "crates/wasi-preview1-component-adapter/verify",
   "crates/wasi-tls-nativetls",
+  "crates/debugger",
   "crates/wizer/fuzz",
   "crates/wizer/tests/regex-test",
   "crates/wizer/benches/regex-bench",
@@ -276,6 +277,7 @@ wasmtime-jit-icache-coherence = { path = "crates/jit-icache-coherence", version
 wasmtime-wit-bindgen = { path = "crates/wit-bindgen", version = "=41.0.0", package = 'wasmtime-internal-wit-bindgen' }
 wasmtime-math = { path = "crates/math", version = "=41.0.0", package = 'wasmtime-internal-math'  }
 wasmtime-unwinder = { path = "crates/unwinder", version = "=41.0.0", package = 'wasmtime-internal-unwinder' }
+wasmtime-debugger = { path = "crates/debugger", version = "=41.0.0", package = "wasmtime-internal-debugger" }
 wasmtime-wizer = { path = "crates/wizer", version = "41.0.0" }
 
 # Miscellaneous crates without a `wasmtime-*` prefix in their name but still

Cargo.toml

@@ -0,0 +1,27 @@
+[package]
+name = "wasmtime-internal-debugger"
+authors.workspace = true
+version.workspace = true
+description = "INTERNAL: Wasmtime's guest-debugger functionality"
+license = "Apache-2.0 WITH LLVM-exception"
+keywords = ["debugger"]
+edition.workspace = true
+rust-version.workspace = true
+
+[lints]
+workspace = true
+
+[dependencies]
+wasmtime = { workspace = true, features = ["debug", "std", "async"] }
+tokio = { workspace = true, features = ["rt", "sync", "macros"] }
+anyhow = { workspace = true }
+log = { workspace = true }
+
+[dev-dependencies]
+# Depend on `wasmtime` again to get `cranelift` and `wat` so we can
+# write unit tests that build modules from textual WAT.
+wasmtime = { workspace = true, features = ["cranelift", "wat"] }
+env_logger = { workspace = true }
+
+[features]
+default = []