Debug: implement breakpoints and single-stepping. (#12133)

* Debug: implement breakpoints and single-stepping.

This is a PR that puts together a bunch of earlier pieces (patchable
calls in #12061 and #12101, private copies of code in #12051, and all
the prior debug event and instrumentation infrastructure) to implement
breakpoints in the guest debugger.

These are implemented in the way we have planned in #11964: each
sequence point (location prior to a Wasm opcode) is now a patchable call
instruction, patched out (replaced with NOPs) by default. When patched
in, the breakpoint callsite calls a trampoline with the `patchable` ABI
which then invokes the `breakpoint` hostcall. That hostcall emits the
debug event and nothing else.

A few of the interesting bits in this PR include:
- Implementations of "unpublish" (switch permissions back to read/write
  from read/execute) for mmap'd code memory on all our platforms.
- Infrastructure in the frame-tables (debug info) metadata producer and
  parser to record "breakpoint patches".
- A tweak to the NOP metadata packaged with the `MachBuffer` to allow
  multiple NOP sizes. This lets us use one 5-byte NOP on x86-64, for
  example (did you know x86-64 had these?!) rather than five 1-byte
  NOPs.

This PR also implements single-stepping with a global-per-`Store` flag,
because at this point why not; it's a small additional bit of logic to
do *all* patches in all modules registered in the `Store` when that flag
is enabled.

A few realizations for future work:
- The need for an introspection API available to a debugger to see the
  modules within a component is starting to become clear; either that,
  or the "module and PC" location identifier for a breakpoint switches
  to a "module or component" sum type. Right now, the tests for this
  feature use only core modules. Extending to components should not
  actually be hard at all, we just need to build the API for it.
- The interaction between inlining and `patchable_call` is interesting:
  what happens if we inline a `patchable_call` at a `try_call` callsite?
  Right now, we do *not* update the `patchable_call` to a `try_call`,
  because there is no `patchable_try_call`; this is fine in the Wasmtime
  embedding in practice because we never (today!) throw exceptions from
  a breakpoint handler. This does suggest to me that maybe we should
  make patchability a property of any callsite, and allow try-calls to
  be patchable too (with the same restriction about no return values as
  the only restriction); but happy to discuss that one further.

* Add missing debug.wat disas test.

* Review feedback.

* Fix comment on `CodeMemory::text_mut`.

* Review feedback.

* Review feedback: abort process on failure to re-apply executable permissions.

* Implement icache flush for aarch64.

This appears to be necessary as we otherwise see a failure in CI on
macOS/aarch64 that is consistent with patched-in breakpoint calls still
being incorrectly cached after we remove them and republish the code.

There is a longstanding issue in #3310 tracking proper icache coherence
handling on aarch64. We implemented this for Linux with the `membarrier`
syscall but never did so for macOS. Maybe this is the first point at
which it matters, because code was always loaded at new addresses (hence
did not have coherence issues because nothing would have been cached)
previously.

prtest:full

* Review feedback: use `next_multiple_of`.

Author: cfallin

Cargo.lock

@@ -79,6 +79,7 @@ capstone = { workspace = true, optional = true }
 termcolor = { workspace = true, optional = true }
 gimli = { workspace = true, optional = true }
 pulley-interpreter = { workspace = true, optional = true }
+smallvec = { workspace = true }
 
 async-trait = { workspace = true }
 bytes = { workspace = true }

Cargo.toml

@@ -227,6 +227,14 @@ pub(crate) fn do_inlining(
                     // Can't inline indirect calls; need to have some earlier
                     // pass rewrite them into direct calls first, when possible.
                 }
+                ir::InstructionData::Call {
+                    opcode: ir::Opcode::PatchableCall,
+                    ..
+                } => {
+                    // Can't inline patchable calls; they need to
+                    // remain patchable and inlining the whole body is
+                    // decidedly *not* patchable!
+                }
                 _ => {
                     debug_assert!(
                         !cursor.func.dfg.insts[inst].opcode().is_call(),
@@ -497,7 +505,19 @@ fn inline_one(
                         call_opcode == ir::Opcode::TryCall,
                         call_exception_table.is_some()
                     );
-                    if call_opcode == ir::Opcode::TryCall {
+                    // Note that we do not fix up patchable calls
+                    // inlined at a try-call to a try-call, because
+                    // the "patchable ABI" does not support catching
+                    // exceptions. This does mean that we cannot have
+                    // an exception-throw propagate out of a
+                    // breakpoint when we use patchable calls to set
+                    // up breakpoints, but we don't expect that to
+                    // occur.
+                    //
+                    // FIXME: consider making patchability an aspect
+                    // of any call; then we can remove this special
+                    // case.
+                    if call_opcode == ir::Opcode::TryCall && opcode != ir::Opcode::PatchableCall {
                         allocs
                             .calls_needing_exception_table_fixup
                             .push(inlined_inst);

cranelift/codegen/src/inline.rs

@@ -1111,8 +1111,8 @@ impl MachInst for Inst {
         Inst::Nop4
     }
 
-    fn gen_nop_unit() -> SmallVec<[u8; 8]> {
-        smallvec![0x1f, 0x20, 0x03, 0xd5]
+    fn gen_nop_units() -> Vec<Vec<u8>> {
+        vec![vec![0x1f, 0x20, 0x03, 0xd5]]
     }
 
     fn rc_for_type(ty: Type) -> CodegenResult<(&'static [RegClass], &'static [Type])> {

cranelift/codegen/src/isa/aarch64/inst/mod.rs

@@ -10,6 +10,8 @@ use crate::isa::pulley_shared::abi::PulleyMachineDeps;
 use crate::{CodegenError, CodegenResult, settings};
 use crate::{machinst::*, trace};
 use alloc::string::{String, ToString};
+use alloc::vec;
+use alloc::vec::Vec;
 use regalloc2::RegClass;
 use smallvec::SmallVec;
 
@@ -535,14 +537,14 @@ where
         todo!()
     }
 
-    fn gen_nop_unit() -> SmallVec<[u8; 8]> {
-        let mut bytes = smallvec::smallvec![];
+    fn gen_nop_units() -> Vec<Vec<u8>> {
+        let mut bytes = vec![];
         let nop = pulley_interpreter::op::Nop {};
         nop.encode(&mut bytes);
         // NOP needs to be a 1-byte opcode so it can be used to
         // overwrite a callsite of any length.
         assert_eq!(bytes.len(), 1);
-        bytes
+        vec![bytes]
     }
 
     fn rc_for_type(ty: Type) -> CodegenResult<(&'static [RegClass], &'static [Type])> {

cranelift/codegen/src/isa/pulley_shared/inst/mod.rs

@@ -811,8 +811,8 @@ impl MachInst for Inst {
         Inst::Nop4
     }
 
-    fn gen_nop_unit() -> SmallVec<[u8; 8]> {
-        smallvec![0x13, 0x00, 0x00, 0x00]
+    fn gen_nop_units() -> Vec<Vec<u8>> {
+        vec![vec![0x13, 0x00, 0x00, 0x00]]
     }
 
     fn rc_for_type(ty: Type) -> CodegenResult<(&'static [RegClass], &'static [Type])> {

cranelift/codegen/src/isa/riscv64/inst/mod.rs

@@ -1164,8 +1164,8 @@ impl MachInst for Inst {
         }
     }
 
-    fn gen_nop_unit() -> SmallVec<[u8; 8]> {
-        smallvec::smallvec![0x07, 0x07]
+    fn gen_nop_units() -> Vec<Vec<u8>> {
+        vec![vec![0x07, 0x07]]
     }
 
     fn rc_for_type(ty: Type) -> CodegenResult<(&'static [RegClass], &'static [Type])> {

cranelift/codegen/src/isa/s390x/inst/mod.rs

@@ -11,6 +11,8 @@ use crate::isa::{CallConv, FunctionAlignment};
 use crate::{CodegenError, CodegenResult, settings};
 use crate::{machinst::*, trace};
 use alloc::boxed::Box;
+use alloc::vec;
+use alloc::vec::Vec;
 use core::slice;
 use cranelift_assembler_x64 as asm;
 use smallvec::{SmallVec, smallvec};
@@ -1308,6 +1310,7 @@ impl MachInst for Inst {
         match self {
             Inst::CallKnown { .. }
             | Inst::CallUnknown { .. }
+            | Inst::PatchableCallKnown { .. }
             | Inst::ElfTlsGetAddr { .. }
             | Inst::MachOTlsGetAddr { .. } => CallType::Regular,
 
@@ -1391,8 +1394,13 @@ impl MachInst for Inst {
         Inst::nop(std::cmp::min(preferred_size, 9) as u8)
     }
 
-    fn gen_nop_unit() -> SmallVec<[u8; 8]> {
-        smallvec![0x90]
+    fn gen_nop_units() -> Vec<Vec<u8>> {
+        vec![
+            // Standard 1-byte NOP.
+            vec![0x90],
+            // 5-byte NOP useful for patching out patchable calls.
+            vec![0x0f, 0x1f, 0x44, 0x00, 0x00],
+        ]
     }
 
     fn rc_for_type(ty: Type) -> CodegenResult<(&'static [RegClass], &'static [Type])> {

cranelift/codegen/src/isa/x64/inst/mod.rs

@@ -355,7 +355,7 @@ impl MachBufferFinalized<Stencil> {
             unwind_info: self.unwind_info,
             alignment: self.alignment,
             frame_layout: self.frame_layout,
-            nop: self.nop,
+            nop_units: self.nop_units,
         }
     }
 }
@@ -406,7 +406,10 @@ pub struct MachBufferFinalized<T: CompilePhase> {
     /// This allows a consumer of a `MachBufferFinalized` to disable
     /// patchable call sites (which are enabled by default) without
     /// specific knowledge of the target ISA.
-    pub nop: SmallVec<[u8; 8]>,
+    ///
+    /// Each entry is one form of nop, and these are required to be
+    /// sorted in ascending-size order.
+    pub nop_units: Vec<Vec<u8>>,
 }
 
 const UNKNOWN_LABEL_OFFSET: CodeOffset = 0xffff_ffff;
@@ -1586,7 +1589,7 @@ impl<I: VCodeInst> MachBuffer<I> {
             unwind_info: self.unwind_info,
             alignment,
             frame_layout: self.frame_layout,
-            nop: I::gen_nop_unit(),
+            nop_units: I::gen_nop_units(),
         }
     }
 
@@ -1841,6 +1844,12 @@ impl<T: CompilePhase> MachBufferFinalized<T> {
         &self.data[..]
     }
 
+    /// Get a mutable slice of the code bytes, allowing patching
+    /// post-passes.
+    pub fn data_mut(&mut self) -> &mut [u8] {
+        &mut self.data[..]
+    }
+
     /// Get the list of external relocations for this code.
     pub fn relocs(&self) -> &[FinalizedMachReloc] {
         &self.relocs[..]
@@ -1900,10 +1909,7 @@ impl<T: CompilePhase> MachBufferFinalized<T> {
     /// region is guaranteed to be an integer multiple of that NOP
     /// unit size.)
     pub fn patchable_call_sites(&self) -> impl Iterator<Item = &MachPatchableCallSite> + '_ {
-        self.patchable_call_sites.iter().map(|call_site| {
-            debug_assert!(call_site.len as usize % self.nop.len() == 0);
-            call_site
-        })
+        self.patchable_call_sites.iter()
     }
 }
 

cranelift/codegen/src/machinst/buffer.rs

@@ -177,9 +177,9 @@ pub trait MachInst: Clone + Debug {
     /// the instruction must have a nonzero size if preferred_size is nonzero.
     fn gen_nop(preferred_size: usize) -> Self;
 
-    /// The smallest possible NOP, as a unit that can be used to patch
-    /// out code.
-    fn gen_nop_unit() -> SmallVec<[u8; 8]>;
+    /// The various kinds of NOP, with size, sorted in ascending-size
+    /// order.
+    fn gen_nop_units() -> Vec<Vec<u8>>;
 
     /// Align a basic block offset (from start of function).  By default, no
     /// alignment occurs.