Gdb support for mshv guests (#327)

* dbg: change visibility of debugging related functions/structs

Signed-off-by: Doru Blânzeanu <[email protected]>

* dbg: move kvm vcpu debug type to another file

- this is done in preparation of accomodating support for other hypervisors
which means some of the functionality whould be common.
- moving the debug code to a separate file can enable us to
group common behavior in traits

Signed-off-by: Doru Blânzeanu <[email protected]>

* dbg: move hardware breakpoints functionality to separate file

- also add a trait that contains all the needed methods for interacting
with a vCPU

Signed-off-by: Doru Blânzeanu <[email protected]>

* dbg: move read/write registers and get_stop_reason to vCPU functionality trait

Signed-off-by: Doru Blânzeanu <[email protected]>

* dbg: modify hw breakpoints and move common functionality in trait

Signed-off-by: Doru Blânzeanu <[email protected]>

* dbg: move read/write address and add/remove sw breakpoint to separate file

- add trait to define common behavior to be used by other hypervisors
  later

Signed-off-by: Doru Blânzeanu <[email protected]>

* dbg: remove specific method that adds entry point breakpoint

- it can be done using the generic way to add hw breakpoints
- also remove the entry point breakpoint after the vCPU stops to
avoid hanging there

Signed-off-by: Doru Blânzeanu <[email protected]>

* dbg: improve vCPU exit reason checking

- verify the debug registers that report what kind of exception was triggered
- in case there is an unknown reason, report it as a SwBp so that the gdb
client can inspect what happened

Signed-off-by: Doru Blânzeanu <[email protected]>

* dbg: prepare for mshv guest debugging implementation

Signed-off-by: Doru Blânzeanu <[email protected]>

* dbg: add basic skeleton for mshv guest debug

Signed-off-by: Doru Blânzeanu <[email protected]>

* dbg: add mshv intercepts for #DB and #BP exceptions

Signed-off-by: Doru Blânzeanu <[email protected]>

* dbg: mshv guest debugging support for vCPU operations

Signed-off-by: Doru Blânzeanu <[email protected]>

* dbg: mshv guest debugging support for memory inspection

Signed-off-by: Doru Blânzeanu <[email protected]>

* dbg: read/write addresses check offset calculation

- use checked_sub to avoid issues from addresses that could cause
underflow

Signed-off-by: Doru Blânzeanu <[email protected]>

* dbg: enable ci guest debugging tests on mshv

- change gdb commands in test to be usable with older gdb versions
- change Justfile to provide mshv3 feature
- change gdb test to invoke cargo test with correct features for mshv

Signed-off-by: Doru Blânzeanu <[email protected]>

* dbg: update documentation to specify the mshv debug support

Signed-off-by: Doru Blânzeanu <[email protected]>

* dbg: add specific logging for each debug command error encountered

Signed-off-by: Doru Blânzeanu <[email protected]>

---------

Signed-off-by: Doru Blânzeanu <[email protected]>

Author: dblnz

.github/workflows/dep_rust.yml

@@ -124,11 +124,11 @@ jobs:
         run: just run-rust-examples-linux ${{ matrix.config }} ${{ matrix.hypervisor == 'mshv3' && 'mshv3' || ''}}
 
       - name: Run Rust Gdb tests - linux
-        if: runner.os == 'Linux' && matrix.hypervisor == 'kvm'
+        if: runner.os == 'Linux'
         env:
           CARGO_TERM_COLOR: always
           RUST_LOG: debug
-        run: just test-rust-gdb-debugging ${{ matrix.config }}
+        run: just test-rust-gdb-debugging ${{ matrix.config }} ${{ matrix.hypervisor == 'mshv3' && 'mshv3' || ''}}
 
       ### Benchmarks ###
       - name: Install github-cli (Linux mariner)

Justfile

@@ -117,9 +117,9 @@ test-rust-feature-compilation-fail target=default-target:
     {{ if os() == "linux" { "! cargo check -p hyperlight-host --no-default-features 2> /dev/null"} else { "" } }}
 
 # Test rust gdb debugging
-test-rust-gdb-debugging target=default-target: (build-rust target)
-    {{ set-trace-env-vars }} cargo test --profile={{ if target == "debug" { "dev" } else { target } }} --example guest-debugging --features gdb
-    {{ set-trace-env-vars }} cargo test --profile={{ if target == "debug" { "dev" } else { target } }} --features gdb -- test_gdb
+test-rust-gdb-debugging target=default-target features="": (build-rust target)
+    {{ set-trace-env-vars }} cargo test --profile={{ if target == "debug" { "dev" } else { target } }} --example guest-debugging {{ if features =="" {'--features gdb'} else { "--features gdb," + features } }}
+    {{ set-trace-env-vars }} cargo test --profile={{ if target == "debug" { "dev" } else { target } }} {{ if features =="" {'--features gdb'} else { "--features gdb," + features } }} -- test_gdb
 
 test target=default-target: (test-rust target)
 

docs/debugging-hyperlight.md

@@ -42,3 +42,7 @@ cargo test --package hyperlight-host --test integration_test --features print_de
 To dump the details of the memory configuration, the virtual processors register state and the contents of the VM memory set the feature `crashdump` and run a debug build. This will result in a dump file being created in the temporary directory. The name and location of the dump file will be printed to the console and logged as an error message.
 
 There are no tools at this time to analyze the dump file, but it can be useful for debugging.
+
+## Debugging guests
+
+For more information on how to debug the Hyperlight guests check the following [link](./how-to-debug-a-hyperlight-guest.md).

docs/how-to-debug-a-hyperlight-guest.md

@@ -1,13 +1,13 @@
-# How to debug a Hyperlight **KVM** guest using gdb
+# How to debug a Hyperlight guest using gdb on Linux
 
-Hyperlight supports gdb debugging of a **KVM** guest running inside a Hyperlight sandbox.
-When Hyperlight is compiled with the `gdb` feature enabled, a Hyperlight KVM sandbox can be configured
+Hyperlight supports gdb debugging of a **KVM** or **MSHV** guest running inside a Hyperlight sandbox on Linux.
+When Hyperlight is compiled with the `gdb` feature enabled, a Hyperlight sandbox can be configured
 to start listening for a gdb connection.
 
 ## Supported features
 
-The Hyperlight `gdb` feature enables **KVM** guest debugging:
-   - an entry point breakpoint is automatically set for the guest to stop
+The Hyperlight `gdb` feature enables **KVM** and **MSHV** guest debugging to:
+   - stop at an entry point breakpoint which is automatically set by Hyperlight
    - add and remove HW breakpoints (maximum 4 set breakpoints at a time)
    - add and remove SW breakpoints
    - read and write registers
@@ -18,7 +18,7 @@ The Hyperlight `gdb` feature enables **KVM** guest debugging:
 ## Expected behavior
 
 Below is a list describing some cases of expected behavior from a gdb debug 
-session of a guest binary running inside a KVM Hyperlight sandbox.
+session of a guest binary running inside a Hyperlight sandbox on Linux.
 
 - when the `gdb` feature is enabled and a SandboxConfiguration is provided a
   debug port, the created sandbox will wait for a gdb client to connect on the
@@ -154,7 +154,7 @@ is sent over the communication channel to the hypervisor handler for the sandbox
 to resolve.
 
 Below is a sequence diagram that shows the interaction between the entities
-involved in the gdb debugging of a Hyperlight guest running inside a KVM sandbox.
+involved in the gdb debugging of a Hyperlight guest running inside a **KVM** or **MSHV** sandbox.
 
 ```
                                ┌───────────────────────────────────────────────────────────────────────────────────────────────┐

src/hyperlight_host/build.rs

@@ -89,7 +89,7 @@ fn main() -> Result<()> {
     // Essentially the kvm and mshv features are ignored on windows as long as you use #[cfg(kvm)] and not #[cfg(feature = "kvm")].
     // You should never use #[cfg(feature = "kvm")] or #[cfg(feature = "mshv")] in the codebase.
     cfg_aliases::cfg_aliases! {
-        gdb: { all(feature = "gdb", debug_assertions, feature = "kvm", target_os = "linux") },
+        gdb: { all(feature = "gdb", debug_assertions, any(feature = "kvm", feature = "mshv2", feature = "mshv3"), target_os = "linux") },
         kvm: { all(feature = "kvm", target_os = "linux") },
         mshv: { all(any(feature = "mshv2", feature = "mshv3"), target_os = "linux") },
         // inprocess feature is aliased with debug_assertions to make it only available in debug-builds.

src/hyperlight_host/examples/guest-debugging/main.rs

@@ -107,7 +107,7 @@ mod tests {
 
                 set pagination off
                 set logging file {out_file_path}
-                set logging enabled on
+                set logging on
 
                 break hyperlight_main
                     commands
@@ -118,7 +118,7 @@ mod tests {
 
                 continue
 
-                set logging enabled off
+                set logging off
                 quit
             "
             )
@@ -134,12 +134,17 @@ mod tests {
         write_cmds_file(&cmd_file_path, &out_file_path)
             .expect("Failed to write gdb commands to file");
 
+        #[cfg(mshv3)]
+        let features = "gdb,mshv3";
+        #[cfg(not(mshv3))]
+        let features = "gdb";
+
         let mut guest_child = Command::new("cargo")
             .arg("run")
             .arg("--example")
             .arg("guest-debugging")
             .arg("--features")
-            .arg("gdb")
+            .arg(features)
             .stdin(Stdio::piped())
             .stdout(Stdio::piped())
             .spawn()

src/hyperlight_host/src/hypervisor/gdb/event_loop.rs

@@ -16,14 +16,15 @@ limitations under the License.
 
 use gdbstub::common::Signal;
 use gdbstub::conn::ConnectionExt;
-use gdbstub::stub::run_blocking::{self, WaitForStopReasonError};
-use gdbstub::stub::{BaseStopReason, DisconnectReason, GdbStub, SingleThreadStopReason};
+use gdbstub::stub::{
+    run_blocking, BaseStopReason, DisconnectReason, GdbStub, SingleThreadStopReason,
+};
 use libc::{pthread_kill, SIGRTMIN};
 
 use super::x86_64_target::HyperlightSandboxTarget;
 use super::{DebugResponse, GdbTargetError, VcpuStopReason};
 
-pub struct GdbBlockingEventLoop;
+struct GdbBlockingEventLoop;
 
 impl run_blocking::BlockingEventLoop for GdbBlockingEventLoop {
     type Connection = Box<dyn ConnectionExt<Error = std::io::Error>>;
@@ -57,13 +58,10 @@ impl run_blocking::BlockingEventLoop for GdbBlockingEventLoop {
                             signal: Signal(SIGRTMIN() as u8),
                         },
                         VcpuStopReason::Unknown => {
-                            log::warn!("Unknown stop reason - resuming execution");
+                            log::warn!("Unknown stop reason received");
 
-                            target
-                                .resume_vcpu()
-                                .map_err(WaitForStopReasonError::Target)?;
-
-                            continue;
+                            // Marking as a SwBreak so the gdb inspect where/why it stopped
+                            BaseStopReason::SwBreak(())
                         }
                     };
 
@@ -115,7 +113,7 @@ impl run_blocking::BlockingEventLoop for GdbBlockingEventLoop {
     }
 }
 
-pub fn event_loop_thread(
+pub(crate) fn event_loop_thread(
     debugger: GdbStub<HyperlightSandboxTarget, Box<dyn ConnectionExt<Error = std::io::Error>>>,
     target: &mut HyperlightSandboxTarget,
 ) {