Unexpected VM Exit

[Magicloud]

What happened?

Following README, built on Linux, and cargo run --example hello-world. I got Error: Error("Unexpected VM Exit \"Unexpected KVM Exit Shutdown\""). kvmexit showed that:

PID      TID      KVM_EXIT_REASON                     COUNT
1687589  1688748  TRIPLE_FAULT                        1
1687589  1688748  EPT_VIOLATION                       6

What did you expect to happen?

Example running with some output.

Steps to reproduce the behavior

1. Clone this repo
2. vim Justfile to comment all lines contains "msvc".
3. just build
4. just rg
5. cargo run --example hello-world

Hyperlight Version

Git commit: c89f72d

OS version

On Linux:
$ cat /etc/os-release
ANSI_COLOR="0;38;2;126;186;228"
BUG_REPORT_URL="https://github.com/NixOS/nixpkgs/issues"
BUILD_ID="25.05pre782401.2631b0b7abce"
CPE_NAME="cpe:/o:nixos:nixos:25.05"
DEFAULT_HOSTNAME=nixos
DOCUMENTATION_URL="https://nixos.org/learn.html"
HOME_URL="https://nixos.org/"
ID=nixos
ID_LIKE=""
IMAGE_ID=""
IMAGE_VERSION=""
LOGO="nix-snowflake"
NAME=NixOS
PRETTY_NAME="NixOS 25.05 (Warbler)"
SUPPORT_URL="https://nixos.org/community.html"
VARIANT=""
VARIANT_ID=""
VENDOR_NAME=NixOS
VENDOR_URL="https://nixos.org/"
VERSION="25.05 (Warbler)"
VERSION_CODENAME=warbler
VERSION_ID="25.05"


$ uname -a
Linux nixos 6.12.20 #1-NixOS SMP PREEMPT_DYNAMIC Sat Mar 22 19:54:28 UTC 2025 x86_64 GNU/Linux

Additional Information

No response

[devigned]

@Magicloud did you sort out what the problem was?

[Magicloud]

@devigned No.

It works well in Windows and WSL. In NixOS, I cannot built if not comment out msvc lines (even I copied Windows msvcrt.lib and oldnames.lib, it still reported SUBSYSTEM unknown). I do not know if those are the key to the issue.

I also noticed some warning during building that "cc" in NixOS is a wraper, and some options are ignored. It could be the cause. But I am not sure.

[syntactically]

@Magicloud I have used Hyperlight on NixOS before. I use this flake.nix:

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  inputs.nixpkgs-mozilla.url = "github:mozilla/nixpkgs-mozilla/master";
  outputs = { self, nixpkgs, nixpkgs-mozilla, ... } @ inputs:
{
    devShells.x86_64-linux.default = with import nixpkgs { system = "x86_64-linux"; overlays = [ (import (nixpkgs-mozilla + "/rust-overlay.nix")) ]; }; let
      stable = rustChannelOf {
        date = "2025-02-20";
        channel = "stable";
        sha256 = "sha256-AJ6LX/Q/Er9kS15bn9iflkUwcgYqRQxiOIL2ToVAXaU=";
      };
      nightly = rustChannelOf {
        date = "2025-02-20";
        channel = "nightly";
        sha256 = "sha256-/FZ/oWuPrugpO0AYM22eGFaqIxEbr5r6rHO+MH8sz0M=";
      };
      targets = [
        "x86_64-unknown-linux-gnu"
        "x86_64-pc-windows-msvc"
        "x86_64-unknown-none"
        "wasm32-wasip1"
      ];
      rust_stable = stable.rust.override { inherit targets; };
      rust_nightly = nightly.rust.override { inherit targets; };
      rust-platform = makeRustPlatform {
        cargo = rust_stable;
        rustc = rust_stable // {
          targetPlatforms = [ "x86_64-linux" ];
          badTargetPlatforms = [];
        };
      };
      buildRustPackage = rust-platform.buildRustPackage.override {
        stdenv = clangStdenv;
      };
    in (buildRustPackage rec {
      pname = "hyperlight";
      version = "0.0.0";
      src = lib.cleanSource ./.;
      cargoHash = "sha256-3smh8AUWiKHR4nKrqQI03B5DR8qSK8j2dqXuQ/jLWVE=";
      nativeBuildInputs = [
        azure-cli
        just
        dotnet-sdk_8
        clang
        llvmPackages_18.llvm
        gh
        lld
        valgrind
        pkg-config
        ffmpeg
        mkvtoolnix
        wasm-tools
      ];
      buildInputs = [
        pango
        cairo
      ];
      auditable = false;
      KVM_SHOULD_BE_PRESENT = "true";
      LIBCLANG_PATH = "${pkgs.llvmPackages_18.libclang.lib}/lib";
      shellHook = ''
        rustc_nightly() {
          (export PATH="${rust_nightly}/bin/:$PATH"; ${rust_nightly}/bin/rustc "$@")
        }
        rustc_stable() {
          (export PATH="${rust_stable}/bin/:$PATH"; ${rust_stable}/bin/rustc "$@")
        }
        cargo_nightly() {
          (export PATH="${rust_nightly}/bin/:$PATH"; ${rust_nightly}/bin/cargo "$@")
        }
        cargo_stable() {
          (export PATH="${rust_stable}/bin/:$PATH"; ${rust_stable}/bin/cargo "$@")
        }
      '';
    }).overrideAttrs(oA: {
      hardeningDisable = [ "all" ];
    });
  };
}

and this flake.lock:

{
  "nodes": {
    "nixpkgs": {
      "locked": {
        "lastModified": 1744463964,
        "narHash": "sha256-LWqduOgLHCFxiTNYi3Uj5Lgz0SR+Xhw3kr/3Xd0GPTM=",
        "owner": "NixOS",
        "repo": "nixpkgs",
        "rev": "2631b0b7abcea6e640ce31cd78ea58910d31e650",
        "type": "github"
      },
      "original": {
        "owner": "NixOS",
        "ref": "nixos-unstable",
        "repo": "nixpkgs",
        "type": "github"
      }
    },
    "nixpkgs-mozilla": {
      "locked": {
        "lastModified": 1744624473,
        "narHash": "sha256-S6zT/w5SyAkJ//dYdjbrXgm+6Vkd/k7qqUl4WgZ6jjk=",
        "owner": "mozilla",
        "repo": "nixpkgs-mozilla",
        "rev": "2292d4b35aa854e312ad2e95c4bb5c293656f21a",
        "type": "github"
      },
      "original": {
        "owner": "mozilla",
        "ref": "master",
        "repo": "nixpkgs-mozilla",
        "type": "github"
      }
    },
    "root": {
      "inputs": {
        "nixpkgs": "nixpkgs",
        "nixpkgs-mozilla": "nixpkgs-mozilla"
      }
    }
  },
  "root": "root",
  "version": 7
}

instead of the tool installation procedure in the README, so you might want to try them?

That said, this sounds like it's probably a deeper problem in our paging somewhere; it's somewhat concerning that we are unable to reproduce. Would you be willing to share some details about the hardware that you're using?

[Magicloud]

@syntactically Thanks. Just gave it a try. Updated the SHA in flake to latest git commit. It failed the same as I did before, at just rg step:

   Compiling callbackguest v0.1.0 (/home/magicloud/src/Git/hyperlight/src/tests/rust_guests/callbackguest)
error: linking with `rust-lld` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/nix/store/zqbz1qbylqlk7z60afkznchz9nl3bwrl-rust-1.85.0-2025-02-17-4d91de4e4/lib/rustlib/x86_64-unknown-linux-gnu/bin:/nix/store/pv8ackl9m0j6yrb98sv11gcf0xr91rrq-python3.12-azure-cli-2.71.0/bin:/nix/store/hr7z6wz0lafjc4dxshfmwcscyggw2c1m-python3.12-antlr4-python3-runtime-4.13.2/bin:/nix/store/f2krmq3iv5nibcvn4rw7nrnrciqprdkh-python3-3.12.9/bin:/nix/store/zjqsdznfnnzndwj01jwqhmwxz7rmxwzb-python3.12-charset-normalizer-3.4.1/bin:/nix/store/dds9yi6lc5zh41n5k4qnijq4xjk7069v-python3.12-argcomplete-3.5.3/bin:/nix/store/qhq3bs52xgccxcgdkimkfnlsihmjvqpm-python3.12-distro-1.9.0/bin:/nix/store/10f1pskdfvj5f5cal1bd2alr2k1wb55f-python3.12-humanfriendly-10.0/bin:/nix/store/wyl3l3iyql0nlbx47pmmdds0mjnf8qr4-python3.12-jmespath-1.0.1/bin:/nix/store/bh3mfvhgh1h7aycaf4pz47wl9ym7ax4h-python3.12-pygments-2.19.1/bin:/nix/store/zz7pyhkz57yr9jc32rnzryi6ar062qqm-python3.12-tabulate-0.9.0/bin:/nix/store/qm868va7rkd37g0qrk0yzv0ywn9jr26c-python3.12-pkginfo-1.12.1.2/bin:/nix/store/chxmq43cm5fly6xdqjpyzc43dgbkpja5-python3.12-chardet-5.2.0/bin:/nix/store/isi9598400qvvjpxnmk2nmp7bjf5cy0i-python3.12-fabric-3.2.2/bin:/nix/store/qr1ll140lbx261p2h8azgy57cmqfff9y-python3.12-invoke-2.2.0/bin:/nix/store/vhrdlgk0mf84pdpv51axrn36nx60v8rh-python3.12-jsondiff-2.2.1/bin:/nix/store/nib2yzakc7q605ap7rmnxfs4p7vnfjdf-python3.12-semver-3.0.4/bin:/nix/store/c12ssawmdxi5b118a97dk3584d9pj9qf-python3.12-sshtunnel-0.4.0/bin:/nix/store/l6l1b49kmazvxlrb2zwsbbh6vb9l8g91-python3.12-websocket-client-1.8.0/bin:/nix/store/37kp3wrwd3hib68vqg3h2f6sfi1g8cq7-just-1.40.0/bin:/nix/store/yjg0bi4zf6f15kv11xcyq9f0wkw9c8ai-dotnet-sdk-wrapped-8.0.408/bin:/nix/store/q87df3dw0x7r1k9gagir9a5sdi3gmw4a-clang-wrapper-19.1.7/bin:/nix/store/qla374n3avx7nzaw2kvq6wj9y4agiw1l-clang-19.1.7/bin:/nix/store/nq61v7a601gjndijq5nndprkzpwz4q9g-glibc-2.40-66-bin/bin:/nix/store/yh6qg1nsi5h2xblcr67030pz58fsaxx3-coreutils-9.6/bin:/nix/store/b1y4f1xab6svz8in8azp7c8bl5r6sx2s-binutils-wrapper-2.43.1/bin:/nix/store/s40y31bdn82sj6daaxid1bn3p7la03lv-binutils-2.43.1/bin:/nix/store/0v0hyvw4aq7a71vbn1wrgn1ij39hna16-llvm-18.1.8-dev/bin:/nix/store/k6p7j4dfvzhdb844vgqjsizbvlmprkzq-ncurses-6.5-dev/bin:/nix/store/80l1sb3vcmrkcdd7ihlizkcnv19rq9fj-ncurses-6.5/bin:/nix/store/zhxadq2h983jzmc5v3876d1k5lf6yllz-llvm-18.1.8/bin:/nix/store/a42dil5xrgf74ir5nnryhyg4v0ymkpi0-gh-2.69.0/bin:/nix/store/6gxr0kllxxkv24hva8k27x1cqrs6d454-lld-19.1.7/bin:/nix/store/cawr37dyifswi0zfkx1gygc7h16489g4-valgrind-3.24.0/bin:/nix/store/2ijmf0rg27gn78widiljfx9m7cfggjm0-pkg-config-wrapper-0.29.2/bin:/nix/store/gjzf3dca4v804mx46z1xnr2vqmi8r90n-ffmpeg-7.1.1-bin/bin:/nix/store/9cj05gmzhgn5jzfvcb2rknr2dd52ia6j-mkvtoolnix-91.0/bin:/nix/store/cg53g4m948fzqhhsxd1nh1lj9zzd31z7-wasm-tools-1.228.0/bin:/nix/store/zqbz1qbylqlk7z60afkznchz9nl3bwrl-rust-1.85.0-2025-02-17-4d91de4e4/bin:/nix/store/nk427kbyf9l6fpfdxb8hmrq2nlkl8nfl-patchelf-0.15.0/bin:/nix/store/yh6qg1nsi5h2xblcr67030pz58fsaxx3-coreutils-9.6/bin:/nix/store/xq5f95pp297afc2xjgrmhmf9w631qp7m-findutils-4.10.0/bin:/nix/store/wj1c3gsiajabnq50ifxqnlv60i5rhqj7-diffutils-3.10/bin:/nix/store/xhql0ilzbiqwnmz4z8y0phk611wynxf2-gnused-4.9/bin:/nix/store/40yifhx34v4g4llrdn3v2ag8w02j10fv-gnugrep-3.11/bin:/nix/store/r66s4mjwqi7zkqqyry5yhhv0wm5rw96a-gawk-5.3.1/bin:/nix/store/hfkclq54gppdb9fshnkwfdph1safyvli-gnutar-1.35/bin:/nix/store/4q2sv4sqb7y8y4pp3n9bf2aabmgqmwmy-gzip-1.13/bin:/nix/store/pcf18502sz0b25qya2lzm5f7wy0pd7a6-bzip2-1.0.8-bin/bin:/nix/store/zzgldw3dwyhzn7sw9cyq7d7k57h0yx46-gnumake-4.4.1/bin:/nix/store/58br4vk3q5akf4g8lx0pqzfhn47k3j8d-bash-5.2p37/bin:/nix/store/lf3ln60nya8xmkymabap9i7gqj60k9py-patch-2.7.6/bin:/nix/store/f246qzchy7zy00z1f42dqz9jiwcf5n7f-xz-5.6.4-bin/bin:/nix/store/hyp4b779nh5y4m48zvvhd4y3z1gm8hd5-file-5.46/bin:/home/magicloud/.cargo/bin:/home/magicloud/.local/bin:/run/wrappers/bin:/home/magicloud/.nix-profile/bin:/nix/profile/bin:/home/magicloud/.local/state/nix/profile/bin:/etc/profiles/per-user/magicloud/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin" VSLANG="1033" "rust-lld" "-flavor" "link" "/NOLOGO" "/tmp/nix-shell.VM11x1/rustcql0I5Q/symbols.o" "<9 object files omitted>" "/home/magicloud/src/Git/hyperlight/src/tests/rust_guests/callbackguest/target/x86_64-pc-windows-msvc/debug/deps/{libhyperlight_guest-460b0c401cbe2c77.rlib,libserde_json-09086660906fe6fe.rlib,libmemchr-2298fecb9570289b.rlib,libitoa-a756b307f5bb1722.rlib,libryu-c69c17daddba31a6.rlib,libserde-f7d8120d2249015d.rlib,libspin-df1e7d83f2863e68.rlib,libhyperlight_common-bbf6af34a9ac9a32.rlib,libstrum-0f2c9429b394f43a.rlib,liblog-ef357f236da06391.rlib,libanyhow-e6e393e97a300ed7.rlib,libflatbuffers-93690509da9d672f.rlib,libbitflags-28b634de4aeb28b0.rlib,libbuddy_system_allocator-711404ff7b32e300.rlib,libspin-576e1c4a1698ece4.rlib,liblock_api-2dd6bdb46ee6ccf4.rlib,libscopeguard-ae63359ecb7d4ecc.rlib}" "/nix/store/6ynk08k69z7y62x6f89giv6hps66m1f4-rust-std/lib/rustlib/x86_64-pc-windows-msvc/lib/{liballoc-4f54ad4ac4b0f4c5.rlib,librustc_std_workspace_core-ff8db640d177ed2a.rlib,libcore-745ff350a54e4299.rlib,libcompiler_builtins-b2f39c5f2779068f.rlib}" "/defaultlib:msvcrt" "/NXCOMPAT" "/LIBPATH:/nix/store/snfxmriwav4i0k1fxp78xk5w12hbv4q9-postgresql-17.4/lib" "/LIBPATH:/home/magicloud/src/Git/hyperlight/src/tests/rust_guests/callbackguest/target/x86_64-pc-windows-msvc/debug/build/hyperlight-guest-92b206c9ed72f431/out" "/OUT:/home/magicloud/src/Git/hyperlight/src/tests/rust_guests/callbackguest/target/x86_64-pc-windows-msvc/debug/deps/callbackguest.exe" "/OPT:REF,NOICF" "/DEBUG" "/PDBALTPATH:%_PDB%"
  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: rust-lld: error: could not open 'msvcrt.lib': No such file or directory
          rust-lld: error: could not open 'oldnames.lib': No such file or directory


error: could not compile `callbackguest` (bin "callbackguest") due to 1 previous error
error: Recipe `build-rust-guests` failed on line 45 with exit code 101

[Magicloud]

@syntactically For hardware info, I am not sure what exactly do you need. Following is the last core of cpuinfo:

processor       : 23
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           X5675  @ 3.07GHz
stepping        : 2
microcode       : 0x1f
cpu MHz         : 1599.626
cache size      : 12288 KB
physical id     : 1
siblings        : 12
core id         : 9
cpu cores       : 6
apicid          : 51
initial apicid  : 51
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb pti ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid dtherm ida arat vnmi flush_l1d
vmx flags       : vnmi preemption_timer invvpid ept_x_only ept_1gb flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest ple
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips        : 6131.70
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

[Magicloud]

@syntactically Also, if I removed lines in Justfile contains "msvc", I can build but hello-world example got the same error as above.