How to add a passthrogh device to the L2 Guest?

[i-yyi]

Hi there,

I'm trying the pKVM in qemu now, and I'd like to see if I can pass through some device to the L2 Guest.
I encountered some question when passing through device via crosvm

My system environment:

CPU: Intel Core i9-13980HX (Raptor Lake)
Host Kernel: Linux 6.14.0
Qemu Version: qemu 10.1.2
pKVM Version: commit f21553 (HEAD of pvVMCS-POC-v6.12)
crosVM Version(runs on L1 Host): commit 463076 (Last Update at Oct 27)

In my qemu command, I added two NVMe disks emulated by qemu beyond the normal pKVM boot.

exec sudo "${QEMU_BIN}" \
	-drive "file=${DISK_IMG},format=qcow2" \
	-machine q35,accel=kvm,kernel-irqchip=split \
	-cpu host,kvm-pv-unhalt=off,kvm-pv-ipi=off,kvm-pv-sched-yield=off \
	-device intel-iommu,aw-bits=48 \
	-m "${MEM}" -smp "${SMP}" \
	-name debug-threads=on \
	-vga none \
	-display none \
	-device virtio-net-pci,netdev=net0,mac=52:54:00:12:34:56 \
	-netdev user,id=net0,hostfwd=tcp:127.0.0.1:2222-:22,hostfwd=tcp:127.0.0.1:1235-:1234,hostfwd=tcp:127.0.0.1:2223-:3333 \
	-serial stdio \
	-kernel "${KERNEL_IMG}" \
	-append "root=/dev/sda2 console=ttyS0,115200n8 debug earlyprintk=ttyS0,115200n8 kvm-intel.pkvm=1 nokaslr systemd.mask=NetworkManager-wait-online.service systemd.mask=systemd-networkd-wait-online.service" \
	-gdb tcp:127.0.0.1:1234 \
	**-device pcie-root-port,id=rp_host,chassis=1,slot=2 \
	-drive if=none,id=nvme_host,file="${HOST_NVME}",format=raw \
	-device nvme,serial=HOSTNVME,drive=nvme_host,bus=rp_host \
	-device pcie-root-port,id=rp_guest,chassis=2,slot=3 \
	-drive if=none,id=nvme_guest,file="${GUEST_NVME}",format=raw \
	-device nvme,serial=GUESTNVME,drive=nvme_guest,bus=rp_guest \**
	${EXTRA_ARGS}

I unbound the GUESTNVME in L1 Host, and bound it to the vfio-pcie.

When I start L2 Guest with no passthrough, it can boot as expected.

taskset -c 7 crosvm run \
        --disable-sandbox \
        --protected-vm-without-firmware \
        --cpus 1 --mem 512 \
        --block "path=./ubuntu.qcow2,root=false" \
        --serial type=stdout,hardware=serial,console=true,stdin=true,earlycon=true \
        --params "root=/dev/vda2 console=ttyS0,115200n8 earlyprintk=ttyS0,115200n8" \
        ./bzImage_pkvm_guest

However when I add a passthrough device, crosvm will report a vcpu crash

The command:

taskset -c 7 crosvm run \
	--disable-sandbox \
	--protected-vm-without-firmware \
	--cpus 1 --mem 512 \
	--vfio /sys/bus/pci/devices/0000:02:00.0,iommu=viommu \
	--block "path=./ubuntu.qcow2,root=false" \
	--serial type=stdout,hardware=serial,console=true,stdin=true,earlycon=true \
	--params "root=/dev/vda2 console=ttyS0,115200n8 earlyprintk=ttyS0,115200n8" \
	./bzImage_pkvm_guest

and the crash info:

......
[    0.728000] PCI: Using E820 reservations for host bridge windows
[    0.732000] ACPI: Enabled 1 GPEs in block 00 to FF
[    0.736000] ACPI: \_SB_.PC00.PC02.PE00.PRIC: New power resource
[2025-11-19T05:49:57.243328124+00:00 ERROR crosvm::crosvm::sys::linux::vcpu] vcpu hit unknown error: Bad address (os error 14)
[2025-11-19T05:49:57.243628112+00:00 INFO  crosvm::crosvm::sys::linux] vcpu crashed
[2025-11-19T05:49:57.243663656+00:00 ERROR crosvm::crosvm::sys::linux::vcpu] failed to send VcpuControl: sending on a closed channel

I'm not sure where to ask for help, and have no idea whether it's an issue about pKVM. But you may have experience with providing L2 Guest devices that pass through to QEMU emulation during development, so I'd like to ask if you have any tips.

Thank you for your patience and help! I will provide as much additional information as possible.

Upd:
Btw, the L0 Hyp & L1 Host 's dmesg didn't show anything about this fault

[maluka-dmytro]

TL;DR device assignment to protected VMs is not supported by pKVM-IA yet.

It used to be supported to some extent in an earlier PoC implementation, and there is still some rudimentary code for that in the v6.12 branch, but I think it stopped working at least since we stopped using Virtualization Exceptions (#VE) for emulated MMIO for pVMs and started requiring pVMs to directly use hypercalls instead. Now pVMs use hypercalls for any MMIO accesses (see pv_ops.mmio setup in arch/x86/coco/pkvm/pkvm.c), i.e. de facto assuming that any MMIO is emulated by the host. Thus it fails to access physical MMIO of passthrough devices (since it tries to use those hypercalls for that MMIO as well).

To fix that, the guest kernel would need to somehow distinguish which MMIO addresses are physical MMIO of a passed-through device, to access that MMIO directly instead of using hypercalls. (But even then that is just one of many things that are yet to be designed and implemented to properly support assigning physical devices to pVMs. Especially if we are talking about assigning them securely.)

You may try running your VM as a non-protected VM instead, then it might work (although I'm not entirely sure about that either, I haven't tried that myself since quite a while).