GPU passthrough support in KVM

[li-liwen]

Hi all. I've been trying to setup a proper GPU enabled KVM cluster for weeks. I cannot find any official document or complete tutorial on this topic. It seems that GPU is working on Xen, but I'm a fan of Ceph and KVM, so I really would like to make it work on KVM.

Currently, I could only find some recent discussion indicating it's not working on KVM: #10112 (comment)
This issue has been resolved, but the writer indicates it's only working in Xen, and I can't get it working even in 4.20. #9573

There's also a tutorial which not accessible any more on how to use extra config to pass GPU to one instance.
https://lab.piszki.pl/cloudstack-kvm-and-running-vm-with-vgpu/

I'm looking for a solution that can create a compute offering with GPU support, so that users can create instances with GPU passed through without the help of administrator (or other type of API). I'm wondering if it's possible to do that with KVM, and other non-datacenter GPUs like RTX 4090. If it can be done I'm happy to update the document for that.

Thanks for answering.

[rajujith]

@li-liwen can you try the workarounds mentioned here: #10076 (reply in thread)

[li-liwen]

@rajujith Thanks for answering! I have tried the method and got GPU Passthrough working. Cheers!

However, I do need some sort of orchestration. For example, I have four 4090 GPUs on one machine, and I would like to provide different compute offerings with one, two or all four GPUs. If a user tries to deploy a new instance (or scale from the old one) when not enough GPUs are available, the hook script will fail and the instance will go to Error state, which requires administrator operation. The desired behavior is to let CloudStack know how many GPU available in some way, and prevent trying to deploy instance that cannot fit.

Is there any recommended way to do this? Should I let the Groovy hook script interact with CloudStack management sql database directly? Or should I modify the GPU offering in the CloudStack UI somehow?

Many thanks for helping me and to every contributor of this project.

[vishesh92]

@li-liwen Are you doing GPU passthrough using the extra config or are you using libvirt hooks?

[li-liwen]

@vishesh92 I'm using Libvirt hooks. Specifically, the groovy agent hooks introduced in #3839. I'm trying to write a script that can dynamically check available GPUs (by maintaining an extra json file containing all GPUs). I'm still working on this, but I think it should work. However, when there's no available GPU, the instance will either boot normally without GPU (throw an exception in the groovy script), or enter error state (use sys.exit(1) instead). Neither is ideal. But I cannot think of a better way to do that.

[li-liwen]

Thank you all for answering. I've used the libvirt hooks as a work around for GPU passthrough. However, if there's not enough GPU, the VM will enter error state. Is there any way for user or admin to modify the service offering of the error VM, so that it can be reattempted to deploy?

[rajujith]

@li-liwen this talk about the GPU feature proposal should have the answer for you, the short answer is it will need some development. https://www.youtube.com/watch?v=UyJTcy69ncg