GPU Support forKVM

[hstan77]

Hi

My cloudstack 4.15, on Ubuntu. 20 , running on KVM

I bought Nvidia Grid K1 , as mentioned in the Service Offering mentioned.

I create a new cluster and host for this.

However, after I created the host with Grid K1 card, the service offering plan not able to create, and obviously it seem the VM in Hypervisor not detected hav Grapic card.

When create a service Offering, it show error , which i aldy choose.
Request Failed (431)
Please specify the pciDevice and vgpuType correctly.

Can I know if this GPU function actually i s not working ?

[tdtmusic2]

I don't think you'll have any chance on using CS with a gpu these days, the only way that seem to work is by having a xen host with a supported gpu and full passthrough mode only.

[hstan77]

i bought Nvidia Grid K1 ( a refurbish unit) , just to test out . But seems not able to work .

May be this Cloudstack functions is not more working. The Cloudstack GUI just put there to show only.

[correajl]

Hi! I've used Nvidia GPUs with CloudStack in a totally manual way. You will need to do PCI passthrough. Some steps:

• enable some global configs like 'allow.additional.vm.configuration.list.kvm = devices,hostdev,driver,source,address,alias' and 'enable.additional.vm.configuration = true' in ACS.

• enable features in hosts BIOS like Intel VT-d or AMD IOV.

• use IOMMU to create groups and isolate PCIe bus.

  IOMMU Group 34:
      0000:17:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104 [GeForce RTX 3070] [10de:2484] (rev a1)
      0000:17:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1)
  IOMMU Group 74:
      0000:b3:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104 [GeForce RTX 3070] [10de:2484] (rev a1)
      0000:b3:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1)
  

• use some parameters in hosts kernel like 'GRUB_CMDLINE_LINUX="video=efifb:off intel_iommu=on iommu=pt kvm.ignore_msrs=1 rd.driver.pre=vfio-pci"' or 'GRUB_CMDLINE_LINUX="video=efifb:off amd_iommu=on iommu=pt kvm.ignore_msrs=1 rd.driver.pre=vfio-pci"'

• discover the PCI IDs and change grub again to something like 'GRUB_CMDLINE_LINUX="video=efifb:off intel_iommu=on intel_iommu=on kvm.ignore_msrs=1 rd.driver.pre=vfio-pci vfio-pci.ids=10de:2484,10de:228b"' (then GPUs is not used by host)

• create new compute offerings that use "extra-config"
  cmk -p profile create serviceoffering name="VM with 2x RTX3070" displaytext="VM with 2x RTX3070" customized=true storagetype=shared provisioningtype=fat cpuspeed=1 mincpunumber=2 maxcpunumber=12 minmemory=2000 maxmemory=30000 offerha=false dynamicscalingenabled=false hosttags=rtx3070-bus17 diskofferingstrictness=false serviceofferingdetails[1].key=extraconfig-1 serviceofferingdetails[1].value="CHANGEINDATABASE"

• find it in database:
  select so.name, so.id, sod.service_offering_id, sod.name, sod.value from service_offering as so left join service_offering_details as sod on so.id = sod.service_offering_id where so.state = "Active" and so.name = "VM with 2x RTX3070";

• chage it to an extra-config that uses your GPU (pci bus, etc):
  Something like (here I'm using 2 GPUs, 0x17 and 0xb3):
  <devices><hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x17' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x17' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0xb3' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0xb3' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x00' function='0x0'/></hostdev></devices>

  update using something like:
  update service_offering_details set value = "<devices><hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x17' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x17' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0xb3' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0xb3' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x00' function='0x0'/></hostdev></devices>" where service_offering_id = 44 and name = "extraconfig-1" and value = "CHANGEINDATABASE";

Well, at this point you "could" passthrough a GPU to an instance. But, you will need to launch instance as ROOT, so you can choose the host (ACS knows nothing about your host that has GPU). If you are using a compute offering that inject an extra-config that makes the instance use some pci bus, so you should launch it in a host that has this pci bus id.

In the end, you should have an instance that access de PCI where is the GPU. Install the nvidia driver in instance and be happy.

These steps aren't so simples, there are many other details, but, this is a fast guideline to some way to do this work.

:)

[dcontiveros-nf]

I have successfully passed through some disks via this manner. I did have to edit the cloudstack configuration to enable a lot of privileges and had to URL baseencode the request per this link:

https://lab.piszki.pl/cloudstack-kvm-and-running-vm-with-vgpu/