GPU in passthrough mode on KVM

[tdtmusic2]

Hi all. Did anyone manage to get a gpu to be usable in ACS in passthrough mode on a KVM host? For me, it seems that the host is not added to the host_gpu_groups table in the database, because I get the error bellow when trying to deploy a vm with a gpu-enabled offering:

Unable to orchestrate start VM instance {"id":2284,"instanceName":"i-2-2284-VM","type":"User","uuid":"10bf3221-bb21-4879-8691-f3dc6cb5f920"} due to [DB Exception on: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT host_gpu_groups.id, host_gpu_groups.group_name, host_gpu_groups.host_id FROM host_gpu_groups INNER JOIN vgpu_types groupId ON host_gpu_groups.id=groupId.gpu_group_id WHERE host_gpu_groups.host_id = 176 AND host_gpu_groups.group_name = x'47726f7570206f66204e766964696120436f72706f726174696f6e205455313034474c205b5465736c612054345d2047505573' AND (groupId.vgpu_type = x'706173737468726f756768' AND groupId.remaining_capacity > 0 ) ORDER BY vgpu_types.remaining_capacity DESC ].

I have this in the agent.properties:

enable.pci.passthrough=true
pci.devices.in.vm=nvidia,10de,1eb8
pci.device.mode=nvidia-tesla-t4
nvidia.gpu.enable=true

If I try to addlibvirt.vif.driver=com.cloud.hypervisor.kvm.resource.LibvirtVifDriver to it the agent service does not start anymore, so I left it out. A virsh dumpxml command on the gpu's pci id seems to report ok:

<device>
  <name>pci_0000_61_00_0</name>
  <path>/sys/devices/pci0000:60/0000:60:03.1/0000:61:00.0</path>
  <parent>pci_0000_60_03_1</parent>
  <driver>
    <name>vfio-pci</name>
  </driver>
  <capability type='pci'>
    <class>0x030200</class>
    <domain>0</domain>
    <bus>97</bus>
    <slot>0</slot>
    <function>0</function>
    <product id='0x1eb8'>TU104GL [Tesla T4]</product>
    <vendor id='0x10de'>NVIDIA Corporation</vendor>
    <capability type='virt_functions' maxCount='16'/>
    <iommuGroup number='65'>
      <address domain='0x0000' bus='0x61' slot='0x00' function='0x0'/>
    </iommuGroup>
    <numa node='3'/>
    <pci-express>
      <link validity='cap' port='0' speed='8' width='16'/>
      <link validity='sta' speed='8' width='8'/>
    </pci-express>
  </capability>
</device>

[tdtmusic2]

A later addition to this. Seeing as the kvm route is pointless, I tried via xcp-ng 8.2. Here the gpu is detected and after adding the host to ACS I can see that the host_gpu_groups has now a row with "Group of NVIDIA Corporation TU104GL [Tesla T4] GPUs" for that host (also the integrated gpu but that doesn't matter). vgpu_types table is also populated, with availability 1 and type passthrough. However when trying to deploy a vm with a gpu-enabled offering, I get the same sql query error in ACS...

I think the sql query that ACS is trying to run is wrong. The query in the logs is this:

SELECT host_gpu_groups.id, host_gpu_groups.group_name, host_gpu_groups.host_id FROM host_gpu_groups INNER
JOIN vgpu_types groupId ON host_gpu_groups.id=groupId.gpu_group_id WHERE host_gpu_groups.host_id = 185 
AND host_gpu_groups.group_name =x'47726f7570206f66204e766964696120436f72706f726174696f6e205455313034474c205b5465736c612054345d2047505573' AND (groupId.vgpu_type = x'706173737468726f756768' AND groupId.remaining_capacity > 0 )
ORDER BY vgpu_types.remaining_capacity DESC;

But if I'm running it directly in workbench I get this error: Error Code: 1054. Unknown column 'vgpu_types.remaining_capacity' in 'order clause'
The query should look like this (and it returns what it should, the available gpu):

SELECT host_gpu_groups.id, host_gpu_groups.group_name, host_gpu_groups.host_id 
FROM host_gpu_groups 
INNER JOIN vgpu_types groupId ON host_gpu_groups.id = groupId.gpu_group_id 
WHERE host_gpu_groups.host_id = 185 
  AND host_gpu_groups.group_name = x'47726f7570206f66204e766964696120436f72706f726174696f6e205455313034474c205b5465736c612054345d2047505573' 
  AND (groupId.vgpu_type = x'706173737468726f756768' 
       AND groupId.remaining_capacity > 0) 
ORDER BY groupId.remaining_capacity DESC;

[rajujith]

@ tdtmusic2 this needs a fix, could you open a bug ticket? here is another thread with some information on the GPU support: #10076

[tdtmusic2]

@ tdtmusic2 this needs a fix, could you open a bug ticket? here is another thread with some information on the GPU support: #10076

Hello. I did that: #10119

[rajujith]

can you try Cloudstack 4.20? #9573 seems to be fixed there.

[tdtmusic2]

can you try Cloudstack 4.20? #9573 seems to be fixed there.

Unfortunatelly no, it would mean a lot of changes for us. Currently I'm on 4.19.1.2, maybe it will also be implemented in the next minor release?

[rajujith]

Yes, it should be in the 4.19.2.