Wait for parent devices (VFs) before applying vGPU config

[karthikvetrivel]

Wait for VFs in vGPU Manager Validation

Problem

When switching workloads from vm-passthrough to vm-vgpu, the vgpu-device-manager can start before vgpu-manager finishes creating Virtual Functions (VFs), causing "no parent devices found" errors. This race condition is more pronounced on nodes with many GPUs (8+) where VF creation takes longer.

Root Cause

The vgpu-manager-ready status file is created when nvidia-smi works, but VF creation via sriov-manage -e happens after that. The vgpu-device-manager needs VFs to exist before it can configure vGPU types on them.

The sriov-manage script enables VFs by writing sriov_totalvfs to sriov_numvfs for each GPU sequentially. On systems with 8 GPUs (160 total VFs), this can take 30+ seconds to complete.

Solution

Add waitForVFs() to VGPUManager.validate() in the nvidia-validator. After the nvidia-smi check passes, poll SR-IOV sysfs via the nvpci library until all VFs are created:

• Find all NVIDIA PCI devices that are SR-IOV Physical Functions (SriovInfo.IsPF())
• Poll until sum(NumVFs) == sum(TotalVFs) across all GPUs

Testing

Verified on DGX A100 (8 GPUs, 160 VFs):

time="2025-12-18T00:32:43+02:00" level=info msg="Waiting for VFs: 0/160 enabled across 8 GPU(s)"
time="2025-12-18T00:32:45+02:00" level=info msg="Waiting for VFs: 0/160 enabled across 8 GPU(s)"
time="2025-12-18T00:32:47+02:00" level=info msg="Waiting for VFs: 0/160 enabled across 8 GPU(s)"
time="2025-12-18T00:32:52+02:00" level=info msg="All 160 VF(s) enabled on 8 NVIDIA GPU(s)"

Tested by building a standalone binary that calls waitForVFs() directly, then enabling VFs via sriov_numvfs while the test was running to simulate the vgpu-manager's sriov-manage -e behavior.

VFs were enabled by binding each GPU to pci-pf-stub and writing to sriov_numvfs.