Exploration: GPU setup on Canonical k8s

[misohu]

Why it needs to get done

We want to make sure we understand how to setup Nvidia GPU operator on top of Canonical k8s for different use cases.

What needs to get done

1. Test Nvidia GPU operator on device with up to date Nvidia driver
2. Test Nvidia GPU operator on device with outdated Nvidia driver
3. Test Nvidia GPU operator on device with no Nvidia driver

When is the task considered done

1. Document the steps above.

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6896.

This message was autogenerated

[misohu]

Exploration Report: NVIDIA GPU Operator Behavior in Different Driver Scenarios

Introduction

The objective of this exploration was to evaluate how the NVIDIA GPU Operator behaves under different conditions related to NVIDIA GPU drivers. Specifically, the following scenarios were tested:

1. Device with an up-to-date NVIDIA driver
2. Device with an outdated NVIDIA driver
3. Device with no NVIDIA driver installed

Each scenario was tested by deploying the NVIDIA GPU Operator using Helm and monitoring its behavior, including potential errors.

---

1. Test: NVIDIA GPU Operator on Device with Up-to-Date NVIDIA Driver

Setup

• The latest NVIDIA driver was manually installed on the host system before deploying the NVIDIA GPU Operator.

• The GPU Operator was deployed using the default Helm installation:

  helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update
  helm install --wait --generate-name -n gpu-operator --create-namespace nvidia/gpu-operator
  

Observed Behavior

• The operator attempted to install its own driver despite the presence of a valid, up-to-date driver.
• However, since the driver was already installed, there were no critical failures.
• nvidia-smi correctly listed the GPU without errors.

Logs & Potential Warnings

• From nvidia-driver-daemonset logs:

  Driver already installed, skipping installation.
  

• From nvidia-smi output:

  +-----------------------------------------------------------------------------+
  | NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
  +-----------------------------------------------------------------------------+
  

Conclusion

• The operator works correctly when the latest driver is pre-installed, but it still attempts installation unless explicitly disabled.
• Best practice: Use --set driver.enabled=false to avoid redundant installations:

  helm install --wait --generate-name nvidia/gpu-operator --set driver.enabled=false
  

---

2. Test: NVIDIA GPU Operator on Device with Outdated NVIDIA Driver

Setup

• A deliberately outdated NVIDIA driver (e.g., 470.xx) was installed.
• The GPU Operator was deployed using the same Helm installation.

Observed Behavior

• The GPU Operator attempted to install a new driver.
• There was a driver mismatch error because the old driver could not support the required CUDA version.
• The nvidia-device-plugin failed to detect GPUs due to the mismatch.

Error Messages & Logs

• From nvidia-smi:

  NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.
  Make sure that the latest NVIDIA driver is installed and running.
  

• From kubectl logs -n gpu-operator -l app=nvidia-driver-daemonset:

  modprobe: ERROR: could not insert 'nvidia': Exec format error
  

• From kubectl describe node <node-name>:

  Warning  MissingGPUs  10m  nvidia-device-plugin  No NVIDIA GPUs detected on this node
  

Conclusion

• The outdated driver conflicted with the operator’s expected version, preventing proper GPU detection.
• Fix:
  • Upgrade the host driver manually (sudo apt update && sudo apt install -y nvidia-driver-latest), OR
  • Allow the operator to install its own driver by removing the existing driver:

    sudo apt remove --purge '^nvidia-.*'
    reboot
    

---

3. Test: NVIDIA GPU Operator on Device with No NVIDIA Driver Installed

Setup

• No NVIDIA driver was installed on the host before deploying the operator.
• The GPU Operator was installed using Helm.

Observed Behavior

• The GPU Operator successfully installed the NVIDIA driver using a containerized driver daemonset.
• nvidia-smi worked after installation, and GPUs were detected.
• The deployment took longer than usual due to the time needed to install the driver.

Error Messages & Warnings (Before the Driver Was Installed)

• Initially, nvidia-smi failed:

  NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.
  

• After driver installation completed:

  +-----------------------------------------------------------------------------+
  | NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
  +-----------------------------------------------------------------------------+
  

• From kubectl get pods -n gpu-operator (while installing driver):

  nvidia-driver-daemonset-xxxxx   0/1     Init:0/2   5m
  

• After driver installation completed:

  nvidia-driver-daemonset-xxxxx   1/1     Running   8m
  

Conclusion

• If no driver is present, the operator correctly installs it.
• The process takes longer compared to using a preinstalled driver.
• Recommendation: If you prefer to use a manually installed driver, install it before deploying the GPU Operator and use:

  helm install --wait --generate-name nvidia/gpu-operator --set driver.enabled=false
  

---

Final Recommendations

Scenario	Expected Behavior	Recommended Action
Up-to-date NVIDIA driver present	Works fine, but operator still attempts driver installation	Use --set driver.enabled=false to avoid redundant installation
Outdated NVIDIA driver present	Driver mismatch, GPU not detected	Upgrade the host driver manually or remove it before deploying the operator
No NVIDIA driver present	Operator installs the driver successfully	Let the operator manage driver installation

---

Conclusion

This exploration confirmed that the NVIDIA GPU Operator handles driver installation differently depending on the state of the host system:

• If the driver is up-to-date, it works fine but installs an unnecessary driver unless explicitly disabled.
• If the driver is outdated, conflicts arise, requiring manual intervention.
• If no driver is present, the operator installs it correctly but takes longer to deploy.

For production deployments, it is recommended to explicitly configure driver installation based on the existing system state to avoid conflicts and unnecessary delays.