NVIDIA_VISIBLE_DEVICES is set to void but nvidia-smi shows correct gpu assigned by k8s

[mmolisch]

Important Note: NVIDIA AI Enterprise customers can get support from NVIDIA Enterprise support. Please open a case here.

Describe the bug
After updating to latest operator and reconfiguring to run with CDI, I can see correct amounts of GPU beign assigned via nvidia-smi, but NVIDIA_VISIBLE_DEVICES is set to void. This breaks e.g. reporting of GPU usage into clear-ml. Happens on all gpu-nodes (RTX Titan, RTX 6000 Ada, L40s...)

To Reproduce

1. Setup RKE2 - https://docs.rke2.io/add-ons/gpu_operators?GPUoperator=v25.10.x
2. Install gpu-operator with values, default running with CDI:

node-feature-discovery:
  worker:
    priorityClassName: system-node-critical
  master:
    priorityClassName: system-node-critical
driver:
  upgradePolicy:
    # autoUpgrade (default=true): Switch which enables / disables the driver upgrade controller.
    autoUpgrade: false
  usePrecompiled: false
  version: "580.105.08"
toolkit:
  env:
  - name: CONTAINERD_SOCKET
    value: /run/k3s/containerd/containerd.sock
operator:
  upgradeCRD: true
dcgmExporter:
  config:
    name: custom-dcgm-exporter-metrics
    create: true
    data: |-
       .... (shortened, as it's not relevant)

3. Request 1 GPU (doesn't matter if you specify containerRuntimeClass: nvidia or leave empty as they say in RKE2 config) for pod
4. NVIDIA_VISIBLE_DEVICES is set to void (CUDA_VISIBLE_DEVICES not present at all)

Expected behavior
NVIDIA_VISIBLE_DEVICES or CUDA_VISIBLE_DEVICES is set to correct ID(s)

Environment (please provide the following information):

• GPU Operator Version: v25.10.1
• OS: Ubuntu 22.04 LTS
• Kernel Version: 5.15.0-163-generic
• Container Runtime Version: v2.1.5-k3s1
• Kubernetes Distro and Version: RKE2 v1.34.2+rke2r1

Information to attach (optional if deemed irrelevant)

• Output from running nvidia-smi from the driver container: kubectl exec DRIVER_POD_NAME -n OPERATOR_NAMESPACE -c nvidia-driver-ctr -- nvidia-smi

Collecting full debug bundle (optional):

curl -o must-gather.sh -L https://raw.githubusercontent.com/NVIDIA/gpu-operator/main/hack/must-gather.sh
chmod +x must-gather.sh
./must-gather.sh

NOTE: please refer to the must-gather script for debug data collected.

This bundle can be submitted to us via email: operator_feedback@nvidia.com

[cdesiniotis]

Hi @mmolisch it is expected behavior that NVIDIA_VISIBLE_DEVICES gets set to void. Starting with GPU Operator 25.10.0, we leverage native-CDI support in runtimes (like containerd, cri-o) for injecting GPUs into workload containers. We no longer use the NVIDIA Container Runtime for this purpose. We purposely set NVIDIA_VISIBLE_DEVICES=void in our generated CDI specs so that even if the nvidia runtime happens to be invoked it should not be performing any modifications to the already modified container (modified by containerd / cri-o).

With regards to CUDA_VISIBLE_DEVICES this is not something that the operator sets so the behavior you are seeing is expected.

[cdesiniotis]

This breaks e.g. reporting of GPU usage into clear-ml.

Could you provide more information on what exactly is breaking?

[mmolisch]

Thank you for a swift response! ClearML uses those env vars to provide GPU metrics directly in their web-ui https://github.com/clearml/clearml/blob/3e77cd00798a5e166d40f3788badff69a02d28c2/clearml/utilities/resource_monitor.py#L109

Is there any way how to inject those env vars or do I have to open issue at ClearML repo?

[cdesiniotis]

I am not familiar with ClearML, but we do not recommend setting the NVIDIA_VISIBLE_DEVICES environment variable yourself or depending on the presence of it in a container. If the ClearML application is depending on the presence of this environment variable, then yes, filing an issue against their repository would be a good next step.

[mmolisch]

Thank you, will do.