[Driver CRD] repoConfig not honored

[Archimonde666]

Describe the bug
When provided both in the values.yaml and in the CRD appropriated flag (

gpu-operator/deployments/gpu-operator/crds/nvidia.com_nvidiadrivers.yaml

Line 580 in d2be90c

repoConfig:

) repoConfig.configMapName and repoConfig.destinationDir for custom sources.list ; they are not honored in the driver daemonset, neither volume nor volumemount.

To Reproduce
Install gpu-operator with driver CRD and custom or overrided APT repository

Expected behavior
sources.list mounted in the driver container

Environment (please provide the following information):

• GPU Operator Version: [v25.3.2]
• OS: [Ubuntu22.04]
• Kernel Version: [6.8.0-generic]

Remark : i'm not sure it is actually useful to make this configuration through CRD. The repoconfig should in almost every case be the same clusterwide.

=> i'd suggest to drop that from CRD and honor the global value from the helm chart configuration (which is dropped at the moment)

While you working on that, i'd suggest you also add a flag for custom GPG key in the same CRD ; i have to manually edit the daemonset after creation. Consider simple customVolumeMount.

[rahulait]

Hi,

The issue description says you tried to use gpu-operator version v25.3.2 and tried to set the flags repoConfig.configMapName and repoConfig.destinationDir. However, there is no repoConfig.destinationDir flag in v25.3.2. It has been dropped since v1.9.0.

Can you share your values.yaml file which was used to render the default nvidiadriver crd? Or if you are using your own custom nvidiadriver crd, please share the rendered crd for us to make sure values are set correctly. I tried to reproduce the issue but I do see configmap getting added to the driver daemonset pods. I followed the steps as mentioned in this guide and it worked fine for me: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/install-gpu-operator-air-gapped.html#local-package-repository

[rahulait]

While you working on that, i'd suggest you also add a flag for custom GPG key in the same CRD ; i have to manually edit the daemonset after creation. Consider simple customVolumeMount.

Can you also share a sample example of how you are doing the changes so that we can decide if we can do it or not. Thanks.

[Archimonde666]

Hello,

I had to downgrade to gpu-operator v24.9.2 and unable to upgrade at the moment due to issues with container toolkit with k0s based distributions.

I can confirm that the content of the configmap passed to repoConfig.configMapName isn't mounted when i look at the yaml of the driver pod.

My configmap looks like this :

{
 	"sources.list": 
 		
 		deb [arch=amd64] http://ubuntu.ipms.qual.internal/jammy jammy           main universe multiverse restricted
 		deb [arch=amd64] http://ubuntu.ipms.qual.internal/jammy jammy-updates   main universe multiverse restricted
 		deb [arch=amd64] http://ubuntu.ipms.qual.internal/jammy jammy-backports main universe multiverse restricted
 		deb [arch=amd64] http://ubuntu.ipms.qual.internal/jammy jammy-security  main universe multiverse restricted
 		
}

and the relevant values i'm using are as follow :

driver:
  enabled: true
  nvidiaDriverCRD:
    enabled: true
    deployDefaultCR: false
    driverType: gpu
    nodeSelector: {}
  kernelModuleType: "auto"
  usePrecompiled: false
  repository: nvidia
  image: driver
  version: ""
  imagePullPolicy: IfNotPresent
  imagePullSecrets: []
  startupProbe:
    initialDelaySeconds: 60
    periodSeconds: 10
    timeoutSeconds: 60
    failureThreshold: 120
  rdma:
    enabled: false
    useHostMofed: false
  upgradePolicy:
    autoUpgrade: true
    maxParallelUpgrades: 1
    maxUnavailable: 50%
    waitForCompletion:
      timeoutSeconds: 0
      podSelector: ""
    gpuPodDeletion:
      force: false
      timeoutSeconds: 3
      deleteEmptyDir: true
    drain:
      enable: false
      force: false
      podSelector: ""
      timeoutSeconds: 300
      deleteEmptyDir: false
  manager:
    image: k8s-driver-manager
    repository: nvidia/cloud-native
    version: v0.8.0
    imagePullPolicy: IfNotPresent
    env:
      - name: ENABLE_GPU_POD_EVICTION
        value: "true"
      - name: ENABLE_AUTO_DRAIN
        value: "false"
      - name: DRAIN_USE_FORCE
        value: "false"
      - name: DRAIN_POD_SELECTOR_LABEL
        value: ""
      - name: DRAIN_TIMEOUT_SECONDS
        value: "0s"
      - name: DRAIN_DELETE_EMPTYDIR_DATA
        value: "false"
  env: []
  resources: {}
  repoConfig:
    configMapName: "config-apt"
    destinationDir: "/etc/apt/sources.list.d"
  certConfig:
    name: ""
  licensingConfig:
    configMapName: ""
    nlsEnabled: true
  virtualTopology:
    config: ""
  kernelModuleConfig:
    name: ""

Here is also the rendered CRD :

apiVersion: nvidia.com/v1alpha1
kind: NVIDIADriver
metadata:
  annotations:
  creationTimestamp: '2025-10-28T09:44:00Z'
  generation: 1
  managedFields:
    - apiVersion: nvidia.com/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:spec:
          .: {}
          f:driverType: {}
          f:env: {}
          f:image: {}
          f:imagePullPolicy: {}
          f:imagePullSecrets: {}
          f:kernelModuleType: {}
          f:manager: {}
          f:nodeSelector:
            .: {}
            f:kubernetes.io/hostname: {}
          f:repository: {}
          f:usePrecompiled: {}
          f:version: {}
      manager: kubectl-client-side-apply
      operation: Update
      time: '2025-10-28T09:44:00Z'
    - apiVersion: nvidia.com/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:conditions: {}
          f:state: {}
      manager: gpu-operator
      operation: Update
      subresource: status
      time: '2025-10-28T11:57:48Z'
  name: a100
  resourceVersion: '234925775'
  uid: 8331430a-2a5d-4a1c-90b7-6290038a9fb5
spec:
  driverType: gpu
  env: []
  image: driver
  imagePullPolicy: IfNotPresent
  imagePullSecrets: []
  kernelModuleType: auto
  manager: {}
  nodeSelector:
    kubernetes.io/hostname: uapqpdatalab01
  repository: nvidia
  usePrecompiled: true
  version: '570'
status:
  conditions:
    - lastTransitionTime: '2025-10-28T11:57:48Z'
      message: All resources have been successfully reconciled
      reason: Reconciled
      status: 'True'
      type: Ready
    - lastTransitionTime: '2025-10-28T11:57:48Z'
      message: ''
      reason: Ready
      status: 'False'
      type: Error
  state: ready

Nota : from this issue and since the catalog expanded i'm not anymore editing daemonset manually and transitionned to precompiled drivers. I'm open modifing this for testing purpose but i suppose the configmap should be mounted regardless of the kind of driver selected

[Archimonde666]

While you working on that, i'd suggest you also add a flag for custom GPG key in the same CRD ; i have to manually edit the daemonset after creation. Consider simple customVolumeMount.

Can you also share a sample example of how you are doing the changes so that we can decide if we can do it or not. Thanks.

For this part, the way i was doint it was adding thoses flags in it directly after the daemonset yaml was rendered ; assuming the configmap was already created from the gpg key (using kubectl create configmap --from-file). One must also provide the custom sources.list with the appropriated signed-by flag :

deb [arch=amd64 signed-by=/etc/apt/sources.list.d/key.gpg] ...

- mountPath: /etc/apt/sources.list.d/key.gpg
          name: apt-gpg
          readOnly: true
          subPath: key.gpg

- configMap:
          defaultMode: 420
          items:
          - key: key.gpg
            path: key.gpg
          name: apt-configmap
        name: apt-gpg

[rahulait]

In your config I see deployDefaultCR: false:

driver:
  enabled: true
  nvidiaDriverCRD:
    enabled: true
    deployDefaultCR: false
...

On helm template, this will not render any NVIDIADriver CR and you might be creating NVIDIADriver CR yourself. Can you share how its generated/created because it looks like you don't have this config set: https://github.com/NVIDIA/gpu-operator/blob/main/deployments/gpu-operator/templates/nvidiadriver.yaml#L44-L47. If you are generating NVIDIADriver CR manually, you'll need to set the values as shared in the link.

[rahulait]

For this part, the way i was doint it was adding thoses flags in it directly after the daemonset yaml was rendered ; assuming the configmap was already created from the gpg key (using kubectl create configmap --from-file).

You can create a configmap with data from two files and it will place both of them in the same dir. Example:

kubectl create configmap repo-config -n gpu-operator --from-file=/root/custom-repo.list --from-file=/root/key.gpg

Let me know if this doesn't work for you.

[rahulait]

I'll close this issue in a week if there is no further activity. Let me know if the above mentioned steps doesn't work for you.

[rahulait]

Closing the issue as we tested and found that it works fine. Correct config needs to be added to nvidiaDriver CR to make sure it mounts volumes correctly. Please feel free to re-open or create new ticket in case you face issues.

[Archimonde666]

Sorry for the delay, been rly busy lately.
I can confirm correct setting of the repoconfig.name in the CRD mount the configmap correctly. Maybe should harmonise as i was confused with the repoconfig.configMapName where is it theoreticaly injected from.

Therefore i can also confirm creating the configmap from both a text file (ie sources.list) and a binary file (ie gpg.key) as proposed doesn't mount both of them inside the driver container

The rendered yaml of the daemonset created by the operator is as follow even if there is two keys inside the configmap, not allowing for gpg key injection :

    - configMap:
        defaultMode: 420
        items:
          - key: sources.list
            path: sources.list
        name: config-apt
      name: config-apt

i figured out it actually works with two text files, indicating that it is probably due to the nature of the gpg key, being injected into the configmap under the binaryData: key and not under the data: key

@rahulait thanks for the explanation, i don't know if it's still a bug or a feature request now as the "classic" repoconfig actually works