How to troubleshoot feature.d changes?

[cmontemuino]

I don't get why NFD keeps adding labels to some of my GPU nodes, when it shouldn't. What I need is some pointers to identify the root cause.

If I get the labels from a node and ...| grep nvidia:

nvidia.com/cuda.driver-version.full: 570.124.06
nvidia.com/cuda.driver-version.major: "570"
nvidia.com/cuda.driver-version.minor: "124"
nvidia.com/cuda.driver-version.revision: "06"
nvidia.com/cuda.driver.major: "545"
nvidia.com/cuda.driver.minor: "23"
nvidia.com/cuda.driver.rev: "06"
nvidia.com/cuda.runtime-version.full: "12.8"
nvidia.com/cuda.runtime-version.major: "12"
nvidia.com/cuda.runtime-version.minor: "8"
nvidia.com/cuda.runtime.major: "12"
nvidia.com/cuda.runtime.minor: "3"
nvidia.com/gfd.timestamp: "1710967106"
nvidia.com/gpu.compute.major: "8"
nvidia.com/gpu.compute.minor: "0"
nvidia.com/gpu.count: "4"
nvidia.com/gpu.deploy.container-toolkit: "true"
nvidia.com/gpu.deploy.dcgm: "true"
nvidia.com/gpu.deploy.dcgm-exporter: "true"
nvidia.com/gpu.deploy.device-plugin: "true"
nvidia.com/gpu.deploy.driver: "true"
nvidia.com/gpu.deploy.gpu-feature-discovery: "true"
nvidia.com/gpu.deploy.mig-manager: "true"
nvidia.com/gpu.deploy.node-status-exporter: "true"
nvidia.com/gpu.deploy.operator-validator: "true"
nvidia.com/gpu.engines.copy: "7"
nvidia.com/gpu.engines.decoder: "5"
nvidia.com/gpu.engines.encoder: "0"
nvidia.com/gpu.engines.jpeg: "1"
nvidia.com/gpu.engines.ofa: "1"
nvidia.com/gpu.family: ampere
nvidia.com/gpu.machine: ThinkSystem-SR670-V2
nvidia.com/gpu.memory: "80640"
nvidia.com/gpu.mode: compute
nvidia.com/gpu.multiprocessors: "98"
nvidia.com/gpu.present: "true"
nvidia.com/gpu.product: NVIDIA-A100-80GB-PCIe-MIG-7g.80gb
nvidia.com/gpu.replicas: "1"
nvidia.com/gpu.sharing-strategy: none
nvidia.com/gpu.slices.ci: "7"
nvidia.com/gpu.slices.gi: "7"
nvidia.com/mig.capable: "true"
nvidia.com/mig.config: all-disabled
nvidia.com/mig.config.state: success
nvidia.com/mig.strategy: single
nvidia.com/mps.capable: "false"
nvidia.com/vgpu.present: "false"

These are the labels that does not make sense from above:

nvidia.com/cuda.driver.major: "545"
nvidia.com/cuda.driver.minor: "23"
nvidia.com/cuda.driver.rev: "06"
nvidia.com/gpu.product: NVIDIA-A100-80GB-PCIe-MIG-7g.80gb
nvidia.com/gpu.slices.ci: "7"
nvidia.com/gpu.slices.gi: "7"

So, I've removed the labels and added this one: nvidia.com/gpu.product: NVIDIA-A100-80GB-PCIe

Note: I've also tried to remove annotation nfd.node.kubernetes.io/feature-labels first.

👉 After a few seconds, everything gets reverted back, and this is what I don't understand 🤔.

This is a GPU node and I have the gpu-operator running behind the scenes. This operator has a deamonset writing a set of labels in a directory for NFD to add labels afterwards. Logs:

I0519 09:40:50.420211      51 output.go:82] Writing labels to output file /etc/kubernetes/node-feature-discovery/features.d/gfd
I0519 09:40:50.420737      51 main.go:287] Sleeping for 30000000000

Looking at that file:

cat /etc/kubernetes/node-feature-discovery/features.d/gfd
nvidia.com/cuda.runtime.minor=8
nvidia.com/gpu.replicas=1
nvidia.com/gpu.family=ampere
nvidia.com/gpu.count=4
nvidia.com/gpu.memory=81920
nvidia.com/gfd.timestamp=1746706627
nvidia.com/vgpu.present=false
nvidia.com/cuda.runtime.major=12
nvidia.com/cuda.runtime-version.minor=8
nvidia.com/mps.capable=false
nvidia.com/cuda.driver-version.full=570.124.06
nvidia.com/gpu.compute.minor=0
nvidia.com/cuda.driver-version.minor=124
nvidia.com/mig.strategy=single
nvidia.com/gpu.machine=ThinkSystem-SR670-V2
nvidia.com/mig.capable=true
nvidia.com/gpu.product=NVIDIA-A100-80GB-PCIe
nvidia.com/cuda.driver.rev=06
nvidia.com/cuda.runtime-version.major=12
nvidia.com/cuda.driver.major=570
nvidia.com/cuda.runtime-version.full=12.8
nvidia.com/cuda.driver-version.revision=06
nvidia.com/gpu.sharing-strategy=none
nvidia.com/gpu.compute.major=8
nvidia.com/cuda.driver.minor=124
nvidia.com/cuda.driver-version.major=570
nvidia.com/gpu.mode=compute

Logs from nfd-worker running in the node looks pretty normal:

I0519 09:41:42.634897       1 nfd-worker.go:564] "starting feature discovery..."
I0519 09:41:42.635283       1 nfd-worker.go:577] "feature discovery completed"
I0519 09:42:42.635994       1 nfd-worker.go:564] "starting feature discovery..."
I0519 09:42:42.636399       1 nfd-worker.go:577] "feature discovery completed"

Logs from nfd-master:

I0502 10:52:28.072390       1 nfd-master.go:1230] "configuration file parsed" path="/etc/kubernetes/node-feature-discovery/nfd-master.conf"
I0502 10:52:28.072705       1 nfd-master.go:1275] "configuration successfully updated" configuration=<
        AutoDefaultNs: true
        DenyLabelNs: {}
        EnableTaints: true
        ExtraLabelNs:
          nvidia.com: {}
        Klog: {}
        LabelWhiteList: null
        LeaderElection:
          LeaseDuration:
            Duration: 15000000000
          RenewDeadline:
            Duration: 10000000000
          RetryPeriod:
            Duration: 2000000000
        NfdApiParallelism: 10
        NoPublish: false
        Restrictions:
          AllowOverwrite: true
          DenyNodeFeatureLabels: false
          DisableAnnotations: false
          DisableExtendedResources: false
          DisableLabels: false
          NodeFeatureNamespaceSelector: null
        ResyncPeriod:
          Duration: 3600000000000

ConfigMaps:

# kubectl -n node-feature-discovery get cm nfd-node-feature-discovery-master-conf -oyaml
apiVersion: v1
data:
  nfd-master.conf: "null"
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/instance: nfd
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: node-feature-discovery
    app.kubernetes.io/version: v0.17.3
    helm.sh/chart: node-feature-discovery-0.17.3
  name: nfd-node-feature-discovery-master-conf
  namespace: node-feature-discovery

---

# kubectl -n node-feature-discovery get cm nfd-node-feature-discovery-worker-conf -oyaml
apiVersion: v1
data:
  nfd-worker.conf: |-
    core:
      cacheTTL: 30s
      sleepInterval: 60s
    sources:
      cpu:
        cpuid:
          attributeBlacklist:
          - BMI1
          - BMI2
          - CLMUL
          - CMOV
          - CX16
          - ERMS
          - F16C
          - LZCNT
          - MMX
          - MMXEXT
          - NX
          - POPCNT
          - RDRAND
          - RDSEED
          - RDTSCP
          - SGX
          - SGXLC
          - SSE
          - SSE2
          - SSE3
          - SSE4.1
          - SSE4.2
          - SSSE3
          - TDX_GUEST
      local:
        dir: /etc/kubernetes/node-feature-discovery/features.d
        labels:
        - nvidia.com/*
      pci:
        deviceClassWhitelist:
        - "02"
        - "0200"
        - "0207"
        - "0300"
        - "0302"
        deviceLabelFields:
        - vendor
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/instance: nfd
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: node-feature-discovery
    app.kubernetes.io/version: v0.17.3
    helm.sh/chart: node-feature-discovery-0.17.3
  name: nfd-node-feature-discovery-worker-conf
  namespace: node-feature-discovery

I'm using version v0.17.3.

🥺 Any help is highly appreciated!!

[marquiz]

@cmontemuino looks strange, indeed. Could you run nfd-worker with -vmodule local=4 (or core.klog.vmodule: "local=4" in the config) to see what the nfd-worker actually reads from the features.d file?

@ArangoGutierrez PTAL

[cmontemuino]

@marquiz thanks for the help!

I've took another cluster where the same problem is present, just to confirm it's not a one-case scenario.

According to worker's logs, the features.d file is properly loaded and NodeFeature object is updated:

I0520 07:20:05.872605       1 nfd-worker.go:314] "Node Feature Discovery Worker" version="v0.17.3" nodeName="k8sworkergpu-001" namespace="node-feature-discovery"
I0520 07:20:05.872947       1 nfd-worker.go:519] "configuration file parsed" path="/etc/kubernetes/node-feature-discovery/nfd-worker.conf"
I0520 07:20:05.877340       1 nfd-worker.go:554] "configuration successfully updated" configuration={"Core":{"Klog":{"vmodule":"local=4"},"LabelWhiteList":"","NoPublish":false,"NoO
I0520 07:20:05.877460       1 metrics.go:44] "metrics server starting" port=":8081"
I0520 07:20:05.885632       1 local.go:274] "feature file read" fileName="gfd" features=<
        nvidia.com/cuda.driver-version.full: 570.124.06
        nvidia.com/cuda.driver-version.major: "570"
        nvidia.com/cuda.driver-version.minor: "124"
        nvidia.com/cuda.driver-version.revision: "06"
        nvidia.com/cuda.driver.major: "570"
        nvidia.com/cuda.driver.minor: "124"
        nvidia.com/cuda.driver.rev: "06"
        nvidia.com/cuda.runtime-version.full: "12.8"
        nvidia.com/cuda.runtime-version.major: "12"
        nvidia.com/cuda.runtime-version.minor: "8"
        nvidia.com/cuda.runtime.major: "12"
        nvidia.com/cuda.runtime.minor: "8"
        nvidia.com/gfd.timestamp: "1747637203"
        nvidia.com/gpu.compute.major: "8"
        nvidia.com/gpu.compute.minor: "0"
        nvidia.com/gpu.count: "2"
        nvidia.com/gpu.family: ampere
        nvidia.com/gpu.machine: ThinkSystem-SR670-V2
        nvidia.com/gpu.memory: "81920"
        nvidia.com/gpu.mode: compute
        nvidia.com/gpu.product: NVIDIA-A100-80GB-PCIe
        nvidia.com/gpu.replicas: "1"
        nvidia.com/gpu.sharing-strategy: none
        nvidia.com/mig.capable: "true"
        nvidia.com/mig.strategy: single
        nvidia.com/mps.capable: "false"
        nvidia.com/vgpu.present: "false"
 >
I0520 07:20:05.886054       1 local.go:137] "discovered features" featureSource="local" features=<
        attributes:
          feature:
            elements:
              nvidia.com/cuda.driver-version.full: 570.124.06
              nvidia.com/cuda.driver-version.major: "570"
              nvidia.com/cuda.driver-version.minor: "124"
              nvidia.com/cuda.driver-version.revision: "06"
              nvidia.com/cuda.driver.major: "570"
              nvidia.com/cuda.driver.minor: "124"
              nvidia.com/cuda.driver.rev: "06"
              nvidia.com/cuda.runtime-version.full: "12.8"
              nvidia.com/cuda.runtime-version.major: "12"
              nvidia.com/cuda.runtime-version.minor: "8"
              nvidia.com/cuda.runtime.major: "12"
              nvidia.com/cuda.runtime.minor: "8"
              nvidia.com/gfd.timestamp: "1747637203"
              nvidia.com/gpu.compute.major: "8"
              nvidia.com/gpu.compute.minor: "0"
              nvidia.com/gpu.count: "2"
              nvidia.com/gpu.family: ampere
              nvidia.com/gpu.machine: ThinkSystem-SR670-V2
              nvidia.com/gpu.memory: "81920"
              nvidia.com/gpu.mode: compute
              nvidia.com/gpu.product: NVIDIA-A100-80GB-PCIe
              nvidia.com/gpu.replicas: "1"
              nvidia.com/gpu.sharing-strategy: none
              nvidia.com/mig.capable: "true"
              nvidia.com/mig.strategy: single
              nvidia.com/mps.capable: "false"
              nvidia.com/vgpu.present: "false"
          label:
            elements:
              nvidia.com/cuda.driver-version.full: 570.124.06
              nvidia.com/cuda.driver-version.major: "570"
              nvidia.com/cuda.driver-version.minor: "124"
              nvidia.com/cuda.driver-version.revision: "06"
              nvidia.com/cuda.driver.major: "570"
              nvidia.com/cuda.driver.minor: "124"
              nvidia.com/cuda.driver.rev: "06"
              nvidia.com/cuda.runtime-version.full: "12.8"
              nvidia.com/cuda.runtime-version.major: "12"
              nvidia.com/cuda.runtime-version.minor: "8"
              nvidia.com/cuda.runtime.major: "12"
              nvidia.com/cuda.runtime.minor: "8"
              nvidia.com/gfd.timestamp: "1747637203"
              nvidia.com/gpu.compute.major: "8"
              nvidia.com/gpu.compute.minor: "0"
              nvidia.com/gpu.count: "2"
              nvidia.com/gpu.family: ampere
              nvidia.com/gpu.machine: ThinkSystem-SR670-V2
              nvidia.com/gpu.memory: "81920"
              nvidia.com/gpu.mode: compute
              nvidia.com/gpu.product: NVIDIA-A100-80GB-PCIe
              nvidia.com/gpu.replicas: "1"
              nvidia.com/gpu.sharing-strategy: none
              nvidia.com/mig.capable: "true"
              nvidia.com/mig.strategy: single
              nvidia.com/mps.capable: "false"
              nvidia.com/vgpu.present: "false"
        flags: {}
        instances: {}
 >
I0520 07:20:05.909956       1 nfd-worker.go:564] "starting feature discovery..."
I0520 07:20:05.910453       1 nfd-worker.go:577] "feature discovery completed"
I0520 07:20:05.962177       1 nfd-worker.go:693] "updating NodeFeature object" nodefeature="node-feature-discovery/k8sworkergpu-001"

I notice labels do not get updated. The worker keeps showing "feature discovery completed", but the "updating NodeFeature object" message does not keep showing.

So I've removed a couple of labels to verify, including annotation nfd.node.kubernetes.io/feature-labels, killed the worker pod in that node, and then all labels with old values have been brought back to life. Annotation nfd.node.kubernetes.io/feature-labels too with old values in it.

Also noticed nvidia.com/gfd.timestamp has a very old value does not get updated in the node.

• In labels: 1706195200 --> Thursday, January 25, 2024 3:06:40 PM
• In features.d file: 1747637203 --> Monday, May 19, 2025 6:46:43 AM

---

I also increased the master's logging and this is what I see:

I0520 07:54:24.206237       1 nfd-master.go:277] "Node Feature Discovery Master" version="v0.17.3" nodeName="k8sworker-003" namespace="node-feature-discovery"
I0520 07:54:24.206422       1 nfd-master.go:1230] "configuration file parsed" path="/etc/kubernetes/node-feature-discovery/nfd-master.conf"
I0520 07:54:24.206599       1 nfd-master.go:1275] "configuration successfully updated" configuration=<
        AutoDefaultNs: true
        DenyLabelNs: {}
        EnableTaints: true
        ExtraLabelNs:
          nvidia.com: {}
        Klog:
          vmodule: local=4
        LabelWhiteList: null
        LeaderElection:
          LeaseDuration:
            Duration: 15000000000
          RenewDeadline:
            Duration: 10000000000
          RetryPeriod:
            Duration: 2000000000
        NfdApiParallelism: 10
        NoPublish: false
        Restrictions:
          AllowOverwrite: true
          DenyNodeFeatureLabels: false
          DisableAnnotations: false
          DisableExtendedResources: false
          DisableLabels: false
          NodeFeatureNamespaceSelector: null
        ResyncPeriod:
          Duration: 3600000000000
 >
I0520 07:54:24.206650       1 nfd-master.go:1359] "starting the nfd api controller"
I0520 07:54:24.807781       1 nfd-api-controller.go:202] "informer caches synced" duration="601.017775ms"
I0520 07:54:24.807804       1 updater-pool.go:143] "starting the NFD master updater pool" parallelism=10
I0520 07:54:24.809862       1 metrics.go:44] "metrics server starting" port=":8081"
I0520 07:54:24.809895       1 leaderelection.go:257] attempting to acquire leader lease node-feature-discovery/nfd-master.nfd.kubernetes.io...
I0520 07:54:24.810004       1 component.go:34] [core][Server #1]Server created
I0520 07:54:24.810031       1 nfd-master.go:366] "gRPC health server serving" port=8082
I0520 07:54:24.810085       1 component.go:34] [core][Server #1 ListenSocket #2]ListenSocket created

I've removed labels and annotations as before, and then master's logs shows the following:

I0520 07:59:03.870608       1 nfd-master.go:1132] "node updated" nodeName="k8sworkergpu-001"

All values reverted back.

Where is the NodeFeature object being stored? Perhaps I could take a look and see if it has outdated info in it.

[cmontemuino]

🗣️ Found the issue: NodeFeature resources in the gpu-operator namespace.

Lesson learnt: look if there are multiple NodeFeature resources for the same node in different namespaces.

@marquiz, perhaps an action point for the project would be to log the NodeFeature resource name and namespace that's being updated/read.

I'll close the issue.

[marquiz]

Lesson learnt: look if there are multiple NodeFeature resources for the same node in different namespaces.

@marquiz, perhaps an action point for the project would be to log the NodeFeature resource name and namespace that's being updated/read.

Good that you found that. @cmontemuino if you will, could you create an issue about the logging enhancement?

Some thinking aloud: The NodeFeature business is a possible source of confusion, I admit, as the nfd-master (by default) blindly watches all NodeFeatures from all namespaces in the cluster. The thinking behind was to make it easy to write nfd extensions: create NodeFeature in your namespace and nfd will pick that up. There are now restrictions config with which we can limit the namespaces to look into, but the defaults pick everything. That being said, this wouldn't have helped in your case if the gpu-operator was not working correctly. In any case we could consider changing the default configuration so that nfd only picks NodeFeatures from it's own namespace.

Thoughts @ArangoGutierrez? Does the gpu-operator still create its own NodeFeature objects?