Node SystemUUID does not match VM ID in xen-orchestra

[deefdragon]

Bug Report

Description

I have installed xo-ccm and it is able to look up information about the given nodes, however, it is failing to find the ID of the VMs in XO because the systemUUID it is using to look up the node is not the same UUID as the vm in XO.

I am unsure how to rectify this to get the ccm to correctly identify the node.

(In general, it would be nice for the documentation to list what the expectations are for getting the CCM to connect everything, and a general "how this works". I had to dig into the code to determine what the potential problem was, and why the UUID getting used (the systemUUID) was not the VM uuid)

Logs

I0705 04:11:12.986523       1 instances.go:117] "instances.InstanceMetadata() called" node="k31-w-ad076d"
I0705 04:11:12.986563       1 instances.go:127] "instances.InstanceMetadata() empty providerID, trying find node" node="k31-w-ad076d" uuid="6a87cb0f-ca4c-ffa5-3ca2-fc398fb25eac"
E0705 04:11:12.990870       1 node_controller.go:285] Error getting instance metadata for node addresses: instances.InstanceMetadata() - failed to find instance by uuid k31-w-ad076d: API error: 404 Not Found - {
  "error": "no such VM 6a87cb0f-ca4c-ffa5-3ca2-fc398fb25eac"
}, skipped

(This VM's uuid for context is 0fcb876a-4cca-a5ff-3ca2-fc398fb25eac)

Environment

• Plugin version:
• Kubernetes version: [kubectl version --short]

Client Version: v1.31.2
Kustomize Version: v5.4.2
Server Version: v1.31.1

• Node describe: [kubectl describe node <node>]

Name:               k31-w-ad076d
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=k31-w-ad076d
                    kubernetes.io/os=linux
Annotations:        csi.volume.kubernetes.io/nodeid: {"org.democratic-csi.truenas-nfs":"k31-w-ad076d"}
                    flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"2e:60:de:75:f0:55"}
                    flannel.alpha.coreos.com/backend-type: vxlan
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: 192.168.35.155
                    kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/cri-dockerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Fri, 15 Nov 2024 21:03:37 -0600
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  k31-w-ad076d
  AcquireTime:     <unset>
  RenewTime:       Fri, 04 Jul 2025 23:23:14 -0500
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Thu, 19 Jun 2025 23:15:45 -0500   Thu, 19 Jun 2025 23:15:45 -0500   FlannelIsUp                  Flannel is running on this node
  MemoryPressure       False   Fri, 04 Jul 2025 23:20:37 -0500   Thu, 16 Jan 2025 03:33:39 -0600   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Fri, 04 Jul 2025 23:20:37 -0500   Fri, 30 May 2025 18:56:02 -0500   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Fri, 04 Jul 2025 23:20:37 -0500   Thu, 16 Jan 2025 03:33:39 -0600   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Fri, 04 Jul 2025 23:20:37 -0500   Thu, 16 Jan 2025 03:33:39 -0600   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  192.168.35.155
  Hostname:    k31-w-ad076d
Capacity:
  cpu:                6
  ephemeral-storage:  74942236Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             24600764Ki
  pods:               110
Allocatable:
  cpu:                6
  ephemeral-storage:  69066764584
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             24498364Ki
  pods:               110
System Info:
  Machine ID:                 9bae21fe1e10d9da92da72599fd7b4a3
  System UUID:                6a87cb0f-ca4c-ffa5-3ca2-fc398fb25eac
  Boot ID:                    f03bfe3a-59a3-47af-9022-13c37e348fbf
  Kernel Version:             6.8.0-60-generic
  OS Image:                   Ubuntu 24.04.1 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://27.4.1
  Kubelet Version:            v1.31.2
  Kube-Proxy Version:         v1.31.2
PodCIDR:                      10.244.31.0/24
PodCIDRs:                     10.244.31.0/24
Non-terminated Pods:          (30 in total)
  Namespace                   Name                                                               CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                                               ------------  ----------  ---------------  -------------  ---

...removed...

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                4521m (75%)   7675m (127%)
  memory             5811Mi (24%)  24970Mi (104%)
  ephemeral-storage  50Mi (0%)     2Gi (3%)
  hugepages-1Gi      0 (0%)        0 (0%)
  hugepages-2Mi      0 (0%)        0 (0%)
Events:              <none>

[deefdragon]

Doing some more digging, this appears to be a result of the VM UUID not matching the UUID of the vm as listed in xen-orcestra. I checked using dmidecode and the UUID matches the systemUUID in kubernetes, but that is NOT the uuid of the VM in xen-orchestra.

It appears that the value pulled in to systemUUID in kubernetes was the value at /sys/devices/virtual/dmi/id/product_uuid and not the value at /sys/devices/virtual/dmi/id/product_serial (the latter of which matches the UUID of the VM in xen-orchestra).

I tried doing a bit more digging in the k8s code itself. I attempted to trace how kubernetes pulls this information, but was unable to follow past here as the sysFS that gets passed in is set when the cadvisor is constructed here and cant easily follow it beyond that without actually pulling the k8s codebase.

[deefdragon]

Going to cross post the forum discussion I had on this issue, trying to get the UUID to be updated to the correct value. Comment from Oliver was that the ideal would be more to use a xenstore read vs getting the data from dmidecode.

Im wondering if it would be possible to have the ccm, as an option, deploy a daemonset whos only job is to grab the value from xenstore read might be a way arround this. Not ideal, and Im not sure if a daemonset'ed container would be able to get the data, but a suggestion.

[gCyrille]

Hi @deefdragon thank you for the bug report!

That's true the documentation is missing of "how it works", I still have work to do on the docs 😅

I will search an alternative to the "systemUUID". The xenstore must be the most reliable way todo (but maybe not the easiest).

I'm looking to reproduce the issue in our lab to make it easier to understand and fix it.

[gCyrille]

I was unable to reproduce the issue. Thanks to the XCP-ng team, the bug was identified on their end.

I am looking for a workaround on the CCM. It seems difficult to use something other than the system UUID with the current code architecture.

[gCyrille]

What you can do is to set manually the provider ID:

Set the --provider-id flag in the kubelet arguments to specify the VM ID --provider-id=xenorchestra://<pool_ID/<vm_ID

[deefdragon]

unfortunately as these vms are cloned from a source vm, and deployed via infra as code, I would have to manually edit them after deployment, which is quite time consuming.

I would like to note, having the ability to pull data from the xenstore would be useful beyond just the uuid, as it could allow setting up labels on the worker by configuring the xenstore, (which can be done at vm creation time in terraform atleast)

that's probably worth pulling out to it's own issue for long term tracking tho, as you said it would require some rearchitecting, and it's better to not have that muddied by this.

Thank you for the assistance here.

[gCyrille]

I'm note sure if it can help, but you can set the providerID with kubectl patch node <node_name> -p '{"spec": "providerID":"<providerID>}}' which can be done afterward and scripted from the output of your "infra as code".

But this should be done before running the CCM, because node updates may not change providerID except from "" to valid.

For the labels and all xenstore stuff, this can be done from the XO API, but it requires to know the VM ID first 😅

The "re-architecturing" might require to run a deamonset to have a (privileged) pod on each node that pool data from the VM. I don't know if this is acceptable or not 🤔