need assistance on disk partition and talos, k8s version upgrade

[sivasudheerch]

I'm setting up a cluster initially on talos v.11.5, k8s 1.34.1, later updating the talos and k8s to 1.12.2, 1.35.0 changed the wipe: false and rebooted the nodes before upgrade, I'm encountering two issues

1. k8s upgrade is halted at kubelet restart
2. disk partition is missing in volumestatus or mount status (we're using this disk partition for longhorn deployment)

disk partition:

install:
disk: /dev/vda
image: factory.talos.dev/installer/613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245:v1.11.5
wipe: true #TRUE: Essential for applying new partition layouts on fresh install
sysctls:
vm.max_map_count: "262144"

KUBELET CONFIG

kubelet:
extraMounts:
- destination: /var/lib/longhorn
type: bind
source: /var/mnt/longhorn-data
options:
- bind
- rshared
- rw

---

1. Limit the System (Ephemeral) partition

apiVersion: v1alpha1
kind: VolumeConfig
name: EPHEMERAL
provisioning:
maxSize: 10GiB

---

2. Create the Persistent Partition

apiVersion: v1alpha1
kind: UserVolumeConfig
name: longhorn-data
provisioning:
diskSelector:
match: disk.dev_path == '/dev/vda'
minSize: 10GiB

[rothgar]

I don't think a 10GB Ephemeral partition is going to be big enough to do an upgrade. Ephemeral is where all your container images are stored and each version of Talos + k8s is going to take at least a couple GB of container images.

Do you have any logs showing why the k8s upgrade haulted?

[sivasudheerch]

(Talos: Store-4959-CN-dev) root@adcphltadvtes03:~/my-cluster-infra# talosctl -n 10.149.51.169 logs kubelet | grep -i Error
10.149.51.169: {"ts":1771263226217.132,"caller":"kubelet/kubelet_node_status.go:404","msg":"Error getting the current node from lister","err":"node \"sr495901\" not found"}
10.149.51.169: {"ts":1771263226291.1729,"caller":"kubelet/kubelet.go:2451","msg":"Skipping pod synchronization","err":"PLEG is not healthy: pleg has yet to be successful","errCauses":[{"error":"PLEG is not healthy: pleg has yet to be successful"}]}
10.149.51.169: {"ts":1771263226400.3154,"caller":"kubelet/kubelet_node_status.go:486","msg":"Error updating node status, will retry","err":"error getting node \"sr495901\": node \"sr495901\" not found"}
10.149.51.169: {"ts":1771263226421.6091,"caller":"kubelet/kubelet_node_status.go:404","msg":"Error getting the current node from lister","err":"node \"sr495901\" not found"}
10.149.51.169: {"ts":1771263226522.2686,"caller":"kubelet/kubelet_node_status.go:404","msg":"Error getting the current node from lister","err":"node \"sr495901\" not found"}
10.149.51.169: {"ts":1771263226623.2285,"caller":"kubelet/kubelet_node_status.go:404","msg":"Error getting the current node from lister","err":"node \"sr495901\" not found"}
10.149.51.169: {"ts":1771263226723.8157,"caller":"kubelet/kubelet_node_status.go:404","msg":"Error getting the current node from lister","err":"node \"sr495901\" not found"}
10.149.51.169: {"ts":1771263226824.6785,"caller":"kubelet/kubelet_node_status.go:404","msg":"Error getting the current node from lister","err":"node \"sr495901\" not found"}
10.149.51.169: {"ts":1771263226925.8315,"caller":"kubelet/kubelet_node_status.go:404","msg":"Error getting the current node from lister","err":"node \"sr495901\" not found"}
10.149.51.169: {"ts":1771263227026.873,"caller":"kubelet/kubelet_node_status.go:404","msg":"Error getting the current node from lister","err":"node \"sr495901\" not found"}
10.149.51.169: {"ts":1771263227127.081,"caller":"kubelet/kubelet_node_status.go:404","msg":"Error getting the current node from lister","err":"node \"sr495901\" not found"}
10.149.51.169: {"ts":1771264733820.151,"caller":"nestedpendingoperations/nestedpendingoperations.go:348","msg":"Operation for \"{volumeName:kubernetes.io/secret/9cf6633a-04c8-4681-8e6b-a275013c9f20-memberlist podName:9cf6633a-04c8-4681-8e6b-a275013c9f20 nodeName:}\" failed. No retries permitted until 2026-02-16 17:58:54.320072263 +0000 UTC m=+1508.646238553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume \"memberlist\" (UniqueName: \"kubernetes.io/secret/9cf6633a-04c8-4681-8e6b-a275013c9f20-memberlist\") pod \"sr495900-metallb-speaker-4bjsv\" (UID: \"9cf6633a-04c8-4681-8e6b-a275013c9f20\") : secret \"sr495900-metallb-memberlist\" not found"}
10.149.51.169: {"ts":1771264734420.7437,"caller":"nestedpendingoperations/nestedpendingoperations.go:348","msg":"Operation for \"{volumeName:kubernetes.io/secret/9cf6633a-04c8-4681-8e6b-a275013c9f20-memberlist podName:9cf6633a-04c8-4681-8e6b-a275013c9f20 nodeName:}\" failed. No retries permitted until 2026-02-16 17:58:55.420712679 +0000 UTC m=+1509.746878877 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume \"memberlist\" (UniqueName: \"kubernetes.io/secret/9cf6633a-04c8-4681-8e6b-a275013c9f20-memberlist\") pod \"sr495900-metallb-speaker-4bjsv\" (UID: \"9cf6633a-04c8-4681-8e6b-a275013c9f20\") : secret \"sr495900-metallb-memberlist\" not found"}
10.149.51.169: {"ts":1771264735428.2825,"caller":"nestedpendingoperations/nestedpendingoperations.go:348","msg":"Operation for \"{volumeName:kubernetes.io/secret/9cf6633a-04c8-4681-8e6b-a275013c9f20-memberlist podName:9cf6633a-04c8-4681-8e6b-a275013c9f20 nodeName:}\" failed. No retries permitted until 2026-02-16 17:58:57.428239103 +0000 UTC m=+1511.754405318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume \"memberlist\" (UniqueName: \"kubernetes.io/secret/9cf6633a-04c8-4681-8e6b-a275013c9f20-memberlist\") pod \"sr495900-metallb-speaker-4bjsv\" (UID: \"9cf6633a-04c8-4681-8e6b-a275013c9f20\") : secret \"sr495900-metallb-memberlist\" not found"}
10.149.51.169: {"ts":1771264737444.063,"caller":"nestedpendingoperations/nestedpendingoperations.go:348","msg":"Operation for \"{volumeName:kubernetes.io/secret/9cf6633a-04c8-4681-8e6b-a275013c9f20-memberlist podName:9cf6633a-04c8-4681-8e6b-a275013c9f20 nodeName:}\" failed. No retries permitted until 2026-02-16 17:59:01.443932488 +0000 UTC m=+1515.770098730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume \"memberlist\" (UniqueName: \"kubernetes.io/secret/9cf6633a-04c8-4681-8e6b-a275013c9f20-memberlist\") pod \"sr495900-metallb-speaker-4bjsv\" (UID: \"9cf6633a-04c8-4681-8e6b-a275013c9f20\") : secret \"sr495900-metallb-memberlist\" not found"}
10.149.51.169: {"ts":1771264752672.582,"logger":"UnhandledError","caller":"kuberuntime/kuberuntime_manager.go:1449","msg":"Unhandled Error","err":"container db-initializer start failed in pod db-init-job-mjvlp_stores-isp(3da18b2e-0c6e-4b26-9c4c-54c848f534fe): CreateContainerConfigError: secret \"postgres-db-superuser\" not found"}
10.149.51.169: {"ts":1771264752672.8833,"caller":"kubelet/pod_workers.go:1324","msg":"Error syncing pod, skipping","pod":{"name":"db-init-job-mjvlp","namespace":"stores-isp"},"podUID":"3da18b2e-0c6e-4b26-9c4c-54c848f534fe","err":"failed to \"StartContainer\" for \"db-initializer\" with CreateContainerConfigError: \"secret \\\"postgres-db-superuser\\\" not found\"","errCauses":[{"error":"failed to \"StartContainer\" for \"db-initializer\" with CreateContainerConfigError: \"secret \\\"postgres-db-superuser\\\" not found\""}]}
10.149.51.169: {"ts":1771264753023.4902,"logger":"UnhandledError","caller":"kuberuntime/kuberuntime_manager.go:1449","msg":"Unhandled Error","err":"container db-initializer start failed in pod db-init-job-mjvlp_stores-isp(3da18b2e-0c6e-4b26-9c4c-54c848f534fe): CreateContainerConfigError: secret \"postgres-db-superuser\" not found"}
10.149.51.169: {"ts":1771264753023.5354,"caller":"kubelet/pod_workers.go:1324","msg":"Error syncing pod, skipping","pod":{"name":"db-init-job-mjvlp","namespace":"stores-isp"},"podUID":"3da18b2e-0c6e-4b26-9c4c-54c848f534fe","err":"failed to \"StartContainer\" for \"db-initializer\" with CreateContainerConfigError: \"secret \\\"postgres-db-superuser\\\" not found\"","errCauses":[{"error":"failed to \"StartContainer\" for \"db-initializer\" with CreateContainerConfigError: \"secret \\\"postgres-db-superuser\\\" not found\""}]}
10.149.51.169: {"ts":1771264764294.5107,"logger":"UnhandledError","caller":"kuberuntime/kuberuntime_manager.go:1449","msg":"Unhandled Error","err":"container db-initializer start failed in pod db-init-job-mjvlp_stores-isp(3da18b2e-0c6e-4b26-9c4c-54c848f534fe): CreateContainerConfigError: secret \"postgres-db-superuser\" not found"}
10.149.51.169: {"ts":1771264764294.5598,"caller":"kubelet/pod_workers.go:1324","msg":"Error syncing pod, skipping","pod":{"name":"db-init-job-mjvlp","namespace":"stores-isp"},"podUID":"3da18b2e-0c6e-4b26-9c4c-54c848f534fe","err":"failed to \"StartContainer\" for \"db-initializer\" with CreateContainerConfigError: \"secret \\\"postgres-db-superuser\\\" not found\"","errCauses":[{"error":"failed to \"StartContainer\" for \"db-initializer\" with CreateContainerConfigError: \"secret \\\"postgres-db-superuser\\\" not found\""}]}
10.149.51.169: {"ts":1771264778294.639,"logger":"UnhandledError","caller":"kuberuntime/kuberuntime_manager.go:1449","msg":"Unhandled Error","err":"container db-initializer start failed in pod db-init-job-mjvlp_stores-isp(3da18b2e-0c6e-4b26-9c4c-54c848f534fe): CreateContainerConfigError: secret \"postgres-db-superuser\" not found"}
10.149.51.169: {"ts":1771264778294.8953,"caller":"kubelet/pod_workers.go:1324","msg":"Error syncing pod, skipping","pod":{"name":"db-init-job-mjvlp","namespace":"stores-isp"},"podUID":"3da18b2e-0c6e-4b26-9c4c-54c848f534fe","err":"failed to \"StartContainer\" for \"db-initializer\" with CreateContainerConfigError: \"secret \\\"postgres-db-superuser\\\" not found\"","errCauses":[{"error":"failed to \"StartContainer\" for \"db-initializer\" with CreateContainerConfigError: \"secret \\\"postgres-db-superuser\\\" not found\""}]}
10.149.51.169: {"ts":1771264790296.331,"logger":"UnhandledError","caller":"kuberuntime/kuberuntime_manager.go:1449","msg":"Unhandled Error","err":"container db-initializer start failed in pod db-init-job-mjvlp_stores-isp(3da18b2e-0c6e-4b26-9c4c-54c848f534fe): CreateContainerConfigError: secret \"postgres-db-superuser\" not found"}
10.149.51.169: {"ts":1771264790296.388,"caller":"kubelet/pod_workers.go:1324","msg":"Error syncing pod, skipping","pod":{"name":"db-init-job-mjvlp","namespace":"stores-isp"},"podUID":"3da18b2e-0c6e-4b26-9c4c-54c848f534fe","err":"failed to \"StartContainer\" for \"db-initializer\" with CreateContainerConfigError: \"secret \\\"postgres-db-superuser\\\" not found\"","errCauses":[{"error":"failed to \"StartContainer\" for \"db-initializer\" with CreateContainerConfigError: \"secret \\\"postgres-db-superuser\\\" not found\""}]}
10.149.51.169: {"ts":1771264803293.9497,"logger":"UnhandledError","caller":"kuberuntime/kuberuntime_manager.go:1449","msg":"Unhandled Error","err":"container db-initializer start failed in pod db-init-job-mjvlp_stores-isp(3da18b2e-0c6e-4b26-9c4c-54c848f534fe): CreateContainerConfigError: secret \"postgres-db-superuser\" not found"}
10.149.51.169: {"ts":1771264803294.0042,"caller":"kubelet/pod_workers.go:1324","msg":"Error syncing pod, skipping","pod":{"name":"db-init-job-mjvlp","namespace":"stores-isp"},"podUID":"3da18b2e-0c6e-4b26-9c4c-54c848f534fe","err":"failed to \"StartContainer\" for \"db-initializer\" with CreateContainerConfigError: \"secret \\\"postgres-db-superuser\\\" not found\"","errCauses":[{"error":"failed to \"StartContainer\" for \"db-initializer\" with CreateContainerConfigError: \"secret \\\"postgres-db-superuser\\\" not found\""}]}
10.149.51.169: {"ts":1771264816294.306,"logger":"UnhandledError","caller":"kuberuntime/kuberuntime_manager.go:1449","msg":"Unhandled Error","err":"container db-initializer start failed in pod db-init-job-mjvlp_stores-isp(3da18b2e-0c6e-4b26-9c4c-54c848f534fe): CreateContainerConfigError: secret \"postgres-db-superuser\" not found"}
10.149.51.169: {"ts":1771264816294.3455,"caller":"kubelet/pod_workers.go:1324","msg":"Error syncing pod, skipping","pod":{"name":"db-init-job-mjvlp","namespace":"stores-isp"},"podUID":"3da18b2e-0c6e-4b26-9c4c-54c848f534fe","err":"failed to \"StartContainer\" for \"db-initializer\" with CreateContainerConfigError: \"secret \\\"postgres-db-superuser\\\" not found\"","errCauses":[{"error":"failed to \"StartContainer\" for \"db-initializer\" with CreateContainerConfigError: \"secret \\\"postgres-db-superuser\\\" not found\""}]}
10.149.51.169: {"ts":1771264828295.542,"logger":"UnhandledError","caller":"kuberuntime/kuberuntime_manager.go:1449","msg":"Unhandled Error","err":"container db-initializer start failed in pod db-init-job-mjvlp_stores-isp(3da18b2e-0c6e-4b26-9c4c-54c848f534fe): CreateContainerConfigError: secret \"postgres-db-superuser\" not found"}
10.149.51.169: {"ts":1771264828295.603,"caller":"kubelet/pod_workers.go:1324","msg":"Error syncing pod, skipping","pod":{"name":"db-init-job-mjvlp","namespace":"stores-isp"},"podUID":"3da18b2e-0c6e-4b26-9c4c-54c848f534fe","err":"failed to \"StartContainer\" for \"db-initializer\" with CreateContainerConfigError: \"secret \\\"postgres-db-superuser\\\" not found\"","errCauses":[{"error":"failed to \"StartContainer\" for \"db-initializer\" with CreateContainerConfigError: \"secret \\\"postgres-db-superuser\\\" not found\""}]}
10.149.51.169: {"ts":1771265331732.3254,"caller":"internal/log.go:32","msg":"ContainerStatus from runtime service failed","containerID":"65cb5addf0bdd06c73fcd74c7edac46239682f1d5282c0f3fe65c04e04567d50","err":"rpc error: code = NotFound desc = an error occurred when try to find container \"65cb5addf0bdd06c73fcd74c7edac46239682f1d5282c0f3fe65c04e04567d50\": not found"}
10.149.51.169: {"ts":1771265331732.3965,"caller":"kubelet/pod_container_deletor.go:53","msg":"DeleteContainer returned error","v":0,"containerID":{"Type":"containerd","ID":"65cb5addf0bdd06c73fcd74c7edac46239682f1d5282c0f3fe65c04e04567d50"},"err":"failed to get container status \"65cb5addf0bdd06c73fcd74c7edac46239682f1d5282c0f3fe65c04e04567d50\": rpc error: code = NotFound desc = an error occurred when try to find container \"65cb5addf0bdd06c73fcd74c7edac46239682f1d5282c0f3fe65c04e04567d50\": not found"}
10.149.51.169: {"ts":1771265331732.926,"caller":"internal/log.go:32","msg":"ContainerStatus from runtime service failed","containerID":"8cb206a1f6ec1b48b341367cfdfaa67c791e63304a2caf9fee2b9d6e1285ec9b","err":"rpc error: code = NotFound desc = an error occurred when try to find container \"8cb206a1f6ec1b48b341367cfdfaa67c791e63304a2caf9fee2b9d6e1285ec9b\": not found"}
10.149.51.169: {"ts":1771265331733.0867,"caller":"kubelet/pod_container_deletor.go:53","msg":"DeleteContainer returned error","v":0,"containerID":{"Type":"containerd","ID":"8cb206a1f6ec1b48b341367cfdfaa67c791e63304a2caf9fee2b9d6e1285ec9b"},"err":"failed to get container status \"8cb206a1f6ec1b48b341367cfdfaa67c791e63304a2caf9fee2b9d6e1285ec9b\": rpc error: code = NotFound desc = an error occurred when try to find container \"8cb206a1f6ec1b48b341367cfdfaa67c791e63304a2caf9fee2b9d6e1285ec9b\": not found"}
10.149.51.169: {"ts":1771265537199.5288,"caller":"internal/log.go:32","msg":"ContainerStatus from runtime service failed","containerID":"58a1f27f102ac11a8c783fa4b42de6120f47d6169d7cd0529059a3548f0335f4","err":"rpc error: code = NotFound desc = an error occurred when try to find container \"58a1f27f102ac11a8c783fa4b42de6120f47d6169d7cd0529059a3548f0335f4\": not found"}
10.149.51.169: {"ts":1771265537199.674,"caller":"kubelet/pod_container_deletor.go:53","msg":"DeleteContainer returned error","v":0,"containerID":{"Type":"containerd","ID":"58a1f27f102ac11a8c783fa4b42de6120f47d6169d7cd0529059a3548f0335f4"},"err":"failed to get container status \"58a1f27f102ac11a8c783fa4b42de6120f47d6169d7cd0529059a3548f0335f4\": rpc error: code = NotFound desc = an error occurred when try to find container \"58a1f27f102ac11a8c783fa4b42de6120f47d6169d7cd0529059a3548f0335f4\": not found"}
10.149.51.169: {"ts":1771265796945.5789,"caller":"internal/log.go:32","msg":"ContainerStatus from runtime service failed","containerID":"deb07cad66adaa920379709cdced77c95a63dbcbef1b7941367ffc7947c0e3c0","err":"rpc error: code = NotFound desc = an error occurred when try to find container \"deb07cad66adaa920379709cdced77c95a63dbcbef1b7941367ffc7947c0e3c0\": not found"}
10.149.51.169: {"ts":1771265796945.6238,"caller":"kubelet/pod_container_deletor.go:53","msg":"DeleteContainer returned error","v":0,"containerID":{"Type":"containerd","ID":"deb07cad66adaa920379709cdced77c95a63dbcbef1b7941367ffc7947c0e3c0"},"err":"failed to get container status \"deb07cad66adaa920379709cdced77c95a63dbcbef1b7941367ffc7947c0e3c0\": rpc error: code = NotFound desc = an error occurred when try to find container \"deb07cad66adaa920379709cdced77c95a63dbcbef1b7941367ffc7947c0e3c0\": not found"}
(Talos: Store-4959-CN-dev) root@adcphltadvtes03:~/my-cluster-infra# talosctl -n 10.149.51.169 logs containerd
10.149.51.169: {"level":"info","msg":"starting containerd","revision":"c74fd8780002eb26bd5940ae339d690d891221c2","time":"2026-02-16T17:33:41.999939323Z","version":"v2.1.6"}
10.149.51.169: {"id":"io.containerd.content.v1.content","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.011403767Z","type":"io.containerd.content.v1"}
10.149.51.169: {"id":"io.containerd.image-verifier.v1.bindir","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.011557014Z","type":"io.containerd.image-verifier.v1"}
10.149.51.169: {"id":"io.containerd.warning.v1.deprecations","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.011627534Z","type":"io.containerd.warning.v1"}
10.149.51.169: {"id":"io.containerd.snapshotter.v1.native","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.011664716Z","type":"io.containerd.snapshotter.v1"}
10.149.51.169: {"id":"io.containerd.snapshotter.v1.overlayfs","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.011763899Z","type":"io.containerd.snapshotter.v1"}
10.149.51.169: {"id":"io.containerd.event.v1.exchange","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.012084554Z","type":"io.containerd.event.v1"}
10.149.51.169: {"id":"io.containerd.monitor.task.v1.cgroups","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.013850446Z","type":"io.containerd.monitor.task.v1"}
10.149.51.169: {"id":"io.containerd.metadata.v1.bolt","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.014354134Z","type":"io.containerd.metadata.v1"}
10.149.51.169: {"level":"info","msg":"metadata content store policy set","policy":"shared","time":"2026-02-16T17:33:42.014496318Z"}
10.149.51.169: {"id":"io.containerd.gc.v1.scheduler","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.015846477Z","type":"io.containerd.gc.v1"}
10.149.51.169: {"id":"io.containerd.differ.v1.walking","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.015934663Z","type":"io.containerd.differ.v1"}
10.149.51.169: {"id":"io.containerd.lease.v1.manager","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.015982697Z","type":"io.containerd.lease.v1"}
10.149.51.169: {"id":"io.containerd.service.v1.containers-service","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016016460Z","type":"io.containerd.service.v1"}
10.149.51.169: {"id":"io.containerd.service.v1.content-service","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016040154Z","type":"io.containerd.service.v1"}
10.149.51.169: {"id":"io.containerd.service.v1.diff-service","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016060711Z","type":"io.containerd.service.v1"}
10.149.51.169: {"id":"io.containerd.service.v1.images-service","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016094960Z","type":"io.containerd.service.v1"}
10.149.51.169: {"id":"io.containerd.service.v1.introspection-service","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016121574Z","type":"io.containerd.service.v1"}
10.149.51.169: {"id":"io.containerd.service.v1.namespaces-service","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016143060Z","type":"io.containerd.service.v1"}
10.149.51.169: {"id":"io.containerd.service.v1.snapshots-service","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016180112Z","type":"io.containerd.service.v1"}
10.149.51.169: {"id":"io.containerd.shim.v1.manager","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016202141Z","type":"io.containerd.shim.v1"}
10.149.51.169: {"id":"io.containerd.runtime.v2.task","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016229758Z","type":"io.containerd.runtime.v2"}
10.149.51.169: {"id":"io.containerd.service.v1.tasks-service","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016384723Z","type":"io.containerd.service.v1"}
10.149.51.169: {"id":"io.containerd.grpc.v1.containers","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016444086Z","type":"io.containerd.grpc.v1"}
10.149.51.169: {"id":"io.containerd.grpc.v1.content","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016472240Z","type":"io.containerd.grpc.v1"}
10.149.51.169: {"id":"io.containerd.grpc.v1.diff","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016512754Z","type":"io.containerd.grpc.v1"}
10.149.51.169: {"id":"io.containerd.grpc.v1.events","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016559796Z","type":"io.containerd.grpc.v1"}
10.149.51.169: {"id":"io.containerd.grpc.v1.images","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016594242Z","type":"io.containerd.grpc.v1"}
10.149.51.169: {"id":"io.containerd.grpc.v1.introspection","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016623435Z","type":"io.containerd.grpc.v1"}
10.149.51.169: {"id":"io.containerd.grpc.v1.leases","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016645106Z","type":"io.containerd.grpc.v1"}
10.149.51.169: {"id":"io.containerd.grpc.v1.namespaces","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016697446Z","type":"io.containerd.grpc.v1"}
10.149.51.169: {"id":"io.containerd.grpc.v1.snapshots","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016722027Z","type":"io.containerd.grpc.v1"}
10.149.51.169: {"id":"io.containerd.streaming.v1.manager","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016753463Z","type":"io.containerd.streaming.v1"}
10.149.51.169: {"id":"io.containerd.grpc.v1.streaming","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016798183Z","type":"io.containerd.grpc.v1"}
10.149.51.169: {"id":"io.containerd.grpc.v1.tasks","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016823386Z","type":"io.containerd.grpc.v1"}
10.149.51.169: {"id":"io.containerd.transfer.v1.local","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016864828Z","type":"io.containerd.transfer.v1"}
10.149.51.169: {"id":"io.containerd.grpc.v1.transfer","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016911071Z","type":"io.containerd.grpc.v1"}
10.149.51.169: {"id":"io.containerd.grpc.v1.version","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016935114Z","type":"io.containerd.grpc.v1"}
10.149.51.169: {"id":"io.containerd.grpc.v1.healthcheck","level":"info","msg":"loading plugin","time":"2026-02-16T17:33:42.016963898Z","type":"io.containerd.grpc.v1"}
10.149.51.169: {"address":"/system/run/containerd/containerd.sock.ttrpc","level":"info","msg":"serving...","time":"2026-02-16T17:33:42.017233451Z"}
10.149.51.169: {"address":"/system/run/containerd/containerd.sock","level":"info","msg":"serving...","time":"2026-02-16T17:33:42.017325715Z"}
10.149.51.169: {"level":"info","msg":"containerd successfully booted in 0.018794s","time":"2026-02-16T17:33:42.017376191Z"}
10.149.51.169: {"address":"unix:///run/containerd/s/814eeaf75962b52d1228c72642b2a1e34f6ad18299621a2bc0a0d887e86f9db6","level":"info","msg":"connecting to shim apid","namespace":"system","protocol":"ttrpc","time":"2026-02-16T17:33:44.167897231Z","version":3}
10.149.51.169: {"address":"unix:///run/containerd/s/5fa90b83e183f45a761a15c4d691cc248504e7544f0fd8787e47e42f16c45345","level":"info","msg":"connecting to shim trustd","namespace":"system","protocol":"ttrpc","time":"2026-02-16T17:33:44.381776564Z","version":3}
10.149.51.169: {"address":"unix:///run/containerd/s/36dd51478408a3c10c2d441d38a580ba39abeafd0089845e6f7add11599099c0","level":"info","msg":"connecting to shim ext-iscsid","namespace":"system","protocol":"ttrpc","time":"2026-02-16T17:33:45.431408433Z","version":3}
(Talos: Store-4959-CN-dev) root@adcphltadvtes03:~/my-cluster-infra#

[sivasudheerch]

10.149.51.169: user: warning: [2026-02-16T18:32:39.616021229Z]: [talos] syncfs(/var/mnt/longhorn-data) ok {"component": "controller-runtime", "controller": "block.MountController"}
10.149.51.169: user: warning: [2026-02-16T18:32:39.619319229Z]: [talos] controller failed {"component": "controller-runtime", "controller": "block.MountController", "error": "failed to unmount "u-longhorn-data": 2 error(s) occurred:\n\tdevice or resource busy\n\ttimeout"}

[sivasudheerch]

talosctl -n 10.149.51.159 service kubelet
NODE                  10.149.51.159
ID                    kubelet
STATE                 Waiting
HEALTH                Fail
LAST HEALTH MESSAGE   service not running
EVENTS                [Waiting]: Waiting for volumes /var/mnt to be mounted (1h31m26s ago)
                     [Waiting]: Waiting for volumes /var/log/audit/kube, /var/mnt to be mounted (1h32m23s ago)
                     [Waiting]: Waiting for volumes /var/lib/kubelet/seccomp/profiles, /var/log/audit/kube, /var/mnt to be mounted (1h33m6s ago)
                     [Waiting]: Waiting for volumes /var/lib/kubelet/seccomp, /var/lib/kubelet/seccomp/profiles, /var/log/audit/kube, /var/mnt to be mounted (1h34m0s ago)
                     [Waiting]: Waiting for volumes /var/log/pods, /var/lib/kubelet/seccomp, /var/lib/kubelet/seccomp/profiles, /var/log/audit/kube, /var/mnt to be mounted (1h34m53s ago)
                     [Waiting]: Waiting for volumes /var/log/containers, /var/log/pods, /var/lib/kubelet/seccomp, /var/lib/kubelet/seccomp/profiles, /var/log/audit/kube, /var/mnt to be mounted (1h36m4s ago)
                     [Waiting]: Waiting for volumes /var/log/audit, /var/log/containers, /var/log/pods, /var/lib/kubelet/seccomp, /var/lib/kubelet/seccomp/profiles, /var/log/audit/kube, /var/mnt to be mounted (1h36m52s ago)
                     [Waiting]: Waiting for volumes /var/log, /var/log/audit, /var/log/containers, /var/log/pods, /var/lib/kubelet/seccomp, /var/lib/kubelet/seccomp/profiles, /var/log/audit/kube, /var/mnt to be mounted (1h38m16s ago)
                     [Waiting]: Waiting for volumes /var/lib/kubelet, /var/log, /var/log/audit, /var/log/containers, /var/log/pods, /var/lib/kubelet/seccomp, /var/lib/kubelet/seccomp/profiles, /var/log/audit/kube, /var/mnt to be mounted (1h42m24s ago)
                     [Waiting]: Waiting for volumes /var/lib, /var/lib/kubelet, /var/log, /var/log/audit, /var/log/containers, /var/log/pods, /var/lib/kubelet/seccomp, /var/lib/kubelet/seccomp/profiles, /var/log/audit/kube, /var/mnt to be mounted (1h43m56s ago)
                     [Waiting]: Waiting for volumes /var/lib, /var/lib/kubelet, /var/log, /var/log/audit, /var/log/containers, /var/log/pods, /var/lib/kubelet/seccomp, /var/lib/kubelet/seccomp/profiles, /var/log/audit/kube, /var/mnt to be mounted, service "cri" to be "up", time sync, network (1h43m57s ago)
                     [Starting]: Starting service (1h43m57s ago)
                     [Failed]: Condition failed: context canceled (1h43m57s ago)
                     [Waiting]: Waiting for volumes /var/mnt to be mounted (2h10m34s ago)
                     [Waiting]: Waiting for volumes /var/log/audit/kube, /var/mnt to be mounted (2h10m45s ago)
                     [Waiting]: Waiting for volumes /var/lib/kubelet/seccomp/profiles, /var/log/audit/kube, /var/mnt to be mounted (2h11m0s ago)
                     [Waiting]: Waiting for volumes /var/lib/kubelet/seccomp, /var/lib/kubelet/seccomp/profiles, /var/log/audit/kube, /var/mnt to be mounted (2h11m12s ago)
                     [Waiting]: Waiting for volumes /var/log/pods, /var/lib/kubelet/seccomp, /var/lib/kubelet/seccomp/profiles, /var/log/audit/kube, /var/mnt to be mounted (2h11m22s ago)
                     [Waiting]: Waiting for volumes /var/log/containers, /var/log/pods, /var/lib/kubelet/seccomp, /var/lib/kubelet/seccomp/profiles, /var/log/audit/kube, /var/mnt to be mounted (2h11m29s ago)
                     [Waiting]: Waiting for volumes /var/log/audit, /var/log/containers, /var/log/pods, /var/lib/kubelet/seccomp, /var/lib/kubelet/seccomp/profiles, /var/log/audit/kube, /var/mnt to be mounted (2h11m35s ago)
                     [Waiting]: Waiting for volumes /var/log, /var/log/audit, /var/log/containers, /var/log/pods, /var/lib/kubelet/seccomp, /var/lib/kubelet/seccomp/profiles, /var/log/audit/kube, /var/mnt to be mounted (2h11m41s ago)
                     [Waiting]: Waiting for volumes /var/lib/kubelet, /var/log, /var/log/audit, /var/log/containers, /var/log/pods, /var/lib/kubelet/seccomp, /var/lib/kubelet/seccomp/profiles, /var/log/audit/kube, /var/mnt to be mounted (2h11m47s ago)
                     [Waiting]: Waiting for volumes /var/lib, /var/lib/kubelet, /var/log, /var/log/audit, /var/log/containers, /var/log/pods, /var/lib/kubelet/seccomp, /var/lib/kubelet/seccomp/profiles, /var/log/audit/kube, /var/mnt to be mounted (2h11m52s ago)
                     [Waiting]: Waiting for volumes /var/lib, /var/lib/kubelet, /var/log, /var/log/audit, /var/log/containers, /var/log/pods, /var/lib/kubelet/seccomp, /var/lib/kubelet/seccomp/profiles, /var/log/audit/kube, /var/mnt to be mounted, service "cri" to be "up", time sync, network (2h11m53s ago)
                     [Starting]: Starting service (2h11m53s ago)
                     [Finished]: Service finished successfully (2h11m53s ago)
                     [Stopping]: Sending SIGTERM to task kubelet (PID 2669, container kubelet) (2h11m53s ago)
                     [Running]: Health check successful (2h26m50s ago)
                     [Running]: Started task kubelet (PID 2669) for container kubelet (2h26m52s ago)
                     [Preparing]: Creating service runner (2h26m52s ago)
                     [Preparing]: Running pre state (2h26m52s ago)
                     [Waiting]: Waiting for volumes /var/lib, /var/lib/kubelet, /var/log, /var/log/audit, /var/log/containers, /var/log/pods, /var/lib/kubelet/seccomp, /var/lib/kubelet/seccomp/profiles, /var/log/audit/kube, /var/mnt to be mounted, service "cri" to be "up", time sync, network (2h26m53s ago)
                     [Starting]: Starting service (2h26m53s ago)

[sivasudheerch]

(Talos: Store-4959-CN-dev) root@adcphltadvtes03:~/my-cluster-infra# talosctl --nodes 10.149.51.169 upgrade-k8s --to 1.35.0
automatically detected the lowest Kubernetes version 1.34.1
discovered controlplane nodes ["10.149.51.169" "10.149.51.170" "10.149.51.173"]
discovered worker nodes []

"10.149.51.170": Talos version 1.12.2 is compatible with Kubernetes version 1.35.0
"10.149.51.173": Talos version 1.12.2 is compatible with Kubernetes version 1.35.0
"10.149.51.169": Talos version 1.12.2 is compatible with Kubernetes version 1.35.0
checking for removed Kubernetes component flags
checking for removed Kubernetes API resource versions
"10.149.51.169": pre-pulling registry.k8s.io/kube-apiserver:v1.35.0
"10.149.51.170": pre-pulling registry.k8s.io/kube-apiserver:v1.35.0
"10.149.51.173": pre-pulling registry.k8s.io/kube-apiserver:v1.35.0
"10.149.51.169": pre-pulling registry.k8s.io/kube-controller-manager:v1.35.0
"10.149.51.170": pre-pulling registry.k8s.io/kube-controller-manager:v1.35.0
"10.149.51.173": pre-pulling registry.k8s.io/kube-controller-manager:v1.35.0
"10.149.51.169": pre-pulling registry.k8s.io/kube-scheduler:v1.35.0
"10.149.51.170": pre-pulling registry.k8s.io/kube-scheduler:v1.35.0
"10.149.51.173": pre-pulling registry.k8s.io/kube-scheduler:v1.35.0
"10.149.51.169": pre-pulling ghcr.io/siderolabs/kubelet:v1.35.0
"10.149.51.170": pre-pulling ghcr.io/siderolabs/kubelet:v1.35.0
"10.149.51.173": pre-pulling ghcr.io/siderolabs/kubelet:v1.35.0
updating "kube-apiserver" to version "1.35.0"
"10.149.51.169": starting update
update kube-apiserver: v1.34.1 -> 1.35.0
"10.149.51.169": machine configuration patched
"10.149.51.169": waiting for kube-apiserver pod update
"10.149.51.169": kube-apiserver: waiting, config version mismatch: got "1", expected "2"
"10.149.51.169": kube-apiserver: waiting, config version mismatch: got "1", expected "2"
"10.149.51.169": kube-apiserver: waiting, config version mismatch: got "1", expected "2"
"10.149.51.169": kube-apiserver: waiting, config version mismatch: got "1", expected "2"
"10.149.51.169": kube-apiserver: waiting, config version mismatch: got "1", expected "2"
"10.149.51.169": kube-apiserver: waiting, config version mismatch: got "1", expected "2"
"10.149.51.169": kube-apiserver: waiting, config version mismatch: got "1", expected "2"
< "10.149.51.169": successfully updated
"10.149.51.170": starting update
update kube-apiserver: v1.34.1 -> 1.35.0
"10.149.51.170": machine configuration patched
"10.149.51.170": waiting for kube-apiserver pod update
"10.149.51.170": kube-apiserver: waiting, config version mismatch: got "1", expected "2"
"10.149.51.170": kube-apiserver: waiting, config version mismatch: got "1", expected "2"
"10.149.51.170": kube-apiserver: waiting, config version mismatch: got "1", expected "2"
"10.149.51.170": kube-apiserver: waiting, config version mismatch: got "1", expected "2"
"10.149.51.170": kube-apiserver: waiting, config version mismatch: got "1", expected "2"
"10.149.51.170": kube-apiserver: waiting, config version mismatch: got "1", expected "2"
"10.149.51.170": kube-apiserver: waiting, config version mismatch: got "1", expected "2"
"10.149.51.170": kube-apiserver: pod is not ready, waiting
< "10.149.51.170": successfully updated
"10.149.51.173": starting update
update kube-apiserver: v1.34.1 -> 1.35.0
"10.149.51.173": machine configuration patched
"10.149.51.173": waiting for kube-apiserver pod update
"10.149.51.173": kube-apiserver: waiting, config version mismatch: got "1", expected "2"
"10.149.51.173": kube-apiserver: waiting, config version mismatch: got "1", expected "2"
"10.149.51.173": kube-apiserver: waiting, config version mismatch: got "1", expected "2"
"10.149.51.173": kube-apiserver: waiting, config version mismatch: got "1", expected "2"
"10.149.51.173": kube-apiserver: waiting, config version mismatch: got "1", expected "2"
"10.149.51.173": kube-apiserver: waiting, config version mismatch: got "1", expected "2"
"10.149.51.173": kube-apiserver: waiting, config version mismatch: got "1", expected "2"
< "10.149.51.173": successfully updated
updating "kube-controller-manager" to version "1.35.0"
"10.149.51.169": starting update
update kube-controller-manager: v1.34.1 -> 1.35.0
"10.149.51.169": machine configuration patched
"10.149.51.169": waiting for kube-controller-manager pod update
"10.149.51.169": kube-controller-manager: waiting, config version mismatch: got "1", expected "2"
"10.149.51.169": kube-controller-manager: waiting, config version mismatch: got "1", expected "2"
"10.149.51.169": kube-controller-manager: waiting, config version mismatch: got "1", expected "2"
"10.149.51.169": kube-controller-manager: pod is not ready, waiting
"10.149.51.169": kube-controller-manager: pod is not ready, waiting
"10.149.51.169": kube-controller-manager: pod is not ready, waiting
< "10.149.51.169": successfully updated
"10.149.51.170": starting update
update kube-controller-manager: v1.34.1 -> 1.35.0
"10.149.51.170": machine configuration patched
"10.149.51.170": waiting for kube-controller-manager pod update
"10.149.51.170": kube-controller-manager: waiting, config version mismatch: got "1", expected "2"
"10.149.51.170": kube-controller-manager: waiting, config version mismatch: got "1", expected "2"
"10.149.51.170": kube-controller-manager: waiting, config version mismatch: got "1", expected "2"
"10.149.51.170": kube-controller-manager: pod is not ready, waiting
"10.149.51.170": kube-controller-manager: pod is not ready, waiting
< "10.149.51.170": successfully updated
"10.149.51.173": starting update
update kube-controller-manager: v1.34.1 -> 1.35.0
"10.149.51.173": machine configuration patched
"10.149.51.173": waiting for kube-controller-manager pod update
"10.149.51.173": kube-controller-manager: waiting, config version mismatch: got "1", expected "2"
"10.149.51.173": kube-controller-manager: waiting, config version mismatch: got "1", expected "2"
"10.149.51.173": kube-controller-manager: waiting, config version mismatch: got "1", expected "2"
"10.149.51.173": kube-controller-manager: waiting, config version mismatch: got "1", expected "2"
"10.149.51.173": kube-controller-manager: pod is not ready, waiting
"10.149.51.173": kube-controller-manager: pod is not ready, waiting
< "10.149.51.173": successfully updated
updating "kube-scheduler" to version "1.35.0"
"10.149.51.169": starting update
update kube-scheduler: v1.34.1 -> 1.35.0
"10.149.51.169": machine configuration patched
"10.149.51.169": waiting for kube-scheduler pod update
"10.149.51.169": kube-scheduler: waiting, config version mismatch: got "1", expected "2"
"10.149.51.169": kube-scheduler: waiting, config version mismatch: got "1", expected "2"
"10.149.51.169": kube-scheduler: waiting, config version mismatch: got "1", expected "2"
"10.149.51.169": kube-scheduler: waiting, config version mismatch: got "1", expected "2"
"10.149.51.169": kube-scheduler: pod is not ready, waiting
"10.149.51.169": kube-scheduler: pod is not ready, waiting
"10.149.51.169": kube-scheduler: pod is not ready, waiting
< "10.149.51.169": successfully updated
"10.149.51.170": starting update
update kube-scheduler: v1.34.1 -> 1.35.0
"10.149.51.170": machine configuration patched
"10.149.51.170": waiting for kube-scheduler pod update
"10.149.51.170": kube-scheduler: pod is not ready, waiting
"10.149.51.170": kube-scheduler: pod is not ready, waiting
< "10.149.51.170": successfully updated
"10.149.51.173": starting update
update kube-scheduler: v1.34.1 -> 1.35.0
"10.149.51.173": machine configuration patched
"10.149.51.173": waiting for kube-scheduler pod update
"10.149.51.173": kube-scheduler: waiting, config version mismatch: got "1", expected "2"
"10.149.51.173": kube-scheduler: waiting, config version mismatch: got "1", expected "2"
"10.149.51.173": kube-scheduler: waiting, config version mismatch: got "1", expected "2"
"10.149.51.173": kube-scheduler: waiting, config version mismatch: got "1", expected "2"
"10.149.51.173": kube-scheduler: pod is not ready, waiting
"10.149.51.173": kube-scheduler: pod is not ready, waiting
"10.149.51.173": kube-scheduler: pod is not ready, waiting
< "10.149.51.173": successfully updated
updating kube-proxy to version "1.35.0"
"10.149.51.169": starting update
"10.149.51.170": starting update
"10.149.51.173": starting update
updating kubelet to version "1.35.0"
"10.149.51.169": starting update
update kubelet: 1.34.1 -> 1.35.0
"10.149.51.169": machine configuration patched
"10.149.51.169": waiting for kubelet restart

---

[sivasudheerch]

(Talos: Store-4959-CN-dev) root@adcphltadvtes03:~/my-cluster-infra# talosctl -n 10.149.51.169 get mountstatus u-longhorn-data
NODE   NAMESPACE   TYPE   ID   VERSION   SOURCE   TARGET   FILESYSTEM   VOLUME
1 error occurred:
        * rpc error: code = NotFound desc = resource MountStatuses.block.talos.dev(runtime/u-longhorn-data@undefined) doesn't exist


(Talos: Store-4959-CN-dev) root@adcphltadvtes03:~/my-cluster-infra# talosctl -n 10.149.51.170 get mountstatus u-longhorn-data
NODE            NAMESPACE   TYPE          ID                VERSION   SOURCE      TARGET                   FILESYSTEM   VOLUME
10.149.51.170   runtime     MountStatus   u-longhorn-data   2         /dev/vda7   /var/mnt/longhorn-data   xfs          u-longhorn-data
(Talos: Store-4959-CN-dev) root@adcphltadvtes03:~/my-cluster-infra# talosctl -n 10.149.51.173 get mountstatus u-longhorn-data
NODE            NAMESPACE   TYPE          ID                VERSION   SOURCE      TARGET                   FILESYSTEM   VOLUME
10.149.51.173   runtime     MountStatus   u-longhorn-data   2         /dev/vda7   /var/mnt/longhorn-data   xfs          u-longhorn-data
(Talos: Store-4959-CN-dev) root@adcphltadvtes03:~/my-cluster-infra# talosctl -n 10.149.51.169 get VolumeStatus u-longhorn-data -o yaml
node: 10.149.51.169
metadata:
    namespace: runtime
    type: VolumeStatuses.block.talos.dev
    id: u-longhorn-data
    version: 2
    owner: block.VolumeManagerController
    phase: running
    created: 2026-02-16T17:33:44Z
    updated: 2026-02-16T17:33:44Z
    labels:
        talos.dev/user-volume:
    finalizers:
        - block.MountController
spec:
    phase: ready
    type: partition
    location: /dev/vda7
    mountLocation: /dev/vda7
    partitionIndex: 7
    parentLocation: /dev/vda
    uuid: 902b6fa3-24d7-4040-b0a4-7528a9c72d70
    partitionUUID: 6e6ab443-b802-4c45-9974-4b8afba6f544
    size: 29901193216
    prettySize: 30 GB
    filesystem: xfs
    mountSpec:
        targetPath: longhorn-data
        selinuxLabel: system_u:object_r:ephemeral_t:s0
        projectQuotaSupport: false
        parentId: /var/mnt
        fileMode: 493
(Talos: Store-4959-CN-dev) root@adcphltadvtes03:~/my-cluster-infra# talosctl -n 10.149.51.169 get VolumeConfig u-longhorn-data
NODE            NAMESPACE   TYPE           ID                VERSION
10.149.51.169   runtime     VolumeConfig   u-longhorn-data   2
(Talos: Store-4959-CN-dev) root@adcphltadvtes03:~/my-cluster-infra#

[rothgar]

Can you please use github code formatting for the logs with three backticks (```). It's very difficult to follow any of the text without proper formatting.

[sivasudheerch]

Hi @rothgar
I reported a bug for the same issue, requesting you to please check
#12797

[sivasudheerch]

apologies, I have updated the logs format as well now, you should be able to go through the logs @rothgar

[smira]

Thanks for reporting, it will be fixed via #12797 and backported.