k3s 1.35.1 not starting flannel CNI after power cut

[BinaryMan32]

I have a 9 node cluster of turing RK1s running k3s 1.35.1 with HA (3 control plane/servers). I started with 4 nodes about 2 years ago and have gradually grown the cluster to 9 nodes. A few days ago I was adding a 10th node and accidentally cut power to a couple of the existing nodes instead of shutting them down gracefully. When they restarted, the worker rejoined with no issues but the control plane node would not rejoin.

I see many messages in the logs similar to:

Feb 24 20:24:49 griffin7 k3s[1280]: E0224 20:24:49.426843    1280 remote_available_controller.go:462] "Unhandled Error" err="v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.42.3.217:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.42.3.217:10250/apis/metrics.k8s.io/v1beta1\": context deadline exceeded"

These are just symptoms though, not root cause.

On the last SUCCESSFUL k3s startup, I saw messages like this:

Feb 15 15:43:02 griffin7 k3s[3107129]: time="2026-02-15T15:43:02Z" level=info msg="Annotations and labels have been set successfully on node: griffin7"
Feb 15 15:43:02 griffin7 k3s[3107129]: time="2026-02-15T15:43:02Z" level=info msg="Starting flannel with backend vxlan"

Now, these messages do not appear in the logs. I can confirm there are no cni or flannel interfaces:

$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 22:71:0b:2b:2d:bf brd ff:ff:ff:ff:ff:ff

On other working nodes I see:

$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether b2:b1:be:5d:84:4f brd ff:ff:ff:ff:ff:ff
3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default 
    link/ether 2e:75:83:c2:55:98 brd ff:ff:ff:ff:ff:ff
4: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether c2:36:d7:3b:9b:0b brd ff:ff:ff:ff:ff:ff
...

I tried uninstalling and re-installing k3s on this node, but the behavior has not changed.

I don't see anything sensitive in the logs, so I'll attach the log of the failed startup in case something in there is interesting to diagnose the issue.
k3s-failed-startup-logs.txt

The only other brute force approach I can think of is to reflash the operating system. I'll try that in a few days, but I'd like to understand what happened in case there's an easier way to fix this or prevent it happening again.

[BinaryMan32]

Not sure if I should have created an issue instead of a discussion, let me know if that would be more appropriate.

[brandond]

This sounds kind of like #13277 - maybe take a look at the causes people have identified there?

[BinaryMan32]

At first it sounded like a different issue, but one of the comments mentioned longhorn/longhorn#12578 which could be related. I updated to longhorn 1.11.0 on 2026-01-28 though and was able to restart control plane nodes a few times after that. I'll look into this more Friday evening and reply with my findings. Thanks for the tip.

[brandond]

Yeah it's gonna be that.

[BinaryMan32]

And it was, thank you