Help! Nodes are going to not ready state!

[bulnv]

Hey guys! I am running a cluster of 1 CPX21 master and 2 CPX21 worker nodes. Unexpectedly I started to get NodeNotReady from workers, one or both at once. I've checked events and can't see there are any issues. A node becomes unready cause kubelet stops responding. I guess this is happening cause CPU, Mem, and IO go too high in hetzner pannel. I can't check whats happening inside node, cause ssh is unavailable but once i managed to do it i think i saw there k3swapd process. Wellp, any ideas appreciated

[mysticaltech]

@bulnv Sorry just saw this now, just for the record so we can have an answer, what was the issue?

[tandemblind]

Hello, any idea about what was the issue?
I'm having constantly some node of the cluster going in NodeNotReady.
I've a cluster with 3 nodes and 2 masters and the cluster is going constantly down, because some node don't reply anymore.
even to kill a pod running on this node is taking a lot of time so I'm obliged to drain and reboot the node via the hetzner console.

Any help will be highly appreciated :)

[tandemblind]

describing the node, I've the following.

Warning InvalidDiskCapacity 93m kubelet invalid capacity 0 on image filesystem
Normal NodeHasNoDiskPressure 93m kubelet Node k3s-master3-gbl status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 93m kubelet Node k3s-master3-gbl status is now: NodeHasSufficientPID
Normal NodeHasSufficientMemory 93m kubelet Node k3s-master3-gbl status is now: NodeHasSufficientMemory
Normal NodeAllocatableEnforced 92m kubelet Updated Node Allocatable limit across pods
Warning InvalidDiskCapacity 85m kubelet invalid capacity 0 on image filesystem
Normal Starting 85m kubelet Starting kubelet.
Normal NodeHasSufficientMemory 85m kubelet Node k3s-master3-gbl status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 85m kubelet Node k3s-master3-gbl status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 85m kubelet Node k3s-master3-gbl status is now: NodeHasSufficientPID
Normal NodeNotReady 85m kubelet Node k3s-master3-gbl status is now: NodeNotReady
Normal Starting 74m kubelet Starting kubelet.
Warning InvalidDiskCapacity 74m kubelet invalid capacity 0 on image filesystem
Normal NodeNotReady 74m kubelet Node k3s-master3-gbl status is now: NodeNotReady
Normal NodeReady 70m kubelet Node k3s-master3-gbl status is now: NodeReady
Normal NodeHasNoDiskPressure 70m (x3 over 74m) kubelet Node k3s-master3-gbl status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 70m (x3 over 74m) kubelet Node k3s-master3-gbl status is now: NodeHasSufficientPID
Normal NodeHasSufficientMemory 70m (x3 over 74m) kubelet Node k3s-master3-gbl status is now: NodeHasSufficientMemory
Normal Starting 17m kubelet Starting kubelet.
Warning InvalidDiskCapacity 17m kubelet invalid capacity 0 on image filesystem
Normal NodeHasSufficientMemory 17m (x2 over 17m) kubelet Node k3s-master3-gbl status is now: NodeHasSufficientMemory
Warning SystemOOM 17m kubelet System OOM encountered, victim process: java, pid: 11084
Normal NodeHasSufficientPID 17m (x2 over 17m) kubelet Node k3s-master3-gbl status is now: NodeHasSufficientPID
Normal NodeReady 17m kubelet Node k3s-master3-gbl status is now: NodeReady
Normal NodeAllocatableEnforced 17m kubelet Updated Node Allocatable limit across pods
Normal NodeHasNoDiskPressure 17m (x2 over 17m) kubelet Node k3s-master3-gbl status is now: NodeHasNoDiskPressure
Normal NodeNotReady 7m2s (x2 over 17m) kubelet Node k3s-master3-gbl status is now: NodeNotReady
Warning SystemOOM 7m2s kubelet System OOM encountered, victim process: java, pid: 11631

can it be some issue with hetzner? I don't have a lot of pods running, the cluster is used at 25-30%

[mysticaltech]

@tandemblind, to resolve the issue you're facing, it's important that you first update to the latest version of the module. You'll find this in the pinned discussions in the discussion section.

Once you've done that, recreate your nodes by adding nodepools with the same count as you had previously. Next, you should drain old nodes from your old nodepools. This will force all your workloads/pods to move to your new nodepools.

Then, reduce the count of your old nodepools to 0. Remember not to delete them, just reduce the count.

By following these steps, you will be using the latest version of both the module and the nodes, and the bug you've been encountering should no longer be present. From my recollection, this bug was related to the kured helmchart definition having the wrong setting in the previous versions.

[tandemblind]

Hello @mysticaltech ,
thanks a lot for your kind answer,

I've deleted the project and started again from scratch (from an empty folder),
and I've configured the kube.tf to have :
-3 master CPX11
-2 slaves CP21

• ClipperEnabled(I haven't changed anything else from the kube.tf)

Once I've deployed again all the deployments( lens metrics, 2 springboot app, postgresql, openfire) I'm again in the same situation as before -> randomly some node is down.

Sometimes I cannot even use kubectl because I receive timeouts.

Do you have some idea what else can be the issue?

[mysticaltech]

@tandemblind Please make sure to run terraform init -upgrade.. Also maybe you need more powerful workers.