Cannot access node via ssh

[v-petukhov]

Hi everyone!

Two of my cluster nodes gone to bad state:

MemoryPressure       Unknown   Tue, 02 Apr 2024 10:32:12 +0300   Tue, 02 Apr 2024 10:35:44 +0300   NodeStatusUnknown   Kubelet stopped posting node status.
 DiskPressure         Unknown   Tue, 02 Apr 2024 10:32:12 +0300   Tue, 02 Apr 2024 10:35:44 +0300   NodeStatusUnknown   Kubelet stopped posting node status.
 PIDPressure          Unknown   Tue, 02 Apr 2024 10:32:12 +0300   Tue, 02 Apr 2024 10:35:44 +0300   NodeStatusUnknown   Kubelet stopped posting node status.
 Ready                Unknown   Tue, 02 Apr 2024 10:32:12 +0300   Tue, 02 Apr 2024 10:35:44 +0300   NodeStatusUnknown   Kubelet stopped posting node status.

So, my decision was to access them via SSH and restart kubelet service, but, unfortunately, nodes are in such bad condition, that I'm not able to use ssh:

ssh [email protected] -i ~/.ssh/key -o StrictHostKeyChecking=no                                                                                                                                         
kex_exchange_identification: read: Connection reset by peer
Connection reset by xxx.xxx.xxx.xxx port 22

All other nodes are accessible via ssh.

So, my thoughts were about to reboot nodes via htz console. But, I don't know how to do it in a right way. I need to cordon node, than drain node and than reboot via htz console? Or how?

Thx for your answers!

[cobrani]

Hi @v-petukhov. The problem sounds a bit like the issue I was facing in #1287 . From one moment to the other, I was not able to connect to the nodes via SSH anymore and kubelet was not reachable. In my case, it was somehow related to the system upgrades. I didn't have the time yet to investigate the issues, however I solved the problem temporarily by

• Manually restart the node via Hetzner console, choosing Start from snapshot in the bootloader, selecting an older snapshot version (cause in my case, I missed to set up backups)
• Rollback via sudo snapper rollback
• Turn off system updates via systemctl --now disable transactional-update.timer

Of course this doesn't fix the problem, but at least I was able to get the cluster back to a healthy state.

[v-petukhov]

Hi, @cobrani!

I've cured my cluster without rollback to older snapshots, but i'm also had thought about auto-updates.

In my case helped such steps: drain node, restart node, uncordon node.