LoadBalancer Alerting

[maaft]

Description

This morning, suddenly all targets of my load balancer went unhealthy without me noticing.

https://status.hetzner.com/incident/8becfe86-b077-4983-b7d0-af81b4fc1496

Somehow a restart of all k3s nodes helped and all targets where healty again.

Anyway, it took me a while to find out that the load balancer was the problem.

I already opened this issue here hetznercloud/hcloud-cloud-controller-manager#976

If those metrics are added, maybe kube-hetzner module could also add options to generate & deploy monitoring / alerting manifests to the cluster.

[loomsen]

Happened to me as well! Unfortunately, a reboot didn't fix it for me. Still everything down and the LB is not picking up the targets.

[askanhesse]

The non working LBs seem to be fixed when hetzner cloud-controller-manager pod gets restarted. Simply deleting it should also work.

[maaft]

@askanhesse thanks for the info! Deletion would give you a new load-balancer IP though, right? Just keep in mind that this is potentially dangerous if your DNS records point directly to your LB.

@loomsen TBH I only suspected that restarting k3s on all nodes solved the issue. Maybe this is not the full solution. What I actually did:

1. use_control_plane_lb = true
2. terraform apply

This added a 2nd LB for control-plane API, and restarted all nodes. Afterwards the regular LB was working for me.

For this reason I have two servers in front of my LB with haproxy installed + a small script to reconfigure them with a new target IP. This allows me to move to new clusters or new loadbalancers with minimum downtime. Could also be a nice additione to this project @mysticaltech ;)

[loomsen]

@maaft sounds reasonable, after todays outage I was also thinking about something similar. But not much you can do, when hetzner has issues on the load balancer infra itself.
It just started working again some half an hour ago or so without restarting anything on my side.
Fortunately I could get my prod projects back to working with restarting the nodes, however, for another dev project, this didn't have any effect up until it recovered itself just now.
I'm using a control plane lb in all projects, too :)

[askanhesse]

@askanhesse thanks for the info! Deletion would give you a new load-balancer IP though, right? Just keep in mind that this is potentially dangerous if your DNS records point directly to your LB.

No, this is not the case. It will sync configuration to existing LBs. It is quite the same as restarting the whole cluster, resulting in freshly started pod.

[mysticaltech]

Good ideas folks. Personally, I have started using Cloudflare Tunnels with cloudflared (and Cloudflare Zero Trust) instead of LBs, I find it much easier to work with and more reliable, and also free, unless you get mega worldscale traffic.

[raakraka]

As an alternative you can also point Cloudflare Loadbalancer directly to your worker's public ip's with a custom ingress controller (to disable deploying a hetzner lb).
This feels like a good way to have HA