LB always reporting failures or unknown health status

[BlueRaven1975]

Hi guys, I recently discovered this project and I'm playing with it since a couple of days.
First of all, let me give you all my greatest kudos, you've really made a great work with this thing and you're going to save the life of many people trying to figure out how to properly build a K8s cluster on Hetzner.
Unfortunately, Hetzner itself doesn't seem to be as cool, I made at least 30-40 runs and everytime a different hiccup/issue pops up: I already hit the nefarious CSI image pull error on a single node (I'm trying to workaround it by setting a proxy with our corporate JFrog, but for the moment I can't make it work), but I also have an issue with the LB that is not mentioned anywhere in previous discussions... or, at least, I couldn't find anything on it.
Long story short, the Hetzner LB always reports unhealthy or unknown status, this is what I see for my last attempt:

and this is what I saw in several previous attempts (where I was experimenting with the control plane LB):

I don't know if it could be normal, but honestly I don't expect to see such a thing... if you put in place an LB that expects to find 2 active services behind it, I take for granted that there should be something on the nodes providing them.

Can you please help me understand what's going on here?

Thanks in advance, keep up this wonderful work you're doing!

[aleksasiriski]

Please send the full kube.tf file with sensitive info removed

[BlueRaven1975]

Hey, thanks for this extremely quick reply!
Here is my kube.tf (it's a really barebones one):

locals {
  hcloud_token = "MY-TOKEN"
}

module "kube-hetzner" {

  providers = {
    hcloud = hcloud
  }

  hcloud_token    = local.hcloud_token
  source          = "kube-hetzner/kube-hetzner/hcloud"
  ssh_public_key  = file("~/.ssh/hetzner-k8s-playground.key.pub")
  ssh_private_key = file("~/.ssh/hetzner-k8s-playground.key")
  network_region  = "eu-central"

  control_plane_nodepools = [
    {
      name        = "control-plane",
      server_type = "cpx11",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 3
    }
  ]

  agent_nodepools = [
    {
      name        = "worker",
      server_type = "cpx11",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 3
    }
  ]

  load_balancer_type     = "lb11"
  load_balancer_location = "fsn1"

  autoscaler_nodepools = [
    {
      name        = "autoscaler"
      server_type = "cpx11"
      location    = "fsn1"
      min_nodes   = 0
      max_nodes   = 6
    }
  ]

  cluster_name         = "k8s-playground"
  create_kubeconfig    = false
  create_kustomization = false

}

provider "hcloud" {
  token = local.hcloud_token
}

terraform {
  required_version = ">= 1.3.3"
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = ">= 1.35.2"
    }
  }
}

output "kubeconfig" {
  value     = module.kube-hetzner.kubeconfig
  sensitive = true
}

[aleksasiriski]

Hey, thanks for this extremely quick reply! Here is my kube.tf (it's a really barebones one):

locals {
  hcloud_token = "MY-TOKEN"
}

module "kube-hetzner" {

  providers = {
    hcloud = hcloud
  }

  hcloud_token    = local.hcloud_token
  source          = "kube-hetzner/kube-hetzner/hcloud"
  ssh_public_key  = file("~/.ssh/hetzner-k8s-playground.key.pub")
  ssh_private_key = file("~/.ssh/hetzner-k8s-playground.key")
  network_region  = "eu-central"

  control_plane_nodepools = [
    {
      name        = "control-plane",
      server_type = "cpx11",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 3
    }
  ]

  agent_nodepools = [
    {
      name        = "worker",
      server_type = "cpx11",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 3
    }
  ]

  load_balancer_type     = "lb11"
  load_balancer_location = "fsn1"

  autoscaler_nodepools = [
    {
      name        = "autoscaler"
      server_type = "cpx11"
      location    = "fsn1"
      min_nodes   = 0
      max_nodes   = 6
    }
  ]

  cluster_name         = "k8s-playground"
  create_kubeconfig    = false
  create_kustomization = false

}

provider "hcloud" {
  token = local.hcloud_token
}

terraform {
  required_version = ">= 1.3.3"
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = ">= 1.35.2"
    }
  }
}

output "kubeconfig" {
  value     = module.kube-hetzner.kubeconfig
  sensitive = true
}

Did you make sure you're on the latest ver? We had a couple releases to fix bugs in the past 2 days.

terraform init --upgrade

[BlueRaven1975]

Yeah, absolutely, I'm always fetching the latest code changes before trying every run and I'm also always starting from scratch, destroying and recreating both the project on Hetzner - just to be sure nothing stays around - and my local Terraform state/cache, so that every module is downloaded from the upstream again every time I launch the command above.
I also tried to compare my kube.tf with the latest one to spot if some parameter/config flag was changed in a significant way, but it looks like it's not the case.

[mysticaltech]

@BlueRaven1975, It's completely normal. We are not defining an active URL in the settings for it to do active health checks, and even if it tries to hit the IPs with GET request for instance, the ingress controller will not respond unless it has an ingress definition configured to respond to all routes, and targeting a service.

IMHO the hcloud cli is everything you need, the UI is really not needed, and it will be easier to ignore this LB state! :)

As for the Control Plane LB, it is not connected to an ingress controller, but directly to the Kubernetes API, that actually responds to HTTP requests by default, that's why it's all green.

See, more explanations below (source: chatgpt):

Hope that clarifies it!

[BlueRaven1975]

Hey, sorry for the late response.
Yes, thanks, now it's clear... I still find it confusing to set health checks for a service you already know is not going to work by default, but I guess it's just me! :-)
How can I mark your answer as the accepted one? it seems I have no way to do that...

[mysticaltech]

Health Check for Ingress Controller

@BlueRaven1975 Cool! Yeah, it can be a bit confusing. But here's what you can do to make sure your LB shows up as healthy!

If a hostname is provided, we can perform a health check. The user just needs to take care of the DNS. We already have the option of providing a hostname for the ingress controller via lb_hostname.

To add a health check, we need to set up a small deployment with a single pod that will respond to health checks. The method you choose to set up the health check will depend on the ingress controller you're using (either Traefik or Nginx). You can either:

1. Set custom Helm values at the bottom of the kube.tf file for the ingress controller, using the required health check annotations. Check the locals.tf file for the current default.

2. Edit your service directly by running kubectl get svc -A to grab the service name, then running kubectl edit service-name -n namespace, and adding the appropriate health check endpoint annotations. This is a faster option.

You can find all the supported load balancer (LB) annotations in the following link: https://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/main/internal/annotation/load_balancer.go

Finally, you will need to set up a small deployment (such as an Nginx "Hello World"), a service to make it available, and an ingress definition to expose it on the lb_hostname.

If everything is set up correctly, you should see everything turn green! If it works for you and you want to contribute, don't hesitate to open a pull request (PR). If lb_hostname is present, we should be able to perform a valid health check.