Use existing floating_ip assigned to server not working for ingress

[sebastianlutter]

I have an existing floating IP (DNS points to this IP) and try to use it with a kube-hetzner cluster. I start the cluster with the config below, then assign the floating_ip to the worker node (using hcloud or UI) and then create an Ingress to a nginx dummy pod running on the worker with some domain foo.bar.org. Lets say the node has IP A and there is the floating IP.

• I can ssh to the server using the floating IP (so the eth0 seems to be properly configured)
• With IP A I can reach the nginx with 404 (on port 80 and 443)
• With domain pointing to IP A I get the nginx starting side (so ingress properly routes to the pods service)
• When using the floating IP the nginx does not answer at all (80 and 443)

The ServiceLB (klipper) only lists the node IPs as external IPs, the floating IP is not there. I guess this is why the ingress-nginx cannot be reached, it is bound to the nodes first IP, not the second floating IP I assigned. I found no way to tell klipper to add my floating_ip to the list. Documentation is very limited, and I guess it all depends on k3s options. With metalLB I could configure the single floating IP as available IP pool and it should work, how to do this with ServiceLB?

I found the floating_ip attribute of agent_nodes, and read the discussion about it: #611
As far as I understood giving an agent_node a floating_ip = true creates a hetzner floating IP and assigns it to the node. There is currently no way to specify to use an existing floating IP, right? But this is what I would need (Define to use an existing floating_ip, assign it to the node, configure eth0, inform ServiceLB to add it as external IP).

Goal is to have a floating IP I keep, and be able to use the IP with Ingress rules. We often start and stop test cluster, and have domains pointing to the floating_ip to make them quickly available to customers (without DNS cache/sync problems because of changing IPs every restart.

How would you do deal with this in general?

Thanks,
Sebastian

My kube.tf

locals {
  hcloud_token = "xxxxxxxxxxx"
}

module "kube-hetzner" {
  providers = {
    hcloud = hcloud
  }
  hcloud_token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
  source = "kube-hetzner/kube-hetzner/hcloud"
  ssh_public_key = file("foobar-k8s.pub")
  ssh_private_key = file("foobar-k8s")
  network_region = "eu-central"

  control_plane_nodepools = [
    {
      name        = "control-plane",
      server_type = "cpx11"
      location    = "nbg1",
      labels      = [  "node-type=control-node" ],
      count       = 3,
    }
  ]
  agent_nodepools = [
    {
      name        = "app-pool",
      server_type =  "cpx11",
      location    = "nbg1",
      labels      = [ "node-type=app" ],
      count       = 1,
    }
 ]

  load_balancer_type     = "lb11"
  load_balancer_location = "nbg1"
  ingress_controller = "nginx"
  enable_klipper_metal_lb = "true"

  preinstall_exec = [
    # make eth0 interface listen on floating IP requests on all nodes
   "ip addr add ${var.floating_ip} dev eth0"
  ]
  enable_cert_manager = true

  # IP Addresses to use for the DNS Servers, set to an empty list to use the ones provided by Hetzner, defaults to ["1.1.1.1", "8.8.8.8", "9.9.9.9"].
  # The number of different DNS servers is limited to 3 by Kubernetes itself.
  dns_servers = ["185.12.64.1", "185.12.64.2", "8.8.8.8"]


provider "hcloud" {
  token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
}

terraform {
  required_version = ">= 1.3.3"
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = ">= 1.39.0"
    }
  }
}

output "kubeconfig" {
  value     = module.kube-hetzner.kubeconfig
  sensitive = true
}

variable "hcloud_token" {
  description = "Hetzner API tokey"
  sensitive   = true
  type        = string
  default   = ""
}

[mysticaltech]

@sebastianlutter I wonder why you would need to start and stop a cluster. And by that you mean recreating correct? Normally that shouldn't be needed. Instead of a floating IP you may want to put a manual Hetzner LB in front of your cluster(s) LBs, so that they serve traffic to the cluster that is not down if that's what you need.

Also have a look at ExternalDNS, if your cluster is really just stopped, then that is the way to go, it will update the DNS immediately if the ingress controller deployed LB IP changes.

Just ideas, I do not know what else you could do. @kube-hetzner/core Any more ideas?

[mysticaltech]

@sebastianlutter More info on servicelb and why you cannot really configure it. k3s-io/k3s#3032 (comment)

[sebastianlutter]

@mysticaltech Ok, ServiceLB simply is not made for this. Thanks!

To fully recreate the cluster is part of the project I do, and that is not common (I know). If I could assign the floating IP to a LoadBalancer everything would be easy, but that is not possible with hetzner cloud yet.

I'll test to set a HetznerLB in front of my cluster.

Another way would be if the floating_ip feature of the agent nodes introduced here would allow to assign an already existing floating IP (e.g. by adding some parameter floating_ip_existing_id, and then only assign the floating IP without creating a new one).

As far as I can understand the code the current agent node floating_ip works like this:

1. In agents.tf#L145-L151 the floating IP is provisioned (using this hetzner API)
2. The part agents.tf#L153-L162 then fetches the information from the created floating_ip resource and does the assignment to the cloud server (using this hetzner API)
3. Then the agent node is configured (eth0 and route) in agents.tf#L164-L198
4. The for_each = { for k, v in local.agent_nodes : k => v if coalesce(lookup(v, "floating_ip"), false) } line at the top of the null-resource makes sure this block is only executed for agent nodes that have floating_ip=true set.

Then with the following changes the use of an existing floating_ip should be possible:

1. Only provision floating_ips when floating_ip_existing_id is set in agent node (in this each statement)
2. If both variables floating_ip_existing_id and floating_ip=true are set then the floating_ip should be imported (which is supported by states the hetzner documentation). It is unclear to me if this can be done by configuration or if the terraform importstatement needs to be used. Also my terraform foo is not powerful enough to know how this imported resource can then be used in the assignment part.
3. Also the floating_ip should then be locked/prevent that it is destroyed

But does this sounds sane? Possible?

[mysticaltech]

@sebastianlutter It does sound sane. Currently if floating_ip=true we create it. But if we had floating_ip=true and floating_ip_id="1234567" we would not create it and it would be managed by the user outside of terraform.

Basically, if floating_ip_id == null then the old "circuit" would be used, if not a simpler newer circuit would be used. Please don't heistate to create a PR for this.

[mashkovd]

@sebastianlutter
I have the same problem.
DNS connected with existing floating IPs. I often recreate my cluster and every time I need to connect to control-plane and type command sudo ip addr add {my floating IPs} dev eth0

params floating_ip=true create new floating_ip. It's bad options, because you need also connect it to DNS name

My solution - I delete all agent and save control-plane (so I pay only for control-plane control-plane)

[sebastianlutter]

Did not find the free time to work on this atm, hope this will change in the next weeks.

My current workaround is like this:

In kube.tf I set the IP on the node with this:

  preinstall_exec = [
    # Reconfigure eth0:
    #  - add floating_ip as first and other IP as second address
    #  - add 172.31.1.1 as default gateway (In the Hetzner Cloud, the
    #    special private IP address 172.31.1.1 is the default
    #    gateway for the public network)
    # The configuration is stored in file /etc/NetworkManager/system-connections/cloud-init-eth0.nmconnection
    # see https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/issues/869
    <<-EOT
    [[ "$(hostname)" == "YOUR_SERVERS_HOSTNAME"* ]] && nmcli connection modify "$(nmcli -g GENERAL.CONNECTION device show eth0)" \
      ipv4.method manual \
      ipv4.addresses YOUR_FLOATING_IP_HERE/32,$(ip addr show eth0 | grep "inet\b" | awk '{print $2}' | cut -d/ -f1)/32 gw4 172.31.1.1 \
      ipv4.route-metric 100 \
    && nmcli connection up "$(nmcli -g GENERAL.CONNECTION device show eth0)"
    EOT
 ]

But the problem is that this will stuck the installation if the floating ip is not assigned to the node (which is not the case since terraform does not assign it). To workaround this I wrote this script:

#!/bin/bash
export APP_NODE_NAME=your_node_name

export FLOATING_IP_NAME=your_floating_ip_name

echo "Wait until the app server is available to assign the floating IP then"
until [ "$(hcloud server list -o noheader -o columns=name | grep $APP_NODE_NAME )" != "" ]; do
  sleep 3
  echo "### WAIT FOR APP SERVER: floating IP not assigned yet ###"
done
# now assign the floating IP to the app node in the cluster
APP_NODE_NAME=$(hcloud server list -o columns=name|grep $APP_NODE_NAME)
FLOATING_IP_ID=$(hcloud floating-ip list -o columns=id -o noheader)
echo "Assign floating IP ${FLOATING_IP_NAME} (${FLOATING_IP_IPv4}) to server node ${APP_NODE_NAME} now . . ."
hcloud floating-ip assign ${FLOATING_IP_ID} ${APP_NODE_NAME}
echo "Floating IP has been assigned"
hcloud floating-ip list -o noheader

Before calling terraform apply . . . I start this script as background process in my shell:

./assign_floating_ip.sh &

It polls in background until the node has been started by terraform and assigns the IP then. Its just a dirty hack, but maybe it helps somebody