Unable to deploy

[maxtomczyk]

Hello,

I'm trying to deploy very basic cluster to one of the german region and it looks like it's impossible task.

I'm using Gitlab CI template to run terraform. When deploying to Falksnstein region there is issue with very slow MicroOS download speed (it's downloading for 20 minutes) and whole deployment fails. When deploying to Nuremberg one node is not responding and the error is timeout - last error: dial tcp xxx.xxx.xxx.xxx:xxxx i/o timeout. When I ssh into IP of node that failed I can see that it's still in rescue mode and Micro OS is not installed.

My configuration is (it's minimal as I started to remove many parts after I encountered problems):

locals {
  hcloud_token = "${HETZNER_TOKEN}"
}

module "kube-hetzner" {
  providers = {
    hcloud = hcloud
  }
  hcloud_token = local.hcloud_token

  source = "kube-hetzner/kube-hetzner/hcloud"

  ssh_port = 22
  ssh_public_key = file("~/.ssh/key_name.pub")
  ssh_private_key = file("~/.ssh/key_name")
  network_region = "eu-central"

  control_plane_nodepools = [
    {
      name        = "control-plane-s",
      server_type = "cx21",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 3
    }
  ]

  agent_nodepools = [
    {
      name        = "agent-s",
      server_type = "cx21",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 3
    }
  ]
  load_balancer_type     = "lb11"
  load_balancer_location = "nbg1"

  enable_nginx = true
  enable_traefik = false

  initial_k3s_channel = "v1.25"
  cluster_name = "nsc-1"

  create_kubeconfig = false
  create_kustomization = false
}

provider "hcloud" {
  token = local.hcloud_token
}

terraform {
  required_version = ">= 1.3.3"
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = ">= 1.35.2"
    }
  }
}

output "kubeconfig" {
  value     = module.kube-hetzner.kubeconfig
  sensitive = true
}

Is there anything wrong with this kube.tf?

[maxtomczyk]

Interesting thing is that after multiple runs I noticed that the node that stucks in rescue mode has the same IP address in all runs (I delete whole project on Hetzner after each failure)

[maxtomczyk]

I just noticed that gitlab ci pipeline runner cannot even Ping affected IPs (my local machine can ping it), so it looks like either gitlab CI blocks certain IP addresses or hetzner drops connection from gitlab for this addresses.

Is there any way to investigate it further?

[captnCC]

I experienced some similar problems with gitlab.com, at least for the image registry. One node was blocked from the registry completely (but with a 403, so there is a block on application level, not network).
It's common for hetzner to reassign the same IP if a node is just recreated. The only workaround I can think of, is to create a new node and then delete the old one.

As a quick fix, you can use a custom runner instead of the shared ones, to avoid the IP blocks.
But the general problem of hetzner and gitlab is outside of this projects scope.

[maxtomczyk]

Creating new node with problematic IP assigned to other node was my first idea. Unfortunately I git another IP adres Thassos not working. I will try with own runner.

[mysticaltech]

There is another solution folks, shared by @aleksasiriski. Just create static IPs a good bunch of them, they also reuse the previous node IPs. So that when you create a node, it uses a brand new IP, then you delete the static IPs. It's a neat and cheap trick to pull off.