Additional service on loadbalancer with kube.tf

[leisefuxX]

Description

hey git'sters!

in addition to port 80 and 443 on the loadbalancer, I would like to forward port 22 to my worker nodes (for a git installation) aka "add a service to the loadbalancer". i want this state to be reflected by my kube.tf because when i add it by hand, the service tends to dissappear within minutes

For this I first asked the KH assistant, which gave me a code that is conclusive for me

load_balancer_services = [
  {
    protocol          = "tcp"
    listen_port       = 22
    destination_port  = 30022
    node_pool_selector = {
      type = "all"
    }
  }
]

if I now enter this code in my kube.tf and try to use it, I get the error:

│ Error: Unsupported argument
│
│ on kube.tf line 300, in module "kube-hetzner":
│ 300: load_balancer_services = [
│
│ An argument named "load_balancer_services" is not expected here.

I've fed the error code back to KH Agent (which imho knows more than me about this topic) which responded with

The latest version (e.g. v1.6.4 or newer) of the module on Terraform Registry supports load_balancer_services according to $WEBSITE
. To use this version, add block version = "..." to the module "kube-hetzner":

i am using version 2.17.4 according to my .terraform/modules/modules.json
❯ more modules.json {"Modules":[{"Key":"","Source":"","Dir":"."},{"Key":"kube-hetzner","Source":"registry.terraform.io/kube-hetzner/kube-hetzner/hcloud","Version":"2.17.4","Dir":".terraform/modules/kube-hetzner"},{"Key":"kube-hetzner.agents","Source":"./modules/host","Dir":".terraform/modules/kube-hetzner/modules/host"},{"Key":"kube-hetzner.control_planes","Source":"./modules/host","Dir":".terraform/modules/kube-hetzner/modules/host"}]}

What am I doing wrong, or what is the correct code?

Kube.tf file

locals {
  hcloud_token = "MYSUPERSECUREHETZNERAPITOKEN"
}
module "kube-hetzner" {
  providers = {
    hcloud = hcloud
  }
  hcloud_token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
  source = "kube-hetzner/kube-hetzner/hcloud"
  ssh_public_key = file("/Users/leisefuxx/.ssh/fleckenbühl/flnet-k3s-prod.pub")
  ssh_private_key = file("/Users/leisefuxx/.ssh/fleckenbühl/flnet-k3s-prod")
  network_region = "eu-central" # change to `us-east` if location is ash
  control_plane_nodepools = [
    {
      name        = "control-plane-fsn1",
      server_type = "cx22",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "control-plane-nbg1",
      server_type = "cx22",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "control-plane-hel1",
      server_type = "cx22",
      location    = "hel1",
      labels      = [],
      taints      = [],
      count       = 1
    }
  ]
  agent_nodepools = [
    {
      name        = "agent-small",
      server_type = "cx22",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "agent-large",
      server_type = "cx32",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "storage",
      server_type = "cx32",
      location    = "fsn1",
      labels      = [
        "node.kubernetes.io/server-usage=storage"
      ],
      taints      = [],
      count       = 1
    },
    {
      name        = "egress",
      server_type = "cx22",
      location    = "fsn1",
      labels = [
        "node.kubernetes.io/role=egress"
      ],
      taints = [
        "node.kubernetes.io/role=egress:NoSchedule"
      ],
      floating_ip = true
      count = 1
    },
    {
      name        = "agent-arm-small",
      server_type = "cax11",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 0
    },
    {
      name        = "agent-arm-medium",
      server_type = "cax21",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 0
    },
  ]
  load_balancer_type     = "lb11"
  load_balancer_location = "fsn1"
  base_domain = "fleckenbuehler.net"
  hetzner_ccm_use_helm = true
  system_upgrade_use_drain = true
  /* k3s_registries = <<-EOT
    mirrors:
      hub.my_registry.com:
        endpoint:
          - "hub.my_registry.com"
    configs:
      hub.my_registry.com:
        auth:
          username: username
          password: password
  EOT */
  dns_servers = [
    "1.1.1.1",
    "8.8.8.8",
    "2606:4700:4700::1111",
  ]
  /*   cilium_values = <<EOT
ipam:
  mode: kubernetes
k8s:
  requireIPv4PodCIDR: true
kubeProxyReplacement: true
routingMode: native
ipv4NativeRoutingCIDR: "10.0.0.0/8"
endpointRoutes:
  enabled: true
loadBalancer:
  acceleration: native
bpf:
  masquerade: true
encryption:
  enabled: true
  type: wireguard
MTU: 1450
  EOT */
  /*   cert_manager_values = <<EOT
crds:
  enabled: true
  keep: true
replicaCount: 3
webhook:
  replicaCount: 3
cainjector:
  replicaCount: 3
  EOT */
  /*   csi_driver_smb_values = <<EOT
controller:
  name: csi-smb-controller
  replicas: 1
  runOnMaster: false
  runOnControlPlane: false
  resources:
    csiProvisioner:
      limits:
        memory: 300Mi
      requests:
        cpu: 10m
        memory: 20Mi
    livenessProbe:
      limits:
        memory: 100Mi
      requests:
        cpu: 10m
        memory: 20Mi
    smb:
      limits:
        memory: 200Mi
      requests:
        cpu: 10m
        memory: 20Mi
  EOT */
  /*   longhorn_values = <<EOT
defaultSettings:
  defaultDataPath: /var/longhorn
persistence:
  defaultFsType: ext4
  defaultClassReplicaCount: 3
  defaultClass: true
  EOT */
  /*   traefik_values = <<EOT
deployment:
  replicas: 1
globalArguments: []
service:
  enabled: true
  type: LoadBalancer
  annotations:
    "load-balancer.hetzner.cloud/name": "k3s"
    "load-balancer.hetzner.cloud/use-private-ip": "true"
    "load-balancer.hetzner.cloud/disable-private-ingress": "true"
    "load-balancer.hetzner.cloud/location": "nbg1"
    "load-balancer.hetzner.cloud/type": "lb11"
    "load-balancer.hetzner.cloud/uses-proxyprotocol": "true"
ports:
  web:
    redirections:
      entryPoint:
        to: websecure
        scheme: https
        permanent: true
    proxyProtocol:
      trustedIPs:
        - 127.0.0.1/32
        - 10.0.0.0/8
    forwardedHeaders:
      trustedIPs:
        - 127.0.0.1/32
        - 10.0.0.0/8
  websecure:
    proxyProtocol:
      trustedIPs:
        - 127.0.0.1/32
        - 10.0.0.0/8
    forwardedHeaders:
      trustedIPs:
        - 127.0.0.1/32
        - 10.0.0.0/8
  EOT */
  /*   nginx_values = <<EOT
controller:
  watchIngressWithoutClass: "true"
  kind: "DaemonSet"
  config:
    "use-forwarded-headers": "true"
    "compute-full-forwarded-for": "true"
    "use-proxy-protocol": "true"
  service:
    annotations:
      "load-balancer.hetzner.cloud/name": "k3s"
      "load-balancer.hetzner.cloud/use-private-ip": "true"
      "load-balancer.hetzner.cloud/disable-private-ingress": "true"
      "load-balancer.hetzner.cloud/location": "nbg1"
      "load-balancer.hetzner.cloud/type": "lb11"
      "load-balancer.hetzner.cloud/uses-proxyprotocol": "true"
  EOT */
  /*   haproxy_values = <<EOT
  EOT */
  /*   rancher_values = <<EOT
ingress:
  tls:
    source: "rancher"
hostname: "rancher.example.com"
replicas: 1
bootstrapPassword: "supermario"
  EOT */
}
provider "hcloud" {
  token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
}
terraform {
  required_version = ">= 1.5.0"
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = ">= 1.49.1"
    }
  }
}
output "kubeconfig" {
  value     = module.kube-hetzner.kubeconfig
  sensitive = true
}
variable "hcloud_token" {
  sensitive = true
  default   = ""
}

Screenshots

No response

Platform

Linux

[mysticaltech]

Hi @leisefuxX,

This is not a bug - the load_balancer_services variable doesn't exist in this module. The KH assistant gave you incorrect information (Hallucination, we will try to improve that in the future with actively running terraform plan and catching these things before replying).

To add custom ports to your load balancer, you need to use the ingress controller configuration. Since you have several ingress controllers commented out in your config, I'll show you how to do it for each one:

For Traefik (default):

# Add this to your module configuration:
ingress_controller = "traefik"  # This is the default

traefik_additional_ports = [
  {
    name        = "git-ssh"
    port        = 22
    exposedPort = 22
  }
]

# Then configure your traefik_values to handle the port:
traefik_values = <<EOT
ports:
  git-ssh:
    port: 22
    exposedPort: 22
    expose: true
    protocol: TCP
# ... rest of your traefik configuration
EOT

For Nginx:

If you're using nginx, you would need to configure it through the nginx_values:

ingress_controller = "nginx"

nginx_values = <<EOT
tcp:
  22: "default/your-git-service:22"
controller:
  # ... your existing nginx config
EOT

Important Notes:

1. Port 22 will forward directly to your service, not to NodePort 30022
2. You'll need to create a Kubernetes Service that exposes your git installation on the appropriate port
3. The load balancer annotations are automatically added by the module based on your configuration

The module version 2.17.4 you're using is recent and supports these configurations. There's no need to downgrade or look for a non-existent load_balancer_services variable.

Would you like help setting up the complete configuration for your specific ingress controller?

[leisefuxX]

hello @mysticaltech

i am using the defaults (traefik).

i added your snippet (traefik_additional_ports) and added your code to my traefik_values codeblock to my so my kube.tf, planned and applied my kube.tf to my cluster and then traefik won't configure and keeps crashing...

i am using stock and default kube.tf, only added a hostname to my loadbalancer so far...

i want to open port 22 on my loadbalancer and forward it (ideally) to port 2222 on my forgejo (selfhosted git) pod ...

i added a ssh service definition on my forgejo.yaml like so:

service:
  type: ClusterIP
  ports:
    http: 3000
  ssh:
    enabled: true
    type: ClusterIP 
    port: 2222
    externalTrafficPolicy: Cluster 

what am i missing? why does your code crash my traefik?

and please keep the window to respond longer than 8 hours (:

[mysticaltech]

@leisefuxX traefik_values override the rest, when set, it configures the helm values directly. So have a look at your helm values, make sure it's correct.

[leisefuxX]

it`s a stock installation. where can i find the helm values? i only applied the kube.tf to a fresh cluster.

[mysticaltech]

@leisefuxX Something along these lines, and you can also find the helm values of the traefik helm chart somewhere on github.

Here's a recap of all the commands to get the helm values for traefik:

Direct Helm Commands:

# Get user-supplied values only
helm get values traefik -n traefik

# Get ALL values (defaults + user-supplied)
helm get values traefik -n traefik -a

# Save to file
helm get values traefik -n traefik -a > traefik-values.yaml

# List helm releases to find namespace
helm list -A | grep traefik

k3s HelmChart CRD:

# Get the HelmChart resource
kubectl get helmchart traefik -n kube-system -o yaml

# Get just the values content
kubectl get helmchart traefik -n kube-system -o jsonpath='{.spec.valuesContent}'

k3s ConfigMap (where k3s stores helm values):

# Get values from the ConfigMap
kubectl get configmap -n kube-system traefik-traefik-values -o jsonpath='{.data.values\.yaml}'

# Or full ConfigMap
kubectl get configmap -n kube-system traefik-traefik-values -o yaml

# List all helm values ConfigMaps
kubectl get configmap -n kube-system | grep values

Check Deployed Resources:

# See the actual deployed configuration
kubectl get deployment -n traefik traefik -o yaml
kubectl get service -n traefik traefik -o yaml

The helm get values command is the most direct way if you have helm installed and configured with your kubeconfig.