Dangerous behavior when removing a worker nodegroup.

[mlinares1998]

Hi! @M4t7e 👋
Hope everything is going well! 😄
Today I encountered a potentially dangerous plan proposal when attempting to remove one of my node groups that was positioned earlier in the worker_nodepools array (not the last one).

After investigating the module, I discovered that subnet allocation IP ranges are directly tied to the nodepool array indices. When the array shrinks by removing a nodegroup from the middle, the module proposes to shift the remaining nodepool ranges and node IPs to "fill the gaps."

Problematic code reference:
https://github.com/hcloud-k8s/terraform-hcloud-kubernetes/blob/main/network.tf#L103
There may be additional references throughout the codebase that exhibit similar behavior.

This creates a critical issue, since it will try to delete healthy nodegroup subnets and recreate the nodes in the removed nodegroup's subnet.
The potential chaos this could create is significant 😅

Example scenario:

worker_nodepools = [
    {
      name     = "worker-platform-egress-nbg1"
      type     = "cax21"
      location = "nbg1"
      count    = 1
      labels = {}
      annotations = {}
      taints      = []
    },
    # {
    #   name     = "worker-platform-egress-fsn1"
    #   type     = "cax21"
    #   location = "fsn1"
    #   count    = 1
    #   labels = {}
    #   annotations = {}
    #   taints      = []
    # },
    {
      name     = "worker-platform-fsn1"
      type     = "cax21"
      location = "fsn1"
      count    = 1
      labels = {}
      annotations = {}
      taints      = []
    },
  ]`

TF Plan:

14:44:22.028 STDOUT tofu:   # module.kubernetes.hcloud_network_subnet.worker["worker-platform-egress-fsn1"] will be destroyed
14:44:22.028 STDOUT tofu:   # (because key ["worker-platform-egress-fsn1"] is not in for_each map)
14:44:22.028 STDOUT tofu:   - resource "hcloud_network_subnet" "worker" {
14:44:22.028 STDOUT tofu:       - gateway      = "10.10.0.1" -> null
14:44:22.028 STDOUT tofu:       - id           = "REDACTED-10.10.65.128/25" -> null
14:44:22.028 STDOUT tofu:       - ip_range     = "10.10.65.128/25" -> null
14:44:22.028 STDOUT tofu:       - network_id   = REDACTED -> null
14:44:22.028 STDOUT tofu:       - network_zone = "eu-central" -> null
14:44:22.028 STDOUT tofu:       - type         = "cloud" -> null
14:44:22.028 STDOUT tofu:     }
14:44:22.028 STDOUT tofu:   # module.kubernetes.hcloud_network_subnet.worker["worker-platform-fsn1"] must be replaced
14:44:22.028 STDOUT tofu: -/+ resource "hcloud_network_subnet" "worker" {
14:44:22.028 STDOUT tofu:       ~ gateway      = "10.10.0.1" -> (known after apply)
14:44:22.028 STDOUT tofu:       ~ id           = "REDACTED-10.10.66.0/25" -> (known after apply)
14:44:22.028 STDOUT tofu:       ~ ip_range     = "10.10.66.0/25" -> "10.10.65.128/25" # forces replacement
14:44:22.028 STDOUT tofu:         # (3 unchanged attributes hidden)
14:44:22.029 STDOUT tofu:     }
14:44:22.029 STDOUT tofu:   # module.kubernetes.hcloud_placement_group.worker["eb-hcloud-ops-worker-platform-egress-fsn1-pg-1"] will be destroyed
14:44:22.029 STDOUT tofu:   # (because key ["eb-hcloud-ops-worker-platform-egress-fsn1-pg-1"] is not in for_each map)
14:44:22.029 STDOUT tofu:   - resource "hcloud_placement_group" "worker" {
14:44:22.029 STDOUT tofu:       - id      = "1100212" -> null
14:44:22.029 STDOUT tofu:       - labels  = {
14:44:22.029 STDOUT tofu:           - "cluster"  = "REDACTED"
14:44:22.029 STDOUT tofu:           - "nodepool" = "worker-platform-egress-fsn1"
14:44:22.029 STDOUT tofu:           - "role"     = "worker"
14:44:22.029 STDOUT tofu:         } -> null
14:44:22.029 STDOUT tofu:       - name    = "REDACTED-worker-platform-egress-fsn1-pg-1" -> null
14:44:22.029 STDOUT tofu:       - servers = [
14:44:22.029 STDOUT tofu:           - 106345027,
14:44:22.029 STDOUT tofu:         ] -> null
14:44:22.029 STDOUT tofu:       - type    = "spread" -> null
14:44:22.029 STDOUT tofu:     }
14:44:22.029 STDOUT tofu:   # module.kubernetes.hcloud_server.worker["eb-hcloud-ops-worker-platform-egress-fsn1-1"] will be destroyed
14:44:22.029 STDOUT tofu:   # (because key ["eb-hcloud-ops-worker-platform-egress-fsn1-1"] is not in for_each map)
14:44:22.029 STDOUT tofu:   - resource "hcloud_server" "worker" {
14:44:22.029 STDOUT tofu:       - allow_deprecated_images    = false -> null
14:44:22.029 STDOUT tofu:       - backups                    = false -> null
14:44:22.029 STDOUT tofu:       - datacenter                 = "fsn1-dc14" -> null
14:44:22.029 STDOUT tofu:       - delete_protection          = false -> null
14:44:22.029 STDOUT tofu:       - firewall_ids               = [
14:44:22.029 STDOUT tofu:           - 2299076,
14:44:22.029 STDOUT tofu:         ] -> null
14:44:22.029 STDOUT tofu:       - id                         = "REDACTED" -> null
14:44:22.029 STDOUT tofu:       - ignore_remote_firewall_ids = false -> null
14:44:22.029 STDOUT tofu:       - image                      = "REDACTED" -> null
14:44:22.029 STDOUT tofu:       - ipv6_network               = "<nil>" -> null
14:44:22.029 STDOUT tofu:       - keep_disk                  = false -> null
14:44:22.029 STDOUT tofu:       - labels                     = {
14:44:22.029 STDOUT tofu:           - "cluster"                 = "REDACTED"
14:44:22.029 STDOUT tofu:           - "nodepool"                = "worker-platform-egress-fsn1"
14:44:22.029 STDOUT tofu:           - "role"                    = "worker"
14:44:22.029 STDOUT tofu:         } -> null
14:44:22.029 STDOUT tofu:       - location                   = "fsn1" -> null
14:44:22.029 STDOUT tofu:       - name                       = "REDACTED-worker-platform-egress-fsn1-1" -> null
14:44:22.029 STDOUT tofu:       - placement_group_id         = 1100212 -> null
14:44:22.029 STDOUT tofu:       - primary_disk_size          = 80 -> null
14:44:22.029 STDOUT tofu:       - rebuild_protection         = false -> null
14:44:22.029 STDOUT tofu:       - server_type                = "cax21" -> null
14:44:22.029 STDOUT tofu:       - shutdown_before_deletion   = true -> null
14:44:22.029 STDOUT tofu:       - ssh_keys                   = [
14:44:22.029 STDOUT tofu:           - "100479455",
14:44:22.029 STDOUT tofu:         ] -> null
14:44:22.029 STDOUT tofu:       - status                     = "running" -> null
14:44:22.029 STDOUT tofu:       - network {
14:44:22.029 STDOUT tofu:           - alias_ips   = [] -> null
14:44:22.029 STDOUT tofu:           - ip          = "10.10.65.129" -> null
14:44:22.029 STDOUT tofu:           - mac_address = "86:00:00:ac:6d:bd" -> null
14:44:22.029 STDOUT tofu:           - network_id  = 11263756 -> null
14:44:22.029 STDOUT tofu:         }
14:44:22.029 STDOUT tofu:       - public_net {
14:44:22.030 STDOUT tofu:           - ipv4         = 0 -> null
14:44:22.030 STDOUT tofu:           - ipv4_enabled = false -> null
14:44:22.030 STDOUT tofu:           - ipv6         = 0 -> null
14:44:22.030 STDOUT tofu:           - ipv6_enabled = false -> null
14:44:22.030 STDOUT tofu:         }
14:44:22.030 STDOUT tofu:     }
14:44:22.030 STDOUT tofu:   # module.kubernetes.hcloud_server.worker["eb-hcloud-ops-worker-platform-fsn1-1"] will be updated in-place
14:44:22.030 STDOUT tofu:   ~ resource "hcloud_server" "worker" {
14:44:22.030 STDOUT tofu:         id                         = "REDACTED"
14:44:22.030 STDOUT tofu:         name                       = "REDACTED"
14:44:22.030 STDOUT tofu:         # (18 unchanged attributes hidden)
14:44:22.030 STDOUT tofu:       - network {
14:44:22.030 STDOUT tofu:           - alias_ips   = [] -> null
14:44:22.030 STDOUT tofu:           - ip          = "10.10.66.1" -> null
14:44:22.030 STDOUT tofu:           - mac_address = "86:00:00:a9:87:80" -> null
14:44:22.030 STDOUT tofu:           - network_id  = 11263756 -> null
14:44:22.030 STDOUT tofu:         }
14:44:22.030 STDOUT tofu:       + network {
14:44:22.030 STDOUT tofu:           + alias_ips   = []
14:44:22.030 STDOUT tofu:           + ip          = "10.10.65.129"
14:44:22.030 STDOUT tofu:           + mac_address = (known after apply)
14:44:22.030 STDOUT tofu:           + network_id  = REDACTED
14:44:22.030 STDOUT tofu:         }
14:44:22.030 STDOUT tofu:         # (1 unchanged block hidden)
14:44:22.030 STDOUT tofu:     }
14:44:22.030 STDOUT tofu:   # module.kubernetes.talos_machine_configuration_apply.worker["eb-hcloud-ops-worker-platform-egress-fsn1-1"] will be destroyed
14:44:22.030 STDOUT tofu:   # (because key ["eb-hcloud-ops-worker-platform-egress-fsn1-1"] is not in for_each map)
14:44:22.030 STDOUT tofu:   - resource "talos_machine_configuration_apply" "worker" {
14:44:22.030 STDOUT tofu:       - apply_mode                  = "auto" -> null
14:44:22.030 STDOUT tofu:       - client_configuration        = {
14:44:22.030 STDOUT tofu:           - ca_certificate     = "REDACTED" -> null
14:44:22.030 STDOUT tofu:           - client_certificate = "REDACTED" -> null
14:44:22.030 STDOUT tofu:           - client_key         = (sensitive value) -> null
14:44:22.030 STDOUT tofu:         } -> null
14:44:22.030 STDOUT tofu:       - endpoint                    = "10.10.65.129" -> null
14:44:22.030 STDOUT tofu:       - id                          = "machine_configuration_apply" -> null
14:44:22.030 STDOUT tofu:       - machine_configuration       = (sensitive value) -> null
14:44:22.030 STDOUT tofu:       - machine_configuration_input = (sensitive value) -> null
14:44:22.030 STDOUT tofu:       - node                        = "10.10.65.129" -> null
14:44:22.030 STDOUT tofu:       - on_destroy                  = {
14:44:22.030 STDOUT tofu:           - graceful = true -> null
14:44:22.030 STDOUT tofu:           - reboot   = false -> null
14:44:22.030 STDOUT tofu:           - reset    = true -> null
14:44:22.030 STDOUT tofu:         } -> null
14:44:22.030 STDOUT tofu:     }
14:44:22.031 STDOUT tofu:   # module.kubernetes.talos_machine_configuration_apply.worker["eb-hcloud-ops-worker-platform-egress-nbg1-1"] will be updated in-place
14:44:22.031 STDOUT tofu:   ~ resource "talos_machine_configuration_apply" "worker" {
14:44:22.031 STDOUT tofu:         id                          = "machine_configuration_apply"
14:44:22.031 STDOUT tofu:       ~ machine_configuration       = (sensitive value)
14:44:22.031 STDOUT tofu:       ~ machine_configuration_input = (sensitive value)
14:44:22.031 STDOUT tofu:         # (5 unchanged attributes hidden)
14:44:22.031 STDOUT tofu:     }
14:44:22.031 STDOUT tofu:   # module.kubernetes.talos_machine_configuration_apply.worker["eb-hcloud-ops-worker-platform-fsn1-1"] will be updated in-place
14:44:22.031 STDOUT tofu:   ~ resource "talos_machine_configuration_apply" "worker" {
14:44:22.031 STDOUT tofu:       ~ endpoint                    = "10.10.66.1" -> "10.10.65.129"
14:44:22.031 STDOUT tofu:         id                          = "machine_configuration_apply"
14:44:22.031 STDOUT tofu:       ~ machine_configuration       = (sensitive value)
14:44:22.031 STDOUT tofu:       ~ machine_configuration_input = (sensitive value)
14:44:22.031 STDOUT tofu:       ~ node                        = "10.10.66.1" -> "10.10.65.129"
14:44:22.031 STDOUT tofu:         # (3 unchanged attributes hidden)
14:44:22.031 STDOUT tofu:     }
14:44:22.031 STDOUT tofu:   # module.kubernetes.terraform_data.talos_health_data will be updated in-place
14:44:22.031 STDOUT tofu:   ~ resource "terraform_data" "talos_health_data" {
14:44:22.031 STDOUT tofu:         id     = "REDACTED"
14:44:22.031 STDOUT tofu:       ~ input  = {
14:44:22.031 STDOUT tofu:           ~ worker_nodes        = [
14:44:22.031 STDOUT tofu:               - "10.10.65.129",
14:44:22.031 STDOUT tofu:                 "10.10.65.1",
14:44:22.031 STDOUT tofu:               - "10.10.66.1",
14:44:22.031 STDOUT tofu:               + "10.10.65.129",
14:44:22.031 STDOUT tofu:             ]
14:44:22.031 STDOUT tofu:             # (4 unchanged attributes hidden)
14:44:22.031 STDOUT tofu:         }
14:44:22.031 STDOUT tofu:       ~ output = {
14:44:22.031 STDOUT tofu:           - control_plane_nodes = [
14:44:22.031 STDOUT tofu:               - "10.10.64.11",
14:44:22.031 STDOUT tofu:               - "10.10.64.21",
14:44:22.031 STDOUT tofu:               - "10.10.64.1",
14:44:22.031 STDOUT tofu:             ]
14:44:22.031 STDOUT tofu:           - current_ip          = []
14:44:22.031 STDOUT tofu:           - endpoints           = [
14:44:22.031 STDOUT tofu:               - "10.10.64.11",
14:44:22.031 STDOUT tofu:               - "10.10.64.21",
14:44:22.031 STDOUT tofu:               - "10.10.64.1",
14:44:22.031 STDOUT tofu:             ]
14:44:22.031 STDOUT tofu:           - kube_api_url        = "https://REDACTED:6443"
14:44:22.031 STDOUT tofu:           - worker_nodes        = [
14:44:22.031 STDOUT tofu:               - "10.10.65.129",
14:44:22.031 STDOUT tofu:               - "10.10.65.1",
14:44:22.031 STDOUT tofu:               - "10.10.66.1",
14:44:22.031 STDOUT tofu:             ]
14:44:22.031 STDOUT tofu:         } -> (known after apply)
14:44:22.031 STDOUT tofu:     }
14:44:22.031 STDOUT tofu: Plan: 1 to add, 4 to change, 5 to destroy.

• It will remove worker-platform-egress-fsn1 nodegroup which is expected
• As the index shrinks, it wants to assign the deleted nodepool subnet range (10.10.65.128/25) to the healthy worker-platform-fsn1 nodepool (changing from 10.10.66.0/25)
• This forces subnet replacement, IP changes (10.10.66.1 → 10.10.65.129), and Talos machineconfig updates.

@M4t7e what are your thoughts on this?
Which approach do you think would work best while maintaining backward compatibility?

Regards!

[M4t7e]

Hey @mlinares1998, this actually works as expected. Unfortunately there's no straightforward way to dynamically assign subnets to resources without passing an index to cidrsubnet, since the CIDRs have to be calculated from that input. The solution here is to set the nodepool count to 0 instead of removing the entire slot. See also: #150 (comment)

Ideally, we'd need some kind of stateful assignment function for this, but AFAIK Terraform doesn't support that. Alternatively, we could manually assign the index or set the CIDR in the worker nodepool config variable, but both options feel pretty clunky and inelegant to me.

[M4t7e]

Another possible solution would be to use a single subnet for all worker nodepools and let Hetzner assign IPs dynamically. At first glance, this seems to be already supported if you omit the ip field, as described here: https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/server_network

(Optional, string) IP to request to be assigned to this server. If you do not provide this then you will be auto assigned an IP address.

However, note the documentation for subnet_id:

(Optional, string) ID of the sub-network which should be added to the Server. Required if network_id is not set. Note: if the ip property is missing, the Server is currently added to the last created subnet.

This means the server will receive an IP from the last created subnet, not necessarily from the subnet specified via subnet_id. This behavior is quite frustrating, and I've already spent weeks experimenting with different approaches. Unfortunately, the only real fix would need to come from Hetzner's (server-side) API or an implementation in their Terraform provider. If either option allowed auto-assigning IPs within a specified subnet, the solution would be straightforward. Other cloud providers typically support such a functionality.

See also:

• [Feature]: hcloud_server_network: assign IP from supplied subnet hetznercloud/terraform-provider-hcloud#672
• [Feature]: Load balancers should honor selected subnet. hetznercloud/terraform-provider-hcloud#844

Btw, the Autoscaler's IP assignment runs into the same issue. Hopefully this will be fixed soon by kubernetes/autoscaler#8334

[mlinares1998]

Thanks for your quick response as always @M4t7e!
I have reached yesterday to the same cluster-autoscaler issue aswell.
I was trying to figure out why autoscaler nodes were scheduling on one of my nodepools ranges 😆

Didn't knew anything about the current IP assignment behaviors, that certainly explains some things...

After reading the threads you have linked, i completely agree with you that resolving this lies on Hetzner's side rather than us developing dirty hacks.

Shall we keep this open, so we can follow-up progress by their side on this?

Meanwhile i guess the best solution for now is scale to 0 those "dead" nodepools and reuse their slots.

Regards! 🚀

[M4t7e]

I'll close this issue for now, as in my experience, solutions from Hetzner, even for these basic things, can unfortunately take several years if they happen at all. There's not much we can do on our end at the moment, but if anything changes we can always revisit this.

[mlinares1998]

Hi @M4t7e!
Now that Hetzner has shipped the required API changes 🥳, shall we reopen this?

Subnet support is already merged upstream in the cluster-autoscaler repo and will likely land in the next release.

I’m preparing a PR with the corresponding cluster-autoscaler integration so we can address this once the new version is out, it doesn’t affect existing deployments.

On the Terraform side, the required provider PR also looks close to being merged.

Regards!

[M4t7e]

Hey @mlinares1998, I want to finally solve this issue in the upcoming v4 release and it might require a manual migration step.

My current plan is to change the default to use a single subnet for all worker nodes, using the new subnet IP allocation to automatically assign IPs. I'm just waiting for hetznercloud/terraform-provider-hcloud#1278 to be merged. Thank you so much for taking care of that, it's amazing! 🚀

I'm also considering an option to allow custom subnets (CIDR) per worker nodepool, which would create dedicated subnets like we do today. This could also provide a migration path for users coming from v3. If they want to stick with per-nodepool subnets, they can reuse the existing subnet CIDRs. This approach would also solve the ordering issue. I remember that you also had a use case for dedicated per-nodepool subnets, so hopefully this is a good solution for you as well.

For now, I would leave cluster autoscaler out of scope. The subnet IP allocation feature was added as a feature and not as a fix. Therefore I assume K8s/CA 1.35 will be the earliest possible version where it will be released, which means it will probably land around Q3 in this module.

[mlinares1998]

Hi @M4t7e!
Thank you very much for addressing this! 🚀

Hey @mlinares1998, I want to finally solve this issue in the upcoming v4 release and it might require a manual migration step.

My current plan is to change the default to use a single subnet for all worker nodes, using the new subnet IP allocation to automatically assign IPs. I'm just waiting for hetznercloud/terraform-provider-hcloud#1278 to be merged. Thank you so much for taking care of that, it's amazing! 🚀

As you can see, I've been working around this issue for a while 😆

I submitted hetznercloud/terraform-provider-hcloud#1278 while working on a PoC local branch that required exactly those changes to make this approach fully functional.
The current provider's hcloud_server_network resource fails if the server does not have, at least another private/public IP attached, so the fix was necessary (see hetznercloud/terraform-provider-hcloud#1265).

I didn't want to discuss this further with you before the changes were fully merged 😅 , so we could unify the CA/subnet changes in a single release.

I'm also considering an option to allow custom subnets (CIDR) per worker nodepool, which would create dedicated subnets like we do today. This could also provide a migration path for users coming from v3. If they want to stick with per-nodepool subnets, they can reuse the existing subnet CIDRs. This approach would also solve the ordering issue. I remember that you also had a use case for dedicated per-nodepool subnets, so hopefully this is a good solution for you as well.

This is exactly the approach I took in my PoC local branch! My idea was to move the worker subnet ahead of the current non-working Autoscaler subnet (which is currently at the end of the CIDR range), and split the free ranges between control planes/LBs and the unified worker subnet into user-defined CIDR slots which could be used for fixed node pools or autoscaler pools altogether.

Here's the allocation schema I wanted to propose you initially:

When skip_first_subnet = false:

• ID 0: Control Plane
• ID 1: Load Balancer
• IDs 2-46: Manual assignment pool (45 IDs, indices 0-45)
• ID 47: Worker shared subnet
• ID 48: Autoscaler shared subnet

When skip_first_subnet = true (all IDs shift +1):

• ID 0: Skipped
• ID 1: Control Plane
• ID 2: Load Balancer
• IDs 3-46: Manual assignment pool (44 IDs, indices 0-44)
• ID 47: Worker shared subnet
• ID 48: Autoscaler shared subnet

For v3 users who want to keep their existing CIDRs, they would need to migrate their current nodegroups to Index 0, then Index 1, and so on. If no index is specified, then the nodes would be assigned to the unified worker CIDR.

However, now that you mention it, maybe allowing users to fully define the CIDR is the right approach here, rather than using automated pre-calculated slots. This would give users more flexibility while still solving the core issue.

For now, I would leave cluster autoscaler out of scope. The subnet IP allocation feature was added as a feature and not as a fix. Therefore I assume K8s/CA 1.35 will be the earliest possible version where it will be released, which means it will probably land around Q3 in this module.

#226 addresses this. We could revisit the PR and migrate it to this new approach in advance. Based on my current tests, it doesn't break current CA since the subnet field isn't used yet.

So, if anyone wants to manually set v1.35's CA Helm Chart to have autoscaler nodepools in the correct subnet, they could do so before going GA.
I think we could also test it once it's released and verify whether it works fine with K8s v1.34, as the API doesn't seem to have breaking changes that would make it incompatible with older K8s versions, especially for this critical fix.

Maybe for this time, we could make an exception here if everything seems to work correctly.
It's unfortunate the fix wasn't backported to the v1.33-v1.34 series. I was expecting that when I made this PR initially.

Let me know if you'd like to discuss this further or if you need any help implementing these changes!

Regards and happy new year!