Issues scaling servers: network is already full

[Niek]

I have an autoscaler group that I can't seem to grow above 94 instances. The error I see in the cluster-autoscaler-x logs is:

E0417 10:47:10.721880       1 hetzner_node_group.go:117] failed to create error: could not create server type cpx21 in region ash: network 1234567 is already full (invalid_input)

If I check in the Hetzner UI I see only 95 resources/routes (94 + control plane), that does't seem too much to me? Any idea what to do?

[ifeulner]

Contact the Hetzner support, maybe they are reserving additional IPs for the node internally... Seems your cluster is quite big.

Let us know the outcome, please!

[mysticaltech]

@Niek Definitely vertical scaling is the solution here, you can have much much bigger nodes, especially now that we support ARM instances, you can save a lot of 💵.

[mysticaltech]

I will update the docs ASAP too.

[ifeulner]

That's really a pity - I always thought this is per subnet, but obviously it's per network. And that's quite a limitation, at least compared to the big hyperscalers.

[ifeulner]

Yeah, for now multiple clusters seems like the easiest solution. I run rather small cpx21 nodes, I could also go with much larger nodes (like cpx51) and run more pods per node on it.
I think this section requires some updating though, as it's much more limited than thought: https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner#scaling-nodes

In general this is not a good practice, as it raises extremely the internode-communication. So it's better to find a sweet spot between node size and number of nodes depending on your requirements. It could also make sense to increase the number of pods per node (200 by default I think...) .

[mysticaltech]

Good point @ifeulner, however 163-32GB nodes seems obviously ok to use. @Niek If I were you, I would shoot for the CAX41 instances, 16 vCPU / 32GB @ 29 euros!

[mysticaltech]

@apricot FYI above, some of our users need more than 100 nodes and it seems that private networks do not support that. It seems like a really low limit. Any recommendations on getting around that? And are you guys planning on scaling that number up?