How to enable autoscaler?

[dstockhammer]

I provisioned a cluster with a static node pool plus an autoscaled node pool. The autoscaled pool should scale to 0 if the static nodes are sufficient. So far so good; this works as expected (or so I thought).

  autoscaler_nodepools = [
    {
      name        = "worker-asg-arm-fsn1"
      server_type = "cax41",
      location    = "fsn1",
      min_nodes   = 0
      max_nodes   = 6

      labels = [
        "type=dynamic"
      ],
    }
  ]

Now let's assume the number of nodes in the static pool is n. If I set the autoscaled pool to min_nodes = 1, I would expect there to be n + 1 worker nodes in total. However, when I do that, the autoscaler doesn't provision any new nodes. I also tried setting it to a higher min number, just in case it counts all the worker nodes.

I can see that the change is correctly applied to the cluster-autoscaler pod: --nodes=1:6:cax41:fsn1:worker-asg-arm-fsn1. But it still thinks that no scaling is required:

node_instances_cache.go:156] Start refreshing cloud provider node instances cache                                                                                               │
node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 16.881µs                                                                        │
hetzner_servers_cache.go:94] Fetching servers from Hetzner API                                                                                                                  │
hetzner_node_group.go:438] Set node group worker-asg-arm-fsn1 size from 0 to 0, expected delta 0                                                                                │
hetzner_node_group.go:438] Set node group draining-node-pool size from 0 to 0, expected delta 0                                                                                 │
static_autoscaler.go:511] No unschedulable pods                                                                                                                                 │
legacy.go:298] No candidates for scale down                                                                                                                                     │
hetzner_node_group.go:438] Set node group draining-node-pool size from 0 to 0, expected delta 0                                                                                 │
hetzner_node_group.go:438] Set node group worker-asg-arm-fsn1 size from 0 to 0, expected delta 0                                                                                │
static_autoscaler.go:511] No unschedulable pods                                                                                                                                 │
legacy.go:298] No candidates for scale down                                                                                                                                     │

Am I doing something wrong? Is there anything else I need to enable for the autoscaler to start adding nodes?

Thanks!

[mysticaltech]

@dstockhammer Thank you for providing a detailed description of your setup and the issue you're facing.

From your configuration and the logs you've shared, it seems that the cluster-autoscaler is correctly configured with the desired node range (--nodes=1:6:cax41:fsn1:worker-asg-arm-fsn1). However, the autoscaler is not triggering any scaling actions.

The cluster-autoscaler primarily scales based on pod requirements. If there are pods that are unschedulable due to resource constraints, the autoscaler will attempt to add nodes. Conversely, if there are nodes running at low capacity, the autoscaler will consider scaling down.

Here are a few things to consider:

1. Unschedulable Pods: Ensure that there are unschedulable pods that require scaling. If all your pods are running without any pending status, the autoscaler won't add new nodes, even if min_nodes is set to a value greater than 0.

2. Resource Requests: Ensure that your pods have appropriate resource requests set. The autoscaler uses these requests to determine if a new node is needed.

3. Autoscaler Logs: The logs you've shared indicate that there are "No unschedulable pods" and "No candidates for scale down". This suggests that the current node count is sufficient for the workload. If you believe this isn't the case, try deploying a pod with resource requirements that can't be met by the current nodes and observe the autoscaler's behavior.

4. Autoscaler Version: Ensure that you're using a compatible version of the cluster-autoscaler with your Kubernetes cluster. Sometimes, version mismatches can lead to unexpected behaviors.

5. Other Constraints: Check if there are other constraints like taints, affinities, or node selectors that might be preventing the autoscaler from scheduling pods on the autoscaled node pool.

If you've checked all the above and the issue persists, it might be helpful to increase the log verbosity of the cluster-autoscaler to get more detailed information about its decision-making process.

[dstockhammer]

Thanks for the very detailed response!

Unschedulable Pods: Ensure that there are unschedulable pods that require scaling. If all your pods are running without any pending status, the autoscaler won't add new nodes, even if min_nodes is set to a value greater than 0.

That was it! Seems a bit unintuitive, bit I can see now that that's just how the cluster autoscaler works. Passing --enforce-node-group-min-size=true in cluster_autoscaler_extra_args solved the issue for me. There's actually pretty good documentation on the autoscaler in the terraform file, but I missed it because I had an older version where all that was missing 🙃

[mysticaltech]

Good to hear @dstockhammer!!

[pvtrn]

Same problem. Autoscale not working

[mysticaltech]

@PavelTaran You just need to configure your workloads correctly, this is not related to this project. Make sure to increase logging verbosity for the autoscaler, you have variables for this.

[mysticaltech]

@PavelTaran See the above response!