How to make autoscaler scale down again

[jonas-tm]

In my test cluster I have 2 agent-nodes and an autoscale pool with min 0

Since my agent-nodes got deleted and recreated most pods were moved to a newly started autoscale node.

Now all agent nodes are running fine again. However, the pods do not get moved to the agent-nodes back again keeping the autoscale node alive.

Is there a config I can apply to force desired behavior of always preferring agent nodes enabling the autoscale node to be scaled down?

[mysticaltech]

@jonas-tm You could try draining the autoscaled node(s) in question, that will force all workloads to move to the agent nodes. kubectl drain node-name

[jonas-tm]

@mysticaltech Yes I did exactly that (also worked as expected). However, I am looking for an automated approach since I sometimes have a bit of load which ramps up the autoscale pool, but it should automatically scale down to 0 nodes again (pushing pods to agent pool).

[ifeulner]

If the load is low enough to be taken care of by the other, general agents, the cluster autoscaler removes the autoscaler nodes automatically. It's important to have proper resource settings on the pods.

[TimHeckel]

Hi @ifeulner - thanks for the implementation work on the autoscaler, it's pretty incredible. I am hopeful you can help me understand "It's important to have proper resource settings on the pods." a bit more -- the autoscaler scaled up when I increased the replicas in my deployment, but when I reduced them, it didn't scale down. A number of other pods were also scheduled on the autoscaled node at that point.

At this point, the autoscaler won't automatically drain the node to push these other pods back to the "general" agent nodes, correct? Is there a strategy you use in this scenario to ensure that any additionally scheduled pods on the autoscale node are drained and moved back to the general agent nodes, or do you recommend some kind of taint to be applied to all pods except those that you want scheduled on the autoscaled node?

Thanks for helping clarify!

[mysticaltech]

@TimHeckel Just to get you going while @ifeulner addresses your question more directly, here's what GPT-4 has to say about it:

---

Hello,

I'm glad you're finding the Kubernetes autoscaler helpful. I can definitely help clarify your question about proper resource settings on pods and how it affects the autoscaler behavior.

When you reduce the number of replicas in your deployment, the Kubernetes Cluster Autoscaler (CA) should ideally scale down the nodes if they are underutilized. However, the autoscaler might not scale down immediately if there are other pods running on the node that prevents it from being safely removed. This can happen if the other pods have no other suitable nodes to be rescheduled on or if they have a local storage that would be lost during the rescheduling process.

To address this issue and ensure that the additional pods on the autoscale node are drained and moved back to the general agent nodes, you can consider the following strategies:

1. Taints and Tolerations: You can apply a taint on the autoscaled nodes to prevent other pods from being scheduled on them. Taints allow nodes to repel a set of pods. Then, add tolerations to the specific pods that you want to be scheduled on these nodes. This way, only the pods with the matching tolerations will be scheduled on the autoscaled nodes.

2. Pod Disruption Budgets: You can define Pod Disruption Budgets (PDBs) to limit the number of concurrent disruptions that your application experiences, allowing for more controlled rescheduling of your pods. PDBs can help make your application more resilient to disruptions caused by the autoscaler during scale-down events.

3. Pod Priority and Preemption: By assigning different priority classes to your pods, you can influence which pods are more likely to be evicted during a scale-down event. Pods with lower priority will be the first to be removed when Kubernetes needs to evict one or more pods. This way, you can ensure that important pods stay running and the less important ones are removed first.

As for the initial quote, "It's important to have proper resource settings on the pods," this refers to setting appropriate resource requests and limits (CPU and memory) for your pods. Proper resource settings help the Kubernetes scheduler to make better decisions about placing pods on nodes, and it also helps the autoscaler make better decisions about scaling up or down. If your pods don't have proper resource settings, the autoscaler may not work optimally, leading to situations like the one you're experiencing.

I hope this helps clarify your concerns. Let me know if you have any further questions!