Cluster autoscaler. Nodes are not scaling down.

[v-petukhov]

Hello there!

I have interesting problem with cluster autoscaler.

I've enabled autoscaler pool and all is great, it creates new nodes and add them into cluster. Pods (my application pods) are scheduling onto them with load and later pods are scheduling down. But, created nodes not scaling down neither after 24h, not 36h. They only scaling down after i'll mark them as drained. What can be wrong with cluster configuration?

P.S. Also, I've noticed, that traefik (i'm using it as ingress controller) schedules own pods on cluster autoscaler nodes and after reducing load did not schedules pods on normal nodes, maybe because of this autoscaler nodes are not scaling down?

[fatelgit]

We struggled with autoscaling as well. What helped me a lot was changing the cluster-autoscaler start command to deliver more verbose output. You can do that by adding a higher verbosity level. Example:

         command:
            - ./cluster-autoscaler
            - '--v=4'
            - '--cloud-provider=hetzner'
            - '--stderrthreshold=info'
            - .....

After that you should see a lot more info why it can't scale down any of your nodes. In my case there were some pods with mounts that were not annotated as safe to evict.

cluster-autoscaler.kubernetes.io/safe-to-evict: "true"

[mysticaltech]

FYI folks, we are going to give more options to allow to better control logging for the autoscaler, as it seems to be very important. #859

[mysticaltech]

@v-petukhov @fatelgit You can now customize all logging options for maximum flexibility and to understand why in some cases the autoscaler does not want to scale.