Autoscaler not adding node based on nodeSelector

[jonas-tm]

Hi,
I am having issues with the autoscaler.
I am trying to schedule a pod on an autoscaler node via nodeSelector. It is working fine when targeting the labels of agent node pool but when targeting an autoscale node pool (init pool size 0) autoscaler is not picking up the node.
Am I missing a certain config in kube.tf or is there a way to debug this to know why it is not scheduled?

Setup:

• kube-hetzner 1.9.1
• Kubernetes v1.25

Cluster Autoscaler logs

I0219 17:57:56.723532       1 hetzner_node_group.go:437] Set node group draining-node-pool size from 0 to 0, expected delta 0
I0219 17:57:56.723568       1 hetzner_node_group.go:437] Set node group kube-hetzner-autoscaler-small size from 0 to 0, expected delta 0
I0219 17:57:56.724498       1 klogx.go:86] Pod my-app/my-app-54876b48c-vvm7f is unschedulable
I0219 17:57:56.724789       1 scale_up.go:462] No expansion options
I0219 17:57:56.724933       1 legacy.go:727] No candidates for scale down

Part of kube.tf

  autoscaler_nodepools = [
    {
      name        = "autoscaler-small"
      server_type = "cpx21", # must be same or better than the control_plane server type (regarding disk size)!
      location    = "ash",
      labels = [
        "location=ash",
        "cores=3",
        "memory=4",
        "disk=40",
        "type=autoscaler",
        "size=small"
      ],
      min_nodes = 0
      max_nodes = 15
    },

Deployment YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: my-app
  name: my-app
  namespace: test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
    spec:
      containers:
          image: 'myimage.com/myimage'
          imagePullPolicy: Always
          name: my-app
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
          resources: {}
      nodeSelector:
        size: small
        type: autoscaler
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
# ....

[mysticaltech]

@ifeulner Any ideas on this?

[ifeulner]

@jonas-tm Hi, just seen this. Actually, I think you should not do this. Autoscaler decides to start new nodes based on memory / cpu requirements / constraints. With an explicit node selector you are asking to schedule on a node which might actually not be available (because there was no reason to ramp up new nodes...). So don't do this.
You can force the creation of a new node by creating a pod with memory / cpu requirements that exceed your current cluster capabilities.

[v-petukhov]

@jonas-tm Hi, just seen this. Actually, I think you should not do this. Autoscaler decides to start new nodes based on memory / cpu constraints. With an explicit node selector you are asking to schedule on a node which might actually not be available (because there was no reason to ramp up new nodes...).

Hey, maybe this is not the right question for this discussion. But, is there any possibilities to set label on autoscaler nodes? As I can understand approach:

 autoscaler_nodepools = [
    {
      name        = "autoscaler-small"
      server_type = "cpx21", # must be same or better than the control_plane server type (regarding disk size)!
      location    = "ash",
      labels = [
        "location=ash",
        "cores=3",
        "memory=4",
        "disk=40",
        "type=autoscaler",
        "size=small"
      ],
      min_nodes = 0
      max_nodes = 15
    },

Will not work. Or am I wrong?

[jonas-tm]

@ifeulner what approach would you recommend to use if one has multiple different autoscaler pools and want to target a specific one?

[mysticaltech]

@v-petukhov Indeed not the right place to ask that, but the answer is no, labels are not supported for autoscaler nodes at this point, but it could be done, please do not hesitate to open a feature request or submit a PR.

[valkenburg-prevue-ch]

I actually looked into this a little while back, and I understood that the hetzner autoscaling does not support node labels and that the auto-scaling alhorithm does not look at nodeselectors of unschedulable pods vs labels of autoscalable nodes. But I hope my understanding was wrong: I'd love this feature.

[mysticaltech]

Actually, it's possible to add labels, just because when an autoscaled node starts, it uses our own cloud-init, and in there we can pipe the labels and give it to the k3s node config.

But you are right, it would be informational only, not really related to the logic of the autoscaler itself.

[valkenburg-prevue-ch]

Thanks for clarifying and confirming!

[konnextv]

On EKS this works by constructing a 'pseudo node' and checking if the nodeSelector/affinity matches the node:
https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-can-i-scale-a-node-group-to-0

But I suppose this is not possible with the Hetzner CA provider? Maybe this is something worth implementing to make it possible to use scale-to-0-nodegroups in combination with nodeSelector/affinity.

[mysticaltech]

@konnextv Interesting! @ifeulner FYI, you might know more about this than me.