[Bug]: Unable to schedule Zookeeper cluster using node-selector plugin

[michal0000000]

Bug Description

I'm trying to create a HA Kafka cluster across multiple availability domains. I have 5 worker nodes, 3 of which I have labeled nodepool=test
After applying
scheduler.alpha.kubernetes.io/node-selector: nodepool=test
to the kafka namespace and deploying the cluster I get an error during Zookeeper's pod scheduling saying:
3 node(s) didn't satisfy existing pods anti-affinity rules (there are no anti-affinity rules)

The thing is:

1. All nodes seem to be properly labeled
2. The same configuration WORKS on kafka nodes, only ZK is buggy
3. There are no anti-affinity rules neither in the node's or pods' configuration
4. The issue exists only with three specific nodes, the other two workers in the cluster are schedulable with the same configuration
5. I tried adding other nodes to the pool and noticed there was no problem with scheduling on them, just the initial three ones are problematic. However, the configuration between all nodes is the same as far as k8s goes. The only difference is the nodepool label.

I also checked for antiAffinity rules of other pods but found nothing.

Kubernetes cluster info:

• 3 master nodes, 5 worker nodes
• containerd, 1.6.16

Full error message:

le.kubernetes.io/master: }. preemption: 0/8 nodes are available: 3 No preemption victims found for incoming pod, 5 Preemption is not helpful for scheduling..

Steps to reproduce

1. Create a 5 node cluster and label three of them with scheduler.alpha.kubernetes.io/node-selector: nodepool=test
2. Create a namespace using the config below
3. Create a Kafka cluster using the config below
4. Don't profit

Expected behavior

Two zookeeper instances should have been scheduled somewhere on the three nodes labeled with nodepool=test

Strimzi version

latest

Kubernetes version

Kubernetes 1.26.1

Installation method

Downloaded YAMLs from https://strimzi.io/install/latest?namespace=kafka

Infrastructure

Bare-metal

Configuration files and logs

Kafka config

kind: Kafka
metadata:
  name: kafka-broker
spec:
  kafka:
    version: 3.4.0
    replicas: 2
    logging:
      type: inline
      loggers:
        kafka.root.logger.level: "ERROR"
    listeners:
	...
    config:
      ...
    storage:
      type: ephemeral
    template:
      pod:
        affinity:
	  # Basically the same setting that is present in the namespace config, works
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                  - key: nodepool
                    operator: In
                    values:
                    - test
	  # Spreads the pods out across all nodes
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values:
                        - kafka-broker-kafka
                topologyKey: "kubernetes.io/hostname"
  zookeeper:
    replicas: 2
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}

namespace config

kind: Namespace
metadata:
  name: kafka
  annotations:
    scheduler.alpha.kubernetes.io/node-selector: nodepool=test

kubectl get nodes --show-labels

worker1   Ready    <none>          5d1h   v1.26.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,nodepool=test,kubernetes-host=,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker1,kubernetes.io/os=linux,node-role=worker
worker2   Ready    <none>          5d1h   v1.26.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,nodepool=test,kubernetes-host=,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker2,kubernetes.io/os=linux,node-role=worker
worker3   Ready    <none>          5d1h   v1.26.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,nodepool=test,testpool=test,kubernetes-host=,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker3,kubernetes.io/os=linux,node-role=worker
master1              Ready    control-plane   5d2h   v1.26.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes-host=,kubernetes.io/arch=amd64,kubernetes.io/hostname=master1,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node-role=master,node.kubernetes.io/exclude-from-external-load-balancers=
master2              Ready    control-plane   5d2h   v1.26.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes-host=,kubernetes.io/arch=amd64,kubernetes.io/hostname=master2,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node-role=master,node.kubernetes.io/exclude-from-external-load-balancers=
master3              Ready    control-plane   5d2h   v1.26.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes-host=,kubernetes.io/arch=amd64,kubernetes.io/hostname=master3,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node-role=master,node.kubernetes.io/exclude-from-external-load-balancers=
worker4              Ready    <none>          5d1h   v1.26.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,nodepool=not-test,kubernetes-host=,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker4,kubernetes.io/os=linux,node-role=worker
worker5     Ready    <none>          5d1h   v1.26.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,nodepool=not-test2,kubernetes-host=,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker5,kubernetes.io/os=linux,node-role=worker

kubectl describe node worker1 (node belongs to test nodepool)

Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    nodepool=test
                    kubernetes-host=
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=worker1
                    kubernetes.io/os=linux
                    node-role=worker
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 30 Aug 2023 17:07:24 +0200
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  worker1
  AcquireTime:     <unset>
  RenewTime:       Mon, 04 Sep 2023 19:14:46 +0200
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  NetworkUnavailable   False   Wed, 30 Aug 2023 17:08:40 +0200   Wed, 30 Aug 2023 17:08:40 +0200   CalicoIsUp                   Calico is running on this node
  MemoryPressure       False   Mon, 04 Sep 2023 19:14:51 +0200   Wed, 30 Aug 2023 17:07:23 +0200   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Mon, 04 Sep 2023 19:14:51 +0200   Wed, 30 Aug 2023 17:07:23 +0200   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Mon, 04 Sep 2023 19:14:51 +0200   Wed, 30 Aug 2023 17:07:23 +0200   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Mon, 04 Sep 2023 19:14:51 +0200   Wed, 30 Aug 2023 17:08:26 +0200   KubeletReady                 kubelet is posting ready status. AppArmor enabled
Addresses:
  InternalIP:  1.2.3.4
  Hostname:    worker1
Capacity:
  cpu:                8
  ephemeral-storage:  15398944Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             32866384Ki
  pods:               110
Allocatable:
  cpu:                8
  ephemeral-storage:  14191666767
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             32763984Ki
  pods:               110
System Info:
  Machine ID:                 <redacted>
  System UUID:                <redacted>
  Boot ID:                    <redacted>
  Kernel Version:             6.1.0-10-amd64
  OS Image:                   Debian GNU/Linux 12 (bookworm)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.6.16
  Kubelet Version:            v1.26.1
  Kube-Proxy Version:         v1.26.1
PodCIDR:                      10.10.10.0/24
PodCIDRs:                     10.10.10.0/24
Non-terminated Pods:          (6 in total)
  Namespace                   Name                                         CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  kafka                       kafka-broker-kafka-0                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         60m
  kafka                       strimzi-cluster-operator-7f6dc46d44-dqvzq    200m (2%)     1 (12%)     384Mi (1%)       384Mi (1%)     68m
  kube-system                 calico-node-v9j57                            150m (1%)     300m (3%)   64M (0%)         500M (1%)      5d2h
  kube-system                 kube-proxy-x4468                             0 (0%)        0 (0%)      0 (0%)           0 (0%)         5d2h
  kube-system                 nodelocaldns-wxgww                           100m (1%)     0 (0%)      70Mi (0%)        200Mi (0%)     4d3h
  openebs                     openebs-ndm-pv65k                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         4d6h
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests       Limits
  cpu                450m (5%)      1300m (16%)
  memory             527396Ki (1%)  1112368384 (3%)
  ephemeral-storage  0 (0%)         0 (0%)
  hugepages-1Gi      0 (0%)         0 (0%)
  hugepages-2Mi      0 (0%)         0 (0%)
Events:              <none>

kubectl describe node worker4 (node does not belong to test nodepool)

Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    nodepool=not-test
                    kubernetes-host=
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=worker4
                    kubernetes.io/os=linux
                    node-role=worker
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 30 Aug 2023 17:07:24 +0200
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  worker4
  AcquireTime:     <unset>
  RenewTime:       Mon, 04 Sep 2023 19:19:25 +0200
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Wed, 30 Aug 2023 17:08:39 +0200   Wed, 30 Aug 2023 17:08:39 +0200   CalicoIsUp                   Calico is running on this node
  MemoryPressure       False   Mon, 04 Sep 2023 19:19:33 +0200   Wed, 30 Aug 2023 17:07:22 +0200   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Mon, 04 Sep 2023 19:19:33 +0200   Wed, 30 Aug 2023 17:07:22 +0200   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Mon, 04 Sep 2023 19:19:33 +0200   Wed, 30 Aug 2023 17:07:22 +0200   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Mon, 04 Sep 2023 19:19:33 +0200   Wed, 30 Aug 2023 17:08:25 +0200   KubeletReady                 kubelet is posting ready status. AppArmor enabled
Addresses:
  InternalIP:  1.2.3.5
  Hostname:    worker4
Capacity:
  cpu:                8
  ephemeral-storage:  10239936Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             32866384Ki
  pods:               110
Allocatable:
  cpu:                8
  ephemeral-storage:  9437125002
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             32763984Ki
  pods:               110
System Info:
  Machine ID:                 <redacted>
  System UUID:                <redacted>
  Boot ID:                    <redacted>
  Kernel Version:             6.1.0-10-amd64
  OS Image:                   Debian GNU/Linux 12 (bookworm)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.6.16
  Kubelet Version:            v1.26.1
  Kube-Proxy Version:         v1.26.1
PodCIDR:                      10.10.10.0/24
PodCIDRs:                     10.10.10.0/24
Non-terminated Pods:          (4 in total)
  Namespace                   Name                  CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                  ------------  ----------  ---------------  -------------  ---
  kube-system                 calico-node-wxbkw     150m (1%)     300m (3%)   64M (0%)         500M (1%)      5d2h
  kube-system                 kube-proxy-f4x6g      0 (0%)        0 (0%)      0 (0%)           0 (0%)         5d2h
  kube-system                 nodelocaldns-t7fc2    100m (1%)     0 (0%)      70Mi (0%)        200Mi (0%)     4d4h
  openebs                     openebs-ndm-ff8fz     0 (0%)        0 (0%)      0 (0%)           0 (0%)         4d6h
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests        Limits
  --------           --------        ------
  cpu                250m (3%)       300m (3%)
  memory             137400320 (0%)  709715200 (2%)
  ephemeral-storage  0 (0%)          0 (0%)
  hugepages-1Gi      0 (0%)          0 (0%)
  hugepages-2Mi      0 (0%)          0 (0%)
Events:              <none>

Additional context

Any help would be greatly appreciated. Thank you!

[scholzj]

Not sure if any of this helps, but ...

• Not sure why are you using any namespace annotations. The node affinity should be sufficient. I would suggest to remove it to avoid any mismatches.
• Without any logs and pod listings,it is not clear what the issue is. You talk about ZooKeeper cluster in the title - so are the ZooKeeper pods stuck? You do not have any node affinity in the .spec.zookeeper section, so the pod will have no affinity configured.
• What does it mean that yo labaled the nodes scheduler.alpha.kubernetes.io/node-selector: nodepool=test as you say? The label should be nodepool: test for the affinity you have configured.
• You say you do not have any anti-affinity rules. But the Kafka CR you shared does have them.

[michal0000000]

1. Checking pod node affinity rules:

kubectl describe pod kafka-broker-zookeeper-0 (method 1)

Node-Selectors:              <none>
Tolerations:           node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  7m29s  default-scheduler  0/8 nodes are available: 1 node(s) didn't satisfy existing pods anti-affinity rules. preemption: 0/8 nodes are available: 8 No preemption victims found for incoming pod..

kubectl get pod kafka-broker-zookeeper-0 -o yaml (method 2)

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
                - matchExpressions:
                  - key: nodepool
                    operator: In
                    values:
                    - test

So the two outputs show different information, I dont know what to think of it.

2. Other info.

I read in other issues that ZK is supposed to spawn a stateful set for itself. I do not see it when i run kubectl get all:

NAME                                            READY   STATUS    RESTARTS   AGE
pod/kafka-broker-zookeeper-0                    0/1     Pending   0          13m
pod/kafka-broker-zookeeper-1                    0/1     Pending   0          13m
pod/strimzi-cluster-operator-7f6dc46d44-dqvzq   1/1     Running   0          17h

NAME                                    TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                      AGE
service/kafka-broker-zookeeper-client   ClusterIP   10.70.15.28   <none>        2181/TCP                     13m
service/kafka-broker-zookeeper-nodes    ClusterIP   None          <none>        2181/TCP,2888/TCP,3888/TCP   13m

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/strimzi-cluster-operator   1/1     1            1           18h

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/strimzi-cluster-operator-7f6dc46d44   1         1         1       18h

[scholzj]

So the two outputs show different information, I dont know what to think of it.

I'm not sure what exactly you mean with different information. It has the affinity in the Pod in the YAML. It just seems it cannot satisfy it.

I read in other issues that ZK is supposed to spawn a stateful set for itself. I do not see it when i run kubectl get all:

It is not clear what Strimzi version you use. Butwe do not use StatefulSets anymore.

[michal0000000]

1. When running kubectl describe pod kafka-broker-zookeeper-0 the NodeSelector column has a value of <none>. That's what I meant.
2. I'm using the latest version, so this seems to be ok.
   Any other ideas ?

[scholzj]

But you are not using a node selector. You are using node affinity which is not the same. So normally it is expected that node selector would no be set.

[michal0000000]

Obviously there was some kind of bug. I reinstalled the same configs using a different namespace name and it worked. Maybe something was improperly removed from etcd.