Pod Anti-Affinity not working when using Persistent Volume

[hsiaand]

Please use this to only for bug reports. For questions or when you need help, you can use the GitHub Discussions, our #strimzi Slack channel or out user mailing list.

Describe the bug
When creating a 3 replica cluster on OpenEBS StorageClass with AntiAffinity, the PVCs are created on the same node, which prevents 2nd and 3rd Pods from starting.

To Reproduce
Steps to reproduce the behavior:
Following Kafka Definition are used to create a cluster.

% kubectl apply -f - <<EOD
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: sdn-cluster
  namespace: strimzi
spec:
  kafka:
    version: 3.1.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.1"
    storage:
      deleteClaim: true
      size: 100Gi
      type: persistent-claim
      class: vg01-lvmpv
    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values:
                        - sdn-cluster-kafka
                topologyKey: "kubernetes.io/hostname"
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 20Gi
      deleteClaim: true
      class: vg01-lvmpv
    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values:
                        - sdn-cluster-zookeeper
                topologyKey: "kubernetes.io/hostname"
  entityOperator:
    topicOperator: {}
    userOperator: {}
EOD

The Pod creation never completes

% kubectl get po -n strimzi
NAME                                        READY   STATUS    RESTARTS   AGE
sdn-cluster-zookeeper-0                     0/1     Running   0          39s
sdn-cluster-zookeeper-1                     0/1     Pending   0          39s
sdn-cluster-zookeeper-2                     0/1     Pending   0          39s
strimzi-cluster-operator-587cb79468-qh2nf   1/1     Running   1          2d17h

3 PVCs are created

 % kubectl get pvc -n strimzi
NAME                           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-sdn-cluster-zookeeper-0   Bound    pvc-0345f4b8-7b72-4518-b58c-1c3c810a9950   20Gi       RWO            vg01-lvmpv     46s
data-sdn-cluster-zookeeper-1   Bound    pvc-3eee7aca-d11e-4a4d-a4ef-723627516346   20Gi       RWO            vg01-lvmpv     46s
data-sdn-cluster-zookeeper-2   Bound    pvc-d08a9ab0-d3f5-417a-b44d-73ae0ef223ef   20Gi       RWO            vg01-lvmpv     46s

Inspect the PVs are created on same worker node

% kubectl get pv pvc-0345f4b8-7b72-4518-b58c-1c3c810a9950 pvc-3eee7aca-d11e-4a4d-a4ef-723627516346 pvc-d08a9ab0-d3f5-417a-b44d-73ae0ef223ef -o yaml|grep wrk
            - sjt-c2g1wrk3
            - sjt-c2g1wrk3
            - sjt-c2g1wrk3

Here is StatefulSet created by operator

 % kubectl get sts sdn-cluster-zookeeper -o yaml -n strimzi
apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    strimzi.io/storage: '{"type":"persistent-claim","size":"20Gi","class":"vg01-lvmpv","deleteClaim":true}'
  creationTimestamp: "2022-03-20T12:24:57Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: sdn-cluster
    app.kubernetes.io/managed-by: strimzi-cluster-operator
    app.kubernetes.io/name: zookeeper
    app.kubernetes.io/part-of: strimzi-sdn-cluster
    strimzi.io/cluster: sdn-cluster
    strimzi.io/kind: Kafka
    strimzi.io/name: sdn-cluster-zookeeper
  name: sdn-cluster-zookeeper
  namespace: strimzi
  ownerReferences:
  - apiVersion: kafka.strimzi.io/v1beta2
    blockOwnerDeletion: false
    controller: false
    kind: Kafka
    name: sdn-cluster
    uid: c282e13d-d6be-450b-adbd-68fe1f1524f9
  resourceVersion: "131932960"
  uid: 9947715c-6a2a-4dad-8cfc-3fcf7528d020
spec:
  podManagementPolicy: Parallel
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      strimzi.io/cluster: sdn-cluster
      strimzi.io/kind: Kafka
      strimzi.io/name: sdn-cluster-zookeeper
  serviceName: sdn-cluster-zookeeper-nodes
  template:
    metadata:
      annotations:
        strimzi.io/cluster-ca-cert-generation: "0"
        strimzi.io/generation: "0"
        strimzi.io/logging-hash: 0f057cb0003c78f02978b83e4fabad5bd508680c
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: sdn-cluster
        app.kubernetes.io/managed-by: strimzi-cluster-operator
        app.kubernetes.io/name: zookeeper
        app.kubernetes.io/part-of: strimzi-sdn-cluster
        strimzi.io/cluster: sdn-cluster
        strimzi.io/kind: Kafka
        strimzi.io/name: sdn-cluster-zookeeper
      name: sdn-cluster-zookeeper
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: strimzi.io/name
                operator: In
                values:
                - sdn-cluster-zookeeper
            topologyKey: kubernetes.io/hostname
      containers:
      - command:
        - /opt/kafka/zookeeper_run.sh
        env:
        - name: ZOOKEEPER_METRICS_ENABLED
          value: "false"
        - name: ZOOKEEPER_SNAPSHOT_CHECK_ENABLED
          value: "true"
        - name: STRIMZI_KAFKA_GC_LOG_ENABLED
          value: "false"
        - name: KAFKA_HEAP_OPTS
          value: -Xms128M
        - name: ZOOKEEPER_CONFIGURATION
          value: |
            tickTime=2000
            initLimit=5
            syncLimit=2
            autopurge.purgeInterval=1
        image: quay.io/strimzi/kafka:0.28.0-kafka-3.1.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - /opt/kafka/zookeeper_healthcheck.sh
          failureThreshold: 3
          initialDelaySeconds: 15
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: zookeeper
        ports:
        - containerPort: 2888
          name: tcp-clustering
          protocol: TCP
        - containerPort: 3888
          name: tcp-election
          protocol: TCP
        - containerPort: 2181
          name: tcp-clients
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - /opt/kafka/zookeeper_healthcheck.sh
          failureThreshold: 3
          initialDelaySeconds: 15
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp
          name: strimzi-tmp
        - mountPath: /var/lib/zookeeper
          name: data
        - mountPath: /opt/kafka/custom-config/
          name: zookeeper-metrics-and-logging
        - mountPath: /opt/kafka/zookeeper-node-certs/
          name: zookeeper-nodes
        - mountPath: /opt/kafka/cluster-ca-certs/
          name: cluster-ca-certs
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 0
      serviceAccount: sdn-cluster-zookeeper
      serviceAccountName: sdn-cluster-zookeeper
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir:
          medium: Memory
          sizeLimit: 1Mi
        name: strimzi-tmp
      - configMap:
          defaultMode: 420
          name: sdn-cluster-zookeeper-config
        name: zookeeper-metrics-and-logging
      - name: zookeeper-nodes
        secret:
          defaultMode: 292
          secretName: sdn-cluster-zookeeper-nodes
      - name: cluster-ca-certs
        secret:
          defaultMode: 292
          secretName: sdn-cluster-cluster-ca-cert
  updateStrategy:
    type: OnDelete
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 20Gi
      storageClassName: vg01-lvmpv
      volumeMode: Filesystem
    status:
      phase: Pending
status:
  collisionCount: 0
  currentReplicas: 3
  currentRevision: sdn-cluster-zookeeper-66565f6954
  observedGeneration: 1
  replicas: 3
  updateRevision: sdn-cluster-zookeeper-66565f6954
  updatedReplicas: 3

Expected behavior
The operator should not pre-allocate PVC on the same worker node.

Environment (please complete the following information):

• Strimzi version: 0.28.0
• Installation method: Helm Chart
• Kubernetes cluster: 1.21.2
• Infrastructure: on-prim K8S

YAML files and logs

Attach or copy paste the custom resources you used to deploy the Kafka cluster and the relevant YAMLs created by the Cluster Operator.
Attach or copy and paste also the relevant logs.

2022-03-20 12:22:54 INFO  AbstractOperator:373 - Reconciliation #502(timer) Kafka(strimzi/sdn-cluster): Reconciliation is in progress
2022-03-20 12:22:58 INFO  OperatorWatcher:38 - Reconciliation #505(watch) Kafka(strimzi/sdn-cluster): Kafka sdn-cluster in namespace strimzi was ADDED
2022-03-20 12:23:54 INFO  ClusterOperator:123 - Triggering periodic reconciliation for namespace strimzi
2022-03-20 12:23:54 INFO  AbstractOperator:373 - Reconciliation #502(timer) Kafka(strimzi/sdn-cluster): Reconciliation is in progress
2022-03-20 12:24:54 INFO  AbstractOperator:373 - Reconciliation #502(timer) Kafka(strimzi/sdn-cluster): Reconciliation is in progress
2022-03-20 12:24:54 ERROR Util:149 - Reconciliation #502(timer) Kafka(strimzi/sdn-cluster): Exceeded timeout of 300000ms while waiting for Pods resource sdn-cluster-zookeeper-1 in namespace strimzi to be ready
2022-03-20 12:24:54 ERROR Util:149 - Reconciliation #502(timer) Kafka(strimzi/sdn-cluster): Exceeded timeout of 300000ms while waiting for Pods resource sdn-cluster-zookeeper-0 in namespace strimzi to be ready
2022-03-20 12:24:54 ERROR Util:149 - Reconciliation #502(timer) Kafka(strimzi/sdn-cluster): Exceeded timeout of 300000ms while waiting for Pods resource sdn-cluster-zookeeper-2 in namespace strimzi to be ready
2022-03-20 12:24:54 ERROR AbstractOperator:247 - Reconciliation #502(timer) Kafka(strimzi/sdn-cluster): createOrUpdate failed
io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for Pods resource sdn-cluster-zookeeper-0 in namespace strimzi to be ready
	at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:150) ~[io.strimzi.operator-common-0.28.0.jar:0.28.0]
	at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:69) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) [io.netty.netty-transport-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2022-03-20 12:24:54 INFO  OperatorWatcher:38 - Reconciliation #507(watch) Kafka(strimzi/sdn-cluster): Kafka sdn-cluster in namespace strimzi was MODIFIED
2022-03-20 12:24:54 INFO  CrdOperator:113 - Reconciliation #502(timer) Kafka(strimzi/sdn-cluster): Status of Kafka sdn-cluster in namespace strimzi has been updated
2022-03-20 12:24:54 WARN  AbstractOperator:532 - Reconciliation #502(timer) Kafka(strimzi/sdn-cluster): Failed to reconcile
io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for Pods resource sdn-cluster-zookeeper-0 in namespace strimzi to be ready
	at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:150) ~[io.strimzi.operator-common-0.28.0.jar:0.28.0]
	at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:69) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) [io.netty.netty-transport-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2022-03-20 12:24:54 INFO  AbstractOperator:226 - Reconciliation #507(watch) Kafka(strimzi/sdn-cluster): Kafka sdn-cluster will be checked for creation or modification
2022-03-20 12:25:54 INFO  ClusterOperator:123 - Triggering periodic reconciliation for namespace strimzi
2022-03-20 12:25:54 INFO  AbstractOperator:373 - Reconciliation #507(watch) Kafka(strimzi/sdn-cluster): Reconciliation is in progress
2022-03-20 12:26:54 INFO  AbstractOperator:373 - Reconciliation #507(watch) Kafka(strimzi/sdn-cluster): Reconciliation is in progress
2022-03-20 12:27:54 INFO  ClusterOperator:123 - Triggering periodic reconciliation for namespace strimzi
2022-03-20 12:27:54 INFO  AbstractOperator:373 - Reconciliation #507(watch) Kafka(strimzi/sdn-cluster): Reconciliation is in progress

To easily collect all YAMLs and logs, you can use our report script which will automatically collect all files and prepare a ZIP archive which can be easily attached to this issue.
The usage of this script is:
./report.sh --namespace <string> --cluster <string> [--bridge <string>] [--connect <string>] [--mm2 <string>]

Additional context
Add any other context about the problem here.

[scholzj]

Strimzi does not schedule any pods or schedule any PVs. This is done by Kubernetes and the rest of your infrastructure (in this case probably OpenEBS). Strimzi just passes the information from the configuration into the PVC and into the StatefulSet / Pod definition. So there is not much Strimzi can do about this, you need to confgure your infrastructure components to match the architecture you want to use.

If needed, you can consider:

• Pre-create the PVCs and PVs
• Use different storage class per-broker

But the details might depend on your infrastructure.

[hsiaand]

@scholzj Thank you for your prompt response.
OpenEBS has an auto-provisioner. It does not require preallocating PVC.

I am using the following test StatefulSet to prove my point.
When the pods are created, it is creating one pod and associated PV at a time. That means 2nd Pod and PV will not be created if 1st Pod is not in a ready state.

 % kubectl apply -f - <<EOD
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: local-test
spec:
  serviceName: "local-service"
  replicas: 3
  selector:
    matchLabels:
      app: local-test
  template:
    metadata:
      labels:
        app: local-test
    spec:
      containers:
      - name: test-container
        image: k8s.gcr.io/busybox
        command:
        - "/bin/sh"
        args:
        - "-c"
        - "sleep 100000"
        volumeMounts:
        - name: local-vol
          mountPath: /usr/test-pod
  volumeClaimTemplates:
  - metadata:
      name: local-vol
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "vg01-lvmpv"
      resources:
        requests:
          storage: 5Gi
EOD

All storage are been schedule automatically on different worker nodes

%  kubectl get pvc
NAME                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
local-vol-local-test-0   Bound    pvc-7747b568-10eb-4771-bc23-7cbea8552033   5Gi        RWO            vg01-lvmpv     5m6s
local-vol-local-test-1   Bound    pvc-9cd17195-c2c6-49aa-b658-5703fa87f751   5Gi        RWO            vg01-lvmpv     5m1s
local-vol-local-test-2   Bound    pvc-84f594c8-424e-462a-870f-07e86a8d75b2   5Gi        RWO            vg01-lvmpv     4m56s

% kubectl get pv pvc-7747b568-10eb-4771-bc23-7cbea8552033 pvc-9cd17195-c2c6-49aa-b658-5703fa87f751 pvc-84f594c8-424e-462a-870f-07e86a8d75b2 -o yaml|grep wrk
            - sjt-c2g1wrk5
            - sjt-c2g2wrk0
            - sjt-c2g2wrk1

Regarding your recommendation on pre-create PV, it is won't help in case of failed worker node to auto-recover.
Also using different storage class still need manual intervention and working against Kubernetes automation...

[scholzj]

The pods are created all at the same time. Whether the PVC is create when the pod is created or when the PVC is created is something you can control in your Storage Class with the volumeBindingMode.

[hsiaand]

@scholzj Thank you for the suggestion on volumeBindingMode.
I never run into Pod scheduling issue before, but looking at the volumeBindingMode, it defaults to immediate.
After recreating the StorageClass with volumeBindingMode to WaitForFirstConsumer, the Kafka cluster is created immediately.