strimzi-cluster-operator failing to strart zookeeper cluster

[koushyk]

Please use this to only for bug reports. For questions or when you need help, you can use the GitHub Discussions, our #strimzi Slack channel or out user mailing list.

Describe the bug
After updating chart from version 0.22.1 to 0.26.0 zookeper cluster failing to start. Operator runs 1 replica of Zookeeper and it never goes green. Scaling zookeeper statefulset to 3 helps.

To Reproduce
Steps to reproduce the behavior:

1. 3 kafka and 3 zookeeper node cluster
2. Scale down deployment strimzi-cluster-operator
3. Scale down cluster-kafka-exporter and cluster-entity-operator deployments
4. Scaling Kafka statefulset down
5. Scaling Zookeeper statefulset down
6. Env stopped
7. Scale up deployment strimzi-cluster-operator and there would be one Zookepeer pod with probes failed

Expected behavior
strimzi-cluster-operator should scale Zookeeper to 3 nodes and kafka to 3 after Zookeeper is up. When setting zookeeper to 1 node it starts fine

Environment (please complete the following information):

• Strimzi version: 0.26.1
• Installation method: Helm chart
• Kubernetes cluster: Kubernetes 1.20
• Infrastructure: Amazon EKS

YAML files and logs

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  annotations:
    meta.helm.sh/release-name: eksdev3
    meta.helm.sh/release-namespace: eksdev3
  creationTimestamp: "2021-12-15T12:05:54Z"
  generation: 5
  labels:
    app.kubernetes.io/managed-by: Helm
  name:
  kafka-cluster
  namespace: eksdev3
  resourceVersion: "74818778"
  uid: 55c8acef-7bc2-4d54-8955-1adf0230f285
spec:
  entityOperator:
    topicOperator:
      logging:
        type: external
        valueFrom:
          configMapKeyRef:
            key: log4j2.xml
            name:
            kafka-log4j2
    userOperator:
      logging:
        type: external
        valueFrom:
          configMapKeyRef:
            key: log4j2.xml
            name:
            kafka-log4j2
  kafka:
    config:
      auto.create.topics.enable: false
      log.message.format.version: 3.0.0
      offsets.topic.replication.factor: 3
      transaction.state.log.min.isr: 1
      transaction.state.log.replication.factor: 3
    jvmOptions:
      -Xms: 1g
      -Xmx: 1g
    listeners:
    - name: plain
      port: 9092
      tls: false
      type: internal
    - name: tls
      port: 9093
      tls: true
      type: internal
    logging:
      type: external
      valueFrom:
        configMapKeyRef:
          key: log4j.properties
          name:
          kafka-log4j
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: kafka-metrics-config.yml
          name: kafka-metrics
    replicas: 3
    storage:
      class:
      kafka-sc
      deleteClaim: false
      size: 100Gi
      type: persistent-claim
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: node.kubernetes.io/role
                  operator: In
                  values:
                  - kafka
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchExpressions:
                  - key: strimzi.io/name
                    operator: In
                    values:
                    -
                    kafka-cluster-kafka
                topologyKey: kubernetes.io/hostname
              weight: 100
        metadata:
          annotations:
            linkerd.io/inject: disabled
        tolerations:
        - effect: NoSchedule
          key: dedicated
          operator: Equal
          value: kafka
      statefulset:
        metadata:
          annotations:
            linkerd.io/inject: disabled
    version: 3.0.0
  kafkaExporter:
    enableSaramaLogging: false
    groupRegex: .*
    logging: info
    topicRegex: .*
  zookeeper:
    logging:
      type: external
      valueFrom:
        configMapKeyRef:
          key: log4j.properties
          name:
          kafka-log4j
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: zookeeper-metrics-config.yml
          name: zookeeper-metrics
    replicas: 3
    storage:
      class:
      kafka-sc
      deleteClaim: false
      size: 10Gi
      type: persistent-claim
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: node.kubernetes.io/role
                  operator: In
                  values:
                  - kafka
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchExpressions:
                  - key: strimzi.io/name
                    operator: In
                    values:
                    -
                    kafka-cluster-kafka
                topologyKey: kubernetes.io/hostname
              weight: 100
        metadata:
          annotations:
            linkerd.io/inject: disabled
        tolerations:
        - effect: NoSchedule
          key: dedicated
          operator: Equal
          value: kafka
      statefulset:
        metadata:
          annotations:
            linkerd.io/inject: disabled

Attach or copy paste the custom resources you used to deploy the Kafka cluster and the relevant YAMLs created by the Cluster Operator.
Attach or copy and paste also the relevant logs.

To easily collect all YAMLs and logs, you can use our report script which will automatically collect all files and prepare a ZIP archive which can be easily attached to this issue.
The usage of this script is:
./report.sh [--namespace <string>] [--cluster <string>]

Additional context
Add any other context about the problem here.

[scholzj]

Can you please format the custom resource you shared as code so that it is readable? Also, you will probably need to provide some logs.

[koushyk]

I meant that strimzi-cluster-operator scales Zookeeper one replica and waits for it to be ready than scales to 3. Logs contain only that pod zookeeper-0 could not connect to zookeeper-1 and zookeeper-2 and that is fine they are not running yet. But Zookeeper-0 never gets ready because probes failing

[scholzj]

Well, I guess you need to scale the ZooKeeper StatefulSet first manually maybe to get all 3 pods running before you scale the operator.

[koushyk]

yep, I'm using such action now but I'm not satisfied with it, as it working ok before the update

[scholzj]

I guess as an alternative you can just delete the stateful set instead of scaling it down 🤷‍♂️ . But this is not necessarily something what you are supposed to use for anything. You are not supposed to modify the resources created by Strimzi. So you cannot expect this to behave the same in every version.

[djotanov]

This does not sound like a desired behavior. In previous versions strimzi was scaling zookeeper to number of replicas specified in Kafka resource. Now that is not happening, which looks like a bug.
To replicate:

• scale down strimzi deployment to 0
• scale down zookeeper/kafka statefulset to 0 (number of replicas in Kafka resource is 3)
• scale up strimzi to 1
• observe that strimzi scales zookeeper only to 1 replica, which never starts because it can't reach the quorum

Logs:
2022-01-06 09:14:11 INFO KafkaAssemblyOperator:1339 - Reconciliation #1(watch) Kafka(default/test-cluster): Scaling Zookeeper up from 0 to 3 replicas
2022-01-06 09:14:11 INFO AbstractScalableResourceOperator:69 - Reconciliation #1(watch) Kafka(default/test-kafka-cluster): Scaling up to 1 replicas