kafka pods starting one by one

[shreyasarani23]

Hi @scholzj we have 3 kafka and 3 zookeeper pods running in our AKS(Azure kubernetes service) cluster. Below is the deployment yaml file.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-kafka-cluster
spec:
  kafka:
    template:
      podDisruptionBudget:
        maxUnavailable: 2
      pod:
        metadata:
          labels:
            app: kafkaPod
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - kafkaPod
                topologyKey: "kubernetes.io/hostname"
    version: 2.8.0
    replicas: 3
    logging:
      type: inline
      loggers:
        kafka.root.logger.level: "INFO"
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: 2.8
      inter.broker.protocol.version: 2.8
      auto.leader.rebalance.enable: "false"
      delete.topic.enable: "true"
      auto.create.topics.enable: "false"
    storage:
      type: persistent-claim
      size: 1Gi
      class: kafka-storage-class
      deleteClaim: false
  zookeeper:
    template:
      podDisruptionBudget:
        maxUnavailable: 2
      pod:
        metadata:
          labels:
            app: zookeeperPod
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - zookeeperPod
                topologyKey: "kubernetes.io/hostname"
    replicas: 3
    logging:
      type: inline
      loggers:
        zookeeper.root.logger: "INFO"
    storage:
      type: persistent-claim
      size: 1Gi
      class: kafka-storage-class
      deleteClaim: false
  entityOperator:
    topicOperator:
      watchedNamespace: kafka
      reconciliationIntervalSeconds: 60
      logging:
        type: inline
        loggers:
          rootLogger.level: "INFO"

Below is our deployement file for topic creation

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: user-login-from-api
  labels:
    strimzi.io/cluster: my-kafka-cluster
spec:
  topicName: user_login_from_api
  partitions: 2
  replicas: 2

The problem is that we are stopping our AKS cluster daily to save the expense. And our cluster daily autostarts at [9:30 AM IST]. we have 4 node AKS cluster, when the cluster is started one by one pods starts running. For example:
first the kafka-pod-0 will come to running state, whereas kafka-pod-1 and kafka-pod-2 will not be in running state
Next the kafka-pod-1 will come to running state, whereas kafka-pod-2 will not be in running state
finally kafka-pod-2 will not be in running state will also be in running state.

Note: I guess order may differ but not sure. But all the 3 kafka pods will not be up. It will be up one by one.

So by checking the logs of kafka-pod-1 logs at 9:38 AM IST

2021-12-03 04:08:50,820 ERROR [Broker id=1] Received LeaderAndIsrRequest with correlation id 1 from controller 1 epoch 14 for partition user_login_from_api-1 (last update controller epoch 13) but cannot become follower since the new leader 0 is unavailable. (state.change.logger) [data-plane-kafka-request-handler-1]
2021-12-03 04:08:50,820 INFO [Partition user_login_from_api-1 broker=1] Log loaded for partition user_login_from_api-1 with initial high watermark 1 (kafka.cluster.Partition) [data-plane-kafka-request-handler-1]

From the logs it is clear that it is broker 1 is trying to become follower for that topic but failing because the kafka-pod-0 which is the leader is still not in running state I guess.

But after some time i.e. 9:42 AM IST I see below logs which says ISR update to 1,0

2021-12-03 04:12:33,531 INFO [Partition user_login_from_api-1 broker=1] ISR updated to 1,0 and version updated to [23] (kafka.cluster.Partition) [broker-1-to-controller-send-thread]

But the problem is that consumer pods may be up before all the 3 kafka pods are up. Due to which consumer will not run properly.
How to resolve the issue kindly help

[scholzj]

I don't know I'm afraid. You can probably use network policies to block the clients from accessing the brokers before you know they are all up and running. But not much else comes to my mind. Normally, I would expect the clients to simply recover when all brokers are up. But I'm not an expert on the clients to be able to help why they don't.

[shreyasarani23]

ohk got it but I have used liveness and readiness probe to my kafka pods. Only if the readiness probe succeeds the pod will accept the traffic right? And also any work around to avoid the error scenario occurring in the pods?

[scholzj]

Well, but readiness pod passing on one pod does not mean that the topic (its partitions) are ready. I don't tknow what the broker message means -> but in the light of what you are doing it could for example mean that you are starting the broker in different order than they were shutdown and the leader is not ready?

[shreyasarani23]

yes I guess the order of shutting/starting pods might be different because the pods are deployed on different nodes and its up to the node to schedule/shutdown pod. suppose in one of the node only few pods are running and this node consists one kafka pod then this kafka pod will the first kafka pod to be in running state among other kafka pods. By the way I am using pod antiaffinity so that 3 kafka pods gets scheduled on 3 different nodes. And yes initially leader will not be in ready state.