Strimzi Kafka failing to start up after server issue

[JaswanthP6878]

We have a custom storage PVC setup that went offline for >30 minutes. Then the kafka cluster is failing to start properly. not facing this issue with any other solution.

kafka.yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-cluster
spec:
  kafka:
    version: 3.7.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
    config:
      auto.offset.reset: "latest"
      num.network.threads: 18
      num.io.threads: 8 
      num.partitions: 45
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      log.retention.minutes: 60
      log.roll.ms: 1200000
      delete.topic.enable: true
      offsets.retention.minutes: 60
      log.segment.minutes: 60
      inter.broker.protocol.version: "3.7"
    storage:
      type: persistent-claim
      size: 1Ti
      deleteClaim: true
      class: csi-rdma-sc-xfs
    template:
      pod:
        affinity:
          nodeAffinity:
            preferredDuringSchedulingIgnoredDuringExecution: 
            - weight: 60
              preference:
                matchExpressions:
                - key: kubernetes.io/hostname
                  operator: In
                  values:
                  - p8.assetsense.com
            - weight: 40
              preference:
                matchExpressions:
                - key: kubernetes.io/hostname
                  operator: In
                  values:
                  - p7.assetsense.com
  zookeeper:
    replicas: 3
    config:
      jute.maxbuffer: 0x9ffffff
    storage:
      type: persistent-claim
      size: 20Gi
      deleteClaim: true
      class: csi-rdma-sc-xfs
    template:
      pod:
        affinity:
          nodeAffinity:
            preferredDuringSchedulingIgnoredDuringExecution: 
            - weight: 60
              preference:
                matchExpressions:
                - key: kubernetes.io/hostname
                  operator: In
                  values:
                  - p8.assetsense.com
            - weight: 40
              preference:
                matchExpressions:
                - key: kubernetes.io/hostname
                  operator: In
                  values:
                  - p7.assetsense.com
  entityOperator:
    topicOperator: {}
    userOperator: {}

Error logs :
kafla logs
2025-08-16 04:54:25,217 WARN Session 0x0 for server kafka-cluster-zookeeper-client/10.101.123.30:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn) [main-SendThread(kafka-cluster-zookeeper-client:2181)] EndOfStreamException: channel for sessionid 0x0 is lost at org.apache.zookeeper.ClientCnxnSocketNetty.doTransport(ClientCnxnSocketNetty.java:286) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289) 2025-08-16 04:54:26,745 INFO Opening socket connection to server kafka-cluster-zookeeper-client/10.101.123.30:2181. (org.apache.zookeeper.ClientCnxn) [main-SendThread(kafka-cluster-zookeeper-client:2181)]

zookeeper logs:
2025-08-16 05:00:18,236 INFO Created server with tickTime 2000 ms minSessionTimeout 4000 ms maxSessionTimeout 40000 ms clientPortListenBacklog -1 datadir /var/lib/zookeeper/data/version-2 snapdir /var/lib/zookeeper/data/version-2 (org.apache.zookeeper.server.ZooKeeperServer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)] 2025-08-16 05:00:18,236 ERROR Couldn't bind to kafka-cluster-zookeeper-0.kafka-cluster-zookeeper-nodes.kafka.svc/:2888 (org.apache.zookeeper.server.quorum.Leader) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)] java.net.SocketException: Unresolved address at java.base/java.net.ServerSocket.bind(ServerSocket.java:380) at java.base/java.net.ServerSocket.bind(ServerSocket.java:342) at org.apache.zookeeper.server.quorum.Leader.createServerSocket(Leader.java:322) at org.apache.zookeeper.server.quorum.Leader.lambda$new$0(Leader.java:301) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base/java.util.concurrent.ConcurrentHashMap$KeySpliterator.forEachRemaining(ConcurrentHashMap.java:3573) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) at org.apache.zookeeper.server.quorum.Leader.(Leader.java:304) at org.apache.zookeeper.server.quorum.QuorumPeer.makeLeader(QuorumPeer.java:1340) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1551) 2025-08-16 05:00:18,237 WARN Unexpected exception (org.apache.zookeeper.server.quorum.QuorumPeer) [QuorumPeermyid=1(secure=[0:0:0:0:0:0:0:0]:2181)] java.io.IOException: Leader failed to initialize any of the following sockets: [kafka-cluster-zookeeper-0.kafka-cluster-zookeeper-nodes.kafka.svc/:2888] at org.apache.zookeeper.server.quorum.Leader.(Leader.java:307) at org.apache.zookeeper.server.quorum.QuorumPeer.makeLeader(QuorumPeer.java:1340) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1551)

anyone has any idea why this is occuring. Thank you.

[scholzj]

Unresolved address looks like some networking / DNS issue. But hard to tell what exactly to look into. ZooKeeper had also some known issues with re-resolving DNS etc. So restarting the ZooKeeper Pods manually might help (just delete them all with kubectl delete pod ...). The Kafka error seems to be just with connecting to ZooKeeper which is exoected issue if ZooKeeper does not form a quorum.

[JaswanthP6878]

Thank you. After deleting pods, it is able to continue to running state, but are there any mechanisms/configurations for this to resolve automatically without manually restarting the pods?

[scholzj]

I don't think so. You can of course move to KRaft to get rid of ZooKeeper. But that likely has its own issues (that said, as ZooKeeper is gone from Kafka, you will need to do it sooner or later).