Zookeeper crash loop after node restart

[tmahring]

Hello

I have a problem with my kafka cluster, running on a self-hosted kubernetes cluster.
Sometimes, when i node gets drained and restarted (due to automatic updates), once it comes back up, zookeeper fails to reconnect to the other zookeeper instances.

Looking at the failing instance i get the following log entries over and over again:

WARN Cannot open channel to 1 at election address flexogrow-kafka-zookeeper-0.flexogrow-kafka-zookeeper-nodes.flexogrow-kafka.svc/10.42.7.19:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=3]-1]

java.net.ConnectException: Connection refused (Connection refused)

On the other instance i can't see any log entries refering to the connection attempt, however, restarting it fixes the problem.

I'm more or less out of ideas on how to fix this, can anyone help me?

Here is the configuration i currently use:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: flexogrow-kafka
spec:
  kafka:
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 200Gi
        deleteClaim: false
    config:
      auto.create.topics.enable: "false"
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
    template:
      pod:
        metadata:
          labels:
            role: kafka
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                  - key: role
                    operator: In
                    values:
                    - kafka
                topologyKey: "kubernetes.io/hostname"
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
    template:
      pod:
        metadata:
          labels:
            role: zookeeper
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                  - key: role
                    operator: In
                    values:
                    - zookeeper
                topologyKey: "kubernetes.io/hostname"
  entityOperator:
    topicOperator: {}
    userOperator: {}