Kafka upgrade issue 2.8.0 -> 3.0.0 (zookeeper failures)

[edgarkz]

Hello,
We are running into zookeeper issues while upgrading kafka using the operator from 2.8.0 -> 3.0.0
Our setup includes 3 zookeepers and 3 kafka brokers with below config (trimmed)

  kafka:
    version: 2.8.0
    replicas: 3
  zookeeper:
    replicas: 3
    resources:
      requests:
        memory: 1Gi
        cpu: 200m
      limits:
        memory: 1Gi
        cpu: 200m
    storage:
      type: persistent-claim
      size: 10Gi
      deleteClaim: false
 config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      min.insync.replicas: 2
      log.message.format.version: "2.8"
      inter.broker.protocol.version: "2.8"
      auto.create.topics.enable: "false"
      message.max.bytes: 5242880

While upgrading the kafka version to 3.0.0:

1. stimzi operator detects zookeper vesion upgrade is required and starts rolling restarts,
   1 zookeeper instance was upgraded fine,
   2 zookeeper instance starts and eventually throws the below log and keeps restarting:

2024-01-07 07:48:40,934 INFO Using org.apache.zookeeper.server.watch.WatchManager as watch manager (org.apache.zookeeper.server.watch.WatchManagerFactory) [main]
2024-01-07 07:48:40,934 INFO Using org.apache.zookeeper.server.watch.WatchManager as watch manager (org.apache.zookeeper.server.watch.WatchManagerFactory) [main]
2024-01-07 07:48:40,936 INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase) [main]
2024-01-07 07:48:40,936 INFO zookeeper.commitLogCount=500 (org.apache.zookeeper.server.ZKDatabase) [main]
2024-01-07 07:48:41,233 INFO Using TLS encrypted quorum communication (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
2024-01-07 07:48:41,233 INFO Port unification disabled (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
2024-01-07 07:48:41,233 INFO multiAddress.enabled set to false (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
2024-01-07 07:48:41,234 INFO multiAddress.reachabilityCheckEnabled set to true (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
2024-01-07 07:48:41,234 INFO multiAddress.reachabilityCheckTimeoutMs set to 1000 (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
2024-01-07 07:48:41,234 INFO QuorumPeer communication is not secured! (SASL auth disabled) (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
2024-01-07 07:48:41,234 INFO quorum.cnxn.threads.size set to 20 (org.apache.zookeeper.server.quorum.QuorumPeer) [main]
2024-01-07 07:48:41,235 INFO Reading snapshot /var/lib/zookeeper/data/version-2/snapshot.600026f9b (org.apache.zookeeper.server.persistence.FileSnap) [main]
2024-01-07 07:48:42,535 WARN Got EOF exception while reading the digest, likely due to the reading an older snapshot. (org.apache.zookeeper.server.DataTree) [main]

Eventually after some restarts all 3 zookeeper instances enter unhealthy state and keep restarting

Can you please advise how to proceed from here?

Zookeeper data directory1:

2#

3#

Strimzi operator version: 0.27.1
K8s EKS 1.24

[edgarkz]

closed - new issue started as bug.
#9527

[scholzj]

Sorry, but I think you should close the bug and keep this open. Even if it is a Strimzi bug, what do you think happens with a bug in a 2-year-old release with Kafka versions not supported for a long time?

[edgarkz]

@scholzj Sure, done.
I am looking for ideas on how to handle this problem. I would appreciate any suggestions on how to proceed.

[scholzj]

I'm afraid I have no idea what this means. The log snippet you shared doesn't even show any errors or anything like that.

[edgarkz]

Zookeeper just constantly restarts once reaching snapshot loading state - the below is the latest log line

2024-01-07 07:48:41,235 INFO Reading snapshot /var/lib/zookeeper/data/version-2/snapshot.600026f9b (org.apache.zookeeper.server.persistence.FileSnap) [main]
2024-01-07 07:48:42,535 WARN Got EOF exception while reading the digest, likely due to the reading an older snapshot. (org.apache.zookeeper.server.DataTree) [main]

[edgarkz]

we ended up by completely cleaning up zookeeper data and have performed upgrades without old snapshots..