Mismatched datadir volumeMount causing zookeeper dataloss after pod recreation

[Kiritow]

Category:

Kubernetes apps

Type:

• Bug
• Feature Request
• Process

---

We are using the GKE click-to-deploy feature to deploy Kafka in our cluster, but after the underlying nodes are replaced, Kafka fails to start properly and throws the following error:

ERROR Exiting Kafka due to fatal exception during startup. (kafka.Kafka$)
java.lang.RuntimeException: Invalid cluster.id in: /kafka/logs/meta.properties. Expected <redacted>, but read <redacted>

After our investigation, we found that there might be an issue with the Zookeeper configuration in the chart. The datadir volumeMount configuration for Zookeeper is inconsistent with the value of the ZK_DATA_DIR environment variable, causing data loss in Zookeeper after pod migration.

In k8s/kafka/chart/kafka/templates/zk-statefulset.yaml, ZK_DATA_DIR is set to /data but volumeMounts is configured as:

volumeMounts:
- name: config
  mountPath: /config-scripts
- name: datadir
  mountPath: /opt/zookeeper

when we checked the /opt/zookeeper folder inside the pod, we found it was empty.