Migration of zookeeper data

[zawistowskipiotr]

Hi. We are running 5-kafka nodes and 3-zookeeper nodes cluster using strimzi (on AKS). Kafka nodes storage is set up correctly using azuredisk, but zookeeper uses azurefile storageclass. Snapshots and logs on zookeeper are being crashed from time to time and we want to migrate our zookeeper to azuredisk storage. Since jbod storage is not provided for zookeeper - it is hard to copy the data and migrate successfully.

What path should we take to do the migration (without losing data 😊)?

[scholzj]

I'm afraid that is not completely simple. Assuming you want to avoid downtimes, I guess there are two ways how to do it. The first one is:

• Stop the operator / pause the reconciliation of the Kafka CR
• Do a non-cascading delete of the StatefulSet / StrimziPodSet (depends on which Strimzi version you use)
• Delete the first pod and its PVC
• Create a new PVC with the right storage class
• Start the operator / unpause the reconciliation
• Wait until it recreates the pod and until the data resync
• Start from the beginning and proceed with the next step

The second option might be to:

• Edit the Kafka custom resource and change the storage class
• Edit the StrimziPodSet / StatefulSet (depending on which Strimzi version you use) and change the storage class in the strimzi.io/storage annotation as well
• Delete the first pod with its PVC (there is a Strimzi annotation you can use to tell the operator to do it)
• Wait until it is recreated with a new storage class and resyncs from the other nodes
• Proceed with the next ZooKeeper pod etc.

You should try both of them first in some test environment to double check they work. Its not something what is done / tested regularly.