Strimzi Kafka external PVC [High Availability and Disaster Recovery]

[brunooliveiramac]

Hi everyone,

I'm testing Strimzi Kafka and we are trying to see how to achieve high availability and disaster recovery.
We have recently seen a weird behavior where a cluster was missing and we were only able to have this back by deleting the operator and recreating everything again by Argo.

Based on that we started to see how to ensure that the PVCs didn't get deleted.

Would you know some approaches? For example, using a mirror maker and having two operators in order to switch the traffic in case of an issue?
Or some way to use an external PVC, so if I delete the entire operator I still have the data stored somewhere.

[scholzj]

There are some issues with Argo and PVC deletion - there are some other threads about that.

Using a Mirror Maker to duplicate the Kafka cluster and its data can definitely be used for DR / backup. But it is a fairly expensive way as you need to run a second cluster and in public clouds also the data transfers. There is no simple way to switch traffic between the clusters. You would need to reconfigure all clients to make sure they connect to the backup cluster or use some network infrastructure that would do it for you (including severing existing connections to the old cluster etc.). And assuming you use them as active-passive which would be typical for backup scenarios, there is also no good way how to revert the mirroring flow. You would basically start with a new cluster as a new backup after the switch. So operations-wise, it is not completely straightforward either.

Deleting the operator Deployment itself should not really delete anything. That should happen only when you delete the custom resources (e.g. by deleting them directly or deleting the CRDs which will cause garbage collection of the custom resources). You can to some extent prevent that for example by setting finalizers on the CRDs / CRs. It is up to you to evaluate what is the right measure for your situation and what the scenarios you want to have covered are - I'm just offering this as an option.

[brunooliveiramac]

Oh yeh, that was my mistake. We deleted the whole application, so it deleted the CRD's as you mention.
The issue was related to a node that disappeared, for no reason. We couldn't find any log or events that would explain it. So to have that node back we had to crush everything. We tried to delete the stateful set and I didn't come back also. I think we should have gone for the operator, restarting it.
anyway. I'm trying to patch the CRD using Kustomize, I'm struggling a little but sounds like a good idea.

[brunooliveiramac]

Hi @scholzj, I saw that you are pretty active on the answers, thanks a lot for your help.
I have just one more quick question

I found this answer in one of the topics regarding disaster recovery.
https://github.com/orgs/strimzi/discussions/5894#discussioncomment-1651589
You pointed this doc: https://github.com/orgs/strimzi/discussions/4892#:~:text=https%3A//strimzi.io/docs/operators/latest/full/using.html%23cluster%2Drecovery_str

This is a little outdated, I couldn't find it.
That is what I found, using mirror maker: https://strimzi.io/docs/operators/latest/full/configuring.html#cluster_configuration
Is that what is expected to handle disaster recovery?

[scholzj]

Maybe the is the link that works today? https://strimzi.io/docs/operators/latest/full/deploying.html#cluster-recovery-str

[scholzj]

In general, I do not think Kafka has some super great DR story. Mirror Maker is the only thing inside the Kafka project itself that can help with it. But it is not completely straight forward (and I'm not really an expert on it to be honest)