Migrating clusters to k8s

[jiahuijiang]

We are doing a migration for an existing Kafka stack to be running in k8s. The hope is to do the migration live and automatic without shutting down applications using Kafka, because we need to do this migration with many clients for multiple application each.

We have looked into using MirrorMaker, but MirrorMaker doesn’t guarantee persisting offsets across clusters. Even with KIP-545, there is only support for consumers if they are using consumer group and have Kafka tracks the consumer offset.

Unfortunately our application track offset internally instead, and use them very heavily for job progress and replay purpose. So migrating clusters using MirrorMaker means we have to do a lot of migration and translating in the application to keep it running correctly.

So we started to look into whether it’s possible to do a Mitosis style migration - adding some nodes in k8s, increase replicate factor, wait for data to be replicated, drop old nodes.
But since the nodes running in k8s are managed by Strimzi while the rest are not, we are see several challenges in this approach:
(1) TLS: for the internal communication between Strimzi brokers, it’s using certs generated by Strimzi operator, which our other nodes won’t have access to.
(2) It looks like Strimzi loads the list of brokers from zookeeper, what’s going to happen if some of them do not run in k8s? Is it possible to support a set of ‘unmanaged brokers’ that Strimzi will ignore the lifecycle of?
(3) Strimzi uses different port for control plane and replication listener than Kafka defaults. Will there be any issue when some of the brokers are using a different config than others?

Given the complexity, is there any other way to migrate the clusters and keep the record offsets?

Thank you!

[scholzj]

So we started to look into whether it’s possible to do a Mitosis style migration - adding some nodes in k8s, increase replicate factor, wait for data to be replicated, drop old nodes.

This is not supported by Strimzi and to get something like that work for your migration would require quite a lot of hacking and lot of things you would need to figure out. Leaving that aside, if you wanna give it a try below are the answers for the sub-questions.

(1) TLS: for the internal communication between Strimzi brokers, it’s using certs generated by Strimzi operator, which our other nodes won’t have access to.

Well, the certificates are in Secrets. So you can always extract them and use them elsewhere. It is just a certiicate.

(2) It looks like Strimzi loads the list of brokers from zookeeper, what’s going to happen if some of them do not run in k8s? Is it possible to support a set of ‘unmanaged brokers’ that Strimzi will ignore the lifecycle of?

Well, this is not Strimzi. This is standard Kafka thing. It always works like that (well, the KIP-500 / Kraft mode changes that of course, but that is not production ready and not supported by Strimzi). For this kind of migration, you would probably need to first somehow move your existing Kafka cluster to use the Strimzi ZooKeeper - otherwise you would not be able to have it as one cluster. But I have no idea how to merge two ZooKeeper clusters with completely different data into one.

(3) Strimzi uses different port for control plane and replication listener than Kafka defaults. Will there be any issue when some of the brokers are using a different config than others?

AFAIK the information about how to connect to the other brokers is shared through ZooKeeper. So that might not be a problem if you figure out how to share the ZooKeeper cluster. In the worst case, I guess you can always reconfigure your existing cluster to use different ports. But I don't think it would be needed.

[jiahuijiang]

Thank you so much for the fast response! We will look into the certs part.
For 'merging zookeeper', we were hoping to do a similar style of 'adding new nodes in k8s, remove original nodes'. We will prototype this out and see what issues come up :)

[scholzj]

For 'merging zookeeper', we were hoping to do a similar style of 'adding new nodes in k8s, remove original nodes'. We will prototype this out and see what issues come up :)

That will be quite hard. ZooKeeper needs to have the quorum configured by listing all the Zoo nodes. The operator checks the configuration during the reconciliation. So I do not think you would be able to add any nodes to the Strimzi Zoo cluster or add these nodes to some other cluster without some temporary code modifications to the operator.

---

Another problem which comes to my mind is actually about the replication in the Kafka cluster. While the different ports might not matter, the Strimzi Kafka clusters uses internal DNS names of the Kube cluster as advertised hostnames. It would be hard to connect to those from the outside nodes. You would again either need to modify the code or do some hacking around the networking or something.

[vishwa35]

ZooKeeper needs to have the quorum configured by listing all the Zoo nodes.

We could modify these config files as we add/remove nodes (even if this requires hacking the operator to skip/adjust checking this for the period of the migration, as you mentioned). Does that sound reasonable to do?

the Strimzi Kafka clusters uses internal DNS names of the Kube cluster as advertised hostnames

similarly, if we modded the operator to use external hostnames this would work?

---

Currently am heavily considering going down this path, even if it requires modding the strimzi operator.

• would this be something worth contributing back in any fashion?
• do you see any hard blockers here, or are they just inconveniences?

[scholzj]

Well, give it a try if you want ... but I think it is not as trivial as it sounds.

• would this be something worth contributing back in any fashion?
• do you see any hard blockers here, or are they just inconveniences?

I think contributing it in any way would be quite hard as all of these things will be alternatives to existing code. It will be hard to maintain and hard to test etc. And TBH, most users I know would simply go with Mirror Maker if they need the cluster with maximum availability or just copy around the data if they can take the cluster down for some time.

But if you go for it, your best chance is probably just do some dirty replacements for the changes you need -> just delete old methods and replace with new methods doing what you want etc. Anything else would be IMHO too complicated.

[Manicben]

We have the same problem, however our Kafka clusters are already on k8s.
If Strimzi supported external Zookeeper clusters, then it may be possible to point both Strimzi brokers and old brokers to the same Zookeeper and use Cruise Control to help move over partitions, or increase replication factor temporarily. This of course depends on the brokers not sharing IDs (which for us isn't the case, due to us also using a StatefulSet, so we're kinda stuck 🙃 ).

However, it does seem like this is a no, with perfectly valid reasons (KRaft ftw!). Merging Zookeeper clusters would also be very hard as @scholzj has mentioned.
This means that it wouldn't really be possible, e.g. by using MirrorMaker2.0, to have a smooth migration without some downtime, potential data loss or consistency issues, especially for our customers connecting externally. In addition, it would be beneficial to migrate from the old Zookeeper to the one Strimzi deploys, for its added benefits such as mTLS, and I don't see an easy way of doing that without some downtime either.

However for most people, if they can tolerate some downtime and data loss, MirrorMaker2.0 would be the way to go.
Once replication is mostly up-to-date and during a quiet window (use metrics to determine this), use ACLs or networking to stop all clients from producing/consuming to/from the old cluster (will cause some downtime and client-side errors), then point clients to the new cluster and rolling restart them. Any uncommitted messages would be picked up by consumers again and reprocessed; ideally all applications are able to retry and are idempotent. Any in-flight messages to the old brokers before the switch might be lost, so this might need extra planning around if it matters, especially if produced from external clients you do not control.
There's most likely other measures to help minimise downtime and data loss, such as only stopping producers with ACLs and waiting for consumer lag to go to 0 (however, some of our consumers produce to downstream topics before committing, so that approach wouldn't work).
This is at least my understanding of how it would be done in that scenario, if that helps anyone 😃 . Would be interested to hear how others have achieved the migration to Strimzi.

[lenin-joseph]

@jiahuijiang @vishwa35 @Manicben, I'm just curious to know if you were able to achieve the migration from apache-kafka to strimzi without downtime.

I am looking for options to migrate the apache-kafka cluster to strimzi(by cloning PVC) without MM2, so I was wondering if any of you were able to succeed with your implementation.

Thank you