Kafka Mirroring between on prem and strimzi kafka in cloud

[seema-sandeep-jaisinghani-db]

Hi team,

We have requirement to move our on prem kafka data like topics, messages, acl etc to strimzi kafka on cloud.
Is it possible any how?
Has someone tried to move data from on prem linux machines to kafka setup on cloud.
Lets us know as we are almost in last phase of decision to move to strimzi kafka.
Appreciate any help!

[scholzj]

To move the messages, you can use for example Mirror Maker 2. That should be fairly straight forward. If you wanna keep the original topic names, you can use the IdentityReplicationPolicy which will make sure they will not be renamed. It will also mirror some topic configurations but not all of them => for example replication factor is not copied exactly IIRC since it does not know how many nodes there are in the future cluster etc.

Things such as ACLs and users are a bit more complicated. Strimzi has user operator to manage users and ACLs. If you would want to use it, you will need to prepare the KafkaUser custom resources for them which will deal with the authentication and ACLs. You could also disable it and just do things directly in Kafka using the Kafka APIs. In that case the Mirror Maker 2 should be able to also sync the ACLs.

[seema-sandeep-jaisinghani-db]

@scholzj Thank you for inputs!
In our case source cluster is not the setup on cloud, its on prem setup on one of the machines. And our target kafka cluster would be in GCP using strimzi.

WIll Mirror maker 2.0 help in this situation? as their documents refer to cluster endpoints for source.

[scholzj]

Mirror Marker does not really care about where do the cluster runs. But it of course needs to be able to connect to it. So you need to make sure that your old cluster is accessible from the cloud (or if that is not the option, you can also do it the other way => make the new cluster accessible and run the Mirror Maker 2 in your old environment).