Multi Cluster Deployment in Strimzi

[deshpande-pavan]

Dear Strimzi experts,

Could you please let us know if strimzi supports multi cluster deployment? If yes, could you also refer me to the documentation please?
I tried to search it on official documentation pages, it did not help me.
Thank you so much in advance,

[scholzj]

I think you need to be more exact what does multi cluster deployment mean for you.

[deshpande-pavan]

Dear @scholzj , I could not respond to your response as I wasn't keeping well. I wanted to know if strimzi supports multi region cluster. In our company, we want to deploy kafka over different regions, so I was looking if strimzi kafka operator could support that. Thanks in advance.

[scholzj]

Deploying over multiple regions will always be complicated because many parts of Kafka require reasonably low latency. So that is not just a question of what Strimzi does or does not support. It is in the first place a question whether such architecture makes any sense and whether Kafka itself can handle it. Today, it in general cannot handle multiregion deployments. In the future, who knows, maybe it will be easier.

Today, Strimzi supports deploying only over a single Kubernetes cluster. It does not really care about the number of regions or zones it runs on, but if the latency is not good enough, it will not be stable enough and usable. As a rule of thumb, it the latency is too big to span a single Kubernetes cluster over your regions, it will not be good enough for Kafka anyway.

[deshpande-pavan]

thanks for your response. I will discuss it with my architect and get back to you if we further have any queries.

[deshpande-pavan]

Dear @scholzj , latency and bandwidth are a given in our ecosystem and it is not an issue. So, we have to change our values.yaml file and our kafka custom resource to accomodate two different regions on the same stretched cluster and we would be good to go.
Is my assessment correct?

[scholzj]

Yes, in general, if you have stretch Kubernetes cluster, Strimzi it self does not care about the regions and can deploy Kafka on top of it.

[deshpande-pavan]

Okay, thanks for your response. We don't have a stretch kubernetes cluster right now. We have to create and test these things out. I tried to find different sources to create a stretch kubernetes cluster, but failed to find the correct solution. I am still in the middle of finding it. If you can point me to a resource to create a stretch cluster, that would be of great help.

[scholzj]

I'm sorry, I don't have any resources about stretch Kubernetes clusters or any real experience with them. I know you will similarly to Kafka need a very good latency for it to be stable and performant. But that is it.

…

On Wed, Jul 13, 2022, 03:40 Pavan Deshpande ***@***.***> wrote: Okay, thanks for your response. We don't have a stretch kubernetes cluster right now. We have to create and test this things out. I tried to find different sources to create a stretch kubernetes cluster, but failed to find the correct solution. I am still in the middle of finding it. If you can point me to a resource to create a stretch cluster, that would be of great help. — Reply to this email directly, view it on GitHub <#6968 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABLFOR6IXO3HY5BVATUHXN3VTYNADANCNFSM5ZOXM6NQ> . You are receiving this because you were mentioned.Message ID: <strimzi/strimzi-kafka-operator/repo-discussions/6968/comments/3133470@ github.com>

[DavidNavalho]

Fully aware of that - but the stretched architecture is for Kafka - Confluent notwithstanding. When I mentioned I've done stretched clusters this includes Apache Kafka deployments. You mentioned not having any resources and I just provided two links you could read up on the architecture of a "stretched cluster".

edit: granted you may find Confluent specific stuff in there, but on the topics of a stretched cluster, it's Apache Kafka through-and-through, and that's my point.

[scholzj]

Well, things like cluster linking or replicator are not Apache Kafka things.

Also, I said that I do not have any resources about stretched Kubernetes clusters. Not about stretched Kafka clusters. Strimzi currently operates Kafka clusters within a single Kubernetes cluster. It does not care whether this cluster is stretched over multiple DCs or not. There is some demand for being able to deploy a Kafka cluster over multiple Kubernetes clusters and we have it on the roadmap. But right now, we do not support it -> hence the Kubernetes stretched cluster part. For many users, this is not an issue since they have stretched Kubernetes clusters anyway (across multiple AZs in public clouds which are different DCs etc.).

[DavidNavalho]

Apologies - I probably didn’t make myself very clear and the links likely don’t help without reading them carefully - and maybe some additional context. Stretched clusters require neither replicators nor cluster linking - just wanted to make that clear :)

It’s great to know you have this in the roadmap. Cheers

[belaboros]

Hi David, Jakub and All,

AKS (Azure Kubernetes Service) supports multi-AZ/stretched deployment.
https://learn.microsoft.com/en-us/azure/aks/availability-zones

Strimzi will just see it as one K8S cluster, but you are protected from DC outages.

"Strimzi on multiple K8S clusters" would be a business value for us as well.
We had an incident during a K8S upgrade when somehow the DNS resolution stopped working at K8S level and all the services hosted in that K8S (including our Strimzi) were impacted.

In this case a Strimzi on multiple K8S clusters could be useful.

[scholzj]

That kind of stretch cluster is tracked in #3697