How to build a cloud agnostic multi-node cluster with external access ?

[ViniciussSantos]

I'm building a project with Kafka and K8s where I need to deploy a Kafka cluster and let applications outside of the cluster send messages to it so applications inside of the cluster can consume and store it. In this project, we have a bunch of IoT devices from multiple companies producing a lot of data and all this data needs to be processed. The architecture is as follows:

• An MQTT broker gathers all the data from the devices and publishes to a topic in the cluster.

• Because of the nature of the problem (Multiple companies with different cloud providers), the MQTT broker lives outside of our Kafka cluster.

• So I need to deploy a load balancer or ingress service that allows external applications to publish messages.

(In this example I gave, there is only one MQTT broker, but in production I will be working with multiple MQTT brokers.)

For development purposes, I started prototyping by running a cluster locally using vagrant, virtualbox, kubeadm and flannel. This worked initially, until I found out that kubeadm doesn’t offer a load balancer service (type: loadbalancer doesn’t work, the external IP stays pending indefinitely), so I did some research and found out that maybe K3s could solve some of my problems or a could deploy a custom load balancer and make it work with kubeadm.

So I have two questions:

• Given my problem, which would work best with Strimzi and Kafka, a load balancer or an Ingress service ?

• From what I have seen, Strimzi doesn’t work well with K3s, so should I stay in kubeadm and add an ingress/load balancer or moving to K3s is the best choice here ?

[scholzj]

Given my problem, which would work best with Strimzi and Kafka, a load balancer or an Ingress service ?

I'm not sure if there is a single thing that works everywhere. That is not really a Strimzi issue but a Kubernetes issue. All of them have some limitations caused by your infrastructure and environment.

NodePorts are probably the least dependent on the outside infrastructure. But even they need your nodes to be publicly accessible which is what many users don't want. Load balancers are fine in public clouds. But might not be available outside of them. I personally use MetalLB for load balancers in my home cluster and I'm happy with it - but I have a physical cluster and not Vagrant / VirtualBox VM, so not sure how well it works there. Ingress is fine, but it basically requires node port or load balancer as well (just not so many of them, which is sometimes an advantage). There are also other technologies such as Skupper that can link various clusters / networks. But they will always require some loadbalancer / nodeport / ingress to bootstrap.

I personally think that using various mechanisms in various places is relatively simple and would simply use the suitable mechanism for given environment. E.g. node ports for local development, load balancers in public cloud etc. For your client based Kafka applications (the MQTT brokers) the address will differ anyway. And changing the type in the Kafka custom resource on the Strimzi side is normally reasonably easy.

From what I have seen, Strimzi doesn’t work well with K3s, so should I stay in kubeadm and add an ingress/load balancer or moving to K3s is the best choice here ?

I'm not aware of any limitations. I was running Strimzi on a K3s cluster for almost 6 months recently without any major issues.

[ViniciussSantos]

I personally use MetalLB for load balancers in my home cluster and I'm happy with it - but I have a physical cluster and not Vagrant / VirtualBox VM, so not sure how well it works there.

As far as I know, there shouldn't be any kind of big difference between running it in a VM vs bare metal. Also, I probably didn't convey it clearly, but my main objective is to create an on-premise solution, that's why I decided to set up a Vagrant / VirtualBox environment because it's the closely I could come to a "production" environment without having to use a bunch of physical machines, but much of the content/docs/info out there regarding external access using load balancers/ingress is very cloud-centric, so I decided to share my problem to see if anyone could help me. If MetalLB works for your home cluster, maybe it will work for me.

There are also other technologies such as Skupper that can link various clusters / networks.

Skupper seems nice, but I, unfortunately, don't know if those companies are running their infra on k8s or if they would be willing to move it to k8s if not (I'm assuming Skupper only works between k8s clusters).

I personally think that using various mechanisms in various places is relatively simple and would simply use the suitable mechanism for given environment. E.g. node ports for local development, load balancers in public cloud etc.

This is what I have been doing, while I was testing it locally, I used node ports, but when I decided to start building something more robust, I started having the problems I described in my OP.

I'm not aware of any limitations. I was running Strimzi on a K3s cluster for almost 6 months recently without any major issues.

I saw some relatively old issues talking about some problems using Strimzi with K3s but I couldn't trace down anything more. Happy to know that it works perfectly with K3s.

I'm under the impression that my best possible course of action is to set up MetalLB, what do you think?

[scholzj]

Yeah, I think It should be worth trying MetalLB. I'm happy with it in my home cluster that I use for development and testing. But to be clear, I have never done any performance testing or anything like that on it. So I have no idea if you run into any scale issues etc.