External Listener

[yasin-amadmia-mck]

So we running Kafka Strimzi operator within AKS. We want to connect to this kafka from our on-premise network too (all network etc is configured).
What we configured is an external Listener of type loadbalancer but with an annotation of Azure Internal IP. This works fine.

      - name: external
        port: 9095
        type: loadbalancer
        tls: false
        authentication:
          type: scram-sha-512
        configuration:
          bootstrap:
            annotations:
              service.beta.kubernetes.io/azure-load-balancer-internal: "true"
              external-dns.alpha.kubernetes.io/hostname: kafka.aks.cloud
          brokers:
          - broker: 0
            annotations:
              service.beta.kubernetes.io/azure-load-balancer-internal: "true"
              external-dns.alpha.kubernetes.io/hostname: kafka-broker-0.aks.cloud
          - broker: 1
            annotations:
              service.beta.kubernetes.io/azure-load-balancer-internal: "true"
              external-dns.alpha.kubernetes.io/hostname: kafka-broker-1.aks.cloud
          - broker: 2
            annotations:
              service.beta.kubernetes.io/azure-load-balancer-internal: "true"
              external-dns.alpha.kubernetes.io/hostname: kafka-broker-2.aks.cloud

The above creates 4 services. An external-0,1,2 - One per broker, and one external-bootstrap.

Question:
(1) What is the use of the external-0,1,2 services ?
(2) When connecting from on-premise for example, we should use the external-bootstrap service and NOT the external-0,1,2 right ?

[scholzj]

Kafka requires clients to have direct access to each of the Kafka brokers which are part of the clusters. So the bootstrap service is used by the clients to get the metadata for the Kafka cluster and the per-broker services will be used by the clients to access directly the brokers hosting the partition replica they want to connect to.

In the clients, you should probably either configure the DNS name you are trying to use (kafka.aks.cloud) or the address of the bootstrap loadbalancer. When you do kubectl get kafka -o yaml you should see the bootstrap address of the different listeners in the status section.

One thing to note ... the configuration you have right not above will give the per-broker loadbalancers the DNS names kafka-broker-X.aks.cloud. But these will not be really used. If you would want them to be used by the clients, you should also configure the advertisedHost fields: https://strimzi.io/docs/operators/latest/full/using.html#type-GenericKafkaListenerConfigurationBroker-reference

[yasin-amadmia-mck]

Thanks @scholzj
So If i don't use the advertisedHost, what is used by default ? The EXTERNAL-IP I suppose (what i get when i do kubectl get service. ?

[scholzj]

Yeah, it will be the IP or DNS of the loadbalancer (TBH I cannot remember now if AKS uses DNS or IP as the default one).