external access - ingress question

[tirelibirefe]

Hello,
I've a EKS Kubernetes, Strimzi-Kafka and I'm working on external-access of kafka.
I've read @scholzj tutorials about external access.
That one works very well but cons were already listed by Jakub.

I know only Nginx was tested but I would like use a different ingress controller (Tyk).

I need to understand a few things about Kafka before start, as I am a newbie about Kafka.

In order to pass data to Kafka, we pass "--bootstrap-server" parameter like "-b mykafkacluster-kafka-bootstrap.kafka.svc:9092"

I see brookers were exposed too. Also each brooker was exposed individually.

Questions:

1. Why do we need to expose brookers outside, is exposing them outside necessary to send data to Kafka?
2. Why do we need expose these brookers individually? Why don't we load-balance them as all-in-one, as single entrypoint?

Thanks & Regards

[scholzj]

Why do we need to expose brookers outside, is exposing them outside necessary to send data to Kafka?

Yes.

Why do we need expose these brookers individually? Why don't we load-balance them as all-in-one, as single entrypoint?

Have you read the all parts of the blog post series you linked above? That explains how the Kafka protocol works and why this is needed.

[tirelibirefe]

Sorry, I admit , I have missed reading some parts.
..but I read "introduction". Thanks

[tirelibirefe]

...I changed my mind; I followed your LoadBalancer walkthrough and completed external-access. Thank you!

[cferrera]

Hey, sorry to bringing this discussion up - but since it's exactly the same issue I thought would be good to avoid a duplicated topic;

I've followed the same tutorial as described in the topic description (ingress access) and after so many tries I don't know anymore what I'm doing wrong.

I'm trying to deploy it on AWS EKS. This is the basic config.

kafka:
  replicas: ${kafka.brokers}
  listeners:
    - name: plain
      port: 9092
      type: internal
      tls: false
    - name: external
      port: 9094
      type: ingress
      tls: true
      configuration:
        createBootstrapService: true
        class: nginx
        bootstrap:
          host: "kafka-bootstrap.${clusterDomain}"
          annotations:
            nginx.ingress.kubernetes.io/ssl-passthrough: "true"
            external-dns.alpha.kubernetes.io/hostname: "kafka-bootstrap.${clusterDomain}"
        brokers: 
  %{~ for idx, broker in range(kafka.brokers) ~}
          - broker: ${idx}
            host: "kafka-${idx}.${clusterDomain}"
            advertisedHost: "kafka-${idx}.${clusterDomain}"
            advertisedPort: 9094
            annotations:
              external-dns.alpha.kubernetes.io/hostname: "kafka-${idx}.${clusterDomain}"
              ingress.kubernetes.io/ssl-passthrough: true
              nginx.ingress.kubernetes.io/backend-protocol: HTTPS
              nginx.ingress.kubernetes.io/ssl-passthrough: true
  %{~ endfor ~}

I have external-dns installed to manage the records on Route53 and nginx as ingress controller. The weird stuff is at Route53 the values for my dns records are pointing to my service cluster IP instead of to my nginx load balancer dns.

Already tried openssl and hangs forever

openssl s_client -connect kafka-bootstrap.{domain}:443 -showcerts

Also tried console-producer example + the keytool -import - always give a disconnected after a few seconds

./bin/kafka-console-producer.sh --broker-list kafka-bootstrap.{domain}:443 --producer-property security.protocol=SSL --producer-property ssl.truststore.password=password --producer-property ssl.truststore.location=./truststore.jks --topic teste
>[2022-03-30 14:50:54,324] WARN [Producer clientId=console-producer] Bootstrap broker kafka-bootstrap.{domain}:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

* Internally, using cluster dns (svc.cluster.local) everything works just fine

Any suggestion or guidance will be really appreciate, I don't have more ideas to try.
Thank you!
FF.

[scholzj]

Already tried openssl and hangs forever

The Ingress works on TLS-SNI. So you cannot just specify the connect part. You will also need to pass the TLS-SNI hostname (I think the option is -servername, but not 100% out of my head) and the value should be kafka-bootstrap.{domain}. That should normally get you at least some certificates which you can share back here. If it doesn't give you anything, it might not be connecting to the ingress I guess or your Ingress has somehow disabled TLS I guess. Whether it exists or hangs after printing the certificates might not matter that much - the certifictes might be useful.

[scholzj]

(You should also doublecheck that the operator properly applied these changes and that there are no errors in its logs and the Kafka custom resource is ready.)

[cferrera]

1. tried with -servername and nothing changed
2. checked the kafka-operator and there are no complains

...
2022-04-01 12:27:49 INFO  ClusterOperator:123 - Triggering periodic reconciliation for namespace kafka-operator
2022-04-01 12:27:49 INFO  ClusterOperator:123 - Triggering periodic reconciliation for namespace kafka
2022-04-01 12:27:49 INFO  AbstractOperator:226 - Reconciliation #1437(timer) Kafka(kafka/kafka-cluster): Kafka kafka-cluster will be checked for creation or modification
2022-04-01 12:27:51 INFO  AbstractOperator:517 - Reconciliation #1437(timer) Kafka(kafka/kafka-cluster): reconciled

3. Custom resource is ready

NAME            DESIRED KAFKA REPLICAS   DESIRED ZK REPLICAS   READY   WARNINGS
kafka-cluster   3                        1                     True    True

* the warnings are not related to the issue (log format, replication factor and ISR)
 

If it doesn't give you anything, it might not be connecting to the ingress

I think this is the tricky part - that I've commented before - looks like the operator is configuring external-dns with the service ClusterIP instead of nginx's DNS. Does it make any sense?

[scholzj]

I think this is the tricky part - that I've commented before - looks like the operator is configuring external-dns with the service ClusterIP instead of nginx's DNS. Does it make any sense?

TBH, I do not know how ExternalDNS finds out the Ingress address to configure for the DNS records - maybe from the Ingress resources status? But I'm not sure - for me it always just worked out of the box, so I never diged any deeper into the EternalDNS details. Maybe its some External DNS bug? Strimzi is not really involved in the DNS process. It just takes the annotations from the Kafka CR and passes them to the Ingress resource. That is where ExternalDNS would pick them up and does whatever it does. But I guess such issue would explain what you describe => the operator doesn't know about this and will be happy. But from outside you will be connecting somewhere else.

[cferrera]

@scholzj
I would like to thank you for all your replies 🙏
After a LOT of tries and digging I was able to come with a solution that I would like to share here.

External DNS

By default ExternalDNS watches Ingresses and Services and prioritises Service's Annotations it means that if you are using an ingress controller like traefik or nginx it will use the service's IP instead of the controller's LoadBalancer IP.
If you don't want to change ExternalDNS configs to watch only ingress, you can "force" using ingress annotations like these:

# this will override some annotations for bootstrap and pod services allowing externalDNS to pick up ingress configs;
kafka:
  ...
  template:
    externalBootstrapService:
      metadata:
        annotations:
          external-dns.alpha.kubernetes.io/exclude: 'true'
          external-dns.alpha.kubernetes.io/hostname: ""
    perPodService:
      metadata:
        annotations:
          external-dns.alpha.kubernetes.io/exclude: 'true'
          external-dns.alpha.kubernetes.io/hostname: ""
  ...

Hope it helps someone in the future 🤞

[cferrera]

@scholzj
I'm using Traefik as my ingress controller and I have some specific issue with the ingress and the operator.
To make tlsPassthrough on Traefik I need to create a router and an ingress like these:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: "kafka-bootstrap"
  namespace: ${local.namespace.name}
  annotations:
    kubernetes.io/ingress.class: traefik
  labels:
    app: kafka-strimzi
spec:
  entryPoints:
    - websecure
  routes:
  - match: HostSNI(`kafka-bootstrap.${var.cluster_domain}`)
    kind: Rule
    services:
    - name: kafka-cluster-kafka-external-bootstrap
      port: 9094
  tls:
    passthrough: true
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kafka-cluster-kafka-0
  namespace: ${local.namespace.name}
spec:
  rules:
    - host: kafka-0.experimental
  tls: []

but the operator is generating an Ingress like this:

spec:
  ingressClassName: traefik
  rules:
  - host: kafka-0.experimental
    http:
      paths:
      - backend:
          service:
            name: kafka-cluster-kafka-0
            port:
              number: 9094
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - kafka-0.experimental

Traefik will not work if the ingress has tls and http configuration (they need to be managed by IngressRouteTCP).

Is there a way to remove those configurations from ingress? I've tried to apply a patch but the reconciliation will rollback my changes.

[scholzj]

Strimzi supports Ingress API and the support is developed and tested with Kubernetes Nginx Ingress controller. It uses the TLS passthrough feature to get the Kafka TCP traffic through Ingress. If you want to use the type: ingress listener, you will need to use this pattern. Unfortunately, the Ingress API does not offer very good compatibility between controllers. So I do not know if this works with Traefik or not. In any case, I do not think we are planning to add any support for any custom resources Traefik is using.

What you can do is use an internal listener and manage the infrastructure your self -> so you can create your IngressRouteTCP as you need and you can configure the advertised listener addresses in the Kafka CR in the listener's configuration section. That should make it possible to use pretty much whatever you want.