How to configure loadbalancer type in AWS with lets-encrypt certificate

[jslusher]

I have the following configuration for my scram listener:

    listeners:
    - authentication:
        type: scram-sha-512
      configuration:
        bootstrap:
          alternativeNames:
          - strimzi-kafka-broker-0.staging.company.com
          - strimzi-kafka-broker-1.staging.company.com
          - strimzi-kafka-broker-2.staging.company.com
          annotations:
            external-dns.alpha.kubernetes.io/hostname: strimzi-kafka-bootstrap.staging.company.com
            external-dns.alpha.kubernetes.io/ttl: "60"
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-type: external
        brokerCertChainAndKey:
          certificate: tls.crt
          key: tls.key
          secretName: strimzi-kafka-tls
        brokers:
        - annotations:
            external-dns.alpha.kubernetes.io/hostname: strimzi-kafka-broker-0.staging.company.com
            external-dns.alpha.kubernetes.io/ttl: "60"
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-type: external
          broker: 0
        - annotations:
            external-dns.alpha.kubernetes.io/hostname: strimzi-kafka-broker-1.staging.company.com
            external-dns.alpha.kubernetes.io/ttl: "60"
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-type: external
          broker: 1
        - annotations:
            external-dns.alpha.kubernetes.io/hostname: strimzi-kafka-broker-2.staging.company.com
            external-dns.alpha.kubernetes.io/ttl: "60"
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-type: external
          broker: 2
      name: scram
      port: 9096
      tls: true
      type: loadbalancer

I'm trying to get a avro-schema-registry cluster to connect to it by configuring its bootstrap setting like this:

        - name: SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS
          value: SASL_SSL://strimzi-kafka-broker-0.staging.company.com:9096,strimzi-kafka-broker-1.staging.company.com:9096,strimzi-kafka-broker-2.staging.company.com:9096

The certificate specified in brokerCertChainAndKey looks like this:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: strimzi-kafka-sdlc
spec:
  secretName: strimzi-kafka-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
    group: cert-manager.io
  dnsNames:
  - strimzi-kafka-bootstrap.staging.company.com
  - strimzi-kafka-broker-0.staging.company.com
  - strimzi-kafka-broker-1.staging.company.com
  - strimzi-kafka-broker-2.staging.company.com

I get the following error from the avro server:

Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching k8s-strimzik-myclust-f4784b7673-7xxxxxxxxxxxxxx.elb.us-west-2.amazonaws.com found.

Do I need to specify the ELB hostnames in my certificate? I was under the impression that this wouldn't need them since the hostnames in the bootstrap configuration is for the hosts in the domain I have the certificate for.

[scholzj]

I think you will need to set the advertised hostname override for each broker: https://strimzi.io/docs/operators/latest/full/configuring.html#type-GenericKafkaListenerConfigurationBroker-reference

Strimzi lets you set the ExternalDNS annotations. But it does not really parse them or understand them (which would be quite hard since there are multiple tools like that). So if you want the broker to actually use the hostname, you need to set the advertisedHost field as well. Hope this helps.

[jslusher]

That did the trick, thanks!

For anyone else who might run into this, you need both the external-dns. annotations and the advertisedHost and advertisedPort settings to be in place. One creates the DNS records and then the other tells the brokers where to listen.

[scholzj]

The advertisedPort should be needed only in case you change the port somewhere on the way. If the load balancer exposes the same port as the listener uses (9096 in your case) it should not be needed.