Added brokers SSL Handshake failures

[jslusher]

I have a 3 broker Strimzi cluster that has been working fine. I just tried adding another 3 brokers and the new brokers are showing these errors in the logs:

2023-07-26 22:35:31,131 INFO [SocketServer listenerType=ZK_BROKER, nodeId=3] Failed authentication with /172.25.13.162 (channelId=172.25.12.220:9096-172.25.13.162:32089-636) (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-3-ListenerName(SCRAM-9096)-SASL_SSL-8]

Not sure what to check since the other 3 brokers seem fine.

[scholzj]

That suggests some misconfiguration in the clients. How is the broker listener on port 9096 configured with regard to TLS? How are the clients configured for TLS?

[jslusher]

I found at least one place that is showing the SSL handshake failure. I have a akhq pod that is showing this in its logs:

Caused by: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching k8s-strimzik-xxxxxx-d9c1938396-xxxxxxxxxxx.elb.us-west-2.amazonaws.com found.

[scholzj]

Shouldn't the broker listeners be configured the same as the first three brokers?

It should, yes. That is why I suggested a client configuration issue. If your clients are failing because of No subject alternative DNS name, you need to share how you configured the Kafka cluster and the clients. Without that it is not possible to pinpoint the issue.

[jslusher]

After some troubleshooting, I think I figured out what happened:

• I changed spec.kafka.replicas to 6 from 3
• I did not add their respective spec.kafka.listeners configuration
  • we're using scram-sha-512 listeners using a loadbalancer type
  • each listener has an advertisedHost with external-dns annotations correlating to a certManager certificate entry
• the operator created the broker pods which came up and joined the kafka cluster
• because their listeners weren't configured and their hostnames weren't present in the certificate, the clients were giving the auth errors

I tried later adding the listener/certificate configuration, but the operator was stuck giving this exception:

Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://10.100.0.1:443/api/v1/namespaces/strimzi-kafka-cluster/services/kafka-example-kafka-scram-3. Message: Service "example-kafka-example-kafka-scram-3" is invalid: spec.loadBalancerClass: Invalid value: "null": may not change once set. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.loadBalancerClass, message=Invalid value: "null": may not change once set, reason=FieldValueInvalid, additionalProperties={})], group=null, kind=Service, name=example-kafka-example-kafka-scram-3, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Service "example-kafka-example-kafka-scram-3" is invalid: spec.loadBalancerClass: Invalid value: "null": may not change once set, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).

Restarting pods/operator didn't help, so I had to roll back the added brokers entirely.

I just tried first adding the hostname to the certificate, and only after that adding a broker, but I'm still getting the same exception from the operator.

[jslusher]

Listener configuration for reference:

  kafka:
    version: 3.4.0
    replicas: 4
...
    listeners:
    - name: scram
      authentication:
        type: scram-sha-512
      port: 9096
      type: loadbalancer
      tls: true
      configuration:
        bootstrap:
          annotations:
            external-dns.alpha.kubernetes.io/hostname: strimzi-kafka-bootstrap.staging.example.com
            external-dns.alpha.kubernetes.io/ttl: "60"
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-type: external
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
        brokers:
        - broker: 0
          advertisedHost: strimzi-kafka-broker-0.staging.example.com
          advertisedPort: 9096
          annotations:
            external-dns.alpha.kubernetes.io/hostname: strimzi-kafka-broker-0.staging.example.com
            external-dns.alpha.kubernetes.io/ttl: "60"
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-type: external
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip

[jslusher]

Certificate configuration as reference:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: strimzi-kafka-sdlc
spec:
  secretName: strimzi-kafka-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
    group: cert-manager.io
  dnsNames:
  - strimzi-kafka-bootstrap.staging.example.com
  - strimzi-kafka-broker-0.staging.example.com
  - strimzi-kafka-broker-1.staging.example.com
  - strimzi-kafka-broker-2.staging.example.com
  - strimzi-kafka-broker-3.staging.example.com

[jslusher]

After some more troubleshooting, it looks like I ran into this issue. Upgrading the operator to 0.36.0 seems to have fixed it.