Kafka listener certificate

[MathieuTheerensKlarrio]

I've been trying to setup Strimzi using a listener certificate but so far I've always run into troubles. I am using the latest version of the operator and my setup is:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: strimzi-cluster
  namespace: kafka
spec:
  kafka:
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      log.message.format.version: "3.1"
      inter.broker.protocol.version: "3.1"
      auto.create.topics.enable: true
      default.replication.factor: 1
      num.partitions: 6
      ssl.enabled.protocols: "TLSv1,TLSv1.2,TLSv1.1"
    version: 3.1.0
    storage:
      type: persistent-claim
      size: 5Gi
      deleteClaim: false
    logging:
      type: inline
      loggers:
        kafka.root.logger.level: "INFO"
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: true
        configuration:
          brokerCertChainAndKey:
            secretName: private-cert
            certificate: ca.crt
            key: ca.key

Where private-cert is a certificate I create via the following commands:

### Create CA
openssl genrsa -out rootCA.key 2048
openssl req -x509 -sha256 -new -nodes -key rootCA.key -days 3650 -out rootCA.crt
openssl pkcs12 -export -in rootCA.crt --nokeys -out rootCA.p12 -password pass:my-password -caname ca.crt

### Create certificate signed by CA
cat > client.ext << EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
subjectAltName = @alt_names

[alt_names]
DNS.1 = *.strimzi-cluster-kafka-brokers
DNS.2 = *.strimzi-cluster-kafka-brokers.kafka.svc
DNS.3 = strimzi-cluster-kafka-bootstrap
DNS.4 = strimzi-cluster-kafka-bootstrap.kafka.svc
EOF


openssl genrsa -out private-cert.key 2048
openssl req -new -key private-cert.key -out private-cert.csr
openssl x509 -req -in private-cert.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out private-cert.crt -days 825 -sha256 -extfile client.ext
openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in private-cert.key -out private-cert-pkcs8.key

I create the secret via

kubectl create secret generic private-cert --from-file=ca.crt=private-cert.crt --from-file=ca.key=private-cert-pkcs8.key -n kafka

Every time I try to setup a kafka producer or consumer by deploying the kafka-cli pod to K8s and afterwards connecting to shell and executing:

cat > /tmp/props << EOF
security.protocol=SSL
ssl.protocol=SSL
ssl.truststore.type=PEM
ssl.truststore.location=/tmp/private-cert/ca.crt
EOF


kafka-console-producer.sh --producer.config /tmp/props --bootstrap-server strimzi-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092 --topic test-topic-1

it gives me the error:

javax.net.ssl.SSLProtocolException: Unexpected handshake message: client_hello

Can anyone help me with this?

[scholzj]

The Kafka CR looks good. So without the logs from operator, brokers, clients and the operator ir is hard to say what is the issue TBH. ssl.protocol configures the version of the SSL protocol. Not sure what exactly setting it to SSL means ... SSLv3? that is probably not supported out of the box by the broker. So I would try to remove that option and use only

security.protocol=SSL
ssl.truststore.type=PEM
ssl.truststore.location=/tmp/private-cert/ca.crt

[MathieuTheerensKlarrio]

Thanks for the quick reply!
I just tried removing it but unfortunately doesn't seem to change anything.
Please find here the logs of the kafka-broker-0 during the request.

The kafka client gave the following error:

[2022-05-13 13:15:49,657] ERROR [Producer clientId=console-producer] Connection to node -1 (strimzi-cluster-kafka-bootstrap.kafka.svc.cluster.local/10.96.46.174:9092) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2022-05-13 13:15:49,657] WARN [Producer clientId=console-producer] Bootstrap broker strimzi-cluster-kafka-bootstrap.kafka.svc.cluster.local:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

[scholzj]

You can use -Djava.net.debug=ssl system property to get some TLS debug infor from the client. You should also try to remove the special TLS options you have in the broker and start witht he most simple configuration rather than starting with something special.

[MathieuTheerensKlarrio]

Ok, I will try that.
But can you also verify that the broker certificate works with the latest version of Kafka and Strimzi? If so, what commands do you use to test it?

[scholzj]

I'm afraid I'm traveling this week. I can try it next week if you want. But there are system tests for it which are run regularly.

The way I would test it:

• (if needed) I would use openssl s_client to verify that the broker returns the right certificate
• Use the Java client to connect / debug the issues in it

[vjvel]

@scholzj

I need to add one more dns name for the tls certs for the route53 failover purpose. how can we additional altdnsnmae for the external listener.

configuration:
bootstrap:
annotations:
external-dns.alpha.kubernetes.io/hostname: kafka-bootstrap.mydomain.com.
external-dns.alpha.kubernetes.io/ttl: "60"

[scholzj]

When you use Kafka listener certificate, it is you who creates the certificate. So you can add any DNS names you want to it.

[vjvel]

@scholzj

I have created own rootCA and certificate which has all dns and added in the external tls configuration. created the secret with crt and key and pod also running fine.

i want to try with producer.config to connect this external listener. I tried with below option
security.protocol=SSL
ssl.truststore.type=PEM
ssl.truststore.location=/tmp/private-cert/ca.crt

getting the error

[2022-08-12 11:17:22,973] WARN [Producer clientId=console-producer] Bootstrap broker x.x.x.x:9094 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2022-08-12 11:17:23,139] ERROR [Producer clientId=console-producer] Connection to node -1 (/x,x,x.x:9094) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2022-08-12 11:17:23,139] WARN [Producer clientId=console-producer] Bootstrap broker x.x.x.x:9094 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2022-08-12 11:17:23,674] ERROR [Producer clientId=console-producer] Connection to node -1 (/x.x.x.x:9094) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)

I am bit confused whether we need to sign this certificate with Cluster CA or how can we do that not able to find clear documentation. please refer some configuration example for this external listener tls and producer.properties what to use... Thanks

[scholzj]

Then you need to debug the TLS to find out what problem does the client have.

[vjvel]

I found the issue, My external LB is using IP address for broker not DNS name. So I added the broker IP in alt-dnsnames and it worked now.. Thanks @scholzj for your quick response...

[chelsiezhen-db]

@vjvel Hi Vjvel, I got similar issue with the external load balancer: SSL handshake failed with "Unexpected handshake message: client_hello". Could you please help to explain more about your solution? I don't understand what broker IP you added into alternative DNS names of the broker cert. Doesn't the broker IP change with every restart of the broker pod?

[scholzj]

Doesn't the broker IP change with every restart of the broker pod?

It does. But not sure you care about it since the broker IP is normally not used for anything. Loadbalancers have their own addresses which do not change when broker restarts.

I don't understand what broker IP you added into alternative DNS names of the broker cert.

You need to get the load balancer address. You should be able to get it with kubectl get service.

[chelsiezhen-db]

I see. Thank for the prompt response!!