SSL handshake while using custom cert for Kafka listeners

[arjun180]

Hi,

I have been dealing with an SSL handshake failure while dealing with my Kafka cluster setup using Strimzi. I've been using this guide https://strimzi.io/docs/operators/latest/full/using.html#ref-alternative-subjects-certs-for-listeners-str to setup custom certificates on my Kafka listener. The following are the certs I have (The CA is a third party vendor) :

1. Root cert (root.crt)
2. intermediate cert (intermediate.crt)
3. Server crt (server.crt)
4. Server key (server key)

I do not have the root key

My Kafka cluster yaml looks like the following :

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  clusterCa:
    generateCertificateAuthority: true
  kafka:
    version: 2.6.0
    replicas: 1
    listeners:
      - name: external
        port: 9094
        type: loadbalancer
        tls: true
        authentication :
          type: tls
        configuration:
          brokerCertChainAndKey:
            secretName: my-secret
            certificate: server.crt
            key: server.key
    storage:
      type: ephemeral
    config:
      num.partitions: 8
      num.recovery.threads.per.data.dir: 1
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      ssl.cipher.suites: "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
      ssl.enabled.protocols: "TLSv1.2"
      ssl.protocol: "TLSv1.2"
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral

Before deploying this Kafka cluster I created my-secret using the following command :

kubectl create secret generic my-secret --from-file=server.key --from-file=server.crt

The deployment goes off okay. The next step is to setup my TLS/SSL connection so that I can connect using the Kafka CLI (to send messages through the producer/consumer)

1. Create a trust store file where the root cert is added

keytool -keystore kafka.truststore.jks -alias CARoot -import -file root.crt

2. Create a keystore and add the server private key. Since, a private key can't be directly added to the keystore. So I create a p12 file and then add it to the keystore

openssl pkcs12 -export -in server.crt -inkey server.key -out server.p12 -certfile root.crt
# Add it to the keystore
keytool -importkeystore -srckeystore server.p12 -srcstoretype pkcs12 -destkeystore keystore.jks -deststoretype JKS

3. Import the root CA into the keystore

keytool -keystore keystore.jks -alias CARoot -import -file root.crt

4. Import the signed server certificate into the keystore

keytool -keystore keystore.jks -import -file server.crt 

To try the Kafka CLI, I created a client-ssl.properties file with the following configs :

bootstrap.servers=[LOADBALANCER_PUBLIC_IP]:9094
security.protocol=SSL
ssl.truststore.location=[TRUSTSTORE_LOCATION]
ssl.truststore.password=<>
ssl.keystore.location=[KEYSTORE_LOCATION]
ssl.keystore.password=<>
ssl.key.password=<>

I verified the public IP is correct by using kubectl get svc and looking at the loadbalancer public IP. I then ran the following command

$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list $LOADBALANCER_PUBLIC_IP:9094 --topic $TOPIC_NAME --producer.config client-ssl.properties

And the error I kept getting was :

[2021-06-15 15:02:08,445] ERROR [Producer clientId=console-producer] Connection to node -1 (<LOADBALANCER_HOSTNAME>/<IP>:9094) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)

On looking into the pod logs :

2021-06-15 22:02:08,443 INFO [SocketServer brokerId=0] Failed authentication with /<IP> (SSL handshake failed) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-8]

I was wondering if someone could please shed some light into what I'm doing wrong here?

[scholzj]

I think there are several issues in what you are doing:

• Your description suggests the server.crt contains only the server public key without the CAs. In that case, your truststore should IMHO contain not just the Root CA but also the Intermediate CA to trust it. Or, you should have the server.crt contain the whole chain (so not just the server public key but also the CA public keys).
• You are adding the server key to the keystore. But the keystore needs to contain the user certificate for authentication. You can create one for example using the User Operator => create the KafkaUser resource and the operator will create a secret with the user certificate for authentication.

[arjun180]

Hi Jakub - thanks for this. I did some more reading on this (https://itnext.io/kafka-on-kubernetes-the-strimzi-way-part-3-19cfdfe86660) for mutual TLS authentication between the client and the listener.

Couple of questions :

1. Based on some reading I was doing, the truststore needs the cluster CA certificate. Which means the secret stored in

kubectl get secret $CLUSTER_NAME-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 --decode > ca.crt
kubectl get secret $CLUSTER_NAME-cluster-ca-cert -o jsonpath='{.data.ca\.password}' | base64 --decode > ca.password

I don't have a CA, just server certs. So should I be using the strimzi CA certs within my truststore? If so, where does the section below come into play ?

configuration:
          brokerCertChainAndKey:
            secretName: my-secret
            certificate: server.crt
            key: server.key

2. Thanks for the suggestion for using the Kafka user, as far as I understand it creates a user.crt and user.key which can be then be imported into my keystore.

I guess I'm confused w.r.t how my sever certs are being used in the authentication process. Thanks for your help!

[scholzj]

Re 1) Assuming we are still talking about the same configuration ... You can add the server.crt into your truststore. That would make your client trust this server. Using the CA certificates which signed it has one main advantage => after some time, once you get new server.crt from the same CA, you would need to update the truststore. Where as when you use the CA certificates, you would not need to change your truststore. On the beginning you mentioned that you have the CA certificates (root.crt and intermediate.crt) => these are the CA certs, you do not need the private keys in the truststore so you should be able to use them.

When you configure the custom listener certificate as you did, the Strimzi Cluster CA is not used at all by your clients

Re 2) Yes, correct. There are the certs to be used as the keystore (you can use the p12 file directly or create your own keystore from the crt and key files)

[arjun180]

When you configure the custom listener certificate as you did, the Strimzi Cluster CA is not used at all by your clients

^ I see. That makes sense. Alright, let me try this. I'll do the following :

1. Add the server.crt into the truststore (I understand your point on using root.crt + intermediate.crt as a the CA bundle, but for now I'll just try with the server.crt)

2. Create the Kafka user resource. Extract the user.crt and password, and add it into the keystore.

Let me try this and get back to you. Thanks!

[arjun180]

Alright. So, I ended up with the same error :

1. I added the server.crt to the truststore. I checked the contents of the truststore to make sure it had only one entry, which was the server.crt.

2. I created a Kafka user resource and deployed it. The yaml file looks like this

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: kafka-tls-client-credentials
  labels:
    strimzi.io/cluster: my-cluster
spec:
  authentication:
    type: tls

3. A secret called Kafka-tls-client-credentials was created. I extracted the user credentials from it

export KAFKA_USER_NAME=kafka-tls-client-credentials
kubectl get secret $KAFKA_USER_NAME -o jsonpath='{.data.user\.crt}' | base64 --decode > user.crt
kubectl get secret $KAFKA_USER_NAME -o jsonpath='{.data.user\.key}' | base64 --decode > user.key
kubectl get secret $KAFKA_USER_NAME -o jsonpath='{.data.user\.p12}' | base64 --decode > user.p12
kubectl get secret $KAFKA_USER_NAME -o jsonpath='{.data.user\.password}' | base64 --decode > user.password

# Created a keystore
export USER_P12_FILE_PATH=user.p12
export USER_KEY_PASSWORD_FILE_PATH=user.password
export KEYSTORE_NAME=kafka-auth-new-keystore.jks
export KEYSTORE_PASSWORD=foobar
export PASSWORD=`cat $USER_KEY_PASSWORD_FILE_PATH`
sudo keytool -importkeystore -deststorepass $KEYSTORE_PASSWORD -destkeystore $KEYSTORE_NAME -srckeystore $USER_P12_FILE_PATH -srcstorepass $PASSWORD -srcstoretype PKCS12

4. I then checked the keystore to make sure the user authentication key was present. It was the only entry there.

Enter keystore password:  
kafka-tls-client-credentials, Jun 16, 2021, PrivateKeyEntry,

5. My Kafka cluster is running with the same yaml configurations based on the server.crt and server. key. I change the client-ssl.properties to be :

bootstrap.servers=[LOADBALANCER_PUBLIC_IP]:9094
security.protocol=SSL
ssl.truststore.location=kafka.truststore.jks
ssl.truststore.password=<password>
ssl.keystore.location=kafka-auth-new-keystore.jks
ssl.keystore.password=<password>
ssl.key.password=[contents of user.password]

I seem to have done everything correctly. But the SSL error remains when I try to connect through the Kafka CLI.

$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list $LOADBALANCER_PUBLIC_IP:9094 --topic $TOPIC_NAME --producer.config client-ssl.properties
>[2021-06-16 12:32:41,320] ERROR [Producer clientId=console-producer] Connection to node -1 (<DNS_HOSTNAME>/<IP>:9094) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)

[scholzj]

• I do not think this is needed ssl.key.password=[contents of user.password]
• Do you have a full exception form the client? IMHO it is better to start with consumer - it gives better errors
• You can try to use Java system property -Djavax.net.debug=ssl (for example set through the KAFKA_OPTS env var) => that should print some TLS debug information.

[hari819]

this setup for enabling TLS auth worked for me without any issue , the contents of user.password are not required ,
thankyou @scholzj , @arjun180