Full cert-manager support for listener keystore and client truststore

[sergeiwaigant]

Hi guys.

It might be, that I missed some other discussions or issues that tackled this issue already. In that case please just link it if you find it...
I am not quite experienced with Java, therefore I a not able to find the correct placed in the code to provide a PR to suggest a change.

Our requirement is to use an own CA for the external and internal listener.
Furthermore these listeners should be enabled with mTLS client authentication.

Implementing this requirements is failing because of two issues:

1. cert-manager TLS certificate chain with an own issuer

TL;DR: The missing ca.crt in the tls.crt is causing the Kafka Broker to use a certificate in the keystore which is not having the full certificate chain... Therefore the client cannot validate the issuer properly and TLS handshake is failing.

When using cert-manager with an own issuer you are getting a secret with 3 keys: ca.crt, tls.crt and tls.key.
When referencing this certificate in the listener via brokerCertChainAndKey with pointing certificate and key to this keys, Strimzi is creating a volume and volumeMount like this:

# .... snip ....
    volumeMounts:
    - mountPath: /opt/kafka/certificates/custom-internal-9093-certs
      name: custom-internal-9093-certs
# .... snip ....
  volumes:
  - name: custom-internal-9093-certs
    secret:
      defaultMode: 292
      items:
      - key: tls.key
        path: tls.key
      - key: tls.crt
        path: tls.crt
      secretName: self-signed-broker-cert
# .... snip ....

The script kafka_tls_prepare_certificates.sh is then looping over the certificates folder and if the folder starts with custom, its executing the create_keystore_without_ca_file function with the key and crt.

https://github.com/strimzi/strimzi-kafka-operator/blob/main/docker-images/kafka-based/kafka/scripts/kafka_tls_prepare_certificates.sh#L38

Why can't we just mount "all" keys provided in the brokerCertChainAndKey.secretName and let the script identify if there is a "ca.crt" and execute the create_keystore function instead?

2. clientsCa secret format incompatible to cert-manager

TL;DR: We want to use the same own issuer as clients CA to issue client certificates.

When setting clientsCa.generateCertificateAuthority to false Strimzi requires you to provide two very specific secrets which include clients-ca-cert with ca.key and clients-ca with ca.p12 and ca.crt.
The volume creation and volumemount is ending up like this:

# .... snip ....
    volumeMounts:
    - mountPath: /opt/kafka/client-ca-certs
      name: client-ca-cert
# .... snip ....
  volumes:
  - name: client-ca-cert
    secret:
      defaultMode: 292
      secretName: clients-ca-cert
# .... snip ....

The files in that folder are then used in the script to create the client truststore.

https://github.com/strimzi/strimzi-kafka-operator/blob/main/docker-images/kafka-based/kafka/scripts/kafka_tls_prepare_certificates.sh#L66

Wouldn't it be possible to just reference a secret which will be used for this volume creation?

Please let me know your thoughts...

Kind regards
Sergei

[sergeiwaigant]

Please find below the output of openssl when connecting to port 9093.
As you can see the certificate chain is only having one certificate, which is causing the error message: unable to verify the first certificate

[kafka@kafka-test-kafka-0 kafka]$ openssl s_client -connect localhost:9093 -showcerts </dev/null
CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 CN = kafka-test.kafka-test.svc.cluster.local
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = kafka-test.kafka-test.svc.cluster.local
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = kafka-test.kafka-test.svc.cluster.local
verify return:1
---
Certificate chain
 0 s:CN = kafka-test.kafka-test.svc.cluster.local
   i:CN = kafka-test
-----BEGIN CERTIFICATE-----
..... snip .....
-----END CERTIFICATE-----
---
Server certificate
subject=CN = kafka-test-kafka-test.svc.cluster.local

issuer=CN = kafka-test

---
Acceptable client certificate CA names
O = io.strimzi, CN = clients-ca v0
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA1:RSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:Ed448:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3247 bytes and written 399 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---
DONE

[scholzj]

Re 1) A long time since I tried it, bu I think you should put the full chain into the tls.crt file.

Re 2) That is currently not possible. There is also no integration with cert-manager at this point (while it is on the roadmap, it does not have any expected timeline at this point). IIRC you do not need to provide the p12 file (and the related password file) for a custom CA. For CLients CA, if you plan to issue user certificates with Cert Manager, you might not need to provide the private key either, but you would need to at least have some dummy value for it. Any you always have to provide the public key (the crt file).