Kafka certificate management - cert manager

[pashafirdous1]

Kafka certificate management

Hello,
Kafka version 0.17.0
i'm trying to configure Kafka to pickup certs generated by the cert-manger

when i use the default certs generated by cert manager (ca.crt, tls.crt, tls.key) for the clients secrets, the operator is able to recognize them and brings up the cluster.

but when i use the same secrets for cluster secrets, it is giving me a NullPointerException.

2021-07-16 13:57:49 WARN  AbstractOperator:268 - Reconciliation #61(timer) Kafka(kafka-17/clusterb): Failed to reconcile
java.lang.NullPointerException: null
        at java.util.Base64$Decoder.decode(Base64.java:549) ~[?:1.8.0_242]
        at io.strimzi.operator.cluster.model.Ca.currentCaKey(Ca.java:710) ~[io.strimzi.operator-common-0.17.0.jar:0.17.0]
        at io.strimzi.operator.cluster.model.Ca.generateSignedCert(Ca.java:282) ~[io.strimzi.operator-common-0.17.0.jar:0.17.0]
        at io.strimzi.operator.cluster.model.Ca.generateSignedCert(Ca.java:327) ~[io.strimzi.operator-common-0.17.0.jar:0.17.0]
        at io.strimzi.operator.cluster.model.ModelUtils.buildSecret(ModelUtils.java:241) ~[io.strimzi.cluster-operator-0.17.0.jar:0.17.0]
        at io.strimzi.operator.cluster.operator.assembly.KafkaAssemblyOperator$ReconciliationState.clusterOperatorSecret(KafkaAssemblyOperator.java:3187) ~[io.strimzi.cluster-operator-0.17.0.jar:0.17.0]
        at io.strimzi.operator.cluster.operator.assembly.KafkaAssemblyOperator.lambda$reconcile$3(KafkaAssemblyOperator.java:254) ~[io.strimzi.cluster-operator-0.17.0.jar:0.17.0]
        at io.vertx.core.Future.lambda$compose$3(Future.java:360) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
        at io.vertx.core.impl.FutureImpl.dispatch(FutureImpl.java:107) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
        at io.vertx.core.impl.FutureImpl.tryComplete(FutureImpl.java:152) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
        at io.vertx.core.impl.FutureImpl.complete(FutureImpl.java:113) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
        at io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:178) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
        at io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:21) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
        at io.vertx.core.impl.FutureImpl.dispatch(FutureImpl.java:107) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
        at io.vertx.core.impl.FutureImpl.tryComplete(FutureImpl.java:152) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
        at io.vertx.core.impl.FutureImpl.complete(FutureImpl.java:113) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
        at io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:178) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
        at io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:21) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
        at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:330) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
        at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:369) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
        at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.45.Final.jar:4.1.45.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [io.netty.netty-common-4.1.45.Final.jar:4.1.45.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [io.netty.netty-transport-4.1.45.Final.jar:4.1.45.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty.netty-common-4.1.45.Final.jar:4.1.45.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.45.Final.jar:4.1.45.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.45.Final.jar:4.1.45.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]

Please let us know

• what are the certificates/in the k8s secrets the clusterca is expecting?
• if i have tls.key instead of ca.key will the operator recognize it?
• how to use cert-manager to generate certificates/secrets for kafka

P.S. i have referred to the LetsEncrypt blog and it wont solve our use case.

CA certificate config

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ${CLUSTERNAME}-clients-ca-cert
spec:
  secretName:  ${CLUSTERNAME}-clients-ca-cert
  commonName: "${CLUSTERNAME}-clients-ca-cert"
  duration: 1h00m5s #default 8760h
  renewBefore: 50m #default 730h
  dnsNames:
  - foo.example.com
  - bar.example.com
  isCA: true
#  revisionHistoryLimit: 1
  issuerRef:
    kind: ClusterIssuer
    name: cluster-issuer   

regards,
Pasha

[scholzj]

You want to use that as a custom CA used by Strimzi or as the server certificates? Strimzi expects a fixed structure in the secrets. So you would need to follow it. That might require you to have something to copy and transform the secret created by CertManager to the right format. The docs - in the chapter about using the custom CA - describes the structure of the C secrets.

[scholzj]

So what certificates and how did you provided?

[pashafirdous1]

i generated certs/secrets for cluster-ca and client-ca from cert-manger.

from that i created a cluster-ca-cert(ca.crt) and cluster-ca(ca.key) (same for client-ca)

when i deployed kafka, the ZK pods launch and have below state. however, they have the error logs i shared above.

cluster-2-zookeeper-0                         2/2     Running   0          2m23s
cluster-2-zookeeper-1                         2/2     Running   0          104s
cluster-2-zookeeper-2                         2/2     Running   0          61s

[scholzj]

So, is the certificate generated by CertManager marked as CA:True? I think the stunnel TLS sidecar cares about this. The error in general IMHO means that the certificate used by the other ZooKeepers is not signed by the trusted CA. I'm not sure how would that happen ... are the key and crt files matching?

TBH, I'm not sure I understand what is the added value of doing this. The end result is that you are using Strimzi with self-signed CA ... which is the same as without Cert Manager.

[pashafirdous1]

Hi ,

isCA: false
the CA is false and cert-manager will be centralized to all our deployments, so will be trusted within the org.
i also referred to one of your discussions and added the below section in the ZK section. still the response is same.

    jvmOptions:
      javaSystemProperties:
        - name: zookeeper.ssl.hostnameVerification
          value: "false"
        - name: zookeeper.ssl.quorum.hostnameVerification
          value: "false"

[scholzj]

The CA flag needs to be true, because Strimzi is using it as a CA (it uses it to issue new certificates). I think this might be the issue here - IIRC the stunnel in the TLS sidecar actually checks the flag (most other tools usually just ignore it).

The disabling of the hostname verification is not related to you - that applies only for newer Strimzi versions where ZooKeeper does the TLS. In your version, TLS seems to be still done by the TLS sidecar. (but even in 0.24, the TLS sidecar is still used in some compoenents. So upgrading might move you forward, but not solve the CA:true issue completely.)