Broker Certificate Renewal Procedure

[abhi1git]

Following are the details of our current Setup and how we are using Strimzi Kafka in our production

• Strimzi Kafka Operator : 0.22.1, Strimzi Kafka : 2.7.0, Kubernetes : 1.19.9
• Kafka 3 brokers & 3 Zookeeper nodes
• Listener of Ingress Type(using Nginx Ingress Controller with Internal Load balancer) is configured with SASL_SSL authentication as follows:

  •   listeners:
      - name: ingress
        port: 9094
        type: ingress
        tls: true
        authentication:
          type: scram-sha-512        
        configuration:
          bootstrap:
            host: kafka-pre-bootstrap.kinfra-internal.example.com
          brokers:
          - broker: 0
            host: kafka-pre-broker-0.kinfra-internal.example.com
        ..........
    

• Our Java Spring Boot Application acts as a client for Kafka and connects to Kafka on kafka-pre-bootstrap.kinfra-internal.example.com with trustsore created from certificate obtained using command kubectl get secret kafka-cluster-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' -n kafka| base64 --decode > ca.crt
• All certificates have default validity(1 year) and default renewal period (30 days)

Now the facts which I know(please correct me if I am wrong) are as follows:

• All certificates get auto-renewed in the renewal period
• Because of the Nginx TLS passthrough, certificates from the Kafka brokers will directly be used for authentication and not from the Ingress
• After auto-renewal I just need to give a rolling restart to zookeeper and Kafka pods so that they can start using new certificate
• I will have to edit the truststore of my Java clients with the newly generated certificate and they will again be able to connect to Kafka

So following are the Questions which I have :

• On what day in the renewal period will the certificate get renewed?
• Will my Java client face connectivity issues to Kafka immediately once the renewal is done?
• So do I need to worry about the certificate shown for the DNS kafka-pre-bootstrap.kinfra-internal.example.com when I check it using openssl s_client -connect kafka-qa-bootstrap.kinfra-internal.example.com:443 -servername kafka-qa-bootstrap.kinfra-internal.example.com (Asking this question just to confirm that renewal of Kafka broker certificate itself should be enough right?)
• Can you point me to a blog/documentation which would have clear steps for the procedure to be followed once a Kafka Cluster certificate gets renewed after 1 year?

@scholzj

[scholzj]

On what day in the renewal period will the certificate get renewed?

In general, right when the 30 days renewal period starts (in the first reconciliation which will be less than 30 days before the expiration) unless:

• you configured the maintenance time window.
• your cluster is in some broker state (for example pods being never ready etc.)

Will my Java client face connectivity issues to Kafka immediately once the renewal is done?

The renewal normally just does a new public certificate but does not regenerate the key. As far as I understood it, this means the client should keep working.

So do I need to worry about the certificate shown for the DNS kafka-pre-bootstrap.kinfra-internal.example.com when I check it using openssl s_client -connect kafka-qa-bootstrap.kinfra-internal.example.com:443 -servername kafka-qa-bootstrap.kinfra-internal.example.com (Asking this question just to confirm that renewal of Kafka broker certificate itself should be enough right?)

I'm not sure I follow this question ... this command should normally show the broker certificates. This should be renewed as part of the renewal. So my answer is you do not need to worry, but I'm not sure I got the question right.

Can you point me to a blog/documentation which would have clear steps for the procedure to be followed once a Kafka Cluster certificate gets renewed after 1 year?

TBH, I'm not sure there is anything more than what is in the documentation: Can you point me to a blog/documentation which would have clear steps for the procedure to be followed once a Kafka Cluster certificate gets renewed after 1 year?