Certs Renewal Issue

[saloni-2404]

Hi Strimzi team,

We are observing an issue recently where the cluster-ca and client-ca certs expired for a product team using Strimzi kafka operator. Post expiration of the certificates, the team tried renewing them by using the "force-renew: true" annotation method, referring to the official documentation, (https://strimzi.io/docs/operators/in-development/configuring.html#proc-renewing-ca-certs-manually-str). We are under the impression that this activity should be performed before the expiration date, while the certificates are still valid - please help to clarify if this is not the case.

Post trying this, "force-renew: true" annotation method, the kafka, entity operator and kafka-exporter pods are in crash loop back-off. Also, there is an error message in the tls-sidecar container.
"SSL_accept: 1408F10B: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number"

The operator logs suggest that it is not able to establish a connection with zookeeper.

Please advise on how to recover the cluster in such a state.

[saloni-2404]

Attaching the logs:

kafka_sacclusterlogs.txt
strimzi-operator-sacclusterlogs.txt
zookeeper_sac-clusterlogs.txt

[scholzj]

So, are you using your own Cluster CA certificates? Or is this with Strimzi generated CA?

In the first case, the annotations should not be used at all. In the second case, if the certs are already expired, it is too late to use any force-renew annotation since the cluster is already broken. If it is Strimzi generated CA, it should normally renew automatically before it expires. It is also not completely clear what happened / what have you done after the certs expired.

Assuming the CA is now renewed, You might need to delete the secrets with the certificates manually together with the pods and have them recreated by the operator with the new CA.

[scholzj]

0.17v strimzi operator version.

Hmm, that is quite old. I do not remember the versions, but there were for sure some bugs around the renewal in the older versions.

Not sure how do we need to delete the pods and secrets together. Can we delete the kafka, zookeeper statefulsets?

You can just delete them with kubectl. Just delete the secrets first and then the pods. It does not need to be exactly in one command.

[srujan07]

Okay, thanks. Any idea why the certs were not auto renewed within the renewal period. It would help us understand the root cause so that we ensure it won't happen to our production clusters going forward. How can we get rid of this in future?

[scholzj]

Well, as I said ... 0.17 is real old. There are many bugs fixed since. So it is hard to speculate what exactly the problem is. The general recommendation would be to upgrade to the latest version for a start. If some problems happen there, they are much easier to debug and investigate because the code is up-to-date.

[srujan07]

got it @scholzj .. Can you please confirm on the secrets we need to delete. Is it cluster-ca and clients-ca or all the secrets created by operator?

[scholzj]

That is not easy without knowing the exact state of the cluster. You should check if the CAs were renewed or not. If they are renewed (that would be my expectation, but not 100% sure), you just delete the server secrets. If they are not renewed you delete them as well. The server secrets are the secrets such as <cluster-name>-kafka-brokers, <cluster-name>-zookeeper-nodes etc.