[Bug]:

[charris-ca]

Bug Description

Strimzi operator has initiated a rollout of kafka pods but the old pods are now on a certificate that is too old for it to communicate with. It started a rollout and the new pods come up successfully, but the partitions on the new pods and the old pods cannot communicate with each other so they are marked as under replicated.

Steps to reproduce

No response

Expected behavior

Strimzi operator should renew and rollout the certificates BEFORE their expiration date. If they do expire, there should be some logic to handle this situation.

Strimzi version

0.33.2

Kubernetes version

1.22.8

Installation method

Helm Chart

Infrastructure

No response

Configuration files and logs

...SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed

Cannot get current offset of topic dr-cluster.dsp-afi-trans-log partition 36: x509: certificate has expired or is not yet valid: current time 2023-07-20T23:59:49Z is after 2023-07-20T22:51:22Z

2023-07-21 00:42:46 INFO KafkaAvailability:127 - Reconciliation #58(timer) Kafka(xxx): jaeger-spans/3 is already under-replicated (ISR={23}, replicas=[18,23,13], min.insync.replicas=2); broker 18 has a replica, so should not be restarted right now (it might be first to catch up).

Additional context

No response

[scholzj]

You did not shared a full log. So it is mostly just guessing. But it looks like the operator was not able to roll some brokers because you have out-of-sync replicas. So your certs expired. Assuming it is only the brokers, you will need to simply manually delete them with kubectl so that they reload the certificates.

[charris-ca]

That is what I had to do, but it did incur a small downtime. When the leader(?) partition went down there was a delay until the controller could figure out that a separate broker with the new cert has an active partition.