[Bug]: Renew CA certificates and private keys with your own (Before Expiry renewal) failing

[sanchayana2007]

Bug Description

I created my own certificate as below and updated the cluster-cert as mentioned in Strimzi documents

openssl req -new -newkey rsa:4096 -days 365 -sha512 -x509 -subj "/O=io.strimzi/CN=clients-ca v0" -keyout ca.key -out ca.crt -nodes

openssl pkcs12 -export -in ca.crt -nokeys -out ca.p12 -password pass:XXXXXX -caname ca.crt

kubectl create secret generic kafka-cluster-ca-cert -n kafka-dev --from-file=ca.crt=ca.crt --from-file=ca.p12=ca.p12 --from-file=ca.password=p12.password #the password is stored on the file p12.password

I deleted the old cluster-ca-cert and recreated the cert as above and restarted the zookeper pod and i see the below in
LOG: ---------------------------------
Setected Zookeeper ID 3

But it fails to restart and on the logs it shows the following
Preparing truststore
Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/zookeeper/cluster.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore is complete
Looking for the right CA
No CA found. Thus exiting.

When i replace the cluster-ca-cert to the old cert the pod again runs fine .
Please help , I am not sure if am missing on something

Steps to reproduce

delete the old cert

openssl req -new -newkey rsa:4096 -days 365 -sha512 -x509 -subj "/O=io.strimzi/CN=clients-ca v0" -keyout ca.key -out ca.crt -nodes

openssl pkcs12 -export -in ca.crt -nokeys -out ca.p12 -password pass:XXXXXX -caname ca.crt

kubectl create secret generic kafka-cluster-ca-cert -n kafka-dev --from-file=ca.crt=ca.crt --from-file=ca.p12=ca.p12 --from-file=ca.password=p12.password #the password is stored on the file p12.password

I deleted the old cluster-ca-cert and recreated the cert as above and restarted the zookeper pod and i see the below in
LOG: ---------------------------------
Setected Zookeeper ID 3

Expected behavior

Zookeper should take the newly generated certificate

Strimzi version

0.27

Kubernetes version

1.24

Installation method

helm insatll

Infrastructure

GKE

Configuration files and logs

Setected Zookeeper ID 3

But it fails to restart and on the logs it shows the following
Preparing truststore
Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/zookeeper/cluster.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore is complete
Looking for the right CA
No CA found. Thus exiting.

Additional context

No response

[scholzj]

This is not how you are supposed to renew the certificate. You seem to generate a brand new private and public key and replace just the public key. You should upgrade to the latest version and follow the documentation for the renewal.

[sanchayana2007]

As suggested by you i tried the existing kafka-cluster-key and kafka-client-key to generate new certs(cluster/client) . But that didnt worked .

I copied the existing cluster/client ca.key from the cluster
openssl req -new -key ca.key -sha512 -x509 -days 365 -out cert.crt -nodes
As my old cert is inn .p12 format i did
openssl pkcs12 -export -in ca.crt -nokeys -out ca.p12 -password pass:xxxxxx -caname ca.crt

Am i doing this wrong ?

[sanchayana2007]

Kafka Cluster and client certs are not getting renewed as We performed the following steps now :

I) Pausing reconciliation
2) creating a new self signed cluster certificate using an existing cluster key from the secrets , both new and old certs are identical
3) Updating both old and new cert in secret and incrementing cert-generation=1
4) Enabling reconciliation

restarting the deployment

[sanchayana2007]

hi @scholzj , @valdar ,

Can you provide me the openssl commands for renewing the certs from the old key . The documentation dosen.t help me to get the commands for doing this .
i) openssl req -new -key ca_cluster.key -out ca.csr -subj "/O=io.strimzi/CN=cluster-ca v0"
ii) openssl x509 -req -days 365 -in ca.csr -signkey ca_cluster.key -out ca.crt -extfile ca.cnf -sha512 -passin pass:symantec -extensions v3_ca
But this cert as generated dosents get validated

openssl verify -CAfile ca.crt crt_from_zookeper.crt
O = io.strimzi, CN = devops-kafka-zookeeper
error 7 at 0 depth lookup: certificate signature failure
error crt_from_zookeper.crt: verification failed

[scholzj]

Sorry, I do not know the OpenSSL commands out of my head to just give them to you like that. The general expectation is that if you decide to use the custom CA, you already have the custom CA, or have some experience managing it and some good reasons why to use it. If you don't, you should probably use the Strimzi CA.