cluster-ca-cert has expired

[Aldjinn]

Hi,
since yesterday, my kafka cluster is broken. The cluster has been created a year ago and the issue seems to be an expired certificate:

Output from a crash looping kafka pod:

+ ./kafka_tls_prepare_certificates.sh
Preparing truststore for replication listener
Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/kafka/cluster.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore for replication listener is complete
Looking for the right CA
Found the right CA: /opt/kafka/cluster-ca-certs/ca.crt
Preparing keystore for replication and clienttls listener
Error certificate has expired getting chain.

NAME                                            READY   STATUS             RESTARTS   AGE
kafka-cluster-entity-operator-f6bb47fcb-g8dlf   1/3     CrashLoopBackOff   11         9m58s
kafka-cluster-kafka-0                           0/1     CrashLoopBackOff   208        17h
kafka-cluster-kafka-1                           0/1     CrashLoopBackOff   199        16h
kafka-cluster-kafka-2                           1/1     Running            1          11d
kafka-cluster-zookeeper-0                       1/1     Running            0          11d
kafka-cluster-zookeeper-1                       1/1     Running            0          11d
kafka-cluster-zookeeper-2                       1/1     Running            0          11d

My cluster is called "kafka-cluster" and the according secret containing the certificate is kafka-cluster-cluster-ca-cert. I've had a look into ca.crt and the result contains:

        Validity
            Not Before: Jun 17 07:30:37 2020 GMT
            Not After : Jun 17 07:30:37 2021 GMT

The cluster was created using the following settings:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: kafka-cluster
  namespace: kafka
spec:
  kafka:
    version: 2.5.0
    replicas: 3
    listeners:
      plain: {}
      tls: {}
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: '2.5'
    storage:
      type: persistent-claim
      size: 5Gi
      deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 5Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

I've tried to add the following clusterCa and clientsCa settings without any success:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: kafka-cluster
  namespace: kafka
spec:
  kafka:
    version: 2.5.0
    replicas: 3
    listeners:
      plain: {}
      tls: {}
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: '2.5'
    storage:
      type: persistent-claim
      size: 5Gi
      deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 5Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}
  clusterCa:
    renewalDays: 30
    validityDays: 3650
    generateCertificateAuthority: true
    certificateExpirationPolicy: renew-certificate
  clientsCa:
    renewalDays: 30
    validityDays: 3650
    generateCertificateAuthority: true
    certificateExpirationPolicy: renew-certificate

Also adding annotations described in the documentation did not change anything:

kubectl -n kafka annotate secret kafka-cluster-cluster-ca-cert strimzi.io/force-renew=true
kubectl -n kafka annotate secret kafka-cluster-clients-ca-cert strimzi.io/force-renew=true

Manually (re-) creating certificates and replacing the secret did not help either:

# create
openssl req -x509 -new -subj '/O=io.strimzi/CN=cluster-ca v0' -key ca.key -sha256 -days 10950 -out ca.crt

# print
openssl x509 -in ca.crt -text

# convert to p12
# password: *******
openssl pkcs12 -export -in ca.crt -inkey ca.key -out ca.p12

# convert to base64
openssl base64 -in ca.p12 -out ca.p12.base64

I'm not sure if this could be related to my issue, but my kafka operator is pending for upgrade.

NAME                                 DISPLAY              VERSION   REPLACES                           PHASE
strimzi-cluster-operator.v0.23.0     Strimzi              0.23.0    strimzi-cluster-operator.v0.22.1   Pending

The update (is is called update) has failed because of

          - lastTransitionTime: '2021-06-07T00:09:28Z'
            lastUpdateTime: '2021-06-07T00:09:28Z'
            message: >-
              pods "strimzi-cluster-operator-v0.23.0-6d48cd845c-" is forbidden:
              error looking up service account
              operators/strimzi-cluster-operator: serviceaccount
              "strimzi-cluster-operator" not found
            reason: FailedCreate
            status: 'True'
            type: ReplicaFailure

I'm a little bit lost. This is my first issue with the combination of a k8s operator and expired certificates. In general, a problem related to one way or the other is challenging, but the combination is... more challenging, at least for me.

Kind regards,
Dennis

[scholzj]

TBH, I'm not sure what the error in the operator upgrade means and how does it impact the old version of the operator - whether that is still functional or not. If the old one is not functional anymore because of the upgrade, but the new one doesn't work either, that can be the root cause of this. I guess the upgrade part is from OLM? Did you upgraded the CRDs before the 0.23 upgrade? The service account error looks weird, but normally you need to upgrade the CRDs manually after the 0.22 installation / before the 0.23 upgrade. So that can cause upgrade issues as well. (+ 0.23 does not support Kafka 2.5.0 anymore)

It would be interesting to see the logs from your cluster operator and the status of your Kafka cluster (i.e. get the full output with kubectl get kafka -o yaml which contains also the status part). That might help us with the initial issue why the CA was not renewed in time.

Apart from that, to fix it, probably the best thing is to do is delete the CA secrets and let the operator create a new one. You might need to help it in the first rolling update by deleting the pod (since they have already expired certificates and might not roll smoothly).

[scholzj]

Just a theory: Maybe the update of the operator from 0.22 to 0.23 starts failing just a few days before the required renewal (30 days) of the certificates. So the operator wasn't able to perform the renewal.

Yeah, I think that would be possible explanation.

Which logs from the operator are helpful?The pod kafka-cluster-entity-operator-f6bb47fcb-g8dlf contains [topic-operator user-operator tls-sidecar].

No, I would need the logs from the Cluster Operators => that is probably in some different namespace deployed by the OLM. Maybe in operators?

[Aldjinn]

Sorry, it's too hot today and my brain is melting. 🌡️

time="2021-06-18T13:20:06Z" level=info msg="checking strimzi-cluster-operator.v0.23.0"
time="2021-06-18T13:20:07Z" level=info msg="csv in operatorgroup" csv=strimzi-cluster-operator.v0.23.0 id=O7B9v namespace=operators opgroup=global-operators phase=Pending
time="2021-06-18T13:20:07Z" level=info msg="requirements were not met" csv=strimzi-cluster-operator.v0.23.0 id=O7B9v namespace=operators phase=Pending
time="2021-06-18T13:20:08Z" level=info msg="couldn't ensure RBAC in target namespaces" csv=strimzi-cluster-operator.v0.23.0 error="no owned roles found" id=E9gIU namespace=operators phase=Pending
time="2021-06-18T13:20:08Z" level=info msg="checking strimzi-cluster-operator.v0.23.0"
time="2021-06-18T13:20:08Z" level=info msg="csv in operatorgroup" csv=strimzi-cluster-operator.v0.23.0 id=BVRYi namespace=operators opgroup=global-operators phase=Pending
time="2021-06-18T13:20:08Z" level=info msg="requirements were not met" csv=strimzi-cluster-operator.v0.23.0 id=BVRYi namespace=operators phase=Pending
time="2021-06-18T13:20:09Z" level=info msg="couldn't ensure RBAC in target namespaces" csv=strimzi-cluster-operator.v0.23.0 error="no owned roles found" id=pCje8 namespace=operators phase=Pending
time="2021-06-18T13:20:09Z" level=info msg="csv in operatorgroup" csv=strimzi-cluster-operator.v0.23.0 id=PaDpL namespace=operators opgroup=global-operators phase=Pending
time="2021-06-18T13:20:09Z" level=info msg="requirements were not met" csv=strimzi-cluster-operator.v0.23.0 id=PaDpL namespace=operators phase=Pending
time="2021-06-18T13:20:09Z" level=info msg="couldn't ensure RBAC in target namespaces" csv=strimzi-cluster-operator.v0.23.0 error="no owned roles found" id=VAnqP namespace=operators phase=Pending
time="2021-06-18T13:20:09Z" level=info msg="csv in operatorgroup" csv=strimzi-cluster-operator.v0.23.0 id=BCYdg namespace=operators opgroup=global-operators phase=Pending
time="2021-06-18T13:20:10Z" level=info msg="requirements were not met" csv=strimzi-cluster-operator.v0.23.0 id=BCYdg namespace=operators phase=Pending
time="2021-06-18T13:20:10Z" level=info msg="couldn't ensure RBAC in target namespaces" csv=strimzi-cluster-operator.v0.23.0 error="no owned roles found" id=9mN3Z namespace=operators phase=Pending
time="2021-06-18T13:20:10Z" level=info msg="csv in operatorgroup" csv=strimzi-cluster-operator.v0.23.0 id=NzjLh namespace=operators opgroup=global-operators phase=Pending
time="2021-06-18T13:20:10Z" level=info msg="requirements were not met" csv=strimzi-cluster-operator.v0.23.0 id=NzjLh namespace=operators phase=Pending
time="2021-06-18T13:20:10Z" level=info msg="couldn't ensure RBAC in target namespaces" csv=strimzi-cluster-operator.v0.23.0 error="no owned roles found" id=ftDce namespace=operators phase=Pending
time="2021-06-18T13:20:10Z" level=info msg="csv in operatorgroup" csv=strimzi-cluster-operator.v0.23.0 id=E8pdw namespace=operators opgroup=global-operators phase=Pending
time="2021-06-18T13:20:10Z" level=info msg="requirements were not met" csv=strimzi-cluster-operator.v0.23.0 id=E8pdw namespace=operators phase=Pending
time="2021-06-18T13:20:10Z" level=info msg="couldn't ensure RBAC in target namespaces" csv=strimzi-cluster-operator.v0.23.0 error="no owned roles found" id=CYH1Y namespace=operators phase=Pending
time="2021-06-18T13:20:10Z" level=info msg="csv in operatorgroup" csv=strimzi-cluster-operator.v0.23.0 id=DQS9B namespace=operators opgroup=global-operators phase=Pending
time="2021-06-18T13:20:10Z" level=info msg="requirements were not met" csv=strimzi-cluster-operator.v0.23.0 id=DQS9B namespace=operators phase=Pending
time="2021-06-18T13:20:11Z" level=info msg="couldn't ensure RBAC in target namespaces" csv=strimzi-cluster-operator.v0.23.0 error="no owned roles found" id=ZDc2e namespace=operators phase=Pending
time="2021-06-18T13:20:11Z" level=info msg="csv in operatorgroup" csv=strimzi-cluster-operator.v0.23.0 id=LA+/w namespace=operators opgroup=global-operators phase=Pending
time="2021-06-18T13:20:11Z" level=info msg="requirements were not met" csv=strimzi-cluster-operator.v0.23.0 id=LA+/w namespace=operators phase=Pending
time="2021-06-18T13:20:11Z" level=info msg="couldn't ensure RBAC in target namespaces" csv=strimzi-cluster-operator.v0.23.0 error="no owned roles found" id=R40n9 namespace=operators phase=Pending
time="2021-06-18T13:20:12Z" level=info msg="csv in operatorgroup" csv=strimzi-cluster-operator.v0.23.0 id=Jq/me namespace=operators opgroup=global-operators phase=Pending
time="2021-06-18T13:20:12Z" level=info msg="requirements were not met" csv=strimzi-cluster-operator.v0.23.0 id=Jq/me namespace=operators phase=Pending
time="2021-06-18T13:20:12Z" level=info msg="couldn't ensure RBAC in target namespaces" csv=strimzi-cluster-operator.v0.23.0 error="no owned roles found" id=D4uoa namespace=operators phase=Pending

This looks interesting:

time="2021-06-18T13:20:07Z" level=info msg="requirements were not met" csv=strimzi-cluster-operator.v0.23.0 id=O7B9v namespace=operators phase=Pending
time="2021-06-18T13:20:08Z" level=info msg="couldn't ensure RBAC in target namespaces" csv=strimzi-cluster-operator.v0.23.0 error="no owned roles found" id=E9gIU namespace=operators phase=Pending

[scholzj]

That is still weird ... not sure what does it mean really and why it has issues creating the roles.

[Aldjinn]

The k8s cluster is an AKS version 1.18.17.

[Aldjinn]

Hi @scholzj,
I was able to resolve the issue.

Removing the "latest" version, adding the last known functional version (for me: 0.21.1), clean up some custom resources and then let the operator do the work (issuing new certs):

# remove
kubectl delete -f https://operatorhub.io/install/strimzi-kafka-operator.yaml

# use old version
kubectl create -f https://operatorhub.io/install/strimzi-0.21.x/strimzi-kafka-operator.yaml

Then remove 0.23.x artifacts from

• InstallPlan
• Subscriptions
• ClusterServerVersions

After that, the 0.21.1 should be up and running:

aldjinn@wsl1:/mnt/c/tmp » kubectl get csv -n operators                                                                                                                                          130 ↵
NAME                               DISPLAY   VERSION   REPLACES   PHASE
strimzi-cluster-operator.v0.21.1   Strimzi   0.21.1               Succeeded

After a few seconds, the certificates are getting renewed and the zookeeper and kafka pods are restarting.

A look into the certs shows a new generation (was 0 before):