Kafka brokers not restarted after external TLS listener certificate renewed

[paulpozicheun]

Hello there,
I've configured my Kafka resource to use my own certificates for internal TLS listeners ans for external TLS listeners using a route

  Kafka:
    Listeners:
      External:
        Configuration:
          Broker Cert Chain And Key:
            Certificate:  tls.crt
            Key:          tls.key
            Secret Name:  net3-eph-listeners-external-route
        Overrides:
          Bootstrap:
            Host:  net3-eph-kafka-bootstrap-kafka-test.app.net3.tst.westeurope.tstcdr.az.xxx.net
          Brokers:
            Broker:  0
            Host:    net3-eph-kafka-0-kafka-test.app.net3.tst.westeurope.tstcdr.az.xxx.net
            Broker:  1
            Host:    net3-eph-kafka-1-kafka-test.app.net3.tst.westeurope.tstcdr.az.xxx.net
        Type:        route
      Plain:
      Tls:
        Configuration:
          Broker Cert Chain And Key:
            Certificate:  tls.crt
            Key:          tls.key
            Secret Name:  net3-eph-listeners-tls

When internal TLS certificate is renewed, dedicated secret is updated and a rolling update is automatically performed so that Kafka broker pods are restarted.
When certificate of external TLS route is renewed, dedicated secret is updated BUT no rolling update is done.
I have to manually restart the Kafka brokers to get my new certificate taken into account.
Can someone tell me if that is normal and/or if my configuration is not good ?
Thanks for your help.
Paul

[scholzj]

What Strimzi version are you using? I think the code is the same for both. So it is a bit weird if it would happen for one listener and not for the other.

[scholzj]

I guess DEBUG logs might be helpful for this. I tried it with the latest builds from the main branch and it seems to work fine for me. But I'm not aware of any changes in this area since 0.20, so I'm not sure whether this is some fixed bug or something else.

[scholzj]

Just FYI: Tried it once again with main and it seems to work fine. At least in the last versions, the thing to watch is the annotation strimzi.io/custom-listener-cert-thumbprints on the Kafka broker pods. That should contain a hash of the certificates and should update when you update the secret. And the updated annotation is what triggers the rolling update. So check it out. It should contain something like this: strimzi.io/custom-listener-cert-thumbprints: {tls=vfgKejF+A38qe2TFsMp5xUVMetKb7bJ/27wzMJZHw2I=}. If you do not have this annotation, then probably the code changed more than I expected since 0.20 and there was probably some bug.

[0xf10413]

Hi there,
I thought the question was interesting, so i tried to reproduce it on my laptop with minikube.

Here's the reproducer i ended up with:
reproducer.zip

Basically, set up your minikube cluster (minikube start), and then run install_everything.sh.
This will create a dedicated strimzi namespace for the experiment, install strimzi via the 0.20.1 helm chart, then request a kafka cluster + an external listener with a custom secret via another helm chart.

The script will create a CA and two pairs key/cert. You can choose which keypair ends in the secret via an helm parameter.

(since i'm relying on LoadBalancers, you need to run minikube tunnel in a separate shell for the LoadBalancers to get external IPs. If you don't do it, the kafka pods won't spawn)

I added commands in the script to see which certificate is in the secret, which certificate is used by kafka, and a command to update the secret.

It seems to work on my machine:

2021-05-19 21:04:31 INFO  KafkaRoller:500 - Reconciliation #0(watch) Kafka(strimzi/ephemeral-cluster): Pod 0 needs to be restarted. Reason: [Pod has old generation, custom certificate one or more listeners changed]
2021-05-19 21:04:32 INFO  PodOperator:65 - Rolling update of strimzi/ephemeral-cluster-kafka: Rolling pod ephemeral-cluster-kafka-0

(although somehow, the strimzi pod crashed many times, making the reconciliation take some time)

@paulpozicheun since it seems that this is working in 0.20.1, i see two possibilities:

1. whatever tool you're using is doing something different from what helm is doing, and strimzi fails to pick up that change (e.g. i'm not sure helm is doing a patch)
2. there's something else in your environment preventing strimzi from reacting (i'm using k8s server 1.20.2)

Could you try my reproducer ? It might help locating more precisely where your issue is.

Cheers!

[paulpozicheun]

Hello Jakub and Flo,
Thanks for all your good remarks.
Considering your replies, I made a few tests in my environment.

1. I changed the secret of my route listener to use a hand made and not the one from my tool

Then I edited the secret to manually change the certificate public key
It worked in that Strimzi performed a rolling update of my broker pod
Here are the interesting logs (IMO) from Strimzi Cluster Operator

2021-05-21 07:58:50 DEBUG StatefulSetDiff:102 - StatefulSet kafka-test/net3-eph-kafka differs: {"op":"replace","path":"/spec/template/metadata/annotations/strimzi.io~1custom-listener-cert-thumbprints","value":"{external=jDA2OsFtcS2dLNWCcCfOU9GB0f9Yym5XJG9guXD+CDE=, tls=PFZ0SATOOn5HGcStJaMzMv0lgDWxPU8xa1G15xwovbo=}"}
2021-05-21 07:58:50 DEBUG KafkaAssemblyOperator:3195 - Rolling update of kafka-test/net3-eph-kafka: pod net3-eph-kafka-0 has strimzi.io/custom-listener-cert-thumbprints={external=JeoQ1/DUWCuxM4bc6FTy2Ij1ha23YpGW/QVoQhMYUHA=, tls=PFZ0SATOOn5HGcStJaMzMv0lgDWxPU8xa1G15xwovbo=}; sts has strimzi.io/custom-listener-cert-thumbprints={external=jDA2OsFtcS2dLNWCcCfOU9GB0f9Yym5XJG9guXD+CDE=, tls=PFZ0SATOOn5HGcStJaMzMv0lgDWxPU8xa1G15xwovbo=}
2021-05-21 07:58:50 DEBUG KafkaAssemblyOperator:3246 - Reconciliation #8(timer) Kafka(kafka-test/net3-eph): Rolling pod net3-eph-kafka-0 due to [Pod has old generation, custom certificate one or more listeners changed]

We can see that statefulset annotation has been updated with new value of custom-listener-cert-thumbprints for external and that leaded to rolling update because of custom certificate change

2. I re-used my tool made secret for route listener and looked for same logs

So, of course, no rolling update occured, and here are the logs I found from Strimzi Cluster Operator

2021-05-20 14:14:48 DEBUG KafkaAssemblyOperator:3195 - Rolling update of kafka-test/net3-eph-kafka: pod net3-eph-kafka-0 has strimzi.io/custom-listener-cert-thumbprints={external=e0nGS+YFw6a7EI3mTdWPX1k+cRT5qvfA4mu5VhCitJE=, tls=PFZ0SATOOn5HGcStJaMzMv0lgDWxPU8xa1G15xwovbo=}; sts has strimzi.io/custom-listener-cert-thumbprints={external=e0nGS+YFw6a7EI3mTdWPX1k+cRT5qvfA4mu5VhCitJE=, tls=PFZ0SATOOn5HGcStJaMzMv0lgDWxPU8xa1G15xwovbo=}
2021-05-20 14:18:45 DEBUG StatusDiff:44 - Status differs: {"op":"replace","path":"/listeners/2/certificates/0","value":"-----BEGIN CERTIFICATE-----\nMIIF6TCCA9GgAwIBAgIQKrp29HJJtLRFx/QdGUSkHDANBgkqhkiG9w0BAQsFADBL...
2021-05-20 14:18:45 DEBUG StatusDiff:45 - Current Status path /listeners/2/certificates/0 has value "-----BEGIN CERTIFICATE-----\nMIIF6TCCA9GgAwIBAgIQKrp29HJJtLRFx/QdGUSkHDANBgkqhkiG9w0BAQsFADBL
2021-05-20 14:18:45 DEBUG StatusDiff:46 - Desired Status path /listeners/2/certificates/0 has value "-----BEGIN CERTIFICATE-----\nMIIF6TCCA9GgAwIBAgIQKrp29HJJtLRFx/QdGUSkHDANBgkqhkiG9w0BAQsFADBL
2021-05-20 14:18:45 DEBUG AbstractOperator:313 - Reconciliation #9(timer) Kafka(kafka-test/net3-eph): Completed status update
2021-05-20 14:18:48 DEBUG KafkaAssemblyOperator:3195 - Rolling update of kafka-test/net3-eph-kafka: pod net3-eph-kafka-0 has strimzi.io/custom-listener-cert-thumbprints={external=e0nGS+YFw6a7EI3mTdWPX1k+cRT5qvfA4mu5VhCitJE=, tls=PFZ0SATOOn5HGcStJaMzMv0lgDWxPU8xa1G15xwovbo=}; sts has strimzi.io/custom-listener-cert-thumbprints={external=e0nGS+YFw6a7EI3mTdWPX1k+cRT5qvfA4mu5VhCitJE=, tls=PFZ0SATOOn5HGcStJaMzMv0lgDWxPU8xa1G15xwovbo=}

We can see that custom-listener-cert-thumbprints for external is never re-computed, and instead of StatefulSetDiff we got the StatusDiff logs
Strimzi did find that the certificate has changed, but I don't understand to which ressource this StatusDiff applies.

Thanks again for your time, hoping these lines could give you a brilliant idea to help.

Regards,
Paul

[scholzj]

@paulpozicheun I guess you will need to give us some examples to reproduce it. From the small log snippets, it looks like your public key changed but the thumbprint remained the same. But that would be weird. I guess to reproduce, we would need:

• The Kafka CR YAML
• Examples of the before and after Secrets with the custom cert which does not trigger the rolling update (as YAML again)