Kafka cluster keeps restarting by Operator

[zkutasi]

Hi,

I have seen there are similar questions here already, but I wanted to post my log to ask why my cluster is constantly kept restarting by the Kafka operator? For me the reason is not obvious.

operator.log

My current Kafka config (I am testing Strimzi currently):

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: strimzi-kafka-cluster
spec:
  clientsCa:
    generateCertificateAuthority: false
  kafka:
    jvmOptions:
      javaSystemProperties:
        - name: javax.net.debug
          value: ssl
    version: 3.0.0
    replicas: 3
    listeners:
      - name: intp
        port: 9092
        type: internal
        tls: false
      - name: inttls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls
        configuration:
          brokerCertChainAndKey:
            secretName: strimzi-kafka-internal-server-cert-secret
            certificate: srvcert.pem
            key: srvprivkey.pem
      - name: extp
        port: 9094
        type: loadbalancer
        tls: false
        configuration:
          bootstrap:
            annotations:
              external-dns.alpha.kubernetes.io/hostname: bootstrap.strimzi.kafka.ezolkut.se
              external-dns.alpha.kubernetes.io/ttl: "60"
          brokers:
          - broker: 0
            advertisedHost: broker0.strimzi.kafka.ezolkut.se
            annotations:
              external-dns.alpha.kubernetes.io/hostname: broker0.strimzi.kafka.ezolkut.se
              external-dns.alpha.kubernetes.io/ttl: "60"
          - broker: 1
            advertisedHost: broker1.strimzi.kafka.ezolkut.se
            annotations:
              external-dns.alpha.kubernetes.io/hostname: broker1.strimzi.kafka.ezolkut.se
              external-dns.alpha.kubernetes.io/ttl: "60"
          - broker: 2
            advertisedHost: broker2.strimzi.kafka.ezolkut.se
            annotations:
              external-dns.alpha.kubernetes.io/hostname: broker2.strimzi.kafka.ezolkut.se
              external-dns.alpha.kubernetes.io/ttl: "60"
      - name: exttls
        port: 9095
        type: loadbalancer
        tls: true
        authentication:
          type: tls
        configuration:
          brokerCertChainAndKey:
            secretName: strimzi-kafka-external-sip-tls-server-cert-secret
            certificate: srvcert.pem
            key: srvprivkey.pem
          bootstrap:
            annotations:
              external-dns.alpha.kubernetes.io/hostname: bootstrap.strimzi.kafka.ezolkut.se
              external-dns.alpha.kubernetes.io/ttl: "60"
          brokers:
          - broker: 0
            advertisedHost: broker0.strimzi.kafka.ezolkut.se
            annotations:
              external-dns.alpha.kubernetes.io/hostname: broker0.strimzi.kafka.ezolkut.se
              external-dns.alpha.kubernetes.io/ttl: "60"
          - broker: 1
            advertisedHost: broker1.strimzi.kafka.ezolkut.se
            annotations:
              external-dns.alpha.kubernetes.io/hostname: broker1.strimzi.kafka.ezolkut.se
              external-dns.alpha.kubernetes.io/ttl: "60"
          - broker: 2
            advertisedHost: broker2.strimzi.kafka.ezolkut.se
            annotations:
              external-dns.alpha.kubernetes.io/hostname: broker2.strimzi.kafka.ezolkut.se
              external-dns.alpha.kubernetes.io/ttl: "60"
    config:
      auto.create.topics.enable: false
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.0"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 10Gi
        deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 10Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator:
      secretPrefix: strimzi-kafka-cluster-user-

[scholzj]

Well, in the log you shared, there were 5 reconciliations but only one seemed to have triggered rolling update. I'm not sure that provides enough data to make any conclusions. But the rolling update in the reconciliation number 5 seemed to have been triggered by this:

2022-02-09 00:13:25 DEBUG KafkaAssemblyOperator:3686 - Reconciliation #5(timer) Kafka(ezolkut-strimzi/strimzi-kafka-cluster): Rolling update of ezolkut-strimzi/strimzi-kafka-cluster-kafka: pod strimzi-kafka-cluster-kafka-2 has strimzi.io/custom-listener-cert-thumbprints={exttls=c7admaGPHdF5HWgMhmgWRolM79kMQ0gUgJyVJfnLuiU=, inttls=hFQK2jEDEf3HNQACDZeEUsCfBNpkTSs1nBelmpHg3/o=}; sts has strimzi.io/custom-listener-cert-thumbprints={exttls=c7admaGPHdF5HWgMhmgWRolM79kMQ0gUgJyVJfnLuiU=, inttls=asTXwDR/al8KAmbJH2LbBpD1TKFpKxyozhU/Gv5mKmw=}

If you check the message, it shows the hashstubs for the two custom listener certificates used by the exttls and inttls certificates. And the one for the inttls interface seemed to have changed. That suggests that the certificate got updated and the broker needs to be rolled to use the new certificate => so the operator rolls it. So the strimzi-kafka-internal-server-cert-secret is the right place to lok at what changed and what is managing it in this case. The hashstub which changed is based on the public key, so it means srvcert.pem changed.

[zkutasi]

Thanks @scholzj , today I will try to watch what happens... as this shall not happen. This cert is made by our kubernetes-running TLS tool. I will try to check this then... I was just a little bit lost as there are many-many logs stating something is different and they are "patched", but maybe those are not triggering a rollout. I will make myself more familiar with the logs in the upcoming days :)

UPDATE: Thank you very much... I have found the issue with our tool's config, it was set up to renew the certs with a TTL of 3600 and Leadtime of 3000, so technically the renewal was done in every 10 minutes, which is crazy ineffective IMO. I raised this issue on our side. Thanks once again for pointing out the one line that I had to look at.