operator keeps restarting the brokers continuously during the listener cert renewal

[hari819]

@scholzj ,

we are using 0.31 version of strimzi , using letsencrypt for providing listener certificates , recently when the cert-manager renewed the certificate , we observed that operator is in a loop and restarting the brokers , we have managed to get this sorted by restarting the kafka cluster by removing kafka object in strimzipodsets and restarting operator .

i have exec into kafka pods and observed that the renewed cert file is present in all the three brokers , but in the operator logs we observed this in a loop,

2023-04-02 07:50:15 DEBUG StatusDiff:39 - Status differs: {"op":"replace","path":"/listeners/1/certificates/0","value":"-----BEGIN CERTIFICATE 

2023-04-02 07:50:15 DEBUG StatusDiff:40 - Current Status path /listeners/1/certificates/0 has value "-----BEGIN CERTIFICATE-----

please can you help , do we need full logs for this ? I can provide by removing certs and dns names ,

Thanks,
Hari

[scholzj]

Sorry, it is not really clear from your description what the issue is or what you actually did. There is no kafka object in strimzipodsets. And the two lines of log you shared do not cause any broker restarts.

[hari819]

Sorry for being not clear,

the issue is the operator was continuously restarting the kafka pods during the listener certificate renewal .
we had to restart kafka to stop the brokers getting restarted ,

so i scaled down operator first , then deleted the kafka-kafka object shown below and scaled up the operator again , and the new kafka broker pods started fine and no restarts were observed later ,

[hari819]

I would like to know why even after the renewed cert was present when the broker was restarted for the first time , the logs show some diff in the cert from some path /listeners ,

2023-04-02 08:03:54 DEBUG StatusDiff:40 - Current Status path /listeners has value [{"type":"plain","name":"plain","addresses":[{"host":"kafka-kafka-bootstrap.kafka.svc","port":9092}],"bootstrapServers":"kafka-kafka-bootstrap.kafka.svc:9092"},{"type":"external","name":"external","addresses":[{"host":"bootstrap.kafka.kafka.xyzcorp.com","port":443}],"bootstrapServers":"bootstrap.kafka.kafka.xyzcorp.com:443","certificates":["-----BEGIN CERTIFICATE

the above log shows new renewed cert after the string "BEGIN CERTIFICATE"

2023-04-02 08:29:41 DEBUG StatusDiff:39 - Status differs: {"op":"add","path":"/listeners","value":[{"type":"plain","name":"plain","addresses":[{"host":"kafka-kafka-bootstrap.kafka.svc","port":9092}],"bootstrapServers":"kafka-kafka-bootstrap.kafka.svc:9092"},{"type":"external","name":"external","addresses":[{"host":"bootstrap.kafka.kafka.xyzcorp.com","port":443}],"bootstrapServers":"bootstrap.kafka.kafka.xyzcorp.com:443","certificates":["-----BEGIN CERTIFICATE-----

this one is old certificate ,

and we can see the log being repeated , and the kafka pods were continuously restarting because of this ,

[scholzj]

And a version of Strimzi you are using would be useful too.

[hari819]

TBH, I don't know. Have you collected the Pod YAMLs as I suggested? You should probably also share the full Kafka CR.

the cluster was restarted after the incident , please is there a way i can collect these now?

i have added the kafka CR below ,

kind: Kafka
metadata:
  creationTimestamp: '2023-01-24T07:16:52Z'
  generation: 1

  name: kafka
  namespace: kafka

spec:
  clientsCa:
    generateCertificateAuthority: false
  clusterCa:
    generateCertificateAuthority: false
  entityOperator:
    template:
      pod:
        metadata:
          annotations:
            prometheus.io/path: /metrics
            prometheus.io/port: 9404
            prometheus.io/scrape: 'true'
    userOperator:
      logging:
        loggers:
          rootLogger.level: DEBUG
        type: inline
  kafka:
    config:
      default.replication.factor: 1
      inter.broker.protocol.version: 3.2.3
      log.message.format.version: 3.2.3
      num.partitions: 1
      offsets.topic.replication.factor: 1
      transaction.state.log.min.isr: 1
      transaction.state.log.replication.factor: 1
    jvmOptions:
      '-Xms': 2048m
      '-Xmx': 2048m
    listeners:
      - name: plain
        port: 9092
        tls: false
        type: internal
      - authentication:
          type: tls
        configuration:
          bootstrap:
            annotations:
              external-dns.alpha.kubernetes.io/target: ingress.orbit.xyzcorp.cloud
              external-dns.alpha.kubernetes.io/ttl: '30'
            host: bootstrap.kafka.kafka.orbit.xyzcorp.cloud
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: tls-secret-kafka
          brokers:
            - annotations:
                external-dns.alpha.kubernetes.io/target: ingress.orbit.xyzcorp.cloud
                external-dns.alpha.kubernetes.io/ttl: '30'
              broker: 0
              host: broker-0.kafka.kafka.orbit.xyzcorp.cloud
            - annotations:
                external-dns.alpha.kubernetes.io/target: ingress.orbit.xyzcorp.cloud
                external-dns.alpha.kubernetes.io/ttl: '30'
              broker: 1
              host: broker-1.kafka.kafka.orbit.xyzcorp.cloud
            - annotations:
                external-dns.alpha.kubernetes.io/target: ingress.orbit.xyzcorp.cloud
                external-dns.alpha.kubernetes.io/ttl: '30'
              broker: 2
              host: broker-2.kafka.kafka.orbit.xyzcorp.cloud
        name: external
        port: 9094
        tls: true
        type: ingress
    livenessProbe:
      failureThreshold: 3
      initialDelaySeconds: 200
      periodSeconds: 2
      timeoutSeconds: 10
    logging:
      loggers:
        kafka.root.logger.level: DEBUG
      type: inline
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: kafka-metrics-config.yml
          name: kafka-metrics
    readinessProbe:
      initialDelaySeconds: 200
      timeoutSeconds: 10
    replicas: 3
    resources:
      requests:
        cpu: 7
        memory: 7Gi
    storage:
      type: jbod
      volumes:
        - class: cfs-ssd
          deleteClaim: false
          id: 0
          size: 500Gi
          type: persistent-claim
        - class: cfs-ssd
          deleteClaim: false
          id: 1
          size: 500Gi
          type: persistent-claim
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: node-role.kubernetes.io/kafka
                      operator: In
                      values:
                        - kafka
        metadata:
          annotations:
            prometheus.io/path: /metrics
            prometheus.io/port: 9404
            prometheus.io/scrape: 'true'
    version: 3.2.3
  kafkaExporter:
    groupRegex: .*
    template:
      pod:
        metadata:
          annotations:
            prometheus.io/path: /metrics
            prometheus.io/port: 9404
            prometheus.io/scrape: 'true'
    topicRegex: .*
  zookeeper:
    jvmOptions:
      '-Xms': 2048m
      '-Xmx': 2048m
      javaSystemProperties:
        - name: zookeeper.ssl.hostnameVerification
          value: 'false'
        - name: zookeeper.ssl.quorum.hostnameVerification
          value: 'false'
        - name: zookeeper.electionPortBindRetry
          value: '0'
    livenessProbe:
      initialDelaySeconds: 200
      timeoutSeconds: 10
    logging:
      loggers:
        zookeeper.root.logger: DEBUG
      type: inline
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: zookeeper-metrics-config.yml
          name: kafka-metrics
    readinessProbe:
      initialDelaySeconds: 200
      timeoutSeconds: 10
    replicas: 3
    storage:
      class: cfs-ssd
      deleteClaim: false
      size: 100Gi
      type: persistent-claim
    template:
      pod:
        metadata:
          annotations:
            prometheus.io/path: /metrics
            prometheus.io/port: 9404
            prometheus.io/scrape: 'true'

[hari819]

And a version of Strimzi you are using would be useful too.

0.31.1 version of strimzi

[scholzj]

TBH, I do not see anything suspicious. I will try to reproduce it with my own certificate, but the config looks fine to me. Are you sure the certificate is not changing? The YAMLs of the Pods / PodSets should help as they might point us into the right direction.

[scholzj]

I tried to reproduce it (obviously its not exactly the same with my own certificates etc) - but I do not seem to have any issue.