[Bug]: Strimzi fails to reconcile a Kafka cluster when restarting (and there are zero Statefulset replicas running) #9101

paoloyx · 2023-09-11T15:50:49Z

paoloyx
Sep 11, 2023

Bug Description

Strimzi operator cannot reconcile an existing Kafka/Zookeeper cluster and Statefulset replicas are not scaled up

Steps to reproduce

No response

Expected behavior

I would expect Strimzi operator to scale up Statefulsets

Strimzi version

0.29.0

Kubernetes version

v1.24.14-gke.1400

Installation method

Yaml file via Kustomize

Infrastructure

Google GKE

Configuration files and logs

Log files:

2023-09-11 04:11:39 INFO  AbstractOperator:517 - Reconciliation #49(timer) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): reconciled
2023-09-11 04:11:44 INFO  AbstractOperator:373 - Reconciliation #43(watch) Kafka([REDACTED]/[REDACTED]): Reconciliation is in progress
2023-09-11 04:11:45 ERROR Util:153 - Reconciliation #43(watch) Kafka([REDACTED]/[REDACTED]): Exceeded timeout of 300000ms while waiting for Pods resource [REDACTED]-zookeeper-0 in namespace [REDACTED] to be ready
2023-09-11 04:11:45 ERROR Util:153 - Reconciliation #43(watch) Kafka([REDACTED]/[REDACTED]): Exceeded timeout of 300000ms while waiting for Pods resource [REDACTED]-zookeeper-1 in namespace [REDACTED] to be ready
2023-09-11 04:11:45 ERROR Util:153 - Reconciliation #43(watch) Kafka([REDACTED]/[REDACTED]): Exceeded timeout of 300000ms while waiting for Pods resource [REDACTED]-zookeeper-2 in namespace [REDACTED] to be ready
2023-09-11 04:11:45 ERROR AbstractOperator:247 - Reconciliation #43(watch) Kafka([REDACTED]/[REDACTED]): createOrUpdate failed
io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for Pods resource [REDACTED]-zookeeper-0 in namespace [REDACTED] to be ready
        at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:154) ~[io.strimzi.operator-common-0.29.0.jar:0.29.0]
        at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
        at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:69) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
        at java.lang.Thread.run(Thread.java:829) ~[?:?]
2023-09-11 04:11:45 WARN  AbstractOperator:532 - Reconciliation #43(watch) Kafka([REDACTED]/[REDACTED]): Failed to reconcile
io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for Pods resource [REDACTED]-zookeeper-0 in namespace [REDACTED] to be ready
        at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:154) ~[io.strimzi.operator-common-0.29.0.jar:0.29.0]
        at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
        at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:69) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
        at java.lang.Thread.run(Thread.java:829) ~[?:?]
2023-09-11 04:11:45 INFO  AbstractOperator:226 - Reconciliation #48(timer) Kafka([REDACTED]/[REDACTED]): Kafka [REDACTED] will be checked for creation or modification
2023-09-11 04:12:45 INFO  AbstractOperator:373 - Reconciliation #48(timer) Kafka([REDACTED]/[REDACTED]): Reconciliation is in progress
2023-09-11 04:13:38 INFO  ClusterOperator:128 - Triggering periodic reconciliation for namespace *

Additional context

We're using the kube-green project to implement nigthly cluster shutdowns in our NON-production environments.

As kube-green does not manage Statefulsets, we implemented a simple solution by shutting down them with CronJobs. It works as expected.

When the cluster wakes up, all deployments are scaled back to their original number of replicas. Among them are some operator pods (i.e. strimzi, prometheus, etc. etc.)

Other operator can successfully scale up Statefulsets they manage, but it seems that Strimzi can't do it...it loops over this error about Exceeded timeout of 300000ms while waiting for Pods....
While searching for similar bugs in this repo I stumbled upon some interesting ones (for example this: https://github.com/orgs/strimzi/discussions/8533)

Can anyone help? Thanks!

scholzj · 2023-09-11T15:54:06Z

scholzj
Sep 11, 2023
Maintainer

You probably have to share more information. In particular:

What exactly are you doing with the Strimzi StatefulSets as that does not seem obvious
Full log from the Cluster Operator
The state it gets into after the recovery. Ii.e. how do the StatefulSets look like, how do the Pods look like etc.

0 replies

paoloyx · 2023-09-12T08:20:48Z

paoloyx
Sep 12, 2023
Author

Hi @scholzj, thanks for you very quick reply...here the is the events unfolding along the timeline:

At t0 (at 22:00 PM of weekdays) the kube-green controller shuts down many deployments into some target namespaces, and Strimzi controller is one of them
At t1 (~5 minutes later) some CronJobs schedule pods that shut down a set of Statefulsets (and the ones managed by Strimzi controller are among them, I mean, Kafka and Zookeeper)
At t2 (8 hours later, so at 06:00AM) the kube-green controller wakes up and scales back all previously shut-down deployments, Strimzi controller included

I was expecting that controller to notice that Kafka and Zookeeper Statefulset are at zero replicas, and to scale them back up the desired number or replicas declared into the Kafka custom resource, but that is not happening, we receive errors.

Here are the data I've been able to collect, hope that is enough. All data below is from t2 onwards, so when the Strimzi controller has been restored

`Kafka` resource

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  labels:
    app.kubernetes.io/instance: kafka-stack
  name: [REDACTED]
  namespace: [REDACTED]
spec:
  kafka:
    authorization:
      superUsers:
      - [REDACTED]
      type: simple
    config:
      auto.create.topics.enable: false
      default.replication.factor: 3
      delete.topic.enable: true
      log.retention.bytes: -1
      log.retention.hours: 168
      min.insync.replicas: 2
      num.partitions: 3
      offsets.topic.replication.factor: 3
      transaction.state.log.min.isr: 2
      transaction.state.log.replication.factor: 3
    jmxOptions:
      authentication:
        type: password
    listeners:
    - authentication:
        type: scram-sha-512
      configuration:
        bootstrap:
          annotations:
            kubernetes.io/ingress.class: nginx
          host: [REDACTED]
        brokerCertChainAndKey:
          certificate: tls.crt
          key: tls.key
          secretName: [REDACTED]
        brokers:
        - annotations:
            kubernetes.io/ingress.class: nginx
          broker: 0
          host: [REDACTED]
        - annotations:
            kubernetes.io/ingress.class: nginx
          broker: 1
          host: [REDACTED]
        - annotations:
            kubernetes.io/ingress.class: nginx
          broker: 2
          host: [REDACTED]
      name: external
      port: 9092
      tls: true
      type: ingress
    - authentication:
        type: scram-sha-512
      name: internal
      port: 9093
      tls: false
      type: internal
    replicas: 3
    resources:
      requests:
        cpu: 500m
        memory: 3Gi
    storage:
      class: ssd
      deleteClaim: false
      size: 110Gi
      type: persistent-claim
    template:
      pod:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: kafka
                    strimzi.io/cluster: [REDACTED]
                topologyKey: kubernetes.io/hostname
              weight: 100
    version: 3.1.0
  zookeeper:
    replicas: 3
    resources:
      requests:
        cpu: 10m
        memory: 750Mi
    storage:
      class: ssd
      deleteClaim: false
      size: 5Gi
      type: persistent-claim
    template:
      pod:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: zookeeper
                    strimzi.io/cluster: [REDACTED]
                topologyKey: kubernetes.io/hostname
              weight: 100
        metadata:
          labels:
            sidecar.istio.io/inject: "false"

Statefulsets

Statefulsets' yaml files have been taken without the Status field, I know that it's not perfect, in case I'll put them again with all proper data

Anyway, here is the Zookeeper Statefulset

apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    strimzi.io/storage: '{"type":"persistent-claim","size":"5Gi","class":"ssd"}'
  labels:
    [...]
  namespace: [REDACTED]
spec:
  podManagementPolicy: Parallel
  replicas: 0
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      strimzi.io/cluster: [REDACTED]
      strimzi.io/kind: Kafka
      strimzi.io/name: [REDACTED]-zookeeper
  serviceName: [REDACTED]-zookeeper-nodes
  template:
    metadata:
      annotations:
        strimzi.io/cluster-ca-cert-generation: "0"
        strimzi.io/generation: "2"
        strimzi.io/logging-hash: 0f057cb0
      creationTimestamp: null
      labels:
        [...]
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: zookeeper
                  strimzi.io/cluster: [REDACTED]
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - command:
        - /opt/kafka/zookeeper_run.sh
        env:
        - name: ZOOKEEPER_METRICS_ENABLED
          value: "false"
        - name: ZOOKEEPER_SNAPSHOT_CHECK_ENABLED
          value: "true"
        - name: STRIMZI_KAFKA_GC_LOG_ENABLED
          value: "false"
        - name: STRIMZI_DYNAMIC_HEAP_PERCENTAGE
          value: "75"
        - name: STRIMZI_DYNAMIC_HEAP_MAX
          value: "2147483648"
        - name: ZOOKEEPER_CONFIGURATION
          value: |
            tickTime=2000
            initLimit=5
            syncLimit=2
            autopurge.purgeInterval=1
        image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
        imagePullPolicy: Always
        livenessProbe:
          exec:
            command:
            - /opt/kafka/zookeeper_healthcheck.sh
          failureThreshold: 3
          initialDelaySeconds: 15
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: zookeeper
        ports:
        - containerPort: 2888
          name: tcp-clustering
          protocol: TCP
        - containerPort: 3888
          name: tcp-election
          protocol: TCP
        - containerPort: 2181
          name: tcp-clients
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - /opt/kafka/zookeeper_healthcheck.sh
          failureThreshold: 3
          initialDelaySeconds: 15
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          requests:
            cpu: 10m
            memory: 750Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp
          name: strimzi-tmp
        - mountPath: /var/lib/zookeeper
          name: data
        - mountPath: /opt/kafka/custom-config/
          name: zookeeper-metrics-and-logging
        - mountPath: /opt/kafka/zookeeper-node-certs/
          name: zookeeper-nodes
        - mountPath: /opt/kafka/cluster-ca-certs/
          name: cluster-ca-certs
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 0
      serviceAccount: [REDACTED]-zookeeper
      serviceAccountName: [REDACTED]-zookeeper
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir:
          medium: Memory
          sizeLimit: 5Mi
        name: strimzi-tmp
      - configMap:
          defaultMode: 420
          name: [REDACTED]-zookeeper-config
        name: zookeeper-metrics-and-logging
      - name: zookeeper-nodes
        secret:
          defaultMode: 292
          secretName: [REDACTED]-zookeeper-nodes
      - name: cluster-ca-certs
        secret:
          defaultMode: 292
          secretName: [REDACTED]-cluster-ca-cert
  updateStrategy:
    type: OnDelete
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
      storageClassName: ssd
      volumeMode: Filesystem
    status:
      phase: Pending

Here is the Kafka Statefulset

apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    strimzi.io/kafka-version: 3.1.0
    strimzi.io/storage: '{"type":"persistent-claim","size":"110Gi","class":"ssd"}'
  labels:
   [...]
  name: [REDACTED]-kafka
  namespace: [REDACTED]
spec:
  podManagementPolicy: Parallel
  replicas: 0
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      strimzi.io/cluster: [REDACTED]
      strimzi.io/kind: Kafka
      strimzi.io/name: [REDACTED]-kafka
  serviceName: [REDACTED]-kafka-brokers
  template:
    metadata:
      annotations:
        strimzi.io/broker-configuration-hash: 0d2142b7
        strimzi.io/clients-ca-cert-generation: "0"
        strimzi.io/cluster-ca-cert-generation: "0"
        strimzi.io/custom-listener-cert-thumbprints: '{external=bbe96e1d}'
        strimzi.io/generation: "5"
        strimzi.io/inter-broker-protocol-version: "3.1"
        strimzi.io/kafka-version: 3.1.0
        strimzi.io/log-message-format-version: "3.1"
        strimzi.io/logging-appenders-hash: e893ac9f
        strimzi.io/storage: '{"type":"persistent-claim","size":"110Gi","class":"ssd"}'
      creationTimestamp: null
      labels:
      [...]
      name: [REDACTED]-kafka
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: kafka
                  strimzi.io/cluster: [REDACTED]
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - command:
        - /opt/kafka/kafka_run.sh
        env:
        - name: KAFKA_METRICS_ENABLED
          value: "false"
        - name: STRIMZI_KAFKA_GC_LOG_ENABLED
          value: "false"
        - name: STRIMZI_DYNAMIC_HEAP_PERCENTAGE
          value: "50"
        - name: STRIMZI_DYNAMIC_HEAP_MAX
          value: "5368709120"
        - name: KAFKA_JMX_ENABLED
          value: "true"
        - name: KAFKA_JMX_USERNAME
          valueFrom:
            secretKeyRef:
              key: jmx-username
              name: [REDACTED]-kafka-jmx
        - name: KAFKA_JMX_PASSWORD
          valueFrom:
            secretKeyRef:
              key: jmx-password
              name: [REDACTED]-kafka-jmx
        image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
        imagePullPolicy: Always
        livenessProbe:
          exec:
            command:
            - /opt/kafka/kafka_liveness.sh
          failureThreshold: 3
          initialDelaySeconds: 15
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: kafka
        ports:
        - containerPort: 9090
          name: tcp-ctrlplane
          protocol: TCP
        - containerPort: 9091
          name: tcp-replication
          protocol: TCP
        - containerPort: 9092
          name: tcp-external
          protocol: TCP
        - containerPort: 9093
          name: tcp-internal
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - test
            - -f
            - /var/opt/kafka/kafka-ready
          failureThreshold: 3
          initialDelaySeconds: 15
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          requests:
            cpu: 500m
            memory: 3Gi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/kafka/data
          name: data
        - mountPath: /tmp
          name: strimzi-tmp
        - mountPath: /opt/kafka/cluster-ca-certs
          name: cluster-ca
        - mountPath: /opt/kafka/broker-certs
          name: broker-certs
        - mountPath: /opt/kafka/client-ca-certs
          name: client-ca-cert
        - mountPath: /opt/kafka/custom-config/
          name: kafka-metrics-and-logging
        - mountPath: /var/opt/kafka
          name: ready-files
        - mountPath: /opt/kafka/certificates/custom-external-9092-certs
          name: custom-external-9092-certs
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 0
      serviceAccount: [REDACTED]-kafka
      serviceAccountName: [REDACTED]-kafka
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir:
          medium: Memory
          sizeLimit: 5Mi
        name: strimzi-tmp
      - name: cluster-ca
        secret:
          defaultMode: 292
          secretName: [REDACTED]-cluster-ca-cert
      - name: broker-certs
        secret:
          defaultMode: 292
          secretName: [REDACTED]-kafka-brokers
      - name: client-ca-cert
        secret:
          defaultMode: 292
          secretName: [REDACTED]-clients-ca-cert
      - configMap:
          defaultMode: 420
          name: [REDACTED]-kafka-config
        name: kafka-metrics-and-logging
      - emptyDir:
          medium: Memory
          sizeLimit: 1Ki
        name: ready-files
      - name: custom-external-9092-certs
        secret:
          defaultMode: 292
          items:
          - key: tls.key
            path: tls.key
          - key: tls.crt
            path: tls.crt
          secretName: [REDACTED]
  updateStrategy:
    type: OnDelete
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 110Gi
      storageClassName: ssd
      volumeMode: Filesystem
    status:
      phase: Pending

Pods

There are no pods, Statefulsets are still at zero replicas

Operator logs

Here are the logs from the operator since Pod startup, until the errors we get

Auto-detected KUBERNETES_SERVICE_DNS_DOMAIN: cluster.local
+ exec /usr/bin/tini -w -e 143 -- java -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom --illegal-access=deny -XX:MinRAMPercentage=10 -XX:MaxRAMPercentage=20 -XX:InitialRAMPercentage=10 -classpath lib/io.strimzi.cluster-operator-0.29.0.jar:lib/io.fabric8.kubernetes-model-discovery-5.12.2.jar:lib/io.netty.netty-transport-4.1.77.Final.jar:lib/io.netty.netty-codec-socks-4.1.77.Final.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.13.1.jar:lib/io.netty.netty-handler-proxy-4.1.77.Final.jar:lib/io.strimzi.kafka-oauth-server-0.10.0.jar:lib/io.fabric8.kubernetes-model-common-5.12.2.jar:lib/io.fabric8.kubernetes-model-autoscaling-5.12.2.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.12.6.jar:lib/io.fabric8.openshift-client-5.12.2.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/io.fabric8.openshift-model-machineconfig-5.12.2.jar:lib/org.apache.zookeeper.zookeeper-jute-3.6.3.jar:lib/io.netty.netty-transport-native-epoll-4.1.77.Final-linux-x86_64.jar:lib/io.strimzi.api-0.29.0.jar:lib/org.apache.zookeeper.zookeeper-3.6.3.jar:lib/io.netty.netty-resolver-dns-4.1.77.Final.jar:lib/net.minidev.accessors-smart-2.4.7.jar:lib/io.netty.netty-handler-4.1.77.Final.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.netty.netty-codec-http-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-metrics-5.12.2.jar:lib/io.fabric8.kubernetes-model-flowcontrol-5.12.2.jar:lib/io.fabric8.openshift-model-hive-5.12.2.jar:lib/io.fabric8.kubernetes-model-apps-5.12.2.jar:lib/com.jayway.jsonpath.json-path-2.6.0.jar:lib/net.minidev.json-smart-2.4.7.jar:lib/io.strimzi.kafka-oauth-server-plain-0.10.0.jar:lib/io.fabric8.openshift-model-operator-5.12.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/io.fabric8.openshift-model-clusterautoscaling-5.12.2.jar:lib/org.yaml.snakeyaml-1.27.jar:lib/io.netty.netty-codec-dns-4.1.77.Final.jar:lib/io.strimzi.certificate-manager-0.29.0.jar:lib/io.netty.netty-resolver-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-core-5.12.2.jar:lib/io.strimzi.kafka-oauth-common-0.10.0.jar:lib/io.fabric8.kubernetes-model-admissionregistration-5.12.2.jar:lib/io.fabric8.kubernetes-model-scheduling-5.12.2.jar:lib/io.netty.netty-transport-native-unix-common-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-batch-5.12.2.jar:lib/io.fabric8.kubernetes-model-coordination-5.12.2.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/io.fabric8.kubernetes-model-extensions-5.12.2.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.fabric8.kubernetes-model-events-5.12.2.jar:lib/io.netty.netty-transport-native-epoll-4.1.77.Final.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/io.vertx.vertx-core-4.2.4.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.4.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/io.fabric8.kubernetes-model-apiextensions-5.12.2.jar:lib/io.fabric8.openshift-model-whereabouts-5.12.2.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/io.strimzi.crd-annotations-0.29.0.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.12.6.1.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.netty.netty-codec-4.1.77.Final.jar:lib/io.fabric8.openshift-model-storageversionmigrator-5.12.2.jar:lib/io.fabric8.kubernetes-model-networking-5.12.2.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/com.nimbusds.nimbus-jose-jwt-9.10.jar:lib/io.fabric8.openshift-model-monitoring-5.12.2.jar:lib/com.fasterxml.jackson.core.jackson-core-2.12.6.jar:lib/io.fabric8.kubernetes-client-5.12.2.jar:lib/com.squareup.okhttp3.okhttp-3.12.12.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/io.fabric8.kubernetes-model-certificates-5.12.2.jar:lib/io.fabric8.openshift-model-console-5.12.2.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/io.strimzi.kafka-oauth-client-0.10.0.jar:lib/io.netty.netty-buffer-4.1.77.Final.jar:lib/org.apache.kafka.kafka-clients-3.2.0.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/io.fabric8.openshift-model-installer-5.12.2.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.fabric8.openshift-model-miscellaneous-5.12.2.jar:lib/io.fabric8.kubernetes-model-storageclass-5.12.2.jar:lib/io.netty.netty-codec-http2-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-node-5.12.2.jar:lib/io.strimzi.config-model-0.29.0.jar:lib/io.fabric8.openshift-model-machine-5.12.2.jar:lib/com.github.luben.zstd-jni-1.5.2-1.jar:lib/io.fabric8.openshift-model-5.12.2.jar:lib/io.netty.netty-transport-classes-epoll-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-policy-5.12.2.jar:lib/io.fabric8.kubernetes-model-rbac-5.12.2.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.strimzi.operator-common-0.29.0.jar:lib/io.fabric8.openshift-model-tuned-5.12.2.jar:lib/io.vertx.vertx-micrometer-metrics-4.2.4.jar:lib/io.fabric8.openshift-model-operatorhub-5.12.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.12.6.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/io.netty.netty-common-4.1.77.Final.jar io.strimzi.operator.cluster.Main
2023-09-12 04:00:29 INFO  Main:60 - ClusterOperator 0.29.0 is starting
2023-09-12 04:00:30 INFO  Main:62 - Cluster Operator configuration is ClusterOperatorConfig(namespaces=[*],reconciliationIntervalMs=120000,operationTimeoutMs=300000,connectBuildTimeoutMs=300000,createClusterRoles=false,networkPolicyGeneration=true,versions=versions{3.0.0={proto: 3.0 msg: 3.0 kafka-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.0 connect-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.0 mirrormaker-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.0 mirrormaker2-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.0}, 3.0.1={proto: 3.0 msg: 3.0 kafka-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.1 connect-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.1 mirrormaker-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.1 mirrormaker2-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.1}, 3.1.0={proto: 3.1 msg: 3.1 kafka-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.0 connect-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.0 mirrormaker-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.0 mirrormaker2-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.0}, 3.1.1={proto: 3.1 msg: 3.1 kafka-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.1 connect-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.1 mirrormaker-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.1 mirrormaker2-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.1}, 3.2.0={proto: 3.2 msg: 3.2 kafka-image: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0 connect-image: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0 mirrormaker-image: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0 mirrormaker2-image: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0}},imagePullPolicy=Always,imagePullSecrets=null,operatorNamespace=kafka,operatorNamespaceLabels=null,customResourceSelector=null,featureGates=FeatureGates(controlPlaneListener=true,ServiceAccountPatching=true,UseStrimziPodSets=false,UseKRaft=false),zkAdminSessionTimeoutMs=10000,dnsCacheTtlSec=30,podSetReconciliationOnly=false,podSetControllerWorkQueueSize=1024)
2023-09-12 04:00:34 WARN  PlatformFeaturesAvailability:157 - API Group route.openshift.io is not supported
2023-09-12 04:00:34 WARN  PlatformFeaturesAvailability:157 - API Group build.openshift.io is not supported
2023-09-12 04:00:34 WARN  PlatformFeaturesAvailability:157 - API Group image.openshift.io is not supported
2023-09-12 04:00:34 INFO  Main:85 - Environment facts gathered: PlatformFeaturesAvailability(KubernetesVersion=1.24,OpenShiftRoutes=false,OpenShiftBuilds=false,OpenShiftImageStreams=false)
2023-09-12 04:00:34 INFO  Util:307 - Using config:
	PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
	container: oci
	STRIMZI_IMAGE_PULL_POLICY: Always
	STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE: quay.io/strimzi/kafka-bridge:0.21.5
	KUBERNETES_PORT: tcp://10.175.240.1:443
	STRIMZI_KAFKA_MIRROR_MAKER_IMAGES: 3.0.0=quay.io/strimzi/kafka:0.29.0-kafka-3.0.0
3.0.1=quay.io/strimzi/kafka:0.29.0-kafka-3.0.1
3.1.0=quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
3.1.1=quay.io/strimzi/kafka:0.29.0-kafka-3.1.1
3.2.0=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0

	STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE: quay.io/strimzi/operator:0.29.0
	JAVA_HOME: /usr/lib/jvm/jre-11
	JAVA_OPTS:  -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom  --illegal-access=deny -XX:MinRAMPercentage=10 -XX:MaxRAMPercentage=20 -XX:InitialRAMPercentage=10
	STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
	KUBERNETES_SERVICE_HOST: 10.175.240.1
	JAVA_CLASSPATH: lib/io.strimzi.cluster-operator-0.29.0.jar:lib/io.fabric8.kubernetes-model-discovery-5.12.2.jar:lib/io.netty.netty-transport-4.1.77.Final.jar:lib/io.netty.netty-codec-socks-4.1.77.Final.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.13.1.jar:lib/io.netty.netty-handler-proxy-4.1.77.Final.jar:lib/io.strimzi.kafka-oauth-server-0.10.0.jar:lib/io.fabric8.kubernetes-model-common-5.12.2.jar:lib/io.fabric8.kubernetes-model-autoscaling-5.12.2.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.12.6.jar:lib/io.fabric8.openshift-client-5.12.2.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/io.fabric8.openshift-model-machineconfig-5.12.2.jar:lib/org.apache.zookeeper.zookeeper-jute-3.6.3.jar:lib/io.netty.netty-transport-native-epoll-4.1.77.Final-linux-x86_64.jar:lib/io.strimzi.api-0.29.0.jar:lib/org.apache.zookeeper.zookeeper-3.6.3.jar:lib/io.netty.netty-resolver-dns-4.1.77.Final.jar:lib/net.minidev.accessors-smart-2.4.7.jar:lib/io.netty.netty-handler-4.1.77.Final.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.netty.netty-codec-http-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-metrics-5.12.2.jar:lib/io.fabric8.kubernetes-model-flowcontrol-5.12.2.jar:lib/io.fabric8.openshift-model-hive-5.12.2.jar:lib/io.fabric8.kubernetes-model-apps-5.12.2.jar:lib/com.jayway.jsonpath.json-path-2.6.0.jar:lib/net.minidev.json-smart-2.4.7.jar:lib/io.strimzi.kafka-oauth-server-plain-0.10.0.jar:lib/io.fabric8.openshift-model-operator-5.12.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/io.fabric8.openshift-model-clusterautoscaling-5.12.2.jar:lib/org.yaml.snakeyaml-1.27.jar:lib/io.netty.netty-codec-dns-4.1.77.Final.jar:lib/io.strimzi.certificate-manager-0.29.0.jar:lib/io.netty.netty-resolver-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-core-5.12.2.jar:lib/io.strimzi.kafka-oauth-common-0.10.0.jar:lib/io.fabric8.kubernetes-model-admissionregistration-5.12.2.jar:lib/io.fabric8.kubernetes-model-scheduling-5.12.2.jar:lib/io.netty.netty-transport-native-unix-common-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-batch-5.12.2.jar:lib/io.fabric8.kubernetes-model-coordination-5.12.2.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/io.fabric8.kubernetes-model-extensions-5.12.2.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.fabric8.kubernetes-model-events-5.12.2.jar:lib/io.netty.netty-transport-native-epoll-4.1.77.Final.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/io.vertx.vertx-core-4.2.4.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.4.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/io.fabric8.kubernetes-model-apiextensions-5.12.2.jar:lib/io.fabric8.openshift-model-whereabouts-5.12.2.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/io.strimzi.crd-annotations-0.29.0.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.12.6.1.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.netty.netty-codec-4.1.77.Final.jar:lib/io.fabric8.openshift-model-storageversionmigrator-5.12.2.jar:lib/io.fabric8.kubernetes-model-networking-5.12.2.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/com.nimbusds.nimbus-jose-jwt-9.10.jar:lib/io.fabric8.openshift-model-monitoring-5.12.2.jar:lib/com.fasterxml.jackson.core.jackson-core-2.12.6.jar:lib/io.fabric8.kubernetes-client-5.12.2.jar:lib/com.squareup.okhttp3.okhttp-3.12.12.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/io.fabric8.kubernetes-model-certificates-5.12.2.jar:lib/io.fabric8.openshift-model-console-5.12.2.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/io.strimzi.kafka-oauth-client-0.10.0.jar:lib/io.netty.netty-buffer-4.1.77.Final.jar:lib/org.apache.kafka.kafka-clients-3.2.0.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/io.fabric8.openshift-model-installer-5.12.2.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.fabric8.openshift-model-miscellaneous-5.12.2.jar:lib/io.fabric8.kubernetes-model-storageclass-5.12.2.jar:lib/io.netty.netty-codec-http2-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-node-5.12.2.jar:lib/io.strimzi.config-model-0.29.0.jar:lib/io.fabric8.openshift-model-machine-5.12.2.jar:lib/com.github.luben.zstd-jni-1.5.2-1.jar:lib/io.fabric8.openshift-model-5.12.2.jar:lib/io.netty.netty-transport-classes-epoll-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-policy-5.12.2.jar:lib/io.fabric8.kubernetes-model-rbac-5.12.2.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.strimzi.operator-common-0.29.0.jar:lib/io.fabric8.openshift-model-tuned-5.12.2.jar:lib/io.vertx.vertx-micrometer-metrics-4.2.4.jar:lib/io.fabric8.openshift-model-operatorhub-5.12.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.12.6.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/io.netty.netty-common-4.1.77.Final.jar
	STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
	STRIMZI_HOME: /opt/strimzi
	PWD: /opt/strimzi
	TINI_VERSION: v0.19.0
	KUBERNETES_PORT_443_TCP: tcp://10.175.240.1:443
	STRIMZI_OPERATION_TIMEOUT_MS: 300000
	JAVA_MAIN: io.strimzi.operator.cluster.Main
	KUBERNETES_PORT_443_TCP_ADDR: 10.175.240.1
	STRIMZI_VERSION: 0.29.0
	STRIMZI_DEFAULT_JMXTRANS_IMAGE: quay.io/strimzi/jmxtrans:0.29.0
	STRIMZI_NAMESPACE: *
	STRIMZI_DEFAULT_KANIKO_EXECUTOR_IMAGE: quay.io/strimzi/kaniko-executor:0.29.0
	TINI_SHA256_S390X: 931b70a182af879ca249ae9de87ef68423121b38d235c78997fafc680ceab32d
	STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES: 3.0.0=quay.io/strimzi/kafka:0.29.0-kafka-3.0.0
3.0.1=quay.io/strimzi/kafka:0.29.0-kafka-3.0.1
3.1.0=quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
3.1.1=quay.io/strimzi/kafka:0.29.0-kafka-3.1.1
3.2.0=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0

	STRIMZI_DEFAULT_MAVEN_BUILDER: quay.io/strimzi/maven-builder:0.29.0
	KUBERNETES_PORT_443_TCP_PROTO: tcp
	STRIMZI_DEFAULT_USER_OPERATOR_IMAGE: quay.io/strimzi/operator:0.29.0
	STRIMZI_FEATURE_GATES: 
	KUBERNETES_SERVICE_PORT: 443
	STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
	TINI_SHA256_ARM64: 07952557df20bfd2a95f9bef198b445e006171969499a1d361bd9e6f8e5e0e81
	STRIMZI_DEFAULT_KAFKA_INIT_IMAGE: quay.io/strimzi/operator:0.29.0
	TINI_SHA256_PPC64LE: 3f658420974768e40810001a038c29d003728c5fe86da211cff5059e48cfdfde
	TINI_SHA256_AMD64: 93dcc18adc78c65a028a84799ecf8ad40c936fdfc5f2a57b1acda5a8117fa82c
	KUBERNETES_SERVICE_DNS_DOMAIN: cluster.local
	HOSTNAME: strimzi-cluster-operator-78d7997fd5-f4k4j
	STRIMZI_KAFKA_CONNECT_IMAGES: 3.0.0=quay.io/strimzi/kafka:0.29.0-kafka-3.0.0
3.0.1=quay.io/strimzi/kafka:0.29.0-kafka-3.0.1
3.1.0=quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
3.1.1=quay.io/strimzi/kafka:0.29.0-kafka-3.1.1
3.2.0=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0

	STRIMZI_KAFKA_IMAGES: 3.0.0=quay.io/strimzi/kafka:0.29.0-kafka-3.0.0
3.0.1=quay.io/strimzi/kafka:0.29.0-kafka-3.0.1
3.1.0=quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
3.1.1=quay.io/strimzi/kafka:0.29.0-kafka-3.1.1
3.2.0=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0

	KUBERNETES_PORT_443_TCP_PORT: 443
	KUBERNETES_SERVICE_PORT_HTTPS: 443
	STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
	SHLVL: 0
	HOME: /home/strimzi
	MALLOC_ARENA_MAX: 2
	STRIMZI_OPERATOR_NAMESPACE: kafka

2023-09-12 04:00:34 INFO  ClusterOperator:79 - Creating ClusterOperator for namespace *
2023-09-12 04:00:34 INFO  ClusterOperator:94 - Starting ClusterOperator for namespace *
2023-09-12 04:00:34 INFO  ClusterOperator:113 - Opened watch for Kafka operator
2023-09-12 04:00:35 INFO  ClusterOperator:113 - Opened watch for KafkaMirrorMaker operator
2023-09-12 04:00:35 INFO  ClusterOperator:113 - Opened watch for KafkaConnect operator
2023-09-12 04:00:35 INFO  ClusterOperator:113 - Opened watch for KafkaBridge operator
2023-09-12 04:00:35 INFO  ClusterOperator:113 - Opened watch for KafkaMirrorMaker2 operator
2023-09-12 04:00:35 INFO  ClusterOperator:125 - Setting up periodic reconciliation for namespace *
2023-09-12 04:00:36 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkaconnects' with unstable version 'v1beta2'
2023-09-12 04:00:36 INFO  ClusterOperator:192 - ClusterOperator is now ready (health server listening on 8080)
2023-09-12 04:00:36 INFO  Main:155 - Cluster Operator verticle started in namespace * without label selector
2023-09-12 04:00:37 INFO  AbstractConnectOperator:225 - Reconciliation #3(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): [REDACTED] [REDACTED] was ADDED
2023-09-12 04:00:37 INFO  OperatorWatcher:38 - Reconciliation #2(watch) Kafka([REDACTED]/[REDACTED]): Kafka [REDACTED] in namespace [REDACTED] was ADDED
2023-09-12 04:00:37 INFO  OperatorWatcher:38 - Reconciliation #1(watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): KafkaConnect [REDACTED]-kafka-connect in namespace [REDACTED] was ADDED
2023-09-12 04:00:37 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkas' with unstable version 'v1beta2'
2023-09-12 04:00:37 INFO  AbstractConnectOperator:466 - Reconciliation #3(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): creating/updating connector: reactions-sink
2023-09-12 04:00:37 INFO  AbstractOperator:226 - Reconciliation #2(watch) Kafka([REDACTED]/[REDACTED]): Kafka [REDACTED] will be checked for creation or modification
2023-09-12 04:00:38 INFO  AbstractConnectOperator:225 - Reconciliation #4(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): [REDACTED] [REDACTED] was ADDED
2023-09-12 04:00:38 INFO  AbstractConnectOperator:225 - Reconciliation #5(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): [REDACTED] [REDACTED] was ADDED
2023-09-12 04:00:38 INFO  AbstractConnectOperator:225 - Reconciliation #6(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): [REDACTED] [REDACTED] was ADDED
2023-09-12 04:00:38 INFO  AbstractConnectOperator:225 - Reconciliation #7(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): [REDACTED] [REDACTED] was ADDED
2023-09-12 04:00:38 INFO  AbstractConnectOperator:225 - Reconciliation #8(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): [REDACTED] [REDACTED] was ADDED
2023-09-12 04:00:38 INFO  AbstractConnectOperator:225 - Reconciliation #9(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): [REDACTED] [REDACTED] was ADDED
2023-09-12 04:00:38 INFO  AbstractConnectOperator:225 - Reconciliation #10(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): [REDACTED] [REDACTED] was ADDED
2023-09-12 04:00:38 INFO  AbstractConnectOperator:225 - Reconciliation #11(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): [REDACTED] [REDACTED] was ADDED
2023-09-12 04:00:38 INFO  AbstractConnectOperator:225 - Reconciliation #12(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): [REDACTED] [REDACTED] was ADDED
2023-09-12 04:00:38 INFO  AbstractConnectOperator:225 - Reconciliation #13(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): [REDACTED] [REDACTED] was ADDED
2023-09-12 04:00:38 INFO  AbstractConnectOperator:225 - Reconciliation #14(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): [REDACTED] [REDACTED] was ADDED
2023-09-12 04:00:38 INFO  AbstractConnectOperator:225 - Reconciliation #15(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): [REDACTED] [REDACTED] was ADDED
2023-09-12 04:00:38 INFO  AbstractConnectOperator:225 - Reconciliation #16(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): [REDACTED] [REDACTED] was ADDED
2023-09-12 04:00:38 INFO  AbstractConnectOperator:225 - Reconciliation #17(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): [REDACTED] [REDACTED] was ADDED
2023-09-12 04:00:38 INFO  AbstractConnectOperator:225 - Reconciliation #18(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): [REDACTED] [REDACTED] was ADDED
2023-09-12 04:00:38 INFO  AbstractConnectOperator:225 - Reconciliation #19(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): [REDACTED] [REDACTED] was ADDED
2023-09-12 04:00:38 INFO  AbstractConnectOperator:225 - Reconciliation #20(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): [REDACTED] [REDACTED] was ADDED
2023-09-12 04:00:39 ERROR ContextImpl: - Unhandled exception
io.vertx.core.json.DecodeException: Failed to decode:Unrecognized token 'no': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: (io.netty.buffer.ByteBufInputStream); line: 1, column: 4]
	at io.vertx.core.json.jackson.DatabindCodec.fromParser(DatabindCodec.java:129) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.json.jackson.DatabindCodec.fromBuffer(DatabindCodec.java:99) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.json.JsonObject.fromBuffer(JsonObject.java:948) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.json.JsonObject.<init>(JsonObject.java:85) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.buffer.impl.BufferImpl.toJsonObject(BufferImpl.java:110) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.strimzi.operator.cluster.operator.assembly.KafkaConnectApiImpl.lambda$doGet$6(KafkaConnectApiImpl.java:142) ~[io.strimzi.cluster-operator-0.29.0.jar:0.29.0]
	at io.vertx.core.impl.future.FutureImpl$1.onSuccess(FutureImpl.java:91) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:54) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:81) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.impl.ContextImpl.execute(ContextImpl.java:260) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:22) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:51) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.impl.future.PromiseImpl.tryComplete(PromiseImpl.java:23) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.http.impl.HttpEventHandler.handleEnd(HttpEventHandler.java:79) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.http.impl.HttpClientResponseImpl.handleEnd(HttpClientResponseImpl.java:250) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$new$0(Http1xClientConnection.java:383) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:240) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:130) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleEnd(Http1xClientConnection.java:610) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:71) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.impl.ContextImpl.execute(ContextImpl.java:267) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.http.impl.Http1xClientConnection.handleResponseEnd(Http1xClientConnection.java:844) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.http.impl.Http1xClientConnection.handleHttpMessage(Http1xClientConnection.java:716) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.http.impl.Http1xClientConnection.handleMessage(Http1xClientConnection.java:680) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:156) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:153) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) ~[io.netty.netty-codec-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) ~[io.netty.netty-codec-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[io.netty.netty-handler-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'no': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: (io.netty.buffer.ByteBufInputStream); line: 1, column: 4]
	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2337) ~[com.fasterxml.jackson.core.jackson-core-2.12.6.jar:2.12.6]
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:720) ~[com.fasterxml.jackson.core.jackson-core-2.12.6.jar:2.12.6]
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3594) ~[com.fasterxml.jackson.core.jackson-core-2.12.6.jar:2.12.6]
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3570) ~[com.fasterxml.jackson.core.jackson-core-2.12.6.jar:2.12.6]
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._matchToken2(UTF8StreamJsonParser.java:2905) ~[com.fasterxml.jackson.core.jackson-core-2.12.6.jar:2.12.6]
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._matchNull(UTF8StreamJsonParser.java:2876) ~[com.fasterxml.jackson.core.jackson-core-2.12.6.jar:2.12.6]
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:849) ~[com.fasterxml.jackson.core.jackson-core-2.12.6.jar:2.12.6]
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:762) ~[com.fasterxml.jackson.core.jackson-core-2.12.6.jar:2.12.6]
	at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4684) ~[com.fasterxml.jackson.core.jackson-databind-2.12.6.1.jar:2.12.6.1]
	at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4561) ~[com.fasterxml.jackson.core.jackson-databind-2.12.6.1.jar:2.12.6.1]
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2798) ~[com.fasterxml.jackson.core.jackson-databind-2.12.6.1.jar:2.12.6.1]
	at io.vertx.core.json.jackson.DatabindCodec.fromParser(DatabindCodec.java:126) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	... 53 more
2023-09-12 04:00:39 WARN  VersionUsageUtils:60 - The client is using resource type 'strimzipodsets' with unstable version 'v1beta2'
2023-09-12 04:01:37 INFO  AbstractOperator:373 - Reconciliation #2(watch) Kafka([REDACTED]/[REDACTED]): Reconciliation is in progress
2023-09-12 04:01:37 INFO  AbstractOperator:373 - Reconciliation #3(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): Reconciliation is in progress
2023-09-12 04:02:35 INFO  ClusterOperator:128 - Triggering periodic reconciliation for namespace *
2023-09-12 04:02:35 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkamirrormakers' with unstable version 'v1beta2'
2023-09-12 04:02:36 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkamirrormaker2s' with unstable version 'v1beta2'
2023-09-12 04:02:36 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkabridges' with unstable version 'v1beta2'
2023-09-12 04:02:36 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkarebalances' with unstable version 'v1beta2'
2023-09-12 04:02:37 INFO  AbstractOperator:373 - Reconciliation #2(watch) Kafka([REDACTED]/[REDACTED]): Reconciliation is in progress
2023-09-12 04:02:37 INFO  AbstractOperator:373 - Reconciliation #3(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): Reconciliation is in progress
2023-09-12 04:02:46 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkaconnectors' with unstable version 'v1beta2'
2023-09-12 04:03:37 INFO  AbstractOperator:373 - Reconciliation #2(watch) Kafka([REDACTED]/[REDACTED]): Reconciliation is in progress
2023-09-12 04:03:37 INFO  AbstractOperator:373 - Reconciliation #3(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): Reconciliation is in progress
2023-09-12 04:04:35 INFO  ClusterOperator:128 - Triggering periodic reconciliation for namespace *
2023-09-12 04:04:37 INFO  AbstractOperator:373 - Reconciliation #2(watch) Kafka([REDACTED]/[REDACTED]): Reconciliation is in progress
2023-09-12 04:04:37 INFO  AbstractOperator:373 - Reconciliation #3(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): Reconciliation is in progress
2023-09-12 04:05:37 INFO  AbstractOperator:373 - Reconciliation #2(watch) Kafka([REDACTED]/[REDACTED]): Reconciliation is in progress
2023-09-12 04:05:37 INFO  AbstractOperator:373 - Reconciliation #3(connector-watch) KafkaConnect([REDACTED]/[REDACTED]-kafka-connect): Reconciliation is in progress
2023-09-12 04:05:40 ERROR Util:153 - Reconciliation #2(watch) Kafka([REDACTED]/[REDACTED]): Exceeded timeout of 300000ms while waiting for Pods resource [REDACTED]-zookeeper-0 in namespace [REDACTED] to be ready
2023-09-12 04:05:40 ERROR Util:153 - Reconciliation #2(watch) Kafka([REDACTED]/[REDACTED]): Exceeded timeout of 300000ms while waiting for Pods resource [REDACTED]-zookeeper-1 in namespace [REDACTED] to be ready
2023-09-12 04:05:40 ERROR Util:153 - Reconciliation #2(watch) Kafka([REDACTED]/[REDACTED]): Exceeded timeout of 300000ms while waiting for Pods resource [REDACTED]-zookeeper-2 in namespace [REDACTED] to be ready
2023-09-12 04:05:40 ERROR AbstractOperator:247 - Reconciliation #2(watch) Kafka([REDACTED]/[REDACTED]): createOrUpdate failed
io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for Pods resource [REDACTED]-zookeeper-0 in namespace [REDACTED] to be ready
	at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:154) ~[io.strimzi.operator-common-0.29.0.jar:0.29.0]
	at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:69) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
2023-09-12 04:05:40 INFO  OperatorWatcher:38 - Reconciliation #25(watch) Kafka([REDACTED]/[REDACTED]): Kafka [REDACTED] in namespace [REDACTED] was MODIFIED
2023-09-12 04:05:40 INFO  CrdOperator:113 - Reconciliation #2(watch) Kafka([REDACTED]/[REDACTED]): Status of Kafka [REDACTED] in namespace [REDACTED] has been updated
2023-09-12 04:05:40 WARN  AbstractOperator:532 - Reconciliation #2(watch) Kafka([REDACTED]/[REDACTED]): Failed to reconcile
io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for Pods resource [REDACTED]-zookeeper-0 in namespace [REDACTED] to be ready
	at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:154) ~[io.strimzi.operator-common-0.29.0.jar:0.29.0]
	at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:69) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) ~[io.netty.netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]

Istio

I forgot to say the both the Strimzi controller and the Kafka pods have the Istio sidecar enabled, while Zookeeper pods do not have the sidecar. I suspect that this is a problem anyway and that we may have overlooked the Strimzi best practices in configuring the Kafka cluster

5 replies

scholzj Sep 12, 2023
Maintainer

Now, if I understand you correctly, by shut down a Statefulsets you mean scale it to 0? That does not work properly because it triggers the scale-up routine when you want to restore it. You should delete the StatefulSet instead and have the operator recreate it when you want to run the cluster again.

paoloyx Sep 12, 2023
Author

Yes, the cronJobs scale the Statefulsets to zero...I was expecting Strimzi controller to scale them back to the desired number of replicas, when the controller is restarted at "cluster wake-up".

Not sure I get this point, but it's my lack of knowledge probably

That does not work properly because it triggers the scale-up routine when you want to restore it

If I delete the StatefulSet, the operator will re-create them by using the same PVCs, right? Because PVCs are not deleted when StatefulSets are deleted.

Thank you :)

scholzj Sep 12, 2023
Maintainer

Yes, it will reuse the PVCs.

paoloyx Sep 12, 2023
Author

Ok, I'll try the sts deletion and I'll write back here the results, thanks again

paoloyx Sep 13, 2023
Author

Thanks @scholzj , the StatefulSet deletion did the trick. We're ok now 🙌

paoloyx · 2023-09-12T08:34:46Z

paoloyx
Sep 12, 2023
Author

Something that I forgot also to say:

I tried to restart the Strimzi operator, but without success. The situation was restored only when I issued a manual kubectl scale statefulset to scale up replicas
During the cluster sleep nodes where scaled down by cluster autoscaler, the manual kubectl command above did issue a node scale up

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

[Bug]: Strimzi fails to reconcile a Kafka cluster when restarting (and there are zero Statefulset replicas running) #9101

Uh oh!

{{title}}

Uh oh!

Replies: 3 comments 5 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Strimzi

[Bug]: Strimzi fails to reconcile a Kafka cluster when restarting (and there are zero Statefulset replicas running) #9101

Uh oh!

paoloyx Sep 11, 2023

Bug Description

Steps to reproduce

Expected behavior

Strimzi version

Kubernetes version

Installation method

Infrastructure

Configuration files and logs

Additional context

Replies: 3 comments · 5 replies

Uh oh!

scholzj Sep 11, 2023 Maintainer

Uh oh!

paoloyx Sep 12, 2023 Author

Kafka resource

Statefulsets

Pods

Operator logs

Istio

Uh oh!

scholzj Sep 12, 2023 Maintainer

Uh oh!

paoloyx Sep 12, 2023 Author

Uh oh!

scholzj Sep 12, 2023 Maintainer

Uh oh!

paoloyx Sep 12, 2023 Author

Uh oh!

paoloyx Sep 13, 2023 Author

Uh oh!

paoloyx Sep 12, 2023 Author

paoloyx
Sep 11, 2023

Replies: 3 comments 5 replies

scholzj
Sep 11, 2023
Maintainer

paoloyx
Sep 12, 2023
Author

`Kafka` resource

scholzj Sep 12, 2023
Maintainer

paoloyx Sep 12, 2023
Author

scholzj Sep 12, 2023
Maintainer

paoloyx Sep 12, 2023
Author

paoloyx Sep 13, 2023
Author

paoloyx
Sep 12, 2023
Author