Kafka Operator does not come to ready state: Liveness and Readiness probes fail

[yianni777]

Hi all,

I've installed strimzi kafka operator using the provided helm charts on a rancher k8s cluster. The kafka operator pod does not come to ready state as the Liveness and Readiness probes failed reporting: "failed: Get "http://10.42.1.16:8080/ready": dial tcp 10.42.1.16:8080: connect: connection refused". There are no other pods running except for the k8s system pods. I can't find any apparent reason for the failure.

NAME READY STATUS RESTARTS AGE
strimzi-cluster-operator-67b78fb598-hz9j6 0/1 Running 805 (4m14s ago) 3d15h

The is the output of "kubectl desrcibe" for the operator pod:

Name: strimzi-cluster-operator-67b78fb598-hz9j6
Namespace: kafka
Priority: 0
Node: ip-10-53-4-45.ec2.internal/10.53.4.45
Start Time: Thu, 15 Dec 2022 22:45:25 +0000
Labels: name=strimzi-cluster-operator
pod-template-hash=67b78fb598
strimzi.io/kind=cluster-operator
Annotations: cni.projectcalico.org/containerID: 5fb17b33beecec7dde015cef29b7e435ef7ecd8f8179422321bf20238454168c
cni.projectcalico.org/podIP: 10.42.1.16/32
cni.projectcalico.org/podIPs: 10.42.1.16/32
kubernetes.io/psp: global-unrestricted-psp
Status: Running
IP: 10.42.1.16
IPs:
IP: 10.42.1.16
Controlled By: ReplicaSet/strimzi-cluster-operator-67b78fb598
Containers:
strimzi-cluster-operator:
Container ID: containerd://fe92cfca4fcde11372f50e5245f3e7fadace1973c484d29eb1c25bed591d874e
Image: quay.io/strimzi/operator:0.32.0
Image ID: quay.io/strimzi/operator@sha256:c5e0e5dca7503a8371b8d366db0f49e3fc68f62ea480262e39b6cc277991c97c
Port: 8080/TCP
Host Port: 0/TCP
Args:
/opt/strimzi/bin/cluster_operator_run.sh
State: Running
Started: Mon, 19 Dec 2022 13:57:56 +0000
Last State: Terminated
Reason: Error
Exit Code: 137
Started: Mon, 19 Dec 2022 13:51:26 +0000
Finished: Mon, 19 Dec 2022 13:57:56 +0000
Ready: False
Restart Count: 805
Limits:
cpu: 4
memory: 2Gi
Requests:
cpu: 1
memory: 2Gi
Liveness: http-get http://:http/healthy delay=300s timeout=1s period=30s #success=1 #failure=3
Readiness: http-get http://:http/ready delay=300s timeout=1s period=30s #success=1 #failure=3
Environment:
STRIMZI_NAMESPACE: kafka (v1:metadata.namespace)
STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
STRIMZI_OPERATION_TIMEOUT_MS: 300000
STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE: quay.io/strimzi/kafka:0.32.0-kafka-3.3.1
STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE: quay.io/strimzi/kafka:0.32.0-kafka-3.3.1
STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE: quay.io/strimzi/kafka:0.32.0-kafka-3.3.1
STRIMZI_KAFKA_IMAGES: 3.2.0=quay.io/strimzi/kafka:0.32.0-kafka-3.2.0
3.2.1=quay.io/strimzi/kafka:0.32.0-kafka-3.2.1
3.2.3=quay.io/strimzi/kafka:0.32.0-kafka-3.2.3
3.3.1=quay.io/strimzi/kafka:0.32.0-kafka-3.3.1

  STRIMZI_KAFKA_CONNECT_IMAGES:                       3.2.0=quay.io/strimzi/kafka:0.32.0-kafka-3.2.0
                                                      3.2.1=quay.io/strimzi/kafka:0.32.0-kafka-3.2.1
                                                      3.2.3=quay.io/strimzi/kafka:0.32.0-kafka-3.2.3
                                                      3.3.1=quay.io/strimzi/kafka:0.32.0-kafka-3.3.1

  STRIMZI_KAFKA_MIRROR_MAKER_IMAGES:                  3.2.0=quay.io/strimzi/kafka:0.32.0-kafka-3.2.0
                                                      3.2.1=quay.io/strimzi/kafka:0.32.0-kafka-3.2.1
                                                      3.2.3=quay.io/strimzi/kafka:0.32.0-kafka-3.2.3
                                                      3.3.1=quay.io/strimzi/kafka:0.32.0-kafka-3.3.1

  STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES:                3.2.0=quay.io/strimzi/kafka:0.32.0-kafka-3.2.0
                                                      3.2.1=quay.io/strimzi/kafka:0.32.0-kafka-3.2.1
                                                      3.2.3=quay.io/strimzi/kafka:0.32.0-kafka-3.2.3
                                                      3.3.1=quay.io/strimzi/kafka:0.32.0-kafka-3.3.1

  STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE:               quay.io/strimzi/operator:0.32.0
  STRIMZI_DEFAULT_USER_OPERATOR_IMAGE:                quay.io/strimzi/operator:0.32.0
  STRIMZI_DEFAULT_KAFKA_INIT_IMAGE:                   quay.io/strimzi/operator:0.32.0
  STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE:                 quay.io/strimzi/kafka-bridge:0.22.3
  STRIMZI_DEFAULT_JMXTRANS_IMAGE:                     quay.io/strimzi/jmxtrans:0.32.0
  STRIMZI_DEFAULT_KANIKO_EXECUTOR_IMAGE:              quay.io/strimzi/kaniko-executor:0.32.0
  STRIMZI_DEFAULT_MAVEN_BUILDER:                      quay.io/strimzi/maven-builder:0.32.0
  STRIMZI_OPERATOR_NAMESPACE:                         kafka (v1:metadata.namespace)
  STRIMZI_FEATURE_GATES:
  STRIMZI_LEADER_ELECTION_ENABLED:                    true
  STRIMZI_LEADER_ELECTION_LEASE_NAME:                 strimzi-cluster-operator
  STRIMZI_LEADER_ELECTION_LEASE_NAMESPACE:            kafka (v1:metadata.namespace)
  STRIMZI_LEADER_ELECTION_IDENTITY:                   strimzi-cluster-operator-67b78fb598-hz9j6 (v1:metadata.name)
Mounts:
  /opt/strimzi/custom-config/ from co-config-volume (rw)
  /tmp from strimzi-tmp (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5nb4d (ro)

Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
strimzi-tmp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: 1Mi
co-config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: strimzi-cluster-operator
Optional: false
kube-api-access-5nb4d:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Warning Unhealthy 12m (x2410 over 3d15h) kubelet Liveness probe failed: Get "http://10.42.1.16:8080/healthy": dial tcp 10.42.1.16:8080: connect: connection refused
Warning Unhealthy 6m7s (x5319 over 3d15h) kubelet Readiness probe failed: Get "http://10.42.1.16:8080/ready": dial tcp 10.42.1.16:8080: connect: connection refused

[scholzj]

TBH, it is not really clear what you are asking about. You share some misformatted operator deployment? Or Pod? But you talk about Kafka broker. So it is not really clear what your issue is. Please share the resources always as YAML (i.e. kubectl get <resource> -o yaml and not with kubecl describe ...). Also, you have to format them as code to make it readable. And you should probably also share your logs and custom resources.

[yianni777]

Sorry, It was a typo. I meant to title the discussion as operator, not broker.

[yianni777]

$ kubectl get po strimzi-cluster-operator-67b78fb598-hz9j6 -n kafka -o yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/containerID: 5fb17b33beecec7dde015cef29b7e435ef7ecd8f8179422321bf20238454168c
    cni.projectcalico.org/podIP: 10.42.1.16/32
    cni.projectcalico.org/podIPs: 10.42.1.16/32
    kubernetes.io/psp: global-unrestricted-psp
  creationTimestamp: "2022-12-15T22:45:25Z"
  generateName: strimzi-cluster-operator-67b78fb598-
  labels:
    name: strimzi-cluster-operator
    pod-template-hash: 67b78fb598
    strimzi.io/kind: cluster-operator
  name: strimzi-cluster-operator-67b78fb598-hz9j6
  namespace: kafka
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: strimzi-cluster-operator-67b78fb598
    uid: 31d985e8-e563-476c-86b9-4ee01b934d34
  resourceVersion: "1031819"
  uid: f01c9786-9b29-463f-b69e-9efff10c09f7
spec:
  containers:
  - args:
    - /opt/strimzi/bin/cluster_operator_run.sh
    env:
    - name: STRIMZI_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: STRIMZI_FULL_RECONCILIATION_INTERVAL_MS
      value: "120000"
    - name: STRIMZI_OPERATION_TIMEOUT_MS
      value: "300000"
    - name: STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE
      value: quay.io/strimzi/kafka:0.32.0-kafka-3.3.1
    - name: STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE
      value: quay.io/strimzi/kafka:0.32.0-kafka-3.3.1
    - name: STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE
      value: quay.io/strimzi/kafka:0.32.0-kafka-3.3.1
    - name: STRIMZI_KAFKA_IMAGES
      value: |
        3.2.0=quay.io/strimzi/kafka:0.32.0-kafka-3.2.0
        3.2.1=quay.io/strimzi/kafka:0.32.0-kafka-3.2.1
        3.2.3=quay.io/strimzi/kafka:0.32.0-kafka-3.2.3
        3.3.1=quay.io/strimzi/kafka:0.32.0-kafka-3.3.1
    - name: STRIMZI_KAFKA_CONNECT_IMAGES
      value: |
        3.2.0=quay.io/strimzi/kafka:0.32.0-kafka-3.2.0
        3.2.1=quay.io/strimzi/kafka:0.32.0-kafka-3.2.1
        3.2.3=quay.io/strimzi/kafka:0.32.0-kafka-3.2.3
        3.3.1=quay.io/strimzi/kafka:0.32.0-kafka-3.3.1
    - name: STRIMZI_KAFKA_MIRROR_MAKER_IMAGES
      value: |
        3.2.0=quay.io/strimzi/kafka:0.32.0-kafka-3.2.0
        3.2.1=quay.io/strimzi/kafka:0.32.0-kafka-3.2.1
        3.2.3=quay.io/strimzi/kafka:0.32.0-kafka-3.2.3
        3.3.1=quay.io/strimzi/kafka:0.32.0-kafka-3.3.1
    - name: STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES
      value: |
        3.2.0=quay.io/strimzi/kafka:0.32.0-kafka-3.2.0
        3.2.1=quay.io/strimzi/kafka:0.32.0-kafka-3.2.1
        3.2.3=quay.io/strimzi/kafka:0.32.0-kafka-3.2.3
        3.3.1=quay.io/strimzi/kafka:0.32.0-kafka-3.3.1
    - name: STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE
      value: quay.io/strimzi/operator:0.32.0
    - name: STRIMZI_DEFAULT_USER_OPERATOR_IMAGE
      value: quay.io/strimzi/operator:0.32.0
    - name: STRIMZI_DEFAULT_KAFKA_INIT_IMAGE
      value: quay.io/strimzi/operator:0.32.0
    - name: STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE
      value: quay.io/strimzi/kafka-bridge:0.22.3
    - name: STRIMZI_DEFAULT_JMXTRANS_IMAGE
      value: quay.io/strimzi/jmxtrans:0.32.0
    - name: STRIMZI_DEFAULT_KANIKO_EXECUTOR_IMAGE
      value: quay.io/strimzi/kaniko-executor:0.32.0
    - name: STRIMZI_DEFAULT_MAVEN_BUILDER
      value: quay.io/strimzi/maven-builder:0.32.0
    - name: STRIMZI_OPERATOR_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: STRIMZI_FEATURE_GATES
    - name: STRIMZI_LEADER_ELECTION_ENABLED
      value: "true"
    - name: STRIMZI_LEADER_ELECTION_LEASE_NAME
      value: strimzi-cluster-operator
    - name: STRIMZI_LEADER_ELECTION_LEASE_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: STRIMZI_LEADER_ELECTION_IDENTITY
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    image: quay.io/strimzi/operator:0.32.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /healthy
        port: http
        scheme: HTTP
      initialDelaySeconds: 300
      periodSeconds: 30
      successThreshold: 1
      timeoutSeconds: 1
    name: strimzi-cluster-operator
    ports:
    - containerPort: 8080
      name: http
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /ready
        port: http
        scheme: HTTP
      initialDelaySeconds: 300
      periodSeconds: 30
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: "4"
        memory: 2Gi
      requests:
        cpu: "1"
        memory: 2Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /tmp
      name: strimzi-tmp
    - mountPath: /opt/strimzi/custom-config/
      name: co-config-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-5nb4d
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: ip-10-53-4-45.ec2.internal
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: strimzi-cluster-operator
  serviceAccountName: strimzi-cluster-operator
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - emptyDir:
      medium: Memory
      sizeLimit: 1Mi
    name: strimzi-tmp
  - configMap:
      defaultMode: 420
      name: strimzi-cluster-operator
    name: co-config-volume
  - name: kube-api-access-5nb4d
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2022-12-15T22:45:25Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2022-12-15T22:45:25Z"
    message: 'containers with unready status: [strimzi-cluster-operator]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2022-12-15T22:45:25Z"
    message: 'containers with unready status: [strimzi-cluster-operator]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2022-12-15T22:45:25Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://0c8e22ca8e7c227f2afc3d20e38fdc4722c646d9133faa4b774edb3dfb477624
    image: quay.io/strimzi/operator:0.32.0
    imageID: quay.io/strimzi/operator@sha256:c5e0e5dca7503a8371b8d366db0f49e3fc68f62ea480262e39b6cc277991c97c
    lastState:
      terminated:
        containerID: containerd://d6d42346cb7bbc2a6e37f9cacd3d31bcf7a0bd94f228211fc485e108202a28a8
        exitCode: 137
        finishedAt: "2022-12-19T15:22:26Z"
        reason: Error
        startedAt: "2022-12-19T15:15:56Z"
    name: strimzi-cluster-operator
    ready: false
    restartCount: 818
    started: true
    state:
      running:
        startedAt: "2022-12-19T15:22:26Z"
  hostIP: 10.53.4.45
  phase: Running
  podIP: 10.42.1.16
  podIPs:
  - ip: 10.42.1.16
  qosClass: Burstable
  startTime: "2022-12-15T22:45:25Z"

[yianni777]

$ kubectl logs strimzi-cluster-operator-67b78fb598-hz9j6 -n kafka

removed '/tmp/hsperfdata_strimzi/14'
removed directory '/tmp/hsperfdata_strimzi'
Auto-detected KUBERNETES_SERVICE_DNS_DOMAIN: cluster.local
+ exec /usr/bin/tini -w -e 143 -- java -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom --illegal-access=deny -XX:+ExitOnOutOfMemoryError -XX:MinRAMPercentage=15 -XX:MaxRAMPercentage=20 -classpath lib/io.strimzi.cluster-operator-0.32.0.jar:lib/io.fabric8.openshift-client-6.2.0.jar:lib/io.netty.netty-transport-4.1.77.Final.jar:lib/io.netty.netty-codec-socks-4.1.77.Final.jar:lib/io.fabric8.openshift-model-tuned-6.2.0.jar:lib/io.strimzi.certificate-manager-0.32.0.jar:lib/io.fabric8.kubernetes-model-rbac-6.2.0.jar:lib/io.netty.netty-handler-proxy-4.1.77.Final.jar:lib/io.fabric8.openshift-model-operator-6.2.0.jar:lib/io.fabric8.kubernetes-model-discovery-6.2.0.jar:lib/io.strimzi.kafka-oauth-server-0.11.0.jar:lib/io.fabric8.kubernetes-model-common-6.2.0.jar:lib/io.fabric8.kubernetes-model-storageclass-6.2.0.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/io.fabric8.kubernetes-model-autoscaling-6.2.0.jar:lib/io.fabric8.kubernetes-model-gatewayapi-6.2.0.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/io.strimzi.config-model-0.32.0.jar:lib/org.apache.zookeeper.zookeeper-jute-3.6.3.jar:lib/io.netty.netty-transport-native-epoll-4.1.77.Final-linux-x86_64.jar:lib/org.apache.zookeeper.zookeeper-3.6.3.jar:lib/io.netty.netty-resolver-dns-4.1.77.Final.jar:lib/net.minidev.accessors-smart-2.4.7.jar:lib/io.netty.netty-handler-4.1.77.Final.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.fabric8.kubernetes-model-certificates-6.2.0.jar:lib/io.netty.netty-codec-http-4.1.77.Final.jar:lib/io.fabric8.openshift-model-miscellaneous-6.2.0.jar:lib/io.strimzi.operator-common-0.32.0.jar:lib/io.fabric8.openshift-model-config-6.2.0.jar:lib/io.fabric8.kubernetes-model-policy-6.2.0.jar:lib/io.strimzi.crd-annotations-0.32.0.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.13.4.jar:lib/io.fabric8.openshift-model-installer-6.2.0.jar:lib/io.fabric8.openshift-model-storageversionmigrator-6.2.0.jar:lib/io.fabric8.kubernetes-model-node-6.2.0.jar:lib/com.jayway.jsonpath.json-path-2.6.0.jar:lib/net.minidev.json-smart-2.4.7.jar:lib/org.yaml.snakeyaml-1.32.jar:lib/io.fabric8.openshift-model-machine-6.2.0.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/io.netty.netty-codec-dns-4.1.77.Final.jar:lib/io.fabric8.openshift-model-operatorhub-6.2.0.jar:lib/io.netty.netty-resolver-4.1.77.Final.jar:lib/io.fabric8.openshift-model-6.2.0.jar:lib/io.strimzi.kafka-oauth-server-plain-0.11.0.jar:lib/io.netty.netty-transport-native-unix-common-4.1.77.Final.jar:lib/io.fabric8.openshift-model-console-6.2.0.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.13.4.1.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.fabric8.openshift-client-api-6.2.0.jar:lib/io.netty.netty-transport-native-epoll-4.1.77.Final.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.4.jar:lib/io.fabric8.kubernetes-model-apiextensions-6.2.0.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/io.vertx.vertx-core-4.3.4.jar:lib/io.strimzi.kafka-oauth-common-0.11.0.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.13.4.jar:lib/io.fabric8.kubernetes-model-events-6.2.0.jar:lib/io.fabric8.openshift-model-whereabouts-6.2.0.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/io.fabric8.openshift-model-monitoring-6.2.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.netty.netty-codec-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-metrics-6.2.0.jar:lib/io.strimzi.api-0.32.0.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/io.fabric8.kubernetes-model-networking-6.2.0.jar:lib/com.nimbusds.nimbus-jose-jwt-9.10.jar:lib/com.squareup.okhttp3.okhttp-3.12.12.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/io.fabric8.kubernetes-client-6.2.0.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.13.4.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/io.netty.netty-buffer-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-flowcontrol-6.2.0.jar:lib/io.fabric8.openshift-model-machineconfig-6.2.0.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/io.fabric8.openshift-model-hive-6.2.0.jar:lib/org.apache.kafka.kafka-clients-3.3.1.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/com.fasterxml.jackson.core.jackson-core-2.13.4.jar:lib/io.fabric8.kubernetes-client-api-6.2.0.jar:lib/io.netty.netty-codec-http2-4.1.77.Final.jar:lib/io.fabric8.kubernetes-httpclient-okhttp-6.2.0.jar:lib/com.github.luben.zstd-jni-1.5.2-1.jar:lib/io.fabric8.kubernetes-model-admissionregistration-6.2.0.jar:lib/io.fabric8.kubernetes-model-apps-6.2.0.jar:lib/io.fabric8.kubernetes-model-scheduling-6.2.0.jar:lib/io.strimzi.kafka-oauth-client-0.11.0.jar:lib/io.fabric8.kubernetes-model-core-6.2.0.jar:lib/io.netty.netty-transport-classes-epoll-4.1.77.Final.jar:lib/io.vertx.vertx-micrometer-metrics-4.3.4.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.fabric8.openshift-model-clusterautoscaling-6.2.0.jar:lib/io.fabric8.kubernetes-model-batch-6.2.0.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/io.fabric8.kubernetes-model-extensions-6.2.0.jar:lib/io.fabric8.kubernetes-model-coordination-6.2.0.jar:lib/io.netty.netty-common-4.1.77.Final.jar io.strimzi.operator.cluster.Main
2022-12-19 15:48:27 INFO  Main:74 - ClusterOperator 0.32.0 is starting
2022-12-19 15:48:27 INFO  Util:314 - Using config:
        PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
        container: oci
        STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE: quay.io/strimzi/operator:0.32.0
        JAVA_OPTS:  -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom  --illegal-access=deny -XX:+ExitOnOutOfMemoryError -XX:MinRAMPercentage=15 -XX:MaxRAMPercentage=20
        STRIMZI_HOME: /opt/strimzi
        PWD: /opt/strimzi
        STRIMZI_LEADER_ELECTION_IDENTITY: strimzi-cluster-operator-67b78fb598-hz9j6
        KUBERNETES_PORT_443_TCP: tcp://10.43.0.1:443
        JAVA_MAIN: io.strimzi.operator.cluster.Main
        STRIMZI_LEADER_ELECTION_ENABLED: true
        STRIMZI_VERSION: 0.32.0
        STRIMZI_DEFAULT_JMXTRANS_IMAGE: quay.io/strimzi/jmxtrans:0.32.0
        STRIMZI_NAMESPACE: kafka
        STRIMZI_DEFAULT_KANIKO_EXECUTOR_IMAGE: quay.io/strimzi/kaniko-executor:0.32.0
        TINI_SHA256_S390X: 931b70a182af879ca249ae9de87ef68423121b38d235c78997fafc680ceab32d
        STRIMZI_DEFAULT_MAVEN_BUILDER: quay.io/strimzi/maven-builder:0.32.0
        STRIMZI_FEATURE_GATES:
        TINI_SHA256_AMD64: 93dcc18adc78c65a028a84799ecf8ad40c936fdfc5f2a57b1acda5a8117fa82c
        KUBERNETES_SERVICE_DNS_DOMAIN: cluster.local
        STRIMZI_LEADER_ELECTION_LEASE_NAME: strimzi-cluster-operator
        KUBERNETES_SERVICE_PORT_HTTPS: 443
        SHLVL: 0
        STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE: quay.io/strimzi/kafka-bridge:0.22.3
        KUBERNETES_PORT: tcp://10.43.0.1:443
        STRIMZI_KAFKA_MIRROR_MAKER_IMAGES: 3.2.0=quay.io/strimzi/kafka:0.32.0-kafka-3.2.0
3.2.1=quay.io/strimzi/kafka:0.32.0-kafka-3.2.1
3.2.3=quay.io/strimzi/kafka:0.32.0-kafka-3.2.3
3.3.1=quay.io/strimzi/kafka:0.32.0-kafka-3.3.1

        JAVA_HOME: /usr/lib/jvm/jre-11
        STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
        KUBERNETES_SERVICE_HOST: 10.43.0.1
        JAVA_CLASSPATH: lib/io.strimzi.cluster-operator-0.32.0.jar:lib/io.fabric8.openshift-client-6.2.0.jar:lib/io.netty.netty-transport-4.1.77.Final.jar:lib/io.netty.netty-codec-socks-4.1.77.Final.jar:lib/io.fabric8.openshift-model-tuned-6.2.0.jar:lib/io.strimzi.certificate-manager-0.32.0.jar:lib/io.fabric8.kubernetes-model-rbac-6.2.0.jar:lib/io.netty.netty-handler-proxy-4.1.77.Final.jar:lib/io.fabric8.openshift-model-operator-6.2.0.jar:lib/io.fabric8.kubernetes-model-discovery-6.2.0.jar:lib/io.strimzi.kafka-oauth-server-0.11.0.jar:lib/io.fabric8.kubernetes-model-common-6.2.0.jar:lib/io.fabric8.kubernetes-model-storageclass-6.2.0.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/io.fabric8.kubernetes-model-autoscaling-6.2.0.jar:lib/io.fabric8.kubernetes-model-gatewayapi-6.2.0.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/io.strimzi.config-model-0.32.0.jar:lib/org.apache.zookeeper.zookeeper-jute-3.6.3.jar:lib/io.netty.netty-transport-native-epoll-4.1.77.Final-linux-x86_64.jar:lib/org.apache.zookeeper.zookeeper-3.6.3.jar:lib/io.netty.netty-resolver-dns-4.1.77.Final.jar:lib/net.minidev.accessors-smart-2.4.7.jar:lib/io.netty.netty-handler-4.1.77.Final.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.fabric8.kubernetes-model-certificates-6.2.0.jar:lib/io.netty.netty-codec-http-4.1.77.Final.jar:lib/io.fabric8.openshift-model-miscellaneous-6.2.0.jar:lib/io.strimzi.operator-common-0.32.0.jar:lib/io.fabric8.openshift-model-config-6.2.0.jar:lib/io.fabric8.kubernetes-model-policy-6.2.0.jar:lib/io.strimzi.crd-annotations-0.32.0.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.13.4.jar:lib/io.fabric8.openshift-model-installer-6.2.0.jar:lib/io.fabric8.openshift-model-storageversionmigrator-6.2.0.jar:lib/io.fabric8.kubernetes-model-node-6.2.0.jar:lib/com.jayway.jsonpath.json-path-2.6.0.jar:lib/net.minidev.json-smart-2.4.7.jar:lib/org.yaml.snakeyaml-1.32.jar:lib/io.fabric8.openshift-model-machine-6.2.0.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/io.netty.netty-codec-dns-4.1.77.Final.jar:lib/io.fabric8.openshift-model-operatorhub-6.2.0.jar:lib/io.netty.netty-resolver-4.1.77.Final.jar:lib/io.fabric8.openshift-model-6.2.0.jar:lib/io.strimzi.kafka-oauth-server-plain-0.11.0.jar:lib/io.netty.netty-transport-native-unix-common-4.1.77.Final.jar:lib/io.fabric8.openshift-model-console-6.2.0.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.13.4.1.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.fabric8.openshift-client-api-6.2.0.jar:lib/io.netty.netty-transport-native-epoll-4.1.77.Final.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.4.jar:lib/io.fabric8.kubernetes-model-apiextensions-6.2.0.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/io.vertx.vertx-core-4.3.4.jar:lib/io.strimzi.kafka-oauth-common-0.11.0.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.13.4.jar:lib/io.fabric8.kubernetes-model-events-6.2.0.jar:lib/io.fabric8.openshift-model-whereabouts-6.2.0.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/io.fabric8.openshift-model-monitoring-6.2.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.netty.netty-codec-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-metrics-6.2.0.jar:lib/io.strimzi.api-0.32.0.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/io.fabric8.kubernetes-model-networking-6.2.0.jar:lib/com.nimbusds.nimbus-jose-jwt-9.10.jar:lib/com.squareup.okhttp3.okhttp-3.12.12.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/io.fabric8.kubernetes-client-6.2.0.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.13.4.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/io.netty.netty-buffer-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-flowcontrol-6.2.0.jar:lib/io.fabric8.openshift-model-machineconfig-6.2.0.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/io.fabric8.openshift-model-hive-6.2.0.jar:lib/org.apache.kafka.kafka-clients-3.3.1.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/com.fasterxml.jackson.core.jackson-core-2.13.4.jar:lib/io.fabric8.kubernetes-client-api-6.2.0.jar:lib/io.netty.netty-codec-http2-4.1.77.Final.jar:lib/io.fabric8.kubernetes-httpclient-okhttp-6.2.0.jar:lib/com.github.luben.zstd-jni-1.5.2-1.jar:lib/io.fabric8.kubernetes-model-admissionregistration-6.2.0.jar:lib/io.fabric8.kubernetes-model-apps-6.2.0.jar:lib/io.fabric8.kubernetes-model-scheduling-6.2.0.jar:lib/io.strimzi.kafka-oauth-client-0.11.0.jar:lib/io.fabric8.kubernetes-model-core-6.2.0.jar:lib/io.netty.netty-transport-classes-epoll-4.1.77.Final.jar:lib/io.vertx.vertx-micrometer-metrics-4.3.4.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.fabric8.openshift-model-clusterautoscaling-6.2.0.jar:lib/io.fabric8.kubernetes-model-batch-6.2.0.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/io.fabric8.kubernetes-model-extensions-6.2.0.jar:lib/io.fabric8.kubernetes-model-coordination-6.2.0.jar:lib/io.netty.netty-common-4.1.77.Final.jar
        STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE: quay.io/strimzi/kafka:0.32.0-kafka-3.3.1
        TINI_VERSION: v0.19.0
        STRIMZI_OPERATION_TIMEOUT_MS: 300000
        KUBERNETES_PORT_443_TCP_ADDR: 10.43.0.1
        STRIMZI_LEADER_ELECTION_LEASE_NAMESPACE: kafka
        STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES: 3.2.0=quay.io/strimzi/kafka:0.32.0-kafka-3.2.0
3.2.1=quay.io/strimzi/kafka:0.32.0-kafka-3.2.1
3.2.3=quay.io/strimzi/kafka:0.32.0-kafka-3.2.3
3.3.1=quay.io/strimzi/kafka:0.32.0-kafka-3.3.1

        KUBERNETES_PORT_443_TCP_PROTO: tcp
        STRIMZI_DEFAULT_USER_OPERATOR_IMAGE: quay.io/strimzi/operator:0.32.0
        KUBERNETES_SERVICE_PORT: 443
        STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE: quay.io/strimzi/kafka:0.32.0-kafka-3.3.1
        TINI_SHA256_ARM64: 07952557df20bfd2a95f9bef198b445e006171969499a1d361bd9e6f8e5e0e81
        STRIMZI_DEFAULT_KAFKA_INIT_IMAGE: quay.io/strimzi/operator:0.32.0
        TINI_SHA256_PPC64LE: 3f658420974768e40810001a038c29d003728c5fe86da211cff5059e48cfdfde
        HOSTNAME: strimzi-cluster-operator-67b78fb598-hz9j6
        STRIMZI_KAFKA_CONNECT_IMAGES: 3.2.0=quay.io/strimzi/kafka:0.32.0-kafka-3.2.0
3.2.1=quay.io/strimzi/kafka:0.32.0-kafka-3.2.1
3.2.3=quay.io/strimzi/kafka:0.32.0-kafka-3.2.3
3.3.1=quay.io/strimzi/kafka:0.32.0-kafka-3.3.1

        STRIMZI_KAFKA_IMAGES: 3.2.0=quay.io/strimzi/kafka:0.32.0-kafka-3.2.0
3.2.1=quay.io/strimzi/kafka:0.32.0-kafka-3.2.1
3.2.3=quay.io/strimzi/kafka:0.32.0-kafka-3.2.3
3.3.1=quay.io/strimzi/kafka:0.32.0-kafka-3.3.1

        KUBERNETES_PORT_443_TCP_PORT: 443
        STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE: quay.io/strimzi/kafka:0.32.0-kafka-3.3.1
        STRIMZI_OPERATOR_NAME: strimzi-cluster-operator-67b78fb598-hz9j6
        HOME: /home/strimzi
        MALLOC_ARENA_MAX: 2
        STRIMZI_OPERATOR_NAMESPACE: kafka

2022-12-19 15:48:27 INFO  Main:77 - Cluster Operator configuration is ClusterOperatorConfig(namespaces=[kafka],reconciliationIntervalMs=120000,operationTimeoutMs=300000,connectBuildTimeoutMs=300000,createClusterRoles=false,networkPolicyGeneration=true,versions=versions{3.2.0={proto: 3.2 msg: 3.2 kafka-image: quay.io/strimzi/kafka:0.32.0-kafka-3.2.0 connect-image: quay.io/strimzi/kafka:0.32.0-kafka-3.2.0 mirrormaker-image: quay.io/strimzi/kafka:0.32.0-kafka-3.2.0 mirrormaker2-image: quay.io/strimzi/kafka:0.32.0-kafka-3.2.0}, 3.2.1={proto: 3.2 msg: 3.2 kafka-image: quay.io/strimzi/kafka:0.32.0-kafka-3.2.1 connect-image: quay.io/strimzi/kafka:0.32.0-kafka-3.2.1 mirrormaker-image: quay.io/strimzi/kafka:0.32.0-kafka-3.2.1 mirrormaker2-image: quay.io/strimzi/kafka:0.32.0-kafka-3.2.1}, 3.2.3={proto: 3.2 msg: 3.2 kafka-image: quay.io/strimzi/kafka:0.32.0-kafka-3.2.3 connect-image: quay.io/strimzi/kafka:0.32.0-kafka-3.2.3 mirrormaker-image: quay.io/strimzi/kafka:0.32.0-kafka-3.2.3 mirrormaker2-image: quay.io/strimzi/kafka:0.32.0-kafka-3.2.3}, 3.3.1={proto: 3.3 msg: 3.3 kafka-image: quay.io/strimzi/kafka:0.32.0-kafka-3.3.1 connect-image: quay.io/strimzi/kafka:0.32.0-kafka-3.3.1 mirrormaker-image: quay.io/strimzi/kafka:0.32.0-kafka-3.3.1 mirrormaker2-image: quay.io/strimzi/kafka:0.32.0-kafka-3.3.1}},imagePullPolicy=null,imagePullSecrets=null,operatorNamespace=kafka,operatorNamespaceLabels=null,customResourceSelector=null,featureGates=FeatureGates(UseStrimziPodSets=true,UseKRaft=false),zkAdminSessionTimeoutMs=10000,dnsCacheTtlSec=30,podSetReconciliationOnly=false,podSetControllerWorkQueueSize=1024,operatorName=strimzi-cluster-operator-67b78fb598-hz9j6,podSecurityProviderClass=io.strimzi.plugin.security.profiles.impl.BaselinePodSecurityProvider,leaderElectionConfig=LeaderElectionConfig{leaseName='strimzi-cluster-operator', namespace='kafka', identity='strimzi-cluster-operator-67b78fb598-hz9j6', leaseDuration=PT15S, renewDeadline=PT10S, retryPeriod=PT2S})

[yianni777]

$ kubectl get crd -A

NAME                                                  CREATED AT
addons.k3s.cattle.io                                  2022-12-15T17:01:07Z
bgpconfigurations.crd.projectcalico.org               2022-12-15T17:01:29Z
bgppeers.crd.projectcalico.org                        2022-12-15T17:01:29Z
blockaffinities.crd.projectcalico.org                 2022-12-15T17:01:29Z
caliconodestatuses.crd.projectcalico.org              2022-12-15T17:01:29Z
clusterinformations.crd.projectcalico.org             2022-12-15T17:01:29Z
felixconfigurations.crd.projectcalico.org             2022-12-15T17:01:29Z
globalnetworkpolicies.crd.projectcalico.org           2022-12-15T17:01:29Z
globalnetworksets.crd.projectcalico.org               2022-12-15T17:01:29Z
helmchartconfigs.helm.cattle.io                       2022-12-15T17:01:07Z
helmcharts.helm.cattle.io                             2022-12-15T17:01:07Z
hostendpoints.crd.projectcalico.org                   2022-12-15T17:01:29Z
ipamblocks.crd.projectcalico.org                      2022-12-15T17:01:29Z
ipamconfigs.crd.projectcalico.org                     2022-12-15T17:01:29Z
ipamhandles.crd.projectcalico.org                     2022-12-15T17:01:29Z
ippools.crd.projectcalico.org                         2022-12-15T17:01:29Z
ipreservations.crd.projectcalico.org                  2022-12-15T17:01:29Z
kafkabridges.kafka.strimzi.io                         2022-12-15T21:10:02Z
kafkaconnectors.kafka.strimzi.io                      2022-12-15T21:10:02Z
kafkaconnects.kafka.strimzi.io                        2022-12-15T21:10:00Z
kafkamirrormaker2s.kafka.strimzi.io                   2022-12-15T21:10:03Z
kafkamirrormakers.kafka.strimzi.io                    2022-12-15T21:10:01Z
kafkarebalances.kafka.strimzi.io                      2022-12-15T21:10:03Z
kafkas.kafka.strimzi.io                               2022-12-15T21:10:00Z
kafkatopics.kafka.strimzi.io                          2022-12-15T21:10:01Z
kafkausers.kafka.strimzi.io                           2022-12-15T21:10:01Z
kubecontrollersconfigurations.crd.projectcalico.org   2022-12-15T17:01:29Z
networkpolicies.crd.projectcalico.org                 2022-12-15T17:01:29Z
networksets.crd.projectcalico.org                     2022-12-15T17:01:29Z
strimzipodsets.core.strimzi.io                        2022-12-15T21:10:00Z

[scholzj]

Great, but can you please edit your posts and use proper formatting? Without it, nobody can read it. https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax