Kafka ingress/contour not create service and ingress resources

[sattha-p]

I'm facing an issue service and ingress not create after apply "ingress listeners".
I follow this document.

K8S Tanzu
Ingress controller: Contour/Envoy

I'm not sure is this because I'm not use the Ingress Nginx or not? However, Ingress Contour also support TLS Pass through. Is there anything I can do with Strimzi and Ingress Contour

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-strimzi
  namespace: kafka-dev
spec:
  kafka:
    version: 3.2.0
    replicas: 3
    resources:
      requests:
        memory: "2Gi"
        cpu: 0.5
      limits:
        memory: "2Gi"
    jvmOptions:
      -Xms: 1024m
      -Xmx: 1024m
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: external
        port: 9094
        type: ingress
        tls: true
        configuration:
          bootstrap:
            host: kafka.dev.store.global.abc.org
            annotations:
              external-dns.alpha.kubernetes.io/hostname: kafka.dev.store.global.abc.org.
              external-dns.alpha.kubernetes.io/ttl: "60"
          brokers:
          - broker: 0
            host: kafka-broker-0.dev.store.global.abc.org
            annotations:
              external-dns.alpha.kubernetes.io/hostname: kafka-broker-0.dev.store.global.abc.org.
              external-dns.alpha.kubernetes.io/ttl: "60"
          - broker: 1
            host: kafka-broker-1.dev.store.global.abc.org
            annotations:
              external-dns.alpha.kubernetes.io/hostname: kafka-broker-1.dev.store.global.abc.org.
              external-dns.alpha.kubernetes.io/ttl: "60"
          - broker: 2
            host: kafka-broker-2.dev.store.global.abc.org
            annotations:
              external-dns.alpha.kubernetes.io/hostname: kafka-broker-2.dev.store.global.abc.org.
              external-dns.alpha.kubernetes.io/ttl: "60"
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.2"
    storage:
      type: persistent-claim
      class: cpt-vsphere-with-tanzu-storage-policy
      size: 50G
      deleteClaim: false
    template:
      pod:
        affinity:
          nodeAffinity:
            preferredDuringSchedulingIgnoredDuringExecution: 
            - weight: 1
              preference:
                matchExpressions:
                - key: service
                  operator: In
                  values:
                  - kafka 

  zookeeper:
    replicas: 3
    resources:
      requests:
        memory: "1Gi"
        cpu: 0.3
      limits:
        memory: "1Gi"
    jvmOptions:
      -Xms: 512m
      -Xmx: 512m
    storage:
      type: persistent-claim
      size: 50G
      class: cpt-vsphere-with-tanzu-storage-policy
      deleteClaim: false
    template:
      pod:
        affinity:
          nodeAffinity:
            preferredDuringSchedulingIgnoredDuringExecution: 
            - weight: 1
              preference:
                matchExpressions:
                - key: service
                  operator: In
                  values:
                  - kafka

  entityOperator:
    topicOperator:
      resources:
        requests:
          memory: 256Mi
          cpu: "0.2"
        limits:
          memory: 256Mi
    userOperator:
      resources:
        requests:
          memory: 512Mi
          cpu: "0.2"
        limits:
          memory: 512Mi

  kafkaExporter:
    resources:
      requests:
        memory: 256Mi
        cpu: "0.2"
      limits:
        memory: 256Mi

[scholzj]

I guess sharing logs from the Cluster Opertor or the full resource including the status might show what the issue it. But the compatibility between different Ingress controllers is not good. So it is quite likely it will be caused by that.

[sattha-p]

Now I try with LB option and restart operator

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"kafka.strimzi.io/v1beta2","kind":"Kafka","metadata":{"annotations":{},"name":"kafka-strimzi","namespace":"kafka-dev"},"spec":{"entityOperator":{"topicOperator":{"resources":{"limits":{"memory":"256Mi"},"requests":{"cpu":"0.2","memory":"256Mi"}}},"userOperator":{"resources":{"limits":{"memory":"512Mi"},"requests":{"cpu":"0.2","memory":"512Mi"}}}},"kafka":{"config":{"default.replication.factor":3,"inter.broker.protocol.version":"3.2","min.insync.replicas":2,"offsets.topic.replication.factor":3,"transaction.state.log.min.isr":2,"transaction.state.log.replication.factor":3},"jvmOptions":{"-Xms":"1024m","-Xmx":"1024m"},"listeners":[{"name":"plain","port":9092,"tls":false,"type":"internal"},{"name":"tls","port":9093,"tls":true,"type":"internal"},{"name":"external","port":9094,"tls":true,"type":"loadbalancer"}],"replicas":3,"resources":{"limits":{"memory":"2Gi"},"requests":{"cpu":0.5,"memory":"2Gi"}},"storage":{"class":"cpt-vsphere-with-tanzu-storage-policy","deleteClaim":false,"size":"50G","type":"persistent-claim"},"template":{"pod":{"affinity":{"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"preference":{"matchExpressions":[{"key":"service","operator":"In","values":["kafka"]}]},"weight":1}]}}}},"version":"3.2.0"},"kafkaExporter":{"resources":{"limits":{"memory":"256Mi"},"requests":{"cpu":"0.2","memory":"256Mi"}}},"zookeeper":{"jvmOptions":{"-Xms":"512m","-Xmx":"512m"},"replicas":3,"resources":{"limits":{"memory":"1Gi"},"requests":{"cpu":0.3,"memory":"1Gi"}},"storage":{"class":"cpt-vsphere-with-tanzu-storage-policy","deleteClaim":false,"size":"50G","type":"persistent-claim"},"template":{"pod":{"affinity":{"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"preference":{"matchExpressions":[{"key":"service","operator":"In","values":["kafka"]}]},"weight":1}]}}}}}}}
  creationTimestamp: '2022-06-08T14:40:20Z'
  generation: 7
  managedFields:
    - apiVersion: kafka.strimzi.io/v1beta2
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:spec:
          .: {}
          f:entityOperator:
            .: {}
            f:topicOperator:
              .: {}
              f:resources:
                .: {}
                f:limits:
                  .: {}
                  f:memory: {}
                f:requests:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
            f:userOperator:
              .: {}
              f:resources:
                .: {}
                f:limits:
                  .: {}
                  f:memory: {}
                f:requests:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
          f:kafka:
            .: {}
            f:config:
              .: {}
              f:default.replication.factor: {}
              f:inter.broker.protocol.version: {}
              f:min.insync.replicas: {}
              f:offsets.topic.replication.factor: {}
              f:transaction.state.log.min.isr: {}
              f:transaction.state.log.replication.factor: {}
            f:jvmOptions:
              .: {}
              f:-Xms: {}
              f:-Xmx: {}
            f:listeners: {}
            f:replicas: {}
            f:resources:
              .: {}
              f:limits:
                .: {}
                f:memory: {}
              f:requests:
                .: {}
                f:cpu: {}
                f:memory: {}
            f:storage:
              .: {}
              f:class: {}
              f:deleteClaim: {}
              f:size: {}
              f:type: {}
            f:template:
              .: {}
              f:pod:
                .: {}
                f:affinity:
                  .: {}
                  f:nodeAffinity:
                    .: {}
                    f:preferredDuringSchedulingIgnoredDuringExecution: {}
            f:version: {}
          f:kafkaExporter:
            .: {}
            f:resources:
              .: {}
              f:limits:
                .: {}
                f:memory: {}
              f:requests:
                .: {}
                f:cpu: {}
                f:memory: {}
          f:zookeeper:
            .: {}
            f:jvmOptions:
              .: {}
              f:-Xms: {}
              f:-Xmx: {}
            f:replicas: {}
            f:resources:
              .: {}
              f:limits:
                .: {}
                f:memory: {}
              f:requests:
                .: {}
                f:cpu: {}
                f:memory: {}
            f:storage:
              .: {}
              f:class: {}
              f:deleteClaim: {}
              f:size: {}
              f:type: {}
            f:template:
              .: {}
              f:pod:
                .: {}
                f:affinity:
                  .: {}
                  f:nodeAffinity:
                    .: {}
                    f:preferredDuringSchedulingIgnoredDuringExecution: {}
      manager: kubectl-client-side-apply
      operation: Update
      time: '2022-06-08T14:40:20Z'
    - apiVersion: kafka.strimzi.io/v1beta2
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:clusterId: {}
          f:conditions: {}
          f:listeners: {}
          f:observedGeneration: {}
      manager: okhttp
      operation: Update
      time: '2022-06-09T04:44:32Z'
  name: kafka-strimzi
  namespace: kafka-dev
  resourceVersion: '184588050'
  uid: e4c12b13-1633-4224-a13a-e69dd2f1f9a9
  selfLink: /apis/kafka.strimzi.io/v1beta2/namespaces/kafka-dev/kafkas/kafka-strimzi
status:
  clusterId: 4Nn0V1PsTi-9qYu3Fw2upg
  conditions:
    - lastTransitionTime: '2022-06-09T04:44:32.697Z'
      status: 'True'
      type: Ready
  listeners:
    - addresses:
        - host: kafka-strimzi-kafka-bootstrap.kafka-dev.svc
          port: 9092
      bootstrapServers: kafka-strimzi-kafka-bootstrap.kafka-dev.svc:9092
      name: plain
      type: plain
    - addresses:
        - host: kafka-strimzi-kafka-bootstrap.kafka-dev.svc
          port: 9093
      bootstrapServers: kafka-strimzi-kafka-bootstrap.kafka-dev.svc:9093
      certificates:
        - |
          -----BEGIN CERTIFICATE-----
          MIIFLTCCAxWgAwIBAgIUfrhPhpPwBZRZ1upfEIAifq2YVVowDQYJKoZIhvcNAQEN
          BQAwLTETMBEGA1UECgwKaW8uc3RyaW16aTEWMBQGA1UEAwwNY2x1c3Rlci1jYSB2
          MDAeFw0yMjA2MDgxNDQwMjRaFw0yMzA2MDgxNDQwMjRaMC0xEzARBgNVBAoMCmlv
          LnN0cmltemkxFjAUBgNVBAMMDWNsdXN0ZXItY2EgdjAwggIiMA0GCSqGSIb3DQEB
          AQUAA4ICDwAwggIKAoICAQCqj9GV6Lw+AFCPBZLYoK8/XjGqZRrldKJkmZp18Qo+
          nn9uFolvrz8vk2ACBiLmEghSXmsXh2xdgrdVofW4jZNodn+c1rsURz9KYCLEivi8
          rtgx0L/8DhCpu2UMUpVenCDBpr3KLRphblsgKGLW+DurD9+qYj/pg6H7rWXsxfek
          Xt/K6WibI8aVMyaJb3iix6BT8U+ZnQy7yF+5tcBmmobizpgIezGxRuw0CUyHsY9z
          9QDKsrMVUgmbGqFIGhUTOdKP3iYcEp1pY9Nkj21+r8ucxmVkNLKMvmzdfyNcwywL
          YVvp+8cU909aCKRD9sDU+3HSOJ8t+cLaru5YLkZmEHl3J1oi9l86D0gyIMWBTiy2
          UfxXt1xB4HPSGp6iV1fWQ+0doAwlhAGNNmTtkdH5fZjhhdBKdvzd5UG/UH6zh57V
          OvqC3adYVTvU/7lyNNunmKPYd3N+fN3NdWHmJ/oqjBnvQXLvW8GmqHHmB3jQRLWC
          gYO1nm6LdQiXyBZyYmd+RykkEERkLcDgqvGtuKIhASr1qvh2hx6M/qGCjSCG02pd
          /T2rQc8tRfRwsn1Xo7ipPqx7ZKRxLESHJG6RPQXhTlfwmswBi0shKd1qDDTK2+qG
          hH3cqOEBSJCfnBIpCroTw5OLlZeeq1fDAJ91Mw7D/Zw33W11CZ5YPcSnmwNo5TmO
          FwIDAQABo0UwQzAdBgNVHQ4EFgQUSfknj0Iewn6odioNPuqCXWHW0qcwEgYDVR0T
          AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAQYwDQYJKoZIhvcNAQENBQADggIB
          AERlS1tu26MdbmVoZ+XD9ONNcfHtxgUcaenM5w7kgr//J/sBs8x2CVhRGv++meju
          vnvAGBrsx2lVO2zUgS8FghZJgQZ27ROZK3MQi7h5bzvyTyAW0eGC1Vfn1eR9cUtI
          QS2Uz93I2mdE1gxYRrP/5aW9E49nYdS/C9cheRxrBLL0C57tc3hFNZbLdYbwIdny
          1IKAH+NS48EWilaKy8dzBH7J5EDPdL4o4p1zFR6ITdiDlMyVSo+pnl0hsVJCT7AY
          vNE0gL2Y4qngMOclp1XZrN0rmGFIontHlmiVvImqLSlcpOHk4/9XiIvgZYvsHoPp
          SbW6xC6Ha9pDjc7U6bI2e44QuMcYUy91KTw/hlnfqmBd1CYGuZ151AYl+BuB/U8g
          zyZDBHCN5fcTjypO6Lx+gZP8TRQf/SIjaTADqJug/Gs0cOHObOwPQZ60C86mmMJT
          U9l3o+JZXzFXPu58jO+d36fflH0QaqeVz2nn/Gh9N81edr9IubTeDHWb5pmIoV7B
          E9gZ3MWqdYAXZJmZu+n46lJczOCN0txh1WTAAyogd/hwc4iyZQ9LAtUp/QnCCmtH
          s1GJ2rouDCtsixkJ1ytYxRYPnSToticquGtDiOpfa1SZTaljOtNTiqMlWm03vh5S
          mfTWf2gEJAkHXGd1WASFCIFvBnn7GSP1H+f4eLZeX30g
          -----END CERTIFICATE-----
      name: tls
      type: tls
  observedGeneration: 1
spec:
  entityOperator:
    topicOperator:
      resources:
        limits:
          memory: 256Mi
        requests:
          cpu: '0.2'
          memory: 256Mi
    userOperator:
      resources:
        limits:
          memory: 512Mi
        requests:
          cpu: '0.2'
          memory: 512Mi
  kafka:
    config:
      default.replication.factor: 3
      inter.broker.protocol.version: '3.2'
      min.insync.replicas: 2
      offsets.topic.replication.factor: 3
      transaction.state.log.min.isr: 2
      transaction.state.log.replication.factor: 3
    jvmOptions:
      '-Xms': 1024m
      '-Xmx': 1024m
    listeners:
      - name: plain
        port: 9092
        tls: false
        type: internal
      - name: tls
        port: 9093
        tls: true
        type: internal
      - name: external
        port: 9094
        tls: true
        type: loadbalancer
    replicas: 3
    resources:
      limits:
        memory: 2Gi
      requests:
        cpu: 0.5
        memory: 2Gi
    storage:
      class: cpt-vsphere-with-tanzu-storage-policy
      deleteClaim: false
      size: 50G
      type: persistent-claim
    template:
      pod:
        affinity:
          nodeAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - preference:
                  matchExpressions:
                    - key: service
                      operator: In
                      values:
                        - kafka
                weight: 1
    version: 3.2.0
  kafkaExporter:
    resources:
      limits:
        memory: 256Mi
      requests:
        cpu: '0.2'
        memory: 256Mi
  zookeeper:
    jvmOptions:
      '-Xms': 512m
      '-Xmx': 512m
    replicas: 3
    resources:
      limits:
        memory: 1Gi
      requests:
        cpu: 0.3
        memory: 1Gi
    storage:
      class: cpt-vsphere-with-tanzu-storage-policy
      deleteClaim: false
      size: 50G
      type: persistent-claim
    template:
      pod:
        affinity:
          nodeAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - preference:
                  matchExpressions:
                    - key: service
                      operator: In
                      values:
                        - kafka
                weight: 1

Operator log after restart

Auto-detected KUBERNETES_SERVICE_DNS_DOMAIN: cluster.local
+ exec /usr/bin/tini -w -e 143 -- java -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom --illegal-access=deny -XX:MinRAMPercentage=10 -XX:MaxRAMPercentage=20 -XX:InitialRAMPercentage=10 -classpath lib/io.strimzi.cluster-operator-0.29.0.jar:lib/io.fabric8.kubernetes-model-discovery-5.12.2.jar:lib/io.netty.netty-transport-4.1.77.Final.jar:lib/io.netty.netty-codec-socks-4.1.77.Final.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.13.1.jar:lib/io.netty.netty-handler-proxy-4.1.77.Final.jar:lib/io.strimzi.kafka-oauth-server-0.10.0.jar:lib/io.fabric8.kubernetes-model-common-5.12.2.jar:lib/io.fabric8.kubernetes-model-autoscaling-5.12.2.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.12.6.jar:lib/io.fabric8.openshift-client-5.12.2.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/io.fabric8.openshift-model-machineconfig-5.12.2.jar:lib/org.apache.zookeeper.zookeeper-jute-3.6.3.jar:lib/io.netty.netty-transport-native-epoll-4.1.77.Final-linux-x86_64.jar:lib/io.strimzi.api-0.29.0.jar:lib/org.apache.zookeeper.zookeeper-3.6.3.jar:lib/io.netty.netty-resolver-dns-4.1.77.Final.jar:lib/net.minidev.accessors-smart-2.4.7.jar:lib/io.netty.netty-handler-4.1.77.Final.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.netty.netty-codec-http-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-metrics-5.12.2.jar:lib/io.fabric8.kubernetes-model-flowcontrol-5.12.2.jar:lib/io.fabric8.openshift-model-hive-5.12.2.jar:lib/io.fabric8.kubernetes-model-apps-5.12.2.jar:lib/com.jayway.jsonpath.json-path-2.6.0.jar:lib/net.minidev.json-smart-2.4.7.jar:lib/io.strimzi.kafka-oauth-server-plain-0.10.0.jar:lib/io.fabric8.openshift-model-operator-5.12.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/io.fabric8.openshift-model-clusterautoscaling-5.12.2.jar:lib/org.yaml.snakeyaml-1.27.jar:lib/io.netty.netty-codec-dns-4.1.77.Final.jar:lib/io.strimzi.certificate-manager-0.29.0.jar:lib/io.netty.netty-resolver-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-core-5.12.2.jar:lib/io.strimzi.kafka-oauth-common-0.10.0.jar:lib/io.fabric8.kubernetes-model-admissionregistration-5.12.2.jar:lib/io.fabric8.kubernetes-model-scheduling-5.12.2.jar:lib/io.netty.netty-transport-native-unix-common-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-batch-5.12.2.jar:lib/io.fabric8.kubernetes-model-coordination-5.12.2.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/io.fabric8.kubernetes-model-extensions-5.12.2.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.fabric8.kubernetes-model-events-5.12.2.jar:lib/io.netty.netty-transport-native-epoll-4.1.77.Final.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/io.vertx.vertx-core-4.2.4.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.4.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/io.fabric8.kubernetes-model-apiextensions-5.12.2.jar:lib/io.fabric8.openshift-model-whereabouts-5.12.2.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/io.strimzi.crd-annotations-0.29.0.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.12.6.1.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.netty.netty-codec-4.1.77.Final.jar:lib/io.fabric8.openshift-model-storageversionmigrator-5.12.2.jar:lib/io.fabric8.kubernetes-model-networking-5.12.2.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/com.nimbusds.nimbus-jose-jwt-9.10.jar:lib/io.fabric8.openshift-model-monitoring-5.12.2.jar:lib/com.fasterxml.jackson.core.jackson-core-2.12.6.jar:lib/io.fabric8.kubernetes-client-5.12.2.jar:lib/com.squareup.okhttp3.okhttp-3.12.12.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/io.fabric8.kubernetes-model-certificates-5.12.2.jar:lib/io.fabric8.openshift-model-console-5.12.2.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/io.strimzi.kafka-oauth-client-0.10.0.jar:lib/io.netty.netty-buffer-4.1.77.Final.jar:lib/org.apache.kafka.kafka-clients-3.2.0.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/io.fabric8.openshift-model-installer-5.12.2.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.fabric8.openshift-model-miscellaneous-5.12.2.jar:lib/io.fabric8.kubernetes-model-storageclass-5.12.2.jar:lib/io.netty.netty-codec-http2-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-node-5.12.2.jar:lib/io.strimzi.config-model-0.29.0.jar:lib/io.fabric8.openshift-model-machine-5.12.2.jar:lib/com.github.luben.zstd-jni-1.5.2-1.jar:lib/io.fabric8.openshift-model-5.12.2.jar:lib/io.netty.netty-transport-classes-epoll-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-policy-5.12.2.jar:lib/io.fabric8.kubernetes-model-rbac-5.12.2.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.strimzi.operator-common-0.29.0.jar:lib/io.fabric8.openshift-model-tuned-5.12.2.jar:lib/io.vertx.vertx-micrometer-metrics-4.2.4.jar:lib/io.fabric8.openshift-model-operatorhub-5.12.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.12.6.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/io.netty.netty-common-4.1.77.Final.jar io.strimzi.operator.cluster.Main
2022-09-06 11:20:21 INFO  Main:60 - ClusterOperator 0.29.0 is starting
2022-09-06 11:20:21 INFO  Main:62 - Cluster Operator configuration is ClusterOperatorConfig(namespaces=[kafka-dev],reconciliationIntervalMs=120000,operationTimeoutMs=300000,connectBuildTimeoutMs=300000,createClusterRoles=false,networkPolicyGeneration=true,versions=versions{3.0.0={proto: 3.0 msg: 3.0 kafka-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.0 connect-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.0 mirrormaker-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.0 mirrormaker2-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.0}, 3.0.1={proto: 3.0 msg: 3.0 kafka-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.1 connect-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.1 mirrormaker-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.1 mirrormaker2-image: quay.io/strimzi/kafka:0.29.0-kafka-3.0.1}, 3.1.0={proto: 3.1 msg: 3.1 kafka-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.0 connect-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.0 mirrormaker-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.0 mirrormaker2-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.0}, 3.1.1={proto: 3.1 msg: 3.1 kafka-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.1 connect-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.1 mirrormaker-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.1 mirrormaker2-image: quay.io/strimzi/kafka:0.29.0-kafka-3.1.1}, 3.2.0={proto: 3.2 msg: 3.2 kafka-image: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0 connect-image: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0 mirrormaker-image: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0 mirrormaker2-image: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0}},imagePullPolicy=null,imagePullSecrets=null,operatorNamespace=kafka-dev,operatorNamespaceLabels=null,customResourceSelector=null,featureGates=FeatureGates(controlPlaneListener=true,ServiceAccountPatching=true,UseStrimziPodSets=false,UseKRaft=false),zkAdminSessionTimeoutMs=10000,dnsCacheTtlSec=30,podSetReconciliationOnly=false,podSetControllerWorkQueueSize=1024)
2022-09-06 11:20:23 WARN  PlatformFeaturesAvailability:157 - API Group route.openshift.io is not supported
2022-09-06 11:20:23 WARN  PlatformFeaturesAvailability:157 - API Group build.openshift.io is not supported
2022-09-06 11:20:23 WARN  PlatformFeaturesAvailability:157 - API Group image.openshift.io is not supported
2022-09-06 11:20:23 INFO  Main:85 - Environment facts gathered: PlatformFeaturesAvailability(KubernetesVersion=1.21,OpenShiftRoutes=false,OpenShiftBuilds=false,OpenShiftImageStreams=false)
2022-09-06 11:20:23 INFO  Util:307 - Using config:
	PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
	container: oci
	KAFKA_STRIMZI_ZOOKEEPER_CLIENT_PORT_2181_TCP_ADDR: 10.250.114.124
	STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE: quay.io/strimzi/operator:0.29.0
	JAVA_OPTS:  -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom  --illegal-access=deny -XX:MinRAMPercentage=10 -XX:MaxRAMPercentage=20 -XX:InitialRAMPercentage=10
	KAFKA_STRIMZI_ZOOKEEPER_CLIENT_SERVICE_HOST: 10.250.114.124
	KAFKA_STRIMZI_KAFKA_BOOTSTRAP_PORT_9091_TCP_ADDR: 10.250.113.28
	KAFKA_STRIMZI_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_CLIENTSTLS: 9093
	KAFKA_STRIMZI_KAFKA_BOOTSTRAP_PORT_9092_TCP_PROTO: tcp
	KAFKA_STRIMZI_ZOOKEEPER_CLIENT_PORT_2181_TCP_PORT: 2181
	STRIMZI_HOME: /opt/strimzi
	PWD: /opt/strimzi
	KUBERNETES_PORT_443_TCP: tcp://10.250.112.1:443
	JAVA_MAIN: io.strimzi.operator.cluster.Main
	KAFKA_STRIMZI_KAFKA_BOOTSTRAP_PORT_9092_TCP_ADDR: 10.250.113.28
	KAFKA_STRIMZI_KAFKA_BOOTSTRAP_SERVICE_PORT: 9091
	STRIMZI_VERSION: 0.29.0
	STRIMZI_DEFAULT_JMXTRANS_IMAGE: quay.io/strimzi/jmxtrans:0.29.0
	STRIMZI_NAMESPACE: kafka-dev,kafka-dev
	STRIMZI_DEFAULT_KANIKO_EXECUTOR_IMAGE: quay.io/strimzi/kaniko-executor:0.29.0
	TINI_SHA256_S390X: 931b70a182af879ca249ae9de87ef68423121b38d235c78997fafc680ceab32d
	STRIMZI_DEFAULT_MAVEN_BUILDER: quay.io/strimzi/maven-builder:0.29.0
	KAFKA_STRIMZI_KAFKA_BOOTSTRAP_PORT_9091_TCP_PROTO: tcp
	STRIMZI_FEATURE_GATES: 
	KAFKA_STRIMZI_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_REPLICATION: 9091
	KAFKA_STRIMZI_ZOOKEEPER_CLIENT_PORT_2181_TCP_PROTO: tcp
	TINI_SHA256_AMD64: 93dcc18adc78c65a028a84799ecf8ad40c936fdfc5f2a57b1acda5a8117fa82c
	KUBERNETES_SERVICE_DNS_DOMAIN: cluster.local
	KAFKA_STRIMZI_KAFKA_BOOTSTRAP_PORT: tcp://10.250.113.28:9091
	KAFKA_STRIMZI_KAFKA_BOOTSTRAP_PORT_9093_TCP_ADDR: 10.250.113.28
	KUBERNETES_SERVICE_PORT_HTTPS: 443
	SHLVL: 0
	KAFKA_STRIMZI_KAFKA_BOOTSTRAP_PORT_9093_TCP: tcp://10.250.113.28:9093
	KAFKA_STRIMZI_KAFKA_BOOTSTRAP_PORT_9093_TCP_PORT: 9093
	KAFKA_STRIMZI_KAFKA_BOOTSTRAP_PORT_9091_TCP: tcp://10.250.113.28:9091
	STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE: quay.io/strimzi/kafka-bridge:0.21.5
	KUBERNETES_PORT: tcp://10.250.112.1:443
	STRIMZI_KAFKA_MIRROR_MAKER_IMAGES: 3.0.0=quay.io/strimzi/kafka:0.29.0-kafka-3.0.0
3.0.1=quay.io/strimzi/kafka:0.29.0-kafka-3.0.1
3.1.0=quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
3.1.1=quay.io/strimzi/kafka:0.29.0-kafka-3.1.1
3.2.0=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0

	JAVA_HOME: /usr/lib/jvm/jre-11
	STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
	KAFKA_STRIMZI_KAFKA_BOOTSTRAP_PORT_9092_TCP: tcp://10.250.113.28:9092
	KUBERNETES_SERVICE_HOST: 10.250.112.1
	KAFKA_STRIMZI_ZOOKEEPER_CLIENT_PORT_2181_TCP: tcp://10.250.114.124:2181
	JAVA_CLASSPATH: lib/io.strimzi.cluster-operator-0.29.0.jar:lib/io.fabric8.kubernetes-model-discovery-5.12.2.jar:lib/io.netty.netty-transport-4.1.77.Final.jar:lib/io.netty.netty-codec-socks-4.1.77.Final.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.13.1.jar:lib/io.netty.netty-handler-proxy-4.1.77.Final.jar:lib/io.strimzi.kafka-oauth-server-0.10.0.jar:lib/io.fabric8.kubernetes-model-common-5.12.2.jar:lib/io.fabric8.kubernetes-model-autoscaling-5.12.2.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.12.6.jar:lib/io.fabric8.openshift-client-5.12.2.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/io.fabric8.openshift-model-machineconfig-5.12.2.jar:lib/org.apache.zookeeper.zookeeper-jute-3.6.3.jar:lib/io.netty.netty-transport-native-epoll-4.1.77.Final-linux-x86_64.jar:lib/io.strimzi.api-0.29.0.jar:lib/org.apache.zookeeper.zookeeper-3.6.3.jar:lib/io.netty.netty-resolver-dns-4.1.77.Final.jar:lib/net.minidev.accessors-smart-2.4.7.jar:lib/io.netty.netty-handler-4.1.77.Final.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.netty.netty-codec-http-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-metrics-5.12.2.jar:lib/io.fabric8.kubernetes-model-flowcontrol-5.12.2.jar:lib/io.fabric8.openshift-model-hive-5.12.2.jar:lib/io.fabric8.kubernetes-model-apps-5.12.2.jar:lib/com.jayway.jsonpath.json-path-2.6.0.jar:lib/net.minidev.json-smart-2.4.7.jar:lib/io.strimzi.kafka-oauth-server-plain-0.10.0.jar:lib/io.fabric8.openshift-model-operator-5.12.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/io.fabric8.openshift-model-clusterautoscaling-5.12.2.jar:lib/org.yaml.snakeyaml-1.27.jar:lib/io.netty.netty-codec-dns-4.1.77.Final.jar:lib/io.strimzi.certificate-manager-0.29.0.jar:lib/io.netty.netty-resolver-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-core-5.12.2.jar:lib/io.strimzi.kafka-oauth-common-0.10.0.jar:lib/io.fabric8.kubernetes-model-admissionregistration-5.12.2.jar:lib/io.fabric8.kubernetes-model-scheduling-5.12.2.jar:lib/io.netty.netty-transport-native-unix-common-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-batch-5.12.2.jar:lib/io.fabric8.kubernetes-model-coordination-5.12.2.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/io.fabric8.kubernetes-model-extensions-5.12.2.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.fabric8.kubernetes-model-events-5.12.2.jar:lib/io.netty.netty-transport-native-epoll-4.1.77.Final.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/io.vertx.vertx-core-4.2.4.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.4.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/io.fabric8.kubernetes-model-apiextensions-5.12.2.jar:lib/io.fabric8.openshift-model-whereabouts-5.12.2.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/io.strimzi.crd-annotations-0.29.0.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.12.6.1.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.netty.netty-codec-4.1.77.Final.jar:lib/io.fabric8.openshift-model-storageversionmigrator-5.12.2.jar:lib/io.fabric8.kubernetes-model-networking-5.12.2.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/com.nimbusds.nimbus-jose-jwt-9.10.jar:lib/io.fabric8.openshift-model-monitoring-5.12.2.jar:lib/com.fasterxml.jackson.core.jackson-core-2.12.6.jar:lib/io.fabric8.kubernetes-client-5.12.2.jar:lib/com.squareup.okhttp3.okhttp-3.12.12.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/io.fabric8.kubernetes-model-certificates-5.12.2.jar:lib/io.fabric8.openshift-model-console-5.12.2.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/io.strimzi.kafka-oauth-client-0.10.0.jar:lib/io.netty.netty-buffer-4.1.77.Final.jar:lib/org.apache.kafka.kafka-clients-3.2.0.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/io.fabric8.openshift-model-installer-5.12.2.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.fabric8.openshift-model-miscellaneous-5.12.2.jar:lib/io.fabric8.kubernetes-model-storageclass-5.12.2.jar:lib/io.netty.netty-codec-http2-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-node-5.12.2.jar:lib/io.strimzi.config-model-0.29.0.jar:lib/io.fabric8.openshift-model-machine-5.12.2.jar:lib/com.github.luben.zstd-jni-1.5.2-1.jar:lib/io.fabric8.openshift-model-5.12.2.jar:lib/io.netty.netty-transport-classes-epoll-4.1.77.Final.jar:lib/io.fabric8.kubernetes-model-policy-5.12.2.jar:lib/io.fabric8.kubernetes-model-rbac-5.12.2.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.strimzi.operator-common-0.29.0.jar:lib/io.fabric8.openshift-model-tuned-5.12.2.jar:lib/io.vertx.vertx-micrometer-metrics-4.2.4.jar:lib/io.fabric8.openshift-model-operatorhub-5.12.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.12.6.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/io.netty.netty-common-4.1.77.Final.jar
	KAFKA_STRIMZI_KAFKA_BOOTSTRAP_SERVICE_PORT_TCP_CLIENTS: 9092
	KAFKA_STRIMZI_KAFKA_BOOTSTRAP_SERVICE_HOST: 10.250.113.28
	STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
	TINI_VERSION: v0.19.0
	STRIMZI_OPERATION_TIMEOUT_MS: 300000
	KUBERNETES_PORT_443_TCP_ADDR: 10.250.112.1
	KAFKA_STRIMZI_KAFKA_BOOTSTRAP_PORT_9093_TCP_PROTO: tcp
	STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES: 3.0.0=quay.io/strimzi/kafka:0.29.0-kafka-3.0.0
3.0.1=quay.io/strimzi/kafka:0.29.0-kafka-3.0.1
3.1.0=quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
3.1.1=quay.io/strimzi/kafka:0.29.0-kafka-3.1.1
3.2.0=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0

	KUBERNETES_PORT_443_TCP_PROTO: tcp
	STRIMZI_DEFAULT_USER_OPERATOR_IMAGE: quay.io/strimzi/operator:0.29.0
	KAFKA_STRIMZI_KAFKA_BOOTSTRAP_PORT_9092_TCP_PORT: 9092
	KUBERNETES_SERVICE_PORT: 443
	KAFKA_STRIMZI_ZOOKEEPER_CLIENT_SERVICE_PORT: 2181
	STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
	TINI_SHA256_ARM64: 07952557df20bfd2a95f9bef198b445e006171969499a1d361bd9e6f8e5e0e81
	STRIMZI_DEFAULT_KAFKA_INIT_IMAGE: quay.io/strimzi/operator:0.29.0
	KAFKA_STRIMZI_ZOOKEEPER_CLIENT_PORT: tcp://10.250.114.124:2181
	TINI_SHA256_PPC64LE: 3f658420974768e40810001a038c29d003728c5fe86da211cff5059e48cfdfde
	HOSTNAME: strimzi-cluster-operator-6c9ff6597b-vbv6j
	KAFKA_STRIMZI_KAFKA_BOOTSTRAP_PORT_9091_TCP_PORT: 9091
	KAFKA_STRIMZI_ZOOKEEPER_CLIENT_SERVICE_PORT_TCP_CLIENTS: 2181
	STRIMZI_KAFKA_CONNECT_IMAGES: 3.0.0=quay.io/strimzi/kafka:0.29.0-kafka-3.0.0
3.0.1=quay.io/strimzi/kafka:0.29.0-kafka-3.0.1
3.1.0=quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
3.1.1=quay.io/strimzi/kafka:0.29.0-kafka-3.1.1
3.2.0=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0

	STRIMZI_KAFKA_IMAGES: 3.0.0=quay.io/strimzi/kafka:0.29.0-kafka-3.0.0
3.0.1=quay.io/strimzi/kafka:0.29.0-kafka-3.0.1
3.1.0=quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
3.1.1=quay.io/strimzi/kafka:0.29.0-kafka-3.1.1
3.2.0=quay.io/strimzi/kafka:0.29.0-kafka-3.2.0

	KUBERNETES_PORT_443_TCP_PORT: 443
	STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE: quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
	HOME: /home/strimzi
	MALLOC_ARENA_MAX: 2
	STRIMZI_OPERATOR_NAMESPACE: kafka-dev

2022-09-06 11:20:23 INFO  ClusterOperator:79 - Creating ClusterOperator for namespace kafka-dev
2022-09-06 11:20:23 INFO  ClusterOperator:94 - Starting ClusterOperator for namespace kafka-dev
2022-09-06 11:20:23 INFO  ClusterOperator:113 - Opened watch for Kafka operator
2022-09-06 11:20:23 INFO  ClusterOperator:113 - Opened watch for KafkaMirrorMaker operator
2022-09-06 11:20:24 INFO  ClusterOperator:113 - Opened watch for KafkaConnect operator
2022-09-06 11:20:24 INFO  ClusterOperator:113 - Opened watch for KafkaBridge operator
2022-09-06 11:20:24 INFO  ClusterOperator:113 - Opened watch for KafkaMirrorMaker2 operator
2022-09-06 11:20:24 INFO  OperatorWatcher:38 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Kafka kafka-strimzi in namespace kafka-dev was ADDED
2022-09-06 11:20:24 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkas' with unstable version 'v1beta2'
2022-09-06 11:20:24 INFO  ClusterOperator:125 - Setting up periodic reconciliation for namespace kafka-dev
2022-09-06 11:20:24 INFO  AbstractOperator:226 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Kafka kafka-strimzi will be checked for creation or modification
2022-09-06 11:20:24 INFO  ClusterOperator:192 - ClusterOperator is now ready (health server listening on 8080)
2022-09-06 11:20:24 INFO  Main:155 - Cluster Operator verticle started in namespace kafka-dev without label selector
2022-09-06 11:20:25 WARN  VersionUsageUtils:60 - The client is using resource type 'strimzipodsets' with unstable version 'v1beta2'
2022-09-06 11:20:45 INFO  KafkaRoller:496 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Pod {kafka-strimzi-kafka-0} needs to be restarted. Reason: [Pod has old generation, server certificates changed]
2022-09-06 11:20:46 INFO  PodOperator:54 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Rolling pod kafka-strimzi-kafka-0
2022-09-06 11:21:19 INFO  KafkaRoller:496 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Pod {kafka-strimzi-kafka-1} needs to be restarted. Reason: [Pod has old generation, server certificates changed]
2022-09-06 11:21:19 INFO  KafkaRoller:298 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Could not roll pod {kafka-strimzi-kafka-1} due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod kafka-strimzi-kafka-1 is currently the controller and there are other pods still to roll, retrying after at least 250ms
2022-09-06 11:21:19 INFO  KafkaRoller:496 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Pod {kafka-strimzi-kafka-2} needs to be restarted. Reason: [Pod has old generation, server certificates changed]
2022-09-06 11:21:19 INFO  KafkaAvailability:135 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): qa.store.product/0 will be under-replicated (ISR={1,2}, replicas=[2,0,1], min.insync.replicas=2) if broker 2 is restarted.
2022-09-06 11:21:19 INFO  KafkaRoller:298 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Could not roll pod {kafka-strimzi-kafka-2} due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod kafka-strimzi-kafka-2 is currently not rollable, retrying after at least 250ms
2022-09-06 11:21:19 INFO  KafkaRoller:496 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Pod {kafka-strimzi-kafka-1} needs to be restarted. Reason: [Pod has old generation, server certificates changed]
2022-09-06 11:21:19 INFO  KafkaRoller:298 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Could not roll pod {kafka-strimzi-kafka-1} due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod kafka-strimzi-kafka-1 is currently the controller and there are other pods still to roll, retrying after at least 500ms
2022-09-06 11:21:19 INFO  KafkaRoller:496 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Pod {kafka-strimzi-kafka-2} needs to be restarted. Reason: [Pod has old generation, server certificates changed]
2022-09-06 11:21:20 INFO  KafkaAvailability:135 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): qa.store.product/0 will be under-replicated (ISR={1,2}, replicas=[2,0,1], min.insync.replicas=2) if broker 2 is restarted.
2022-09-06 11:21:20 INFO  KafkaRoller:298 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Could not roll pod {kafka-strimzi-kafka-2} due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod kafka-strimzi-kafka-2 is currently not rollable, retrying after at least 500ms
2022-09-06 11:21:20 INFO  KafkaRoller:496 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Pod {kafka-strimzi-kafka-1} needs to be restarted. Reason: [Pod has old generation, server certificates changed]
2022-09-06 11:21:20 INFO  KafkaRoller:298 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Could not roll pod {kafka-strimzi-kafka-1} due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod kafka-strimzi-kafka-1 is currently the controller and there are other pods still to roll, retrying after at least 1000ms
2022-09-06 11:21:20 INFO  KafkaRoller:496 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Pod {kafka-strimzi-kafka-2} needs to be restarted. Reason: [Pod has old generation, server certificates changed]
2022-09-06 11:21:20 INFO  KafkaAvailability:135 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): qa.store.product/0 will be under-replicated (ISR={1,2}, replicas=[2,0,1], min.insync.replicas=2) if broker 2 is restarted.
2022-09-06 11:21:20 INFO  KafkaRoller:298 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Could not roll pod {kafka-strimzi-kafka-2} due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod kafka-strimzi-kafka-2 is currently not rollable, retrying after at least 1000ms
2022-09-06 11:21:21 INFO  KafkaRoller:496 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Pod {kafka-strimzi-kafka-1} needs to be restarted. Reason: [Pod has old generation, server certificates changed]
2022-09-06 11:21:21 INFO  KafkaRoller:298 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Could not roll pod {kafka-strimzi-kafka-1} due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod kafka-strimzi-kafka-1 is currently the controller and there are other pods still to roll, retrying after at least 2000ms
2022-09-06 11:21:21 INFO  KafkaRoller:496 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Pod {kafka-strimzi-kafka-2} needs to be restarted. Reason: [Pod has old generation, server certificates changed]
2022-09-06 11:21:21 INFO  KafkaAvailability:135 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): qa.store.product/0 will be under-replicated (ISR={1,2}, replicas=[2,0,1], min.insync.replicas=2) if broker 2 is restarted.
2022-09-06 11:21:21 INFO  KafkaRoller:298 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Could not roll pod {kafka-strimzi-kafka-2} due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod kafka-strimzi-kafka-2 is currently not rollable, retrying after at least 2000ms
2022-09-06 11:21:23 INFO  KafkaRoller:496 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Pod {kafka-strimzi-kafka-1} needs to be restarted. Reason: [Pod has old generation, server certificates changed]
2022-09-06 11:21:23 INFO  KafkaAvailability:135 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): qa.store.product/0 will be under-replicated (ISR={1,2}, replicas=[2,0,1], min.insync.replicas=2) if broker 1 is restarted.
2022-09-06 11:21:23 INFO  KafkaRoller:298 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Could not roll pod {kafka-strimzi-kafka-1} due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod kafka-strimzi-kafka-1 is currently not rollable, retrying after at least 4000ms
2022-09-06 11:21:24 INFO  KafkaRoller:496 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Pod {kafka-strimzi-kafka-2} needs to be restarted. Reason: [Pod has old generation, server certificates changed]
2022-09-06 11:21:24 INFO  KafkaAvailability:135 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): qa.store.product/0 will be under-replicated (ISR={1,2}, replicas=[2,0,1], min.insync.replicas=2) if broker 2 is restarted.
2022-09-06 11:21:24 INFO  KafkaRoller:298 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Could not roll pod {kafka-strimzi-kafka-2} due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod kafka-strimzi-kafka-2 is currently not rollable, retrying after at least 4000ms
2022-09-06 11:21:24 INFO  AbstractOperator:373 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Reconciliation is in progress
2022-09-06 11:21:27 INFO  KafkaRoller:496 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Pod {kafka-strimzi-kafka-1} needs to be restarted. Reason: [Pod has old generation, server certificates changed]
2022-09-06 11:21:27 INFO  KafkaRoller:298 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Could not roll pod {kafka-strimzi-kafka-1} due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod kafka-strimzi-kafka-1 is currently the controller and there are other pods still to roll, retrying after at least 8000ms
2022-09-06 11:21:28 INFO  KafkaRoller:496 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Pod {kafka-strimzi-kafka-2} needs to be restarted. Reason: [Pod has old generation, server certificates changed]
2022-09-06 11:21:28 INFO  KafkaAvailability:135 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): qa.store.product/0 will be under-replicated (ISR={1,2}, replicas=[2,0,1], min.insync.replicas=2) if broker 2 is restarted.
2022-09-06 11:21:28 INFO  KafkaRoller:298 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Could not roll pod {kafka-strimzi-kafka-2} due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod kafka-strimzi-kafka-2 is currently not rollable, retrying after at least 8000ms
2022-09-06 11:21:35 INFO  KafkaRoller:496 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Pod {kafka-strimzi-kafka-1} needs to be restarted. Reason: [Pod has old generation, server certificates changed]
2022-09-06 11:21:36 INFO  KafkaRoller:298 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Could not roll pod {kafka-strimzi-kafka-1} due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod kafka-strimzi-kafka-1 is currently the controller and there are other pods still to roll, retrying after at least 16000ms
2022-09-06 11:21:36 INFO  KafkaRoller:496 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Pod {kafka-strimzi-kafka-2} needs to be restarted. Reason: [Pod has old generation, server certificates changed]
2022-09-06 11:21:36 INFO  PodOperator:54 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Rolling pod kafka-strimzi-kafka-2
2022-09-06 11:22:20 INFO  KafkaRoller:496 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Pod {kafka-strimzi-kafka-1} needs to be restarted. Reason: [Pod has old generation, server certificates changed]
2022-09-06 11:22:20 INFO  PodOperator:54 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Rolling pod kafka-strimzi-kafka-1
2022-09-06 11:22:24 INFO  AbstractOperator:373 - Reconciliation #1(watch) Kafka(kafka-dev/kafka-strimzi): Reconciliation is in progress
2022-09-06 11:22:24 INFO  ClusterOperator:128 - Triggering periodic reconciliation for namespace kafka-dev
2022-09-06 11:22:24 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkamirrormakers' with unstable version 'v1beta2'
2022-09-06 11:22:24 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkaconnects' with unstable version 'v1beta2'
2022-09-06 11:22:24 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkamirrormaker2s' with unstable version 'v1beta2'
2022-09-06 11:22:24 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkabridges' with unstable version 'v1beta2'
2022-09-06 11:22:24 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkarebalances' with unstable version 'v1beta2'
2022-09-06 11:22:24 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkaconnectors' with unstable version 'v1beta2'

[sattha-p]

it works after restart cluster operator for load balance option. I will go with this for now. thanks for help.

[rajjaiswalsaumya]

Hi, I am also facing the same issue where ingress does not work but the load balancer does in GCP.
We have an organization restriction from our secops team to expose only port 443 to the outside world. Currently, we are integrating with a team that is lying on a different VPC due to which we need to expose Kafka on the load balancer. I tried setting port to 443 in the external listener but that doesn't work. Can someone please point out what are the correct configs that must be set to put Kafka exposed on port 443 instead of 9094? How can i make ingress work ?

[scholzj]

Strimzi supports Kubernetes Nginx Ingress using TLS passthrough which can be used to expose the broker on port 443. You can also create an internal listeners with customized advertised hosts and manage the Ingress resources for them yourself which allows you to use basically whatever you want to expose it.

[rajjaiswalsaumya]

@scholzj A sample would help, can you please provide a sample ref what to change and where ?