upgrade error from 20.1; "error ApiVersion: might be caused by broker version < 0.10"

[kaviankarimzadeh]

Hi,

Cluster version: 0.20.1
Deployed as Helm chart.

Situation: after upgrading to any version newer than current one we get following errors and could not connect from external client to cluster.

Error:
with kafkacat:

+connect.config

security.protocol=SSL
ssl.keystore.location=new.p12
ssl.keystore.password=roBiEmbYtftc
ssl.key.password=roBiEmbYtftc

+kafkacat -F connect.config -L -b kafka.example.com:443
% Reading configuration from file connect.config
%6|1668526958.009|FAIL|rdkafka#producer-1| [thrd:ssl://kafka.example.com:443/bootstrap]: ssl://kafka.example.com:443/bootstrap: Disconnected while requesting ApiVersion: might be caused by broker version < 0.10 (see api.version.request) (after 34ms in state APIVERSION_QUERY)
%6|1668526958.326|FAIL|rdkafka#producer-1| [thrd:ssl://kafka.example.com:443/bootstrap]: ssl://kafka.example.com:443/bootstrap: Disconnected while requesting ApiVersion: might be caused by broker version < 0.10 (see api.version.request) (after 36ms in state APIVERSION_QUERY, 1 identical error(s) suppressed)
% ERROR: Failed to acquire metadata: Local: Broker transport failure

Steps for upgrade process from 0.20 to 0.22:
based on Strimzi docs:

#Download 0.22 Helm chart:

{
  wget https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.22.0/strimzi-kafka-operator-helm-3-chart-0.22.0.tgz
  tar xzvf strimzi-kafka-operator-helm-3-chart-0.22.0.tgz
}

#Update values.yaml wih correct wathNamespace and DNSDomain:

kubernetesServiceDnsDomain: kube-lab.example.com
watchNamespaces:
- kafka-cluster #name of ns that kafka-cluster instance is installed.

#Run upgrade command:

helm upgrade -n kafka-operator kafka-operator strimzi-kafka-operator/

#after this command operator, zookeeper, kafka, entityoperator are restarted with no error in logs.

#upgrade crds (step1)

{
  wget https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.22.0/strimzi-kafka-operator-helm-3-chart-0.22.0.tgz
  tar xzvf strimzi-kafka-operator-helm-3-chart-0.22.0.tgz
}

cd strimzi-kafka-operator/crds
kubectl -n kafka-cluster replace -f .

#upgrade crds (step2)

{
  wget https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.22.0/api-conversion-0.22.0.tar.gz
  tar xzvf api-conversion-0.22.0.tar.gz
  cd api-conversion-0.22.0/bin
}

./api-conversion.sh convert-resource --all-namespaces --kind KafkaConnect --kind KafkaMirrorMaker2 --kind KafkaUser --kind KafkaRebalance --kind KafkaMirrorMaker --kind Kafka --kind KafkaBridge --kind KafkaTopic --kind KafkaConnector

./api-conversion.sh crd-upgrade --debug

#after these steps, all crds version changed to v1beta2 version.

at this steps we get error while before upgrade process it worked!

+kafkacat -F connect.config -L -b kafka.example.com:443
% Reading configuration from file connect.config
%6|1668526958.009|FAIL|rdkafka#producer-1| [thrd:ssl://kafka.example.com:443/bootstrap]: ssl://kafka.example.com:443/bootstrap: Disconnected while requesting ApiVersion: might be caused by broker version < 0.10 (see api.version.request) (after 34ms in state APIVERSION_QUERY)
%6|1668526958.326|FAIL|rdkafka#producer-1| [thrd:ssl://kafka.example.com:443/bootstrap]: ssl://kafka.example.com:443/bootstrap: Disconnected while requesting ApiVersion: might be caused by broker version < 0.10 (see api.version.request) (after 36ms in state APIVERSION_QUERY, 1 identical error(s) suppressed)
% ERROR: Failed to acquire metadata: Local: Broker transport failure

Anyway based on docs, we continue to upgrade to latest version but we got same error on version 0.31.1:

#first enable STRIMZI_FEATURE_GATES in kafka operator deployment file:

  - name: STRIMZI_FEATURE_GATES
    value: +ControlPlaneListene

#Download 0.31.1 helm chart version:

{
  wget https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.31.1/strimzi-kafka-operator-helm-3-chart-0.31.1.tgz
  tar xzvf strimzi-kafka-operator-helm-3-chart-0.31.1.tgz
}

#Update values.yaml wih correct wathNamespace and DNSDomain:

kubernetesServiceDnsDomain: kube-lab.example.com
watchNamespaces:
- kafka-cluster #name of ns that kafka-cluster instance is installed.

#update and add new crd(strimzipodset):

cd strimzi-kafka-operator/crds
kubectl -n kafka-cluster replace -f .
kubectl -n kafka-cluster apply -f 042-Crd-strimzipodset.yaml

helm upgrade -n kafka-operator kafka-operator strimzi-kafka-operator/

#at this step, kafka operator complains about incorrect kafka-cluster version.
#so we manually change following fields in kafka-cluster:

      inter.broker.protocol.version: '3.2'
      log.message.format.version: '3.2'
    version: 3.2.3

And again we get same error after all components restarted and up and running!

+kafkacat -F connect.config -L -b kafka.example.com:443
% Reading configuration from file connect.config
%6|1668526958.009|FAIL|rdkafka#producer-1| [thrd:ssl://kafka.example.com:443/bootstrap]: ssl://kafka.example.com:443/bootstrap: Disconnected while requesting ApiVersion: might be caused by broker version < 0.10 (see api.version.request) (after 34ms in state APIVERSION_QUERY)
%6|1668526958.326|FAIL|rdkafka#producer-1| [thrd:ssl://kafka.example.com:443/bootstrap]: ssl://kafka.example.com:443/bootstrap: Disconnected while requesting ApiVersion: might be caused by broker version < 0.10 (see api.version.request) (after 36ms in state APIVERSION_QUERY, 1 identical error(s) suppressed)
% ERROR: Failed to acquire metadata: Local: Broker transport failure

[scholzj]

Some comments to your upgrade process:

• You have the steps around the ControlPlaneListener wrong. You cannot enable it on 0.22 because it does not exist there. You need to first install 0.31 with it disabled (i.e. set to -ControlPlaneListener). And only then, after the upgrade is complete you can enable it (+ControlPlaneListener). Also, the snipper you shared there has a typo => it is missing r. This might be only copy paste error here, not sure about that.
• After every upgrade, you have to install the new CRDs. The api-conversion tool is one-time thing when upgrading from 0.22 to 0.23 and higher. But the new CRDs need to be applied after every upgrade.

Unfortunately, you do not share the actual config you have of the cluster, what is the current condition and what is running or not. SO it is hard to comment on the actual issue. I do not know librdkafka, so I'm not familiar with what it logs and how it works. The URL kafka.example.com:443/bootstrap seems weird as it is not HTTP URL. But that might be simply how it logs.

From the address, I guess you use Ingress, so my best guess would be that this error means that you are not reaching the broker. So the question is if the brokers are ready or not and what are they doing.

[kaviankarimzadeh]

Hi,
Thanks for your comment.
Regarding ControlPlaneListener, Yes you are right. that was my mistake at shared info. As I have checked I have enabled that after upgrade process on 0.31. (now it is configured as +ControlPlaneListener)
I've missed one step in my shared information and that was related to upgrade from 0.22 to 0.23.

In another word, my upgrade process happened as below:
0.20.0 >> 0.22.0 >> 0.23.0 >> 0.31.1

Installing new CRDs happened after each upgrade process. From 0.20 to 0.22, from 0.22 to 0.23 and from 0.23 to 0.31.
The api-conversion tool is only happened after upgrading from 0.22 to 0.23.

The mentioned error happened exactly after first upgrade from 0.20 to 0.22 and exists after next upgrades as well.

Regarding kafkacat or librdkafka, it worked before upgrade. anyway with other tools like docker container to consume or produce got error again while before upgrade it was fine. If there is any specific client you need to have it logs, please let me know to check and shared.

Also "tls" is enabled in kafka cluster. In the following I've shared kafka-cluster config.

And you are right, the address is ingress. it is reachable and mentioned port is open. There is no any error in kafka brokers log.

Here you can find kafka-cluster config file. If any other config is required please let me know.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  annotations:
  creationTimestamp: '2022-11-15T12:44:56Z'
  generation: 3
  name: kafka-cluster
  namespace: kafka-cluster
  resourceVersion: '67728574'
  selfLink: /apis/kafka.strimzi.io/v1beta2/namespaces/kafka-cluster/kafkas/kafka-cluster
  uid: 2f97d4a1-d87d-45a7-8e06-ae3e4c79e7fd
status:
  clusterId: _-9jyWhPT0q6O_Wki3s8Fg
  conditions:
    - lastTransitionTime: '2022-11-15T15:34:14.466133Z'
      message: >-
        min.insync.replicas option is not configured. It defaults to 1 which
        does not guarantee reliability and availability. You should configure
        this option in .spec.kafka.config.
      reason: KafkaMinInsyncReplicas
      status: 'True'
      type: Warning
    - lastTransitionTime: '2022-11-15T15:45:28.392Z'
      status: 'True'
      type: Ready
  listeners:
    - addresses:
        - host: >-
            kafka-cluster-kafka-bootstrap.kafka-cluster.svc.kafka.example.com
          port: 9093
      bootstrapServers: >-
        kafka-cluster-kafka-bootstrap.kafka-cluster.svc.kafka.example.com:9093
      certificates:
        - |
          -----BEGIN CERTIFICATE-----
          MIIDLTCCAhWgAwIBAgIJAM61zG7lnO3MMA0GCSqGSIb3DQEBCwUAMC0xEzARBgNV
          BAoMCmlvLnN0cmltemkxFjAUBgNVBAMMDWNsdXN0ZXItY2EgdjAwHhcNMjIxMTE1
          MTI0NDU3WhcNMzIxMTEyMTI0NDU3WjAtMRMwEQYDVQQKDAppby5zdHJpbXppMRYw
          .
          .
          .
          -----END CERTIFICATE-----
      name: tls
      type: tls
    - addresses:
        - host: kafka.example.com
          port: 443
      bootstrapServers: kafka.example.com:443
      certificates:
        - |
          -----BEGIN CERTIFICATE-----
          MIIFcjCCBFqgAwIBAgISBBmtEdWIT75GQ6sMtYCacqlJMA0GCSqGSIb3DQEBCwUA
          MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
          EwJSMzAeFw0yMjEwMTYyMTM1NTZaFw0yMzAxMTQyMTM1NTVaMCYxJDAiBgNVBAMT
          .
          .
          .
          p8oP5hPU
          -----END CERTIFICATE-----

          -----BEGIN CERTIFICATE-----
          MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
          TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
          cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
          .
          .
          .
          -----END CERTIFICATE-----

          -----BEGIN CERTIFICATE-----
          MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
          MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
          DkRTVCBSb290IENBIFgzMB4XDTIxMDEyMDE5MTQwM1oXDTI0MDkzMDE4MTQwM1ow
          .
          .
          .
          -----END CERTIFICATE-----
      name: external
      type: external
  observedGeneration: 3
spec:
  clientsCa:
    renewalDays: 30
    validityDays: 3650
  clusterCa:
    renewalDays: 30
    validityDays: 3650
  entityOperator:
    tlsSidecar:
      resources:
        limits:
          cpu: 500m
          memory: 512Mi
        requests:
          cpu: 200m
          memory: 256Mi
    topicOperator:
      logging:
        loggers:
          rootLogger.level: INFO
        type: inline
      reconciliationIntervalSeconds: 60
      resources:
        limits:
          cpu: '1'
          memory: 768Mi
        requests:
          cpu: 300m
          memory: 512Mi
      watchedNamespace: kafka-cluster
    userOperator:
      logging:
        loggers:
          rootLogger.level: INFO
        type: inline
      reconciliationIntervalSeconds: 60
      resources:
        limits:
          cpu: '1'
          memory: 768Mi
        requests:
          cpu: 300m
          memory: 512Mi
      watchedNamespace: kafka-cluster
  kafka:
    authorization:
      type: simple
    config:
      auto.create.topics.enable: 'false'
      compression.type: producer
      default.replication.factor: 3
      inter.broker.protocol.version: '3.2'
      log.message.format.version: '3.2'
      num.partitions: 10
      offsets.topic.replication.factor: 1
      security.inter.broker.protocol: PLAINTEXT
      ssl.cipher.suites: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
      ssl.enabled.protocols: TLSv1.2
      ssl.protocol: TLSv1.2
      transaction.state.log.min.isr: 1
      transaction.state.log.replication.factor: 2
    jvmOptions:
      '-Xms': 1024m
      '-Xmx': 1024m
    listeners:
      - authentication:
          type: tls
        configuration:
          useServiceDnsDomain: true
        name: tls
        port: 9093
        tls: true
        type: internal
      - authentication:
          type: tls
        configuration:
          bootstrap:
            host: kafka.example.com
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: kafka-cluster-secret
          brokers:
            - broker: 0
              host: kafka-0.kafka.example.com
            - broker: 1
              host: kafka-1.kafka.example.com
            - broker: 2
              host: kafka-2.kafka.example.com
        name: external
        port: 9094
        tls: true
        type: ingress
    livenessProbe: {}
    logging:
      loggers:
        kafka.root.logger.level: INFO
      type: inline
    readinessProbe:
      initialDelaySeconds: 180
    replicas: 3
    resources:
      limits:
        cpu: '1'
        memory: 2Gi
      requests:
        cpu: '1'
        memory: 1Gi
    storage:
      type: jbod
      volumes:
        - id: 0
          size: 30Gi
          type: persistent-claim
    template:
      pod:
        metadata:
          labels:
            owned-by: nids
        securityContext:
          fsGroup: 1001
          runAsUser: 1001
        terminationGracePeriodSeconds: 300
    version: 3.2.3
  kafkaExporter:
    enableSaramaLogging: true
    image: quay.io/strimzi/kafka:0.31.1-kafka-3.2.3
    logging: debug
    resources:
      limits:
        cpu: 1000m
        memory: 512Mi
      requests:
        cpu: 500m
        memory: 256Mi
    template:
      pod:
        metadata:
          labels:
            label1: kafka-cluster
        securityContext:
          fsGroup: 1001
          runAsUser: 1001
        terminationGracePeriodSeconds: 120
  zookeeper:
    config:
      sslQuorum: true
    jvmOptions:
      '-Xms': 2048m
      '-Xmx': 2048m
    livenessProbe: {}
    readinessProbe: {}
    replicas: 3
    resources:
      limits:
        cpu: '2'
        memory: 4Gi
      requests:
        cpu: '1'
        memory: 512Mi
    storage:
      size: 30Gi
      type: persistent-claim
    template:
      pod:
        metadata:
          labels:
            owned-by: nids
        securityContext:
          fsGroup: 1001
          runAsUser: 1001
        terminationGracePeriodSeconds: 300

##kafka configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: kafka-cluster-kafka-0
  namespace: kafka-cluster
  selfLink: /api/v1/namespaces/kafka-cluster/configmaps/kafka-cluster-kafka-0
  uid: 4ff950d9-dce6-4ef1-b046-e8e3f27c7dcc
  resourceVersion: '67728616'
  creationTimestamp: '2022-11-15T15:34:15Z'
  labels:
    app.kubernetes.io/instance: kafka-cluster
    app.kubernetes.io/managed-by: strimzi-cluster-operator
    app.kubernetes.io/name: kafka
    app.kubernetes.io/part-of: strimzi-kafka-cluster
    strimzi.io/cluster: kafka-cluster
    strimzi.io/kind: Kafka
    strimzi.io/name: kafka-cluster-kafka
    strimzi.io/pod-name: kafka-cluster-kafka-0
data:
  listeners.config: TLS_9093 EXTERNAL_9094
  log4j.properties: >
    # Do not change this generated file. Logging can be configured in the
    corresponding Kubernetes resource.

    log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender

    log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout

    log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} %p %m (%c)
    [%t]%n

    kafka.root.logger.level=INFO

    log4j.rootLogger=${kafka.root.logger.level}, CONSOLE

    log4j.logger.org.I0Itec.zkclient.ZkClient=INFO

    log4j.logger.org.apache.zookeeper=INFO

    log4j.logger.kafka=INFO

    log4j.logger.org.apache.kafka=INFO

    log4j.logger.kafka.request.logger=WARN, CONSOLE

    log4j.logger.kafka.network.Processor=OFF

    log4j.logger.kafka.server.KafkaApis=OFF

    log4j.logger.kafka.network.RequestChannel$=WARN

    log4j.logger.kafka.controller=TRACE

    log4j.logger.kafka.log.LogCleaner=INFO

    log4j.logger.state.change.logger=TRACE

    log4j.logger.kafka.authorizer.logger=INFO
  server.config: >-
    ##############################

    ##############################

    # This file is automatically generated by the Strimzi Cluster Operator

    # Any changes to this file will be ignored and overwritten!

    ##############################

    ##############################


    ##########

    # Broker ID

    ##########

    broker.id=0

    node.id=0


    ##########

    # Zookeeper

    ##########

    zookeeper.connect=kafka-cluster-zookeeper-client:2181

    zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty

    zookeeper.ssl.client.enable=true

    zookeeper.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12

    zookeeper.ssl.keystore.password=${CERTS_STORE_PASSWORD}

    zookeeper.ssl.keystore.type=PKCS12

    zookeeper.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12

    zookeeper.ssl.truststore.password=${CERTS_STORE_PASSWORD}

    zookeeper.ssl.truststore.type=PKCS12


    ##########

    # Kafka message logs configuration

    ##########

    log.dirs=/var/lib/kafka/data-0/kafka-log0


    ##########

    # Control Plane listener

    ##########

    listener.name.controlplane-9090.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12

    listener.name.controlplane-9090.ssl.keystore.password=${CERTS_STORE_PASSWORD}

    listener.name.controlplane-9090.ssl.keystore.type=PKCS12

    listener.name.controlplane-9090.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12

    listener.name.controlplane-9090.ssl.truststore.password=${CERTS_STORE_PASSWORD}

    listener.name.controlplane-9090.ssl.truststore.type=PKCS12

    listener.name.controlplane-9090.ssl.client.auth=required


    ##########

    # Replication listener

    ##########

    listener.name.replication-9091.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12

    listener.name.replication-9091.ssl.keystore.password=${CERTS_STORE_PASSWORD}

    listener.name.replication-9091.ssl.keystore.type=PKCS12

    listener.name.replication-9091.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12

    listener.name.replication-9091.ssl.truststore.password=${CERTS_STORE_PASSWORD}

    listener.name.replication-9091.ssl.truststore.type=PKCS12

    listener.name.replication-9091.ssl.client.auth=required


    ##########

    # Listener configuration: TLS-9093

    ##########

    listener.name.tls-9093.ssl.client.auth=required

    listener.name.tls-9093.ssl.truststore.location=/tmp/kafka/clients.truststore.p12

    listener.name.tls-9093.ssl.truststore.password=${CERTS_STORE_PASSWORD}

    listener.name.tls-9093.ssl.truststore.type=PKCS12


    listener.name.tls-9093.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12

    listener.name.tls-9093.ssl.keystore.password=${CERTS_STORE_PASSWORD}

    listener.name.tls-9093.ssl.keystore.type=PKCS12



    ##########

    # Listener configuration: EXTERNAL-9094

    ##########

    listener.name.external-9094.ssl.client.auth=required

    listener.name.external-9094.ssl.truststore.location=/tmp/kafka/clients.truststore.p12

    listener.name.external-9094.ssl.truststore.password=${CERTS_STORE_PASSWORD}

    listener.name.external-9094.ssl.truststore.type=PKCS12


    listener.name.external-9094.ssl.keystore.location=/tmp/kafka/custom-external-9094.keystore.p12

    listener.name.external-9094.ssl.keystore.password=${CERTS_STORE_PASSWORD}

    listener.name.external-9094.ssl.keystore.type=PKCS12



    ##########

    # Common listener configuration

    ##########

    listeners=CONTROLPLANE-9090://0.0.0.0:9090,REPLICATION-9091://0.0.0.0:9091,TLS-9093://0.0.0.0:9093,EXTERNAL-9094://0.0.0.0:9094

    advertised.listeners=CONTROLPLANE-9090://kafka-cluster-kafka-0.kafka-cluster-kafka-brokers.kafka-cluster.svc:9090,REPLICATION-9091://kafka-cluster-kafka-0.kafka-cluster-kafka-brokers.kafka-cluster.svc:9091,TLS-9093://kafka-cluster-kafka-0.kafka-cluster-kafka-brokers.kafka-cluster.svc.kafka.example.com:9093,EXTERNAL-9094://kafka-0.kafka.example.com:443

    listener.security.protocol.map=CONTROLPLANE-9090:SSL,REPLICATION-9091:SSL,TLS-9093:SSL,EXTERNAL-9094:SSL

    control.plane.listener.name=CONTROLPLANE-9090

    inter.broker.listener.name=REPLICATION-9091

    sasl.enabled.mechanisms=

    ssl.secure.random.implementation=SHA1PRNG

    ssl.endpoint.identification.algorithm=HTTPS


    ##########

    # Authorization

    ##########

    authorizer.class.name=kafka.security.authorizer.AclAuthorizer

    super.users=User:CN=kafka-cluster-kafka,O=io.strimzi;User:CN=kafka-cluster-entity-topic-operator,O=io.strimzi;User:CN=kafka-cluster-entity-user-operator,O=io.strimzi;User:CN=kafka-cluster-kafka-exporter,O=io.strimzi;User:CN=kafka-cluster-cruise-control,O=io.strimzi;User:CN=cluster-operator,O=io.strimzi


    ##########

    # User provided configuration

    ##########

    auto.create.topics.enable=false

    compression.type=producer

    default.replication.factor=3

    inter.broker.protocol.version=3.2

    log.message.format.version=3.2

    num.partitions=10

    offsets.topic.replication.factor=1

    ssl.cipher.suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

    ssl.enabled.protocols=TLSv1.2

    ssl.protocol=TLSv1.2

    transaction.state.log.min.isr=1

    transaction.state.log.replication.factor=2
binaryData: {}

#kafka StrimziPodSet

apiVersion: core.strimzi.io/v1beta2
kind: StrimziPodSet
metadata:
  annotations:
    strimzi.io/kafka-version: 3.2.3
    strimzi.io/storage: >-
      {"volumes":[{"type":"persistent-claim","size":"30Gi","id":0}],"type":"jbod"}
  creationTimestamp: '2022-11-15T15:34:16Z'
  generation: 2
  labels:
    app.kubernetes.io/instance: kafka-cluster
    app.kubernetes.io/managed-by: strimzi-cluster-operator
    app.kubernetes.io/name: kafka
    app.kubernetes.io/part-of: strimzi-kafka-cluster
    strimzi.io/cluster: kafka-cluster
    strimzi.io/kind: Kafka
    strimzi.io/name: kafka-cluster-kafka
  name: kafka-cluster-kafka
  namespace: kafka-cluster
  ownerReferences:
    - apiVersion: kafka.strimzi.io/v1beta2
      blockOwnerDeletion: false
      controller: false
      kind: Kafka
      name: kafka-cluster
      uid: 2f97d4a1-d87d-45a7-8e06-ae3e4c79e7fd
  resourceVersion: '67731176'
  selfLink: >-
    /apis/core.strimzi.io/v1beta2/namespaces/kafka-cluster/strimzipodsets/kafka-cluster-kafka
  uid: b5d5a405-ba82-4b23-8bc0-686ff9b40bb6
status:
  currentPods: 3
  observedGeneration: 2
  pods: 3
  readyPods: 3
spec:
  pods:
    - apiVersion: v1
      kind: Pod
      metadata:
        annotations:
          strimzi.io/broker-configuration-hash: 7328cb7a
          strimzi.io/clients-ca-cert-generation: '0'
          strimzi.io/cluster-ca-cert-generation: '0'
          strimzi.io/custom-listener-cert-thumbprints: '{external=fb3a4698}'
          strimzi.io/inter-broker-protocol-version: '3.2'
          strimzi.io/kafka-version: 3.2.3
          strimzi.io/log-message-format-version: '3.2'
          strimzi.io/logging-appenders-hash: e893ac9f
          strimzi.io/revision: bbc272b2
        labels:
          app.kubernetes.io/instance: kafka-cluster
          app.kubernetes.io/managed-by: strimzi-cluster-operator
          app.kubernetes.io/name: kafka
          app.kubernetes.io/part-of: strimzi-kafka-cluster
          owned-by: nids
          statefulset.kubernetes.io/pod-name: kafka-cluster-kafka-0
          strimzi.io/cluster: kafka-cluster
          strimzi.io/controller: strimzipodset
          strimzi.io/controller-name: kafka-cluster-kafka
          strimzi.io/kind: Kafka
          strimzi.io/name: kafka-cluster-kafka
          strimzi.io/pod-name: kafka-cluster-kafka-0
        name: kafka-cluster-kafka-0
        namespace: kafka-cluster
      spec:
        affinity: {}
        containers:
          - command:
              - /opt/kafka/kafka_run.sh
            env:
              - name: KAFKA_METRICS_ENABLED
                value: 'false'
              - name: STRIMZI_KAFKA_GC_LOG_ENABLED
                value: 'false'
              - name: KAFKA_HEAP_OPTS
                value: '-Xms1024m -Xmx1024m'
            image: quay.io/strimzi/kafka:0.31.1-kafka-3.2.3
            imagePullPolicy: IfNotPresent
            livenessProbe:
              exec:
                command:
                  - /opt/kafka/kafka_liveness.sh
              initialDelaySeconds: 15
              timeoutSeconds: 5
            name: kafka
            ports:
              - containerPort: 9090
                name: tcp-ctrlplane
                protocol: TCP
              - containerPort: 9091
                name: tcp-replication
                protocol: TCP
              - containerPort: 9093
                name: tcp-clientstls
                protocol: TCP
              - containerPort: 9094
                name: tcp-external
                protocol: TCP
            readinessProbe:
              exec:
                command:
                  - test
                  - '-f'
                  - /var/opt/kafka/kafka-ready
              initialDelaySeconds: 180
              timeoutSeconds: 5
            resources:
              limits:
                cpu: '1'
                memory: 2Gi
              requests:
                cpu: '1'
                memory: 1Gi
            volumeMounts:
              - mountPath: /var/lib/kafka/data-0
                name: data-0
              - mountPath: /tmp
                name: strimzi-tmp
              - mountPath: /opt/kafka/cluster-ca-certs
                name: cluster-ca
              - mountPath: /opt/kafka/broker-certs
                name: broker-certs
              - mountPath: /opt/kafka/client-ca-certs
                name: client-ca-cert
              - mountPath: /opt/kafka/custom-config/
                name: kafka-metrics-and-logging
              - mountPath: /var/opt/kafka
                name: ready-files
              - mountPath: /opt/kafka/certificates/custom-external-9094-certs
                name: custom-external-9094-certs
        hostname: kafka-cluster-kafka-0
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext:
          fsGroup: 1001
          runAsUser: 1001
        serviceAccountName: kafka-cluster-kafka
        subdomain: kafka-cluster-kafka-brokers
        terminationGracePeriodSeconds: 300
        volumes:
          - name: data-0
            persistentVolumeClaim:
              claimName: data-0-kafka-cluster-kafka-0
          - emptyDir:
              medium: Memory
              sizeLimit: 5Mi
            name: strimzi-tmp
          - name: cluster-ca
            secret:
              defaultMode: 292
              secretName: kafka-cluster-cluster-ca-cert
          - name: broker-certs
            secret:
              defaultMode: 292
              secretName: kafka-cluster-kafka-brokers
          - name: client-ca-cert
            secret:
              defaultMode: 292
              secretName: kafka-cluster-clients-ca-cert
          - configMap:
              name: kafka-cluster-kafka-0
            name: kafka-metrics-and-logging
          - emptyDir:
              medium: Memory
              sizeLimit: 1Ki
            name: ready-files
          - name: custom-external-9094-certs
            secret:
              defaultMode: 292
              items:
                - key: tls.key
                  path: tls.key
                - key: tls.crt
                  path: tls.crt
              secretName: kafka-cluster-secret
    - apiVersion: v1
      kind: Pod
      metadata:
        annotations:
          strimzi.io/broker-configuration-hash: e9797824
          strimzi.io/clients-ca-cert-generation: '0'
          strimzi.io/cluster-ca-cert-generation: '0'
          strimzi.io/custom-listener-cert-thumbprints: '{external=fb3a4698}'
          strimzi.io/inter-broker-protocol-version: '3.2'
          strimzi.io/kafka-version: 3.2.3
          strimzi.io/log-message-format-version: '3.2'
          strimzi.io/logging-appenders-hash: e893ac9f
          strimzi.io/revision: 389fdbc7
        labels:
          app.kubernetes.io/instance: kafka-cluster
          app.kubernetes.io/managed-by: strimzi-cluster-operator
          app.kubernetes.io/name: kafka
          app.kubernetes.io/part-of: strimzi-kafka-cluster
          owned-by: nids
          statefulset.kubernetes.io/pod-name: kafka-cluster-kafka-1
          strimzi.io/cluster: kafka-cluster
          strimzi.io/controller: strimzipodset
          strimzi.io/controller-name: kafka-cluster-kafka
          strimzi.io/kind: Kafka
          strimzi.io/name: kafka-cluster-kafka
          strimzi.io/pod-name: kafka-cluster-kafka-1
        name: kafka-cluster-kafka-1
        namespace: kafka-cluster
      spec:
        affinity: {}
        containers:
          - command:
              - /opt/kafka/kafka_run.sh
            env:
              - name: KAFKA_METRICS_ENABLED
                value: 'false'
              - name: STRIMZI_KAFKA_GC_LOG_ENABLED
                value: 'false'
              - name: KAFKA_HEAP_OPTS
                value: '-Xms1024m -Xmx1024m'
            image: quay.io/strimzi/kafka:0.31.1-kafka-3.2.3
            imagePullPolicy: IfNotPresent
            livenessProbe:
              exec:
                command:
                  - /opt/kafka/kafka_liveness.sh
              initialDelaySeconds: 15
              timeoutSeconds: 5
            name: kafka
            ports:
              - containerPort: 9090
                name: tcp-ctrlplane
                protocol: TCP
              - containerPort: 9091
                name: tcp-replication
                protocol: TCP
              - containerPort: 9093
                name: tcp-clientstls
                protocol: TCP
              - containerPort: 9094
                name: tcp-external
                protocol: TCP
            readinessProbe:
              exec:
                command:
                  - test
                  - '-f'
                  - /var/opt/kafka/kafka-ready
              initialDelaySeconds: 180
              timeoutSeconds: 5
            resources:
              limits:
                cpu: '1'
                memory: 2Gi
              requests:
                cpu: '1'
                memory: 1Gi
            volumeMounts:
              - mountPath: /var/lib/kafka/data-0
                name: data-0
              - mountPath: /tmp
                name: strimzi-tmp
              - mountPath: /opt/kafka/cluster-ca-certs
                name: cluster-ca
              - mountPath: /opt/kafka/broker-certs
                name: broker-certs
              - mountPath: /opt/kafka/client-ca-certs
                name: client-ca-cert
              - mountPath: /opt/kafka/custom-config/
                name: kafka-metrics-and-logging
              - mountPath: /var/opt/kafka
                name: ready-files
              - mountPath: /opt/kafka/certificates/custom-external-9094-certs
                name: custom-external-9094-certs
        hostname: kafka-cluster-kafka-1
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext:
          fsGroup: 1001
          runAsUser: 1001
        serviceAccountName: kafka-cluster-kafka
        subdomain: kafka-cluster-kafka-brokers
        terminationGracePeriodSeconds: 300
        volumes:
          - name: data-0
            persistentVolumeClaim:
              claimName: data-0-kafka-cluster-kafka-1
          - emptyDir:
              medium: Memory
              sizeLimit: 5Mi
            name: strimzi-tmp
          - name: cluster-ca
            secret:
              defaultMode: 292
              secretName: kafka-cluster-cluster-ca-cert
          - name: broker-certs
            secret:
              defaultMode: 292
              secretName: kafka-cluster-kafka-brokers
          - name: client-ca-cert
            secret:
              defaultMode: 292
              secretName: kafka-cluster-clients-ca-cert
          - configMap:
              name: kafka-cluster-kafka-1
            name: kafka-metrics-and-logging
          - emptyDir:
              medium: Memory
              sizeLimit: 1Ki
            name: ready-files
          - name: custom-external-9094-certs
            secret:
              defaultMode: 292
              items:
                - key: tls.key
                  path: tls.key
                - key: tls.crt
                  path: tls.crt
              secretName: kafka-cluster-secret
    - apiVersion: v1
      kind: Pod
      metadata:
        annotations:
          strimzi.io/broker-configuration-hash: ee9fb2fe
          strimzi.io/clients-ca-cert-generation: '0'
          strimzi.io/cluster-ca-cert-generation: '0'
          strimzi.io/custom-listener-cert-thumbprints: '{external=fb3a4698}'
          strimzi.io/inter-broker-protocol-version: '3.2'
          strimzi.io/kafka-version: 3.2.3
          strimzi.io/log-message-format-version: '3.2'
          strimzi.io/logging-appenders-hash: e893ac9f
          strimzi.io/revision: '63382164'
        labels:
          app.kubernetes.io/instance: kafka-cluster
          app.kubernetes.io/managed-by: strimzi-cluster-operator
          app.kubernetes.io/name: kafka
          app.kubernetes.io/part-of: strimzi-kafka-cluster
          owned-by: nids
          statefulset.kubernetes.io/pod-name: kafka-cluster-kafka-2
          strimzi.io/cluster: kafka-cluster
          strimzi.io/controller: strimzipodset
          strimzi.io/controller-name: kafka-cluster-kafka
          strimzi.io/kind: Kafka
          strimzi.io/name: kafka-cluster-kafka
          strimzi.io/pod-name: kafka-cluster-kafka-2
        name: kafka-cluster-kafka-2
        namespace: kafka-cluster
      spec:
        affinity: {}
        containers:
          - command:
              - /opt/kafka/kafka_run.sh
            env:
              - name: KAFKA_METRICS_ENABLED
                value: 'false'
              - name: STRIMZI_KAFKA_GC_LOG_ENABLED
                value: 'false'
              - name: KAFKA_HEAP_OPTS
                value: '-Xms1024m -Xmx1024m'
            image: quay.io/strimzi/kafka:0.31.1-kafka-3.2.3
            imagePullPolicy: IfNotPresent
            livenessProbe:
              exec:
                command:
                  - /opt/kafka/kafka_liveness.sh
              initialDelaySeconds: 15
              timeoutSeconds: 5
            name: kafka
            ports:
              - containerPort: 9090
                name: tcp-ctrlplane
                protocol: TCP
              - containerPort: 9091
                name: tcp-replication
                protocol: TCP
              - containerPort: 9093
                name: tcp-clientstls
                protocol: TCP
              - containerPort: 9094
                name: tcp-external
                protocol: TCP
            readinessProbe:
              exec:
                command:
                  - test
                  - '-f'
                  - /var/opt/kafka/kafka-ready
              initialDelaySeconds: 180
              timeoutSeconds: 5
            resources:
              limits:
                cpu: '1'
                memory: 2Gi
              requests:
                cpu: '1'
                memory: 1Gi
            volumeMounts:
              - mountPath: /var/lib/kafka/data-0
                name: data-0
              - mountPath: /tmp
                name: strimzi-tmp
              - mountPath: /opt/kafka/cluster-ca-certs
                name: cluster-ca
              - mountPath: /opt/kafka/broker-certs
                name: broker-certs
              - mountPath: /opt/kafka/client-ca-certs
                name: client-ca-cert
              - mountPath: /opt/kafka/custom-config/
                name: kafka-metrics-and-logging
              - mountPath: /var/opt/kafka
                name: ready-files
              - mountPath: /opt/kafka/certificates/custom-external-9094-certs
                name: custom-external-9094-certs
        hostname: kafka-cluster-kafka-2
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext:
          fsGroup: 1001
          runAsUser: 1001
        serviceAccountName: kafka-cluster-kafka
        subdomain: kafka-cluster-kafka-brokers
        terminationGracePeriodSeconds: 300
        volumes:
          - name: data-0
            persistentVolumeClaim:
              claimName: data-0-kafka-cluster-kafka-2
          - emptyDir:
              medium: Memory
              sizeLimit: 5Mi
            name: strimzi-tmp
          - name: cluster-ca
            secret:
              defaultMode: 292
              secretName: kafka-cluster-cluster-ca-cert
          - name: broker-certs
            secret:
              defaultMode: 292
              secretName: kafka-cluster-kafka-brokers
          - name: client-ca-cert
            secret:
              defaultMode: 292
              secretName: kafka-cluster-clients-ca-cert
          - configMap:
              name: kafka-cluster-kafka-2
            name: kafka-metrics-and-logging
          - emptyDir:
              medium: Memory
              sizeLimit: 1Ki
            name: ready-files
          - name: custom-external-9094-certs
            secret:
              defaultMode: 292
              items:
                - key: tls.key
                  path: tls.key
                - key: tls.crt
                  path: tls.crt
              secretName: kafka-cluster-secret
  selector:
    matchLabels:
      strimzi.io/cluster: kafka-cluster
      strimzi.io/kind: Kafka
      strimzi.io/name: kafka-cluster-kafka

[scholzj]

Well, the cluster seems to work fine based on the statuses of all the resources. I do not know how would it relate to the upgrade, but my best guess is that the Ingress is not working. You should double check that it works for exaple using openssl s_client or curl and that you actually reach the brokers. You should also check the Ingress controller logs to doublecheck it is fine with the ingress resources and it is managing them. There were lot of changes to Ingress handling mostly because Since Strimzi 0.20, Kubernetes updated the APIs. So maybe it is missing some ingress class or some option like that?

As I said, I do not know librdkafka -> the error message from the regular Kafka clients would be easier. But my guess is that where ever it connects, it is not the Kafka brokers because it does not understand the other side. Which would point to the Ingress.

You could also try to run something from inside your Kubernetes cluster against the internal 9093 listener which would connect directly to the brokers and not through the Ingress controller. That might help to verify where the issue is.

[kaviankarimzadeh]

Hi
Thank you so much for your help.
Yes, the issue was related to ingress as there were no default ingressclass configured.
Anyway adding below annotation to Kafka external bootstrap could not help as it was going to overwrite by Kafka cluster controller.
ingressclass.kubernetes.io/is-default-class: "true"
so I've tried to add that annotation to IngressClass as a default one for all Ingresses but same issue as nginx Ingress seems too old.
But after upgrading nginx Ingress with below config in Helm, the issue is resolved. :)

  watchIngressWithoutClass: true
  ingressClassResource:
    name: nginx
    enabled: true
    default: true
    controllerValue: "k8s.io/ingress-nginx"

kafkacat -F connect.config -L -b kafka.example.com:443
% Reading configuration from file connect.config
Metadata for all topics (from broker -1: ssl://kafka.example.com:443/bootstrap):
 3 brokers:
  broker 0 at kafka-0.kafka.example.com:443
  broker 2 at kafka-2.kafka.example.com:443
  broker 1 at kafka-1.kafka.example.com:443 (controller)
 4 topics:
  topic "__strimzi_store_topic" with 1 partitions:
    partition 0, leader 0, replicas: 1,0,2, isrs: 0,1,2
  topic "mytesttopic" with 5 partitions:
    partition 0, leader 0, replicas: 0,1,2, isrs: 0,1,2
    partition 1, leader 2, replicas: 2,0,1, isrs: 0,1,2
    partition 2, leader 1, replicas: 1,2,0, isrs: 0,1,2
    partition 3, leader 0, replicas: 0,2,1, isrs: 0,1,2
    partition 4, leader 2, replicas: 2,1,0, isrs: 0,1,2
  topic "__strimzi-topic-operator-kstreams-topic-store-changelog" with 1 partitions:
    partition 0, leader 1, replicas: 1, isrs: 1
  topic "__consumer_offsets" with 50 partitions:
    partition 0, leader 0, replicas: 0, isrs: 0
    partition 1, leader 1, replicas: 1, isrs: 1
    partition 2, leader 2, replicas: 2, isrs: 2
    partition 3, leader 0, replicas: 0, isrs: 0
    partition 4, leader 1, replicas: 1, isrs: 1
    partition 5, leader 2, replicas: 2, isrs: 2
    partition 6, leader 0, replicas: 0, isrs: 0
    partition 7, leader 1, replicas: 1, isrs: 1
    partition 8, leader 2, replicas: 2, isrs: 2
    partition 9, leader 0, replicas: 0, isrs: 0
    partition 10, leader 1, replicas: 1, isrs: 1
    partition 11, leader 2, replicas: 2, isrs: 2
    partition 12, leader 0, replicas: 0, isrs: 0
    partition 13, leader 1, replicas: 1, isrs: 1
    partition 14, leader 2, replicas: 2, isrs: 2
    partition 15, leader 0, replicas: 0, isrs: 0
    partition 16, leader 1, replicas: 1, isrs: 1
    partition 17, leader 2, replicas: 2, isrs: 2
    partition 18, leader 0, replicas: 0, isrs: 0
    partition 19, leader 1, replicas: 1, isrs: 1
    partition 20, leader 2, replicas: 2, isrs: 2
    partition 21, leader 0, replicas: 0, isrs: 0
    partition 22, leader 1, replicas: 1, isrs: 1
    partition 23, leader 2, replicas: 2, isrs: 2
    partition 24, leader 0, replicas: 0, isrs: 0
    partition 25, leader 1, replicas: 1, isrs: 1
    partition 26, leader 2, replicas: 2, isrs: 2
    partition 27, leader 0, replicas: 0, isrs: 0
    partition 28, leader 1, replicas: 1, isrs: 1
    partition 29, leader 2, replicas: 2, isrs: 2
    partition 30, leader 0, replicas: 0, isrs: 0
    partition 31, leader 1, replicas: 1, isrs: 1
    partition 32, leader 2, replicas: 2, isrs: 2
    partition 33, leader 0, replicas: 0, isrs: 0
    partition 34, leader 1, replicas: 1, isrs: 1
    partition 35, leader 2, replicas: 2, isrs: 2
    partition 36, leader 0, replicas: 0, isrs: 0
    partition 37, leader 1, replicas: 1, isrs: 1
    partition 38, leader 2, replicas: 2, isrs: 2
    partition 39, leader 0, replicas: 0, isrs: 0
    partition 40, leader 1, replicas: 1, isrs: 1
    partition 41, leader 2, replicas: 2, isrs: 2
    partition 42, leader 0, replicas: 0, isrs: 0
    partition 43, leader 1, replicas: 1, isrs: 1
    partition 44, leader 2, replicas: 2, isrs: 2
    partition 45, leader 0, replicas: 0, isrs: 0
    partition 46, leader 1, replicas: 1, isrs: 1
    partition 47, leader 2, replicas: 2, isrs: 2
    partition 48, leader 0, replicas: 0, isrs: 0
    partition 49, leader 1, replicas: 1, isrs: 1

[scholzj]

Well, I guess what matters most is if it now works for you. TBH, I also struggled a bit with the Ingress classes as the different Nginx Ingress Controller versions behave a bit differently. Some general notes to what you mentioned:

• You should be able to configure the Ingress class in the Kafka CR in the listener configuration: https://strimzi.io/docs/operators/latest/full/configuring.html#type-GenericKafkaListenerConfiguration-reference
• If you want to configure annotation or label for the Ingress resource, you can do it in the template or in the listener configuration:
  • https://strimzi.io/docs/operators/latest/full/configuring.html#type-KafkaClusterTemplate-reference
  • https://strimzi.io/docs/operators/latest/full/configuring.html#type-GenericKafkaListenerConfiguration-reference

[kaviankarimzadeh]

Well, it's nice that Ingress class can be configured in the listener configuration too.
Anyway, since in our cluster we only have one ingress controller as nginx, setting annotation on IngressClass object is enough.

[amitmiglani91]

Do we Have any Details Doc or Video Step by stem Strimzi Upgrade from 0.20 to latest one. all this i m still confused where to start what to write. can yuo please help me with that.