Roll Back of Kafka not working as expected.

[Sai-Charan-Madhvaraj]

Hi Team, we are testing the upgrade and roll back of strimzi cluster operator and Kafka versions. Here are the steps performed.

1. Running a Strimzi 0.27.1 with Kafka version 3.0 with inter.broker.protocol.version and log.message.format.version both set to 3.0.

2. Upgraded the Strimzi cluster operator to 0.35.1 - Successfully completed - New Pod came up and started throwing errors (which is expected as Kafka 3.0.0 not supported in strimzi 0.35.1

2023-07-19 15:57:36 ERROR AbstractOperator:260 - Reconciliation #1(watch) Kafka(emp-kafka/cdt-west-1-shared-emp-kafka): createOrUpdate failed
io.strimzi.operator.cluster.model.KafkaVersion$UnsupportedKafkaVersionException: Unsupported Kafka.spec.kafka.version: 3.0.0. Supported versions are: [3.3.1, 3.3.2, 3.4.0]

3. Next upgraded Kafka version to 3.4.0 by keeping the inter.broker.protocol.version and log.message.format.version both set to 3.0- Successfully completed. - Roll out restart happened and post upgrade seeing below errors (which are expected)

  Conditions:
    Last Transition Time:  2023-07-19T16:01:57.915347238Z
    Message:               log.message.format.version does not match the Kafka cluster version, which suggests that an upgrade is incomplete.
    Reason:                KafkaLogMessageFormatVersion
    Status:                True
    Type:                  Warning
    Last Transition Time:  2023-07-19T16:01:57.915414739Z
    Message:               inter.broker.protocol.version does not match the Kafka cluster version, which suggests that an upgrade is incomplete.
    Reason:                KafkaInterBrokerProtocolVersion
    Status:                True
    Type:                  Warning
    Last Transition Time:  2023-07-19T16:07:20.986242231Z
    Status:                True
    Type:                  Ready
    Last Transition Time:  2023-07-19T15:59:03.675958945Z
    Message:               In API version kafka.strimzi.io/v1beta2 the tlsSidecar property at path spec.cruiseControl.tlsSidecar has been deprecated.
    Reason:                DeprecatedFields
    Status:                True
    Type:                  Warning

4. Now started to test roll back from 0.35.1 to 0.27.1,
   Note: Have not changed inter.broker.protocol.version and log.message.format.version both set to 3.0.

5. Downgraded the Cluster operator from 0.35.1 to 0.27.1 - Successfully Completed - the CLuster operator came up with 0.27.1 and started throwing errors (which is expected)

2023-07-19 16:34:34 ERROR AbstractOperator:247 - Reconciliation #2(watch) Kafka(emp-kafka/cdt-west-1-shared-emp-kafka): createOrUpdate failed
io.strimzi.operator.cluster.model.InvalidResourceException: Version 3.4.0 is not supported. Supported versions are: 2.8.0, 2.8.1, 3.0.0.

6. Changed the Kafka Version to 3.0.0 with which cluster was running initially before upgrade. - Not working as expected. Cluster operator not able to do rolling restart pods and get stuck and continues to throw below error. Ideally it should pick new version (3.0.0) and start rolling restart.

Note: Kubectl describe kafka command shows version as 3.0.0

2023-07-19 16:38:34 ERROR AbstractOperator:247 - Reconciliation #20(timer) Kafka(emp-kafka/cdt-west-1-shared-emp-kafka): createOrUpdate failed
io.strimzi.operator.cluster.KafkaUpgradeException: Unsupported Kafka.spec.kafka.version: 3.4.0. Supported versions are: [2.1.0, 2.6.2, 2.7.1, 2.8.0, 2.8.1, 2.5.1, 2.6.0, 2.6.1, 2.7.0, 2.3.1, 2.4.0, 2.4.1, 2.5.0, 2.1.1, 2.2.0, 2.2.1, 2.3.0, 3.0.0]
	at io.strimzi.operator.cluster.model.KafkaVersion$Lookup.version(KafkaVersion.java:128) ~[io.strimzi.cluster-operator-0.27.1.jar:0.27.1]
	at io.strimzi.operator.cluster.operator.assembly.KafkaAssemblyOperator$ReconciliationState.lambda$getKafkaClusterDescription$63(KafkaAssemblyOperator.java:1735) ~[io.strimzi.cluster-operator-0.27.1.jar:0.27.1]
	at io.vertx.core.impl.future.Composition.onSuccess(Composition.java:38) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at io.vertx.core.impl.future.Composition$1.onSuccess(Composition.java:62) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at io.vertx.core.impl.future.PromiseImpl.tryComplete(PromiseImpl.java:23) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at io.vertx.core.impl.future.PromiseImpl.onSuccess(PromiseImpl.java:49) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:54) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) [io.netty.netty-transport-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at java.lang.Thread.run(Thread.java:829) [?:?]

Could you please advise am i missing something here ?

[scholzj]

How does your Kafka custom resource look like?

[Sai-Charan-Madhvaraj]

It throws same error in Kafka custom resource.

Here is the Yaml of Kafka custom resource.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: cdt-west-1-shared-emp-kafka
  namespace: emp-kafka
spec:
  kafka:
    version: 3.0.0
    replicas: 3
    resources: 
      requests: 
        memory: 2Gi
        cpu: 1
      limits:
        memory: 4Gi
        cpu: 2
    listeners:
      - name: external
        port: 9094
        type: ingress
        tls: true
        authentication:
          type: scram-sha-512
        configuration:
          bootstrap:
            host: cdt-west-1-shared-emp-kafka-bootstrap.org-digital.net
          brokers:
          - broker: 0
            host: cdt-west-1-shared-emp-kafka-kafka-0.org-digital.net
          - broker: 1
            host: cdt-west-1-shared-emp-kafka-kafka-1.org-digital.net
          - broker: 2
            host: cdt-west-1-shared-emp-kafka-kafka-2.org-digital.net
          
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: cdt-west-1-shared-emp-kafka-listener-cert
      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: saslint
        port: 9095
        type: internal
        tls: true
        authentication:
          type: scram-sha-512
        configuration:
          useServiceDnsDomain: true    
    authorization:
      type: simple
      superUsers:
        - cdt-shared-emp-admin
    config: 
    # https://strimzi.io/blog/2021/06/08/broker-tuning/
    # https://kafka.apache.org/documentation/#brokerconfigs
      num.partitions: 3
      default.replication.factor: 3
      min.insync.replicas: 2
      auto.create.topics.enable: false
      delete.topic.enable: true
      compression.type: producer
      message.max.bytes: 1048576
      unclean.leader.election.enable: false
      auto.leader.rebalance.enable: true
      acks: -1
      # Internal topic - __consumer_offsets settings for offset commits
      offsets.commit.required.acks: -1
      offsets.topic.num.partitions: 50
      offsets.topic.replication.factor: 3
      # Internal topic - __transaction_state settings for transactions (https://strimzi.io/docs/operators/latest/using.html#reliability_guarantees)
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      # It allows us to run brokers of different versions in one cluster, to avoid a downtime of the cluster during the upgrade. An update to the inter.broker.protocol.version is required when upgrading the Kafka version
      inter.broker.protocol.version: "3.0"
      # Message format version the broker will use to append messages to the logs. From Kafka 3.0.0, when the inter.broker.protocol.version is set to 3.0 or higher, the log.message.format.version option is ignored and doesn’t need to be set
      log.message.format.version: "3.0"
      # Log retention policies
      log.retention.check.interval.ms: 300000 # 5 min
      log.retention.bytes: -1
      log.retention.ms: 259200000 # 3 days
      log.roll.ms: 604800000 # 7 days
      log.segment.bytes: 1073741824 # 1 Gibibyte
      log.segment.delete.delay.ms: 60000  
      # Thread settings
      num.network.threads: 3 # Handles requests to the Kafka cluster, such as produce and fetch requests from client applications
      num.io.threads: 8 # The number of I/O threads to pick up the requests from request queue and process them
      num.recovery.threads.per.data.dir: 1 # The number of threads used for log loading at startup and flushing at shutdown
      queued.max.requests: 500 # To limit the number of requests allowed in the request queue
      # The default buffer settings for sending and receiving messages
      socket.send.buffer.bytes: 102400 # 100 Kibibytes
      socket.receive.buffer.bytes: 102400 # 100 Kibibytes
      socket.request.max.bytes: 104857600 # 100 Mebibytes
      # Size of the buffer for network requests
      replica.socket.receive.buffer.bytes: 65536 # 64 kibibytes
      # Log cleaner settings
      log.cleanup.policy: delete
      log.cleaner.backoff.ms: 15000
      log.cleaner.delete.retention.ms: 86400000 # 1 day
      log.cleaner.dedupe.buffer.size: 134217728
      log.cleaner.io.buffer.load.factor: 0.9
      log.cleaner.threads: 8
      log.cleaner.io.max.bytes.per.second: 1.7976931348623157E308
      # Settings for log flush from in-memory to disk
      log.flush.scheduler.interval.ms: 2000
      log.flush.interval.ms: 50000
      log.flush.interval.messages: 100000
      # The lag time before a follower is considered out of sync
      replica.lag.time.max.ms: 30000
      # A maximum size for the messages fetched by each follower that replicates the data from a leader partition.
      replica.fetch.max.bytes: 1048576 # 1 Mebibyte
      # Leader imbalance check      
      leader.imbalance.check.interval.seconds: 300
      leader.imbalance.per.broker.percentage: 10
      # A delay for the group coordinator to wait for members to join before the initial rebalance
      group.initial.rebalance.delay.ms: 3000
      # The max time to establish connection to zookeeper
      zookeeper.connection.timeout.ms: 18000      
    storage:
      type: jbod
      volumes: 
      - id: 1
        type: persistent-claim
        size: 50Gi
        deleteClaim: false
        class: managed-csi-zrs-retain
      - id: 2
        type: persistent-claim
        size: 50Gi
        deleteClaim: false
        class: managed-csi-zrs-retain
      
    rack:
      topologyKey: topology.kubernetes.io/zone
    template:
      pod:
        metadata:
          annotations:
            prometheus.io/scrape: "true"
            prometheus.io/port: "9404"
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: team
                  operator: In
                  values:
                  - emp
        tolerations:
        - key: "kubernetes.azure.com/scalesetpriority"
          operator: "Equal"
          value: "spot"
          effect: "NoSchedule"
        topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/name: cdt-west-1-shared-emp-kafka-kafka
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/name: cdt-west-1-shared-emp-kafka-kafka
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: emp-kafka-metrics
          key: emp-kafka-metrics-config.yml
  zookeeper:
    replicas: 3
    resources:
      requests:   
        memory: 2Gi
        cpu: 1
      limits:
        memory: 4Gi
        cpu: 2
    storage:
      type: persistent-claim
      size: 50Gi
      deleteClaim: false
      class: managed-csi-zrs-retain
    template:
      pod:
        metadata:
          annotations:
            prometheus.io/scrape: "true"
            prometheus.io/port: "9404"
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: team
                  operator: In
                  values:
                  - emp
        tolerations:
        - key: "kubernetes.azure.com/scalesetpriority"
          operator: "Equal"
          value: "spot"
          effect: "NoSchedule"
        topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/name: cdt-west-1-shared-emp-kafka-zookeeper
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/name: cdt-west-1-shared-emp-kafka-zookeeper
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: emp-kafka-metrics
          key: emp-zookeeper-metrics-config.yml
  entityOperator:
    template:
      pod:
        metadata:
          annotations:
            prometheus.io/scrape: "true"
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: team
                  operator: In
                  values:
                  - emp
        tolerations:
        - key: "kubernetes.azure.com/scalesetpriority"
          operator: "Equal"
          value: "spot"
          effect: "NoSchedule"
        topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/name: cdt-west-1-shared-emp-kafka-entity-operator
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/name: cdt-west-1-shared-emp-kafka-entity-operator
    tlsSidecar:
      resources: 
        requests:
          cpu: 100m
          memory: 100Mi
        limits:
          cpu: 500m
          memory: 500Mi
    topicOperator:
      resources:
        requests: 
          memory: 1Gi
          cpu: 500m
        limits:
          memory: 2Gi
          cpu: 1
    userOperator:
      resources:
        requests: 
          memory: 1Gi
          cpu: 500m
        limits:
          memory: 2Gi
          cpu: 1
  cruiseControl:
    template:
      pod:
        metadata:
          annotations:
            prometheus.io/scrape: "true"
            prometheus.io/port: "9404"
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: team
                  operator: In
                  values:
                  - emp
        tolerations:
        - key: "kubernetes.azure.com/scalesetpriority"
          operator: "Equal"
          value: "spot"
          effect: "NoSchedule"
    tlsSidecar:
      resources: 
        requests:
          memory: 500Mi
          cpu: 500m
        limits:
          memory: 1Gi
          cpu: 1
    resources: 
      requests:
        memory: 1Gi
        cpu: 500m
      limits:
        memory: 2Gi
        cpu: 1
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: emp-cruise-control-metrics
          key: emp-metrics-config.yml
  kafkaExporter:
    template:
      pod:
        metadata:
          annotations:
            prometheus.io/scrape: "true"
            prometheus.io/port: "9404"
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: team
                  operator: In
                  values:
                  - emp
    resources: 
      requests:
        memory: 1Gi
        cpu: 500m
      limits:
        memory: 2Gi
        cpu: 1
    topicRegex: ".*"
    groupRegex: ".*"
  clusterCa:
    renewalDays: 30
    validityDays: 365
    generateCertificateAuthority: true
    generateSecretOwnerReference: false
  clientsCa:
    renewalDays: 30
    validityDays: 365
    generateCertificateAuthority: true
    generateSecretOwnerReference: false

[scholzj]

Hmm, I'm afraid that looks like a bug. I opened #8869 to track it. The only workaround I guess is to delete the Kafka StatefulSet / StrimziPodSet and the Pods and let the operator recreate them.

[scholzj]

Oh, I guess another workaround would be to downgrade step by step ... i.e.:

• First keep Strimzi 0.35 and downgrade Kafka to 3.3.1
• Then downgrade operator to Strimzi 0.32
• Then downgrade Kafka to 3.2.0
• Then downgrade operator to 0.29
• Then downgrade Kafka to 3.0.0
• Then downgrade operator to 0.27

[Sai-Charan-Madhvaraj]

I will try this and let you know how it goes