Adding a disk for type JBOD for Kafka cluster isn't working.

[AAR256]

Configuration -

Strimzi version - 0.27.1
K8s version - 1.22.6
Infrastructure - AKS
Storage Class - Standard Zone redundant storage (ZRS)

Kafka Cluster YAML -

---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: west-2-shared-kafka
  namespace: kafka
spec:
  kafka:
    version: 3.0.0
    replicas: 5
    resources:
      requests:
        memory: 8Gi
        cpu: 2
      limits:
        memory: 16Gi
        cpu: 4
    listeners:
      - name: external
        port: 9094
        type: ingress
        tls: true
        authentication:
          type: scram-sha-512
        configuration:
          bootstrap:
            host: west-2-shared-kafka-bootstrap.net
          brokers:
          - broker: 0
            host: west-2-shared-kafka-kafka-0.net
          - broker: 1
            host: west-2-shared-kafka-kafka-1.net
          - broker: 2
            host: west-2-shared-kafka-kafka-2.net
          - broker: 3
            host: west-2-shared-kafka-kafka-3.net
          - broker: 4
            host: west-2-shared-kafka-kafka-4.net
          
      - name: tls
        port: 9093
        type: internal
        tls: true
    authorization:
      type: simple
      superUsers:
        - admin
    config: 
    # https://strimzi.io/blog/2021/06/08/broker-tuning/
    # https://kafka.apache.org/documentation/#brokerconfigs
      num.partitions: 3
      default.replication.factor: 3
      min.insync.replicas: 2
      auto.create.topics.enable: false
      delete.topic.enable: true
      compression.type: producer
      message.max.bytes: 1048576
      unclean.leader.election.enable: false
      auto.leader.rebalance.enable: true
      acks: -1
      # Internal topic - __consumer_offsets settings for offset commits
      offsets.commit.required.acks: -1
      offsets.topic.num.partitions: 50
      offsets.topic.replication.factor: 3
      # Internal topic - __transaction_state settings for transactions (https://strimzi.io/docs/operators/latest/using.html#reliability_guarantees)
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      # It allows us to run brokers of different versions in one cluster, to avoid a downtime of the cluster during the upgrade. An update to the inter.broker.protocol.version is required when upgrading the Kafka version
      inter.broker.protocol.version: "3.0"
      # Message format version the broker will use to append messages to the logs. From Kafka 3.0.0, when the inter.broker.protocol.version is set to 3.0 or higher, the log.message.format.version option is ignored and doesn’t need to be set
      log.message.format.version: "3.0"
      # Log retention policies
      log.retention.check.interval.ms: 300000 # 5 min
      log.retention.bytes: -1
      log.retention.ms: 259200000 # 3 days
      log.roll.ms: 604800000 # 7 days
      log.segment.bytes: 1073741824 # 1 Gibibyte
      log.segment.delete.delay.ms: 60000  
      # Thread settings
      num.network.threads: 3 # Handles requests to the Kafka cluster, such as produce and fetch requests from client applications
      num.io.threads: 8 # The number of I/O threads to pick up the requests from request queue and process them
      num.recovery.threads.per.data.dir: 1 # The number of threads used for log loading at startup and flushing at shutdown
      queued.max.requests: 500 # To limit the number of requests allowed in the request queue
      # The default buffer settings for sending and receiving messages
      socket.send.buffer.bytes: 102400 # 100 Kibibytes
      socket.receive.buffer.bytes: 102400 # 100 Kibibytes
      socket.request.max.bytes: 104857600 # 100 Mebibytes
      # Size of the buffer for network requests
      replica.socket.receive.buffer.bytes: 65536 # 64 kibibytes
      # Log cleaner settings
      log.cleanup.policy: delete
      log.cleaner.backoff.ms: 15000
      log.cleaner.delete.retention.ms: 86400000 # 1 day
      log.cleaner.dedupe.buffer.size: 134217728
      log.cleaner.io.buffer.load.factor: 0.9
      log.cleaner.threads: 8
      log.cleaner.io.max.bytes.per.second: 1.7976931348623157E308
      # Settings for log flush from in-memory to disk
      log.flush.scheduler.interval.ms: 2000
      log.flush.interval.ms: 50000
      log.flush.interval.messages: 100000
      # The lag time before a follower is considered out of sync
      replica.lag.time.max.ms: 30000
      # A maximum size for the messages fetched by each follower that replicates the data from a leader partition.
      replica.fetch.max.bytes: 1048576 # 1 Mebibyte
      # Leader imbalance check      
      leader.imbalance.check.interval.seconds: 300
      leader.imbalance.per.broker.percentage: 10
      # A delay for the group coordinator to wait for members to join before the initial rebalance
      group.initial.rebalance.delay.ms: 3000
      # The max time to establish connection to zookeeper
      zookeeper.connection.timeout.ms: 18000      
    storage:
      type: jbod
      volumes:
      - id: 1
        type: persistent-claim
        size: 50Gi
        deleteClaim: false
        class: managed-csi-zrs 
      - id: 2
        type: persistent-claim
        size: 50Gi
        deleteClaim: false
        class: managed-csi-zrs
      - id: 3
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
        class: managed-csi-zrs
    rack:
      topologyKey: topology.kubernetes.io/zone
    template:
      pod:
        tolerations:
        - key: "kubernetes.azure.com/scalesetpriority"
          operator: "Equal"
          value: "spot"
          effect: "NoSchedule"
        topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/name: west-2-shared-kafka-kafka
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/name: west-2-shared-kafka-kafka
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
  zookeeper:
    replicas: 3
    resources:
      requests:
        memory: 2Gi
        cpu: 1
      limits:
        memory: 4Gi
        cpu: 2
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
      class: managed-csi-zrs
    template:
      pod:
        tolerations:
        - key: "kubernetes.azure.com/scalesetpriority"
          operator: "Equal"
          value: "spot"
          effect: "NoSchedule"
        topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/name: west-2-shared-kafka-zookeeper
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/name: west-2-shared-kafka-zookeeper
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: zookeeper-metrics-config.yml
  entityOperator:
    template:
      pod:
        tolerations:
        - key: "kubernetes.azure.com/scalesetpriority"
          operator: "Equal"
          value: "spot"
          effect: "NoSchedule"
        topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/name: west-2-shared-kafka-entity-operator
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/name: west-2-shared-kafka-entity-operator
    tlsSidecar:
      resources:
        requests:
          cpu: 100m
          memory: 100Mi
        limits:
          cpu: 500m
          memory: 500Mi
    topicOperator:
      resources:
        requests:
          memory: 500Mi
          cpu: 500m
        limits:
          memory: 1Gi
          cpu: 1
    userOperator:
      resources:
        requests:
          memory: 500Mi
          cpu: 500m
        limits:
          memory: 1Gi
          cpu: 1
  cruiseControl:
    template:
      pod:
        tolerations:
        - key: "kubernetes.azure.com/scalesetpriority"
          operator: "Equal"
          value: "spot"
          effect: "NoSchedule"
    tlsSidecar:
      resources:
        requests:
          memory: 500Mi
          cpu: 500m
        limits:
          memory: 1Gi
          cpu: 1
    resources:
      requests:
        memory: 500Mi
        cpu: 500m
      limits:
        memory: 1Gi
        cpu: 1
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: cruise-control-metrics
          key: metrics-config.yml
  kafkaExporter:
    resources:
      requests:
        cpu: 500m
        memory: 500Mi
      limits:
        cpu: 1
        memory: 1Gi
    topicRegex: ".*"
    groupRegex: ".*"

While running kubectl describe Kafka Cluster-name I'm getting the below error in the events -

Status:
  Cluster Id:  zaj_pqkgTpySqWKkTqQZ_Q
  Conditions:
    Last Transition Time:  2022-04-18T15:18:38.025465Z
    Message:               The desired Kafka storage configuration contains changes which are not allowed. As a result, all storage changes will be ignored. Use DEBUG level logging for more information about the detected changes.
    Reason:                KafkaStorage
    Status:                True
    Type:                  Warning
    Last Transition Time:  2022-04-18T15:18:41.301Z
    Status:                True
    Type:                  Ready

Also, when adding a new disk from the operator logs, we could see that it's not reconciling.

Anything we can do here to either add a new disk successfully or increase the size of existing disk without restarting the pods?

[scholzj]

I think you need to share more information. These things would be useful:

• Full logs form the operator
• The original Kafka CR before you added the volume
• The YAML of the Kafka StatefulSet

[Sai-Charan-Madhvaraj]

Stateful set YAML :

kind: StatefulSet
metadata:
  annotations:
    strimzi.io/kafka-version: 3.0.0
    strimzi.io/storage: '{"volumes":[{"type":"persistent-claim","size":"100Gi","class":"managed-csi-zrs","id":1},{"type":"persistent-claim","size":"100Gi","class":"managed-csi-zrs","id":2}],"type":"jbod"}'
  creationTimestamp: "2022-04-18T13:52:29Z"
  generation: 5
  labels:
    app.kubernetes.io/instance: pp-west-2-shared-kafka
    app.kubernetes.io/managed-by: strimzi-cluster-operator
    app.kubernetes.io/name: kafka
    app.kubernetes.io/part-of: strimzi-pp-west-2-shared-kafka
    strimzi.io/cluster: pp-west-2-shared-kafka
    strimzi.io/kind: Kafka
    strimzi.io/name: pp-west-2-shared-kafka-kafka
  name: pp-west-2-shared-kafka-kafka
  namespace: kafka
  ownerReferences:
  - apiVersion: kafka.strimzi.io/v1beta2
    blockOwnerDeletion: false
    controller: false
    kind: Kafka
    name: pp-west-2-shared-kafka
    uid: cb81816f-71d6-43dd-853a-24d5be1f2bcf
  resourceVersion: "15693361"
  uid: 9df1063f-f229-416d-85dd-2ff5bcabb5b7
spec:
  podManagementPolicy: Parallel
  replicas: 5
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      strimzi.io/cluster: pp-west-2-shared-kafka
      strimzi.io/kind: Kafka
      strimzi.io/name: pp-west-2-shared-kafka-kafka
  serviceName: pp-west-2-shared-kafka-kafka-brokers
  template:
    metadata:
      annotations:
        strimzi.io/broker-configuration-hash: 173fb5bda1f2ed0278400631dad73687f0c4ea3655bb257c68612b48ba07c7f83aa2cd89ceb51512
        strimzi.io/clients-ca-cert-generation: "0"
        strimzi.io/cluster-ca-cert-generation: "0"
        strimzi.io/generation: "3"
        strimzi.io/inter-broker-protocol-version: "3.0"
        strimzi.io/kafka-version: 3.0.0
        strimzi.io/log-message-format-version: "3.0"
        strimzi.io/logging-appenders-hash: e893ac9f32f7958042d1c82477e3484b0f8ebad5
        strimzi.io/storage: '{"volumes":[{"type":"persistent-claim","size":"100Gi","class":"managed-csi-zrs","id":1},{"type":"persistent-claim","size":"100Gi","class":"managed-csi-zrs","id":2}],"type":"jbod"}'
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: pp-west-2-shared-kafka
        app.kubernetes.io/managed-by: strimzi-cluster-operator
        app.kubernetes.io/name: kafka
        app.kubernetes.io/part-of: strimzi-pp-west-2-shared-kafka
        strimzi.io/cluster: pp-west-2-shared-kafka
        strimzi.io/kind: Kafka
        strimzi.io/name: pp-west-2-shared-kafka-kafka
      name: pp-west-2-shared-kafka-kafka
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: Exists
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  strimzi.io/cluster: pp-west-2-shared-kafka
                  strimzi.io/name: pp-west-2-shared-kafka-kafka
              topologyKey: topology.kubernetes.io/zone
            weight: 100
      containers:
      - command:
        - /opt/kafka/kafka_run.sh
        env:
        - name: KAFKA_METRICS_ENABLED
          value: "true"
        - name: STRIMZI_KAFKA_GC_LOG_ENABLED
          value: "false"
        - name: DYNAMIC_HEAP_FRACTION
          value: "0.5"
        - name: DYNAMIC_HEAP_MAX
          value: "5368709120"
        image: acrempeuwcdt.azurecr.io/strimzi/kafka:0.27.1-kafka-3.0.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - /opt/kafka/kafka_liveness.sh
          failureThreshold: 3
          initialDelaySeconds: 15
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: kafka
        ports:
        - containerPort: 9090
          name: tcp-ctrlplane
          protocol: TCP
        - containerPort: 9091
          name: tcp-replication
          protocol: TCP
        - containerPort: 9094
          name: tcp-external
          protocol: TCP
        - containerPort: 9093
          name: tcp-clientstls
          protocol: TCP
        - containerPort: 9404
          name: tcp-prometheus
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - test
            - -f
            - /var/opt/kafka/kafka-ready
          failureThreshold: 3
          initialDelaySeconds: 15
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          limits:
            cpu: "4"
            memory: 16Gi
          requests:
            cpu: "2"
            memory: 8Gi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/kafka/data-1
          name: data-1
        - mountPath: /var/lib/kafka/data-2
          name: data-2
        - mountPath: /tmp
          name: strimzi-tmp
        - mountPath: /opt/kafka/cluster-ca-certs
          name: cluster-ca
        - mountPath: /opt/kafka/broker-certs
          name: broker-certs
        - mountPath: /opt/kafka/client-ca-certs
          name: client-ca-cert
        - mountPath: /opt/kafka/custom-config/
          name: kafka-metrics-and-logging
        - mountPath: /var/opt/kafka
          name: ready-files
        - mountPath: /opt/kafka/init
          name: rack-volume
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: azure-secret
      initContainers:
      - args:
        - /opt/strimzi/bin/kafka_init_run.sh
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: RACK_TOPOLOGY_KEY
          value: topology.kubernetes.io/zone
        image: acrempeuwcdt.azurecr.io/strimzi/operator:0.27.1
        imagePullPolicy: IfNotPresent
        name: kafka-init
        resources:
          limits:
            cpu: "4"
            memory: 16Gi
          requests:
            cpu: "2"
            memory: 8Gi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /opt/kafka/init
          name: rack-volume
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 0
      serviceAccount: pp-west-2-shared-kafka-kafka
      serviceAccountName: pp-west-2-shared-kafka-kafka
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoSchedule
        key: kubernetes.azure.com/scalesetpriority
        operator: Equal
        value: spot
      topologySpreadConstraints:
      - labelSelector:
          matchLabels:
            strimzi.io/name: pp-west-2-shared-kafka-kafka
        maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
      - labelSelector:
          matchLabels:
            strimzi.io/name: pp-west-2-shared-kafka-kafka
        maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway
      volumes:
      - emptyDir:
          medium: Memory
          sizeLimit: 1Mi
        name: rack-volume
      - emptyDir:
          medium: Memory
          sizeLimit: 1Mi
        name: strimzi-tmp
      - name: cluster-ca
        secret:
          defaultMode: 292
          secretName: pp-west-2-shared-kafka-cluster-ca-cert
      - name: broker-certs
        secret:
          defaultMode: 292
          secretName: pp-west-2-shared-kafka-kafka-brokers
      - name: client-ca-cert
        secret:
          defaultMode: 292
          secretName: pp-west-2-shared-kafka-clients-ca-cert
      - configMap:
          defaultMode: 420
          name: pp-west-2-shared-kafka-kafka-config
        name: kafka-metrics-and-logging
      - emptyDir:
          medium: Memory
          sizeLimit: 1Ki
        name: ready-files
  updateStrategy:
    type: OnDelete
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: data-1
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 100Gi
      storageClassName: managed-csi-zrs
      volumeMode: Filesystem
    status:
      phase: Pending
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: data-2
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 100Gi
      storageClassName: managed-csi-zrs
      volumeMode: Filesystem
    status:
      phase: Pending
status:
  availableReplicas: 5
  collisionCount: 0
  currentRevision: pp-west-2-shared-kafka-kafka-5c5467dd6c
  observedGeneration: 5
  readyReplicas: 5
  replicas: 5
  updateRevision: pp-west-2-shared-kafka-kafka-5bf8ccdbdf
  updatedReplicas: 5

[Sai-Charan-Madhvaraj]

Operator logs -

Auto-detected KUBERNETES_SERVICE_DNS_DOMAIN: cluster.local
+ exec /usr/bin/tini -w -e 143 -- java -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom --illegal-access=deny -classpath lib/io.strimzi.cluster-operator-0.27.1.jar:lib/io.netty.netty-common-4.1.71.Final.jar:lib/io.fabric8.kubernetes-model-flowcontrol-5.10.2.jar:lib/io.fabric8.kubernetes-model-node-5.10.2.jar:lib/io.fabric8.kubernetes-model-metrics-5.10.2.jar:lib/io.netty.netty-codec-http2-4.1.71.Final.jar:lib/io.fabric8.openshift-model-machineconfig-5.10.2.jar:lib/io.fabric8.openshift-model-machine-5.10.2.jar:lib/org.apache.kafka.kafka-clients-3.0.0.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.11.3.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/io.fabric8.openshift-model-5.10.2.jar:lib/io.fabric8.openshift-model-operatorhub-5.10.2.jar:lib/org.apache.zookeeper.zookeeper-jute-3.6.3.jar:lib/io.netty.netty-handler-4.1.71.Final.jar:lib/io.fabric8.kubernetes-model-admissionregistration-5.10.2.jar:lib/io.fabric8.kubernetes-model-apps-5.10.2.jar:lib/org.apache.logging.log4j.log4j-core-2.17.1.jar:lib/io.fabric8.kubernetes-model-extensions-5.10.2.jar:lib/org.apache.zookeeper.zookeeper-3.6.3.jar:lib/net.minidev.accessors-smart-2.4.7.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.5.jar:lib/io.fabric8.kubernetes-model-batch-5.10.2.jar:lib/io.fabric8.kubernetes-model-scheduling-5.10.2.jar:lib/io.fabric8.openshift-model-clusterautoscaling-5.10.2.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.strimzi.certificate-manager-0.27.1.jar:lib/org.xerial.snappy.snappy-java-1.1.8.1.jar:lib/io.strimzi.crd-annotations-0.27.1.jar:lib/io.fabric8.kubernetes-client-5.10.2.jar:lib/org.yaml.snakeyaml-1.26.jar:lib/io.fabric8.kubernetes-model-coordination-5.10.2.jar:lib/io.netty.netty-resolver-dns-4.1.71.Final.jar:lib/io.fabric8.kubernetes-model-discovery-5.10.2.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/io.fabric8.kubernetes-model-apiextensions-5.10.2.jar:lib/io.strimzi.kafka-oauth-client-0.8.1.jar:lib/io.fabric8.openshift-model-whereabouts-5.10.2.jar:lib/io.netty.netty-codec-socks-4.1.71.Final.jar:lib/io.netty.netty-handler-proxy-4.1.71.Final.jar:lib/io.netty.netty-resolver-4.1.71.Final.jar:lib/io.strimzi.operator-common-0.27.1.jar:lib/com.jayway.jsonpath.json-path-2.6.0.jar:lib/net.minidev.json-smart-2.4.7.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/io.strimzi.kafka-oauth-server-0.8.1.jar:lib/io.vertx.vertx-micrometer-metrics-4.2.1.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.netty.netty-codec-dns-4.1.71.Final.jar:lib/io.netty.netty-transport-native-unix-common-4.1.71.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.71.Final.jar:lib/com.github.luben.zstd-jni-1.5.0-2.jar:lib/io.strimzi.config-model-0.27.1.jar:lib/io.vertx.vertx-core-4.2.1.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.1.jar:lib/io.netty.netty-buffer-4.1.71.Final.jar:lib/io.fabric8.openshift-model-console-5.10.2.jar:lib/io.fabric8.kubernetes-model-certificates-5.10.2.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/io.fabric8.openshift-model-storageversionmigrator-5.10.2.jar:lib/io.fabric8.kubernetes-model-policy-5.10.2.jar:lib/io.strimzi.api-0.27.1.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/io.fabric8.openshift-model-miscellaneous-5.10.2.jar:lib/io.netty.netty-transport-native-epoll-4.1.71.Final-linux-x86_64.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.fabric8.kubernetes-model-core-5.10.2.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.11.3.jar:lib/io.fabric8.openshift-model-operator-5.10.2.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/io.netty.netty-codec-4.1.71.Final.jar:lib/com.nimbusds.nimbus-jose-jwt-9.10.jar:lib/io.netty.netty-codec-http-4.1.71.Final.jar:lib/com.fasterxml.jackson.core.jackson-core-2.11.3.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/io.fabric8.kubernetes-model-autoscaling-5.10.2.jar:lib/io.fabric8.openshift-model-tuned-5.10.2.jar:lib/io.fabric8.kubernetes-model-common-5.10.2.jar:lib/io.strimzi.kafka-oauth-server-plain-0.8.1.jar:lib/io.fabric8.kubernetes-model-events-5.10.2.jar:lib/io.fabric8.openshift-client-5.10.2.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.fabric8.kubernetes-model-rbac-5.10.2.jar:lib/org.apache.logging.log4j.log4j-api-2.17.1.jar:lib/io.strimzi.kafka-oauth-common-0.8.1.jar:lib/io.netty.netty-transport-classes-epoll-4.1.71.Final.jar:lib/io.fabric8.kubernetes-model-networking-5.10.2.jar:lib/io.fabric8.kubernetes-model-storageclass-5.10.2.jar:lib/io.fabric8.openshift-model-monitoring-5.10.2.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.13.0.jar:lib/io.netty.netty-tcnative-classes-2.0.46.Final.jar:lib/io.netty.netty-transport-4.1.71.Final.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar io.strimzi.operator.cluster.Main

[Sai-Charan-Madhvaraj]

OLD KAFKA CR before adding new volume

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: west-2-shared-kafka
  namespace: kafka
spec:
  kafka:
    version: 3.0.0
    replicas: 5
    resources:
      requests:
        memory: 8Gi
        cpu: 2
      limits:
        memory: 16Gi
        cpu: 4
    listeners:
      - name: external
        port: 9094
        type: ingress
        tls: true
        authentication:
          type: scram-sha-512
        configuration:
          bootstrap:
            host: west-2-shared-kafka-bootstrap.net
          brokers:
          - broker: 0
            host: west-2-shared-kafka-kafka-0.net
          - broker: 1
            host: west-2-shared-kafka-kafka-1.net
          - broker: 2
            host: west-2-shared-kafka-kafka-2.net
          - broker: 3
            host: west-2-shared-kafka-kafka-3.net
          - broker: 4
            host: west-2-shared-kafka-kafka-4.net
          
      - name: tls
        port: 9093
        type: internal
        tls: true
    authorization:
      type: simple
      superUsers:
        - admin
    config: 
    # https://strimzi.io/blog/2021/06/08/broker-tuning/
    # https://kafka.apache.org/documentation/#brokerconfigs
      num.partitions: 3
      default.replication.factor: 3
      min.insync.replicas: 2
      auto.create.topics.enable: false
      delete.topic.enable: true
      compression.type: producer
      message.max.bytes: 1048576
      unclean.leader.election.enable: false
      auto.leader.rebalance.enable: true
      acks: -1
      # Internal topic - __consumer_offsets settings for offset commits
      offsets.commit.required.acks: -1
      offsets.topic.num.partitions: 50
      offsets.topic.replication.factor: 3
      # Internal topic - __transaction_state settings for transactions (https://strimzi.io/docs/operators/latest/using.html#reliability_guarantees)
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      # It allows us to run brokers of different versions in one cluster, to avoid a downtime of the cluster during the upgrade. An update to the inter.broker.protocol.version is required when upgrading the Kafka version
      inter.broker.protocol.version: "3.0"
      # Message format version the broker will use to append messages to the logs. From Kafka 3.0.0, when the inter.broker.protocol.version is set to 3.0 or higher, the log.message.format.version option is ignored and doesn’t need to be set
      log.message.format.version: "3.0"
      # Log retention policies
      log.retention.check.interval.ms: 300000 # 5 min
      log.retention.bytes: -1
      log.retention.ms: 259200000 # 3 days
      log.roll.ms: 604800000 # 7 days
      log.segment.bytes: 1073741824 # 1 Gibibyte
      log.segment.delete.delay.ms: 60000  
      # Thread settings
      num.network.threads: 3 # Handles requests to the Kafka cluster, such as produce and fetch requests from client applications
      num.io.threads: 8 # The number of I/O threads to pick up the requests from request queue and process them
      num.recovery.threads.per.data.dir: 1 # The number of threads used for log loading at startup and flushing at shutdown
      queued.max.requests: 500 # To limit the number of requests allowed in the request queue
      # The default buffer settings for sending and receiving messages
      socket.send.buffer.bytes: 102400 # 100 Kibibytes
      socket.receive.buffer.bytes: 102400 # 100 Kibibytes
      socket.request.max.bytes: 104857600 # 100 Mebibytes
      # Size of the buffer for network requests
      replica.socket.receive.buffer.bytes: 65536 # 64 kibibytes
      # Log cleaner settings
      log.cleanup.policy: delete
      log.cleaner.backoff.ms: 15000
      log.cleaner.delete.retention.ms: 86400000 # 1 day
      log.cleaner.dedupe.buffer.size: 134217728
      log.cleaner.io.buffer.load.factor: 0.9
      log.cleaner.threads: 8
      log.cleaner.io.max.bytes.per.second: 1.7976931348623157E308
      # Settings for log flush from in-memory to disk
      log.flush.scheduler.interval.ms: 2000
      log.flush.interval.ms: 50000
      log.flush.interval.messages: 100000
      # The lag time before a follower is considered out of sync
      replica.lag.time.max.ms: 30000
      # A maximum size for the messages fetched by each follower that replicates the data from a leader partition.
      replica.fetch.max.bytes: 1048576 # 1 Mebibyte
      # Leader imbalance check      
      leader.imbalance.check.interval.seconds: 300
      leader.imbalance.per.broker.percentage: 10
      # A delay for the group coordinator to wait for members to join before the initial rebalance
      group.initial.rebalance.delay.ms: 3000
      # The max time to establish connection to zookeeper
      zookeeper.connection.timeout.ms: 18000      
    storage:
      type: jbod
      volumes:
      - id: 1
        type: persistent-claim
        size: 50Gi
        deleteClaim: false
        class: managed-csi-zrs 
      - id: 2
        type: persistent-claim
        size: 50Gi
        deleteClaim: false
        class: managed-csi-zrs
    rack:
      topologyKey: topology.kubernetes.io/zone
    template:
      pod:
        tolerations:
        - key: "kubernetes.azure.com/scalesetpriority"
          operator: "Equal"
          value: "spot"
          effect: "NoSchedule"
        topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/name: west-2-shared-kafka-kafka
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/name: west-2-shared-kafka-kafka
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
  zookeeper:
    replicas: 3
    resources:
      requests:
        memory: 2Gi
        cpu: 1
      limits:
        memory: 4Gi
        cpu: 2
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
      class: managed-csi-zrs
    template:
      pod:
        tolerations:
        - key: "kubernetes.azure.com/scalesetpriority"
          operator: "Equal"
          value: "spot"
          effect: "NoSchedule"
        topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/name: west-2-shared-kafka-zookeeper
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/name: west-2-shared-kafka-zookeeper
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: zookeeper-metrics-config.yml
  entityOperator:
    template:
      pod:
        tolerations:
        - key: "kubernetes.azure.com/scalesetpriority"
          operator: "Equal"
          value: "spot"
          effect: "NoSchedule"
        topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/name: west-2-shared-kafka-entity-operator
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/name: west-2-shared-kafka-entity-operator
    tlsSidecar:
      resources:
        requests:
          cpu: 100m
          memory: 100Mi
        limits:
          cpu: 500m
          memory: 500Mi
    topicOperator:
      resources:
        requests:
          memory: 500Mi
          cpu: 500m
        limits:
          memory: 1Gi
          cpu: 1
    userOperator:
      resources:
        requests:
          memory: 500Mi
          cpu: 500m
        limits:
          memory: 1Gi
          cpu: 1
  cruiseControl:
    template:
      pod:
        tolerations:
        - key: "kubernetes.azure.com/scalesetpriority"
          operator: "Equal"
          value: "spot"
          effect: "NoSchedule"
    tlsSidecar:
      resources:
        requests:
          memory: 500Mi
          cpu: 500m
        limits:
          memory: 1Gi
          cpu: 1
    resources:
      requests:
        memory: 500Mi
        cpu: 500m
      limits:
        memory: 1Gi
        cpu: 1
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: cruise-control-metrics
          key: metrics-config.yml
  kafkaExporter:
    resources:
      requests:
        cpu: 500m
        memory: 500Mi
      limits:
        cpu: 1
        memory: 1Gi
    topicRegex: ".*"
    groupRegex: ".*"

[scholzj]

I guess you need to complete the increase of the disk space first and not shrink it back.

[Sai-Charan-Madhvaraj]

seems we should not do any operation ( increase / decrease) on existing disks..either add or delete the disks .. is my understanding right ?

If we increase space on disk space it will fail because PV is already mounted to a running pod.

[scholzj]

You can do the supported operations. That includes increasing the disk size if supported by your storage / platform. That should be indicated in your Storage Class which it seems was the case since the operator accepted the change.

[Sai-Charan-Madhvaraj]

Yeah true, we had allowVolume expansion = true in storage class..

But Strimzi is not bouncing the pods to allow the increase capacity operation hence increase operation is not use.

[scholzj]

Strimzi will restart the pod when it is indicated that file system resizing needs the pod to be restarted through the PVC status. Is that the case for you? If it is and it does not restart it then it might be a bug. If your storage doesn't indicate it, then Strimzi has no way to know that restart is needed and oyu might need to do it manually.