How to manage clean up when the PVCs associated with Strimzi become full due to logs and data

[vanyagyosheva]

Hello, we faced the problem that our strimzi cluster was crashing because of full disks. Even cleaning the tmp dir, was not enough and we detached and deleted the PVCs which is not the best way to maintain the cleaning properly. Could you please have a look at our kafka config and suggest better solution to maintain a regular cleanup of old and temporary data.
Also if you see something that could be overall improved , please note.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-svc
  namespace: ${namespace}
spec:
  kafka:
    version: 3.4.0
    replicas: 1
    resources:
      requests:
        memory: 4Gi
        cpu: "1"
      limits:
        memory: 4Gi
        cpu: "2"
    jvmOptions:
      -Xms: 3276m
      -Xmx: 3276m
    logging:
      type: inline
      loggers:
        logger.clients.level: INFO
        logger.toc.level: INFO
        logger.toc.name: io.strimzi.operator.topic.TopicOperator
        logger.top.level: INFO
        logger.top.name: io.strimzi.operator.topic
        rootLogger.level: INFO
        log4j.logger.kafka.controller: INFO
        log4j.logger.org.apache.kafka.controller: INFO
        log4j.logger.kafka.log.LogCleaner: INFO
        log4j.logger.state.change.logger: INFO
    listeners:
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls
      - name: external
        port: 9092
        type: loadbalancer
        tls: false
        configuration:
          bootstrap:
            annotations:
              service.beta.kubernetes.io/azure-load-balancer-internal: "true"
          brokers:
            - broker: 0
              annotations:
                service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    template:
      externalBootstrapService:
        metadata:
          annotations:
            service.beta.kubernetes.io/azure-load-balancer-internal: "true"
      perPodService:
        metadata:
          annotations:
            service.beta.kubernetes.io/azure-load-balancer-internal: "true"
      pod:
        tmpDirSizeLimit: 5Gi
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values:
                        - kafka-svc-kafka
                topologyKey: "kubernetes.io/hostname"
        tolerations:
          - key: "${toleration.key}"
            operator: Equal
            value: "yes"
            effect: "${toleration.effect}"
    config:
      offsets.topic.replication.factor: 1
      offsets.retention.minutes: 5259600
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      inter.broker.protocol.version: "3.4"
      zookeeper.connection.timeout.ms: 120000
      log.retention.check.interval.ms: 300000
      log.retention.hours: ${log_retention_hours}
      log.segment.bytes: 1073741824
      log.cleaner.enable: true
      log.flush.interval.messages: 50000
      log.flush.interval.ms: 10000
      message.max.bytes: 100000000
      max.poll.interval.ms: 600000
      group.initial.rebalance.delay.ms: 0
      group.min.session.timeout.ms: 6000
      group.max.session.timeout.ms: 1800000
      heartbeat.interval.ms: 10000
      session.timeout.ms: 31000
      fetch.max.bytes: 100000000
      replica.fetch.max.bytes: 3303304
      delivery.timeout.ms: 300000
      num.io.threads: 8
      num.network.threads: 5
      num.partitions: 8
      num.recovery.threads.per.data.dir: 2
      socket.receive.buffer.bytes: 31460040
      socket.request.max.bytes: 104857600
      socket.send.buffer.bytes: 31460040
      ssl.cipher.suites: TLS_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
      ssl.enabled.protocols: TLSv1.2
      ssl.endpoint.identification.algorithm: HTTPS
      ssl.protocol: TLSv1.2
    storage:
      type: jbod
      volumes:
        - id: 0
          type: persistent-claim
          size: 20Gi
          class: managed-csi
          deleteClaim: true
        - id: 1
          type: persistent-claim
          size: 20Gi
          class: managed-csi
          deleteClaim: true
        - id: 2
          type: persistent-claim
          size: 20Gi
          class: managed-csi
          deleteClaim: true
        - id: 3
          type: persistent-claim
          size: 20Gi
          class: managed-csi
          deleteClaim: true
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
  zookeeper:
    replicas: 3
    resources:
      requests:
        memory: 2Gi
        cpu: "1"
      limits:
        memory: 2Gi
        cpu: "1.5"
    storage:
      size: 10Gi
      type: persistent-claim
      class: managed-csi
    logging:
      type: inline
      loggers:
        zookeeper.root.logger: "INFO"
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: zookeeper-metrics-config.yml
    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values:
                        - kafka-svc-zookeeper
                topologyKey: "kubernetes.io/hostname"
        tolerations:
          - key: "${toleration.key}"
            operator: Equal
            value: "yes"
            effect: "${toleration.effect}"
  entityOperator:
    template:
      pod:
        tolerations:
          - key: "${toleration.key}"
            operator: Equal
            value: "yes"
            effect: "${toleration.effect}"
    topicOperator:
      resources:
        requests:
          memory: 2Gi
          cpu: "1"
        limits:
          memory: 2Gi
          cpu: "1"
    userOperator: {}
  kafkaExporter:
    template:
      pod:
        tolerations:
          - key: "${toleration.key}"
            operator: Equal
            value: "yes"
            effect: "${toleration.effect}"

[scholzj]

I guess first of all, using 4 disks with 20Gi each is a wrong way to run the Kafka cluster. The JBOD storage is useful when your disks are already big - for example in multiple Ti and you are reaching the disk size limits and performance limits. With 20Gi, it will bring lot more troubles then anything else. Please check this video: https://youtu.be/7gaOUjkuR68 which covers some of the issues.

/tmp has nothing to do with full PVCs. It is completely independent of them. The PVCs are where your Kafka topics / partitions are stored. So the best way to protect them from getting full is to properly configure the retention on them to make sure they fit the disks. This blog post should give you some insights into how to do it: https://strimzi.io/blog/2021/12/17/kafka-segment-retention/

If the disk is full and your Kafka cluster does not want to start anymore, the only way forward is to mount the disk into some helper pod and manually delete some files from it.

[vanyagyosheva]

Thank you for the answer. Another question, could we create init container within the Kafka pod itself to perform cleanup tasks, rather than creating a separate helper pod. Could that be handled in the Kafka config?

[scholzj]

No, you cannot. Keep in mind that:

• The cleanup means you have misconfigured the broker, so it should be an exceptional operation.
• the cleanup has to decide which messages need to be deleted => as in lost. There are no unused files that you can just delete like that. So not sure this is something you can easily automate as it requires knowledge of your topics etc.