Kafka PVC's become full almost immediately. ZooKeeper is fine.

[towens]

We have a basic test cluster running strimzi 0.23.0, k8s v1.18.12, KOPS, AWS. Our volumes are filling almost immediately. Probably something simple, we have yet to determine the issue and any insight would be much appreciated.

We are generating ~10k events per second at 2.5 bytes.
Lag spikes initially and becomes stable.
Messages in per second and message consumed per second (grafana) are nearly 1:1.

We have slowly been reducing topic retention, it seems to have no effect.

Topic

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: redacted-test
  namespace: kafka
  labels:
    strimzi.io/cluster: redacted
spec:
  partitions: 50
  replicas: 3
  topicName: redacted-test
  config:
    retention.ms: 1000

Cluster

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: redacted
  namespace: kafka
spec:
  kafka:
    replicas: 3
    version: 2.8.0
    logging:
      type: inline
      loggers:
        kafka.root.logger.level: "ERROR"
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
    storage:
      type: jbod
      volumes:
        - id: 0
          type: persistent-claim
          class: kops-ssd-1-17
          size: 80Gi
          deleteClaim: true
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
  zookeeper:
    replicas: 1
    logging:
      type: inline
      loggers:
        zookeeper.root.logger: "ERROR"
    storage:
      type: persistent-claim
      class: kops-ssd-1-17
      size: 10Gi
      deleteClaim: true
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: zookeeper-metrics-config.yml
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"

Skipped including the configmap

[scholzj]

retention.ms: 1000 does make absolutely no sense for Kafka and most probably does not do what you expect. Kafka retention does not work as message TTL or expiration in traditional messaging broker. If you want your messages to be so short lived, you should maybe try something more suitable such as ActiveMQ Artemis etc. Kafka documentation should explain more about how the retention works.

If you want to reduce the disk usage, you should for example consider:

• Reducing the segment size
• Configuring how often are new segments rolled out
• Using byte retention

[towens]

Good to know, we don't need/want retention.ms: 1000 It's just what it got dropped down to. We will return to including byte retention in the topic and investigate segment size and rollout.

Thanks for the suggestions

[scholzj]

Ok, good ... if you do not need it really, I would focus on byte retention and segment size + segment roll out. Basically, Kafka drops only whole segments. So with the default 1GB per segment, you always need to count with having 1+GB for partition-replica. So playing with that might help a lot.

[towens]

Excellent. Thanks for imparting that info. Encouraging!

[towens]

@scholzj reducing segement.bytes and re-introducing retention.bytes fixed the issue. Thank you!