Kafka topic data retention issue

[kmandal-volvo]

Deployed Kafka using strimzi operator version 0.20. Kafka version used 2.6 I am facing an issue with data retention, it unable to persist data for more than 2 hours. The message disappears automatically from that broker, the topic for that partition and offset. when I add more data into the topic in the same partition it adding from the next offset from the last inserted offset location.

Data retention configuration used
log.message.format.version: "2.3"
log.retention.bytes: 107374182400
log.retention.check.interval.ms: 300000
log.retention.hours: 168
log.segment.bytes: 1073741824

Storage use persistence storage

storage:
type: persistent-claim
persistentVolumeReclaimPolicy: Retain
size: 30Gi
class: managed-premium-retain
deleteClaim: false

[scholzj]

Strimzi does not do anything special about retention configuration. It is just running Kafka pretty much exactly as you would when running it outside of Kubernetes. You do not share the full configuration neither for the broker nor for the topic, so it is hard to understand how did you configured things. For example there is no persistentVolumeReclaimPolicy field in the storage configuration (this in particular should not cause what you described - but the questions is whether there is something else wrong).

[kmandal-volvo]

Thanks for your input it is because of my topic configuration retention.ms limit did not notice. I will update based on my neeed

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaTopic
metadata:
name: icoe-ext
labels:
strimzi.io/cluster: vcc-external-esp
spec:
partitions: 1
replicas: 1
config:
retention.ms: 7200000
segment.bytes: 1073741824

[scholzj]

Yeah, the retention.ms option will always overwrite the broker setting => the configuration in the broker is just a default in case you don't specifiy the retention in the topic it self.