Ingest storage kafka topic compression

[z0rc]

Documentation mentions that ingest storage uses compression https://github.com/grafana/mimir/blob/mimir-3.0.0/docs/sources/mimir/configure/configure-zone-aware-replication.md?plain=1#L122. But it doesn't specify what compression algorightm it uses.

I'm interested in this, because when inspecting individual messages in kafka topic, they look uncompressed.

Also I wasn't able to find place in code that supposedly configures compression for ingest storage.

Does it make sense to enable compression on kafka topic in this case? To reduce storage allocated for topics.

[deniszh]

According to the code github.com/twmb/franz-go/ used as Kafka client, and looks like it picks ingest_storage.kafka section from config and construct client options from it. But I doi not see any compression options in KafkaConfig.
Probably it's not implemented.

[simioa]

You can check if compression is used with the DumpLogSegments tool in Kafka: https://stackoverflow.com/a/36625558

In my case compresscodec is set to snappy

[z0rc]

I'm using AutoMQ, so DumpLogSegments doesn't work for my case, it isn't able to parse logs uploaded to S3.

[deniszh]

Hi @simioa ,
Where do you setting compression? in broker settings?

[simioa]

I did not set any compression related settings, It looks as if this is set by the producer.

[z0rc]

@simioa isn't it just default compression.codec=producer which assumes to use whatever compression the producer sets, which effectively is actually none in case of mimir.

[simioa]

@z0rc mimir uses kgo as its Kafka client, which appears to use the snappy codec by default.: https://github.com/twmb/franz-go/blob/master/pkg/kgo/config.go#L559

However, I'm not entirely sure because I haven't looked at the rest of the code in both repositories.

[z0rc]

@simioa interesting, thanks. Though it isn't entirely clear which compression codec is picked from array []CompressionCodec{SnappyCompression(), NoCompression()}.