Kafka source consumer group rebalance

[dimohammed328]

Hi guys. I have a general question about the kafka source and I don't see a great answer for it in the documentation so I'm wondering if anybody here has some insight. Suppose I have multiple vector aggregators with a kafka source in the same consumer group, configured to listen to multiple topics/partitions, with e2e acknowledgements enabled. Downstream, i have aws_s3 sink with a large batching window (30 minutes). If I scale up the number of aggregators in the same consumer group, this will trigger a partition rebalance amongst consumers. If a partition is revoked from an aggregator that has a batch stored in buffer, will this batch be discarded, or will the batch be written, resulting in duplicate messages?

[pront]

Hi @dimohammed328, I think you are asking if there's potential for losing events while Kafka is rebalancing. The events that were already emitted from the source are not lost. I think Vector will try to consume existing partitions. Also, they will have a notifier associated with them because acknowledgements are enabled, so the source knows if the sinks are done with them or not.

Also, see https://vector.dev/docs/reference/configuration/sources/kafka/#drain_timeout_ms