-
Notifications
You must be signed in to change notification settings - Fork 44
Description
Just wanted to know the use of the disableRedeliveryBefore argument of the marker queue?
And as per the logic, I feel the following case scenario would be a failure.
for code reference :
kmq/core/src/main/scala/com.softwaremill.kmq/redelivery/ConsumeMarkersActor.scala
Line 71 in 391f9f9
assignedPartitions += p -> AssignedPartition(new MarkersQueue(endOffset - 1), redeliverActor, None, None) |
Case Scenario:
For a Topic(X) there is a marker topic(MT). One redelivery tracker is listening to the marker topic.
The order of markers on a partition are SM1, SM2, EM1, EM2, where SMx is start-marker for some key x and EMx is end-marker for some key x. the redelivery tracker commits on Kafka till EM2.
Then redelivery tracker disconnects and a new redelivery consumer is created, and in between a new SM3 is published. But as the consumer connects after SM3, endOffset/ disabledRedeliveryBefore for MarkerQueue of partition is after the SM3 offset.
Problem:
SM3 would not be tracked by any redelivery tracker.
Possible Solution:
Rather than endOffset, the redelivery tracker can disable redelivery before the last committed offset.