Skip to content

EndOffset of a redelivery consumer #12

@shubhamDhoble

Description

@shubhamDhoble

Just wanted to know the use of the disableRedeliveryBefore argument of the marker queue?

And as per the logic, I feel the following case scenario would be a failure.
for code reference :

assignedPartitions += p -> AssignedPartition(new MarkersQueue(endOffset - 1), redeliverActor, None, None)

Case Scenario:
For a Topic(X) there is a marker topic(MT). One redelivery tracker is listening to the marker topic.
The order of markers on a partition are SM1, SM2, EM1, EM2, where SMx is start-marker for some key x and EMx is end-marker for some key x. the redelivery tracker commits on Kafka till EM2.
Then redelivery tracker disconnects and a new redelivery consumer is created, and in between a new SM3 is published. But as the consumer connects after SM3, endOffset/ disabledRedeliveryBefore for MarkerQueue of partition is after the SM3 offset.

Problem:
SM3 would not be tracked by any redelivery tracker.

Possible Solution:
Rather than endOffset, the redelivery tracker can disable redelivery before the last committed offset.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions