Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EndOffset of a redelivery consumer #12

Open
shubhamDhoble opened this issue Jul 27, 2018 · 3 comments
Open

EndOffset of a redelivery consumer #12

shubhamDhoble opened this issue Jul 27, 2018 · 3 comments

Comments

@shubhamDhoble
Copy link

shubhamDhoble commented Jul 27, 2018

Just wanted to know the use of the disableRedeliveryBefore argument of the marker queue?

And as per the logic, I feel the following case scenario would be a failure.
for code reference :

assignedPartitions += p -> AssignedPartition(new MarkersQueue(endOffset - 1), redeliverActor, None, None)

Case Scenario:
For a Topic(X) there is a marker topic(MT). One redelivery tracker is listening to the marker topic.
The order of markers on a partition are SM1, SM2, EM1, EM2, where SMx is start-marker for some key x and EMx is end-marker for some key x. the redelivery tracker commits on Kafka till EM2.
Then redelivery tracker disconnects and a new redelivery consumer is created, and in between a new SM3 is published. But as the consumer connects after SM3, endOffset/ disabledRedeliveryBefore for MarkerQueue of partition is after the SM3 offset.

Problem:
SM3 would not be tracked by any redelivery tracker.

Possible Solution:
Rather than endOffset, the redelivery tracker can disable redelivery before the last committed offset.

@adamw
Copy link
Member

adamw commented Jul 31, 2018

The disableRedeliveryBefore argument is there to enable redeliveries only after all known data has been loaded from Kafka to build the MarkersQueue state on startup or after a restart.

It doesn't affect which markers are tracked or not. So in your example, SM3 would still be tracked and considered for redelivery (provided that there's some movement on the topic, and more markers are seen).

The setting only affects when markers will be checked for redeliveries.

@shubhamDhoble
Copy link
Author

Ok, I agree SM3 would be tracked but it would not be redelivered as redelivery would be disabled for that offset. This should not be the required behavior as redelivery is required to be done, as it was not even done by the first instance of redelivery tracker.

@adamw
Copy link
Member

adamw commented Aug 1, 2018

It would be redelivered, but possibly not immediately, but at a later time, after some more markers are published to the topic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants