-
Notifications
You must be signed in to change notification settings - Fork 46
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
EndOffset of a redelivery consumer #12
Comments
The It doesn't affect which markers are tracked or not. So in your example, The setting only affects when markers will be checked for redeliveries. |
Ok, I agree SM3 would be tracked but it would not be redelivered as redelivery would be disabled for that offset. This should not be the required behavior as redelivery is required to be done, as it was not even done by the first instance of redelivery tracker. |
It would be redelivered, but possibly not immediately, but at a later time, after some more markers are published to the topic. |
Just wanted to know the use of the disableRedeliveryBefore argument of the marker queue?
And as per the logic, I feel the following case scenario would be a failure.
for code reference :
kmq/core/src/main/scala/com.softwaremill.kmq/redelivery/ConsumeMarkersActor.scala
Line 71 in 391f9f9
Case Scenario:
For a Topic(X) there is a marker topic(MT). One redelivery tracker is listening to the marker topic.
The order of markers on a partition are SM1, SM2, EM1, EM2, where SMx is start-marker for some key x and EMx is end-marker for some key x. the redelivery tracker commits on Kafka till EM2.
Then redelivery tracker disconnects and a new redelivery consumer is created, and in between a new SM3 is published. But as the consumer connects after SM3, endOffset/ disabledRedeliveryBefore for MarkerQueue of partition is after the SM3 offset.
Problem:
SM3 would not be tracked by any redelivery tracker.
Possible Solution:
Rather than endOffset, the redelivery tracker can disable redelivery before the last committed offset.
The text was updated successfully, but these errors were encountered: