-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add example of reordering the events of a Kafka Streams application #53
Conversation
Added functionality to reorder Kafka Streams by the timestamp embedded in the message payload. Created new Java classes ReorderStreams, ReorderProcessorSupplier and Event, and respective unit tests to support the functionality. Updated the project structure and build.gradle as needed.
Imported java.time.Instant and refactored the test method in the ReorderProcessorTest class for better readability. Also updated the version of Kafka streams and clients dependencies from 3.6.0 to 3.7.0 in the build.gradle files of both the ReorderStreams and common modules.
…rder events The README for the reordering-streams/kstreams module has been significantly improved to provide more detailed instructions on how to reorder out-of-order events in Kafka Streams. Additionally, the ReorderingProcessorSupplier.java file has been slightly edited to improve its readability.
The method name `punctuate` was updated to `forwardOrderedByEventTime` to more accurately reflect its function. The variable `grace` was also renamed to `reorderWindow` for more clarity. Both code and README file were updated with a detailed explanation of the reordering strategy for events in the stream; including aspects such as restoring the original key, limitations to a single partition, and considerations for choosing the window size for reordering.
\cc @mjsax @sshcherbakov |
|
||
1. It's essential to have a way to restore the incoming key, you don't want to lose the original key-partition mapping. | ||
2. This re-ordering strategy only applies to a single partition, not across multiple partitions. | ||
3. Since an event stream is infinite, re-ordering can only be applied to distinct windows of time, and you'll balance the trade-off of large windows and iterating over the entire contents of a state store. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
3. Since an event stream is infinite, re-ordering can only be applied to distinct windows of time, and you'll balance the trade-off of large windows and iterating over the entire contents of a state store. | |
3. Since an event stream is infinite, reordering can only be applied to distinct windows of time, and you'll balance the trade-off of large windows and iterating over the entire contents of a state store. |
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
delete whitespace
/** | ||
* Bill Bejeck | ||
* 6/7/24 | ||
*/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remove? did IDE add this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But I don't want to be forgotten! Ha!
Yep it's IDE generated, I'll have to disable for this repo
/** | ||
* User: Bill Bejeck | ||
* Date: 6/7/24 | ||
* Time: 1:54 PM | ||
*/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remove? did IDE add this?
The unnecessary comments from several files, specifically Event.java and ReorderingProcessorSupplier.java, are removed. Also, redundant lines in the markdown file README.md are deleted for a cleaner and more concise document.
Co-authored-by: Dave Troiano <[email protected]>
A tutorial demonstrating how to re-order a Kafka Streams application by timestamps embedded in the event payload.
This tutorial originally began as a PR to kafka-tutorials-examples with the discussions of migrating the content in the kafka-streams-examples repo as individual tutorials in this repository it made sense to convert the above ks-examples PR to one for this repository.