Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add example of reordering the events of a Kafka Streams application #53

Merged
merged 6 commits into from
Jul 1, 2024

Conversation

bbejeck
Copy link
Contributor

@bbejeck bbejeck commented Jun 13, 2024

A tutorial demonstrating how to re-order a Kafka Streams application by timestamps embedded in the event payload.

This tutorial originally began as a PR to kafka-tutorials-examples with the discussions of migrating the content in the kafka-streams-examples repo as individual tutorials in this repository it made sense to convert the above ks-examples PR to one for this repository.

bbejeck added 4 commits June 7, 2024 18:35
Added functionality to reorder Kafka Streams by the timestamp embedded in the message payload. Created new Java classes ReorderStreams, ReorderProcessorSupplier and Event, and respective unit tests to support the functionality. Updated the project structure and build.gradle as needed.
Imported java.time.Instant and refactored the test method in the ReorderProcessorTest class for better readability. Also updated the version of Kafka streams and clients dependencies from 3.6.0 to 3.7.0 in the build.gradle files of both the ReorderStreams and common modules.
…rder events

The README for the reordering-streams/kstreams module has been significantly improved to provide more detailed instructions on how to reorder out-of-order events in Kafka Streams. Additionally, the ReorderingProcessorSupplier.java file has been slightly edited to improve its readability.
The method name `punctuate` was updated to `forwardOrderedByEventTime` to more accurately reflect its function. The variable `grace` was also renamed to `reorderWindow` for more clarity. Both code and README file were updated with a detailed explanation of the reordering strategy for events in the stream; including aspects such as restoring the original key, limitations to a single partition, and considerations for choosing the window size for reordering.
@bbejeck bbejeck requested a review from a team as a code owner June 13, 2024 16:29
@bbejeck bbejeck changed the title feat: Add example of reorder Kafka Streams feat: Add example of reordering the events of a Kafka Streams application Jun 13, 2024
@bbejeck
Copy link
Contributor Author

bbejeck commented Jun 13, 2024

\cc @mjsax @sshcherbakov

reordering-streams/kstreams/README.md Outdated Show resolved Hide resolved
reordering-streams/kstreams/README.md Outdated Show resolved Hide resolved
reordering-streams/kstreams/README.md Outdated Show resolved Hide resolved
reordering-streams/kstreams/README.md Outdated Show resolved Hide resolved
reordering-streams/kstreams/README.md Outdated Show resolved Hide resolved
reordering-streams/kstreams/README.md Outdated Show resolved Hide resolved

1. It's essential to have a way to restore the incoming key, you don't want to lose the original key-partition mapping.
2. This re-ordering strategy only applies to a single partition, not across multiple partitions.
3. Since an event stream is infinite, re-ordering can only be applied to distinct windows of time, and you'll balance the trade-off of large windows and iterating over the entire contents of a state store.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
3. Since an event stream is infinite, re-ordering can only be applied to distinct windows of time, and you'll balance the trade-off of large windows and iterating over the entire contents of a state store.
3. Since an event stream is infinite, reordering can only be applied to distinct windows of time, and you'll balance the trade-off of large windows and iterating over the entire contents of a state store.

Comment on lines 117 to 124








Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

delete whitespace

Comment on lines 3 to 6
/**
* Bill Bejeck
* 6/7/24
*/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove? did IDE add this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But I don't want to be forgotten! Ha!
Yep it's IDE generated, I'll have to disable for this repo

Comment on lines 19 to 23
/**
* User: Bill Bejeck
* Date: 6/7/24
* Time: 1:54 PM
*/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove? did IDE add this?

The unnecessary comments from several files, specifically Event.java and ReorderingProcessorSupplier.java, are removed. Also, redundant lines in the markdown file README.md are deleted for a cleaner and more concise document.
@bbejeck bbejeck requested a review from a team as a code owner July 1, 2024 19:25
@bbejeck bbejeck merged commit 6984828 into master Jul 1, 2024
@bbejeck bbejeck deleted the reorder-streams branch July 1, 2024 19:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants