Kafka Observer Sample Pattern #80
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
qu1queee
force-pushed
the
qu1queee/kafka_observer
branch
3 times, most recently
from
b433c75 to
c89841a
Compare
reggeenr
requested changes
Mar 19, 2024
There was a problem hiding this comment.
Hi team, great job. Based on the on the observer pattern I was able to cut down costs in my sample scenarios by roughly 50% while still staying the acceptable performance ranges for the consumption latency.
Based on the extensive tests that I ran, I suggest a few tweaks and changes in the observer and consumer implementation. Besides some performance tweaks, I would like to revisit the solution design w.r.t. how the observer consumes messages from the topics.
Again, great job!
qu1queee
force-pushed
the
qu1queee/kafka_observer
branch
6 times, most recently
from
46e04ef to
c882b9c
Compare
Introduce kafkadata file Add run.sh Add related Dockerfile's Improve README Signed-off-by: encalada <[email protected]>
Signed-off-by: Karan Kumar <[email protected]> Signed-off-by: encalada <[email protected]>
Signed-off-by: encalada <[email protected]>
Signed-off-by: encalada <[email protected]>
Enhance config fields Signed-off-by: encalada <[email protected]>
Adopt the code to read from the embedded env Vars into our Jobs. Adjust the run.sh script and the README accordingly. Signed-off-by: encalada <[email protected]>
Signed-off-by: encalada <[email protected]>
Signed-off-by: encalada <[email protected]>
- Implement the custom KafkaTopic interface. This allow us to operate on Topics and Consumer Groups offsets. It also uses goroutines and channels, to determine when a change is done in terms of offsets comparisons. We dont longer commit neither read topic messages. - Improve logging - Add missing functions comments - Fixes for job_invoker - Improve handling on env vars for both consumers and observers. - Move images from golang to alpine - Introduce IDLE_TIMEOUT env var Signed-off-by: encalada <[email protected]> Signed-off-by: Mahesh Kumawat <[email protected]> Signed-off-by: encalada <[email protected]>
Co-authored-by: qu1queee <[email protected]> Signed-off-by: encalada <[email protected]>
- when new topic and consumer group - make ticker configurable Co-authored-by: qu1queee <[email protected]> Signed-off-by: encalada <[email protected]>
qu1queee
force-pushed
the
qu1queee/kafka_observer
branch
from
645139c to
5fd21fe
Compare
reggeenr
previously approved these changes
Apr 16, 2024
|
While testing, I realized that from time to time one can see large outliers. First I thought this is due to a timing issue in which the observer does not start a new instance because there seems to be still a jobrun around but then the jobrun stops right after and so misses new messages that arrive. However, I can see that in the log below the observer did not noticed new messages sent to message |
reggeenr
dismissed
their stale review
I want to better understand the ticker logic, as I have a concern that there is a bug
You would not always see in the logs the new messages(offset change), due to the ticker. But the latest offset change being logged should reflect the change from all previous arrived messages. This is ok, as long as a Jobrun was already running. |
|
Sorry that it took so long. I finally was able to recreate a test setup. Long story short, I wasn't able to reproduce the issue that I observed once in a long running test scenario. This was my test setup By sending messages to both topics, I was able to observe that the corresponding consumer were woken up by the observer. Hence. I am approving :) Sorry for the delay. Great job team |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This introduces a Kafka Observer Pattern for consumes a stream of messages from Kafka.
A so called Observer entity is provided, this Observer has the task to consume messages from multiple Kafka Topics and wake-up the so called Consumers whenever new messages arrived.
Consumers are the second entity in this sample. They will consume messages from a particular Kafka Topic after being wake-up by the Observer. Note that users might bring their own Consumer Kafka Clients code. In this example, a Consumer client written in go is provided. Consumers will gracefully shut-down after one minute, if no new messages are coming through the stream.
A comprehensive README is provided, on how to use this sample.