Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SwimlaneDispatcher configurable - hashing algorithm, threadpool size #57

Open
cer opened this issue Aug 4, 2022 · 3 comments
Open

SwimlaneDispatcher configurable - hashing algorithm, threadpool size #57

cer opened this issue Aug 4, 2022 · 3 comments

Comments

@cer
Copy link
Contributor

cer commented Aug 4, 2022

Currently,

  • the SwimlaneDispatcher parallelizes message processing by partition.
  • Messaged from different topics with the same partition are processed by the same thread.
  • As a result, the degree of parallelism is limited

Proposal:

  • Make the (topic, partition) -> thread algorithm configurable
cer added a commit that referenced this issue Aug 4, 2022
…hreadpool size - defined TopicPartitionToSwimLaneMapping strategy
cer added a commit that referenced this issue Aug 5, 2022
…hreadpool size - defined TopicPartitionToSwimLaneMapping strategy
@cer
Copy link
Contributor Author

cer commented Aug 5, 2022

To configure the SwimlaneBasedDispatcher, configure an @Bean that implements TopicPartitionToSwimLaneMapping .

For example, see SwimlanePerTopicPartition

@cer
Copy link
Contributor Author

cer commented Aug 6, 2022

One more thing to consider: pass the message key to toSwimLane(TopicPartition topicPartition) to enable parallelism within a TopicPartition.

cer added a commit that referenced this issue Aug 6, 2022
… to toSwimLane(TopicPartition topicPartition) to enable parallelism within a TopicPartition.
cer added a commit that referenced this issue Aug 6, 2022
cer added a commit to eventuate-foundation/eventuate-cdc that referenced this issue Aug 7, 2022
@cer
Copy link
Contributor Author

cer commented Aug 24, 2022

I believe that MultipleSwimlanesPerTopicPartitionMapping is unnecessarily complicated.
A simpler implementation could simply hash(topic and messageKey) % desiredNumberOfSwimlanes.
This will preserve ordering of all messages published to a topic (e.g. Aggregate type) with the same message key (e.g. aggregate ID)

cer added a commit to eventuate-foundation/eventuate-messaging-kafka-quarkus that referenced this issue Oct 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant