Skip to content
This repository has been archived by the owner on Nov 20, 2020. It is now read-only.

Parallelize consumer fetch requests #82

Open
eulerfx opened this issue Jan 3, 2017 · 0 comments
Open

Parallelize consumer fetch requests #82

eulerfx opened this issue Jan 3, 2017 · 0 comments

Comments

@eulerfx
Copy link
Contributor

eulerfx commented Jan 3, 2017

Currently, the logic to route requests to brokers is contained in the connection. Fetch, Produce and Offset requests can be addressed to multiple brokers based on the partitions involved. When the consumer sends fetch requests, it does so for the entire set of partitions assigned to it. The connection then routes them to the appropriate brokers in a fork-join fashion. This join however is needless and reduces utilization/parallelism. It is better to have a fetch process per broker independent of the fetch processes of other brokers. Either way, the fetch processes publish messages into an exchange which distributes into per-partition streams.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant