ProducerPool - easier remote publishing #311
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
go-nsq does not provide good primitives to handle failures when publishing to a nsq instances; Handling publish errors is left as an exercise for the developer when that logic is important for reliable message creation. Common guidance has been to prefer publishing to a colocated nsqd instance, but that configuration is less desirable in many cloud environments.
I'm proposing we add a
ProducerPool
which provides a Publish interface that can be configured with multiple nsqd instances and which will retry publishing to another instance on error. This will provide a simple way for applications publishing messages to handle write failures in a fault tolerant way.Based on outcome of nsqio/nsq#1300 this producer may also lazily organize nsqd instances into groupings of node local, zone local, region local or global and provide prioritization base on topology.
Functionally this is similar to the reliable publishing in
nsq_to_nsq
where publish happens in a round robin, or host pool to a pool of nsqd instances (but without the same backpressure).cc: nsqio/nsq#1254 which tracks facilitating use of nsq in a cloud environment.