FEATURE REQUEST: Reliable (redis) broker

[kamikazechaser]

The Redis broker implementation uses BLPOP. If the process crashes after a dequeue, the job will be lost. I think it should be possible to use a combination of BLMOVE, the success callback to properly delete from the "in-progress" queue and a re-queuer to somewhat guarantee that a job lost during a process crash can be retried. However, extending the broker interface would allow writing reliable brokers which use different storage backends.

Has this been planned or is it within the scope of this library?

[kalbhor]

Ideally if anyone requires durability then they shouldn't be using redis as a broker anyway. I like the implementation idea in general (ie: BLMOVE over BLPOP) to make the redis broker more robust, but it doesn't provide an actual guarantee. Consider the redis node going down or the program crashing right before the success callback (which means job was actually processed) and numerous other scenarios.

For durable workloads its better to use something like kafka as the broker and backend. I plan to add an implementation soon. That said, if you think its possible to improve the existing redis broker implementation, I'd be open to a PR (or a more detailed plan).

[kalbhor]

Btw I'm looking into doing a v3 release with breaking changes, so if you have any ideas to improve the broker interfaces, would be good to know