parallel-process returns a lazy seq that is never materialized

[tampix]

Hello and thanks for your library.

I found something that looks like a bug.

Currently, parallel-process returns a lazy seq, produced by the call to claypoole/upmap.

Consumer code being :

(fn []
  (reset! (:running config) true)
  (while @(:running config)
    ;; neither dequeue or run-dequeue-fn (here core/default-top-level-error-handler)
    ;; read the value returned by process-fn
    (run-dequeue-fn
     #(dequeue config process-fn)))
  (reset! (:finished-shutdown config) true))

This means that dequeue will be called without waiting for materialization of the result of process-fn (here, parallel-process).
This can lead to receiving messages up until reaching SQS Messages in flight limit or saturating the JVM's heap as you can't have any control on the amount of messages hold in memory.

One possible solution would be to force the materialization of the lazy seq with either dorun or doall, which would seem more correct :

• receive up to 10 messages
• process them in parallel
• when all were processed, receive the next 10 messages