fix(digest): write digest on middleware call

[mhenrixon]

Closes #766

@JeremiahChurch can you check if this solves your problem?

[JeremiahChurch]

Apologies for the delayed response.

Local shake down of this has been positive. I'm going to get it in to our production this week - should know within a few hours if it solves our problems or not. Really appreciate the work on it! <3

[JeremiahChurch]

quick follow up - went in to prod today with the few recent changes on main (here). Unusual issue required it come right back out, unfortunately.

Only one or two workers were starting on each box (concurrency of 25 + ~15 more in capsules)

I'll pull the log details tomorrow and provide as well as look for what may have caused it.

[JeremiahChurch]

Apologies for the massive delay on this.

We've been running this fix against the top of main for the last ~month and it has greatly improved our processing.

I haven't found a single overlapping/duplicate unique key for different jobs - I'd call this a good fix!

This fix has 10+ fold increased #821 reoccurrence as we're generating so many additional keys.

[mhenrixon]

This fix has 10+ fold increased #821 reoccurrence as we're generating so many additional keys.

Alright, I will release this after fixing the number of keys in the digests set.

[mhenrixon]

@JeremiahChurch I believe that the last commit cddcc08 will help you. There was potential for lingering keys in the original if statement.