switch to procedural approach for delivery receipts

[terrazoon]

Description

The previous implementation of checking delivery receipts did not scale. It was calling cloudwatch filter_log_events() for each phone number, which worked fine at low volumes. But at high volumes AWS started throttling us and we ended up in retry hell.

This new approach makes use of Log Insights to grab all delivery receipts for all services in 10 minute time chunks. It runs every 8 minutes, so there is some intentional overlap to avoid losing any records due to edge cases or timing.

Once we fetch a batch of receipts, we are going to update the db with a batch system so we don't move the throttling from AWS to our own db.

Security Considerations

N/A

[ccostino]

Thanks, @terrazoon!

I've gone through this and overall I think this will be a big improvement, even sticking with filter_log_events for now. 🎉

I flagged a few things that I want us to dig a bit deeper in, most notably the DB update statements because I want to make sure those don't become bottlenecks and take advantage of larger batch sizes. We may not even have to chunk them like that at all if we're only talking a few thousand records at a time.

One last thing - I see a couple of test updates and lots of older tests removed; do we need any other new tests for the new code? I'm not sure I saw them.

Thanks again for diving into this, @terrazoon!

[ccostino]

Thanks, @terrazoon, let's see how this goes!