Refactor delivery receipts

[terrazoon]

Right now we get a message_id for a notification and then we use FilterLogEvents to search the AWS logs using that message_id to obtain the delivery status. This results in many many calls to CloudWatch and we are getting throttled.

We could refactor as follows:

1. At send time, write the message_id to the notification table
2. Create a scheduled task to periodically fetch logs in a given time range (last ten minutes, let's say)
3. Loop through the result set and update all the DELIVERED and FAILED notifications using the message id
4. When we feel confident of the new approach, remove the old approach.

Updated approach

After more experimentation, research, and guidance, we've determined that using the CloudWatch Log Insights would work even better for this:

• Use a scheduled task to run log insights every 8 minutes.
• Log insights itself will scan a 10 minute chunk (use overlap to avoid edge cases).
• Database with be batch updated.

[terrazoon]

PR to add message_id to notifications table: #1474

[terrazoon]

1. write one scheduled task that runs every ten minutes and downloads all delivery results from t-15 to t-5.
2. write one schedule task the runs every 12 minutes and downloads all delivery receipts from t-22 to t-10

When we get the delivery receipts, make them into a batch statement and do a batch update of the notifications table.

The reason for doing two passes is to improve the accuracy of the mass download, where there are sometimes edge cases. If we need further accuracy, add a 13 minute run etc.