Fix "Pending Forever" forever

[terrazoon]

When running jobs for the US Census, we observed that some notifications got stuck in a Pending state after the 4 hour mark when all statuses should be resolved.

We are going to refactor our delivery receipts checking in the near future and allow up to 72 hours for the status to resolve, but we need a "final" way to clear Pending regardless of what happens.

I suggest that we make a task that runs hourly and pushes any notifications more than 72 hours old from Pending to Failed.

[ccostino]

Copying here for more context, @terrazoon!

Right now with delivery receipts we are getting a message id for each notification, and then we are going and filtering log events for that message id. Which means if we have 28000 notifications we are making 28000 calls, sometimes multiple times.

Maybe it would make more sense when a job is running periodically FilterLogEvents by job_id and "phoneCarrier" and get back 5000, 10000, 20000 statuses at one time and update that way. This might avoid the throttling which is definitely happening, and it might (?) be faster.

There is probably some way to filter like "give me all the new ones since the last time I asked"