Investigate 'adaptive' mode for retries and improve concurrency of message sends from multiple services

[terrazoon]

Right now when we do large sends we are hitting some ThrottlingExceptions with a "max retries exceeded: 4" message. This is not our celery retries, this is built into AWS. This occurs both for publishing messages and for calling FilterLogEvents when we are getting the delivery receipts.

Apparently there is a new, and currently experimental, 'adaptive' mode for AWS retries which takes into account service limits and hopefully reduces the number of AWS retries:

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html

Investigate switching over to this adaptive mode when it becomes less experimental.

Some additional guidance is also available here:

• https://aws.amazon.com/blogs/messaging-and-targeting/how-to-utilise-amazon-pinpoint-to-retry-unsuccessful-sms-delivery/
• https://aws.amazon.com/blogs/messaging-and-targeting/queueing-amazon-pinpoint-api-calls-to-distribute-sms-spikes/

Ultimately, we also want to make sure that a large batch from one partner doesn't become a blocker for all other work to go through the system from other partners.