Add retry logic (and delays?) to our task handling

[rsyring]

See:

• retries
• async invocation
• configuring async

We'd have to put the retry config into the task invocation event and then handle it ourselves. We could handle short (<10s?, <1m?) retry delays by sleeping. By sleeping we pay for the execution time. So we'd probably only want to use this method for quick retries like a failed network/db connection.

If we want a longer delay, we'd need to put a message in an sqs queue with a timer. Max delay on that is only 15 mins. No obvious way to do a longer delay (hours) like we can do with Celery.