Handle 503 Slow Down

[maartenberg]

The S3 API may reject requests with a custom error (503 Slow Down) when a rate
limit has been reached. Boto automatically retries these requests, this does
not always work. Example traceback:

Traceback (most recent call last):
  File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/boto3/s3/transfer.py", line 279, in upload_file
    future.result()
  File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
  File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/s3transfer/upload.py", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/botocore/client.py", line 320, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/botocore/client.py", line 623, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (SlowDown) when calling the PutObject operation (reached max retries: 4): Please reduce your request rate.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 6, in <module>
    cli.main()
  File "/home/username/pxl/pxl/cli.py", line 224, in main
    cli()
  File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/username/pxl/pxl/cli.py", line 127, in upload_cmd
    image = upload.public_image_with_size(client, entry)
  File "/home/username/pxl/pxl/upload.py", line 102, in public_image_with_size
    public_image(client, local_filename, object_name)
  File "/home/username/pxl/pxl/upload.py", line 124, in public_image
    Key=object_name,
  File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/boto3/s3/inject.py", line 131, in upload_file
    extra_args=ExtraArgs, callback=Callback)
  File "/home/username/.local/share/virtualenvs/pxl-IyaRoL09/lib/python3.7/site-packages/boto3/s3/transfer.py", line 287, in upload_file
    filename, '/'.join([bucket, key]), e))
boto3.exceptions.S3UploadFailedError: Failed to upload /tmp/photofile.jpeg to the-bucket-name/abcdef12-3456-7890-abcd-ef1234567890_w_1600.jpg: An error occurred (SlowDown) when calling the PutObject operation (reached max retries: 4): Please reduce your request rate.

Some sort of backoff handling would probably be useful. I found
https://pypi.org/project/backoff/, which would probably work.