Reuse AWS client between S3 requests #1430
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
At the moment, we create new AWS clients whenever we download or head an object. Each time we do this, the client requests a new set of AWS credentials using the pod's (or the node's?) metadata endpoint.
This causes a problem when asset-manager is under high load, because the AWS metadata endpoint has a rate limit. Eventually, we start getting 429 errors back from the metadata endpoint, which the AWS client reports as:
(At least according to issue #2823 in aws-sdk-ruby)
We see this as a spike of 500 errors every night when the mirror script runs:
The
S3Storage
instance in asset-manager is created once when the app loads, and set asServices.cloud_storage
. In other words it's effectively a singleton - the same instance will be used by all requests.Aws::S3::Client
instances are thread safe, and can be reused between requests (again, according to issue #2823 in aws-sdk-ruby).By creating the client once, instead of once per request, we'll avoid hitting the AWS metadata endpoint on every request to S3, which should mean that we don't get rate limited.
I've lightly tested this on integration by making sure I can still upload and download assets. No errors in the logs.
This application is owned by the publishing platform team. Please let us know in #govuk-publishing-platform when you raise any PRs.