Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reuse AWS client between S3 requests #1430

Merged
merged 1 commit into from
Jul 2, 2024
Merged

Conversation

richardTowers
Copy link
Contributor

@richardTowers richardTowers commented Jul 2, 2024

At the moment, we create new AWS clients whenever we download or head an object. Each time we do this, the client requests a new set of AWS credentials using the pod's (or the node's?) metadata endpoint.

This causes a problem when asset-manager is under high load, because the AWS metadata endpoint has a rate limit. Eventually, we start getting 429 errors back from the metadata endpoint, which the AWS client reports as:

Aws::Sigv4::Errors::MissingCredentialsError: missing credentials

(At least according to issue #2823 in aws-sdk-ruby)

We see this as a spike of 500 errors every night when the mirror script runs:

image

The S3Storage instance in asset-manager is created once when the app loads, and set as Services.cloud_storage. In other words it's effectively a singleton - the same instance will be used by all requests.

Aws::S3::Client instances are thread safe, and can be reused between requests (again, according to issue #2823 in aws-sdk-ruby).

By creating the client once, instead of once per request, we'll avoid hitting the AWS metadata endpoint on every request to S3, which should mean that we don't get rate limited.

I've lightly tested this on integration by making sure I can still upload and download assets. No errors in the logs.


This application is owned by the publishing platform team. Please let us know in #govuk-publishing-platform when you raise any PRs.

⚠️ This repo is Continuously Deployed: make sure you follow the guidance ⚠️

At the moment, we create new AWS clients whenever we download or head an object. Each time we do this, the client requests a new set of AWS credentials using the pod's (or the node's?) metadata endpoint.

This causes a problem when asset-manager is under high load, because the AWS metadata endpoint has a rate limit. Eventually, we start getting 429 errors back from the metadata endpoint, which the AWS client reports as:

    Aws::Sigv4::Errors::MissingCredentialsError: missing credentials

(At least according to issue #2823 in aws-sdk-ruby)

The S3Storage instance in asset-manager is created once when the app loads, and set as Services.cloud_storage. In other words it's effectively a singleton - the same instance will be used by all requests.

Aws::S3::Client instances are thread safe, and can be reused between requests (again, according to issue #2823 in aws-sdk-ruby).

By creating the client once, instead of once per request, we'll avoid hitting the AWS metadata endpoint on every request to S3, which should mean that we don't get rate limited.
@richardTowers richardTowers marked this pull request as ready for review July 2, 2024 16:16
@richardTowers richardTowers merged commit 342141a into main Jul 2, 2024
9 checks passed
@richardTowers richardTowers deleted the reuse-aws-client branch July 2, 2024 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants