Skip to content
This repository has been archived by the owner on Feb 19, 2021. It is now read-only.

ListBucket SPIKES in collector #57

Open
mikegrima opened this issue Jan 12, 2018 · 0 comments
Open

ListBucket SPIKES in collector #57

mikegrima opened this issue Jan 12, 2018 · 0 comments
Assignees
Labels

Comments

@mikegrima
Copy link
Contributor

mikegrima commented Jan 12, 2018

The S3 ListBuckets call will spike in the collector when there are frequent bucket events that occur.

ListBuckets is an expensive call to AWS, and we will need to significantly reduce the number of times it is called.

Things that need to be done:

  1. Need to pass in a session name to cloudaux so that the S3 collector function's ListBucket call is visible in CloudTrail.
  2. On collection events, check if the S3 bucket already exists in the current table. If it does, then DO NOT attempt to get the bucket creation date (which calls ListBuckets under the hood). Instead, grab the existing creation and reuse it. If it doesn't -- then the bucket is brand new and we need to grab the creation date.

While spikes in bucket creation can still spam ListBuckets, it will massively reduce the number of calls to that API whenever buckets are modified. Thus a major win.

@mikegrima mikegrima self-assigned this Jan 12, 2018
@mikegrima mikegrima added the bug label Jan 12, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

1 participant