Reduce housekeeping pressure on CloudFormation #17
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dagguh
force-pushed
the
reduce-housekeeping-pressure-on-cloudformation
branch
2 times, most recently
from
a4ae713
to
f9fb678
Compare
When there's a lot of expired stacks (e.g. due to housekeeping outage), then listing them all requires a lot of Cloudformation requests. Then we get throttled, so AWS SDK makes a couple retries underneath, but we finally time out. Then we don't reach the stack deletion part. So the list remains long and it's a self-perpetuating problem. Instead, start deleting stacks as soon as we get a batch listed.
We haven't been throttled by EC2 like for CloudFormation, but do it for consistency with CloudFormation housekeeping.
Network stacks contain VPCs, and VPCs depend on security groups. Some security groups are provisioned outside of the stack, so deleting such a stack will fail due to the dependency. Delete stacks at the end, so that all external dependencies are already cleaned up.
Avoid false-positives like (notice lack of stacktrace logging too):
```
16:36:00,592 ERROR {} Ec2Instance(instanceId = i-080edd5933a74418f) failed to release itself
16:36:30,577 ERROR {} Ec2Instance(instanceId = i-061f9c1f94990848e) failed to release itself
```
When you hit the 5k key limit, starting 5k threads is ok for Java, but not ok for AWS. We get immediately throttled. It's actually faster to do it sequentially. And AWS is quick to delete.
dagguh
force-pushed
the
reduce-housekeeping-pressure-on-cloudformation
branch
from
2100663
to
b0952f3
Compare
mgrzaslewicz
approved these changes
Jan 4, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.