[DH-377] Increase ephemeral ports by default #6441
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
deployed to logodev.d.b.e and the updated port range has been applied! |
shaneknapp
changed the title
|
this took down all hubs that already had the other hubs were unaffected. ofc the hubs that went down are the ones that get the most usage. :) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
instead of manually patching proxy pods that run out of ephemeral ports in a terrible game of whack-a-mole, just give every hub's chp more ports by default.
it's a band-aid for jupyterhub/configurable-http-proxy#557, and while it doesn't fix the problem outright it definitely lessens the impact significantly.
instead of ~15-45m of "503 service unavailable" being presented to the users when we hit the default limit of 28231 ephemeral ports (60999 - 32768), we give ourselves 55000 ephemeral ports (65000 - 10000). this limits downtime to a few minutes of high hub response times, and possibly a few users seeing 503.
we haven't seen any impact on other services, and remain slightly optimistic of it's efficacy in smoothing out the road.
we've also given the worst impacted chp pods 3G of ram each, but could also consider scaling that down to 2.5G. under max load, when the port count is ~32000, the chp pod ram usage is typically between ~1.2-1.6G.
i will test this on logodev.d.b.e