-
-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a more aggressive GC watchdog to help in bursty memory usage situations #8798
Labels
kind/enhancement
A net-new feature or improvement to an existing feature
P2
Medium: Good to have, but can wait until someone steps up
Comments
Jorropo
added
the
kind/enhancement
A net-new feature or improvement to an existing feature
label
Mar 17, 2022
@Jorropo : cool, sounds good. We look forward to hearing what you learn from experimenting. |
3 tasks
3 tasks
Update: go implemented this somewhat recently however it's disabled by default. |
Jorropo
added a commit
to Jorropo/go-ipfs
that referenced
this issue
Dec 3, 2022
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Jorropo
added a commit
to Jorropo/go-ipfs
that referenced
this issue
Dec 3, 2022
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Jorropo
added a commit
to Jorropo/go-ipfs
that referenced
this issue
Dec 3, 2022
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Jorropo
added a commit
to Jorropo/go-ipfs
that referenced
this issue
Dec 3, 2022
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Jorropo
added a commit
to Jorropo/go-ipfs
that referenced
this issue
Dec 3, 2022
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Jorropo
added a commit
to Jorropo/go-ipfs
that referenced
this issue
Dec 5, 2022
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Jorropo
added a commit
to Jorropo/go-ipfs
that referenced
this issue
Mar 30, 2023
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Jorropo
added a commit
to Jorropo/go-ipfs
that referenced
this issue
Mar 30, 2023
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Jorropo
added a commit
to Jorropo/go-ipfs
that referenced
this issue
May 3, 2023
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Jorropo
added a commit
to Jorropo/go-ipfs
that referenced
this issue
May 4, 2023
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Jorropo
added a commit
to Jorropo/go-ipfs
that referenced
this issue
Jun 30, 2023
I have a rather big collection of profiles where someone claims that Kubo is ooming on XGiB. Then you open the profile and it is using half of that, this is due to the default GOGC=200%. That means, go will only run the GC once it's twice as being as the previous alive set. This situation happen more than it should / almost always because many parts of Kubo are memory garbage factories. Adding a GOMEMLIMIT helps by trading off more and more CPU running GC more often when memory is about to run out, it's not healthy to run at the edge of the limit because the GC will continously run killing performance. So this doesn't double the effective memory usable by Kubo, but we should expect to be able to use ~1.5x~1.75x before performance drastically falling off. Closes: ipfs#8798
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
kind/enhancement
A net-new feature or improvement to an existing feature
P2
Medium: Good to have, but can wait until someone steps up
Checklist
Description
In #8797 it happen often that bitswap use 20+Gigs of ram and will continue even after all requests has been canceled / finished.
However due to how golang GC's work, that lead to an accoutumance to high memory usage, and the GC wont get agressive to reclaim space.
(note the way the GC works, is that the GC tries to run when it is a factor X above the last GC size, so if your last size was 20Gig, golang might as well wait for 40gig of memory usage before gcing)
I belive lotus use https://github.com/raulk/go-watchdog to help in memory bursty conditions, maybe we should try it ? (I'm creating the issue to log any experiments)
The text was updated successfully, but these errors were encountered: