fix: retrieve from memory when waiting for snapshots #5161
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/try |
|
Okay, starting a try! I'll update this comment once it's running...\n |
zacharyhamm
approved these changes
Dec 18, 2024
@zacharyhamm |
&str means you can pass a String as easily as a &str or an Arc, and means you don't have to do a ref count operation in the read loop |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The original implementation of these calls went directly to the memory cache, knowing that we were waiting for something new and it couldn't be on the disk yet. When foyer was added, this changed to go directly to the cache, meaning that we are knowingly spawning a thread that reads from disk with every attempt here, every millisecond, for up to 2000 milliseconds. This creates a TON of disk pressure and additional allocations. I think this might be part of the source of our memory pressure issues, but this really leads me to believe that there is a leak internal to foyer around how they load things from the disk (maybe specifically around misses?).
Another curious thing is that I don't seem to have any traces that point to the disk misses, of which there should be many thousand.
Not that this undoes what was done earlier to move us to a just-memory cache. I would like to see how this performs in tools prod overnight at least to see if it has a significant impact on the memory allocations.