Improves daemon <-> pubsub memory footprint when many objects #400
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
cgalibern
commented
Aug 23, 2023
•
cgalibern
commented
- improves daemon memory footprint (many objects)
- prevents daemondata <-> pubsub hang during event storms
Rename SubscriptionError.Error error to Errs string
log warn when free size is < 1/4 of full size, publish SubscriptionQueueThreshold with level=warn log error when free size is < 1/2 of full size, publish SubscriptionQueueThreshold with level=info
main daemon components needs dedicated subscription queue size values to avoid hang during
activity storm (boot, peer lost, peer join...) because they are "slow" or they filter many events.
This commit also defines dedicated queue sizes as public values to allow updates from package consumers.
With this commit it will be possible to reduce the default pubsub subscription queue size to consume less memory for
<path> dedicated consumers such as imon, omon, icfg...
New defaults:
daemondata: SubscriptionQueueSize = 40000
discover.cfg: SubscriptionQueueSizeCfg = 30000
discover.omon: SubscriptionQueueSizeOmon = 16000
dns: SubscriptionQueueSize = 1000
istats: SubscriptionQueueSize = 1000
scheduler: SubscriptionQueueSize = 16000
…cription queue size since huge pubsub subscription queue consumers have dedicated value, we can now reduce the default bus subscription queue size from 4000 -> 200. Impact example for 500 simple objects on 3 node cluster: 300Mo -> 100Mo
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.