Feature/bulk redis payload processing #368
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ability to process redis payloads in bulk to reduce the number of requeries that large updates on a server can cause
Findings in production
Bulk updates that send many redis payloads to our meteor servers causes expensive database requeries for limit-sort publications and can spike cpu for other publications which attempt to process these payloads one at a time.
Goal
Reduce the large number of database lookups and requeries that happen when a large number of redis payloads are processed in bulk
Instead of doing document lookups and requeries one by one as the are needed, we process events in bulk, perform these database lookups in bulk and store them in a temporary
documentMap
and then iterate over the payloads triggering only a single requery when necessary.This also gives us the ability to set
maxRedisEventsToProcess
threshold which prevents us from spiking server cpu for extended periods of time in the case where we have received too many redis payloads to process effectively.Test Fixes
Looks like mocha tests had started failing due to some updates to the underlying mocha libraries in
meteortesting:mocha-core
. I pinned down the versions so that the issues with the test suite are resolved without needing to modify all the tests to no longer make use of async and done