SAI-5162: Experimental downnode approach to use CoreContainer if available #232
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
As introduced in this fix, downnode calls will now discover replicas on a node via zk which is more reliable then
CoreContainer
, which might not have the full picture of the replicas if it's not loaded yet (during startup). This however, might impose some extra overhead for big cluster with alot of collections, especially when a large number of collections do NOT have actual replica on the target node, since those are "lazy" collection ref that has no watch which might require a full collection state fetch (exists, and fetch from ZK).A detailed breakdown of the overhead changes with such fix:
Node shutdown
Before fix : No ZK fetch as it gets list of replicas from CoreContainer which only contains replicas this node hosts. Therefore they are watched already (no lazy collection ref).
After fix: No ZK fetch on collections that have replica on this node since they are watched by the
ZkStateReader
, however other collections will trigger full ZK fetch on stateNode startup
Before fix: No ZK fetch, as the downnode has no effect due to the bug
After fix: Zk fetch on all collections since none of them is being watched yet (core was not loaded/registered to the container yet)
Solution
This is purely just an experiment to understand the scope of code change required to implement the approach to use core container if possible . We might not want such code complexity if the new overhead is not great enough to justify the change!
In theory, we can still use the
CoreContainer
approach for non-startup. Factored out a new methodgetReplicasPerCollectionOnThisNode
to achieve thatRemarks
Take note that this PR also slightly optimizes the existing flow of reading from ZK. Since the existing code actually does a full fetch once and then does another exists check (docCollection is obtained twice), while this PR fetches doCollection only once.
Though this likely makes little difference as it only saves an
exists
call, which is fast.