Caching total number of primary shards per node for evaluating constraints on avg primary shards across all indices per node #14992
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…on avg primary shards across all indices per node Signed-off-by: RS146BIJAY <[email protected]>
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #14992 +/- ##
============================================
+ Coverage 71.78% 71.89% +0.10%
- Complexity 62694 62794 +100
============================================
Files 5160 5161 +1
Lines 294211 294366 +155
Branches 42553 42579 +26
============================================
+ Hits 211212 211635 +423
+ Misses 65599 65291 -308
- Partials 17400 17440 +40 ☔ View full report in Codecov by Sentry. |
RS146BIJAY
requested review from
anasalkouz,
andrross,
ashking94,
Bukhtawar,
CEHENKLE,
dblock,
dbwiddis,
gbbafna,
kotwanikunal,
mch2,
msfroh,
nknize,
owaiskazi19,
reta,
Rishikesh1159,
sachinpkale,
saratvemulapalli,
shwetathareja,
sohami and
VachaShah
as code owners
RS146BIJAY
changed the title
vikasvb90
approved these changes
Jul 30, 2024
Bukhtawar
approved these changes
Jul 31, 2024
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Jul 31, 2024
…on avg primary shards across all indices per node (#14992) Signed-off-by: RS146BIJAY <[email protected]> (cherry picked from commit 4707885) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
andrross
pushed a commit
that referenced
this pull request
Jul 31, 2024
…on avg primary shards across all indices per node (#14992) (#15045) (cherry picked from commit 4707885) Signed-off-by: RS146BIJAY <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
harshavamsi
pushed a commit
to harshavamsi/OpenSearch
that referenced
this pull request
Aug 20, 2024
…on avg primary shards across all indices per node (opensearch-project#14992) Signed-off-by: RS146BIJAY <[email protected]>
wdongyu
pushed a commit
to wdongyu/OpenSearch
that referenced
this pull request
Aug 22, 2024
…on avg primary shards across all indices per node (opensearch-project#14992) Signed-off-by: RS146BIJAY <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Allocation constraints specify conditions which, if breached, reduce the priority of a node for receiving unassigned shard allocations. When evaluating the cluster constraint on avg primary shards across all indices per node, we calculate total number of primary shards on a node by summing up the shard count of all indices on the node. Since this computation is performed for each node during shard allocation, it becomes computationally expensive. As there is only single thread on master node which handles all the operations including the deciders and make allocation decisions, allocation deciders execution may continue to block these threads which may prevent execution of certain high priority tasks like applying/sending cluster state update, index create, etc.
This PR does an offline calculation of total shards of all indices on a node and caches it so that AllocationConstraint does not needs to traverse all the indices metadata during constraint evaluation.
Issue Resolved
#15114
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.