Adding QueryGroup schema #13669

kaushalmahi12 · 2024-05-14T19:45:30Z

Description

This change introduces the fundamental construct which we will use to enforce node level resiliency as part of this RFC: #12342.

Related Issues

RFC

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
API changes companion pull request created.
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-05-14T20:45:14Z

❌ Gradle check result for c7e56f4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/cluster/metadata/ResourceLimitGroup.java

server/src/main/java/org/opensearch/cluster/metadata/ResourceLimitGroupMetadata.java

github-actions · 2024-05-15T22:07:41Z

❌ Gradle check result for bd1f864: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-05-20T21:22:15Z

❌ Gradle check result for 9119813: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/cluster/metadata/ResourceLimitGroup.java

github-actions · 2024-06-03T18:51:57Z

❌ Gradle check result for 9e11acf: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-06-13T16:57:22Z

❌ Gradle check result for 1b0e331: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/cluster/metadata/QueryGroupMetadata.java

Signed-off-by: Kaushal Kumar <[email protected]>

github-actions · 2024-07-02T19:13:13Z

✅ Gradle check result for 5a7f405: SUCCESS

github-actions · 2024-07-02T19:19:52Z

❌ Gradle check result for c5a70e9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

jainankitk · 2024-07-02T19:40:14Z

@kaushalmahi12 - Can you run ./gradlew :server:spotlessApply locally and push to fix the checkstyle issues?

Signed-off-by: Kaushal Kumar <[email protected]>

github-actions · 2024-07-02T21:21:30Z

✅ Gradle check result for 42d3494: SUCCESS

codecov · 2024-07-02T21:23:11Z

Codecov Report

Attention: Patch coverage is 79.36508% with 39 lines in your changes missing coverage. Please review.

Project coverage is 71.88%. Comparing base (f1f4f89) to head (8a45648).
Report is 3 commits behind head on main.

Files	Patch %	Lines
...va/org/opensearch/cluster/metadata/QueryGroup.java	85.24%	8 Missing and 10 partials ⚠️
...java/org/opensearch/cluster/metadata/Metadata.java	0.00%	10 Missing ⚠️
...pensearch/cluster/metadata/QueryGroupMetadata.java	80.39%	8 Missing and 2 partials ⚠️
...search/backpressure/SearchBackpressureService.java	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #13669      +/-   ##
============================================
+ Coverage     71.69%   71.88%   +0.19%     
- Complexity    62292    62490     +198     
============================================
  Files          5140     5142       +2     
  Lines        293020   293206     +186     
  Branches      42347    42377      +30     
============================================
+ Hits         210090   210781     +691     
+ Misses        65614    65094     -520     
- Partials      17316    17331      +15

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

server/src/main/java/org/opensearch/search/ResourceType.java

server/src/main/java/org/opensearch/cluster/metadata/QueryGroup.java

Signed-off-by: Kaushal Kumar <[email protected]>

github-actions · 2024-07-04T00:00:34Z

✅ Gradle check result for 8a45648: SUCCESS

server/src/main/java/org/opensearch/search/ResourceType.java

opensearch-trigger-bot · 2024-07-08T21:40:26Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-13669-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0684342b9e3711118614f38fb0648d8e6428477e
# Push it to GitHub
git push --set-upstream origin backport/backport-13669-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-13669-to-2.x.

* rebase with opensearch/main Signed-off-by: Kaushal Kumar <[email protected]> * add resourceLimitGroupId propagation logic from coordinator to data nodes Signed-off-by: Kaushal Kumar <[email protected]> * add sandbox schema Signed-off-by: Kaushal Kumar <[email protected]> * add resourceLimitGroupTests Signed-off-by: Kaushal Kumar <[email protected]> * add resourceLimitGroupMetadata tests Signed-off-by: Kaushal Kumar <[email protected]> * run spotlessApply Signed-off-by: Kaushal Kumar <[email protected]> * add mode field in ResourceLimitGroup schema Signed-off-by: Kaushal Kumar <[email protected]> * fix breaking testcases Signed-off-by: Kaushal Kumar <[email protected]> * add task cancellation skeleton Signed-off-by: Kaushal Kumar <[email protected]> * add multitenant labels in searchSource builder Signed-off-by: Kaushal Kumar <[email protected]> * write custom xcontent parser for ResourceLimitGroup Signed-off-by: Kaushal Kumar <[email protected]> * remove unrelated changes Signed-off-by: Kaushal Kumar <[email protected]> * remove non-existing import fro cluster settings Signed-off-by: Kaushal Kumar <[email protected]> * remove non releated changes Signed-off-by: Kaushal Kumar <[email protected]> * add _id as the resourceLimitGroup key Signed-off-by: Kaushal Kumar <[email protected]> * add change to register resource limit group metadata Signed-off-by: Kaushal Kumar <[email protected]> * add updatedAt in resource limit group Signed-off-by: Kaushal Kumar <[email protected]> * rename resourceLimitGroup to queryGroup Signed-off-by: Kaushal Kumar <[email protected]> * address the comments on PR Signed-off-by: Kaushal Kumar <[email protected]> * rename the mode member var to resiliency mode Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> * add change in CHANGELOG Signed-off-by: Kaushal Kumar <[email protected]> * add tests for custom namedWritable QueryGroupMetadata Signed-off-by: Kaushal Kumar <[email protected]> * structure resourceLimits into an object Signed-off-by: Kaushal Kumar <[email protected]> * add QueryGroup.toXContent test case Signed-off-by: Kaushal Kumar <[email protected]> * fix precommit errors Signed-off-by: Kaushal Kumar <[email protected]> * fix precommit errors Signed-off-by: Kaushal Kumar <[email protected]> * fix assemble errors Signed-off-by: Kaushal Kumar <[email protected]> * fix checkstyle errors Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> --------- Signed-off-by: Kaushal Kumar <[email protected]>

* Adding QueryGroup schema (#13669) * rebase with opensearch/main Signed-off-by: Kaushal Kumar <[email protected]> * add resourceLimitGroupId propagation logic from coordinator to data nodes Signed-off-by: Kaushal Kumar <[email protected]> * add sandbox schema Signed-off-by: Kaushal Kumar <[email protected]> * add resourceLimitGroupTests Signed-off-by: Kaushal Kumar <[email protected]> * add resourceLimitGroupMetadata tests Signed-off-by: Kaushal Kumar <[email protected]> * run spotlessApply Signed-off-by: Kaushal Kumar <[email protected]> * add mode field in ResourceLimitGroup schema Signed-off-by: Kaushal Kumar <[email protected]> * fix breaking testcases Signed-off-by: Kaushal Kumar <[email protected]> * add task cancellation skeleton Signed-off-by: Kaushal Kumar <[email protected]> * add multitenant labels in searchSource builder Signed-off-by: Kaushal Kumar <[email protected]> * write custom xcontent parser for ResourceLimitGroup Signed-off-by: Kaushal Kumar <[email protected]> * remove unrelated changes Signed-off-by: Kaushal Kumar <[email protected]> * remove non-existing import fro cluster settings Signed-off-by: Kaushal Kumar <[email protected]> * remove non releated changes Signed-off-by: Kaushal Kumar <[email protected]> * add _id as the resourceLimitGroup key Signed-off-by: Kaushal Kumar <[email protected]> * add change to register resource limit group metadata Signed-off-by: Kaushal Kumar <[email protected]> * add updatedAt in resource limit group Signed-off-by: Kaushal Kumar <[email protected]> * rename resourceLimitGroup to queryGroup Signed-off-by: Kaushal Kumar <[email protected]> * address the comments on PR Signed-off-by: Kaushal Kumar <[email protected]> * rename the mode member var to resiliency mode Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> * add change in CHANGELOG Signed-off-by: Kaushal Kumar <[email protected]> * add tests for custom namedWritable QueryGroupMetadata Signed-off-by: Kaushal Kumar <[email protected]> * structure resourceLimits into an object Signed-off-by: Kaushal Kumar <[email protected]> * add QueryGroup.toXContent test case Signed-off-by: Kaushal Kumar <[email protected]> * fix precommit errors Signed-off-by: Kaushal Kumar <[email protected]> * fix precommit errors Signed-off-by: Kaushal Kumar <[email protected]> * fix assemble errors Signed-off-by: Kaushal Kumar <[email protected]> * fix checkstyle errors Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> --------- Signed-off-by: Kaushal Kumar <[email protected]> * add min supported version in query group metadata Signed-off-by: Kaushal Kumar <[email protected]> --------- Signed-off-by: Kaushal Kumar <[email protected]>

* rebase with opensearch/main Signed-off-by: Kaushal Kumar <[email protected]> * add resourceLimitGroupId propagation logic from coordinator to data nodes Signed-off-by: Kaushal Kumar <[email protected]> * add sandbox schema Signed-off-by: Kaushal Kumar <[email protected]> * add resourceLimitGroupTests Signed-off-by: Kaushal Kumar <[email protected]> * add resourceLimitGroupMetadata tests Signed-off-by: Kaushal Kumar <[email protected]> * run spotlessApply Signed-off-by: Kaushal Kumar <[email protected]> * add mode field in ResourceLimitGroup schema Signed-off-by: Kaushal Kumar <[email protected]> * fix breaking testcases Signed-off-by: Kaushal Kumar <[email protected]> * add task cancellation skeleton Signed-off-by: Kaushal Kumar <[email protected]> * add multitenant labels in searchSource builder Signed-off-by: Kaushal Kumar <[email protected]> * write custom xcontent parser for ResourceLimitGroup Signed-off-by: Kaushal Kumar <[email protected]> * remove unrelated changes Signed-off-by: Kaushal Kumar <[email protected]> * remove non-existing import fro cluster settings Signed-off-by: Kaushal Kumar <[email protected]> * remove non releated changes Signed-off-by: Kaushal Kumar <[email protected]> * add _id as the resourceLimitGroup key Signed-off-by: Kaushal Kumar <[email protected]> * add change to register resource limit group metadata Signed-off-by: Kaushal Kumar <[email protected]> * add updatedAt in resource limit group Signed-off-by: Kaushal Kumar <[email protected]> * rename resourceLimitGroup to queryGroup Signed-off-by: Kaushal Kumar <[email protected]> * address the comments on PR Signed-off-by: Kaushal Kumar <[email protected]> * rename the mode member var to resiliency mode Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> * add change in CHANGELOG Signed-off-by: Kaushal Kumar <[email protected]> * add tests for custom namedWritable QueryGroupMetadata Signed-off-by: Kaushal Kumar <[email protected]> * structure resourceLimits into an object Signed-off-by: Kaushal Kumar <[email protected]> * add QueryGroup.toXContent test case Signed-off-by: Kaushal Kumar <[email protected]> * fix precommit errors Signed-off-by: Kaushal Kumar <[email protected]> * fix precommit errors Signed-off-by: Kaushal Kumar <[email protected]> * fix assemble errors Signed-off-by: Kaushal Kumar <[email protected]> * fix checkstyle errors Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> --------- Signed-off-by: Kaushal Kumar <[email protected]>

…pensearch-project#14690) * Adding QueryGroup schema (opensearch-project#13669) * rebase with opensearch/main Signed-off-by: Kaushal Kumar <[email protected]> * add resourceLimitGroupId propagation logic from coordinator to data nodes Signed-off-by: Kaushal Kumar <[email protected]> * add sandbox schema Signed-off-by: Kaushal Kumar <[email protected]> * add resourceLimitGroupTests Signed-off-by: Kaushal Kumar <[email protected]> * add resourceLimitGroupMetadata tests Signed-off-by: Kaushal Kumar <[email protected]> * run spotlessApply Signed-off-by: Kaushal Kumar <[email protected]> * add mode field in ResourceLimitGroup schema Signed-off-by: Kaushal Kumar <[email protected]> * fix breaking testcases Signed-off-by: Kaushal Kumar <[email protected]> * add task cancellation skeleton Signed-off-by: Kaushal Kumar <[email protected]> * add multitenant labels in searchSource builder Signed-off-by: Kaushal Kumar <[email protected]> * write custom xcontent parser for ResourceLimitGroup Signed-off-by: Kaushal Kumar <[email protected]> * remove unrelated changes Signed-off-by: Kaushal Kumar <[email protected]> * remove non-existing import fro cluster settings Signed-off-by: Kaushal Kumar <[email protected]> * remove non releated changes Signed-off-by: Kaushal Kumar <[email protected]> * add _id as the resourceLimitGroup key Signed-off-by: Kaushal Kumar <[email protected]> * add change to register resource limit group metadata Signed-off-by: Kaushal Kumar <[email protected]> * add updatedAt in resource limit group Signed-off-by: Kaushal Kumar <[email protected]> * rename resourceLimitGroup to queryGroup Signed-off-by: Kaushal Kumar <[email protected]> * address the comments on PR Signed-off-by: Kaushal Kumar <[email protected]> * rename the mode member var to resiliency mode Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> * add change in CHANGELOG Signed-off-by: Kaushal Kumar <[email protected]> * add tests for custom namedWritable QueryGroupMetadata Signed-off-by: Kaushal Kumar <[email protected]> * structure resourceLimits into an object Signed-off-by: Kaushal Kumar <[email protected]> * add QueryGroup.toXContent test case Signed-off-by: Kaushal Kumar <[email protected]> * fix precommit errors Signed-off-by: Kaushal Kumar <[email protected]> * fix precommit errors Signed-off-by: Kaushal Kumar <[email protected]> * fix assemble errors Signed-off-by: Kaushal Kumar <[email protected]> * fix checkstyle errors Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> --------- Signed-off-by: Kaushal Kumar <[email protected]> * add min supported version in query group metadata Signed-off-by: Kaushal Kumar <[email protected]> --------- Signed-off-by: Kaushal Kumar <[email protected]> Signed-off-by: kkewwei <[email protected]>

* rebase with opensearch/main Signed-off-by: Kaushal Kumar <[email protected]> * add resourceLimitGroupId propagation logic from coordinator to data nodes Signed-off-by: Kaushal Kumar <[email protected]> * add sandbox schema Signed-off-by: Kaushal Kumar <[email protected]> * add resourceLimitGroupTests Signed-off-by: Kaushal Kumar <[email protected]> * add resourceLimitGroupMetadata tests Signed-off-by: Kaushal Kumar <[email protected]> * run spotlessApply Signed-off-by: Kaushal Kumar <[email protected]> * add mode field in ResourceLimitGroup schema Signed-off-by: Kaushal Kumar <[email protected]> * fix breaking testcases Signed-off-by: Kaushal Kumar <[email protected]> * add task cancellation skeleton Signed-off-by: Kaushal Kumar <[email protected]> * add multitenant labels in searchSource builder Signed-off-by: Kaushal Kumar <[email protected]> * write custom xcontent parser for ResourceLimitGroup Signed-off-by: Kaushal Kumar <[email protected]> * remove unrelated changes Signed-off-by: Kaushal Kumar <[email protected]> * remove non-existing import fro cluster settings Signed-off-by: Kaushal Kumar <[email protected]> * remove non releated changes Signed-off-by: Kaushal Kumar <[email protected]> * add _id as the resourceLimitGroup key Signed-off-by: Kaushal Kumar <[email protected]> * add change to register resource limit group metadata Signed-off-by: Kaushal Kumar <[email protected]> * add updatedAt in resource limit group Signed-off-by: Kaushal Kumar <[email protected]> * rename resourceLimitGroup to queryGroup Signed-off-by: Kaushal Kumar <[email protected]> * address the comments on PR Signed-off-by: Kaushal Kumar <[email protected]> * rename the mode member var to resiliency mode Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> * add change in CHANGELOG Signed-off-by: Kaushal Kumar <[email protected]> * add tests for custom namedWritable QueryGroupMetadata Signed-off-by: Kaushal Kumar <[email protected]> * structure resourceLimits into an object Signed-off-by: Kaushal Kumar <[email protected]> * add QueryGroup.toXContent test case Signed-off-by: Kaushal Kumar <[email protected]> * fix precommit errors Signed-off-by: Kaushal Kumar <[email protected]> * fix precommit errors Signed-off-by: Kaushal Kumar <[email protected]> * fix assemble errors Signed-off-by: Kaushal Kumar <[email protected]> * fix checkstyle errors Signed-off-by: Kaushal Kumar <[email protected]> * address comments Signed-off-by: Kaushal Kumar <[email protected]> --------- Signed-off-by: Kaushal Kumar <[email protected]>

kaushalmahi12 requested review from anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89, gbbafna, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, reta, Rishikesh1159, sachinpkale, saratvemulapalli, shwetathareja, sohami, tlfeng and VachaShah as code owners May 14, 2024 19:45

jainankitk reviewed May 15, 2024

View reviewed changes

server/src/main/java/org/opensearch/cluster/metadata/ResourceLimitGroup.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/cluster/metadata/ResourceLimitGroup.java Outdated Show resolved Hide resolved

deshsidd reviewed May 15, 2024

View reviewed changes

sgup432 reviewed May 28, 2024

View reviewed changes

server/src/main/java/org/opensearch/cluster/metadata/ResourceLimitGroup.java Outdated Show resolved Hide resolved

ruai0511 reviewed Jun 13, 2024

View reviewed changes

server/src/main/java/org/opensearch/cluster/metadata/QueryGroupMetadata.java Show resolved Hide resolved

fix assemble errors

c5a70e9

Signed-off-by: Kaushal Kumar <[email protected]>

fix checkstyle errors

42d3494

Signed-off-by: Kaushal Kumar <[email protected]>

mch2 reviewed Jul 3, 2024

View reviewed changes

address comments

8a45648

Signed-off-by: Kaushal Kumar <[email protected]>

mch2 approved these changes Jul 6, 2024

View reviewed changes

server/src/main/java/org/opensearch/search/ResourceType.java Show resolved Hide resolved

mch2 linked an issue Jul 8, 2024 that may be closed by this pull request

[Proposal] Writable Warm Status API Model #14640

Open

mch2 removed a link to an issue Jul 8, 2024

[Proposal] Writable Warm Status API Model #14640

Open

mch2 merged commit 0684342 into opensearch-project:main Jul 8, 2024
32 checks passed

opensearch-trigger-bot bot added the backport-failed label Jul 8, 2024

kaushalmahi12 deleted the feature/sandbox-schemaPR branch July 8, 2024 21:48

kaushalmahi12 mentioned this pull request Jul 9, 2024

[Backport 2.x] Adding QueryGroup schema (#13669) #14690

Merged

finnegancarroll mentioned this pull request Jul 15, 2024

Make remote snapshot (local)block size configurable #14753

Closed

3 tasks

kiranprakash154 assigned kaushalmahi12 Jul 16, 2024

jainankitk mentioned this pull request Jul 24, 2024

Add Create QueryGroup API Logic #14680

Merged

8 tasks

BrewTestBot mentioned this pull request Aug 8, 2024

opensearch 2.16.0 Homebrew/homebrew-core#180446

Merged

Bukhtawar mentioned this pull request Aug 19, 2024

Adding @jainankitk as a Maintainer #15304

Merged

opensearch-ci-bot mentioned this pull request Sep 11, 2024

[AUTOCUT] Gradle Check Flaky Test Report for RemoteStoreStatsIT #14310

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding QueryGroup schema #13669

Adding QueryGroup schema #13669

kaushalmahi12 commented May 14, 2024

github-actions bot commented May 14, 2024

github-actions bot commented May 15, 2024

github-actions bot commented May 20, 2024

github-actions bot commented Jun 3, 2024

github-actions bot commented Jun 13, 2024

github-actions bot commented Jul 2, 2024

github-actions bot commented Jul 2, 2024

jainankitk commented Jul 2, 2024

github-actions bot commented Jul 2, 2024

codecov bot commented Jul 2, 2024 •

edited

Loading

github-actions bot commented Jul 4, 2024

opensearch-trigger-bot bot commented Jul 8, 2024

Adding QueryGroup schema #13669

Adding QueryGroup schema #13669

Conversation

kaushalmahi12 commented May 14, 2024

Description

Related Issues

Check List

github-actions bot commented May 14, 2024

github-actions bot commented May 15, 2024

github-actions bot commented May 20, 2024

github-actions bot commented Jun 3, 2024

github-actions bot commented Jun 13, 2024

github-actions bot commented Jul 2, 2024

github-actions bot commented Jul 2, 2024

jainankitk commented Jul 2, 2024

github-actions bot commented Jul 2, 2024

codecov bot commented Jul 2, 2024 • edited Loading

Codecov Report

github-actions bot commented Jul 4, 2024

opensearch-trigger-bot bot commented Jul 8, 2024

codecov bot commented Jul 2, 2024 •

edited

Loading