Update index settings during remote store migration #12748

ltaragi · 2024-03-19T05:12:39Z

Description

For MIXED compatibility mode and REMOTE_STORE migration direction:
- New indices should be remote store backed
- New indices restored from a snapshot must be remote store backed irrespective of whether the original index was remote store backed or not
- This is accomplished as follows:
  - updateRemoteStoreSettings looks for the MIGRATION_DIRECTION_SETTING and REMOTE_STORE_COMPATIBILITY_MODE_SETTING inside the clusterSettings object
  - Upon finding REMOTE_STORE and MIXED values respectively, it obtains the REMOTE_STORE_TRANSLOG_REPOSITORY_NAME and REMOTE_STORE_SEGMENT_REPOSITORY_NAME from any of the remote nodes in the mixed cluster. Information of nodes is obtained from the clusterState object.
  - Upon finding both repositories, it marks the SETTING_REMOTE_STORE_ENABLED as true for the index settings.

Related Issues

Resolves [Remote Store] Update remote store index settings for mixed cluster #12852

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
~~Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)~~
Commits are signed per the DCO using --signoff
~~Commit changes are listed out in CHANGELOG.md file (See: Changelog)~~
~~Public documentation issue/PR created~~
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-03-19T05:22:30Z

❌ Gradle check result for c19e881: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-03-19T05:34:30Z

Compatibility status:

Checks if related components are compatible with change 53c1989

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/performance-analyzer.git]

github-actions · 2024-03-19T07:33:52Z

❌ Gradle check result for 501b052: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-03-19T08:24:40Z

❌ Gradle check result for e605b95: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

linuxpi · 2024-03-22T05:07:52Z

@ltaragi can you please add more details to the description and link the related issue

...nternalClusterTest/java/org/opensearch/remotemigration/RemoteStoreMigrationAllocationIT.java

.../java/org/opensearch/action/admin/cluster/settings/TransportClusterUpdateSettingsAction.java

server/src/main/java/org/opensearch/cluster/metadata/MetadataCreateIndexService.java

server/src/main/java/org/opensearch/snapshots/RestoreService.java

linuxpi · 2024-03-22T09:38:31Z

if the PR is ready, lets publish it. its currently in draft

.../java/org/opensearch/action/admin/cluster/settings/TransportClusterUpdateSettingsAction.java

server/src/main/java/org/opensearch/cluster/metadata/MetadataCreateIndexService.java

server/src/main/java/org/opensearch/snapshots/RestoreService.java

github-actions · 2024-03-23T12:25:22Z

❌ Gradle check result for 51cc9df: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-04-01T12:08:52Z

❌ Gradle check result for 5e3b82a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-04-01T16:42:22Z

✅ Gradle check result for 4c23b6b: SUCCESS

codecov · 2024-04-01T16:44:51Z

Codecov Report

Attention: Patch coverage is 88.09524% with 5 lines in your changes are missing coverage. Please review.

Project coverage is 71.45%. Comparing base (b15cb0c) to head (53c1989).
Report is 156 commits behind head on main.

Files	Patch %	Lines
...h/cluster/metadata/MetadataCreateIndexService.java	93.93%	0 Missing and 2 partials ⚠️
.../java/org/opensearch/snapshots/RestoreService.java	66.66%	0 Missing and 2 partials ⚠️
...earch/node/remotestore/RemoteStoreNodeService.java	66.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #12748      +/-   ##
============================================
+ Coverage     71.42%   71.45%   +0.03%     
- Complexity    59978    60556     +578     
============================================
  Files          4985     5033      +48     
  Lines        282275   284972    +2697     
  Branches      40946    41299     +353     
============================================
+ Hits         201603   203636    +2033     
- Misses        63999    64511     +512     
- Partials      16673    16825     +152

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-04-01T18:44:54Z

✅ Gradle check result for 8e55204: SUCCESS

Signed-off-by: Lakshya Taragi <[email protected]>

github-actions · 2024-04-12T05:28:33Z

✅ Gradle check result for fe8d1b5: SUCCESS

Signed-off-by: Lakshya Taragi <[email protected]>

...src/internalClusterTest/java/org/opensearch/remotemigration/ShardAllocationBaseTestCase.java

server/src/test/java/org/opensearch/cluster/metadata/MetadataCreateIndexServiceTests.java

linuxpi

LGTM, some minor changes in UTs

github-actions · 2024-04-12T10:46:33Z

❌ Gradle check result for 51c5705: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/snapshots/RestoreService.java

gbbafna

Good to see that docrep direction is also potentially handled in this PR itself . Thanks Lakshya and Varun.

github-actions · 2024-04-12T11:48:24Z

❌ Gradle check result for 2931a1c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Lakshya Taragi <[email protected]>

github-actions · 2024-04-12T12:44:25Z

❌ Gradle check result for b0549e5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-04-12T13:20:51Z

❕ Gradle check result for 0028922: UNSTABLE

TEST FAILURES:

      1 org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2024-04-12T13:28:18Z

❕ Gradle check result for 53c1989: UNSTABLE

TEST FAILURES:

      1 org.opensearch.remotestore.RemoteStoreStatsIT.testDownloadStatsCorrectnessSinglePrimaryMultipleReplicaShards
      1 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

server/src/main/java/org/opensearch/cluster/metadata/MetadataCreateIndexService.java

opensearch-trigger-bot · 2024-04-12T14:17:34Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-12748-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c168e1c9216ba914fcd5c310ffb070df8ef5e6b5
# Push it to GitHub
git push --set-upstream origin backport/backport-12748-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-12748-to-2.x.

…ct#12748) Signed-off-by: Lakshya Taragi <[email protected]> (cherry picked from commit c168e1c)

Signed-off-by: Lakshya Taragi <[email protected]> (cherry picked from commit c168e1c)

harishbhakuni · 2024-05-08T21:36:54Z

server/src/main/java/org/opensearch/snapshots/RestoreService.java

                        if (RemoteStoreNodeAttribute.isRemoteStoreAttributePresent(clusterService.getSettings()) == true
-                            || clusterSettings.get(IndicesService.CLUSTER_INDEX_RESTRICT_REPLICATION_TYPE_SETTING) == true) {


this logic might not work well during remote store -> docrep migration. so, this code runs in cluster manager node, if during migration from remote store -> docrep, user updates setting of one of the non cluster manager node first and if cluster manager still have the remote store attributes this will restore the indices as remote store enabled.

we should maybe check (if mode is STRICT and remote store attribute is present) || (migration to remote is ongoing).

ltaragi force-pushed the remote-store-migration-settings branch 2 times, most recently from 501b052 to e605b95 Compare March 19, 2024 07:33

linuxpi reviewed Mar 22, 2024

View reviewed changes

server/src/main/java/org/opensearch/snapshots/RestoreService.java Outdated Show resolved Hide resolved

github-actions bot added enhancement Enhancement or improvement to existing feature or request Storage:Durability Issues and PRs related to the durability framework Storage:Remote labels Mar 23, 2024

ltaragi force-pushed the remote-store-migration-settings branch from 51cc9df to 5e3b82a Compare April 1, 2024 10:32

ltaragi added the skip-changelog label Apr 1, 2024

ltaragi removed enhancement Enhancement or improvement to existing feature or request Storage:Durability Issues and PRs related to the durability framework Storage:Remote labels Apr 1, 2024

ltaragi force-pushed the remote-store-migration-settings branch from 4c23b6b to 8e55204 Compare April 1, 2024 17:58

github-actions bot added enhancement Enhancement or improvement to existing feature or request Storage:Durability Issues and PRs related to the durability framework Storage:Remote labels Apr 1, 2024

ltaragi force-pushed the remote-store-migration-settings branch from 8e55204 to c0aca30 Compare April 1, 2024 18:29

Address comments

fe8d1b5

Signed-off-by: Lakshya Taragi <[email protected]>

ltaragi force-pushed the remote-store-migration-settings branch from b1c049e to fe8d1b5 Compare April 12, 2024 04:39

Ensure remote node and strict mode for remote cluster

51c5705

Signed-off-by: Lakshya Taragi <[email protected]>

linuxpi reviewed Apr 12, 2024

View reviewed changes

...src/internalClusterTest/java/org/opensearch/remotemigration/ShardAllocationBaseTestCase.java Outdated Show resolved Hide resolved

linuxpi reviewed Apr 12, 2024

View reviewed changes

server/src/test/java/org/opensearch/cluster/metadata/MetadataCreateIndexServiceTests.java Outdated Show resolved Hide resolved

linuxpi reviewed Apr 12, 2024

View reviewed changes

server/src/test/java/org/opensearch/cluster/metadata/MetadataCreateIndexServiceTests.java Outdated Show resolved Hide resolved

linuxpi approved these changes Apr 12, 2024

View reviewed changes

gbbafna reviewed Apr 12, 2024

View reviewed changes

server/src/main/java/org/opensearch/snapshots/RestoreService.java Show resolved Hide resolved

gbbafna approved these changes Apr 12, 2024

View reviewed changes

Address comments

b0549e5

Signed-off-by: Lakshya Taragi <[email protected]>

ltaragi force-pushed the remote-store-migration-settings branch from 2931a1c to b0549e5 Compare April 12, 2024 12:01

Fix incompatibility with IT

53c1989

Signed-off-by: Lakshya Taragi <[email protected]>

ltaragi force-pushed the remote-store-migration-settings branch from 0028922 to 53c1989 Compare April 12, 2024 12:39

gbbafna reviewed Apr 12, 2024

View reviewed changes

server/src/main/java/org/opensearch/cluster/metadata/MetadataCreateIndexService.java Show resolved Hide resolved

gbbafna merged commit c168e1c into opensearch-project:main Apr 12, 2024
31 checks passed

gbbafna added the backport 2.x Backport to 2.x branch label Apr 12, 2024

opensearch-trigger-bot bot added the backport-failed label Apr 12, 2024

This was referenced Apr 12, 2024

Add validation while updating CompatibilityMode setting #13080

Merged

[Backport 2.x] Update index settings during remote store migration (#12748) #13186

Merged

ltaragi added a commit to ltaragi/OpenSearch that referenced this pull request Apr 15, 2024

Update index settings during remote store migration (opensearch-proje…

5aa8c7d

…ct#12748) Signed-off-by: Lakshya Taragi <[email protected]> (cherry picked from commit c168e1c)

gbbafna pushed a commit that referenced this pull request Apr 15, 2024

Update index settings during remote store migration (#12748) (#13186)

87de4c2

Signed-off-by: Lakshya Taragi <[email protected]> (cherry picked from commit c168e1c)

harishbhakuni reviewed May 8, 2024

View reviewed changes

neetikasinghal mentioned this pull request Jul 18, 2024

Add rest, transport layer changes for Tiering #13980

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update index settings during remote store migration #12748

Update index settings during remote store migration #12748

ltaragi commented Mar 19, 2024 •

edited

Loading

github-actions bot commented Mar 19, 2024

github-actions bot commented Mar 19, 2024 •

edited

Loading

github-actions bot commented Mar 19, 2024

github-actions bot commented Mar 19, 2024

linuxpi commented Mar 22, 2024

linuxpi commented Mar 22, 2024

github-actions bot commented Mar 23, 2024

github-actions bot commented Apr 1, 2024

github-actions bot commented Apr 1, 2024

codecov bot commented Apr 1, 2024 •

edited

Loading

github-actions bot commented Apr 1, 2024

github-actions bot commented Apr 12, 2024

linuxpi left a comment

github-actions bot commented Apr 12, 2024

gbbafna left a comment

github-actions bot commented Apr 12, 2024

github-actions bot commented Apr 12, 2024

github-actions bot commented Apr 12, 2024

github-actions bot commented Apr 12, 2024

opensearch-trigger-bot bot commented Apr 12, 2024

harishbhakuni May 8, 2024

harishbhakuni May 8, 2024

		if (RemoteStoreNodeAttribute.isRemoteStoreAttributePresent(clusterService.getSettings()) == true
		\|\| clusterSettings.get(IndicesService.CLUSTER_INDEX_RESTRICT_REPLICATION_TYPE_SETTING) == true) {

Update index settings during remote store migration #12748

Update index settings during remote store migration #12748

Conversation

ltaragi commented Mar 19, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Mar 19, 2024

github-actions bot commented Mar 19, 2024 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Mar 19, 2024

github-actions bot commented Mar 19, 2024

linuxpi commented Mar 22, 2024

linuxpi commented Mar 22, 2024

github-actions bot commented Mar 23, 2024

github-actions bot commented Apr 1, 2024

github-actions bot commented Apr 1, 2024

codecov bot commented Apr 1, 2024 • edited Loading

Codecov Report

github-actions bot commented Apr 1, 2024

github-actions bot commented Apr 12, 2024

linuxpi left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 12, 2024

gbbafna left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 12, 2024

github-actions bot commented Apr 12, 2024

github-actions bot commented Apr 12, 2024

github-actions bot commented Apr 12, 2024

opensearch-trigger-bot bot commented Apr 12, 2024

harishbhakuni May 8, 2024

Choose a reason for hiding this comment

harishbhakuni May 8, 2024

Choose a reason for hiding this comment

ltaragi commented Mar 19, 2024 •

edited

Loading

github-actions bot commented Mar 19, 2024 •

edited

Loading

codecov bot commented Apr 1, 2024 •

edited

Loading