[Segment Replication][Remote Store] Remove commits when remote store is enabled #8050

Poojita-Raj · 2023-06-13T23:30:07Z

Description

Skips performing commits on replica shards that will bump segment generation and segments_n file, which could potentially cause segment conflicts during failover cases when remote store is enabled.

Additional changes:

The lastCommittedSegmentInfos reference in NRTReplicationEngine is now updated since syncSegmentsFromRemoteSegmentStore directly calls store.commitSegmentInfos that doesn't take care of this.

Related Issues

Resolves #8027

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-06-14T00:01:31Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/17510/
CommitID: 8f7fb1d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-26T17:54:44Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/18354/
CommitID: a0cf2e0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-26T17:57:10Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/18362/
CommitID: 8120bb7
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-26T18:36:32Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/18368/
CommitID: 1aceeda
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/index/engine/NRTReplicationEngine.java

Signed-off-by: Poojita Raj <[email protected]>

github-actions · 2023-07-14T22:55:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.remotestore.RemoteStoreIT.testStaleCommitDeletionWithInvokeFlush

URL: https://build.ci.opensearch.org/job/gradle-check/20225/
CommitID: f43fc64
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Poojita-Raj · 2023-07-14T23:07:53Z

Flaky test #8658

server/src/main/java/org/opensearch/index/engine/NRTReplicationEngine.java

server/src/main/java/org/opensearch/index/shard/IndexShard.java

...src/test/java/org/opensearch/indices/recovery/RemoteStorePeerRecoverySourceHandlerTests.java

github-actions · 2023-07-15T01:17:41Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testBasicTaskResourceTracking

URL: https://build.ci.opensearch.org/job/gradle-check/20231/
CommitID: a40c6a5
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-07-17T01:08:27Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/20256/
CommitID: 40ffe8c
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/index/shard/IndexShard.java

github-actions · 2023-07-17T17:14:29Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/20303/
CommitID: 9666949

server/src/main/java/org/opensearch/index/engine/NRTReplicationEngine.java

server/src/main/java/org/opensearch/index/shard/IndexShard.java

server/src/test/java/org/opensearch/index/engine/NRTReplicationEngineTests.java

Signed-off-by: Poojita Raj <[email protected]>

github-actions · 2023-07-18T00:34:10Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication

URL: https://build.ci.opensearch.org/job/gradle-check/20348/
CommitID: 7eb5ddb
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

mch2

lgtm, thanks @Poojita-Raj

server/src/main/java/org/opensearch/index/engine/NRTReplicationEngine.java

…is enabled (#8050) * remove commits + fix failing test Signed-off-by: Poojita Raj <[email protected]> * fix failing tests Signed-off-by: Poojita Raj <[email protected]> * fix precommit failure Signed-off-by: Poojita Raj <[email protected]> * remove logs Signed-off-by: Poojita Raj <[email protected]> * address review comments Signed-off-by: Poojita Raj <[email protected]> --------- Signed-off-by: Poojita Raj <[email protected]> (cherry picked from commit 2f830be) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…is enabled (#8050) (#8753) * remove commits + fix failing test * fix failing tests * fix precommit failure * remove logs * address review comments --------- (cherry picked from commit 2f830be) Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…is enabled (opensearch-project#8050) * remove commits + fix failing test Signed-off-by: Poojita Raj <[email protected]> * fix failing tests Signed-off-by: Poojita Raj <[email protected]> * fix precommit failure Signed-off-by: Poojita Raj <[email protected]> * remove logs Signed-off-by: Poojita Raj <[email protected]> * address review comments Signed-off-by: Poojita Raj <[email protected]> --------- Signed-off-by: Poojita Raj <[email protected]>

…is enabled (opensearch-project#8050) * remove commits + fix failing test Signed-off-by: Poojita Raj <[email protected]> * fix failing tests Signed-off-by: Poojita Raj <[email protected]> * fix precommit failure Signed-off-by: Poojita Raj <[email protected]> * remove logs Signed-off-by: Poojita Raj <[email protected]> * address review comments Signed-off-by: Poojita Raj <[email protected]> --------- Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Kaushal Kumar <[email protected]>

…is enabled (opensearch-project#8050) * remove commits + fix failing test Signed-off-by: Poojita Raj <[email protected]> * fix failing tests Signed-off-by: Poojita Raj <[email protected]> * fix precommit failure Signed-off-by: Poojita Raj <[email protected]> * remove logs Signed-off-by: Poojita Raj <[email protected]> * address review comments Signed-off-by: Poojita Raj <[email protected]> --------- Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

Poojita-Raj added the skip-changelog label Jun 13, 2023

Poojita-Raj force-pushed the remove_commit branch from a0cf2e0 to 8120bb7 Compare June 26, 2023 17:39

Poojita-Raj force-pushed the remove_commit branch from 8120bb7 to 1aceeda Compare June 26, 2023 18:01

mch2 requested changes Jul 5, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/engine/NRTReplicationEngine.java Outdated Show resolved Hide resolved

remove logs

262949b

Signed-off-by: Poojita Raj <[email protected]>

Poojita-Raj force-pushed the remove_commit branch from 951392c to f43fc64 Compare July 14, 2023 22:17

dreamer-89 reviewed Jul 15, 2023

View reviewed changes

...src/test/java/org/opensearch/indices/recovery/RemoteStorePeerRecoverySourceHandlerTests.java Outdated Show resolved Hide resolved

sachinpkale reviewed Jul 17, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/shard/IndexShard.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/index/shard/IndexShard.java Outdated Show resolved Hide resolved

Poojita-Raj force-pushed the remove_commit branch from 40ffe8c to 9666949 Compare July 17, 2023 16:36

mch2 reviewed Jul 17, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/engine/NRTReplicationEngine.java Show resolved Hide resolved

dreamer-89 reviewed Jul 17, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/engine/NRTReplicationEngine.java Show resolved Hide resolved

server/src/main/java/org/opensearch/index/engine/NRTReplicationEngine.java Show resolved Hide resolved

server/src/main/java/org/opensearch/index/shard/IndexShard.java Show resolved Hide resolved

mch2 reviewed Jul 17, 2023

View reviewed changes

server/src/test/java/org/opensearch/index/engine/NRTReplicationEngineTests.java Outdated Show resolved Hide resolved

address review comments

7eb5ddb

Signed-off-by: Poojita Raj <[email protected]>

Poojita-Raj force-pushed the remove_commit branch from 9666949 to 7eb5ddb Compare July 18, 2023 00:00

mch2 approved these changes Jul 18, 2023

View reviewed changes

mch2 added the backport 2.x Backport to 2.x branch label Jul 18, 2023

dreamer-89 approved these changes Jul 18, 2023

View reviewed changes

dreamer-89 reviewed Jul 18, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/engine/NRTReplicationEngine.java Show resolved Hide resolved

mch2 merged commit 2f830be into opensearch-project:main Jul 18, 2023

opensearch-trigger-bot bot mentioned this pull request Jul 18, 2023

[Backport 2.x] [Segment Replication][Remote Store] Remove commits when remote store is enabled #8753

Merged

Poojita-Raj mentioned this pull request Jul 19, 2023

[BUG] Segment_N file cleanup not happening on replicas for indexes with segrep using remote store replication #8777

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Segment Replication][Remote Store] Remove commits when remote store is enabled #8050

[Segment Replication][Remote Store] Remove commits when remote store is enabled #8050

Poojita-Raj commented Jun 13, 2023 •

edited

Loading

github-actions bot commented Jun 14, 2023

github-actions bot commented Jun 26, 2023

github-actions bot commented Jun 26, 2023

github-actions bot commented Jun 26, 2023

github-actions bot commented Jul 14, 2023

Poojita-Raj commented Jul 14, 2023

github-actions bot commented Jul 15, 2023

github-actions bot commented Jul 17, 2023

github-actions bot commented Jul 17, 2023

github-actions bot commented Jul 18, 2023

mch2 left a comment

[Segment Replication][Remote Store] Remove commits when remote store is enabled #8050

[Segment Replication][Remote Store] Remove commits when remote store is enabled #8050

Conversation

Poojita-Raj commented Jun 13, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Jun 14, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 26, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 26, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 26, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 14, 2023

Gradle Check (Jenkins) Run Completed with:

Poojita-Raj commented Jul 14, 2023

github-actions bot commented Jul 15, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 17, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 17, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 18, 2023

Gradle Check (Jenkins) Run Completed with:

mch2 left a comment

Choose a reason for hiding this comment

Poojita-Raj commented Jun 13, 2023 •

edited

Loading