Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Segment Replication] Refactor RemoteStoreReplicationSource (#8767) #9076

Merged
merged 3 commits into from
Aug 3, 2023
Merged

[Segment Replication] Refactor RemoteStoreReplicationSource (#8767) #9076

merged 3 commits into from
Aug 3, 2023

Conversation

dreamer-89
Copy link
Member

@dreamer-89 dreamer-89 commented Aug 2, 2023 •
edited
Loading

Manual backport of #8767 to 2.x

Note: This needs to be merged in after as this PR contain changes on top of below two PRs. Keeping in draft until below are backported.

  1. [Remote Store] Add Segment download stats to remotestore stats API #8718
  2. Restrict user overrides for remote store related index settings #8812

@dreamer-89 dreamer-89 marked this pull request as draft August 2, 2023 18:21
@github-actions
Copy link
Contributor

github-actions bot commented Aug 2, 2023

Gradle Check (Jenkins) Run Completed with:

@dreamer-89
[Segment Replication] Refactor RemoteStoreReplicationSource (opensear…
61d9db4 
…ch-project#8767)

* [Segment Replication] Refactor remote replication source

Signed-off-by: Suraj Singh <[email protected]>

* Unit test updates

Signed-off-by: Suraj Singh <[email protected]>

* Self review

Signed-off-by: Suraj Singh <[email protected]>

* Self review

Signed-off-by: Suraj Singh <[email protected]>

* Segregate shard level tests for node to node and remote store segment replication

Signed-off-by: Suraj Singh <[email protected]>

* Fix failing unit tests

Signed-off-by: Suraj Singh <[email protected]>

* Fix failing UT

Signed-off-by: Suraj Singh <[email protected]>

* Fix failing UT

Signed-off-by: Suraj Singh <[email protected]>

* Address review comments

Signed-off-by: Suraj Singh <[email protected]>

* Fix more unit tests

Signed-off-by: Suraj Singh <[email protected]>

* Improve RemoteStoreReplicationSourceTests, remove unnecessary mocks and use actual failures for failure/exception use cases

Signed-off-by: Suraj Singh <[email protected]>

* Spotless check fix

Signed-off-by: Suraj Singh <[email protected]>

* Address review comments

Signed-off-by: Suraj Singh <[email protected]>

* Ignore files already in store while computing segment file diff with primary

Signed-off-by: Suraj Singh <[email protected]>

* Spotless fix

Signed-off-by: Suraj Singh <[email protected]>

* Fix failing UT

Signed-off-by: Suraj Singh <[email protected]>

* Spotless fix

Signed-off-by: Suraj Singh <[email protected]>

* Move read/writes from IndexInput/Output to RemoteSegmentMetadata

Signed-off-by: Suraj Singh <[email protected]>

* Address review commnt

Signed-off-by: Suraj Singh <[email protected]>

* Update recovery flow to perform commits during recovery

Signed-off-by: Suraj Singh <[email protected]>

* Remove un-necessary char

Signed-off-by: Suraj Singh <[email protected]>

* Address review comments

Signed-off-by: Suraj Singh <[email protected]>

* Update comment nit-pick

Signed-off-by: Suraj Singh <[email protected]>

* Remove deletion logic causing read issues due to deleted segments_N

Signed-off-by: Suraj Singh <[email protected]>

* Spotless fix

Signed-off-by: Suraj Singh <[email protected]>

* Fix unit tests

Signed-off-by: Suraj Singh <[email protected]>

---------

Signed-off-by: Suraj Singh <[email protected]>
@dreamer-89
Rebase 2.x and resolve conflicts
fbed447 
Signed-off-by: Suraj Singh <[email protected]>
@dreamer-89 dreamer-89 force-pushed the backport/backport-8767-to-2.x branch from 2e37b6b to fbed447 Compare August 3, 2023 17:10
@dreamer-89 dreamer-89 marked this pull request as ready for review August 3, 2023 17:11
@github-actions
Copy link
Contributor

github-actions bot commented Aug 3, 2023

Gradle Check (Jenkins) Run Completed with:

@dreamer-89
Fix unit test
0e9c7d7 
Signed-off-by: Suraj Singh <[email protected]>
@dreamer-89 dreamer-89 mentioned this pull request Aug 3, 2023
Merged
6 tasks
@opensearch-trigger-bot
Copy link
Contributor

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/ml-commons.git]
Components skipped due to git failures: [https://github.com/opensearch-project/opensearch-oci-object-storage.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

BUILD SUCCESSFUL in 31m 26s

@opensearch-trigger-bot
Copy link
Contributor

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/ml-commons.git]
Components skipped due to git failures: [https://github.com/opensearch-project/opensearch-oci-object-storage.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/reporting.git]

BUILD SUCCESSFUL in 29m 57s

@github-actions
Copy link
Contributor

github-actions bot commented Aug 3, 2023

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.client.PitIT.testDeleteAllAndListAllPits

@codecov
Copy link

codecov bot commented Aug 3, 2023

Codecov Report

Merging #9076 (0e9c7d7) into 2.x (6ab46bd) will increase coverage by 0.13%.
The diff coverage is 97.53%.

@@             Coverage Diff              @@
##                2.x    #9076      +/-   ##
============================================
+ Coverage     70.72%   70.86%   +0.13%     
- Complexity    57372    57497     +125     
============================================
  Files          4754     4754              
  Lines        271625   271647      +22     
  Branches      40066    40067       +1     
============================================
+ Hits         192102   192494     +392     
+ Misses        62972    62655     -317     
+ Partials      16551    16498      -53
Files Changed Coverage Δ
...ch/indices/recovery/PeerRecoveryTargetService.java 51.96% <0.00%> (-1.97%) ⬇️
...ices/replication/RemoteStoreReplicationSource.java 90.74% <96.42%> (-3.55%) ⬇️
...in/java/org/opensearch/index/shard/IndexShard.java 69.11% <100.00%> (-0.59%) ⬇️
...search/index/shard/RemoteStoreRefreshListener.java 84.61% <100.00%> (-0.09%) ⬇️
...java/org/opensearch/index/shard/StoreRecovery.java 57.25% <100.00%> (ø)
...earch/index/store/RemoteSegmentStoreDirectory.java 87.70% <100.00%> (ø)
...rc/main/java/org/opensearch/index/store/Store.java 80.65% <100.00%> (+0.40%) ⬆️
...x/store/remote/metadata/RemoteSegmentMetadata.java 100.00% <100.00%> (ø)
...h/indices/replication/GetSegmentFilesResponse.java 44.44% <100.00%> (+6.94%) ⬆️
.../indices/replication/SegmentReplicationTarget.java 89.32% <100.00%> (+5.48%) ⬆️

... and 473 files with indirect coverage changes

@dreamer-89 dreamer-89 merged commit 94338e5 into opensearch-project:2.x Aug 3, 2023
@dreamer-89 dreamer-89 deleted the backport/backport-8767-to-2.x branch August 3, 2023 18:14
@dreamer-89 dreamer-89 mentioned this pull request Aug 3, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Poojita-Raj Poojita-Raj Poojita-Raj approved these changes

@mch2 mch2 mch2 approved these changes

@reta reta Awaiting requested review from reta reta is a code owner

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

@dblock dblock Awaiting requested review from dblock dblock is a code owner

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna is a code owner

@setiah setiah Awaiting requested review from setiah

@kartg kartg Awaiting requested review from kartg

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

@nknize nknize Awaiting requested review from nknize nknize is a code owner

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

@ryanbogan ryanbogan Awaiting requested review from ryanbogan

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

@tlfeng tlfeng Awaiting requested review from tlfeng

@VachaShah VachaShah Awaiting requested review from VachaShah VachaShah is a code owner

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

@sachinpkale sachinpkale Awaiting requested review from sachinpkale sachinpkale is a code owner

@sohami sohami Awaiting requested review from sohami sohami is a code owner

Assignees
No one assigned
Labels
skip-changelog
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dreamer-89 @mch2 @Poojita-Raj