Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace internal usages of 'master' term in 'server/src/internalClusterTest' directory #2521

Conversation

tlfeng
Copy link
Collaborator

@tlfeng tlfeng commented Mar 19, 2022

Description

Replacement rules:

  • master -> cluster_manager or clusterManager (variable name) or cluster-manager (code comment)
  • Master -> ClusterManager

Issues Resolved

A part of #1548

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@tlfeng tlfeng requested a review from a team as a code owner March 19, 2022 09:11
@tlfeng tlfeng added enhancement Enhancement or improvement to existing feature or request v2.0.0 Version 2.0.0 labels Mar 19, 2022
@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success 66deace
Log 3552

Reports 3552

@@ -204,9 +204,9 @@ public void runRepeatedlyWhileChangingMaster(Runnable runnable) throws Exception
String value = "none";
while (shutdown.get() == false) {
value = "none".equals(value) ? "all" : "none";
final String nonMasterNode = randomValueOtherThan(masterName, () -> randomFrom(internalCluster().getNodeNames()));
final String nonClusterManagerNode = randomValueOtherThan(masterName, () -> randomFrom(internalCluster().getNodeNames()));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the variable masterName need to change as well?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @VachaShah, thank you so much for your review! You are so careful 👍😁! Yes.. it needs to be changed. I will search for the "master" word completely.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finally I did another scan. The variable name masterName has been changed. 👍

final String nonMasterNode = randomValueOtherThan(masterName, () -> randomFrom(internalCluster().getNodeNames()));
final String claimedMasterName = internalCluster().getMasterName(nonMasterNode);
final String nonClusterManagerNode = randomValueOtherThan(masterName, () -> randomFrom(internalCluster().getNodeNames()));
final String claimedMasterName = internalCluster().getMasterName(nonClusterManagerNode);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change claimedMasterName to claimedClusterManagerName?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed the variable name. 😄

Sets.difference(Sets.newHashSet(internalCluster().getNodeNames()), Collections.singleton(internalCluster().getMasterName()))
);
Settings nonMasterDataPathSettings1 = internalCluster().dataPathSettings(nonMasterNodes.get(0));
Settings nonMasterDataPathSettings2 = internalCluster().dataPathSettings(nonMasterNodes.get(1));
Settings nonMasterDataPathSettings1 = internalCluster().dataPathSettings(nonClusterManagerNodes.get(0));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Variable name change for nonMasterDataPathSettings1 and nonMasterDataPathSettings2?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed the variable name. 👍

);
detachCluster(environment);
unsafeBootstrap(environment); // read-only block will remain same as one before bootstrap, in this case it is false

String masterNode2 = internalCluster().startMasterOnlyNode(masterNodeDataPathSettings);
String masterNode2 = internalCluster().startMasterOnlyNode(clusterManagerNodeDataPathSettings);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

masterNode2 -> clusterManagerNode2?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed the variable name. 👍


DiscoveryNodes discoveryNodes = internalCluster().getInstance(ClusterService.class, nonMasterNode).state().nodes();
DiscoveryNodes discoveryNodes = internalCluster().getInstance(ClusterService.class, nonClusterManagerNode).state().nodes();

TransportService masterTranspotService = internalCluster().getInstance(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

masterTransportService -> clusterManagerTransportService?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed the variable name. 👍

@@ -228,9 +228,9 @@ public void testStaleMasterNotHijackingMajority() throws Exception {
event.state(),
event.previousState()
);
String previousMasterNodeName = previousMaster != null ? previousMaster.getName() : null;
String previousClusterManagerNodeName = previousMaster != null ? previousMaster.getName() : null;
String currentMasterNodeName = currentMaster != null ? currentMaster.getName() : null;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

currentMasterNodeName -> currentClusterManagerNodeName
currentMaster -> currentClusterManager

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed the variable name. 👍

@@ -560,14 +560,14 @@ public void testStartCloneWithSuccessfulShardClonePendingFinalization() throws E
blockMasterOnWriteIndexFile(repoName);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change method name blockMasterOnWriteIndexFile to blockClusterManagerOnWriteIndexFile?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a public method, and it has been published to maven, so I plan to change the name in next major version (3.0), and it's tracked in the issue #1684

@@ -314,11 +314,11 @@ public void testSnapshotRunsAfterInProgressDelete() throws Exception {

blockMasterFromFinalizingSnapshotOnIndexFile(repoName);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Method name blockMasterFromFinalizingSnapshotOnIndexFile -> blockClusterManagerFromFinalizingSnapshotOnIndexFile

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a public method, and it has been published to maven, so I plan to change the name in next major version (3.0), and it's tracked in the issue #1684

@@ -1033,35 +1033,35 @@ public void testStartDeleteDuringFinalizationCleanup() throws Exception {
final String snapshotName = "snap-name";
blockMasterFromDeletingIndexNFile(repoName);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blockMasterFromDeletingIndexNFile -> blockClusterManagerFromDeletingIndexNFile

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a public method, and it has been published to maven, so I plan to change the name in next major version (3.0), and it's tracked in the issue #1684

awaitNumberOfSnapshotsInProgress(1);
blockNodeOnAnyFiles(repoName, dataNode);
final ActionFuture<CreateSnapshotResponse> otherSnapshot = startFullSnapshot(repoName, "other-snapshot");
awaitNumberOfSnapshotsInProgress(2);
assertFalse(blockedSnapshot.isDone());
unblockNode(repoName, masterName);
unblockNode(repoName, clusterManagerName);
awaitNumberOfSnapshotsInProgress(1);

awaitMasterFinishRepoOperations();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awaitMasterFinishRepoOperations -> awaitClusterManagerFinishRepoOperations

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for catching it! I forgot to rename it in my last scan. 😅
I changed the name in the new commit 9fbc35f

nknize
nknize previously requested changes Mar 22, 2022
Copy link
Collaborator

@nknize nknize left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like some changes are in order here. Going to leave this open and we may just need to backport to 2x after the branching.

@tlfeng
Copy link
Collaborator Author

tlfeng commented Mar 22, 2022

Looks like some changes are in order here. Going to leave this open and we may just need to backport to 2x after the branching.

Hi @nknize, thanks for taking care of this PR. No problem, I will create backport PR to 2.x, and I will keep the impact of this PR within this repository.

@tlfeng tlfeng added WIP Work in progress and removed v2.0.0 Version 2.0.0 labels Mar 30, 2022
@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success 370fa9a
Log 4168

Reports 4168

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success 370fa9a
Log 5006

Reports 5006

@tlfeng tlfeng marked this pull request as draft May 4, 2022 22:51
Tianli Feng added 3 commits May 9, 2022 15:59

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
Signed-off-by: Tianli Feng <[email protected]>

# Conflicts:
#	server/src/internalClusterTest/java/org/opensearch/discovery/DiscoveryDisruptionIT.java
Signed-off-by: Tianli Feng <[email protected]>

# Conflicts:
#	server/src/internalClusterTest/java/org/opensearch/action/admin/ClientTimeoutIT.java
#	server/src/internalClusterTest/java/org/opensearch/action/admin/cluster/state/TransportClusterStateActionDisruptionIT.java
#	server/src/internalClusterTest/java/org/opensearch/cluster/coordination/UnsafeBootstrapAndDetachCommandIT.java
#	server/src/internalClusterTest/java/org/opensearch/cluster/coordination/ZenDiscoveryIT.java
#	server/src/internalClusterTest/java/org/opensearch/cluster/routing/AllocationIdIT.java
#	server/src/internalClusterTest/java/org/opensearch/discovery/ClusterDisruptionIT.java
#	server/src/internalClusterTest/java/org/opensearch/discovery/DiscoveryDisruptionIT.java
#	server/src/internalClusterTest/java/org/opensearch/env/NodeRepurposeCommandIT.java
#	server/src/internalClusterTest/java/org/opensearch/snapshots/CloneSnapshotIT.java
#	server/src/internalClusterTest/java/org/opensearch/snapshots/ConcurrentSnapshotsIT.java
#	server/src/internalClusterTest/java/org/opensearch/snapshots/DedicatedClusterSnapshotRestoreIT.java

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
…ernalClusterTest directory

Signed-off-by: Tianli Feng <[email protected]>
@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Check failure 8cb690c
Log 5327

Reports 5327

…ernalClusterTest directory

Signed-off-by: Tianli Feng <[email protected]>
@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Check failure fd5ad03
Log 5328

Reports 5328

Tianli Feng added 3 commits May 15, 2022 12:19
Signed-off-by: Tianli Feng <[email protected]>
…ernalClusterTest directory

Signed-off-by: Tianli Feng <[email protected]>
Signed-off-by: Tianli Feng <[email protected]>
@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success d41b1ed
Log 5340

Reports 5340

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success f5fde30
Log 5341

Reports 5341

@tlfeng tlfeng marked this pull request as ready for review May 15, 2022 21:33
@tlfeng tlfeng added v2.1.0 Issues and PRs related to version 2.1.0 backport 2.x Backport to 2.x branch labels May 15, 2022
@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Check failure 05ef18c
Log 5343

Reports 5343

@tlfeng
Copy link
Collaborator Author

tlfeng commented May 15, 2022

In log 5343:

REPRODUCE WITH: ./gradlew ':qa:remote-clusters:integTest' --tests "org.opensearch.cluster.remote.test.RemoteClustersIT.testHAProxyModeConnectionWorks" -Dtests.seed=D7E3D9B3F65AE94F -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-PY -Dtests.timezone=Africa/Gaborone -Druntime.java=17

WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker
WARNING: System::setSecurityManager will be removed in a future release

org.opensearch.cluster.remote.test.RemoteClustersIT > testHAProxyModeConnectionWorks FAILED
    java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([D7E3D9B3F65AE94F:D0950C78CDDC9F12]:0)
        at org.junit.Assert.fail(Assert.java:87)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.junit.Assert.assertTrue(Assert.java:53)
        at org.opensearch.cluster.remote.test.RemoteClustersIT.testHAProxyModeConnectionWorks(RemoteClustersIT.java:125)

It's reported in #1703 re-run: start gradle check.

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success 05ef18c
Log 5347

Reports 5347

@dblock dblock requested a review from nknize May 16, 2022 12:44
@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Check failure 9fbc35f
Log 5390

Reports 5390

@tlfeng
Copy link
Collaborator Author

tlfeng commented May 17, 2022

In log 5390:

> Task :qa:remote-clusters:integTest

REPRODUCE WITH: ./gradlew ':qa:remote-clusters:integTest' --tests "org.opensearch.cluster.remote.test.RemoteClustersIT.testHAProxyModeConnectionWorks" -Dtests.seed=C884FF6591BB100B -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-KW -Dtests.timezone=America/Metlakatla -Druntime.java=17

org.opensearch.cluster.remote.test.RemoteClustersIT > testHAProxyModeConnectionWorks FAILED
    java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([C884FF6591BB100B:CFF22AAEAA3D6656]:0)
        at org.junit.Assert.fail(Assert.java:87)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.junit.Assert.assertTrue(Assert.java:53)
        at org.opensearch.cluster.remote.test.RemoteClustersIT.testHAProxyModeConnectionWorks(RemoteClustersIT.java:125)

It's reported in issue #1703 .
re-run: start gradle check

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success 9fbc35f
Log 5392

Reports 5392

Copy link
Collaborator

@VachaShah VachaShah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @tlfeng for make all of these changes! I added some comments for this pass but overall the changes look good to me.

There are some tests in this directory whose name contain master keyword. Can we replace those (doesn't have to be the same PR)?

Also there are some method names like assertNoMaster, assertMaster, blockMasterFromFinalizingSnapshotOnIndexFile, blockMasterOnWriteIndexFile, blockMasterFromDeletingIndexNFile and several methods from InternalClusterTest.java which contain the master keyword but they belong to a different directory. Just want to make sure they are addressed in other PRs for those directories.

// The elected master shouldn't have changed, since the unlucky node never could have elected itself as master
assertThat(internalCluster().getMasterName(), equalTo(masterNode));
// The elected cluster-manager shouldn't have changed, since the unlucky node never could have elected itself as cluster-manager
assertThat(internalCluster().getMasterName(), equalTo(clusterManagerNode));
}

private void ensureNoMaster(String node) throws Exception {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ensureNoMaster -> ensureNoClusterManager?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for taking your time in the careful review! 👍👍
I totally missed these method names in test classes, I will rename them in this PR.

Copy link
Collaborator Author

@tlfeng tlfeng May 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some other method names:
For assertMaster, I changed its name in another PR: https://github.com/opensearch-project/OpenSearch/pull/2520/files#diff-8eef20b401000cc2fe94a8d5ee540ee6a77d15b0e6c9bdc2b6eb4da35e6f5dd3
For assertNoMaster, thank you for catching it, I forgot to change its name, and I just renamed it and updated the PR 2520 (dbb085b).
For blockMasterFromFinalizingSnapshotOnIndexFile, blockMasterOnWriteIndexFile, and blockMasterFromDeletingIndexNFile, since they are public method in test/framework directory, I will rename them in the next phase, with other public Java API.

Copy link
Collaborator Author

@tlfeng tlfeng May 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, I searched all the master reference in test method names by using regex test(\w*)Master(\w*)\( 🙂, nothing in the test method names should be missed any more. Haha

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

About the folder name, such as https://github.com/opensearch-project/OpenSearch/tree/2.0.0-rc1/server/src/internalClusterTest/java/org/opensearch/action/support/master, I saw there is a corresponding folder in the source code https://github.com/opensearch-project/OpenSearch/tree/main/server/src/main/java/org/opensearch/action/support/master, so I was thinking to change the folder name along with renaming the public package names in the source code set.
For the method names in the class InternalClusterTest, I'm planning to rename the master terminology along with the other Java APIs, because those methods with master word are all public (https://opensearch.org/javadocs/2.0.0-rc1/OpenSearch/test/framework/build/docs/javadoc/org/opensearch/test/InternalTestCluster.html), I will rename them in the next major version.
The above 2 problems will be resolved in the issue #1684

@@ -135,14 +135,14 @@ private void ensureNoMaster(String node) throws Exception {
}

/**
* Verify that nodes fault detection detects a disconnected node after master reelection
* Verify that nodes fault detection detects a disconnected node after cluster-manager reelection
*/
public void testFollowerCheckerDetectsDisconnectedNodeAfterMasterReelection() throws Exception {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename this test to include cluster manager?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed the name of the test method in the new commit 02af408. Thank you!

@@ -135,14 +135,14 @@ private void ensureNoMaster(String node) throws Exception {
}

/**
* Verify that nodes fault detection detects a disconnected node after master reelection
* Verify that nodes fault detection detects a disconnected node after cluster-manager reelection
*/
public void testFollowerCheckerDetectsDisconnectedNodeAfterMasterReelection() throws Exception {
testFollowerCheckerAfterMasterReelection(NetworkDisruption.DISCONNECT, Settings.EMPTY);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

testFollowerCheckerAfterMasterReelection -> testFollowerCheckerAfterClusterManagerReelection?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed the method name in the new commit 02af408.

*/
public void testFollowerCheckerDetectsDisconnectedNodeAfterMasterReelection() throws Exception {
testFollowerCheckerAfterMasterReelection(NetworkDisruption.DISCONNECT, Settings.EMPTY);
}

/**
* Verify that nodes fault detection detects an unresponsive node after master reelection
* Verify that nodes fault detection detects an unresponsive node after cluster-manager reelection
*/
public void testFollowerCheckerDetectsUnresponsiveNodeAfterMasterReelection() throws Exception {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

testFollowerCheckerDetectsUnresponsiveNodeAfterMasterReelection -> testFollowerCheckerDetectsUnresponsiveNodeAfterClusterManagerReelection?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed the method name in the new commit 02af408.

* Tests that emulates a frozen elected master node that unfreezes and pushes its cluster state to other nodes that already are
* following another elected master node. These nodes should reject this cluster state and prevent them from following the stale master.
* Tests that emulates a frozen elected cluster-manager node that unfreezes and pushes its cluster state to other nodes that already are
* following another elected cluster-manager node. These nodes should reject this cluster state and prevent them from following the stale cluster-manager.
*/
public void testStaleMasterNotHijackingMajority() throws Exception {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

testStaleMasterNotHijackingMajority -> testStaleClusterManagerNotHijackingMajority?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed the method name in the new commit 02af408.

@@ -129,10 +129,10 @@ public void testRecoverAfterNodes() throws Exception {

public void testRecoverAfterMasterNodes() throws Exception {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

testRecoverAfterMasterNodes -> testRecoverAfterClusterManagerNodes?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed the method name in the new commit 02af408.

@@ -390,7 +390,7 @@ private void assertConcreteMappingsOnAll(final String index, final String... fie
}

/**
* Waits for the given mapping type to exists on the master node.
* Waits for the given mapping type to exists on the cluster-manager node.
*/
private void assertMappingOnMaster(final String index, final String... fieldNames) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assertMappingOnMaster -> assertMappingOnClusterManager?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed the method name in the new commit 87d07e5. Thank you for catching it!

@@ -1466,7 +1466,7 @@ public void testDoNotInfinitelyWaitForMapping() {
assertHitCount(client().prepareSearch().get(), numDocs);
}

/** Makes sure the new master does not repeatedly fetch index metadata from recovering replicas */
/** Makes sure the new cluster-manager does not repeatedly fetch index metadata from recovering replicas */
public void testOngoingRecoveryAndMasterFailOver() throws Exception {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

testOngoingRecoveryAndMasterFailOver ->. testOngoingRecoveryAndClusterManagerFailOver?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed the method name in the new commit 02af408.

@@ -338,7 +338,7 @@ public void testBackToBackClonesForIndexNotInCluster() throws Exception {
final String targetSnapshot1 = "target-snapshot";
blockMasterOnShardClone(repoName);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blockMasterOnShardClone -> blockClusterManagerOnShardClone?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed the method name in the new commit 87d07e5. Thank you! 👍

final String masterNode = internalCluster().getMasterName();
blockNodeOnAnyFiles(repoName, masterNode);
final String clusterManagerNode = internalCluster().getMasterName();
blockNodeOnAnyFiles(repoName, clusterManagerNode);
final ActionFuture<CreateSnapshotResponse> snapshotThree = startFullSnapshotFromNonMasterClient(repoName, "snapshot-three");
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

startFullSnapshotFromNonMasterClient -> startFullSnapshotFromNonClusterManagerClient?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed the method name in the new commit 87d07e5. Thank you for catching it!

@opensearch-ci-bot
Copy link
Collaborator

❌   Gradle Check failure 87d07e5
Log 5460

Reports 5460

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success 22d2c3e
Log 5462

Reports 5462

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success 2176adf
Log 5464

Reports 5464

Copy link
Collaborator

@VachaShah VachaShah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you @tlfeng for making all these changes.

@tlfeng tlfeng dismissed nknize’s stale review May 19, 2022 17:23

All comments are resolved. 😁

@tlfeng tlfeng merged commit 507f8cc into opensearch-project:main May 19, 2022
opensearch-trigger-bot bot pushed a commit that referenced this pull request May 19, 2022
…erTest' directory (#2521)

Signed-off-by: Tianli Feng <[email protected]>
(cherry picked from commit 507f8cc)
@tlfeng tlfeng deleted the replace-master-int-server-internalclustertest branch May 19, 2022 17:25
tlfeng pushed a commit that referenced this pull request May 19, 2022
Bukhtawar pushed a commit that referenced this pull request Jun 20, 2022
Bukhtawar added a commit that referenced this pull request Jun 27, 2022
* Bump reactor-netty-core from 1.0.16 to 1.0.19 in /plugins/repository-azure (#3360)

* Bump reactor-netty-core in /plugins/repository-azure

Bumps [reactor-netty-core](https://github.com/reactor/reactor-netty) from 1.0.16 to 1.0.19.
- [Release notes](https://github.com/reactor/reactor-netty/releases)
- [Commits](reactor/reactor-netty@v1.0.16...v1.0.19)

---
updated-dependencies:
- dependency-name: io.projectreactor.netty:reactor-netty-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

* [Type removal] _type removal from mocked responses of scroll hit tests (#3377)

Signed-off-by: Suraj Singh <[email protected]>

* [Type removal] Remove _type deprecation from script and conditional processor (#3239)

* [Type removal] Remove _type deprecation from script and conditional processor

Signed-off-by: Suraj Singh <[email protected]>

* Spotless check apply

Signed-off-by: Suraj Singh <[email protected]>

* [Type removal] Remove _type from _bulk yaml test, scripts, unused constants (#3372)

* [Type removal] Remove redundant _type deprecation checks in bulk request

Signed-off-by: Suraj Singh <[email protected]>

* [Type removal] bulk yaml tests validating deprecation on _type and removal from scripts

Signed-off-by: Suraj Singh <[email protected]>

* Fix Lucene-snapshots repo for jdk 17. (#3396)

Signed-off-by: Marc Handalian <[email protected]>

* Replace internal usages of 'master' term in 'server/src/internalClusterTest' directory (#2521)

Signed-off-by: Tianli Feng <[email protected]>

* [REMOVE] Cleanup deprecated thread pool types (FIXED_AUTO_QUEUE_SIZE) (#3369)

Signed-off-by: Andriy Redko <[email protected]>

* [Type removal] _type removal from tests of yaml tests (#3406)

* [Type removal] _type removal from tests of yaml tests

Signed-off-by: Suraj Singh <[email protected]>

* Fix spotless failures

Signed-off-by: Suraj Singh <[email protected]>

* Fix assertion failures

Signed-off-by: Suraj Singh <[email protected]>

* Fix assertion failures in DoSectionTests

Signed-off-by: Suraj Singh <[email protected]>

* Add release notes for version 2.0.0 (#3410)


Signed-off-by: Rabi Panda <[email protected]>

* [Upgrade] Lucene-9.2.0-snapshot-ba8c3a8 (#3416)

Upgrades to latest snapshot of lucene 9.2.0 in preparation for GA release.

Signed-off-by: Nicholas Walter Knize <[email protected]>

* Fix release notes for 2.0.0-rc1 version (#3418)

This change removes some old commits from the 2.0.0-rc1 release notes. These commits were already released as part of 1.x releases.

Add back some missing type removal commits to the 2.0.0 release notes

Signed-off-by: Rabi Panda <[email protected]>

* Bump version 2.1 to Lucene 9.2 after upgrade (#3424)

Bumps Version.V_2_1_0 lucene version to 9.2 after backporting upgrage.

Signed-off-by: Nicholas Walter Knize <[email protected]>

* Bump com.gradle.enterprise from 3.10 to 3.10.1 (#3425)

Bumps com.gradle.enterprise from 3.10 to 3.10.1.

---
updated-dependencies:
- dependency-name: com.gradle.enterprise
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump reactor-core from 3.4.17 to 3.4.18 in /plugins/repository-azure (#3427)

Bumps [reactor-core](https://github.com/reactor/reactor-core) from 3.4.17 to 3.4.18.
- [Release notes](https://github.com/reactor/reactor-core/releases)
- [Commits](reactor/reactor-core@v3.4.17...v3.4.18)

---
updated-dependencies:
- dependency-name: io.projectreactor:reactor-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

* Bump gax-httpjson from 0.101.0 to 0.103.1 in /plugins/repository-gcs (#3426)

Bumps [gax-httpjson](https://github.com/googleapis/gax-java) from 0.101.0 to 0.103.1.
- [Release notes](https://github.com/googleapis/gax-java/releases)
- [Changelog](https://github.com/googleapis/gax-java/blob/main/CHANGELOG.md)
- [Commits](https://github.com/googleapis/gax-java/commits)

---
updated-dependencies:
- dependency-name: com.google.api:gax-httpjson
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

* [segment replication]Introducing common Replication interfaces for segment replication and recovery code paths (#3234)

* RecoveryState inherits from ReplicationState + RecoveryTarget inherits from ReplicationTarget

Signed-off-by: Poojita Raj <[email protected]>

* Refactoring: mixedClusterVersion error fix + move Stage to ReplicationState

Signed-off-by: Poojita Raj <[email protected]>

* pull ReplicationListener into a top level class + add javadocs + address review comments

Signed-off-by: Poojita Raj <[email protected]>

* fix javadoc

Signed-off-by: Poojita Raj <[email protected]>

* review changes

Signed-off-by: Poojita Raj <[email protected]>

* Refactoring the hierarchy relationship between repl and recovery

Signed-off-by: Poojita Raj <[email protected]>

* style fix

Signed-off-by: Poojita Raj <[email protected]>

* move package common under replication

Signed-off-by: Poojita Raj <[email protected]>

* rename to replication

Signed-off-by: Poojita Raj <[email protected]>

* rename and doc changes

Signed-off-by: Poojita Raj <[email protected]>

* [Type removal] Remove type from BulkRequestParser (#3423)

* [Type removal] Remove type handling in bulk request parser

Signed-off-by: Suraj Singh <[email protected]>

* [Type removal] Remove testTypesStillParsedForBulkMonitoring as it is no longer present in codebase

Signed-off-by: Suraj Singh <[email protected]>

* Adding CheckpointRefreshListener to trigger when Segment replication is turned on and Primary shard refreshes (#3108)

* Intial PR adding classes and tests related to checkpoint publishing

Signed-off-by: Rishikesh1159 <[email protected]>

* Putting a Draft PR with all changes in classes. Testing is still not included in this commit.

Signed-off-by: Rishikesh1159 <[email protected]>

* Wiring up index shard to new engine, spotless apply and removing unnecessary tests and logs

Signed-off-by: Rishikesh1159 <[email protected]>

* Adding Unit test for checkpointRefreshListener

Signed-off-by: Rishikesh1159 <[email protected]>

* Applying spotless check

Signed-off-by: Rishikesh1159 <[email protected]>

* Fixing import statements *

Signed-off-by: Rishikesh1159 <[email protected]>

* removing unused constructor in index shard

Signed-off-by: Rishikesh1159 <[email protected]>

* Addressing comments from last commit

Signed-off-by: Rishikesh1159 <[email protected]>

* Adding package-info.java files for two new packages

Signed-off-by: Rishikesh1159 <[email protected]>

* Adding test for null checkpoint publisher and addreesing PR comments

Signed-off-by: Rishikesh1159 <[email protected]>

* Add docs for indexshardtests and remove shard.refresh

Signed-off-by: Rishikesh1159 <[email protected]>

* Add a new Engine implementation for replicas with segment replication enabled. (#3240)

* Change fastForwardProcessedSeqNo method in LocalCheckpointTracker to persisted checkpoint.

This change inverts fastForwardProcessedSeqNo to fastForwardPersistedSeqNo for use in
Segment Replication.  This is so that a Segrep Engine can match the logic of InternalEngine
where the seqNo is incremented with each operation, but only persisted in the tracker on a flush.
With Segment Replication we bump the processed number with each operation received index/delete/noOp, and
invoke this method when we receive a new set of segments to bump the persisted seqNo.

Signed-off-by: Marc Handalian <[email protected]>

* Extract Translog specific engine methods into an abstract class.

This change extracts translog specific methods to an abstract engine class so that other engine
implementations can reuse translog logic.

Signed-off-by: Marc Handalian <[email protected]>

* Add a separate Engine implementation for replicas with segment replication enabled.

This change adds a new engine intended to be used on replicas with segment replication enabled.
This engine does not wire up an IndexWriter, but still writes all operations to a translog.
The engine uses a new ReaderManager that refreshes from an externally provided SegmentInfos.

Signed-off-by: Marc Handalian <[email protected]>

* Fix spotless checks.

Signed-off-by: Marc Handalian <[email protected]>

* Fix :server:compileInternalClusterTestJava compilation.

Signed-off-by: Marc Handalian <[email protected]>

* Fix failing test naming convention check.

Signed-off-by: Marc Handalian <[email protected]>

* PR feedback.

- Removed isReadOnlyReplica from overloaded constructor and added feature flag checks.
- Updated log msg in NRTReplicationReaderManager
- cleaned up store ref counting in NRTReplicationEngine.

Signed-off-by: Marc Handalian <[email protected]>

* Fix spotless check.

Signed-off-by: Marc Handalian <[email protected]>

* Remove TranslogAwareEngine and build translog in NRTReplicationEngine.

Signed-off-by: Marc Handalian <[email protected]>

* Fix formatting

Signed-off-by: Marc Handalian <[email protected]>

* Add missing translog methods to NRTEngine.

Signed-off-by: Marc Handalian <[email protected]>

* Remove persistent seqNo check from fastForwardProcessedSeqNo.

Signed-off-by: Marc Handalian <[email protected]>

* PR feedback.

Signed-off-by: Marc Handalian <[email protected]>

* Add test specific to translog trimming.

Signed-off-by: Marc Handalian <[email protected]>

* Javadoc check.

Signed-off-by: Marc Handalian <[email protected]>

* Add failEngine calls to translog methods in NRTReplicationEngine.
Roll xlog generation on replica when a new commit point is received.

Signed-off-by: Marc Handalian <[email protected]>

* Rename master to cluster_manager in the XContent Parser of ClusterHealthResponse (#3432)

Signed-off-by: Tianli Feng <[email protected]>

* Bump hadoop-minicluster in /test/fixtures/hdfs-fixture (#3359)

Bumps hadoop-minicluster from 3.3.2 to 3.3.3.

---
updated-dependencies:
- dependency-name: org.apache.hadoop:hadoop-minicluster
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump avro from 1.10.2 to 1.11.0 in /plugins/repository-hdfs (#3358)

* Bump avro from 1.10.2 to 1.11.0 in /plugins/repository-hdfs

Bumps avro from 1.10.2 to 1.11.0.

---
updated-dependencies:
- dependency-name: org.apache.avro:avro
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

* Fix testSetAdditionalRolesCanAddDeprecatedMasterRole() by removing the initial assertion (#3441)

Signed-off-by: Tianli Feng <[email protected]>

* Replace internal usages of 'master' term in 'server/src/test' directory (#2520)

* Replace the non-inclusive terminology "master" with "cluster manager" in code comments, internal variable/method/class names, in `server/src/test` directory.
* Backwards compatibility is not impacted.
* Add a new unit test `testDeprecatedMasterNodeFilter()` to validate using `master:true` or `master:false` can filter the node in [Cluster Stats](https://opensearch.org/docs/latest/opensearch/rest-api/cluster-stats/) API, after the `master` role is deprecated in PR #2424

Signed-off-by: Tianli Feng <[email protected]>

* Removing unused method from TransportSearchAction (#3437)

* Removing unused method from TransportSearchAction

Signed-off-by: Ankit Jain <[email protected]>

* Set term vector flags to false for ._index_prefix field (#1901). (#3119)

* Set term vector flags to false for ._index_prefix field (#1901).

Signed-off-by: Vesa Pehkonen <[email protected]>

* Replaced the FieldType copy ctor with ctor for the prefix field and replaced
setting the field type parameters with setIndexOptions(). (#1901)

Signed-off-by: Vesa Pehkonen <[email protected]>

* Added tests for term vectors. (#1901)

Signed-off-by: Vesa Pehkonen <[email protected]>

* Fixed code formatting error.

Signed-off-by: Vesa Pehkonen <[email protected]>

Co-authored-by: sdp <[email protected]>

* [BUG] Fixing org.opensearch.monitor.os.OsProbeTests > testLogWarnCpuMessageOnlyOnes when cgroups are available but cgroup stats is not (#3448)

Signed-off-by: Andriy Redko <[email protected]>

* [Segment Replication] Add SegmentReplicationTargetService to orchestrate replication events. (#3439)

* Add SegmentReplicationTargetService to orchestrate replication events.

This change introduces  boilerplate classes for Segment Replication and a target service
to orchestrate replication events.

It also includes two refactors of peer recovery components for reuse.
1. Rename RecoveryFileChunkRequest to FileChunkRequest and extract code to handle throttling into
ReplicationTarget.
2. Extracts a component to execute retryable requests over the transport layer.

Signed-off-by: Marc Handalian <[email protected]>

* Code cleanup.

Signed-off-by: Marc Handalian <[email protected]>

* Make SegmentReplicationTargetService component final so that it can not
be extended by plugins.

Signed-off-by: Marc Handalian <[email protected]>

* Bump azure-core-http-netty from 1.11.9 to 1.12.0 in /plugins/repository-azure (#3474)

Bumps [azure-core-http-netty](https://github.com/Azure/azure-sdk-for-java) from 1.11.9 to 1.12.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-java/releases)
- [Commits](Azure/azure-sdk-for-java@azure-core-http-netty_1.11.9...azure-core_1.12.0)

---
updated-dependencies:
- dependency-name: com.azure:azure-core-http-netty
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update to Apache Lucene 9.2 (#3477)

Signed-off-by: Andriy Redko <[email protected]>

* Bump protobuf-java from 3.20.1 to 3.21.1 in /plugins/repository-hdfs (#3472)

Signed-off-by: dependabot[bot] <[email protected]>

* [Upgrade] Lucene-9.3.0-snapshot-823df23 (#3478)

Upgrades to latest snapshot of lucene 9.3.0.

Signed-off-by: Nicholas Walter Knize <[email protected]>

* Filter out invalid URI and HTTP method in the error message of no handler found for a REST request (#3459)

Filter out invalid URI and HTTP method of a error message, which shown when there is no handler found for a REST request sent by user, so that HTML special characters <>&"' will not shown in the error message.

The error message is return as mine-type `application/json`, which can't contain active (script) content, so it's not a vulnerability. Besides, no browsers are going to render as html when the mine-type is that.
While the common security scanners will raise a false-positive alarm for having HTML tags in the response without escaping the HTML special characters, so the solution only aims to satisfy the code security scanners.

Signed-off-by: Tianli Feng <[email protected]>

* Support use of IRSA for repository-s3 plugin credentials (#3475)

* Support use of IRSA for repository-s3 plugin credentials

Signed-off-by: Andriy Redko <[email protected]>

* Address code review comments

Signed-off-by: Andriy Redko <[email protected]>

* Address code review comments

Signed-off-by: Andriy Redko <[email protected]>

* Bump google-auth-library-oauth2-http from 0.20.0 to 1.7.0 in /plugins/repository-gcs (#3473)

* Bump google-auth-library-oauth2-http in /plugins/repository-gcs

Bumps google-auth-library-oauth2-http from 0.20.0 to 1.7.0.

---
updated-dependencies:
- dependency-name: com.google.auth:google-auth-library-oauth2-http
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

* Use variable to define the version of dependency google-auth-library-java

Signed-off-by: Tianli Feng <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tianli Feng <[email protected]>

* [Segment Replication] Added source-side classes for orchestrating replication events (#3470)

This change expands on the existing SegmentReplicationSource interface and its corresponding Factory class by introducing an implementation where the replication source is a primary shard (PrimaryShardReplicationSource). These code paths execute on the target. The primary shard implementation creates the requests to be send to the source/primary shard.

Correspondingly, this change also defines two request classes for the GET_CHECKPOINT_INFO and GET_SEGMENT_FILES requests as well as an abstract superclass.

A CopyState class has been introduced that captures point-in-time, file-level details from an IndexShard. This implementation mirrors Lucene's NRT CopyState implementation.

Finally, a service class has been introduce for segment replication that runs on the source side (SegmentReplicationSourceService) which handles these two types of incoming requests. This includes private handler classes that house the logic to respond to these requests, with some functionality stubbed for now. The service class also uses a simple map to cache CopyState objects that would be needed by replication targets.

Unit tests have been added/updated for all new functionality.

Signed-off-by: Kartik Ganesh <[email protected]>

* [Dependency upgrade] google-oauth-client to 1.33.3 (#3500)

Signed-off-by: Suraj Singh <[email protected]>

* move bash flag to set statement (#3494)

Passing bash with flags to the first argument of /usr/bin/env requires
its own flag to interpret it correctly.  Rather than use `env -S` to
split the argument, have the script `set -e` to enable the same behavior
explicitly in preinst and postinst scripts.

Also set `-o pipefail` for consistency.

Closes: #3492

Signed-off-by: Cole White <[email protected]>

* Support use of IRSA for repository-s3 plugin credentials: added YAML Rest test case (#3499)

Signed-off-by: Andriy Redko <[email protected]>

* Bump azure-storage-common from 12.15.0 to 12.16.0 in /plugins/repository-azure (#3517)

* Bump azure-storage-common in /plugins/repository-azure

Bumps [azure-storage-common](https://github.com/Azure/azure-sdk-for-java) from 12.15.0 to 12.16.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-java/releases)
- [Commits](Azure/azure-sdk-for-java@azure-storage-blob_12.15.0...azure-storage-blob_12.16.0)

---
updated-dependencies:
- dependency-name: com.azure:azure-storage-common
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

* Bump google-oauth-client from 1.33.3 to 1.34.0 in /plugins/discovery-gce (#3516)

* Bump google-oauth-client from 1.33.3 to 1.34.0 in /plugins/discovery-gce

Bumps [google-oauth-client](https://github.com/googleapis/google-oauth-java-client) from 1.33.3 to 1.34.0.
- [Release notes](https://github.com/googleapis/google-oauth-java-client/releases)
- [Changelog](https://github.com/googleapis/google-oauth-java-client/blob/main/CHANGELOG.md)
- [Commits](googleapis/google-oauth-java-client@v1.33.3...v1.34.0)

---
updated-dependencies:
- dependency-name: com.google.oauth-client:google-oauth-client
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

* Fix the support of RestClient Node Sniffer for version 2.x and update tests (#3487)

Fix the support of RestClient Node Sniffer for OpenSearch 2.x, and update unit tests for OpenSearch.
The current code contains the logic to be compatible with Elasticsearch 2.x version, which is conflict with OpenSearch 2.x, so removed that part of legacy code.

* Update the script create_test_nodes_info.bash to dump the response of Nodes Info API GET _nodes/http for OpenSearch 1.0 and 2.0 version, which used for unit test.
* Remove the support of Elasticsearch version 2.x for the Sniffer
* Update unit test to validate the Sniffer compatible with OpenSearch 1.x and 2.x
* Update the API response parser to meet the array notation (in ES 6.1 and above) for the node attributes setting. It will result the value of `node.attr` setting will not be parsed as array in the Sniffer, when using the Sniffer on cluster in Elasticsearch 6.0 and above.
* Replace "master" node role with "cluster_manager" in unit test

Signed-off-by: Tianli Feng <[email protected]>

* Bump com.diffplug.spotless from 6.6.1 to 6.7.0 (#3513)

Bumps com.diffplug.spotless from 6.6.1 to 6.7.0.

---
updated-dependencies:
- dependency-name: com.diffplug.spotless
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump guava from 18.0 to 23.0 in /plugins/ingest-attachment (#3357)

* Bump guava from 18.0 to 23.0 in /plugins/ingest-attachment

Bumps [guava](https://github.com/google/guava) from 18.0 to 23.0.
- [Release notes](https://github.com/google/guava/releases)
- [Commits](google/guava@v18.0...v23.0)

---
updated-dependencies:
- dependency-name: com.google.guava:guava
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

* Add more ingorance of using internal java API sun.misc.Unsafe

Signed-off-by: Tianli Feng <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tianli Feng <[email protected]>

* Added bwc version 2.0.1 (#3452)

Signed-off-by: Kunal Kotwani <[email protected]>

Co-authored-by: opensearch-ci-bot <[email protected]>

* Add release notes for 1.3.3 (#3549)

Signed-off-by: Xue Zhou <[email protected]>

* [Upgrade] Lucene-9.3.0-snapshot-b7231bb (#3537)

Upgrades to latest snapshot of lucene 9.3; including reducing maxFullFlushMergeWaitMillis 
in LuceneTest.testWrapLiveDocsNotExposeAbortedDocuments to 0 ms to ensure aborted 
docs are not merged away in the test with the new mergeOnRefresh default policy.

Signed-off-by: Nicholas Walter Knize <[email protected]>

* [Remote Store] Upload segments to remote store post refresh (#3460)

* Add RemoteDirectory interface to copy segment files to/from remote store

Signed-off-by: Sachin Kale <[email protected]>

Co-authored-by: Sachin Kale <[email protected]>

* Add index level setting for remote store

Signed-off-by: Sachin Kale <[email protected]>

Co-authored-by: Sachin Kale <[email protected]>

* Add RemoteDirectoryFactory and use RemoteDirectory instance in RefreshListener

Co-authored-by: Sachin Kale <[email protected]>
Signed-off-by: Sachin Kale <[email protected]>

* Upload segment to remote store post refresh

Signed-off-by: Sachin Kale <[email protected]>

Co-authored-by: Sachin Kale <[email protected]>

* Fixing VerifyVersionConstantsIT test failure (#3574)

Signed-off-by: Andriy Redko <[email protected]>

* Bump jettison from 1.4.1 to 1.5.0 in /plugins/discovery-azure-classic (#3571)

* Bump jettison from 1.4.1 to 1.5.0 in /plugins/discovery-azure-classic

Bumps [jettison](https://github.com/jettison-json/jettison) from 1.4.1 to 1.5.0.
- [Release notes](https://github.com/jettison-json/jettison/releases)
- [Commits](jettison-json/jettison@jettison-1.4.1...jettison-1.5.0)

---
updated-dependencies:
- dependency-name: org.codehaus.jettison:jettison
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

* Bump google-api-services-storage from v1-rev20200814-1.30.10 to v1-rev20220608-1.32.1 in /plugins/repository-gcs (#3573)

* Bump google-api-services-storage in /plugins/repository-gcs

Bumps google-api-services-storage from v1-rev20200814-1.30.10 to v1-rev20220608-1.32.1.

---
updated-dependencies:
- dependency-name: com.google.apis:google-api-services-storage
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

* Upgrade Google HTTP Client to 1.42.0

Signed-off-by: Xue Zhou <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xue Zhou <[email protected]>

* Add flat_skew setting to node overload decider (#3563)

* Add flat_skew setting to node overload decider

Signed-off-by: Rishab Nahata <[email protected]>

* Bump xmlbeans from 5.0.3 to 5.1.0 in /plugins/ingest-attachment (#3572)

* Bump xmlbeans from 5.0.3 to 5.1.0 in /plugins/ingest-attachment

Bumps xmlbeans from 5.0.3 to 5.1.0.

---
updated-dependencies:
- dependency-name: org.apache.xmlbeans:xmlbeans
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

* Bump google-oauth-client from 1.34.0 to 1.34.1 in /plugins/discovery-gce (#3570)

* Bump google-oauth-client from 1.34.0 to 1.34.1 in /plugins/discovery-gce

Bumps [google-oauth-client](https://github.com/googleapis/google-oauth-java-client) from 1.34.0 to 1.34.1.
- [Release notes](https://github.com/googleapis/google-oauth-java-client/releases)
- [Changelog](https://github.com/googleapis/google-oauth-java-client/blob/main/CHANGELOG.md)
- [Commits](googleapis/google-oauth-java-client@v1.34.0...v1.34.1)

---
updated-dependencies:
- dependency-name: com.google.oauth-client:google-oauth-client
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

* Fix for bug showing incorrect awareness attributes count in AwarenessAllocationDecider (#3428)

* Fix for bug showing incorrect awareness attributes count in AwarenessAllocationDecider

Signed-off-by: Anshu Agarwal <[email protected]>

* Added bwc version 1.3.4 (#3552)

Signed-off-by: GitHub <[email protected]>

Co-authored-by: opensearch-ci-bot <[email protected]>

* Support dynamic node role (#3436)

* Support unknown node role

Currently OpenSearch only supports several built-in nodes like data node
role. If specify unknown node role, OpenSearch node will fail to start.
This limit how to extend OpenSearch to support some extension function.
For example, user may prefer to run ML tasks on some dedicated node
which doesn't serve as any built-in node roles. So the ML tasks won't
impact OpenSearch core function. This PR removed the limitation and user
can specify any node role and OpenSearch will start node correctly with
that unknown role. This opens the door for plugin developer to run
specific tasks on dedicated nodes.

Issue: #2877

Signed-off-by: Yaliang Wu <[email protected]>

* fix cat nodes rest API spec

Signed-off-by: Yaliang Wu <[email protected]>

* fix mixed cluster IT failure

Signed-off-by: Yaliang Wu <[email protected]>

* add DynamicRole

Signed-off-by: Yaliang Wu <[email protected]>

* change generator method name

Signed-off-by: Yaliang Wu <[email protected]>

* fix failed docker test

Signed-off-by: Yaliang Wu <[email protected]>

* transform role name to lower case to avoid confusion

Signed-off-by: Yaliang Wu <[email protected]>

* transform the node role abbreviation to lower case

Signed-off-by: Yaliang Wu <[email protected]>

* fix checkstyle

Signed-off-by: Yaliang Wu <[email protected]>

* add test for case-insensitive role name change

Signed-off-by: Yaliang Wu <[email protected]>

* Rename package 'o.o.action.support.master' to 'o.o.action.support.clustermanager' (#3556)

* Rename package org.opensearch.action.support.master to org.opensearch.action.support.clustermanager

Signed-off-by: Tianli Feng <[email protected]>

* Rename classes with master term in the package org.opensearch.action.support.master

Signed-off-by: Tianli Feng <[email protected]>

* Deprecate classes in org.opensearch.action.support.master

Signed-off-by: Tianli Feng <[email protected]>

* Remove pakcage o.o.action.support.master

Signed-off-by: Tianli Feng <[email protected]>

* Move package-info back

Signed-off-by: Tianli Feng <[email protected]>

* Move package-info to new folder

Signed-off-by: Tianli Feng <[email protected]>

* Correct the package-info

Signed-off-by: Tianli Feng <[email protected]>

* Fixing flakiness of ShuffleForcedMergePolicyTests (#3591)

Signed-off-by: Andriy Redko <[email protected]>

* Deprecate classes in org.opensearch.action.support.master (#3593)

Signed-off-by: Tianli Feng <[email protected]>

* Add release notes for version 2.0.1 (#3595)

Signed-off-by: Kunal Kotwani <[email protected]>

* Fix NPE when minBound/maxBound is not set before being called. (#3605)

Signed-off-by: George Apaaboah <[email protected]>

* Added bwc version 2.0.2 (#3613)

Co-authored-by: opensearch-ci-bot <[email protected]>

* Fix false positive query timeouts due to using cached time (#3454)

* Fix false positive query timeouts due to using cached time

Signed-off-by: Ahmad AbuKhalil <[email protected]>

* delegate nanoTime call to SearchContext

Signed-off-by: Ahmad AbuKhalil <[email protected]>

* add override to SearchContext getRelativeTimeInMillis to force non cached time

Signed-off-by: Ahmad AbuKhalil <[email protected]>

* Fix random gradle check failure issue 3584. (#3627)

* [Segment Replication] Add components for segment replication to perform file copy. (#3525)

* Add components for segment replication to perform file copy.

This change adds the required components to SegmentReplicationSourceService to initiate copy and react to lifecycle events.
Along with new components it refactors common file copy code from RecoverySourceHandler into reusable pieces.

Signed-off-by: Marc Handalian <[email protected]>

* Deprecate public methods and variables with master term in package 'org.opensearch.action.support.master' (#3617)

Signed-off-by: Tianli Feng <[email protected]>

* Add replication orchestration for a single shard (#3533)

* implement segment replication target

Signed-off-by: Poojita Raj <[email protected]>

* test added

Signed-off-by: Poojita Raj <[email protected]>

* changes to tests + finalizeReplication

Signed-off-by: Poojita Raj <[email protected]>

* fix style check

Signed-off-by: Poojita Raj <[email protected]>

* addressing comments + fix gradle check

Signed-off-by: Poojita Raj <[email protected]>

* added test + addressed review comments

Signed-off-by: Poojita Raj <[email protected]>

* [BUG] opensearch crashes on closed client connection before search reply (#3626)

* [BUG] opensearch crashes on closed client connection before search reply

Signed-off-by: Andriy Redko <[email protected]>

* Addressing code review comments

Signed-off-by: Andriy Redko <[email protected]>

* Add all deprecated method in the package with new name 'org.opensearch.action.support.clustermanager' (#3644)

Signed-off-by: Tianli Feng <[email protected]>

* Introduce TranslogManager implementations decoupled from the Engine (#3638)

* Introduce decoupled translog manager interfaces

Signed-off-by: Bukhtawar Khan <[email protected]>

* Adding onNewCheckpoint to Start Replication on Replica Shard when Segment Replication is turned on (#3540)

* Adding onNewCheckpoint and it's test to start replication. SCheck for latestcheckpoint and replaying logic is removed from this commit and will be added in a different PR

Signed-off-by: Rishikesh1159 <[email protected]>

* Changing binding/inject logic and addressing comments from PR

Signed-off-by: Rishikesh1159 <[email protected]>

* Applying spotless check

Signed-off-by: Rishikesh1159 <[email protected]>

* Moving shouldProcessCheckpoint() to IndexShard, and removing some trace logs

Signed-off-by: Rishikesh1159 <[email protected]>

* applying spotlessApply

Signed-off-by: Rishikesh1159 <[email protected]>

* Adding more info to log statement in targetservice class

Signed-off-by: Rishikesh1159 <[email protected]>

* applying spotlessApply

Signed-off-by: Rishikesh1159 <[email protected]>

* Addressing comments on PR

Signed-off-by: Rishikesh1159 <[email protected]>

* Adding teardown() in SegmentReplicationTargetServiceTests.

Signed-off-by: Rishikesh1159 <[email protected]>

* fixing testShouldProcessCheckpoint() in SegmentReplicationTargetServiceTests

Signed-off-by: Rishikesh1159 <[email protected]>

* Removing CheckpointPublisherProvider in IndicesModule

Signed-off-by: Rishikesh1159 <[email protected]>

* spotless check apply

Signed-off-by: Rishikesh1159 <[email protected]>

* Remove class org.opensearch.action.support.master.AcknowledgedResponse (#3662)

* Remove class org.opensearch.action.support.master.AcknowledgedResponse

Signed-off-by: Tianli Feng <[email protected]>

* Remove class org.opensearch.action.support.master.AcknowledgedRequest RequestBuilder ShardsAcknowledgedResponse

Signed-off-by: Tianli Feng <[email protected]>

* Restore AcknowledgedResponse and AcknowledgedRequest to package org.opensearch.action.support.master (#3669)

Signed-off-by: Tianli Feng <[email protected]>

* [BUG] Custom POM configuration for ZIP publication produces duplicit tags (url, scm) (#3656)

* [BUG] Custom POM configuration for ZIP publication produces duplicit tags (url, scm)

Signed-off-by: Andriy Redko <[email protected]>

* Added test case for pluginZip with POM

Signed-off-by: Andriy Redko <[email protected]>

* Support both Gradle 6.8.x and Gradle 7.4.x

Signed-off-by: Andriy Redko <[email protected]>

* Adding 2.2.0 Bwc version to main (#3673)

* Upgraded to t-digest 3.3. (#3634)

* Revert renaming method onMaster() and offMaster() in interface LocalNodeMasterListener (#3686)

Signed-off-by: Tianli Feng <[email protected]>

* Upgrading AWS SDK dependency for native plugins (#3694)

* Merge branch 'feature/point_in_time' of https://github.com/opensearch-project/OpenSearch into fb

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Co-authored-by: Suraj Singh <[email protected]>
Co-authored-by: Marc Handalian <[email protected]>
Co-authored-by: Tianli Feng <[email protected]>
Co-authored-by: Andriy Redko <[email protected]>
Co-authored-by: Rabi Panda <[email protected]>
Co-authored-by: Nick Knize <[email protected]>
Co-authored-by: Poojita Raj <[email protected]>
Co-authored-by: Rishikesh Pasham <[email protected]>
Co-authored-by: Ankit Jain <[email protected]>
Co-authored-by: vpehkone <[email protected]>
Co-authored-by: sdp <[email protected]>
Co-authored-by: Kartik Ganesh <[email protected]>
Co-authored-by: Cole White <[email protected]>
Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>
Co-authored-by: opensearch-ci-bot <[email protected]>
Co-authored-by: Xue Zhou <[email protected]>
Co-authored-by: Sachin Kale <[email protected]>
Co-authored-by: Sachin Kale <[email protected]>
Co-authored-by: Xue Zhou <[email protected]>
Co-authored-by: Rishab Nahata <[email protected]>
Co-authored-by: Anshu Agarwal <[email protected]>
Co-authored-by: Yaliang Wu <[email protected]>
Co-authored-by: Kunal Kotwani <[email protected]>
Co-authored-by: George Apaaboah <[email protected]>
Co-authored-by: Ahmad AbuKhalil <[email protected]>
Co-authored-by: Bukhtawar Khan <[email protected]>
Co-authored-by: Sarat Vemulapalli <[email protected]>
Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch enhancement Enhancement or improvement to existing feature or request v2.1.0 Issues and PRs related to version 2.1.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants