[BUG] Flaky test org.opensearch.discovery.ClusterManagerDisruptionIT.testIsolateClusterManagerAndVerifyClusterStateConsensus

[Poojita-Raj]

Describe the bug

Found this flaky test on gradle check:
org.opensearch.discovery.ClusterManagerDisruptionIT.testIsolateClusterManagerAndVerifyClusterStateConsensus

https://build.ci.opensearch.org/job/gradle-check/32905/

Related component

Cluster Manager

To Reproduce

Might show up on running gradle check

Expected behavior

Expect test to pass

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.discovery.ClusterManagerDisruptionIT.testIsolateClusterManagerAndVerifyClusterStateConsensus" -Dtests.seed=CA2671C1619AE45E -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=da-DK -Dtests.timezone=America/Lima -Druntime.java=21

org.opensearch.discovery.ClusterManagerDisruptionIT > testIsolateClusterManagerAndVerifyClusterStateConsensus FAILED
    java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([CA2671C1619AE45E:7556C7CBC3B6BC4F]:0)
        at org.junit.Assert.fail(Assert.java:87)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.junit.Assert.assertTrue(Assert.java:53)
        at org.opensearch.discovery.ClusterManagerDisruptionIT.lambda$testIsolateClusterManagerAndVerifyClusterStateConsensus$0(ClusterManagerDisruptionIT.java:204)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1089)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1062)
        at org.opensearch.discovery.ClusterManagerDisruptionIT.testIsolateClusterManagerAndVerifyClusterStateConsensus(ClusterManagerDisruptionIT.java:167)

[peternied]

[Triage - attendees 1 2 3 4 5 6 7 8]
@Poojita-Raj thanks for filing, we'd gladly review a pull request

[shwetathareja]

Able to consistently repro the test failure with

./gradlew ':server:internalClusterTest' --tests "org.opensearch.discovery.ClusterManagerDisruptionIT.testIsolateClusterManagerAndVerifyClusterStateConsensus" -Dtests.seed=CA2671C1619AE45E -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=da-DK -Dtests.timezone=America/Lima

Issue is that in test

OpenSearch/server/src/internalClusterTest/java/org/opensearch/discovery/ClusterManagerDisruptionIT.java

Lines 203 to 204 in 5db84d1

	ClusterStateStats clusterStateStats = internalCluster().clusterService().getClusterManagerService().getClusterStateStats();
	assertTrue(clusterStateStats.getUpdateFailed() > 0);

It is picking a random node as specific node is not provided in internalCluster().clusterService() which results in the random behavior.

isolatedNode should be passed here as internalCluster().clusterService(isolatedNode) so that it deterministically checks this node for clusterState update failed count. The test is passing locally consistently. will raise the PR.

[rajiv-kv]

[Triage Attendees - 1, 2, 3]
Fails on verification of failed stats on Isolated node
Next Steps

• Verify if the cluster state is guaranteed to be updated after the isolation