Fix flaky test SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringFetchPhase

[stephen-crawford]

Description

This PR seeks to correct a flaky test SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringFetchPhase.
In the test, a multisearch REST request is created and then cancelled. The test sometimes fails on a runner because the latch had not been set to await for its countdown. By adding the latch.await(2, Seconds) statement the latch should force the following block

CountDownLatch latch = new CountDownLatch(1);
        AtomicReference<Exception> error = new AtomicReference<>();
        Cancellable cancellable = getRestClient().performRequestAsync(searchRequest, new ResponseListener() {
            @Override
            public void onSuccess(Response response) {
                latch.countDown();
            }

            @Override
            public void onFailure(Exception exception) {
                error.set(exception);
                latch.countDown();
            }
        });

        latch.await(2, SECONDS);

to perform the asynchronous request before continuing. This should mean that even on a faster runner, the request will have had to execute before the later assertion can be checked.

Related Issues

Resolves #7980

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/17221/
• CommitID: 163263f
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication

• URL: https://build.ci.opensearch.org/job/gradle-check/17222/
• CommitID: 0f43a5d
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #7985 (50cf7bc) into main (812b3e3) will increase coverage by 0.15%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main    #7985      +/-   ##
============================================
+ Coverage     70.84%   70.99%   +0.15%     
- Complexity    56452    56551      +99     
============================================
  Files          4714     4714              
  Lines        267213   267213              
  Branches      39182    39182              
============================================
+ Hits         189294   189709     +415     
+ Misses        61918    61438     -480     
- Partials      16001    16066      +65     

see 448 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      2 org.opensearch.indices.replication.RemoteStoreReplicationSourceTests.classMethod
      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchTaskCancellationWithHighCpu
      1 org.opensearch.indices.replication.RemoteStoreReplicationSourceTests.testGetCheckpointMetadata

• URL: https://build.ci.opensearch.org/job/gradle-check/17223/
• CommitID: 51f7fa8
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication
      1 org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure

• URL: https://build.ci.opensearch.org/job/gradle-check/17226/
• CommitID: 6576b0f
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu

• URL: https://build.ci.opensearch.org/job/gradle-check/17227/
• CommitID: 50cf7bc
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[stephen-crawford]

Gradle Check (Jenkins) Run Completed with:

* **RESULT:** UNSTABLE ❕

* **TEST FAILURES:**

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu

* **URL:** https://build.ci.opensearch.org/job/gradle-check/17227/

* **CommitID:** [50cf7bc](https://github.com/opensearch-project/OpenSearch/commit/50cf7bcf97ae40f7839d9cfec25d4daa1dd8434f)
  Please review all [flaky tests](https://github.com/opensearch-project/OpenSearch/blob/main/DEVELOPER_GUIDE.md#flaky-tests) that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Looks like the previous change to correct this test did not work...