Extend existing IndexRecoveryIT for remote indexes

[ashking94]

Description

Add integration tests for recovery flow for remote enabled indexes by extending existing recovery indexing integ tests.

Related Issues

Resolves #8476 and #8485

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testPrimaryRelocationWithSegRepFailure

• URL: https://build.ci.opensearch.org/job/gradle-check/19323/
• CommitID: 444731c
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[gbbafna]

Since slowDownRecovery doesn't work for remote backed indices, we won't be getting much value from the tests at this point of time . can we please override this function in RemoteReindexRecovery and add a backlog item to control the recovery speed for remote backed indices ?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19362/
• CommitID: 05aaf4b
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[codecov]

Codecov Report

Merging #8505 (24827ef) into main (9fd1ddb) will decrease coverage by 0.01%.
The diff coverage is 0.00%.

❗ Current head 24827ef differs from pull request most recent head d1cd823. Consider uploading reports for the commit d1cd823 to get more accurate results

@@             Coverage Diff              @@
##               main    #8505      +/-   ##
============================================
- Coverage     70.90%   70.89%   -0.01%     
- Complexity    56961    56973      +12     
============================================
  Files          4758     4758              
  Lines        269362   269363       +1     
  Branches      39408    39409       +1     
============================================
- Hits         190980   190964      -16     
- Misses        62249    62301      +52     
+ Partials      16133    16098      -35     

Impacted Files	Coverage Δ
...n/java/org/opensearch/index/engine/NoOpEngine.java	62.66% <0.00%> (-2.20%)	⬇️

... and 439 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication
      1 org.opensearch.gateway.RecoveryFromGatewayIT.testReuseInFileBasedPeerRecovery

• URL: https://build.ci.opensearch.org/job/gradle-check/19495/
• CommitID: 848df5a
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[ashking94]

The ITs added as part of this PR have been run successfully for 200 iterations.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure

• URL: https://build.ci.opensearch.org/job/gradle-check/19503/
• CommitID: 6f9530d
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/19559/
• CommitID: 3e582ec
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/19560/
• CommitID: e5c7177

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/19561/
• CommitID: 24827ef

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationIT.testDropPrimaryDuringReplication

• URL: https://build.ci.opensearch.org/job/gradle-check/19570/
• CommitID: d1cd823
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.