Fix bugs causing red indexes with remote indexes during translog upload & store recovery

[ashking94]

Description

This PR fixes 2 issues which are causing red indexes while I performed stress testing -

1. [BUG] Store recovery keeps failing for remote indexes if the remote store interaction fails and the shard is red (TranslogCorruptedException) #10400
2. [BUG] Shard failure due to fail engine during trimUnreferencedTranslogFiles in InternalEngine for Remote indexes #10398

Related Issues

Resolves #10398, #10400.

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change 97e5b34

Incompatible components

Incompatible components: [https://github.com/opensearch-project/security-analytics.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27261/
• CommitID: 38dc464
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[ashking94]

These issues have been found while doing stress testing and It's a bit tricky to simulate these kind of failures in integ tests. Still exploring if there is any way to write an IT for the above fixes.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27384/
• CommitID: 2b17e06
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[sachinpkale]

Please add ITs around the changes.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testSearchAggregationWithNetworkDisruption_FailOpenEnabled
      1 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

• URL: https://build.ci.opensearch.org/job/gradle-check/27388/
• CommitID: 2b17e06
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #10449 (97e5b34) into main (2bf63c9) will increase coverage by 0.08%.
Report is 3 commits behind head on main.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##               main   #10449      +/-   ##
============================================
+ Coverage     71.17%   71.26%   +0.08%     
- Complexity    58371    58425      +54     
============================================
  Files          4843     4843              
  Lines        275264   275266       +2     
  Branches      40076    40076              
============================================
+ Hits         195928   196172     +244     
+ Misses        62882    62663     -219     
+ Partials      16454    16431      -23     

Files	Coverage Δ
...c/main/java/org/opensearch/index/IndexService.java	75.86% <ø> (+0.43%)	⬆️
...in/java/org/opensearch/index/shard/IndexShard.java	69.63% <0.00%> (-0.13%)	⬇️

... and 492 files with indirect coverage changes

[ashking94]

Please add ITs around the changes.

@sachinpkale Done. Pls take a relook. Ty!

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27395/
• CommitID: 4657a37
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[ashking94]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27395/
• CommitID: 4657a37
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

Failures -

org.opensearch.script.expression.MoreExpressionIT.testSpecialValueVariable {p0={"search.concurrent_segment_search.enabled":"true"}}
org.opensearch.index.IndexServiceTests.testAsyncTranslogTrimTaskOnClosedIndex

Of the above, #10079 is already a reported flaky test.
On the failure testAsyncTranslogTrimTaskOnClosedIndex, I have made no changes that should affect the specific test. The change made here is only applicable for remote indexes and does not affect the IndexServiceTests.testAsyncTranslogTrimTaskOnClosedIndex test. I am running the specific test on main to see if this fails after high iteration count or low. I have already this test now for more than 250+ iterations with this change and I dont see the failure yet.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27399/
• CommitID: e0c59a6
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[ashking94]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27399/
• CommitID: e0c59a6
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

Flaky tests - #10154, #10046

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/27401/
• CommitID: 97e5b34