[Remote Store] Fix relocation failure due to transport receive timeout

[ashking94]

Description

This PR solves Peer Recovery failure that can happen due to receive timeout exceptions during prepare translog phase of peer recovery. This type of failure can be prominent in zero replica remote store backed indexes. However this can also happen during slow remote interactions. This PR fixes the problem handles both kind of issues.

This PR handles the following cases -

• Removes searchIdle support for remote enabled indexes. This means that we continue to refresh and upload segments periodically at the configured refresh interval.
• Fixes the bug where actual translog retention was higher than the translog flush threshold size.
• Makes the transport timeout configurable for prepare translog phase in peer recovery. This is now controlled through the indices.recovery.internal_action_long_timeout index setting. This, by default, is 30 mins.

Related Issues

Resolves #10727

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[ashking94]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/28493/
• CommitID: 62aeb2d
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

Rebasing with main and retrying.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/28551/
• CommitID: eda5b2f
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/28569/
• CommitID: 9e34d1e
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/28570/
• CommitID: bbc09be
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/28572/
• CommitID: 9589791

[codecov]

Codecov Report

Merging #10761 (9589791) into main (200ad5d) will increase coverage by 0.13%.
Report is 1 commits behind head on main.
The diff coverage is 78.57%.

@@             Coverage Diff              @@
##               main   #10761      +/-   ##
============================================
+ Coverage     71.12%   71.25%   +0.13%     
- Complexity    58545    58659     +114     
============================================
  Files          4859     4859              
  Lines        276252   276280      +28     
  Branches      40191    40196       +5     
============================================
+ Hits         196473   196872     +399     
+ Misses        63347    63016     -331     
+ Partials      16432    16392      -40     

Files	Coverage Δ
...in/java/org/opensearch/index/shard/IndexShard.java	70.10% <100.00%> (+0.49%)	⬆️
...search/index/translog/InternalTranslogManager.java	66.66% <100.00%> (-4.57%)	⬇️
...rg/opensearch/index/translog/RemoteFsTranslog.java	74.50% <100.00%> (+1.13%)	⬆️
...n/java/org/opensearch/index/translog/Translog.java	80.51% <100.00%> (-0.11%)	⬇️
.../indices/replication/common/ReplicationTarget.java	81.25% <100.00%> (+2.30%)	⬆️
...ch/indices/recovery/PeerRecoverySourceService.java	61.93% <50.00%> (+6.74%)	⬆️
.../main/java/org/opensearch/index/IndexSettings.java	86.40% <0.00%> (-0.39%)	⬇️
.../indices/recovery/RemoteRecoveryTargetHandler.java	89.70% <50.00%> (-2.61%)	⬇️
...nsearch/index/store/RemoteStoreFileDownloader.java	92.15% <84.00%> (-0.87%)	⬇️
...ices/replication/RemoteStoreReplicationSource.java	85.24% <75.00%> (-5.83%)	⬇️

... and 461 files with indirect coverage changes

[Bukhtawar]

LGTM