[Backport 2.x] Segment Replication - Fix ShardLockObtained error during corruption cases #10370

[mch2]

Manual backport of #10370 to 2.x

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27193/
• CommitID: 83883b4
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Compatibility status:

Checks if related components are compatible with change 13299ab

Incompatible components

Incompatible components: [https://github.com/opensearch-project/security.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.test.rest.ClientYamlTestSuiteIT.test {p0=search.aggregation/20_terms/string profiler via global ordinals}
      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

• URL: https://build.ci.opensearch.org/job/gradle-check/27197/
• CommitID: 13299ab
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #10418 (13299ab) into 2.x (fc7dc20) will increase coverage by 0.12%.
Report is 7 commits behind head on 2.x.
The diff coverage is 77.07%.

@@             Coverage Diff              @@
##                2.x   #10418      +/-   ##
============================================
+ Coverage     70.75%   70.88%   +0.12%     
- Complexity    58397    58544     +147     
============================================
  Files          4816     4829      +13     
  Lines        276031   276355     +324     
  Branches      40565    40576      +11     
============================================
+ Hits         195306   195891     +585     
+ Misses        64105    63765     -340     
- Partials      16620    16699      +79     

Files	Coverage Δ
...arch/telemetry/metrics/DefaultMetricsRegistry.java	100.00% <100.00%> (ø)
...ch/telemetry/metrics/noop/NoopMetricsRegistry.java	100.00% <100.00%> (ø)
...va/org/opensearch/telemetry/metrics/tags/Tags.java	100.00% <100.00%> (ø)
...org/opensearch/repositories/url/URLRepository.java	72.00% <100.00%> (ø)
...g/opensearch/transport/netty4/Netty4Transport.java	73.65% <100.00%> (ø)
...opensearch/repositories/azure/AzureRepository.java	68.88% <100.00%> (-1.33%)	⬇️
...g/opensearch/repositories/hdfs/HdfsRepository.java	61.90% <100.00%> (ø)
...java/org/opensearch/repositories/s3/S3Service.java	75.00% <100.00%> (-1.05%)	⬇️
.../org/opensearch/telemetry/OTelTelemetryPlugin.java	100.00% <100.00%> (ø)
...rg/opensearch/telemetry/OTelTelemetrySettings.java	100.00% <100.00%> (ø)
... and 43 more

... and 501 files with indirect coverage changes