[Meta] Segment Replication flaky test failures

[dreamer-89]

Coming from #4959, which enables yml based feature flag settings; reveals flakyness in Segment replication tests.

• testAddNewReplicaFailure

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.replication.SegmentReplicationIT.testAddNewReplicaFailure" -Dtests.seed=D3DBF1C239661FF6 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=it -Dtests.timezone=Brazil/West -Druntime.java=19

org.opensearch.indices.replication.SegmentReplicationIT > testAddNewReplicaFailure FAILED
    java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([D3DBF1C239661FF6:F35C3295F5C37311]:0)
        at org.junit.Assert.fail(Assert.java:87)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.junit.Assert.assertFalse(Assert.java:65)
        at org.junit.Assert.assertFalse(Assert.java:75)
        at org.opensearch.indices.replication.SegmentReplicationIT.testAddNewReplicaFailure(SegmentReplicationIT.java:267)

• testSnapshotOnSegRep_RestoreOnSegRepDuringIngestion

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.snapshots.SegmentReplicationSnapshotIT.testSnapshotOnSegRep_RestoreOnSegRepDuringIngestion" -Dtests.seed=857112ECE4907AD3 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=mk-MK -Dtests.timezone=Africa/Porto-Novo -Druntime.java=19

org.opensearch.snapshots.SegmentReplicationSnapshotIT > testSnapshotOnSegRep_RestoreOnSegRepDuringIngestion FAILED
    java.lang.AssertionError: timed out waiting for green state
        at __randomizedtesting.SeedInfo.seed([857112ECE4907AD3:AAFEB48B53C084CE]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureColor(OpenSearchIntegTestCase.java:1001)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureGreen(OpenSearchIntegTestCase.java:932)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureGreen(OpenSearchIntegTestCase.java:921)
        at org.opensearch.snapshots.SegmentReplicationSnapshotIT.testSnapshotOnSegRep_RestoreOnSegRepDuringIngestion(SegmentReplicationSnapshotIT.java:180)

• testReplicationAfterPrimaryRefreshAndFlush

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.replication.SegmentReplicationIT.testReplicationAfterPrimaryRefreshAndFlush" -Dtests.seed=857112ECE4907AD3 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-SY -Dtests.timezone=Antarctica/McMurdo -Druntime.java=19

org.opensearch.indices.replication.SegmentReplicationIT > testReplicationAfterPrimaryRefreshAndFlush FAILED
    java.lang.AssertionError: Count is 0 hits but 73 was expected.  Total shards: 1 Successful shards: 1 & 0 shard failures:
        at __randomizedtesting.SeedInfo.seed([857112ECE4907AD3:6CAB602C42AF5D02]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.test.hamcrest.OpenSearchAssertions.assertHitCount(OpenSearchAssertions.java:303)
        at org.opensearch.indices.replication.SegmentReplicationIT.testReplicationAfterPrimaryRefreshAndFlush(SegmentReplicationIT.java:296)

For tracking added to meta #1715

• org.opensearch.indices.replication.SegmentReplicationIT.testStartReplicaAfterPrimaryIndexesDocs
  Update nebula-publishing-plugin to 19.2.0 #5704 (comment)

• org.opensearch.indices.replication.SegmentReplicationIT.testRestartPrimary
  Fix flaky Segment Replication test testStartReplicaAfterPrimaryIndexesDocs. #5722 (comment)

• [BUG] failing IT test : SegmentReplicationRelocationIT  #6065

• ClientTimeoutIT.testSegmentReplicationStatsWithTimeout test failure #6259

• testSegmentReplicationStatsResponse flaky failure [Segment Replication] Wait for segment replication to be completed and marked done before assertion #6370

[mch2]

All currently linked flaky tests are resolved - If there are any more SR related tests lets pls open separate issues for them.