Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure is flaky #7703

Closed
reta opened this issue May 23, 2023 · 5 comments · Fixed by #7879 or #8547
Assignees
Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run Severity-Blocker Storage:Durability Issues and PRs related to the durability framework v2.9.0 'Issues and PRs related to version v2.9.0'

Comments

@reta
Copy link
Collaborator

reta commented May 23, 2023

Describe the bug
org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure is flaky

java.lang.AssertionError
	at org.junit.Assert.fail(Assert.java:87)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.junit.Assert.assertTrue(Assert.java:53)
	at org.opensearch.remotestore.RemoteStoreRefreshListenerIT.lambda$testRemoteRefreshRetryOnFailure$0(RemoteStoreRefreshListenerIT.java:50)
	at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1084)
	at org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure(RemoteStoreRefreshListenerIT.java:47)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:1589)
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.remotestore.RemoteStoreRefreshListenerIT.lambda$testRemoteRefreshRetryOnFailure$0(RemoteStoreRefreshListenerIT.java:50)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 39 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.remotestore.RemoteStoreRefreshListenerIT.lambda$testRemoteRefreshRetryOnFailure$0(RemoteStoreRefreshListenerIT.java:50)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 39 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.remotestore.RemoteStoreRefreshListenerIT.lambda$testRemoteRefreshRetryOnFailure$0(RemoteStoreRefreshListenerIT.java:50)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 39 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.remotestore.RemoteStoreRefreshListenerIT.lambda$testRemoteRefreshRetryOnFailure$0(RemoteStoreRefreshListenerIT.java:50)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 39 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.remotestore.RemoteStoreRefreshListenerIT.lambda$testRemoteRefreshRetryOnFailure$0(RemoteStoreRefreshListenerIT.java:50)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 39 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.remotestore.RemoteStoreRefreshListenerIT.lambda$testRemoteRefreshRetryOnFailure$0(RemoteStoreRefreshListenerIT.java:50)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 39 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.remotestore.RemoteStoreRefreshListenerIT.lambda$testRemoteRefreshRetryOnFailure$0(RemoteStoreRefreshListenerIT.java:50)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 39 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.remotestore.RemoteStoreRefreshListenerIT.lambda$testRemoteRefreshRetryOnFailure$0(RemoteStoreRefreshListenerIT.java:50)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 39 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.remotestore.RemoteStoreRefreshListenerIT.lambda$testRemoteRefreshRetryOnFailure$0(RemoteStoreRefreshListenerIT.java:50)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 39 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.remotestore.RemoteStoreRefreshListenerIT.lambda$testRemoteRefreshRetryOnFailure$0(RemoteStoreRefreshListenerIT.java:50)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 39 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.remotestore.RemoteStoreRefreshListenerIT.lambda$testRemoteRefreshRetryOnFailure$0(RemoteStoreRefreshListenerIT.java:50)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 39 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.remotestore.RemoteStoreRefreshListenerIT.lambda$testRemoteRefreshRetryOnFailure$0(RemoteStoreRefreshListenerIT.java:50)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 39 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.remotestore.RemoteStoreRefreshListenerIT.lambda$testRemoteRefreshRetryOnFailure$0(RemoteStoreRefreshListenerIT.java:50)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 39 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.remotestore.RemoteStoreRefreshListenerIT.lambda$testRemoteRefreshRetryOnFailure$0(RemoteStoreRefreshListenerIT.java:50)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 39 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.remotestore.RemoteStoreRefreshListenerIT.lambda$testRemoteRefreshRetryOnFailure$0(RemoteStoreRefreshListenerIT.java:50)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 39 more
	Suppressed: java.lang.AssertionError
		at org.junit.Assert.fail(Assert.java:87)
		at org.junit.Assert.assertTrue(Assert.java:42)
		at org.junit.Assert.assertTrue(Assert.java:53)
		at org.opensearch.remotestore.RemoteStoreRefreshListenerIT.lambda$testRemoteRefreshRetryOnFailure$0(RemoteStoreRefreshListenerIT.java:50)
		at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1072)
		... 39 more

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior
The test should pass reliably

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

  • CI

Additional context
https://build.ci.opensearch.org/job/gradle-check/16080/testReport/junit/org.opensearch.remotestore/RemoteStoreRefreshListenerIT/testRemoteRefreshRetryOnFailure/

@reta reta added bug Something isn't working untriaged flaky-test Random test failure that succeeds on second run labels May 23, 2023
@sachinpkale sachinpkale added Storage:Durability Issues and PRs related to the durability framework v2.8.0 'Issues and PRs related to version v2.8.0' and removed untriaged labels May 24, 2023
ashking94 added a commit to ashking94/OpenSearch that referenced this issue Jun 2, 2023
@ashking94
Copy link
Member

There are other exception stack traces as well that I can see -

  2> REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure {seed=[B4035D2FFC2EAD3C:6BBEFB8675A3EC01]}" -Dtests.seed=B4035D2FFC2EAD3C -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-VE -Dtests.timezone=Asia/Kuwait -Druntime.java=19
  2> REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure {seed=[B4035D2FFC2EAD3C:6BBEFB8675A3EC01]}" -Dtests.seed=B4035D2FFC2EAD3C -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-VE -Dtests.timezone=Asia/Kuwait -Druntime.java=19
  2> java.lang.AssertionError
        at org.junit.Assert.fail(Assert.java:87)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.junit.Assert.assertTrue(Assert.java:53)
        at org.opensearch.remotestore.RemoteStoreRefreshListenerIT.lambda$testRemoteRefreshRetryOnFailure$0(RemoteStoreRefreshListenerIT.java:50)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1084)
        at org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure(RemoteStoreRefreshListenerIT.java:47)

    java.lang.AssertionError: java.lang.AssertionError: Could not find index.latest blob for repo [BlobStoreRepository[[my-segment-repo-1], [org.opensearch.snapshots.mockstore.MockRepository$MockBlobStore@1e3751bf]]]
        at org.opensearch.repositories.blobstore.BlobStoreTestUtil.assertConsistency(BlobStoreTestUtil.java:154)
        at org.opensearch.repositories.blobstore.BlobStoreTestUtil.assertRepoConsistency(BlobStoreTestUtil.java:113)
        at org.opensearch.snapshots.AbstractSnapshotIntegTestCase.lambda$assertRepoConsistency$1(AbstractSnapshotIntegTestCase.java:157)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
        at org.opensearch.snapshots.AbstractSnapshotIntegTestCase.assertRepoConsistency(AbstractSnapshotIntegTestCase.java:151)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        at java.base/java.lang.reflect.Method.invoke(Method.java:578)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.base/java.lang.Thread.run(Thread.java:1589)

        Caused by:
        java.lang.AssertionError: Could not find index.latest blob for repo [BlobStoreRepository[[my-segment-repo-1], [org.opensearch.snapshots.mockstore.MockRepository$MockBlobStore@1e3751bf]]]
            at org.opensearch.repositories.blobstore.BlobStoreTestUtil.lambda$assertConsistency$0(BlobStoreTestUtil.java:133)
            at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:73)
            at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88)
            at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806)
            at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
            ... 1 more

gbbafna pushed a commit that referenced this issue Jun 2, 2023
opensearch-trigger-bot bot pushed a commit that referenced this issue Jun 2, 2023
Signed-off-by: Ashish Singh <[email protected]>
(cherry picked from commit 5d5e8ad)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
gbbafna pushed a commit that referenced this issue Jun 2, 2023
(cherry picked from commit 5d5e8ad)

Signed-off-by: Ashish Singh <[email protected]>
@dblock
Copy link
Member

dblock commented Jun 8, 2023

Still flaky in #7969 (comment)

@ashking94
Copy link
Member

I will be prioritising this issue, will try to get this closed before 2.9 release.

@kotwanikunal
Copy link
Member

@ashking94 I think the only issue is that the IOException from MockRepository marks the segment file as failed and it is not uploaded until another refresh kicks in.

I tried adding a force refresh operation here and the test seemed to succeed for 500 times without any issues.

@ashking94
Copy link
Member

ashking94 commented Jul 8, 2023

@kotwanikunal Thanks for the investigation. The tests are failing due to segments have not been uploaded to remote store completely yet. In this case, we are intentionally failing the uploads and have retry mechanism which will ultimately get the remote store in sync with local segments. Doing a force refresh is not recommended as it triggers refresh internally and hence it would make the upload go through ultimately. For now, I am muting the test. I should be able to get the fix real soon.

shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this issue Apr 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run Severity-Blocker Storage:Durability Issues and PRs related to the durability framework v2.9.0 'Issues and PRs related to version v2.9.0'
Projects
None yet
7 participants