Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[AUTOCUT] Gradle Check Flaky Test Report for RemoteSplitIndexIT #14296

Open
opensearch-ci-bot opened this issue Jun 13, 2024 · 4 comments · Fixed by #15107
Open

[AUTOCUT] Gradle Check Flaky Test Report for RemoteSplitIndexIT #14296

opensearch-ci-bot opened this issue Jun 13, 2024 · 4 comments · Fixed by #15107
Assignees
@gbbafna
Labels
autocut flaky-test Random test failure that succeeds on second run Storage:Remote Storage Issues and PRs relating to data and metadata storage >test-failure Test failure from CI, local build, etc.

Comments

@opensearch-ci-bot
Copy link
Collaborator

opensearch-ci-bot commented Jun 13, 2024 •
edited
Loading

Flaky Test Report for RemoteSplitIndexIT

Noticed the RemoteSplitIndexIT has some flaky, failing tests that failed during post-merge actions.

Details

Git Reference Merged Pull Request Build Details Test Name
0cd002d 15055 43778 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testCreateSplitIndex
1386a9b 13930 39885 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.classMethod
3387606 14954 43371 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testCreateSplitIndex
392e666 14487 41458 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testSplitIndexPrimaryTerm
3b0ea65 14863 42979 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testCreateSplitIndex
563375d 14187 41622 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.classMethod
5ad0f5d 14203 40762 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testCreateSplitIndex
5b19454 13801 39904 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testCreateSplitIndex
784f7d3 14214 40795 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.classMethod
887698d 15132 44111 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testCreateSplitIndex
8b10215 14869 43078 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testCreateSplitIndex
8f95735 15172 44301 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.classMethod
913013b 13948 39666 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.classMethod
a211f1d 14820 42761 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.classMethod
a6c86e7 13906 39544 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testCreateSplitIndex
af5a3d1 16147 48706 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testCreateSplitIndex
b9ca5a8 14027 40019 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.classMethod
bc39354 14124 40730 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testCreateSplitIndex
c1e138f 15376 45125 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testCreateSplitIndex
c38dfef 14195 40750 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testCreateSplitIndex
d2757f7 14250 40921 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testCreateSplitIndex
d4bdc7a 15029 44176 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testCreateSplitIndex
d56d8c8 14489 41572 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testCreateSplitIndex
ee17eca 14630 45612 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testCreateSplitIndex
f2fd804 14166 40663 org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testCreateSplitIndex

The other pull requests, besides those involved in post-merge actions, that contain failing tests with the RemoteSplitIndexIT class are:

For more details on the failed tests refer to OpenSearch Gradle Check Metrics dashboard.
@opensearch-ci-bot opensearch-ci-bot added >test-failure Test failure from CI, local build, etc. autocut untriaged labels Jun 13, 2024
@prudhvigodithi prudhvigodithi mentioned this issue Jun 13, 2024
Closed
@soosinha soosinha mentioned this issue Jun 14, 2024
Merged
3 tasks
@prudhvigodithi prudhvigodithi added the flaky-test Random test failure that succeeds on second run label Jun 14, 2024
@andrross
Copy link
Member

The "classMethod" failure appears to be due to a file leak:

RemoteSplitIndexIT > classMethod FAILED
    java.lang.RuntimeException: file handle leaks: [InputStream(/var/jenkins/workspace/gradle-check/search/server/build/testrun/internalClusterTest/temp/org.opensearch.action.admin.indices.create.RemoteSplitIndexIT_3726536651049A1A-001/tempDir-002/repos/cjWLgWIQEY/T11011111101111/6z6VohLmTKe07eW5Z98zQA/1/translog/data/2/translog-11.tlog)]
        at __randomizedtesting.SeedInfo.seed([3726536651049A1A]:0)
        at org.apache.lucene.tests.mockfile.LeakFS.onClose(LeakFS.java:63)
        at org.apache.lucene.tests.mockfile.FilterFileSystem.close(FilterFileSystem.java:69)
        at org.apache.lucene.tests.mockfile.FilterFileSystem.close(FilterFileSystem.java:70)
        at org.apache.lucene.tests.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:223)
        at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.****/java.lang.Thread.run(Thread.java:1583)

./gradlew ':server:internalClusterTest' --tests "org.opensearch.action.admin.indices.create.RemoteSplitIndexIT" -Dtests.seed=3726536651049A1A -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en -Dtests.timezone=Etc/UTC -Druntime.java=21

@andrross andrross added the Indexing Indexing, Bulk Indexing and anything related to indexing label Jun 17, 2024
@reta reta mentioned this issue Jun 19, 2024
Closed
@vikasvb90 vikasvb90 added Storage Issues and PRs relating to data and metadata storage Storage:Remote and removed Indexing Indexing, Bulk Indexing and anything related to indexing labels Jul 8, 2024
@Pranshu-S Pranshu-S mentioned this issue Jul 22, 2024
Merged
3 tasks
@rajiv-kv rajiv-kv mentioned this issue Jul 23, 2024
Merged
3 tasks
@gbbafna gbbafna self-assigned this Aug 5, 2024
@gbbafna gbbafna mentioned this issue Aug 5, 2024
Merged
3 tasks
@github-project-automation github-project-automation bot moved this from 🆕 New to ✅ Done in Storage Project Board Aug 9, 2024
@github-project-automation github-project-automation bot moved this from ✅ Done to 🏗 In progress in Storage Project Board Sep 6, 2024
@dblock dblock removed the untriaged label Sep 9, 2024
@dblock
Copy link
Member

dblock commented Sep 9, 2024

[Catch All Triage - 1, 2, 3, 4, 5]

@prachi-gaonkar
Copy link
Contributor

Hi Team,
we are also facing same issues with ppc64le

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testCreateSplitIndex" -Dtests.seed=71FF58527F2E975B -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=fr-CH -Dtests.timezone=Asia/Saigon -Druntime.java=21

RemoteSplitIndexIT > testCreateSplitIndex FAILED
UncategorizedExecutionException[Failed execution]; nested: IOException[Failed to upload 2 files during transfer];
at __randomizedtesting.SeedInfo.seed([71FF58527F2E975B:9BF8EE873AE5D59]:0)
at app//org.opensearch.action.support.AdapterActionFuture.unwrapEsException(AdapterActionFuture.java:102)
at app//org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:57)
at app//org.opensearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:73)
at app//org.opensearch.action.admin.indices.create.RemoteSplitIndexIT.testCreateSplitIndex(RemoteSplitIndexIT.java:414)

    Caused by:
    java.io.IOException: Failed to upload 2 files during transfer
        at org.opensearch.index.translog.transfer.TranslogTransferManager.transferSnapshot(TranslogTransferManager.java:199)
        at org.opensearch.index.translog.RemoteFsTranslog.upload(RemoteFsTranslog.java:426)
        at org.opensearch.index.translog.RemoteFsTranslog.prepareAndUpload(RemoteFsTranslog.java:409)
        at org.opensearch.index.translog.RemoteFsTranslog.ensureSynced(RemoteFsTranslog.java:341)
        at org.opensearch.index.translog.Translog.ensureSynced(Translog.java:837)
        at org.opensearch.index.translog.InternalTranslogManager.ensureTranslogSynced(InternalTranslogManager.java:184)
        at org.opensearch.index.engine.InternalEngine.ensureTranslogSynced(InternalEngine.java:605)
        at org.opensearch.index.shard.IndexShard.lambda$createTranslogSyncProcessor$44(IndexShard.java:4441)
        at org.opensearch.index.shard.IndexShard$6.write(IndexShard.java:4455)

@prachi-gaonkar
Copy link
Contributor

Hi Team
is there any update on this issue?

@peteralfonsi peteralfonsi mentioned this issue Dec 17, 2024
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gbbafna gbbafna

Labels
autocut flaky-test Random test failure that succeeds on second run Storage:Remote Storage Issues and PRs relating to data and metadata storage >test-failure Test failure from CI, local build, etc.
Projects
Storage Project Board
Status: 🏗 In progress
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Flaky test : Don't use async repo for SplitIndex and wait for translo… gbbafna/OpenSearch
8 participants
@dblock @peternied @andrross @prudhvigodithi @vikasvb90 @opensearch-ci-bot @gbbafna @prachi-gaonkar