Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Flaky test failure org.opensearch.index.WaitUntilRefreshIT.classMethod #7103

Closed
dreamer-89 opened this issue Apr 11, 2023 · 3 comments
Closed

[BUG] Flaky test failure org.opensearch.index.WaitUntilRefreshIT.classMethod #7103

dreamer-89 opened this issue Apr 11, 2023 · 3 comments
Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run Indexing:Replication Issues and PRs related to core replication framework eg segrep >test-failure Test failure from CI, local build, etc.

Comments

@dreamer-89
Copy link
Member

Gradle link

https://build.ci.opensearch.org/job/gradle-check/13858

Identified in PR

#7094 (comment)

Failure trace

org.opensearch.index.WaitUntilRefreshIT > classMethod FAILED
    com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=574, name=opensearch[node_s0][generic][T#7], state=RUNNABLE, group=TGRP-WaitUntilRefreshIT]

        Caused by:
        org.opensearch.core.concurrency.OpenSearchRejectedExecutionException: rejected execution of java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@77e3f6a0[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@53ad6424[Wrapped task = [threaded] fail_stale_replica]] on org.opensearch.threadpool.Scheduler$SafeScheduledThreadPoolExecutor@9ec95c2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 463]
            at __randomizedtesting.SeedInfo.seed([8BFF65004B7D5420]:0)
            at app//org.opensearch.common.util.concurrent.OpenSearchAbortPolicy.rejectedExecution(OpenSearchAbortPolicy.java:67)
            at java.****@19.0.2/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:841)
            at java.****@19.0.2/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
            at java.****@19.0.2/java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562)
            at app//org.opensearch.threadpool.ThreadPool.schedule(ThreadPool.java:421)
            at app//org.opensearch.common.util.concurrent.AbstractAsyncTask.rescheduleIfNecessary(AbstractAsyncTask.java:109)
            at app//org.opensearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:174)
            at app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)
            at java.****@19.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
            at java.****@19.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
            at java.****@19.0.2/java.lang.Thread.run(Thread.java:1589)
@dreamer-89 dreamer-89 added bug Something isn't working >test-failure Test failure from CI, local build, etc. untriaged flaky-test Random test failure that succeeds on second run labels Apr 11, 2023
@sohami sohami mentioned this issue Apr 18, 2023
Merged
6 tasks
@mch2 mch2 removed the untriaged label May 9, 2023
@reta
Copy link
Collaborator

reta commented Jun 7, 2023

Another variation here [1]:

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1037, name=opensearch[node_s2][generic][T#7], state=RUNNABLE, group=TGRP-SearchFieldsIT]
Caused by: org.opensearch.core.concurrency.OpenSearchRejectedExecutionException: rejected execution of java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@6ee84ba2[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@2526cc0a[Wrapped task = [threaded] fail_stale_replica]] on org.opensearch.threadpool.Scheduler$SafeScheduledThreadPoolExecutor@24c49208[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 181]
	at __randomizedtesting.SeedInfo.seed([37F8D9EE5752E8C5]:0)
	at app//org.opensearch.common.util.concurrent.OpenSearchAbortPolicy.rejectedExecution(OpenSearchAbortPolicy.java:67)
	at [email protected]/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:841)
	at [email protected]/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
	at [email protected]/java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562)
	at app//org.opensearch.threadpool.ThreadPool.schedule(ThreadPool.java:438)
	at app//org.opensearch.common.util.concurrent.AbstractAsyncTask.rescheduleIfNecessary(AbstractAsyncTask.java:109)
	at app//org.opensearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:174)
	at app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)
	at [email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at [email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at [email protected]/java.lang.Thread.run(Thread.java:1623)

[1] https://build.ci.opensearch.org/job/gradle-check/17035/testReport/junit/org.opensearch.search.fields/SearchFieldsIT/classMethod/

@reta reta mentioned this issue Jun 7, 2023
Merged
@sejli
Copy link
Member

sejli commented Oct 17, 2023

@MightyTQ, could you pick this up for OSCI? Thanks!

@andrross andrross added Indexing:Replication Issues and PRs related to core replication framework eg segrep and removed distributed framework labels Feb 21, 2024
@dhwanilpatel
Copy link
Contributor

Ran with 500 Iterations, it didn't failed. I see there is not any recent PR has refereed this failure as well.

Closing the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run Indexing:Replication Issues and PRs related to core replication framework eg segrep >test-failure Test failure from CI, local build, etc.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@reta @nknize @mch2 @dhwanilpatel @andrross @saratvemulapalli @sejli @dreamer-89