Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] org.opensearch.search.profile.query.QueryProfilerTests.testBasic {p0=5} is flaky #10941

Open
ashking94 opened this issue Oct 26, 2023 · 3 comments · Fixed by #11397
Open
Assignees
Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run Severity-Major

Comments

@ashking94
Copy link
Member

Describe the bug
org.opensearch.search.profile.query.QueryProfilerTests.testBasic {p0=5} test is flaky.

Below assertion failure can be seen -

java.lang.AssertionError: 
Expected: a value greater than <0L>
     but: <0L> was equal to <0L>

To Reproduce
https://build.ci.opensearch.org/job/gradle-check/29044/testReport/junit/org.opensearch.search.profile.query/QueryProfilerTests/testBasic__p0_5_/

Expected behavior
Test must always pass.

@ashking94 ashking94 added bug Something isn't working untriaged labels Oct 26, 2023
@saratvemulapalli saratvemulapalli added the flaky-test Random test failure that succeeds on second run label Oct 26, 2023
@nknize
Copy link
Collaborator

nknize commented Nov 2, 2023

Similar reproducible error:

./gradlew ':server:test' --tests "org.opensearch.search.profile.query.QueryProfilerTests.testApproximations {p0=5}" -Dtests.seed=D11999824BA0CE71 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=zh-CN -Dtests.timezone=Africa/Bamako
  2> java.lang.AssertionError: 
    Expected: a value greater than <0L>
         but: <0L> was equal to <0L>
        at __randomizedtesting.SeedInfo.seed([D11999824BA0CE71:14FADAE707F9B355]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:964)
        at org.junit.Assert.assertThat(Assert.java:930)
        at org.opensearch.search.profile.query.QueryProfilerTests.testApproximations(QueryProfilerTests.java:349)

This, by definition, is a bug not a flaky test. A flaky test isn't reproducible. This reproduces on all supported jdks.

@nknize nknize added Severity-Major and removed flaky-test Random test failure that succeeds on second run labels Nov 2, 2023
@nknize
Copy link
Collaborator

nknize commented Nov 2, 2023

I removed the flaky test label and added a severity. If I have some cycles I'll look at this but it would be better for someone else to take it.

@peternied peternied added flaky-test Random test failure that succeeds on second run and removed untriaged labels Nov 30, 2023
@github-project-automation github-project-automation bot moved this from Todo to Done in Concurrent Search Nov 30, 2023
@dblock
Copy link
Member

dblock commented Apr 24, 2024

Happened again in #13358, reopening.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run Severity-Major
Projects
Status: In Progress
Development

Successfully merging a pull request may close this issue.

6 participants