[BUG] org.opensearch.search.suggest.CompletionSuggestSearchIT.testSkipDuplicates is flaky #10798

reta · 2023-10-20T17:44:47Z

Describe the bug

The test case org.opensearch.search.suggest.CompletionSuggestSearchIT.testSkipDuplicates is flaky

org.opensearch.search.suggest.CompletionSuggestSearchIT.testSkipDuplicates {p0={"search.concurrent_segment_search.enabled":"false"}}

java.lang.AssertionError: Expected elem 0 in list [suggestion-004, suggestion-005, suggestion-002, suggestion-001, suggestion-003, suggestion-000] to be [suggestion-003] score: 96.0
Expected: is "suggestion-003"
     but: was "suggestion-004"
	at __randomizedtesting.SeedInfo.seed([16FA83231CDA278C:C1B899A779B737AB]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:964)
	at org.opensearch.search.suggest.CompletionSuggestSearchIT.assertSuggestions(CompletionSuggestSearchIT.java:1288)
	at org.opensearch.search.suggest.CompletionSuggestSearchIT.testSkipDuplicates(CompletionSuggestSearchIT.java:1210)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:833)

org.opensearch.search.suggest.CompletionSuggestSearchIT.testSkipDuplicates {p0={"search.concurrent_segment_search.enabled":"true"}}

java.lang.AssertionError: Expected elem 3 in list [suggestion-003, suggestion-004, suggestion-005, suggestion-001, suggestion-002, suggestion-000] to be [suggestion-002] score: 60.0
Expected: is "suggestion-002"
     but: was "suggestion-001"
	at __randomizedtesting.SeedInfo.seed([16FA83231CDA278C:C1B899A779B737AB]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:964)
	at org.opensearch.search.suggest.CompletionSuggestSearchIT.assertSuggestions(CompletionSuggestSearchIT.java:1288)
	at org.opensearch.search.suggest.CompletionSuggestSearchIT.testSkipDuplicates(CompletionSuggestSearchIT.java:1210)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:833)

To Reproduce

 ./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.suggest.CompletionSuggestSearchIT" -Dtests.method="testSkipDuplicates {p0={"search.concurrent_segment_search.enabled":"false"}}" -Dtests.seed=16FA83231CDA278C 

./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.suggest.CompletionSuggestSearchIT" -Dtests.method="testSkipDuplicates {p0={"search.concurrent_segment_search.enabled":"false"}}" -Dtests.seed=16FA83231CDA278C

Expected behavior
The test should always pass

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

CI

Additional context

https://build.ci.opensearch.org/job/gradle-check/28607/

The text was updated successfully, but these errors were encountered:

ashking94 · 2023-10-21T19:22:55Z

This is highly flaky in backport PRs.

shwetathareja · 2023-10-23T09:19:09Z

#10848 (comment)

amkhar · 2023-10-25T07:47:02Z

#10904 (comment)

ashking94 · 2023-10-26T03:40:01Z

#10939 (comment)

reta added bug Something isn't working untriaged flaky-test Random test failure that succeeds on second run and removed untriaged labels Oct 20, 2023

github-actions bot added the untriaged label Oct 20, 2023

reta mentioned this issue Oct 20, 2023

[Backport 2.x] Update Github pull request template to have a task for inspecting failing checks #10792

Merged

linuxpi mentioned this issue Oct 21, 2023

[Backport 2.x] [Remote State] fix lock release before deletion is completed #10778

Merged

This was referenced Oct 21, 2023

[BUG] Flaky test: org.opensearch.search.suggest.CompletionSuggestSearchIT.testSkipDuplicates #10808

Closed

[Backport 2.x] [Remote Store] Fix relocation failure due to transport receive timeout #10788

Merged

linuxpi mentioned this issue Oct 22, 2023

[Backport 2.x] Adds util to compare Metadata for remote state #10824

Merged

shwetathareja mentioned this issue Oct 22, 2023

[Backport 2.x] Fix custom metadata not getting stored for remote store not supporting async write #10828

Merged

This was referenced Oct 23, 2023

[Backport 2.x] Override local disk state if we are able to restore from remote #10833

Merged

[Backport 2.x] fix integ test: check for red indices and then manually restore to green #10843

Merged

shwetathareja mentioned this issue Oct 23, 2023

[Backport 2.x] Fix remote cluster restore for data stream #10848

Merged

ashking94 mentioned this issue Oct 23, 2023

[Backport 2.x] [Remote Store] Add extra buffer before deleting older generations of translog #10850

Merged

rahulkarajgikar mentioned this issue Oct 23, 2023

[Backport 2.x] Make index and global metadata upload timeout dynamic cluster setting #10852

Merged

8 tasks

reta mentioned this issue Oct 23, 2023

[Backport 2.x] Fix flaky testEqualsAndHashcode in SearchRequestTests #10869

Merged

shwetathareja added the >test-failure Test failure from CI, local build, etc. label Oct 24, 2023

RamakrishnaChilaka assigned nknize Oct 24, 2023

Poojita-Raj mentioned this issue Oct 24, 2023

Add cluster setting cluster.restrict.index.replication_type t… (#10866) #10876

Merged

amkhar mentioned this issue Oct 25, 2023

[Backport 2.x] Add cluster state stats #10904

Merged

linuxpi mentioned this issue Oct 25, 2023

[Backport 2.x] link previous cluster uuid to current cluster uuid even if current cluster uuid is not committed #10906

Merged

sachinpkale mentioned this issue Oct 25, 2023

[Backport 2.x] Delete corrupted file to re-download from remote store #10909

Merged

dhwanilpatel mentioned this issue Oct 25, 2023

[Backport 2.x] Fix exception handling for global metadata upload #10920

Merged

shwetathareja removed the untriaged label Oct 25, 2023

mch2 mentioned this issue Oct 25, 2023

[Backport 2.x] Fix flaky testClusterStateBatchedUpdates test #10930

Merged

mch2 mentioned this issue Oct 26, 2023

[Backport 2.x] [Segment Replication] Handle exceptions on local file read during replication #10936

Merged

Poojita-Raj mentioned this issue Oct 27, 2023

[Backport 2.x] [Segment Replication] Add log for failover time #10953

Merged

This was referenced Oct 27, 2023

[Backport 2.x] Add tracing instrumentation for indexing paths #10962

Merged

[Backport 2.x] Remove deprecated classes for Rounding #10963

Merged

linuxpi mentioned this issue Oct 30, 2023

[Backport 2.x] Restore ClusterState version during remote state restore (#10853) #10981

Merged

8 tasks

This was referenced Oct 30, 2023

[Backport 2.x] Disable sort optimization for HALF_FLOAT #11004

Merged

[Backport] [2.x] Bump org.apache.logging.log4j:log4j-core from 2.21.0 to 2.21.1 in /buildSrc/src/testKit/thirdPartyAudit/sample_jars (#11000) #11006

Merged

ticheng-aws mentioned this issue Oct 30, 2023

Backport of PRs for profile related changes in concurrent search path #10898

Merged

8 tasks

sohami added this to Concurrent Search Oct 30, 2023

github-project-automation bot moved this to Todo in Concurrent Search Oct 30, 2023

sohami removed this from Concurrent Search Oct 30, 2023

reta mentioned this issue Oct 30, 2023

[Backport] [2.x] Update bundled JDK to JDK-21.0.1 (#10576) #11003

Merged

dreamer-89 mentioned this issue Oct 31, 2023

[Backport 2.x] Fix SegRep bug where primary shards do not update tracked replica state post failover/relocation #11018

Merged

reta mentioned this issue Nov 1, 2023

[AUTOCUT] Gradle Check Failure on push to 2.x #11041

Closed

jed326 mentioned this issue Nov 1, 2023

[Backport 2.x] Mute flaky concurrent segment search tests pending fixes (#10437) #11050

Merged

8 tasks

gaobinlong mentioned this issue Nov 2, 2023

[Backport 2.x] Fix passing wrong parameter when calling newConfigurationException() in DotExpanderProcessor #11013

Merged

8 tasks

mch2 mentioned this issue Nov 2, 2023

[AUTOCUT] Gradle Check Failure on push to 2.x #10837

Closed

This was referenced Nov 2, 2023

[AUTOCUT] Gradle Check Failure on push to 2.x #11062

Closed

Fix SuggestSearch.testSkipDuplicates by forceing refresh when indexing its test documents #11068

Merged

peternied assigned peternied and unassigned nknize Nov 2, 2023

peternied mentioned this issue Nov 3, 2023

[AUTOCUT] Gradle Check Failure on push to main #11080

Closed

Poojita-Raj mentioned this issue Nov 3, 2023

[2.x] Revert "Add cluster setting cluster.restrict.index.replication_type t… #11073

Merged

8 tasks

mch2 mentioned this issue Nov 6, 2023

[Backport 2.x] Fix flaky pit/scroll tests in SegmentReplicationIT #11098

Merged

reta mentioned this issue Nov 6, 2023

[Backport] [2.x] [Streaming Indexing] Introduce new experimental HTTP transport based on Netty 4 and Project Reactor (Reactor Netty) (#9672) #11108

Merged

ticheng-aws mentioned this issue Nov 6, 2023

[Backport 2.x] Fixing concurrent search tests with one slice #11115

Merged

peternied closed this as completed in #11068 Nov 7, 2023

peternied mentioned this issue Nov 8, 2023

[Backport 2.x] Fix SuggestSearch.testSkipDuplicates by forceing refresh when indexing its test documents (#11068) #11141

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] org.opensearch.search.suggest.CompletionSuggestSearchIT.testSkipDuplicates is flaky #10798

[BUG] org.opensearch.search.suggest.CompletionSuggestSearchIT.testSkipDuplicates is flaky #10798

reta commented Oct 20, 2023

ashking94 commented Oct 21, 2023

shwetathareja commented Oct 23, 2023

amkhar commented Oct 25, 2023

ashking94 commented Oct 26, 2023

[BUG] org.opensearch.search.suggest.CompletionSuggestSearchIT.testSkipDuplicates is flaky #10798

[BUG] org.opensearch.search.suggest.CompletionSuggestSearchIT.testSkipDuplicates is flaky #10798

Comments

reta commented Oct 20, 2023

ashking94 commented Oct 21, 2023

shwetathareja commented Oct 23, 2023

amkhar commented Oct 25, 2023

ashking94 commented Oct 26, 2023