[BUG] org.opensearch.indices.store.IndicesStoreIntegrationIT.testShardActiveElseWhere is flaky #12788

reta · 2024-03-20T13:45:52Z

Describe the bug

The test case org.opensearch.indices.store.IndicesStoreIntegrationIT.testShardActiveElseWhere {p0={"cluster.remote_store.state.enabled":"true"}} is flaky:

java.lang.AssertionError: shard [test][4] is not locked
	at __randomizedtesting.SeedInfo.seed([67FB6E95E72E6169]:0)
	at org.opensearch.env.NodeEnvironment.deleteShardDirectoryUnderLock(NodeEnvironment.java:587)
	at org.opensearch.indices.IndicesService.deleteShardStore(IndicesService.java:1245)
	at org.opensearch.index.IndexService.onShardClose(IndexService.java:678)
	at org.opensearch.index.IndexService$StoreCloseListener.accept(IndexService.java:801)
	at org.opensearch.index.IndexService$StoreCloseListener.accept(IndexService.java:788)
	at org.opensearch.index.store.Store.closeInternal(Store.java:573)
	at org.opensearch.index.store.Store$1.closeInternal(Store.java:194)
	at org.opensearch.common.util.concurrent.AbstractRefCounted.decRef(AbstractRefCounted.java:78)
	at org.opensearch.index.store.Store.decRef(Store.java:546)
	at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1777)
	at org.opensearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:1753)
	at org.opensearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:4498)
	at org.opensearch.index.IndexService.maybeRefreshEngine(IndexService.java:1026)
	at org.opensearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:1170)
	at org.opensearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:159)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Related component

Indexing

To Reproduce

 ./gradlew ':server:internalClusterTest' --tests "org.opensearch.indices.store.IndicesStoreIntegrationIT" -Dtests.method="testShardActiveElseWhere {p0={"cluster.remote_store.state.enabled":"true"}}" -Dtests.seed=67FB6E95E72E6169

Expected behavior

The test must always pass

Additional Details

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

CI

Additional context

https://build.ci.opensearch.org/job/gradle-check/35444/testReport/junit/org.opensearch.indices.store/IndicesStoreIntegrationIT/testShardActiveElseWhere__p0___cluster_remote_store_state_enabled___true___/

The text was updated successfully, but these errors were encountered:

reta · 2024-03-20T13:48:39Z

@Rishikesh1159 seems like came out of #12487, could you please take a look? Thank you.

andrross · 2024-03-20T15:05:36Z

[Triage - attendees 1 2 3 4 5 6]
@reta Thanks for filling.

peternied · 2024-04-16T13:27:01Z

Another failure reported on [AUTOCUT] Gradle Check Failure on push to main #13212

bowenlan-amzn · 2024-06-14T14:31:48Z

org.opensearch.indices.store.IndicesStoreIntegrationIT.testShardActiveElsewhereDoesNotDeleteAnother {p0={"cluster.remote_store.state.enabled":"true"}}

java.lang.AssertionError: Missing cluster-manager, expected nodes: [{node_t6}{iJqdci3PRDiP6K7IA0PQUQ}{BdtuhPTDRni-YavPeeeizg}{127.0.0.1}{127.0.0.1:44987}{dimr}{shard_indexing_pressure_enabled=true}, {node_t0}{bbBIthSXQUmP_ad_EcCOIg}{LF7Q80uUTBSfK1SB3_BUyA}{127.0.0.1}{127.0.0.1:34885}{m}{shard_indexing_pressure_enabled=true}, {node_t5}{lCcpP37hTq2eoUwhSyxohg}{mb4qExFVR6KOmYLzGitr1Q}{127.0.0.1}{127.0.0.1:33571}{dimr}{shard_indexing_pressure_enabled=true}, {node_t4}{0QbUYbQ_S7mk9E4iHjVszw}{kXm5m59DQ_WOl6Uy7mA23A}{127.0.0.1}{127.0.0.1:41315}{d}{shard_indexing_pressure_enabled=true}, {node_t3}{t1h9IsByQJiM5qoGPJ5JKQ}{wJH9PpqMSeKdsIgKnIXEIA}{127.0.0.1}{127.0.0.1:46233}{d}{shard_indexing_pressure_enabled=true}] and actual cluster states [cluster uuid: 3u0GjtdlQZSX__FUSYH31A [committed: true]
version: 22
state uuid: SGQH7ixXSA2NZB3b2bQdsQ
from_diff: false
meta data version: 17
   coordination_metadata:
      term: 1
      last_committed_config: VotingConfiguration{bbBIthSXQUmP_ad_EcCOIg}
      last_accepted_config: VotingConfiguration{bbBIthSXQUmP_ad_EcCOIg}
      voting tombstones: []
   [test/mEe3w-WmSDS2ewJEMHtM4A]: v[12], mv[1], sv[2], av[1]
      0: p_term [1], isa_ids [4E01bHLBTDCFTspIC3bF8A, siiZJzadS3-8Qv32ImLDnA]
      1: p_term [3], isa_ids [HlPd5nBoRRWxF0f46AfJrg, 29nA9fBXStKUVmW1w1JQMg]
      2: p_term [2], isa_ids [jgyNfamcS5OnukvsJYJkEA, TfCvmItBRUqB6lo0t_G2ig]
metadata customs:
   repositories: {"test-remote-store-repo":{"type":"reloadable-fs","settings":{"system_repository":"true","location":"/var/jenkins/workspace/gradle-check/search/server/build/testrun/internalClusterTest/temp/org.opensearch.indices.store.IndicesStoreIntegrationIT_1D9581A57D5AA70F-001/tempDir-018/repos/lipSOdmdQi"},"generation":-2,"pending_generation":-1}}   index-graveyard: IndexGraveyard[[]]
blocks: 
   _global_:
      2,no cluster-manager, blocks WRITE,METADATA_WRITE
nodes: 
   {node_t6}{iJqdci3PRDiP6K7IA0PQUQ}{BdtuhPTDRni-YavPeeeizg}{127.0.0.1}{127.0.0.1:44987}{dimr}{shard_indexing_pressure_enabled=true}
   {node_t0}{bbBIthSXQUmP_ad_EcCOIg}{LF7Q80uUTBSfK1SB3_BUyA}{127.0.0.1}{127.0.0.1:34885}{m}{shard_indexing_pressure_enabled=true}, local
   {node_t3}{t1h9IsByQJiM5qoGPJ5JKQ}{wJH9PpqMSeKdsIgKnIXEIA}{127.0.0.1}{127.0.0.1:46233}{d}{shard_indexing_pressure_enabled=true}
   {node_t4}{0QbUYbQ_S7mk9E4iHjVszw}{kXm5m59DQ_WOl6Uy7mA23A}{127.0.0.1}{127.0.0.1:41315}{d}{shard_indexing_pressure_enabled=true}
routing_table (version 15):
-- index [[test/mEe3w-WmSDS2ewJEMHtM4A]]
----shard_id [test][0]
--------[test][0], node[t1h9IsByQJiM5qoGPJ5JKQ], [P], s[STARTED], a[id=siiZJzadS3-8Qv32ImLDnA]
--------[test][0], node[0QbUYbQ_S7mk9E4iHjVszw], [R], s[STARTED], a[id=4E01bHLBTDCFTspIC3bF8A]
----shard_id [test][1]
--------[test][1], node[null], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[UNASSIGNED], unassigned_info[[reason=NODE_LEFT], at[2024-06-14T02:03:32.716Z], delayed=false, details[node_left [iJqdci3PRDiP6K7IA0PQUQ]], allocation_status[no_valid_shard_copy]]
--------[test][1], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=NODE_LEFT], at[2024-06-14T02:03:32.631Z], delayed=false, details[node_left [lCcpP37hTq2eoUwhSyxohg]], allocation_status[no_attempt]]
----shard_id [test][2]
--------[test][2], node[t1h9IsByQJiM5qoGPJ5JKQ], [P], s[STARTED], a[id=jgyNfamcS5OnukvsJYJkEA]
--------[test][2], node[0QbUYbQ_S7mk9E4iHjVszw], [R], s[STARTED], a[id=TfCvmItBRUqB6lo0t_G2ig]

routing_nodes:
-----node_id[t1h9IsByQJiM5qoGPJ5JKQ][V]
--------[test][0], node[t1h9IsByQJiM5qoGPJ5JKQ], [P], s[STARTED], a[id=siiZJzadS3-8Qv32ImLDnA]
--------[test][2], node[t1h9IsByQJiM5qoGPJ5JKQ], [P], s[STARTED], a[id=jgyNfamcS5OnukvsJYJkEA]
-----node_id[0QbUYbQ_S7mk9E4iHjVszw][V]
--------[test][0], node[0QbUYbQ_S7mk9E4iHjVszw], [R], s[STARTED], a[id=4E01bHLBTDCFTspIC3bF8A]
--------[test][2], node[0QbUYbQ_S7mk9E4iHjVszw], [R], s[STARTED], a[id=TfCvmItBRUqB6lo0t_G2ig]
-----node_id[iJqdci3PRDiP6K7IA0PQUQ][V]
---- unassigned
--------[test][1], node[null], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[UNASSIGNED], unassigned_info[[reason=NODE_LEFT], at[2024-06-14T02:03:32.716Z], delayed=false, details[node_left [iJqdci3PRDiP6K7IA0PQUQ]], allocation_status[no_valid_shard_copy]]
--------[test][1], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=NODE_LEFT], at[2024-06-14T02:03:32.631Z], delayed=false, details[node_left [lCcpP37hTq2eoUwhSyxohg]], allocation_status[no_attempt]]
, cluster uuid: 3u0GjtdlQZSX__FUSYH31A [committed: true]
version: 22
state uuid: SGQH7ixXSA2NZB3b2bQdsQ
from_diff: false
meta data version: 17
   coordination_metadata:
      term: 1
      last_committed_config: VotingConfiguration{bbBIthSXQUmP_ad_EcCOIg}
      last_accepted_config: VotingConfiguration{bbBIthSXQUmP_ad_EcCOIg}
      voting tombstones: []
   [test/mEe3w-WmSDS2ewJEMHtM4A]: v[12], mv[1], sv[2], av[1]
      0: p_term [1], isa_ids [4E01bHLBTDCFTspIC3bF8A, siiZJzadS3-8Qv32ImLDnA]
      1: p_term [3], isa_ids [HlPd5nBoRRWxF0f46AfJrg, 29nA9fBXStKUVmW1w1JQMg]
      2: p_term [2], isa_ids [jgyNfamcS5OnukvsJYJkEA, TfCvmItBRUqB6lo0t_G2ig]
metadata customs:
   repositories: {"test-remote-store-repo":{"type":"reloadable-fs","settings":{"system_repository":"true","location":"/var/jenkins/workspace/gradle-check/search/server/build/testrun/internalClusterTest/temp/org.opensearch.indices.store.IndicesStoreIntegrationIT_1D9581A57D5AA70F-001/tempDir-018/repos/lipSOdmdQi"},"generation":-2,"pending_generation":-1}}   index-graveyard: IndexGraveyard[[]]
blocks: 
   _global_:
      2,no cluster-manager, blocks WRITE,METADATA_WRITE
nodes: 
   {node_t6}{iJqdci3PRDiP6K7IA0PQUQ}{BdtuhPTDRni-YavPeeeizg}{127.0.0.1}{127.0.0.1:44987}{dimr}{shard_indexing_pressure_enabled=true}
   {node_t3}{t1h9IsByQJiM5qoGPJ5JKQ}{wJH9PpqMSeKdsIgKnIXEIA}{127.0.0.1}{127.0.0.1:46233}{d}{shard_indexing_pressure_enabled=true}, local
   {node_t0}{bbBIthSXQUmP_ad_EcCOIg}{LF7Q80uUTBSfK1SB3_BUyA}{127.0.0.1}{127.0.0.1:34885}{m}{shard_indexing_pressure_enabled=true}
   {node_t4}{0QbUYbQ_S7mk9E4iHjVszw}{kXm5m59DQ_WOl6Uy7mA23A}{127.0.0.1}{127.0.0.1:41315}{d}{shard_indexing_pressure_enabled=true}
routing_table (version 15):
-- index [[test/mEe3w-WmSDS2ewJEMHtM4A]]
----shard_id [test][0]
--------[test][0], node[t1h9IsByQJiM5qoGPJ5JKQ], [P], s[STARTED], a[id=siiZJzadS3-8Qv32ImLDnA]
--------[test][0], node[0QbUYbQ_S7mk9E4iHjVszw], [R], s[STARTED], a[id=4E01bHLBTDCFTspIC3bF8A]
----shard_id [test][1]
--------[test][1], node[null], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[UNASSIGNED], unassigned_info[[reason=NODE_LEFT], at[2024-06-14T02:03:32.716Z], delayed=false, details[node_left [iJqdci3PRDiP6K7IA0PQUQ]], allocation_status[no_valid_shard_copy]]
--------[test][1], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=NODE_LEFT], at[2024-06-14T02:03:32.631Z], delayed=false, details[node_left [lCcpP37hTq2eoUwhSyxohg]], allocation_status[no_attempt]]
----shard_id [test][2]
--------[test][2], node[t1h9IsByQJiM5qoGPJ5JKQ], [P], s[STARTED], a[id=jgyNfamcS5OnukvsJYJkEA]
--------[test][2], node[0QbUYbQ_S7mk9E4iHjVszw], [R], s[STARTED], a[id=TfCvmItBRUqB6lo0t_G2ig]

routing_nodes:
-----node_id[t1h9IsByQJiM5qoGPJ5JKQ][V]
--------[test][0], node[t1h9IsByQJiM5qoGPJ5JKQ], [P], s[STARTED], a[id=siiZJzadS3-8Qv32ImLDnA]
--------[test][2], node[t1h9IsByQJiM5qoGPJ5JKQ], [P], s[STARTED], a[id=jgyNfamcS5OnukvsJYJkEA]
-----node_id[0QbUYbQ_S7mk9E4iHjVszw][V]
--------[test][0], node[0QbUYbQ_S7mk9E4iHjVszw], [R], s[STARTED], a[id=4E01bHLBTDCFTspIC3bF8A]
--------[test][2], node[0QbUYbQ_S7mk9E4iHjVszw], [R], s[STARTED], a[id=TfCvmItBRUqB6lo0t_G2ig]
-----node_id[iJqdci3PRDiP6K7IA0PQUQ][V]
---- unassigned
--------[test][1], node[null], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[UNASSIGNED], unassigned_info[[reason=NODE_LEFT], at[2024-06-14T02:03:32.716Z], delayed=false, details[node_left [iJqdci3PRDiP6K7IA0PQUQ]], allocation_status[no_valid_shard_copy]]
--------[test][1], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=NODE_LEFT], at[2024-06-14T02:03:32.631Z], delayed=false, details[node_left [lCcpP37hTq2eoUwhSyxohg]], allocation_status[no_attempt]]
, cluster uuid: 3u0GjtdlQZSX__FUSYH31A [committed: true]
version: 22
state uuid: SGQH7ixXSA2NZB3b2bQdsQ
from_diff: false
meta data version: 17
   coordination_metadata:
      term: 1
      last_committed_config: VotingConfiguration{bbBIthSXQUmP_ad_EcCOIg}
      last_accepted_config: VotingConfiguration{bbBIthSXQUmP_ad_EcCOIg}
      voting tombstones: []
   [test/mEe3w-WmSDS2ewJEMHtM4A]: v[12], mv[1], sv[2], av[1]
      0: p_term [1], isa_ids [4E01bHLBTDCFTspIC3bF8A, siiZJzadS3-8Qv32ImLDnA]
      1: p_term [3], isa_ids [HlPd5nBoRRWxF0f46AfJrg, 29nA9fBXStKUVmW1w1JQMg]
      2: p_term [2], isa_ids [jgyNfamcS5OnukvsJYJkEA, TfCvmItBRUqB6lo0t_G2ig]
metadata customs:
   repositories: {"test-remote-store-repo":{"type":"reloadable-fs","settings":{"system_repository":"true","location":"/var/jenkins/workspace/gradle-check/search/server/build/testrun/internalClusterTest/temp/org.opensearch.indices.store.IndicesStoreIntegrationIT_1D9581A57D5AA70F-001/tempDir-018/repos/lipSOdmdQi"},"generation":-2,"pending_generation":-1}}   index-graveyard: IndexGraveyard[[]]
blocks: 
   _global_:
      2,no cluster-manager, blocks WRITE,METADATA_WRITE
nodes: 
   {node_t6}{iJqdci3PRDiP6K7IA0PQUQ}{BdtuhPTDRni-YavPeeeizg}{127.0.0.1}{127.0.0.1:44987}{dimr}{shard_indexing_pressure_enabled=true}
   {node_t3}{t1h9IsByQJiM5qoGPJ5JKQ}{wJH9PpqMSeKdsIgKnIXEIA}{127.0.0.1}{127.0.0.1:46233}{d}{shard_indexing_pressure_enabled=true}
   {node_t0}{bbBIthSXQUmP_ad_EcCOIg}{LF7Q80uUTBSfK1SB3_BUyA}{127.0.0.1}{127.0.0.1:34885}{m}{shard_indexing_pressure_enabled=true}
   {node_t4}{0QbUYbQ_S7mk9E4iHjVszw}{kXm5m59DQ_WOl6Uy7mA23A}{127.0.0.1}{127.0.0.1:41315}{d}{shard_indexing_pressure_enabled=true}, local
routing_table (version 15):
-- index [[test/mEe3w-WmSDS2ewJEMHtM4A]]
----shard_id [test][0]
--------[test][0], node[t1h9IsByQJiM5qoGPJ5JKQ], [P], s[STARTED], a[id=siiZJzadS3-8Qv32ImLDnA]
--------[test][0], node[0QbUYbQ_S7mk9E4iHjVszw], [R], s[STARTED], a[id=4E01bHLBTDCFTspIC3bF8A]
----shard_id [test][1]
--------[test][1], node[null], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[UNASSIGNED], unassigned_info[[reason=NODE_LEFT], at[2024-06-14T02:03:32.716Z], delayed=false, details[node_left [iJqdci3PRDiP6K7IA0PQUQ]], allocation_status[no_valid_shard_copy]]
--------[test][1], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=NODE_LEFT], at[2024-06-14T02:03:32.631Z], delayed=false, details[node_left [lCcpP37hTq2eoUwhSyxohg]], allocation_status[no_attempt]]
----shard_id [test][2]
--------[test][2], node[t1h9IsByQJiM5qoGPJ5JKQ], [P], s[STARTED], a[id=jgyNfamcS5OnukvsJYJkEA]
--------[test][2], node[0QbUYbQ_S7mk9E4iHjVszw], [R], s[STARTED], a[id=TfCvmItBRUqB6lo0t_G2ig]

routing_nodes:
-----node_id[t1h9IsByQJiM5qoGPJ5JKQ][V]
--------[test][0], node[t1h9IsByQJiM5qoGPJ5JKQ], [P], s[STARTED], a[id=siiZJzadS3-8Qv32ImLDnA]
--------[test][2], node[t1h9IsByQJiM5qoGPJ5JKQ], [P], s[STARTED], a[id=jgyNfamcS5OnukvsJYJkEA]
-----node_id[0QbUYbQ_S7mk9E4iHjVszw][V]
--------[test][0], node[0QbUYbQ_S7mk9E4iHjVszw], [R], s[STARTED], a[id=4E01bHLBTDCFTspIC3bF8A]
--------[test][2], node[0QbUYbQ_S7mk9E4iHjVszw], [R], s[STARTED], a[id=TfCvmItBRUqB6lo0t_G2ig]
-----node_id[iJqdci3PRDiP6K7IA0PQUQ][V]
---- unassigned
--------[test][1], node[null], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[UNASSIGNED], unassigned_info[[reason=NODE_LEFT], at[2024-06-14T02:03:32.716Z], delayed=false, details[node_left [iJqdci3PRDiP6K7IA0PQUQ]], allocation_status[no_valid_shard_copy]]
--------[test][1], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=NODE_LEFT], at[2024-06-14T02:03:32.631Z], delayed=false, details[node_left [lCcpP37hTq2eoUwhSyxohg]], allocation_status[no_attempt]]
, cluster uuid: _na_ [committed: false]
version: 0
state uuid: nwMQ4A8FRoC6r8Tlukz3mA
from_diff: false
meta data version: 0
   coordination_metadata:
      term: 0
      last_committed_config: VotingConfiguration{}
      last_accepted_config: VotingConfiguration{}
      voting tombstones: []
metadata customs:
   index-graveyard: IndexGraveyard[[]]
blocks: 
   _global_:
      1,state not recovered / initialized, blocks READ,WRITE,METADATA_READ,METADATA_WRITE,CREATE_INDEX      2,no cluster-manager, blocks WRITE,METADATA_WRITE
nodes: 
   {node_t5}{lCcpP37hTq2eoUwhSyxohg}{mb4qExFVR6KOmYLzGitr1Q}{127.0.0.1}{127.0.0.1:33571}{dimr}{shard_indexing_pressure_enabled=true}, local
routing_table (version 0):
routing_nodes:
-----node_id[lCcpP37hTq2eoUwhSyxohg][V]
---- unassigned
, cluster uuid: 3u0GjtdlQZSX__FUSYH31A [committed: true]
version: 22
state uuid: SGQH7ixXSA2NZB3b2bQdsQ
from_diff: false
meta data version: 17
   coordination_metadata:
      term: 1
      last_committed_config: VotingConfiguration{bbBIthSXQUmP_ad_EcCOIg}
      last_accepted_config: VotingConfiguration{bbBIthSXQUmP_ad_EcCOIg}
      voting tombstones: []
   [test/mEe3w-WmSDS2ewJEMHtM4A]: v[12], mv[1], sv[2], av[1]
      0: p_term [1], isa_ids [4E01bHLBTDCFTspIC3bF8A, siiZJzadS3-8Qv32ImLDnA]
      1: p_term [3], isa_ids [HlPd5nBoRRWxF0f46AfJrg, 29nA9fBXStKUVmW1w1JQMg]
      2: p_term [2], isa_ids [jgyNfamcS5OnukvsJYJkEA, TfCvmItBRUqB6lo0t_G2ig]
metadata customs:
   repositories: {"test-remote-store-repo":{"type":"reloadable-fs","settings":{"system_repository":"true","location":"/var/jenkins/workspace/gradle-check/search/server/build/testrun/internalClusterTest/temp/org.opensearch.indices.store.IndicesStoreIntegrationIT_1D9581A57D5AA70F-001/tempDir-018/repos/lipSOdmdQi"},"generation":-2,"pending_generation":-1}}   index-graveyard: IndexGraveyard[[]]
blocks: 
   _global_:
      2,no cluster-manager, blocks WRITE,METADATA_WRITE
nodes: 
   {node_t6}{iJqdci3PRDiP6K7IA0PQUQ}{BdtuhPTDRni-YavPeeeizg}{127.0.0.1}{127.0.0.1:44987}{dimr}{shard_indexing_pressure_enabled=true}, local
   {node_t3}{t1h9IsByQJiM5qoGPJ5JKQ}{wJH9PpqMSeKdsIgKnIXEIA}{127.0.0.1}{127.0.0.1:46233}{d}{shard_indexing_pressure_enabled=true}
   {node_t0}{bbBIthSXQUmP_ad_EcCOIg}{LF7Q80uUTBSfK1SB3_BUyA}{127.0.0.1}{127.0.0.1:34885}{m}{shard_indexing_pressure_enabled=true}
   {node_t4}{0QbUYbQ_S7mk9E4iHjVszw}{kXm5m59DQ_WOl6Uy7mA23A}{127.0.0.1}{127.0.0.1:41315}{d}{shard_indexing_pressure_enabled=true}
routing_table (version 15):
-- index [[test/mEe3w-WmSDS2ewJEMHtM4A]]
----shard_id [test][0]
--------[test][0], node[t1h9IsByQJiM5qoGPJ5JKQ], [P], s[STARTED], a[id=siiZJzadS3-8Qv32ImLDnA]
--------[test][0], node[0QbUYbQ_S7mk9E4iHjVszw], [R], s[STARTED], a[id=4E01bHLBTDCFTspIC3bF8A]
----shard_id [test][1]
--------[test][1], node[null], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[UNASSIGNED], unassigned_info[[reason=NODE_LEFT], at[2024-06-14T02:03:32.716Z], delayed=false, details[node_left [iJqdci3PRDiP6K7IA0PQUQ]], allocation_status[no_valid_shard_copy]]
--------[test][1], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=NODE_LEFT], at[2024-06-14T02:03:32.631Z], delayed=false, details[node_left [lCcpP37hTq2eoUwhSyxohg]], allocation_status[no_attempt]]
----shard_id [test][2]
--------[test][2], node[t1h9IsByQJiM5qoGPJ5JKQ], [P], s[STARTED], a[id=jgyNfamcS5OnukvsJYJkEA]
--------[test][2], node[0QbUYbQ_S7mk9E4iHjVszw], [R], s[STARTED], a[id=TfCvmItBRUqB6lo0t_G2ig]

routing_nodes:
-----node_id[t1h9IsByQJiM5qoGPJ5JKQ][V]
--------[test][0], node[t1h9IsByQJiM5qoGPJ5JKQ], [P], s[STARTED], a[id=siiZJzadS3-8Qv32ImLDnA]
--------[test][2], node[t1h9IsByQJiM5qoGPJ5JKQ], [P], s[STARTED], a[id=jgyNfamcS5OnukvsJYJkEA]
-----node_id[0QbUYbQ_S7mk9E4iHjVszw][V]
--------[test][0], node[0QbUYbQ_S7mk9E4iHjVszw], [R], s[STARTED], a[id=4E01bHLBTDCFTspIC3bF8A]
--------[test][2], node[0QbUYbQ_S7mk9E4iHjVszw], [R], s[STARTED], a[id=TfCvmItBRUqB6lo0t_G2ig]
-----node_id[iJqdci3PRDiP6K7IA0PQUQ][V]
---- unassigned
--------[test][1], node[null], [P], recovery_source[existing store recovery; bootstrap_history_uuid=false], s[UNASSIGNED], unassigned_info[[reason=NODE_LEFT], at[2024-06-14T02:03:32.716Z], delayed=false, details[node_left [iJqdci3PRDiP6K7IA0PQUQ]], allocation_status[no_valid_shard_copy]]
--------[test][1], node[null], [R], recovery_source[peer recovery], s[UNASSIGNED], unassigned_info[[reason=NODE_LEFT], at[2024-06-14T02:03:32.631Z], delayed=false, details[node_left [lCcpP37hTq2eoUwhSyxohg]], allocation_status[no_attempt]]
]

https://build.ci.opensearch.org/job/gradle-check/40964/testReport/junit/org.opensearch.indices.store/IndicesStoreIntegrationIT/testShardActiveElsewhereDoesNotDeleteAnother__p0___cluster_remote_store_state_enabled___true___/

reta · 2024-06-20T13:07:49Z

Closing in favour of #14472

reta added bug Something isn't working untriaged flaky-test Random test failure that succeeds on second run labels Mar 20, 2024

github-actions bot added the Indexing Indexing, Bulk Indexing and anything related to indexing label Mar 20, 2024

reta mentioned this issue Mar 20, 2024

[AUTOCUT] Gradle Check Failure on push to main #12787

Closed

reta assigned Rishikesh1159 Mar 20, 2024

andrross removed the untriaged label Mar 20, 2024

This was referenced Mar 22, 2024

[Backport 2.13] Add release-notes for 2.13.0 #12846

Merged

Added missed API visibility annotations for public APIs and enable the check at the build time #12872

Merged

[AUTOCUT] Gradle Check Failure on push to main #12898

Closed

vikasvb90 added ShardManagement:Routing ShardManagement:Placement and removed Indexing Indexing, Bulk Indexing and anything related to indexing labels Mar 27, 2024

github-project-automation bot added this to Shard Management Project Board Mar 27, 2024

github-project-automation bot moved this to 🆕 New in Shard Management Project Board Mar 27, 2024

niyatiagg mentioned this issue Apr 5, 2024

Refactoring globMatch using simpleMatchWithNormalizedStrings from Regex #13104

Merged

7 tasks

kotwanikunal mentioned this issue Apr 10, 2024

[AUTOCUT] Gradle Check Failure on push to 2.x #13142

Closed

peternied mentioned this issue Apr 16, 2024

[AUTOCUT] Gradle Check Failure on push to main #13212

Closed

kotwanikunal mentioned this issue Apr 16, 2024

[AUTOCUT] Gradle Check Failure on push to main #13015

Closed

jainankitk mentioned this issue Apr 19, 2024

Ignoring unavailable shards during search request execution with igno… #13298

Merged

7 tasks

This was referenced Apr 23, 2024

Integrate local recovery with remote store seeding during migration #12922

Merged

[BUG] Multiple tests failing with shard is not locked issue #13342

Open

dblock mentioned this issue Apr 23, 2024

Fix flat_object long field error #13259

Merged

8 tasks

shourya035 mentioned this issue Apr 25, 2024

[Remote Store] Update index settings on shard movement during remote store migration #13316

Merged

8 tasks

bowenlan-amzn mentioned this issue Apr 26, 2024

Support multi ranges traversal when doing date histogram rewrite optimization #13317

Merged

7 tasks

kotwanikunal mentioned this issue May 2, 2024

[AUTOCUT] Gradle Check Failure on push to main #13511

Closed

peternied mentioned this issue May 2, 2024

[BUG] org.opensearch.indices.store.IndicesStoreIntegrationIT.testShardActiveElseWhere {p0={"cluster.remote_store.state.enabled":"true"}} is flaky #13487

Closed

gbbafna mentioned this issue Jun 10, 2024

Add recovery chunk size setting #13997

Merged

9 tasks

gaobinlong mentioned this issue Jun 13, 2024

Fix flaky tests in org.opensearch.cluster.routing.remote.RemoteRoutingTableServiceTests #14264

Merged

soosinha mentioned this issue Jun 14, 2024

Use remote publication flag to decide which custom objects to upload #14338

Merged

3 tasks

bowenlan-amzn mentioned this issue Jun 14, 2024

Apply the date histogram rewrite optimization to range aggregation #13865

Merged

9 tasks

gaobinlong mentioned this issue Jun 17, 2024

Add fingerprint ingest processor #13724

Merged

9 tasks

reta closed this as completed Jun 20, 2024

github-project-automation bot moved this from 🆕 New to ✅ Done in Shard Management Project Board Jun 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] org.opensearch.indices.store.IndicesStoreIntegrationIT.testShardActiveElseWhere is flaky #12788

[BUG] org.opensearch.indices.store.IndicesStoreIntegrationIT.testShardActiveElseWhere is flaky #12788

reta commented Mar 20, 2024

reta commented Mar 20, 2024

andrross commented Mar 20, 2024

peternied commented Apr 16, 2024

bowenlan-amzn commented Jun 14, 2024

reta commented Jun 20, 2024

[BUG] org.opensearch.indices.store.IndicesStoreIntegrationIT.testShardActiveElseWhere is flaky #12788

[BUG] org.opensearch.indices.store.IndicesStoreIntegrationIT.testShardActiveElseWhere is flaky #12788

Comments

reta commented Mar 20, 2024

Describe the bug

Related component

To Reproduce

Expected behavior

Additional Details

reta commented Mar 20, 2024

andrross commented Mar 20, 2024

peternied commented Apr 16, 2024

bowenlan-amzn commented Jun 14, 2024

reta commented Jun 20, 2024