Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix assertion failure while closing remoteStore #10627

Closed
wants to merge 1 commit into from
Closed

Fix assertion failure while closing remoteStore #10627

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion server/src/main/java/org/opensearch/index/IndexService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -654,7 +654,15 @@ private void closeShard(String reason, ShardId sId, IndexShard indexShard, Store
}

if (remoteStore != null && indexShard.isPrimaryMode() && deleted.get()) {
remoteStore.close();
// When an instance of Store is created, a shardlock is created which is released on closing the instance of store.
// Currently, we create 2 instances of store for remote store backed indices: store and remoteStore.
// As there can be only one shardlock acquired for a given shard, the lock is shared between store and remoteStore.
// This creates an issue when we are deleting the index as it results in closing both store and remoteStore. At the time
// of closure of second Store instance, we get the assertion error saying shard is not locked.
// Ideally, we should be closing the remoteStore but until we work on CompositeStore
// (https://github.com/opensearch-project/OpenSearch/issues/3719), we mitigate the test failures by
// closing the remoteDirectory.
indexShard.getRemoteDirectory().close();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you point which assertion fails without this change?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/env/NodeEnvironment.java#L579C15-L579C15

Stacktrace looks like this on test failure:

java.lang.AssertionError: shard [index][1] is not locked
at __randomizedtesting.SeedInfo.seed([892889442ABD4835]:0)
at org.opensearch.env.NodeEnvironment.deleteShardDirectoryUnderLock(NodeEnvironment.java:579)
at org.opensearch.indices.IndicesService.deleteShardStore(IndicesService.java:1194)
at org.opensearch.index.IndexService.onShardClose(IndexService.java:662)
at org.opensearch.index.IndexService$StoreCloseListener.accept(IndexService.java:785)
at org.opensearch.index.IndexService$StoreCloseListener.accept(IndexService.java:772)
at org.opensearch.index.store.Store.closeInternal(Store.java:550)
at org.opensearch.index.store.Store$1.closeInternal(Store.java:190)
at org.opensearch.common.util.concurrent.AbstractRefCounted.decRef(AbstractRefCounted.java:78)
at org.opensearch.index.store.Store.decRef(Store.java:523)
at org.opensearch.index.engine.Engine$1.doClose(Engine.java:766)
at org.opensearch.index.engine.Engine$SearcherSupplier.close(Engine.java:1357)
at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89)
at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131)
at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:114)
at org.opensearch.common.lease.Releasables.close(Releasables.java:54)
at org.opensearch.common.lease.Releasables.close(Releasables.java:64)

}

} catch (Exception e) {
Expand Down
2 changes: 1 addition & 1 deletion server/src/main/java/org/opensearch/index/shard/IndexShard.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1975,7 +1975,7 @@ public void close(String reason, boolean flushEngine, boolean deleted) throws IO
/*
ToDo : Fix this https://github.com/opensearch-project/OpenSearch/issues/8003
*/
private RemoteSegmentStoreDirectory getRemoteDirectory() {
public RemoteSegmentStoreDirectory getRemoteDirectory() {
assert indexSettings.isRemoteStoreEnabled();
assert remoteStore.directory() instanceof FilterDirectory : "Store.directory is not an instance of FilterDirectory";
FilterDirectory remoteStoreDirectory = (FilterDirectory) remoteStore.directory();
Expand Down
Loading