Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix assertion failure while deleting remote backed index #14601

Merged
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Move remote store shard lock creation inline
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Sachin Kale committed Jul 2, 2024
commit 736aa636f463457267b61d698be9456e907a39fb
41 changes: 15 additions & 26 deletions server/src/main/java/org/opensearch/index/IndexService.java
Original file line number Diff line number Diff line change
@@ -597,14 +597,21 @@ public synchronized IndexShard createShard(
this.indexSettings.getRemoteStorePathStrategy()
);
}
remoteStore = new Store(
shardId,
this.indexSettings,
remoteDirectory,
getRemoteStoreLock(shardId),
Store.OnClose.EMPTY,
path
);
// When an instance of Store is created, a shardlock is created which is released on closing the instance of store.
// Currently, we create 2 instances of store for remote store backed indices: store and remoteStore.
// As there can be only one shardlock acquired for a given shard, the lock is shared between store and remoteStore.
// This creates an issue when we are deleting the index as it results in closing both store and remoteStore.
// Sample test failure: https://github.com/opensearch-project/OpenSearch/issues/13871
// The following method provides ShardLock that is not maintained by NodeEnvironment.
// As part of https://github.com/opensearch-project/OpenSearch/issues/13075, we want to move away from keeping 2
// store instances.
ShardLock remoteStoreLock = new ShardLock(shardId) {
@Override
protected void closeInternal() {
// Do nothing for shard lock on remote store
}
};
remoteStore = new Store(shardId, this.indexSettings, remoteDirectory, remoteStoreLock, Store.OnClose.EMPTY, path);
} else {
// Disallow shards with remote store based settings to be created on non-remote store enabled nodes
// Even though we have `RemoteStoreMigrationAllocationDecider` in place to prevent something like this from happening at the
@@ -627,7 +634,6 @@ public synchronized IndexShard createShard(
} else {
directory = directoryFactory.newDirectory(this.indexSettings, path);
}

store = new Store(
shardId,
this.indexSettings,
@@ -685,23 +691,6 @@ public synchronized IndexShard createShard(
}
}

// When an instance of Store is created, a shardlock is created which is released on closing the instance of store.
// Currently, we create 2 instances of store for remote store backed indices: store and remoteStore.
// As there can be only one shardlock acquired for a given shard, the lock is shared between store and remoteStore.
// This creates an issue when we are deleting the index as it results in closing both store and remoteStore.
// Sample test failure: https://github.com/opensearch-project/OpenSearch/issues/13871
// The following method provides ShardLock that is not maintained by NodeEnvironment.
// As part of https://github.com/opensearch-project/OpenSearch/issues/13075, we want to move away from keeping 2
// store instances.
private ShardLock getRemoteStoreLock(ShardId shardId) {
return new ShardLock(shardId) {
@Override
protected void closeInternal() {
// Ignore for remote store
}
};
}

/*
Fetches the shard path based on the index type -
For a remote snapshot index, the cache path is used to initialize the shards.