[BUG] It's not possible to restore an index from snapshot made on remote_store enabled cluster to non remote_store cluster

[eliskovets]

Describe the bug

It's not possible to restore an index from a snapshot made on the remote_state enabled Opensearch cluster to non remote-store Opensearch cluster.

I've tried to add the index.remote_state index setting to being ignored, but it looks like it didn't do anything and remote_state settings are still there. Then it fails to initialize the shard as none of the nodes are remote-store nodes:

[2025-03-07T17:54:13,151][WARN ][o.o.c.r.a.AllocationService] [i-079edbef824e697aa] failing shard [failed shard, shard [sandbox][0], node[ZdN2RlZGQd6LNa495HT6iQ], [P], recovery_source[snapshot recovery [a4Gsp6DxQwmcev1g97a4ZQ] from sandbox-v29-0:sandbox_v29.0/sQYJAaZATHS7liuMbs-IPw], s[INITIALIZING], a[id=CQcnwKb-TcKd7AOFHYk6Mg], unassigned_info[[reason=ALLOCATION_FAILED], at[2025-03-07T17:54:13.053Z], failed_attempts[4], failed_nodes[[ZdN2RlZGQd6LNa495HT6iQ]], delayed=false, details[failed shard on node [ZdN2RlZGQd6LNa495HT6iQ]: failed to create shard, failure IllegalStateException[[{[sandbox][0]}] Cannot initialize shards with remote store index settings on non-remote store nodes]], allocation_status[fetching_shard_data]], expected_shard_size[10783478], message [failed to create shard], failure [IllegalStateException[[{[sandbox][0]}] Cannot initialize shards with remote store index settings on non-remote store nodes]], markAsStale [true]]

Related component

No response

To Reproduce

1. Start cluster with remote-backed storage configured
2. Index some data
3. Create snapshot
4. Deploy another cluster without remote-backed storage configured
5. Register the OpenSearch repo and try to restore the snapshot to the new cluster
6. See error

Expected behavior

Non remote_store clusters ignore remote_store setting in the index or ignore_index_settings allows you to ignore the index.remote_store setting on restore.

Additional Details

Plugins

• discovery-ec2
• repository-s3

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: public.ecr.aws/opensearchproject/opensearch
• Version 2.18.0

Additional context
Add any other context about the problem here.