Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] [Remote State] Data in remote state repository is stored outside of the base_path #16449

Open
ikorchynskyi opened this issue Oct 23, 2024 · 2 comments
Open

[BUG] [Remote State] Data in remote state repository is stored outside of the base_path #16449

ikorchynskyi opened this issue Oct 23, 2024 · 2 comments
Labels
bug Something isn't working Storage:Remote

Comments

@ikorchynskyi
Copy link

Describe the bug

After the migration to remote cluster state the corresponding S3 bucket contains data not only under the specified base_path prefix, but also under the root.

Related component

Storage:Remote

To Reproduce

  1. Setup an empty S3 bucket for a remote state
  2. Configure the remote state for a cluster:
# Enable Remote cluster state cluster settings:
cluster.remote_store.state.enabled: true

# Remote cluster state repository settings
node.attr.remote_store.state.repository: state
node.attr.remote_store.repository.state.type: s3
node.attr.remote_store.repository.state.settings.bucket: my-remote-store
node.attr.remote_store.repository.state.settings.base_path: state
node.attr.remote_store.repository.state.settings.region: us-east-1
  1. Complete a rolling restart of a cluster
  2. Check S3 bucket content:
$ aws s3 ls s3://my-remote-store/ --recursive
2024-10-22 21:16:52       1872 -9U1S2V-Rzg/state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/index/5t0Qx14-S7eHxPjp2M-tiQ/5t0Qx14-S7eHxPjp2M-tiQ/metadata__9223372036854775782__9223370307230164696__2
2024-10-22 21:27:06       2235 -A9zMTlJJm4/state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/index/ZFMebiN2SjS2HUoZ35iZxg/ZFMebiN2SjS2HUoZ35iZxg/metadata__9223372036854775776__9223370307229550119__2
2024-10-22 21:18:40       2324 -TJ76-0S4XA/state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/index/vmNdhpLBQiK0-uBG18Ut7A/vmNdhpLBQiK0-uBG18Ut7A/metadata__9223372036854775778__9223370307230055970__2
2024-10-22 20:37:34      20920 -gHMlojzrB4/state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/index/XtSLmDsXTJ6zaEm8qqnwSw/XtSLmDsXTJ6zaEm8qqnwSw/metadata__9223372036854775774__9223370307232522092__2
2024-10-23 02:02:55       1872 -qj0gdpE0bg/state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/index/XMZF_KcgSr-2aezWtpKx9Q/XMZF_KcgSr-2aezWtpKx9Q/metadata__9223372036854775782__9223370307213001042__2

# omitted

2024-10-22 19:59:49        263 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/global-metadata/coordination__9223372036854754695__9223370307234788933__1
2024-10-22 19:59:49         32 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/global-metadata/custom--data_stream__9223372036854754695__9223370307234788879__1
2024-10-22 19:59:48      25903 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/global-metadata/custom--index-graveyard__9223372036854754695__9223370307234788832__1
2024-10-22 19:59:49      12419 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/global-metadata/custom--index_template__9223372036854754695__9223370307234788853__1
2024-10-22 19:59:49        442 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/global-metadata/custom--ingest__9223372036854754695__9223370307234788824__1
2024-10-22 19:59:49        293 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/global-metadata/custom--repositories__9223372036854754695__9223370307234788883__1
2024-10-22 19:59:48        450 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/global-metadata/settings__9223372036854754695__9223370307234789084__1
2024-10-22 19:59:49         40 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/global-metadata/templates__9223372036854754695__9223370307234788906__1
2024-10-22 20:25:03        250 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/global-metadata/transient-settings__9223372036854754404__9223370307233273771__1
2024-10-23 09:35:11      50077 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest__9223372036854775768__9223372036854745279__C__9223370307185864892__4
2024-10-23 09:35:11      50077 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest__9223372036854775768__9223372036854745279__P__9223370307185865123__4
2024-10-23 09:35:10      50077 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest__9223372036854775768__9223372036854745280__C__9223370307185866298__4
2024-10-23 09:35:10      50077 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest__9223372036854775768__9223372036854745280__P__9223370307185866351__4
2024-10-23 09:35:07      50077 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest__9223372036854775768__9223372036854745281__C__9223370307185869634__4
2024-10-23 09:35:07      50077 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest__9223372036854775768__9223372036854745281__P__9223370307185869753__4
2024-10-23 09:35:05      50077 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest__9223372036854775768__9223372036854745282__C__9223370307185871484__4
2024-10-23 09:35:05      50077 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest__9223372036854775768__9223372036854745282__P__9223370307185871562__4
2024-10-23 09:07:51      50071 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest__9223372036854775768__9223372036854745283__C__9223370307187505540__4
2024-10-23 09:07:51      50071 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest__9223372036854775768__9223372036854745283__P__9223370307187505601__4
2024-10-23 09:05:39      50077 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest__9223372036854775768__9223372036854745284__C__9223370307187637439__4
2024-10-23 09:05:39      50077 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest__9223372036854775768__9223372036854745284__P__9223370307187637491__4
2024-10-23 09:05:35      50049 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest__9223372036854775768__9223372036854745285__C__9223370307187641132__4
2024-10-23 09:05:35      50049 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest__9223372036854775768__9223372036854745285__P__9223370307187641226__4
2024-10-23 09:05:35      50071 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest__9223372036854775768__9223372036854745286__C__9223370307187641603__4
2024-10-23 09:05:35      50071 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest__9223372036854775768__9223372036854745286__P__9223370307187641718__4
2024-10-23 09:05:34      50071 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest__9223372036854775768__9223372036854745287__C__9223370307187641900__4
2024-10-23 09:05:34      50071 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest__9223372036854775768__9223372036854745287__P__9223370307187641986__4
2024-10-22 20:11:47       2393 tFrhHnC1gLI/state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/index/NmfqSC3cSQ2Rtw55NbwsYQ/NmfqSC3cSQ2Rtw55NbwsYQ/metadata__9223372036854775780__9223370307234069245__2
2024-10-22 22:39:20      80504 thHTpF3i0OE/state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/index/Sv_nGZxCTY-H46XUEc49eg/Sv_nGZxCTY-H46XUEc49eg/metadata__9223372036854775596__9223370307225216796__2

# omitted

In total, there is

$ aws s3 ls s3://my-remote-store/ | wc -l
182

The amount of extra folders keeps growing over time.

Expected behavior

Remote state data is stored fully under the state/ base_path

Additional Details

No response
@ikorchynskyi ikorchynskyi added bug Something isn't working untriaged labels Oct 23, 2024
@ikorchynskyi
Copy link
Author

The cluster was on 2.15.0 and has been migrated to 2.17.0 before the switch to the remote state store:

$ opensearch-cli --profile my-cluster curl get --pretty --path '/'
{
  "name" : "my-cluster-data-3",
  "cluster_name" : "my-cluster",
  "cluster_uuid" : "KI73-trAR1WFLGmX3SzYmQ",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.17.0",
    "build_type" : "tar",
    "build_hash" : "8586481dc99b1740ca3c7c966aee15ad0fc7b412",
    "build_date" : "2024-09-13T01:05:31.102264964Z",
    "build_snapshot" : false,
    "lucene_version" : "9.11.1",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

@dblock dblock removed the untriaged label Nov 11, 2024
@dblock
Copy link
Member

dblock commented Nov 11, 2024

[Catch All Triage - 1, 2, 3, 4]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working Storage:Remote
Projects
Storage Project Board
Status: 🆕 New
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dblock @ikorchynskyi