Use ClusterStateRequest with index pattern when searching for expired local indices

[dzane17]

Description

Currently we scan a list of all indices on a domain when looking for expired Top N local indices. This causes extra computation on domains with many indices & greater possibility of mistake.

This PR reduces risk by limiting the indices fetched to those that match the top_queries-* name pattern. It then applies the existing protocol, which includes a stricter check for the index name (top_queries-<date>-<5 digits>), as well as validation of creation time and metadata before any deletions are performed.

Testing

1. Create a few indices including 1 real Query Insights local index

% curl -X GET "localhost:9200/_cat/indices?v&s=index"                                                       
health status index                        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   logs-1                       LUay8iGnTZSSW3rjMUHiKA   1   0          0            0       208b           208b
green  open   logs-2                       I1y6WAWkSF2mTy8QtX1Pcg   1   0          0            0       208b           208b
yellow open   my_index                     h27P91J7TNCdv1rMhFB6fg   1   1          2            0      6.4kb          6.4kb
green  open   top_queries-2020.01.01-12345 _AaDcAN4TEujGrFMjGLidw   1   0          0            0       208b           208b
green  open   top_queries-2025.03.10-21661 cyi2BDaLS42BlO6lmwpO9Q   1   0          1            2       29kb           29kb

2. Update delete_after_days setting to trigger expired index scan + deletion

% curl -XPUT "http://localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'          
{              
  "persistent" : {        
    "search.insights.top_queries.exporter.delete_after_days" : "5"
  }
} 
'
{
  "acknowledged" : true,
  "persistent" : {
    "search" : {
      "insights" : {
        "top_queries" : {
          "exporter" : {
            "delete_after_days" : "5"
          }
        }
      }
    }
  },
  "transient" : { }
}

3. Verified with logging, only indices with name matching top_queries-* pattern were considered for deletion

[2025-03-10T16:24:45,187][INFO ][o.o.c.s.ClusterSettings  ] [integTest-0] updating [search.insights.top_queries.exporter.delete_after_days] from [7] to [5]
[2025-03-10T16:24:45,188][ERROR][o.o.p.i.c.s.QueryInsightsService] [integTest-0] in deleteExpiredTopNIndices
[2025-03-10T16:24:45,188][ERROR][o.o.p.i.c.s.QueryInsightsService] [integTest-0] top_queries-2020.01.01-12345
[2025-03-10T16:24:45,189][ERROR][o.o.p.i.c.s.QueryInsightsService] [integTest-0] top_queries-2025.03.10-21661

Issues Resolved

Resolves #261

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[ansjcy]

I think you also need to make the same changes here:

query-insights/src/main/java/org/opensearch/plugin/insights/core/service/QueryInsightsService.java

Line 622 in 5ef5437

void deleteAllTopNIndices(final Client client, final LocalIndexExporter localIndexExporter) {

[ansjcy]

Another improvement we can do is, on this line

query-insights/src/main/java/org/opensearch/plugin/insights/core/service/TopQueriesService.java

Line 555 in 5ef5437

return parts[2].matches("\\d{5}");

instead of just parts[2].matches("\\d{5}");, we can call this function to verify if the hash is correct as well.

query-insights/src/main/java/org/opensearch/plugin/insights/core/utils/ExporterReaderUtils.java

Line 29 in 5ef5437

public static String generateLocalIndexDateHash(LocalDate date) {

Let me know if you want to include it in this PR as well, if not I'll probably do it in another PR.

[ansjcy]

According to the java doc, the local parameter is defined as: "Return local information, do not retrieve the state from cluster-manager node (default: false).".

using data from cluster manager node requires one extra network hop, it will also have higher latency, but the benifit is we will always have the most accurate info..

But in our case, the absolute consistency is not critical neither.. I'm fine with either approach, just leave the info here in case anyone has other opinions.

[dzane17]

Is cluster state equivalent on cluster manager vs non cluster manager nodes? I am wondering if local node info can be incomplete or stale.

[dzane17]

I will update deleteAllTopNIndices() and the 5 digit check in isTopQueriesIndex().

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/query-insights/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/query-insights/backport-2.x
# Create a new branch
git switch --create backport/backport-262-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 efb871cede2fbb23d5a5c0a0ef9f3c46172743eb
# Push it to GitHub
git push --set-upstream origin backport/backport-262-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/query-insights/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-262-to-2.x.