Top N indices auto deletion config & functionality

[dzane17]

Description

This PR introduces two features related to the retention of Top N historical data:

1. New Cluster Setting: search.insights.top_queries.delete_after_days
2. Scheduled job to automatically deletes expired indices

Details:

• search.insights.top_queries.delete_after_days: This integer setting determines the retention period in days for Top N indices. Default value is 7 days. Acceptable range is 1 to 180 days.
• Setting is only applicable when exporter.type is local_index
• Single setting applies to all metric types
• Deletion job runs when delete_after_days setting is updated and once per day, in line with the top_queries index rollover period (1 day). Looks for indices that match the naming pattern top_queries-YYYY.MM.dd-xxxxx
• Any time exporter.type is changed from local_index, all existing local indices are deleted. This is so that indices do not linger when sink type is not local_index.

Testing

% curl -X GET "localhost:9200/_cat/indices?v&s=index"                                                               
health status index                        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   test2                        I4DeK5ALSOizWRYLbpTGBQ   1   1          4            0      9.2kb          9.2kb
yellow open   top_queries-2025.01.04-62052 kKOZP1ivRlGTtaMP0sXQdw   1   1          7            0     58.1kb         58.1kb

% curl -XGET "http://localhost:9200/_insights/top_queries?pretty&from=2025-01-01T00:00:00.000Z&to=2025-01-05T00:00:00.000Z"
{
  "top_queries" : [
    {
      "timestamp" : 1735954349215,
      "total_shards" : 1,
      "indices" : [ ],
      "phase_latency_map" : {
        "expand" : 0,
        "query" : 42,
        "fetch" : 2
      },
...

search.insights.top_queries.delete_after_days validation:

% curl -XPUT "http://localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'          
{
  "persistent" : {
    "search.insights.top_queries.delete_after_days" : "999" 
  }
}
'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "illegal value can't update [search.insights.top_queries.delete_after_days] from [1] to [999]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "illegal value can't update [search.insights.top_queries.delete_after_days] from [1] to [999]",
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "Invalid exporter delete_after_days setting [999], value should be an integer between 1 and 180."
    },
    "suppressed" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "illegal value can't update [search.insights.top_queries.delete_after_days] from [1] to [999]",
        "caused_by" : {
          "type" : "illegal_argument_exception",
          "reason" : "Invalid exporter delete_after_days setting [999], value should be an integer between 1 and 180."
        }
      },
      {
        "type" : "illegal_argument_exception",
        "reason" : "illegal value can't update [search.insights.top_queries.delete_after_days] from [1] to [999]",
        "caused_by" : {
          "type" : "illegal_argument_exception",
          "reason" : "Invalid exporter delete_after_days setting [999], value should be an integer between 1 and 180."
        }
      }
    ]
  },
  "status" : 400
}

Issues Resolved

Resolves #165

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[ansjcy]

Have we done tests on the actual delete functionalities? Since the min value allowed for delete after is 1 day it might be hard to test. We can probably temporarily modify the min allowed value to 1 minute and change all the TimeUnit.Days to Minutes and test if the deletion work as expected - it should delete all the "top_queries*" indices and shouldn't delete any other indices.

[dzane17]

Have we done tests on the actual delete functionalities? Since the min value allowed for delete after is 1 day it might be hard to test. We can probably temporarily modify the min allowed value to 1 minute and change all the TimeUnit.Days to Minutes and test if the deletion work as expected - it should delete all the "top_queries*" indices and shouldn't delete any other indices.

I manually tested with 1 min rollover & 5-10 min retention. It worked as expected.

[ansjcy]

LGTM. Nice work on the data retention change.