[Tiered Caching] Indices Request cache stalekey management (opensearch-project#12625)

* Introduce IndicesRequestCacheCleanupManager

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* using cleanup mgr to enqueue cleanups

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* readability improvements

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* update updateCleanupKeyToCountMap on new cache entry

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* create IndicesRequestCacheCleanupManager & settings and validators

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* Add IRC specific cache cleaner and remove from IndicesService

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* Unit Tests

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* Update CHANGELOG.md

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* move cachecleaner inside mgr

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* remove processCleanupKeys

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* minor cleanups

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* add updateCleanupKeyToCountMapOnCacheEviction

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* remove locks and make all methods synchronized

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* spotless

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* updateCleanupKeyToCountMapOnCacheEviction

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* Testing

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* add Reschedule back to indices service

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* rename updateStaleKeysCount to incrementStaleKeysCount

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* rename getStaleKeysCountForTesting to getStaleKeysCount

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* rename threshold to stalenessThreshold

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* check for cleanupKey.entity == null

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* use computeIfPresent with keycountmap

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* log both staleKeysInCache & Staleness in debug logs

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* Use HashMap instead of ConcurrentMap

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* Address b/w compatibility

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* remove synchronized for updateCleanupKeyToCountMapOnCacheEviction

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* make cleanCache synchronized

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* remove shouldRemoveKey

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

* spotlessApply

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

---------

Signed-off-by: Kiran Prakash <awskiran@amazon.com>

Author: kiranprakash154

CHANGELOG.md

@@ -115,6 +115,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - [Metrics Framework] Adds support for asynchronous gauge metric type. ([#12642](https://github.com/opensearch-project/OpenSearch/issues/12642))
 - Make search query counters dynamic to support all query types ([#12601](https://github.com/opensearch-project/OpenSearch/pull/12601))
 - [Tiered caching] Add policies controlling which values can enter pluggable caches [EXPERIMENTAL] ([#12542](https://github.com/opensearch-project/OpenSearch/pull/12542))
+- [Tiered caching] Add Stale keys Management and CacheCleaner to IndicesRequestCache ([#12625](https://github.com/opensearch-project/OpenSearch/pull/12625))
 
 ### Dependencies
 - Bump `peter-evans/find-comment` from 2 to 3 ([#12288](https://github.com/opensearch-project/OpenSearch/pull/12288))
@@ -159,7 +160,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Fix `terms` query on `float` field when `doc_values` are turned off by reverting back to `FloatPoint` from `FloatField` ([#12499](https://github.com/opensearch-project/OpenSearch/pull/12499))
 - Fix get task API does not refresh resource stats ([#11531](https://github.com/opensearch-project/OpenSearch/pull/11531))
 - onShardResult and onShardFailure are executed on one shard causes opensearch jvm crashed ([#12158](https://github.com/opensearch-project/OpenSearch/pull/12158))
-- Avoid overflow when sorting missing last on `epoch_millis` datetime field ([#12676](https://github.com/opensearch-project/OpenSearch/pull/12676))
 
 ### Security