[Tiered Caching] Indices Request cache stalekey management (opensearc…

…h-project#12625)

* Introduce IndicesRequestCacheCleanupManager

Signed-off-by: Kiran Prakash <[email protected]>

* using cleanup mgr to enqueue cleanups

Signed-off-by: Kiran Prakash <[email protected]>

* readability improvements

Signed-off-by: Kiran Prakash <[email protected]>

* update updateCleanupKeyToCountMap on new cache entry

Signed-off-by: Kiran Prakash <[email protected]>

* create IndicesRequestCacheCleanupManager & settings and validators

Signed-off-by: Kiran Prakash <[email protected]>

* Add IRC specific cache cleaner and remove from IndicesService

Signed-off-by: Kiran Prakash <[email protected]>

* Unit Tests

Signed-off-by: Kiran Prakash <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Kiran Prakash <[email protected]>

* move cachecleaner inside mgr

Signed-off-by: Kiran Prakash <[email protected]>

* remove processCleanupKeys

Signed-off-by: Kiran Prakash <[email protected]>

* minor cleanups

Signed-off-by: Kiran Prakash <[email protected]>

* add updateCleanupKeyToCountMapOnCacheEviction

Signed-off-by: Kiran Prakash <[email protected]>

* remove locks and make all methods synchronized

Signed-off-by: Kiran Prakash <[email protected]>

* spotless

Signed-off-by: Kiran Prakash <[email protected]>

* updateCleanupKeyToCountMapOnCacheEviction

Signed-off-by: Kiran Prakash <[email protected]>

* Testing

Signed-off-by: Kiran Prakash <[email protected]>

* add Reschedule back to indices service

Signed-off-by: Kiran Prakash <[email protected]>

* rename updateStaleKeysCount to incrementStaleKeysCount

Signed-off-by: Kiran Prakash <[email protected]>

* rename getStaleKeysCountForTesting to getStaleKeysCount

Signed-off-by: Kiran Prakash <[email protected]>

* rename threshold to stalenessThreshold

Signed-off-by: Kiran Prakash <[email protected]>

* check for cleanupKey.entity == null

Signed-off-by: Kiran Prakash <[email protected]>

* use computeIfPresent with keycountmap

Signed-off-by: Kiran Prakash <[email protected]>

* log both staleKeysInCache & Staleness in debug logs

Signed-off-by: Kiran Prakash <[email protected]>

* Use HashMap instead of ConcurrentMap

Signed-off-by: Kiran Prakash <[email protected]>

* Address b/w compatibility

Signed-off-by: Kiran Prakash <[email protected]>

* remove synchronized for updateCleanupKeyToCountMapOnCacheEviction

Signed-off-by: Kiran Prakash <[email protected]>

* make cleanCache synchronized

Signed-off-by: Kiran Prakash <[email protected]>

* remove shouldRemoveKey

Signed-off-by: Kiran Prakash <[email protected]>

* spotlessApply

Signed-off-by: Kiran Prakash <[email protected]>

---------

Signed-off-by: Kiran Prakash <[email protected]>