[Backport 2.x] [Tiered Caching] Make took time policy dynamic and add additional integ tests

[sgup432]

Description

Backport #13063

Related Issues

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• [] Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[sgup432]

@msfroh Can you take a look at this backport PR?

[github-actions]

Compatibility status:

Checks if related components are compatible with change 0d3ba03

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/sql.git]

[github-actions]

✅ Gradle check result for 0d3ba03: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 89.09091% with 6 lines in your changes are missing coverage. Please review.

Project coverage is 71.23%. Comparing base (0dd892c) to head (0d3ba03).
Report is 130 commits behind head on 2.x.

Files	Patch %	Lines
...search/cache/common/tier/TieredSpilloverCache.java	87.50%	3 Missing and 1 partial ⚠️
...opensearch/cache/common/policy/TookTimePolicy.java	75.00%	1 Missing ⚠️
.../cache/common/tier/TieredSpilloverCachePlugin.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##                2.x   #13175      +/-   ##
============================================
- Coverage     71.28%   71.23%   -0.05%     
- Complexity    60145    60641     +496     
============================================
  Files          4957     5003      +46     
  Lines        282799   285293    +2494     
  Branches      41409    41713     +304     
============================================
+ Hits         201591   203234    +1643     
- Misses        64189    64846     +657     
- Partials      17019    17213     +194     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sgup432]

@msfroh @dblock Can someone approve/merge this?