[Backport 2.9] Backport #1302, #1355, #1432, #1435, #1445, #1403, #1430, #1441 to 2.9

[eirsep]

Backports following PRs to 2.9

• optimize doc-level monitor execution workflow for datastreams #1302
• Bulk index findings and sequentially invoke auto-correlations #1355
• fix for MapperException[the [enabled] parameter can't be updated for the object mapping [metadata.source_to_query_index_mapping] #1432
• Add jvm aware setting and max num docs settings for batching docs for percolate queries #1435
• optimize sequence number calculation and reduce search requests in doc level monitor execution #1445
• add distributed locking to jobs in alerting #1403
• Clean up doc level queries on dry run #1430
• optimize to fetch only fields relevant to doc level queries in doc level monitor instead of entire _source for each doc #1441

[codecov]

Codecov Report

Attention: Patch coverage is 78.65330% with 149 lines in your changes are missing coverage. Please review.

Project coverage is 68.64%. Comparing base (0269dd1) to head (e74b166).
Report is 2 commits behind head on 2.9.

Files	Patch %	Lines
.../opensearch/alerting/DocumentLevelMonitorRunner.kt	82.89%	31 Missing and 21 partials ⚠️
...n/org/opensearch/alerting/core/lock/LockService.kt	58.73%	45 Missing and 7 partials ⚠️
.../org/opensearch/alerting/MonitorMetadataService.kt	52.63%	9 Missing and 9 partials ⚠️
...opensearch/alerting/util/DocLevelMonitorQueries.kt	77.14%	5 Missing and 3 partials ⚠️
...in/org/opensearch/alerting/MonitorRunnerService.kt	88.00%	4 Missing and 2 partials ⚠️
.../kotlin/org/opensearch/alerting/util/IndexUtils.kt	70.58%	1 Missing and 4 partials ⚠️
...lerting/transport/TransportDeleteWorkflowAction.kt	60.00%	2 Missing ⚠️
.../alerting/transport/TransportIndexMonitorAction.kt	66.66%	0 Missing and 2 partials ⚠️
...lin/org/opensearch/alerting/core/lock/LockModel.kt	96.55%	0 Missing and 2 partials ⚠️
...opensearch/alerting/model/IndexExecutionContext.kt	88.88%	1 Missing ⚠️
... and 1 more

Additional details and impacted files

@@             Coverage Diff              @@
##                2.9    #1469      +/-   ##
============================================
+ Coverage     68.35%   68.64%   +0.29%     
- Complexity      120      143      +23     
============================================
  Files           160      162       +2     
  Lines         10740    11268     +528     
  Branches       1628     1697      +69     
============================================
+ Hits           7341     7735     +394     
- Misses         2680     2774      +94     
- Partials        719      759      +40     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jowg-amazon]

Are we backporting #1139 to 2.9?