Optimize refresh usage when persisting alerts to reduce Elasticsearch overhead

[mikecote]

Description

Performance tests on the alerting framework have revealed that the current use of refresh: true during the persisting of alerts step can be resource-intensive for Elasticsearch.

An upcoming PR will set refresh: wait_for for times Kibana runs with stateful Elasticsearch. However, refresh: true is kept for times Kibana runs with stateless Elasticsearch where throttling is applied during bursts. While this mitigates some of the overhead, it introduces additional latency whenever persisting alerts.

To further optimize performance, it would be beneficial to dynamically set refresh: false when we don't need to search the data immediately after persisting it. Scenarios requiring searchable alerts after persisting include:

• Maintenance windows
• Conditional actions
• Alert summaries

There may be additional scenarios not identified here. We should audit the use cases as we implement the optimization.

Definition of Done

• Dynamically set refresh based on whether searching after writing is required
• Performance tests confirm reduced overhead
• Tests

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)