[Stack Monitoring] [Alerting] Investigate why "Missing monitoring data" rule is much slower than other rules on the same cluster

[jasonrhodes]

Using APM data to investigate the alerting rules and their performance, it was discovered that the "Missing monitoring data" rule appears to be running much slower than the other rules in a given cluster. (Info provided by @lizozom).

APM graphs

Acceptance Criteria

• Look into why this rule is slower than other rules (e.g. is this query doing something much different in ES, or are we processing data outside of ES?)
• If the fix is simple, we can fix it here. If not, log an implementation bugfix ticket to fix the problem.

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[neptunian]

Haven't look into this much but noticed today while looking at this rule's query that, unlike the others, it doesn't have a term filter for the metricset in the bool query. I noticed while writing unit tests for them and added one. Also left a comment https://github.com/elastic/kibana/pull/124033/files#r794930698

[lizozom]

@neptunian do you believe this should improve performance?
Maybe we could test on cloud staging once those chagnes are merged.
I'm sure that testing this kind of changes might help us understand how we want to monitor and alert on performance degredations in the future.

[neptunian]

@lizozom Yes the filter should improve performance. Will let you know when its merged.

[neptunian]

@lizozom The change to the query was merged!

[lizozom]

@neptunian any thoughts on how to benchmark this?

[neptunian]

I'll setup APM locally with some test data and see what I find.

[neptunian]

We found that having a default 1 day lookback does slow down the query when there is a lot of data in the range. @jasonrhodes did a comparison:

We've opened another issue #126709 to discuss improvements so I am closing this issue.