[Security Solution] Exclude frozen tier nodes from detection rule queries

[banderror]

Summary

Some of our users have had issues with detection rules hitting data_frozen Elasticsearch nodes. This is usually accompanied by spikes in CPU usage and rule execution duration (rules become slow).

We should explicitly disallow rules to query frozen tier nodes, or explicitly allow to query only hot (+ maybe warm) tier. This can be achieved by including a filter by the _tier field into all queries initiated by any detection rules:

https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-tier-field.html

Options

Option 1: non-configurable behavior

One option is to implement this filtering under the hood without letting the user change this behavior.

Option 2: configurable behavior

Another option would be to introduce a new tiers rule parameter which could be shown in the "advanced rule settings" section on the Creation and Editing pages. This would be an array of strings with the following allowed values: data_hot, data_warm, data_cold, and data_frozen. The default value would be ['data_hot'] or ['data_hot', 'data_warm']. The user would be able to change this setting per rule.

In addition, we could have an advanced Kibana setting for letting the user change this default value. Changing it would allow them to have a different default value when a new rule gets created.

cc @yctercero @marshallmain @peluja1012 @approksiu

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[yctercero]

Closing this as a duplicate of #130517

Definitely worth discussing to at least get designs and requirements going.