[Rules and Alerting][Stack Monitoring] during a deployment change from ESS; ECE; ECK alerts should be muted.

[philippkahr]

Describe the feature:

When we run a change e.g. adding nodes, upgrading, .... The built in stack rules go crazy and alert a ton because node changed, version mismatch and so on.

Describe a specific use case for the feature:
I would expect that ESS; ECE; ECK "talks" to the stack alerts and mutes them / reconfigures them, so that during a deployment change, no alert is created. Since there is no need for one, as this is expected.

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[jasonrhodes]

@ravikesarwani I know you've spent time thinking about this in the past, FYI.

[ravikesarwani]

Not something we have discussed and planned so far but feels like something important that we should think about.
I do feel that it will have to be something driven from the platform/cloud team perspective as that can only know when a plan change started/completed. The Kibana/alerting framework should provide some API way that the cloud team can use to enable and disable the rules.

[philippkahr]

I think there are a few edge cases that also need to be taken care off. Like missing monitoring data for node, if I do a downsizing deployment, or an instance gets replaced and thus gets a new name, the old node will not send any more data, thus the missing monitoring data for node will be triggered.

[miltonhultgren]

How does the future Health/Topology APIs play into this? Has there been any thought for how Cloud might "tell" those APIs "I'm changing things under you, chill out for a bit"?

[matschaffer]

If the alert API has a mute option, I'm thinking it might make sense for the orchestrator to call it before operations start. We could potentially tag the rules with metadata about which ones are expected to get noisy during certain workflows (like upgrading).

[smith]

Moved to Refining on our board. We need to determine and decide whether this is something that we could/would implement or otherwise route this issue to the correct place.

[smith]

Closing this. It doesn't look like something @elastic/infra-monitoring-ui would be capable of implementing at this time.

[philippkahr]

Kibana has a mute api, maybe we should revisit this issue?