Stack Monitoring: Cluster alert to detect duplicate UUIDs

[ycombinator]

Describe the feature:

Based on a suggestion by @jsvd (elastic/elasticsearch#55744 (comment)):

Create a cluster alert (one per monitored stack product?) that queries stack monitoring data for a product, aggregating first by UUID and then (sub-aggregation) by host IP. We should never see the same UUID on more than one host, so the sub-aggregation should always return exactly one bucket per UUID. If it returns more than one, then that indicates the UUID is being "reused" by another instance on another host, which indicates a misconfiguration.

Describe a specific use case for the feature:

Occasionally users will copy a stack product's install folder to another host. If the install folder includes the data folder and that data folder contains the original instance's UUID in it, this UUID gets copied over as well. Now we have 2 instances of the product on two separate hosts but both will report the same UUID to Stack Monitoring. As a result, users will incorrectly see only 1 instance in Stack Monitoring.

[ycombinator]

Pinging @elastic/stack-monitoring @elastic/stack-monitoring-ui

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)