[Stack Monitoring] Rules do not support standalone clusters

[neptunian]

if a cluster_uuid has a value of '' or it does not exist, its considered a standalone cluster. We typically do not try to filter by the cluster_uuid if we are in the context of a standalone cluster. However alerts does not get standalone clusters when fetching clusters and then it proceeds to filter the rule query with the cluster ids that will never match this standalone cluster. Like the createQuery helper function that many other queries use, we should not include a filter for standalone clusters.

This was discovered in Logstash but I think Elasticsearch could be a standalone cluster.

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[matschaffer]

This has some intersection with #125401 where we aim to catalog/ensure more products can be monitored in "standalone" mode.

[matschaffer]

With #125401 open it's a little unclear which stack components I should be targeting. I'll start with logstash and https://github.com/elastic/kibana/blob/270adf49587db4f77dbe3c46f41f92e5d739b6c1/x-pack/plugins/monitoring/server/alerts/logstash_version_mismatch_rule.ts since I know that can run in standalone mode.

Could be once I see how to adapt that, it'll just work for all rules.

But given the lack of clarity on that, I think I'm going to call this a medium.

[matschaffer]

Moving this back to ready to pick up some higher priority issues.

[matschaffer]

To "stash" my mental state on this one:

Logstash is easy to run in standalone mode and it has a rule for checking version mixing, so I did this to run 8.0.0 and 8.0.1 at the same time.

two_logstash_versions.zip

With that in hand I think we should be able to modify something in this space to handle standalone (uuid: '') as well

• kibana/x-pack/plugins/monitoring/server/lib/alerts/fetch_clusters.ts

  Line 23 in 7126bc9

  export async function fetchClusters(

• kibana/x-pack/plugins/monitoring/server/alerts/logstash_version_mismatch_rule.ts

  Line 54 in 488d939

  const logstashVersions = await fetchLogstashVersions(

[jasonrhodes]

Everything about cluster_uuid = "" -> "standalone cluster" makes me feel nervous and queasy. I don't want to blow this issue up if there's an easy fix, though, so let's try to figure that out?

However alerts does not get standalone clusters when fetching clusters

@neptunian do you mind explaining what this means a little more? I'm having trouble understanding the issue here and it feels like this ticket is in a limbo state, a little bit.

[matschaffer]

Everything about cluster_uuid = "" -> "standalone cluster" makes me feel nervous and queasy.

Me too, but unfortunately it's how things work today. ;)

kibana/x-pack/plugins/monitoring/server/lib/standalone_clusters/standalone_cluster_query_filter.ts

Lines 8 to 14 in 07f4828

	export const standaloneClusterFilter = {
	bool: {
	should: [
	{
	term: {
	cluster_uuid: {
	value: '',