[Stack Monitoring] Product components should be monitor-able in isolation

[jasonrhodes]

I expect a user to be able to monitor each of the following products in isolation without requiring monitoring of any other product:

• Elasticsearch
• Kibana
• Logstash
• APM
• Enterprise Search

It appears as though this is not the case right now with APM (see below). We should confirm which of the above products have this problem and fix them. If the fixes are different for each, we should create new issues for each one.

FINALLY got this working. The trick? You need to monitor Elasticsearch as well. I was only monitoring APM. Enabling Elasticsearch monitoring enables the Stack Monitoring UI as the UI keys off the cluster_uuid field of .monitoring-es-* first.

Originally posted by @bmorelli25 in elastic/apm-server#5987 (comment)

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[jasonrhodes]

@matschaffer I think you had some additional context here on our expectations re: standalone cluster, but I didn't quite understand (I guess monitoring assumes that it is always set up to monitor "a cluster" rather than monitoring individual products that may or may not be grouped into clusters?). Please feel free to add thoughts via comment or update the description, thanks!

[matschaffer]

So, in SM UI "Standalone Cluster" kicks in when you have monitoring documents with no cluster_uuid set. https://github.com/elastic/observability-dev/blob/main/docs/monitoring/monitoring-simulation-notes.md#standalone-cluster (internal) covers how to make some of those docs.

As far as I've seen only logstash can really enter this mode. Beats will show up in that UI occasionally, but I've only ever seen error documents (can't contact elasticsearch, so no cluster_uuid) show up for beats.

Agreed, I'd be nice to have a clean documented way for each component to be monitored without the requirement of being associated with a monitored elasticsearch cluster.

[miltonhultgren]

I think Kibana works the same way, that you first resolve the Elasticsearch cluster that it is connected to and only then can you display stats about Kibana? @matschaffer isn't that what we saw when trying to mock the UI with synthtrace?

[matschaffer]

I don't recall if I managed to get a synthetic standalone kibana or not. I think the UI should handle it fine when it happens though.

The question on my mind is how many stack components are capable of sending documents like that (not simulated). I think it might be only logstash at the moment.

[neptunian]

Something to be mindful of if we extend the concept of a "standalone cluster" to other products is they are currently auto supported regardless of their license type.

[pmeresanu85]

This works should be done in the existing Stack Monitoring UI, doesn't fit into our Platform Observability initiative.

[klacabane]

The package-based monitoring approaching GA looks like a fair milestone to revive this issue. The dependency on elasticsearch being monitored for other products to show up is a confusing experience for users and also a DX friction point for SM maintainers.

I've put up a dirty PR that allows each product to be monitored independently, whether elasticsearch's cluster_stats documents are being ingested or not. Note that this guard currently exists for licensing reasons and we should revisit that decision before moving forward since such change needs to bypass the license verification.

See draft PR #147183 for more details

[smith]

Reopened and moved to top of the backlog.