[Stack Monitoring] Tolerate lack of elasticsearch metricset

[klacabane]

Summary

When configuring xpack.enabled: true in Elasticsearch beats module, one loses the ability to select the subset of metricsets they want to collect since they are hardcoded. One could set xpack.enabled: false to allow configuration of the metricsets but as a result metricbeat will ship all the elasticsearch module documents to the metricbeat-* indice which is no longer supported by Stack Monitoring as of 8.1.0, so that is not an option if they're still consuming the data through Stack Monitoring UI.

Stopping the collection of a metricset allows one to reduce the disk footprint that can grow expensive in large scale clusters (eg index metricset), as well as removing bottlenecks of linear scaling APIs (eg shard metricset). Code wise it would make the SM modules act like a regular metricbeat module and provide a consistent ux for the API consumers, at the cost of a sparse UI

Implementation option

On the metricbeat side, the change is straightforward and we could remove the hardcoded list in favor of a customer-defined configurations pulled from metricbeat.yml.

The Stack Monitoring UI will need more changes and investigation to determine the impact of a metricset removal, but once identified we could gracefully handle the absence of specific metricsets and potentially provide actionable UI feedback if it's not intended ie No index data found, ensure you have _index_ metricset turned on.

A dependency tree of the metricsets consumed by each Elasticsearch views would be useful to identify what can be removed easily or not.

Acceptance Criteria

• With the exception of cluster_stats, the Stack Monitoring UI can load without crashing due to the lack of existence of any other metricset in the result data
• Nice to have: For some metricesets, a user-friendly message exists letting the user know that a metricset is missing and can be enabled via "x" config

Note: This may need to turn into an epic to track work per metricset, if it turns out to be complicated. We will also need an issue in elastic/beats to represent the configuration change, once the UI has been adapted.

Questions

• Should all the metricsets be configurable considering that cluster_stats documents are mandatory to power Stack Monitoring ?

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[jasonrhodes]

This is also particularly important for customers who may be seeing performance problems based on the collection of a particular metricset but would like to continue collecting the rest of the metrics from Stack Monitoring (index_recovery seems to have this issue at times, in addition to the shards one mentioned in the description).

[jasonrhodes]

• Should all the metricsets be configurable considering that cluster_stats documents are mandatory to power Stack Monitoring ?

I think everything but cluster_stats should be configurable, then, unless we find reasons why others should also be non-configurable.