[Stack Monitoring] Remove internal collection code

[ycombinator]

The work described in this issue is a breaking change. As such, it is intended to be released only in a major release (i.e. no backports).

We now have the ability to monitor Elasticsearch, Kibana, Logstash, Beats, and APM Server with Metricbeat. The previous method of monitoring stack products, using collection code internal to each product that shipped monitoring data to a custom Monitoring Bulk API endpoint, is now deprecated. And in the next major release (likely 8.0.0), all code related to this internal collection approach should be removed.

This issue tracks the removal of one specific item related to the internal collection approach: the code within the Kibana Stack Monitoring plugin that periodically collects monitoring metrics to construct kibana_stats and kibana_settings documents, and ships them to the custom Monitoring Bulk API in Elasticsearch (POST _monitoring/bulk).

By removing this internal collection code Kibana will no longer be able to monitor itself (for the purposes of the Stack Monitoring application in Kibana). Instead, users will have to run Metricbeat along side Kibana and enable the kibana-xpack module. This module knows how to poll the Kibana Stats API (GET /api/stats) to periodically collect monitoring metrics and ship them to the monitoring Elasticsearch cluster.

While removing the internal collection code, be careful to make sure that the Kibana Stats API continues to function as-is, as it is required for Metricbeat collection.

[elasticmachine]

Pinging @elastic/stack-monitoring

[cachedout]

@chrisronline Do you think that this is something we'll need to take on ourselves or should we be involving the Kibana core team at all? I'm just trying to think about when we'll be doing this work and who should take care of it.

[chrisronline]

This will most likely fall onto us, as the majority of the code lives in our plugin.

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)