[Stack Monitoring] UI Improvements for Data Stream Monitoring

[trengrj]

Currently data streams indices are hidden behind "Filter for system indices" in Stack Monitoring and show the .ds-x-y-z index name rather than the data stream name. This is not very discoverable especially for new users and with Elastic Agent most users will be primarily using data streams.

Request is for the Stack Monitoring UI to better display data streams potentially via a dedicated data stream tab similar to Index Management.

[Leaf-Lin]

It appears that Filter for system indices in Kibana is inconsistent with the definition of system indices in Elasticsearch.

• In Kibana, Filter for system indices is based on whether the indices start with a dot or not.
• In Elasticsearch:
  • system indices: Listed in System Indices elasticsearch#50251, but does not have a systematic way to know them.
  • hidden indices: Having index.hidden set to true. All data stream indices are hidden by default.
  • indices start with a dot: Could be either system or hidden or neither. In 8.1.2, users can still create a dot index without having index.hidden set to true.

As @trengrj said, having a new tab for "Data Streams" would be the most intuitive way to monitor them.

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[leandrojmp]

Hello, is this stalled?

We start moving from traditional indices to data streams and we can only see the data stream indexing rate in our monitoring cluster if we enable the Filter for system indices.

We would expect to be able to see the indexing rate for the data stream as we can for the normal indices, for example assuming that we have a data stream named company-data, we would like to see the index rate in the monitoring UI for company-data, not for the .ds-company-data-backing-indice as the date in the backing indice can lead to confusion for operators that are not familiar on how data streams work.

It looks that while Data Streams are now the recommend way for indexing time based data, there are some small places where they are not fully supported as we already have issues with Logstash not fully supporting data streams.

[smith]

This work isn't currently prioritized so there are no immediate plans to work on it.