[Proposal] - Advanced Data Selector

[shanilpa]

Overview

As OpenSearch Dashboards expands to support new data sources (Multiple Data Source, Apache Spark integration, S3 via OpenSearch Observability) there is a growing need to offer a more robust and transparent data selection experience above the simple selector proposed here.

Advanced Data Selection

The advanced data selector aims to solve these key challenges based on the context above:

1. How do we create a scalable data selection experience that supports multiple data connections?
2. How do we allow users to have more transparency and control over the data they are selecting?
3. How do we make it easier for users to create index patterns and other abstractions in context?
4. How do we make complex data selection faster for users that often select the same data?

Advanced.Data.Selector.mp4

Aid in data understanding

Allowing users to inspect a data set to understand meta data such as the total number of fields, field names, and types to aid in the data selection process

Quick selection through favorites and recently selected

Users are able to more quickly select data through favorites (data sources they frequently selected) and recently selected items. Users can add favorites through the star action or via the selected tab for a group of selections.

Contextual index pattern creation

Users no longer have to navigate out of Discover to create an index pattern (however if they would like they still can) - they can simply enter a wild card (*) to match indexes that follow the pattern. The index pattern is created on data selection with a default timestamp auto-selected based on some heuristics. Users can leverage the index pattern feature under Dashboards Management to perform index pattern operations.

States

If data connections are not setup users are presented with an empty state to either refresh or deep link to the data source configuration page to setup connections

In the event data for a specific connection can not be retrieved we present users with an error callout

[paulstn]

If support for multiple queries to appear on the page does eventually come for Discover, how will the datasource selector support all of that? Is it meant to act as an introductory filter for every query that would appear on the page? Would it make more sense to provide every query with its own datasource selector?