[META] Support application building using ingest and search pipelines in OpenSearch Dashboards

[ohltyler]

This is a meta issue capturing all of the details for the plugin's initial release targeted for 2.17.

Overview

Background & problem statement

Over the last several OpenSearch releases, there have been many enhancements made in ingest and search pipelines to support more flexible ML-related use cases, allowing customers to build exactly what they need. But with flexibility, comes complication and difficulty in configuration. It can be hard for users to see how their data flows and is transformed through these pipelines, as well as uncovering where certain problems may lie.

Solution

The [Flow Framework] plugin lets users build out custom search applications using ingest and search pipelines in a user-friendly UI. Users can iteratively build and test these pipelines, and visually see how data is transformed at each step. They can test out different processors, test out different configurations within individual processors, and finally, run concrete ingest and search queries to see how it performs. When users are satisfied with their solution, they can export their produced configuration via a Flow Framework workflow template. This template can then be exported and imported into different clusters, letting users scale and deploy their solution across their workloads/fleets.

Under the hood, this plugin uses the Flow Framework plugin to maintain configuration, created resources, and automatic provisioning and deprovisioning of such resources as users iteratively build out and test their applications.

Next steps

In the future, we would like to expand the plugin's functionality in several different ways. These include:

1. More processor support: onboarding and integrating with more existing ingest / search request / search pipeline processors can increase the usability and flexibility of the plugin
2. Add drag-and-drop support: rather than a form-based, guardrailed approach to configuration, the plugin could utilize the (currently read-only) ReactFlow workspace, into a customizable, drag-and-drop experience for building out their flows.
3. Advanced / DAG-based flows: currently, the flows are tied to ingest and search pipelines, which only allow a single flow of data transformations. Allowing users to build DAG-based flows and parallel executions could greatly enhance the flexibility provided to users.

Checklist

For the 2.17 release, these are the following remaining issues to be completed:

• Onboard to functional-test repo #22
• [META] New workflow page #66
• [Workflow list] Add UT coverage #96
• [New workflow] Add UT coverage #97
• [Workflow editor] Add UT coverage #98
• Integration test coverage  #101
• Re-enable linting and codecov mechanisms #109
• [New workflow] Complete initial use cases #214
• Integration with security plugin #216
• [Workflow editor] Integrate with ML inference search processors #217
• [Workflow editor] Integrate with ML commons interface updates #218
• [Workflow editor] Integrate with other required processors #219
• [FEATURE] Support Multiple Data Source in Flow Framework Dashboards Plugin #228
• Optimize components with React.memo #247
• Integrate with the fine-grained provisioning API #267
• Make form state more granular #272
• Clean up overall form state #286
• Add loading state #287
• Add input sanitization checks #295
• Improve query template building experience #404