From 6721a6b573c8ce027dd464488a01f97501aad8ce Mon Sep 17 00:00:00 2001 From: Daniel Widdis Date: Mon, 12 Feb 2024 10:37:33 -0800 Subject: [PATCH 1/3] Update README and Developer Guide Signed-off-by: Daniel Widdis --- DEVELOPER_GUIDE.md | 22 +++++++++++++++++----- README.md | 8 ++++---- 2 files changed, 21 insertions(+), 9 deletions(-) diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 1a01a02da..e91940112 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -1,7 +1,7 @@ - [Developer Guide](#developer-guide) - [Forking and Cloning](#forking-and-cloning) - [Install Prerequisites](#install-prerequisites) - - [JDK 14](#jdk-14) + - [Java](#java) - [Setup](#setup) - [Build](#build) - [Building from the command line](#building-from-the-command-line) @@ -10,6 +10,7 @@ - [Publishing](#publishing) - [Publishing to Maven Local](#publishing-to-maven-local) - [Generating artifacts](#generating-artifacts) + - [Adding Workflow Steps](#adding-workflow-steps) ## Developer Guide @@ -19,9 +20,11 @@ Fork this repository on GitHub, and clone locally with `git clone`. ### Install Prerequisites -#### JDK 14 +See [OpenSearch requirements](https://github.com/opensearch-project/OpenSearch/blob/main/DEVELOPER_GUIDE.md#install-prerequisites). -OpenSearch components build using Java 14 at a minimum. This means you must have a JDK 14 installed with the environment variable `JAVA_HOME` referencing the path to Java home for your JDK 14 installation, e.g. `JAVA_HOME=/usr/lib/jvm/jdk-14`. +#### Java + +Flow Framework code currently maintains compatibility with JDK 11. Other plugins may require newer Java versions if used. ### Setup @@ -60,6 +63,7 @@ merged to main, the workflow will create a backport PR to the `2.x` branch. ### Publishing #### Publishing to Maven Local + Run the below command to publish the artifacts to maven local. ```./gradlew publishToMavenLocal``` @@ -93,8 +97,16 @@ snapshots/ ├── maven-metadata.xml.sha1 ├── maven-metadata.xml.sha256 └── maven-metadata.xml.sha512 - - ``` + 1. Change the url from ``"https://aws.oss.sonatype.org/content/repositories/snapshots"`` to your local path and comment out the credentials under publishing/repositories in build.gradle. 2. Run ```./gradlew publishPluginZipPublicationToSnapshotsRepository```. + +### Adding Workflow Steps + +To add functionality to workflows, add new Workflow Steps to the [`org.opensearch.flowframework.workflow`](https://github.com/opensearch-project/flow-framework/tree/main/src/main/java/org/opensearch/flowframework/workflow) package. +1. Implement the [Workflow](https://github.com/opensearch-project/flow-framework/blob/main/src/main/java/org/opensearch/flowframework/workflow/WorkflowStep.java) interface. See existing steps for examples for input, output, and API execution. +2. Choose a unique name for the step which is not used by other steps. +3. Add a constructor and call it from the [WorkflowStepFactory](https://github.com/opensearch-project/flow-framework/blob/main/src/main/java/org/opensearch/flowframework/workflow/WorkflowStepFactory.java). +4. Add a configuration to the [`workflow-steps.json`](https://github.com/dbwiddis/flow-framework/blob/main/src/main/resources/mappings/workflow-steps.json) file specifying required inputs, outputs, required plugins, and optionally a different timeout than the default. +5. Write tests. diff --git a/README.md b/README.md index 51bd59d8e..8f99b655f 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,12 @@ -## OpenSearch AI Flow Framework +## OpenSearch Flow Framework -This project is an OpenSearch plugin that enables builders to innovate AI apps on OpenSearch. +This project is an OpenSearch plugin that enables builders to innovate AI applications on OpenSearch. The current process of using ML offerings in OpenSearch, such as Semantic Search, requires users to handle complex setup and pre-processing tasks, and send verbose user queries, both of which can be time-consuming and error-prone. -The directional idea is to provide OpenSearch users with use case templates, which provide a compact description (e.g., JSON document). These templates would describe configurations for automated workflows such as Retrieval Augment Generation (RAG), AI connectors and other components that prime OpenSearch as a backend to leverage generative models—once primed, builders can query OpenSearch directly without building middleware logic to stitch together data flows and ML models. +We want to introduce our customers to a new no-code/low-code builder experience ([Backend RFC](https://github.com/opensearch-project/OpenSearch/issues/9213) and [Frontend RFC](https://github.com/opensearch-project/OpenSearch-Dashboards/issues/4755)) that empowers users to compose AI-augmented query and ingestion flows, integrate ML models supported by ML-Commons, and streamline the OpenSearch app development experience through a drag-and-drop designer. The front end will help users create use case templates, which provide a compact description of configuration steps for automated workflows such as Retrieval Augment Generation (RAG), AI connectors and other components that prime OpenSearch as a backend to leverage generative models. Once primed, builders can query OpenSearch directly without building middleware logic to stitch together data flows and ML models. -See the [RFC](https://github.com/opensearch-project/OpenSearch/issues/9213) on the OpenSearch project for initial design discussions. +See the [Development Plan](https://github.com/opensearch-project/flow-framework/issues/475) to view or comment on current incremental development priorities. ## Security From ffe41e1e8bda8d5057e1e53b59c22cf5d57b9cdb Mon Sep 17 00:00:00 2001 From: Daniel Widdis Date: Mon, 12 Feb 2024 12:00:42 -0800 Subject: [PATCH 2/3] Add instructions for updating WorkflowResources enum Signed-off-by: Daniel Widdis --- DEVELOPER_GUIDE.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index e91940112..ad822dee2 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -106,7 +106,8 @@ snapshots/ To add functionality to workflows, add new Workflow Steps to the [`org.opensearch.flowframework.workflow`](https://github.com/opensearch-project/flow-framework/tree/main/src/main/java/org/opensearch/flowframework/workflow) package. 1. Implement the [Workflow](https://github.com/opensearch-project/flow-framework/blob/main/src/main/java/org/opensearch/flowframework/workflow/WorkflowStep.java) interface. See existing steps for examples for input, output, and API execution. -2. Choose a unique name for the step which is not used by other steps. +2. Choose a unique name for the step which is not used by other steps. This will align with the `step_type` field in the templates and should be descriptive of what the step does. 3. Add a constructor and call it from the [WorkflowStepFactory](https://github.com/opensearch-project/flow-framework/blob/main/src/main/java/org/opensearch/flowframework/workflow/WorkflowStepFactory.java). 4. Add a configuration to the [`workflow-steps.json`](https://github.com/dbwiddis/flow-framework/blob/main/src/main/resources/mappings/workflow-steps.json) file specifying required inputs, outputs, required plugins, and optionally a different timeout than the default. -5. Write tests. +5. If your step provisions a resource that should be deprovisioned, create the corresponding step and add both steps to the [`WorkflowResources`](https://github.com/opensearch-project/flow-framework/blob/main/src/main/java/org/opensearch/flowframework/common/WorkflowResources.java) enum. +6. Write unit and integration tests. From 71ee8d76fc6dc993afb5eb057ee67852b67a4f2d Mon Sep 17 00:00:00 2001 From: Daniel Widdis Date: Mon, 12 Feb 2024 14:10:24 -0800 Subject: [PATCH 3/3] Mention non-ML use cases. Signed-off-by: Daniel Widdis --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 8f99b655f..86510708b 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,8 @@ The current process of using ML offerings in OpenSearch, such as Semantic Search We want to introduce our customers to a new no-code/low-code builder experience ([Backend RFC](https://github.com/opensearch-project/OpenSearch/issues/9213) and [Frontend RFC](https://github.com/opensearch-project/OpenSearch-Dashboards/issues/4755)) that empowers users to compose AI-augmented query and ingestion flows, integrate ML models supported by ML-Commons, and streamline the OpenSearch app development experience through a drag-and-drop designer. The front end will help users create use case templates, which provide a compact description of configuration steps for automated workflows such as Retrieval Augment Generation (RAG), AI connectors and other components that prime OpenSearch as a backend to leverage generative models. Once primed, builders can query OpenSearch directly without building middleware logic to stitch together data flows and ML models. +While the initial development has focused on ML use cases, the framework is generic and can be adapted to other use cases. + See the [Development Plan](https://github.com/opensearch-project/flow-framework/issues/475) to view or comment on current incremental development priorities. ## Security