Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update README and Developer Guide #520

Merged
merged 3 commits into from
Feb 12, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 18 additions & 5 deletions DEVELOPER_GUIDE.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
- [Developer Guide](#developer-guide)
- [Forking and Cloning](#forking-and-cloning)
- [Install Prerequisites](#install-prerequisites)
- [JDK 14](#jdk-14)
- [Java](#java)
- [Setup](#setup)
- [Build](#build)
- [Building from the command line](#building-from-the-command-line)
Expand All @@ -10,6 +10,7 @@
- [Publishing](#publishing)
- [Publishing to Maven Local](#publishing-to-maven-local)
- [Generating artifacts](#generating-artifacts)
- [Adding Workflow Steps](#adding-workflow-steps)

## Developer Guide

Expand All @@ -19,9 +20,11 @@ Fork this repository on GitHub, and clone locally with `git clone`.

### Install Prerequisites

#### JDK 14
See [OpenSearch requirements](https://github.com/opensearch-project/OpenSearch/blob/main/DEVELOPER_GUIDE.md#install-prerequisites).

OpenSearch components build using Java 14 at a minimum. This means you must have a JDK 14 installed with the environment variable `JAVA_HOME` referencing the path to Java home for your JDK 14 installation, e.g. `JAVA_HOME=/usr/lib/jvm/jdk-14`.
#### Java

Flow Framework code currently maintains compatibility with JDK 11. Other plugins may require newer Java versions if used.

### Setup

Expand Down Expand Up @@ -60,6 +63,7 @@ merged to main, the workflow will create a backport PR to the `2.x` branch.
### Publishing

#### Publishing to Maven Local

Run the below command to publish the artifacts to maven local.
```./gradlew publishToMavenLocal```

Expand Down Expand Up @@ -93,8 +97,17 @@ snapshots/
├── maven-metadata.xml.sha1
├── maven-metadata.xml.sha256
└── maven-metadata.xml.sha512


```

1. Change the url from ``"https://aws.oss.sonatype.org/content/repositories/snapshots"`` to your local path and comment out the credentials under publishing/repositories in build.gradle.
2. Run ```./gradlew publishPluginZipPublicationToSnapshotsRepository```.

### Adding Workflow Steps

To add functionality to workflows, add new Workflow Steps to the [`org.opensearch.flowframework.workflow`](https://github.com/opensearch-project/flow-framework/tree/main/src/main/java/org/opensearch/flowframework/workflow) package.
1. Implement the [Workflow](https://github.com/opensearch-project/flow-framework/blob/main/src/main/java/org/opensearch/flowframework/workflow/WorkflowStep.java) interface. See existing steps for examples for input, output, and API execution.
amitgalitz marked this conversation as resolved.
Show resolved Hide resolved
2. Choose a unique name for the step which is not used by other steps. This will align with the `step_type` field in the templates and should be descriptive of what the step does.
3. Add a constructor and call it from the [WorkflowStepFactory](https://github.com/opensearch-project/flow-framework/blob/main/src/main/java/org/opensearch/flowframework/workflow/WorkflowStepFactory.java).
dbwiddis marked this conversation as resolved.
Show resolved Hide resolved
4. Add a configuration to the [`workflow-steps.json`](https://github.com/dbwiddis/flow-framework/blob/main/src/main/resources/mappings/workflow-steps.json) file specifying required inputs, outputs, required plugins, and optionally a different timeout than the default.
5. If your step provisions a resource that should be deprovisioned, create the corresponding step and add both steps to the [`WorkflowResources`](https://github.com/opensearch-project/flow-framework/blob/main/src/main/java/org/opensearch/flowframework/common/WorkflowResources.java) enum.
6. Write unit and integration tests.
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
## OpenSearch AI Flow Framework
## OpenSearch Flow Framework

This project is an OpenSearch plugin that enables builders to innovate AI apps on OpenSearch.
This project is an OpenSearch plugin that enables builders to innovate AI applications on OpenSearch.

The current process of using ML offerings in OpenSearch, such as Semantic Search, requires users to handle complex setup and pre-processing tasks, and send verbose user queries, both of which can be time-consuming and error-prone.

The directional idea is to provide OpenSearch users with use case templates, which provide a compact description (e.g., JSON document). These templates would describe configurations for automated workflows such as Retrieval Augment Generation (RAG), AI connectors and other components that prime OpenSearch as a backend to leverage generative models—once primed, builders can query OpenSearch directly without building middleware logic to stitch together data flows and ML models.
We want to introduce our customers to a new no-code/low-code builder experience ([Backend RFC](https://github.com/opensearch-project/OpenSearch/issues/9213) and [Frontend RFC](https://github.com/opensearch-project/OpenSearch-Dashboards/issues/4755)) that empowers users to compose AI-augmented query and ingestion flows, integrate ML models supported by ML-Commons, and streamline the OpenSearch app development experience through a drag-and-drop designer. The front end will help users create use case templates, which provide a compact description of configuration steps for automated workflows such as Retrieval Augment Generation (RAG), AI connectors and other components that prime OpenSearch as a backend to leverage generative models. Once primed, builders can query OpenSearch directly without building middleware logic to stitch together data flows and ML models.

dbwiddis marked this conversation as resolved.
Show resolved Hide resolved
See the [RFC](https://github.com/opensearch-project/OpenSearch/issues/9213) on the OpenSearch project for initial design discussions.
See the [Development Plan](https://github.com/opensearch-project/flow-framework/issues/475) to view or comment on current incremental development priorities.

## Security

Expand Down
Loading