diff --git a/docs/developer-guides/caching-task-outputs.md b/docs/developer-guides/caching-task-outputs.md index 1d28301f..72020e81 100644 --- a/docs/developer-guides/caching-task-outputs.md +++ b/docs/developer-guides/caching-task-outputs.md @@ -1,6 +1,7 @@ --- slug: "../faqs/task-cache-output" --- + # Caching Task Outputs Some task types support caching, which saves task outputs for reuse in subsequent tasks. This feature can be configured in the task configuration. diff --git a/docs/getting-started/introduction.md b/docs/getting-started/introduction.md new file mode 100644 index 00000000..ae9f15f6 --- /dev/null +++ b/docs/getting-started/introduction.md @@ -0,0 +1,35 @@ +--- +slug: "../category/getting-started" +description: "" +--- + +import {useCurrentSidebarCategory} from '@docusaurus/theme-common'; +import DocCardList from '@theme/DocCardList'; + +# Introduction + +Orkes Conductor is an orchestration platform for building distributed applications in any language. Modern applications often consist of multiple services that must work together to complete the application. An orchestration layer is essential to manage this efficiently—similar to a frontend or backend layer in your tech stack. Orkes Conductor lets you add an orchestration layer to your existing applications or build new ones from scratch. + +Orchestration is the practice of managing the flow and coordination of tasks across multiple services, much like how Kubernetes orchestrates containers or step functions orchestrate cloud operations. Workflows are components used to orchestrate distributed services. As modern applications increasingly rely on distributed services, orchestration becomes essential for ensuring your app’s performance and resilience. + +Whether managing microservices, handling data pipelines, or building AI applications like Agentic RAG (Retrieval Augmented Generation), Conductor helps coordinate services effectively, ensuring reliable performance. +Conductor lets you build complex applications by breaking them down into simple tasks. It automatically handles state management, task sequencing, retries, and error handling, allowing you to focus on developing your core logic without worrying about orchestration. + +Developers can use Conductor to ensure high reliability, transactional consistency, and scalability, no matter where their components are deployed or what languages they use. Simply define the workflow, and the Conductor takes care of the execution. + +## Key features + +- **Scalability**—Run workflows of any size, from small projects to large enterprise systems. +- **Workflow as code**—Write workflows in your preferred language, including Java, Python, Go, C#, TypeScript, Clojure, and more. +- **Task workers**—Build external task workers as microservices, serverless functions, or any deployment method you choose. +- **Fault tolerance**—Automatically recover from failures using built-in retries and compensation mechanisms to ensure high availability and uninterrupted operations. +- **Durable execution**—Create long-running workflows without worrying about system failures, dependencies, or scaling issues. +- **Reusable components**—Package and reuse common functions to reduce duplication across workflows. +- **Access controls**—Maintain control and visibility with built-in access management features. +- **Troubleshooting**—Built-in metrics and logs help identify and resolve issues across distributed executions. +- **Operational insights**—Access detailed metrics to gain insights into performance, bottlenecks, and SLA compliance. +- **Upgrade existing systems**—Integrate microservices, AI agents, and APIs to extend existing systems and improve orchestration. +- **Speed up development cycle**—Speed up development cycles, launch new applications faster, and maintain high standards of security and observability. +- **Ease of integration**—Integrate seamlessly with various systems and services. + + diff --git a/docs/getting-started/quickstart-1.md b/docs/getting-started/quickstart-1.md new file mode 100644 index 00000000..9883e594 --- /dev/null +++ b/docs/getting-started/quickstart-1.md @@ -0,0 +1,487 @@ +--- +slug: "../quickstarts/create-first-workflow" +description: "" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Quickstart 1: Create Your First Workflow + +As you will have learned in [Core Concepts](../core-concepts), Conductor’s core orchestration unit is a workflow. In this quickstart, you will learn how to create your first Conductor workflow. + +**Approaches to create workflows** + +In Conductor, workflow definitions are stored as JSON. To create a workflow, you can use one of the following ways: +* **Workflow as code**—Using the Conductor SDKs, define your workflow in your preferred language. +* **Visual workflow editor**—Using Orkes Platform, define your workflow visually, which is formatted as JSON under the hood. + +**Static vs dynamic workflows** + +With Conductor, you can define workflows statically (ahead of time) or dynamically (at runtime). This quickstart will teach you how to creating static workflows using your preferred approach. Once you master this, you can dive into creating [dynamic workflows as code](../developer-guides/write-workflows-using-code). + +**Tasks in workflows** + +A workflow definition consists of a collection of tasks and operators and specifies the order and execution of the defined tasks. Conductor provides a set of system tasks and operators, but you can also write your own custom worker tasks. This is a powerful feature which you will learn more in [Quickstart 2](write-workers). + +For this quickstart, let’s begin by using system tasks to create your first workflow. + + +## Quickstart overview +1. Create a workflow definition consisting of system tasks and operators. +2. Register the workflow definition to the Conductor server. +3. Run the workflow. + + +## Before you begin + +Ensure that you have [prepared your tools and access](../quickstarts#preparing-your-tools). + + +## A. Workflow as code + +Create a project for your workflow client. + + + + +**Step 1: Configure access and create workflow** + +To define a workflow, you must provide a MetadataClient and a WorkflowExecutor, which requires a Configuration object with the Conductor Server info. Here's an example of how to do that: + +``` python +from conductor.client.configuration.configuration import Configuration +from conductor.client.configuration.settings.authentication_settings import AuthenticationSettings +from conductor.client.orkes.orkes_metadata_client import OrkesMetadataClient +from conductor.client.workflow.conductor_workflow import ConductorWorkflow +from conductor.client.workflow.executor.workflow_executor import WorkflowExecutor + +configuration = Configuration( + server_api_url=SERVER_API_URL, // eg: https://play.orkes.io/api + debug=False, + authentication_settings=AuthenticationSettings(key_id=KEY_ID, key_secret=KEY_SECRET) +) + +metadata_client = OrkesMetadataClient(configuration) + +workflow_executor = WorkflowExecutor(configuration) +workflow = ConductorWorkflow( + executor=workflow_executor, + name='python_workflow_example_from_code', + description='Python workflow example from code' +) +``` + +**Step 2: Add tasks to workflow** + +After creating an instance of a ConductorWorkflow, you can add tasks to it. There are two possible ways to do that: +* method: add +* operator: >> + +``` python +from conductor.client.workflow.task.simple_task import SimpleTask + +simple_task_1 = SimpleTask( + task_def_name='python_simple_task_from_code_1', + task_reference_name='python_simple_task_from_code_1' +) +workflow.add(simple_task_1) + +simple_task_2 = SimpleTask( + task_def_name='python_simple_task_from_code_2', + task_reference_name='python_simple_task_from_code_2' +) +workflow >> simple_task_2 +``` + +You can add input parameters to your workflow: + +``` python +workflow.input_parameters(["a", "b"]) +``` + +**Step 3: Register workflow** + +Register your workflow at the Conductor Server: + +``` python +from conductor.client.http.models.workflow_def import WorkflowDef + +workflowDef = workflow.to_workflow_def() +metadata_client.register_workflow_def(workflowDef, True) +``` + + +**Step 4: Run workflow** + +Your first workflow is now created. Give your workflow a test run: + + +``` python +from conductor.client.http.models import StartWorkflowRequest + +request = StartWorkflowRequest() +request.name = 'python_workflow_example_from_code' +request.version = 1 +request.input = {'name': 'Orkes'} + +workflow_run = workflow_client.execute_workflow( + start_workflow_request=request, + wait_for_seconds=12) + +``` + + + + + + +**Step 1: Create workflow** + +Create a ConductorWorkflow Instance. +``` java +ConductorWorkflow conductorWorkflow = new WorkflowBuilder(executor) + .name("sdk_workflow_example") + .version(1) + .ownerEmail("hello@example.com") + .description("Example Workflow") + .timeoutPolicy(WorkflowDef.TimeoutPolicy.TIME_OUT_WF, 100) + .add(new SimpleTask("calculate_insurance_premium", "calculate_insurance_premium")) + .add(new SimpleTask("send_email", "send_email")) + .build(); +``` + + +**Step 2: Add tasks to workflow** + +After creating an instance of a ConductorWorkflow, you can add tasks to it using the `add` method. The task inputs configured using the `input` method. +``` java +builder.add( + new SimpleTask("send_email", "send_email") + .input("email", "${workflow.input.email}") + .input("subject", "Your insurance quote for the amount ${generate_quote.output.amount}") +); +``` + +**Step 3: Register workflow** + +``` java +//Returns true if the workflow is successfully created +boolean registered = workflow.registerWorkflow(); +``` + +**Step 4: Run workflow** + +Start the execution of the workflow based on the definition registered on the server. Use the register method to register a workflow on the server before executing. +``` java +//Returns a completable future +CompletableFuture execution = conductorWorkflow.execute(input); + +//Wait for the workflow to complete -- useful if workflow completes within a reasonable amount of time +Workflow workflowRun = execution.get(); + +//Get the workflowId +String workflowId = workflowRun.getWorkflowId(); + +//Get the status of workflow execution +WorkflowStatus status = workflowRun.getStatus(); + +``` + + + + + +**Step 1: Configure access and create workflow** +``` javascript +import { + OrkesApiConfig, + orkesConductorClient, + TaskRunner, + simpleTask, +} from "@io-orkes/conductor-javascript"; + +//API client instance with server address and authentication details +const clientPromise = orkesConductorClient({ + keyId: "XXX", + keySecret: "XXXX", + serverUrl: "SERVER_URL", // eg: https://play.orkes.io/api +}); + +const client = await clientPromise; + +//Create new workflow executor +const executor = new WorkflowExecutor(client); + +// Using Factory function to create a workflow +const factoryWf = { + name: "my_first_workflow", + version: 1, + ownerEmail: "user@example.com", + tasks: [simpleTask("simple_task_ref", "simple_task", {})], + inputParameters: [], + outputParameters: {}, + timeoutSeconds: 0, +}; +``` + +**Step 2: Register workflow** + +``` javascript +const workflow = executor.registerWorkflow(true, factoryWf); +``` + + +**Step 3: Run workflow** + +Use Workflow Executor to start the previously-registered workflow. + +``` javascript +const executor = new WorkflowExecutor(client); +const executionId = await executor.startWorkflow({ name, version, input: {} }); +``` + + + + + +**Step 1: Configure access and create workflow** +```csharp +using Conductor.Client; +using Conductor.Definition; +using Conductor.Executor; + +ConductorWorkflow GetConductorWorkflow() +{ + return new ConductorWorkflow() + .WithName("my_first_workflow") + .WithVersion(1) + .WithOwner("developers@orkes.io") + .WithTask(new SimpleTask("simple_task_2", "simple_task_1")) + .WithTask(new SimpleTask("simple_task_1", "simple_task_2")); +} + +var configuration = new Configuration(); + +var conductorWorkflow = GetConductorWorkflow(); +var workflowExecutor = new WorkflowExecutor(configuration); +``` + +**Step 2: Register workflow** +``` csharp +workflowExecutor.RegisterWorkflow( + workflow: conductorWorkflow + overwrite: true +); +``` + +**Step 3: Run workflow** +``` csharp +var workflowId = workflowExecutor.StartWorkflow(conductorWorkflow); +``` + + + + + +**Step 1: Configure access and create workflow** +``` go +// API client instance with server address and authentication details +apiClient := client.NewAPIClient( + settings.NewAuthenticationSettings( + KEY, + SECRET, + ), + settings.NewHttpSettings( + "https://play.orkes.io/api", + )) + +// Create new workflow executor +executor := executor.NewWorkflowExecutor(apiClient) + +// Create a new ConductorWorkflow instance +conductorWorkflow := workflow.NewConductorWorkflow(executor). + Name("my_first_workflow"). + Version(1). + OwnerEmail("developers@orkes.io") +``` + +**Step 2: Add tasks to workflow** +``` go +conductorWorkflow. + Add(workflow.NewSimpleTask("simple_task_2", "simple_task_1")). + Add(workflow.NewSimpleTask("simple_task_1", "simple_task_2")) +``` + +**Step 3: Register workflow** +``` go +conductorWorkflow.Register(true) //Overwrite the existing definition with the new one +``` + + +**Step 4: Execute workflow** + +Use Workflow Executor to start the previously-registered workflow. + +``` go +//Input can be either a map or a struct that is serializable to a JSON map +workflowInput := map[string]interface{}{} + +workflowId, err := executor.StartWorkflow(&model.StartWorkflowRequest{ + Name: conductorWorkflow.GetName(), + Input: workflowInput, +}) +``` + + + + + +**Step 1: Add tasks** + +``` clojure +(defn create-tasks + "Returns workflow tasks" + [] + (vector (sdk/simple-task (:get-user-info constants) (:get-user-info constants) {:userId "${workflow.input.userId}"}) + (sdk/switch-task "emailorsms" "${workflow.input.notificationPref}" {"email" [(sdk/simple-task (:send-email constants) (:send-email constants) {"email" "${get_user_info.output.email}"})] + "sms" [(sdk/simple-task (:send-sms constants) (:send-sms constants) {"phoneNumber" "${get_user_info.output.phoneNumber}"})]} []))) +``` + + +**Step 2: Create workflow** + +```clojure +(defn create-workflow + "Returns a workflow with tasks" + [tasks] + (merge (sdk/workflow (:workflow-name constants) tasks) {:inputParameters ["userId" "notificationPref"]})) + +;; creates a workflow with tasks +(-> (create-tasks) (create-workflow)) +``` + + +**Step 3: Register workflow** + +```clojure +(defn + register-workflow-def-using-client + "Takes a client and a workflow definition in edn, will register a worflow in conductor" + ([client workflow overwrite] + (client "workflow" :method :post :body workflow :query-params {"overwrite" overwrite})) + ([client workflow] (register-workflow-def-using-client client workflow false))) +``` + +**Step 4: Run workflow** + +```clojure +(def workflow-request {:name "SomeWFName" + :version 1 + :input {"userId" "jim" + "notificationPref" "sms"}}) + +(wr/start-workflow options workflow-request) +``` + + + + + +## B. Visual workflow editor + +Use the visual workflow editor in Orkes Platform to create your workflows. + +**To create a workflow:** +1. Log in to your Orkes cluster or the [Orkes Playground](https://play.orkes.io/). +2. In the left navigation menu, go to **Definitions** > **Workflow**. +3. Select **(+) Define workflow**. +4. Enter a Name and Description for your workflow. +5. To add tasks to the workflow, select the **(+)** icon in the visual workflow diagram. +6. To register the workflow, select **Save** > **Confirm**. + +

Screenshot of visual workflow editor in Orkes Platform.

+ + +Once created, you can run your workflow by going to the **Run** tab and selecting **Run workflow**. + + +## Tutorial — your first workflow: + +Follow along to build your first workflow, which is a conditional notification flow based on the user’s location. + +1. The first task will retrieve the user information through an HTTP endpoint. + 1. Add an HTTP task to the workflow. + 2. Enter the task name `get-user` in **Task definition**. + 3. Set the **URL** as [https://randomuser.me/api/](https://randomuser.me/api/) with GET as the **Method**. +2. The next task will evaluate the user information based on a set criteria. In this case, we want the workflow to send a notification only if the user is located in United States. + 1. Add a Switch task to the workflow. + 2. Enter the task name `user-criteria` in **Task definition**. + 3. Set the evaluation criteria to `Value-Param` and enter the evaluation cases in **Switch cases**. In this case, select **Add more switch cases** and enter United States. This creates a new branch, where its tasks are only executed if the evaluation criteria is met. \ + The parameter to be evaluated will be `switchCaseValue`, which will be wired to a variable input based on the previous `get-user` task output. + 4. To do so, in Script params, enter `${get-user_ref.output.response.body.results[0].location.country}` as the value for switchCaseValue. This is a dynamic variable, which is expressed in JSONPath syntax. +3. The final task will send a notification to the user if they are located in United States. + 1. In the United States branch of the Switch task, add a HTTP task. + 2. Enter the task name `send-notification` in **Task definition**. + 3. Set the **URL** as [https://orkes-api-tester.orkesconductor.com/api](https://orkes-api-tester.orkesconductor.com/api), which will serve as the mock notification endpoint, with POST as the **Method**. + + Alternatively, you can copy the JSON code below into the **Code** tab of the workflow builder. + +``` json +// workflow definition in JSON + +{ + "name": "myFirstWorkflow", + "description": "Workflow using a Switch task and HTTP tasks.", + "version": 1, + "tasks": [ + { + "name": "get-user", + "taskReferenceName": "get-user_ref", + "inputParameters": { + "uri": "https://randomuser.me/api/", + "method": "GET", + "accept": "application/json", + "contentType": "application/json", + "encode": true + }, + "type": "HTTP" + }, + { + "name": "user-criteria", + "taskReferenceName": "user-criteria_ref", + "inputParameters": { + "switchCaseValue": "${get-user_ref.output.response.body.results[0].location.country}" + }, + "type": "SWITCH", + "decisionCases": { + "United States": [ + { + "name": "send-notification", + "taskReferenceName": "send-notification_ref", + "inputParameters": { + "uri": "https://orkes-api-tester.orkesconductor.com/api", + "method": "POST", + "accept": "application/json", + "contentType": "application/json", + "encode": true + }, + "type": "HTTP" + } + ] + }, + "defaultCase": [] + } + ], + "inputParameters": [], + "outputParameters": {}, + "failureWorkflow": "", + "schemaVersion": 2 +} +``` + +4. Save and register your workflow. + +Your first workflow is now created. Give it a test run. + +Now that you have gotten a hang of creating workflows, you can make them more powerful by using Worker tasks, which execute custom logic just like any regular function. Head to the next quickstart to learn more. \ No newline at end of file diff --git a/docs/getting-started/quickstart-2.md b/docs/getting-started/quickstart-2.md new file mode 100644 index 00000000..2aa9d4ab --- /dev/null +++ b/docs/getting-started/quickstart-2.md @@ -0,0 +1,520 @@ +--- +slug: "../quickstarts/write-workers" +description: "" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Quickstart 2: Write Workers + +In Conductor, tasks are executed using a worker-queue architecture. System tasks are serviced by Conductor workers, while custom tasks are serviced by the workers that you create. In this quickstart, you will learn how to write your own workers that will execute custom tasks. + +**Decoupled by design** + +The worker code contains your task logic, which is decoupled from both the task definition (number of retries, rate limits) and the workflow-specific task configuration (inputs from other tasks, optionality). + +**Worker deployment** + +Conductor workers can run in a cloud-native environment or on-premise. Like any other application, workers can be easily deployed in a container, VM, or bare metal. + +For the purpose of this quickstart, we will deploy the worker from your own machine. + + +## Quickstart overview +1. Create task worker(s) that poll for scheduled tasks at regular interval +2. Create and register task definitions for these workers. +3. Add the custom task to the workflow definition. +4. Grant execution permission to the worker. + + +## Before you begin + +Ensure that you have [prepared your tools and access](../quickstarts#preparing-your-tools). + + +## Create a worker application + +Create a new project for your worker application, keeping it separate from your workflow client. + + + + +You can create a worker by writing a Python function and annotating it with a @worker_task decorator. + +``` python +from conductor.client.worker.worker_task import worker_task + +@worker_task(task_definition_name='greetings') +def greetings(name: str) -> str: + return f'Hello, {name}' +``` + + +A worker can take inputs which are primitives (str, int, float, bool, and so on) or complex data classes. Here is an example worker that uses `dataclass` as part of the worker input. + + +``` python +from conductor.client.worker.worker_task import worker_task +from dataclasses import dataclass + +@dataclass +class OrderInfo: + order_id: int + sku: str + quantity: int + sku_price: float + + +@worker_task(task_definition_name='process_order') +def process_order(order_info: OrderInfo) -> str: + return f'order: {order_info.order_id}' +``` + +Workers use a polling mechanism (with a long poll) to check for any available tasks from the server periodically. The startup and shutdown of workers are handled by the conductor.client.automator.task_handler.TaskHandler class. + +``` python +from conductor.client.automator.task_handler import TaskHandler +from conductor.client.configuration.configuration import Configuration + +def main(): + api_config = Configuration() + + task_handler = TaskHandler( + workers=[], + configuration=api_config, + scan_for_annotated_workers=True, + import_modules=['greetings'] # import workers from this module - leave empty if all the workers are in the same module + ) + + # start worker polling + task_handler.start_processes() + + # Call to stop the workers when the application is ready to shutdown + task_handler.stop_processes() + + +if __name__ == '__main__': + main() +``` + + + + + +Create a worker class that implements the `Worker` interface and its methods `getTaskDefName()` and `execute(Task task)`. + +``` java +import com.netflix.conductor.client.worker.Worker; +import com.netflix.conductor.common.metadata.tasks.Task; +import com.netflix.conductor.common.metadata.tasks.TaskResult; + +public class SimpleWorker implements Worker { + + @Override + public String getTaskDefName() { + return "simple-java-worker"; + } + + @Override + public TaskResult execute(Task task) { + TaskResult taskResult = new TaskResult(task); + taskResult.setStatus(TaskResult.Status.COMPLETED); + taskResult.getOutputData().put("message", "Hello World!"); + return taskResult; + } +} +``` + +Workers use a polling mechanism as defined by the TaskClient. Here is an example of a Java worker application: + +``` java +package com.netflix.conductor.gettingstarted; + +import java.util.List; + +import com.netflix.conductor.client.automator.TaskRunnerConfigurer; +import com.netflix.conductor.client.http.ConductorClient; +import com.netflix.conductor.client.http.TaskClient; +import com.netflix.conductor.client.worker.Worker; +import com.netflix.conductor.common.metadata.tasks.Task; +import com.netflix.conductor.common.metadata.tasks.TaskResult; + +public class HelloWorker implements Worker { + + @Override + public TaskResult execute(Task task) { + var taskResult = new TaskResult(task); + taskResult.setStatus(TaskResult.Status.COMPLETED); + taskResult.getOutputData().put("message", "Hello World!"); + return taskResult; + } + + @Override + public String getTaskDefName() { + return "hello_task"; + } + + public static void main(String[] args) { + var client = new ConductorClient("http://localhost:8080/api"); + var taskClient = new TaskClient(client); + var runnerConfigurer = new TaskRunnerConfigurer + .Builder(taskClient, List.of(new HelloWorker())) + .withThreadCount(10) + .build(); + runnerConfigurer.init(); + } +} +``` + + + + + + +Create a worker function using the following template: + +``` javascript +import { ConductorWorker, Task } from "@io-orkes/conductor-javascript"; + +const worker: ConductorWorker = { + taskDefName: "task-def-name", + execute: async ( + task: Task + ): Promise> => {}, +}; +``` + + +Use the `TaskRunner` interface to start the workers, which takes care of polling server for the work, executing worker code, and updating the results back to the server. + +``` javascript +import { + OrkesApiConfig, + orkesConductorClient, + TaskRunner, +} from "@io-orkes/conductor-javascript"; + +const clientPromise = orkesConductorClient({ + keyId: "XXX", + keySecret: "XXXX", + serverUrl: "SERVER_URL", // eg: https://play.orkes.io/api +}); + +const client = await clientPromise; + +const taskDefName = "HelloWorldWorker"; + +const customWorker: ConductorWorker = { +taskDefName, + execute: async ({ inputData, taskId }) => { + return { + outputData: { + greeting: "Hello World", + }, + status: "COMPLETED", + }; + }, +}; +// Worker Options will take precedence over options defined in the manager + +const manager = new TaskManager(client, [customWorker], { + options: { pollInterval: 100, concurrency: 1 }, +}); + +manager.startPolling(); +// You can update all worker settings at once using +manager.updatePollingOptions({ pollInterval: 100, concurrency: 1 }); + +// You can update a single worker setting using : +manager.updatePollingOptionForWorker(taskDefName, { + pollInterval: 100, + concurrency: 1, +}); + +manager.isPolling // Will resolve to true + +await manager.stopPolling(); + +manager.isPolling // Will resolve to false +``` + + + + + +Create a worker class that implements the `IWorkflowTask` interface. + +``` csharp +public class SimpleWorker : IWorkflowTask +{ + public string TaskType { get; } + public WorkflowTaskExecutorConfiguration WorkerSettings { get; } + + public SimpleWorker(string taskType = "test-sdk-csharp-task") + { + TaskType = taskType; + WorkerSettings = new WorkflowTaskExecutorConfiguration(); + } + + public TaskResult Execute(Task task) + { + return task.Completed(); + } +} +``` + +Use `WorkflowTaskHost` to create a worker host, which requires a configuration object and workers. + +``` csharp +using Conductor.Client.Worker; +using System; +using System.Threading.Thread; + +var host = WorkflowTaskHost.CreateWorkerHost(configuration, new SimpleWorker()); +await host.startAsync(); +Thread.Sleep(TimeSpan.FromSeconds(100)); +``` + + + + + +Create a worker function using the following template: + +``` go +type ExecuteTaskFunction func(t *Task) (interface{}, error) +``` + +Here is an example worker in Go: + +``` go +func Greet(task *model.Task) (interface{}, error) { + return map[string]interface{}{ + "hello": "Hello, " + fmt.Sprintf("%v", task.InputData["person_to_be_greated"]), + }, nil +} +``` + +Use the `TaskRunner` interface to start the workers, which takes care of polling server for the work, executing worker code, and updating the results back to the server. + +``` go +apiClient := client.NewAPIClient( + settings.NewAuthenticationSettings( + KEY, + SECRET, + ), + settings.NewHttpSettings( + "https://play.orkes.io/api", +)) + +taskRunner := worker.NewTaskRunnerWithApiClient(apiClient) +//Start polling for a task by name "simple_task", with a batch size of 1 and 1 second interval +//Between polls if there are no tasks available to execute +taskRunner.StartWorker("simple_task", examples.SimpleWorker, 1, time.Second*1) +//Add more StartWorker calls as needed + +//Block +taskRunner.WaitWorkers() +``` + + + + + +Create a worker function using the following template: +``` clojure +(def worker + {:name "cool_clj_task_b", + :execute (fn [d] + [:completed (:inputData d)])}) +``` + +Use the `TaskRunner` interface to start the workers, which takes care of polling server for the work, executing worker code, and updating the results back to the server. + +``` clojure +(:require + [io.orkes.taskrunner :refer :all]) + +;; Will poll for tasks +(def shutdown-task-runner (runner-executer-for-workers options [worker])) + +;; Stops polling for tasks +(shutdown-task-runner ) +``` + + + + + +## Add worker task to a workflow + +All worker tasks need to be registered to the Conductor server before it can be added to a workflow. Let’s add a worker to a workflow and give it a test run: +1. Register the task by adding its definition in Conductor. +2. Add a Worker task to a workflow. + +### A. Code + + + + +Register the task definition to Conductor. + +``` python +from conductor.client.http.models.task_def import TaskDef + +taskDef = TaskDef( + name="PYTHON_TASK", + description="Python Task Example", + input_keys=["a", "b"] +) +metadata_client.register_task_def(taskDef) +``` + +Add the Worker task to your workflow. +``` python +workflow >> SimpleTask("simple_task", "simple_task_ref_2") +updatedWorkflowDef = workflow.to_workflow_def() +metadata_client.update_workflow_def(updatedWorkflowDef, True) +``` + + + + + +Create and register the task definition to Conductor. + +``` java +TaskDef taskDef = new TaskDef(); +taskDef.setName(your_task_name); +taskDef.setDescription("task to compress image"); +taskDef3.setOwnerEmail("test@orkes.io"); +taskDef.setRetryCount(3); // Optional + +metadataClient.registerTaskDefs(Arrays.asList(taskDef)); +``` + +Add the Worker task to your workflow. +``` java +builder.add( + new SimpleTask("send_email", "send_email") + .input("email", "${workflow.input.email}") + .input("subject", "Your insurance quote for the amount ${generate_quote.output.amount}") +); +``` + + + + + +Register the task definition to Conductor. + +``` javascript +public registerTask(taskDef: TaskDef): Promise { + return tryCatchReThrow(() => + this._client.metadataResource.registerTaskDef([taskDef]) + ); + } +``` + +Add the Worker task to your workflow. + + + + + +Register the task definition to Conductor. +``` csharp +{ +new TaskDef{Description = ExampleConstants.GetEmailDescription, Name = ExampleConstants.GetEmail }, +new TaskDef{Description = ExampleConstants.SendEmailDescription,Name = ExampleConstants.SendEmail} +}; + + _metaDataClient.RegisterTaskDef(taskDefs); +``` + +Add the Worker task to your workflow. +``` csharp + _metaDataClient.UpdateWorkflowDefinitions(new List(1) { workflow }); +``` + + + + + +Register the task definition to Conductor. + +``` go +client.RegisterTaskDef(ctx, taskDefinitions) +``` + +Add the Worker task to your workflow. + + + + + +Register the task definition to Conductor. + +``` clojure +(defn register-tasks-using-client + "Given a client instance and a list of tasks, + will register the task in consuctor" + [client tasks] + (client "taskdefs" :method :post :body tasks)) +``` + + + + + + +### B. Orkes Platform +1. Register the task definition to Conductor. + 1. In the left navigation menu, go to **Definitions** > **Task**. + 2. Select **(+) Define task**. + 3. Enter the **Name** for the task, which must match the task definition name in your worker code. + 4. Select **Save** > **Confirm Save**. +2. Add the Worker task to your workflow. + 1. In the left navigation menu, go to **Definitions** > **Workflow** and select the workflow to add the task. + 2. In the visual workflow editor, select the **(+)** icon to add a new task. There are two ways to add a worker task: + * Search for your task using its task name and select it. + * Add a **Worker Task (Simple)** and enter the task name in Task Definition. + 3. On the top right, select **Save** > **Confirm**. + + +## Grant execution permission to worker + +Finally, your worker application requires programmatic access to the Conductor server. This can be done by creating an application account for your worker application. + +**To grant execution permission to worker:** +1. In Orkes Platform, go to **Access Control **> **Applications** and create a new application. +2. Enable the **Worker** application role, which allows the application to poll and update tasks. +3. Generate the application access key and set the Key ID and Key Secret in your project environment variables. + +``` +export CONDUCTOR_SERVER_URL= // eg: https://play.orkes.io/api +export CONDUCTOR_AUTH_KEY= +export CONDUCTOR_AUTH_SECRET= +``` + +4. Grant Execute permission to the application. + 1. Under Permissions, select **Add permission**. + 2. Select the **Task** tab and then your worker task. + 3. Enable the **Execute** toggle. + 4. Select **Add Permissions**. + +The application account can now execute the worker task. + + +## Launch the worker + +Launch the worker to begin polling the Conductor server. The method depends on your language and project configuration. + +**Example** + +``` python +python3 main.py +``` + +When you run the workflow with the Worker task, the task should run to completion. Learn how to deploy the workflow in the next quickstart. \ No newline at end of file diff --git a/docs/getting-started/quickstart-3.md b/docs/getting-started/quickstart-3.md new file mode 100644 index 00000000..26da2c86 --- /dev/null +++ b/docs/getting-started/quickstart-3.md @@ -0,0 +1,106 @@ +--- +slug: "../quickstarts/deploy-workflows" +description: "" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Quickstart 3: Deploy Workflows + +Your workflow is now ready to be deployed. This can be done in several different ways: via code, via API, or using the Orkes Platform. + +In this quickstart, you will learn how to deploy your workflow via code. + + + + +``` python +// synchronous execution - applicable for quick-running workflows (0--30 seconds) + + +from conductor.client.http.models import StartWorkflowRequest + +request = StartWorkflowRequest() +request.name = 'hello' +request.version = 1 +request.input = {'name': 'Orkes'} + +workflow_run = workflow_client.execute_workflow( + start_workflow_request=request, + wait_for_seconds=12) +``` + + + + + +Use the WorkflowClient interface to start a workflow: + +``` java +import com.netflix.conductor.client.http.ConductorClient; +import com.netflix.conductor.client.http.WorkflowClient; +import com.netflix.conductor.common.metadata.workflow.StartWorkflowRequest; + +// … other code +var client = new ConductorClient("http://localhost:8080/api"); +var workflowClient = new WorkflowClient(client); +var workflowId = workflowClient.startWorkflow(new StartWorkflowRequest() + .withName("hello_workflow") + .withVersion(1)); + +System.out.println("Started workflow " + workflowId); + +``` + + + + + +``` javascript +const executor = new WorkflowExecutor(client); +const executionId = await executor.startWorkflow({ name, version, input: {} }); +``` + + + + + +``` csharp +using Conductor.Executor; + +var workflowId = workflowExecutor.StartWorkflow(conductorWorkflow); +``` + + + + + +``` go +//Input can be either a map or a struct that is serializable to a JSON map +workflowInput := map[string]interface{}{} + +workflowId, err := executor.StartWorkflow(&model.StartWorkflowRequest{ + Name: conductorWorkflow.GetName(), + Input: workflowInput, +}) +``` + + + + + +``` clojure +(defn start-workflow + "Takes an option map and a start-request map and starts a workflow. + Returns the id of a workflow execution" + ([options wf-request] + (-> (workflow-client options) + (start-workflow-with-client wf-request)))) +``` + + + + + +Ready to dive deeper? Learn more about [running workflows](../developer-guides/running-workflows) \ No newline at end of file diff --git a/docs/getting-started/quickstart-4.md b/docs/getting-started/quickstart-4.md new file mode 100644 index 00000000..237840be --- /dev/null +++ b/docs/getting-started/quickstart-4.md @@ -0,0 +1,34 @@ +--- +slug: "../quickstarts/debug-and-monitor-workflows" +description: "" +--- + +# Quickstart 4: Debug and Monitor Workflows +Orkes Platform provides a dashboard for introspecting each workflow execution, enabling you to debug and monitor while in development or production. + +The introspection dashboard can be found in **Executions** > **Workflow**, where each workflow execution is identified by a workflow ID. + +## Try it out + +Check out the execution of myFirstWorkflow. + +If successful, the workflow should have a Completed status, with each task highlighted in green. Otherwise, the workflow diagram will highlight the failed task in red. + +

Screenshot of the workflow execution screen showing the failed task in red.

+ +However, a workflow can still run until completion even with the wrong logic. You can check if the data have been correctly passed between tasks by selecting a task and selecting its **Input** or **Output** tab. + +For example, the workflow execution below completed successfully. However, the workflow should have flowed through the United States path instead of the defaultCase path, because the user’s location was the United States. Inspecting the Switch task input, we can deduce that the input have not been correctly passed from the get-user task to the Switch task. + +

Screenshot of Task Input tab in the workflow execution screen.

+ +You might also be interested in how long each task took to complete, or to find the bottlenecks in your execution performance. To inspect this, go to **Timeline** in the top navigation bar. + +

Screenshot of Timeline tab in the workflow execution screen.

+ +## What’s next? +Congratulations! You have successfully created, executed, and debugged a workflow in Conductor. Gain deeper mastery by exploring each topic in detail: +* Code with Conductor: [SDK Guides](../category/sdks) +* Build more complex workflows, with LLM chaining, human-in-workflows, eventing, dynamic task inputs, secrets, and more: [Build Workflows](../developer-guides/building-workflows) +* Execute and deploy production-grade workflows, with CI/CD best practices, version control, testing, and scheduling. [Run Workflows](../developer-guides/running-workflows) +* Debug and monitor workflows, by exploring both the introspection dashboard and metrics dashboard in detail. [Deploy and Monitor Workflows](../developer-guides/deploying-workflows) diff --git a/docs/getting-started/quickstart-index.md b/docs/getting-started/quickstart-index.md new file mode 100644 index 00000000..bb34bc54 --- /dev/null +++ b/docs/getting-started/quickstart-index.md @@ -0,0 +1,220 @@ +--- +slug: "../quickstarts" +description: "" +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Quickstarts + +Orchestration lets you develop distributed systems without worrying about coordinating complexity. Conductor is an orchestration engine that runs sequences of tasks, known as workflows, using a worker-task queue architecture. + +In these quickstarts, you will learn the basics of developing with Conductor: +1. How to create workflows +2. How to write custom workers +3. How to deploy workflows to production +4. How to debug workflow + +:::tip +Familiarize yourself with the [Core Concepts](../core-concepts) in Conductor before diving into our quickstarts. +::: + +With Orkes’ suite of SDKs, APIs, and Orkes Platform, you can mix-and-match our products to develop with Conductor. +* **Conductor SDKs**—Manage, test, and deploy workflows; write workers; or integrate Conductor into your applications. Available in Python, Java, Javascript, C#, Go, and Clojure. +* **Conductor APIs**—Manage resources (workflows, tasks, users, etc) programmatically. +* **Orkes Platform**—Manage resources from a user interface. + + +## Preparing your tools + +Before you begin, prepare the following: +* Get UI access to your Orkes Conductor cluster. +* Set up your development environment. +* Configure programmatic access to your Orkes Conductor cluster. + + +### Get UI access + +Orkes Platform offers single-tenancy access. If you have an Orkes account, access your cluster and log in using your organization’s SSO provider. + + +

Screenshot of Orkes log in screen.

+ +### Set up your development environment + +A key part of developing with Conductor involves using our SDKs to write workers, create workflows as code, or develop client applications. Set up your development environment in your preferred language. + + + + +:::info Prerequisites +* [Python 3.9+](https://www.python.org/downloads/) or higher +::: + +In your project directory, create and activate your virtual environment (eg `myProject`). + +**Mac:** +``` bash +// Using venv +python -m venv myProject +source myProject/bin/activate + +// Using virtualenv +virtualenv myProject +source myProject/bin/activate +``` + +**Windows:** +``` shell +// Using venv +python -m venv myProject +myProject\Scripts\activate + +// Using virtualenv +virtualenv myProject +myProject\Scripts\activate +``` + +In your terminal, run the following command to get the Conductor SDK. + +``` bash +python3 -m pip install conductor-python +``` + + + + + +:::info Prerequisites +* JDK 17 from v2.1.2 onwards +* A Gradle or Maven project properly set up +::: + + +**Gradle:** + +In the `build.gradle` file, add the following dependencies. + +``` java +dependencies { + implementation 'org.conductoross:conductor-client:4.0.0' + implementation 'io.orkes:orkes-conductor-client:4.0.0' +} +``` + +**Maven:** + +In the `pom.xml` file, add the following dependencies. + +``` xml + + org.conductoross + conductor-client + 4.0.0 + + + io.orkes + orkes-conductor-client + 4.0.0 + +``` + + + + + +:::info Prerequisites +* NodeJS 18 or higher +::: + +Get the Conductor Javascript package using npm or yarn. + +**npm:** +``` shell +npm i @io-orkes/conductor-javascript +``` + +**yarn:** +``` bash +yarn add @io-orkes/conductor-javascript +``` + + + + + +:::info Prerequisites +* .NET Standard 2.0 or higher +::: + +In your terminal, run the following command to get the Conductor SDK. + +``` shell +dotnet add package conductor-csharp +``` + + + + + +:::info Prerequisites +* Go 1.17 or higher +::: + +Initialize your Go module. + +``` +go mod init +``` + +Add the Conductor SDK to your module. + +``` +go get github.com/conductor-sdk/conductor-go +``` + + + + + +:::info Prerequisites +* Clojure v1.11.0 or higher +::: + +Get the Conductor Clojure package from [clojars](https://clojars.org/io.orkes/conductor-clojure). + +``` +:deps {org.clojure/clojure {:mvn/version "1.11.0"} + io.orkes/conductor-clojure {:mvn/version "0.3.0"}} +``` + + + + + +### Configure programmatic access to Conductor + +Once your development environment is set up, you need to configure your access to the Conductor server. In Orkes, programmatic access to Conductor is enabled by application-based access keys. To get authenticated, you must first create an application in Orkes Platform, then create an access key for your application. + +**To create an application:** +1. Log in to your Orkes cluster or the [Orkes Playground](https://play.orkes.io/). +2. In the left navigation menu, go to **Access Control** > **Applications**. +3. Select **(+) Create application**. +4. Enter the application name, such as “myFirstWorkflow”. Use this application while following along with the quickstarts. +5. Select **Save**. + +The application has been created. You can proceed to retrieve an access key. + +**To retrieve the access key:** + +In the Access Keys section, select **(+) Create access key** to generate a unique Key Id and Key Secret. + +The Key Secret is shown only once. Make sure to copy and store it securely, so you can use it when following along with the quickstarts. + + +## Ready to start? +* **[Quickstart 1: Learn how to create your first workflow](/quickstarts/create-first-workflow)**. You can define workflows as code or on Orkes Platform. +* **[Quickstart 2: Learn how to use custom tasks](/quickstarts/write-workers)**. Write workers in any language using Conductor SDKs. +* **[Quickstart 3: Learn how to deploy your workflow](/quickstarts/deploy-workflows)**. There are many ways to do this, such as creating a client application. +* **[Quickstart 4: Learn how to debug and monitor your workflow](/quickstarts/debug-and-monitor-workflows)**. \ No newline at end of file diff --git a/docs/what-is-orkes-conductor.mdx b/docs/what-is-orkes-conductor.mdx index 6338ce09..6e7aab8b 100644 --- a/docs/what-is-orkes-conductor.mdx +++ b/docs/what-is-orkes-conductor.mdx @@ -23,7 +23,7 @@ import { title: "Getting Started", description: "Learn how to build workflows, discover key concepts, and explore powerful features.", - to: "/getting-started/first-workflow-application", + to: "/quickstarts/create-first-workflow", }, { title: "Examples", @@ -50,9 +50,9 @@ import { href="/content/get-orkes-conductor" target="_self" >setting up your cluster,  - build your first workflow  - and continue through our  - getting started guide to learn the basics of Conductor. + build your first workflow  + and continue through our getting started guide  + to learn the basics of Conductor. Continue your journey by walking through the developer guides. } diff --git a/docusaurus.config.js b/docusaurus.config.js index 8a6f1f3b..e902b45b 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -119,7 +119,7 @@ const config = { items: [ { type: "doc", - docId: "getting-started/first-workflow-application", + docId: "getting-started/quickstart-index", position: "left", label: "Getting Started", }, diff --git a/sidebars.js b/sidebars.js index eef54b90..68981d71 100644 --- a/sidebars.js +++ b/sidebars.js @@ -21,35 +21,48 @@ const sidebars = { className: 'leftMenuHeader', }, { - type: 'doc', - id: 'core-concepts', - className: 'leftMenuHeader', - }, - { - type: 'doc', - id: 'getting-started-orkes-cloud', - className: 'leftMenuHeader', - }, - { - type: 'doc', - id: 'get-orkes-conductor', - className: 'leftMenuHeader', - }, - { - type: 'category', // to move to tutorials + type: 'category', label: 'Getting Started', link: { - type: 'generated-index', - title: 'Getting Started', - description: 'Learn about the most important Orkes Conductor concepts!', - slug: '/category/getting-started', - keywords: ['getting-started', 'installation'] + type: 'doc', + id: 'getting-started/introduction', }, - items: ['getting-started/first-workflow-application', 'getting-started/running-workflows-from-code', 'getting-started/adding-custom-code-worker', 'getting-started/running-an-inline-function', 'getting-started/adding-wait-conditions', 'getting-started/executing-tasks-in-parallel'], + items: [ + { + type: 'doc', + id: 'core-concepts', + className: 'leftMenuHeader', + }, + { + type: 'doc', + id: 'get-orkes-conductor', + className: 'leftMenuHeader', + }, + { + type: 'category', + label: 'Quickstarts', + link: { + type: 'doc', + id: 'getting-started/quickstart-index', + }, + className: 'leftMenuHeader', + items: [ + 'getting-started/quickstart-1', + 'getting-started/quickstart-2', + 'getting-started/quickstart-3', + 'getting-started/quickstart-4', + ] + } + ], collapsible: true, collapsed: false, className: 'leftMenuHeader', }, + { + type: 'doc', + id: 'getting-started-orkes-cloud', + className: 'leftMenuHeader', + }, { type: 'category', label: 'Developer Guides', diff --git a/static/img/getting-started/getting_started-failed_task_introspection.png b/static/img/getting-started/getting_started-failed_task_introspection.png new file mode 100644 index 00000000..d1f89d0b Binary files /dev/null and b/static/img/getting-started/getting_started-failed_task_introspection.png differ diff --git a/static/img/getting-started/getting_started-input_introspection.png b/static/img/getting-started/getting_started-input_introspection.png new file mode 100644 index 00000000..75515b16 Binary files /dev/null and b/static/img/getting-started/getting_started-input_introspection.png differ diff --git a/static/img/getting-started/getting_started-log_in_page.png b/static/img/getting-started/getting_started-log_in_page.png new file mode 100644 index 00000000..9d46fd26 Binary files /dev/null and b/static/img/getting-started/getting_started-log_in_page.png differ diff --git a/static/img/getting-started/getting_started-timeline_introspection.png b/static/img/getting-started/getting_started-timeline_introspection.png new file mode 100644 index 00000000..c380a5ca Binary files /dev/null and b/static/img/getting-started/getting_started-timeline_introspection.png differ diff --git a/static/img/getting-started/getting_started-visual_workflow_editor.png b/static/img/getting-started/getting_started-visual_workflow_editor.png new file mode 100644 index 00000000..1b134c1b Binary files /dev/null and b/static/img/getting-started/getting_started-visual_workflow_editor.png differ