Conductor 1.0.0

Conductor

Conductor is a workflow server built upon Workflow Core that enables you to coordinate multiple services and scripts into workflows so that you can rapidly create complex workflow applications. Workflows are composed of a series of steps, with an internal data object shared between them to pass information around. Conductor automatically runs and tracks each step, and retries when there are errors.

Workflows are written in either JSON or YAML and then added to Conductor's internal registry via the definition API. Then you use the workflow API to invoke them with or without custom data.

Installation

Conductor is available as a Docker image - danielgerlag/conductor

Conductor uses MongoDB as it's datastore, you will also need an instance of MongoDB in order to run Conductor.

Use this command to start a container (with the API available on port 5001) that points to mongodb://my-mongo-server:27017/ as it's datastore.

$ docker run -p 127.0.0.1:5001:80/tcp --env dbhost=mongodb://my-mongo-server:27017/ danielgerlag/conductor

If you wish to run a fleet of Conductor nodes, then you also need to have a Redis instance, which they will use as a backplane. This is not required if you are only running one instance.
Simply have all your conductor instances point to the same MongoDB and Redis instance, and they will operate as a load balanced fleet.

Environment Variables to configure

You can configure the database and Redis backplane by setting environment variables.

dbhost: <<insert connection string to your MongoDB server>>
redis: <<insert connection string to your Redis server>> (optional)

If you would like to setup a conductor container (API on port 5001) and a MongoDB container at the same time and have them linked, use this docker compose file:

version: '3'
services:
  conductor:
    image: danielgerlag/conductor
    ports:
    - "5001:80"
    links:
    - mongo
    environment:
      dbhost: mongodb://mongo:27017/
  mongo:
    image: mongo

Quick example

We'll start by defining a simple workflow that will log "Hello world" as it's first step and then "Goodbye!!!" as it's second and final step. We POST the definition to api/definition in either YAML or JSON.

POST /api/definition
Content-Type: application/yaml

Id: Hello1
Steps:
- Id: Step1
  StepType: EmitLog
  NextStepId: Step2
  Inputs:
    Message: '"Hello world"'
    Level: '"Information"'
- Id: Step2
  StepType: EmitLog
  Inputs:
    Message: '"Goodbye!!!"'
    Level: '"Information"'

Now, lets test it by invoking a new instance of our workflow.
We do this with a POST to /api/workflow/Hello1

POST /api/workflow/Hello1

We can also rewrite our workflow to pass custom data to any input on any of it's steps.

Id: Hello2
Steps:
- Id: Step1
  StepType: EmitLog
  Inputs:
    Message: data.CustomMessage
    Level: '"Information"'

Now, when we start a new instance of the workflow, we also initialize it with some data.

POST /api/workflow/Hello2
Content-Type: application/x-yaml

CustomMessage: foobar

Further reading

• Documentation

Resources

• Download the Postman Collection

License

This project is licensed under the MIT License - see the LICENSE.md file for details

0.6.beta

Activities

An activity is defined as an item on an external queue of work, that a workflow can wait for.
Use the Activity step type to pause your workflow and wait for an external activity worker that you implement.

The following example will wait for an external activity of name my-activity and pass "Hello from the workflow" as the input data to the worker that will process this activity. Once the worker submits a response, the workflow will continue and log the result from the worker.

Id: activity-test
Steps:
- Id: Step1
  StepType: Activity
  NextStepId: Step2
  Inputs:
    ActivityName: '"my-activity"'
    Parameters: '"Hello from the workflow"'
  Outputs:
    ActivityResult: step.Result
- Id: Step2
  StepType: EmitLog
  Inputs:
    Message: 'data.ActivityResult'

We can then use the activity API to implement a worker to process "my-activity" activities.

First, let's start a new workflow instance.

POST /api/workflow/activity-test
{}

Response:

{
    "workflowId": "5d274481ec9ce50001bc9c34",
    "data": {},
    "definitionId": "activity-test",
    "version": 1,
    "status": "Runnable",
    "reference": null,
    "startTime": "2019-07-11T14:15:29.86Z",
    "endTime": null
}

In our external worker process, we can fetch a waiting activity of an active workflow with the following request.

GET /api/activity/my-activity?timeout=30

This will wait yp to 30 seconds for some work be become ready to process, if there is no workflow waiting on the activity requested, then a 404 Not Found will be returned. If there is work waiting for that activity queue, then an exclusive token will be issued and the reponse will look as follows

{
    "token": "eyJTdWJzY3JpcHRpb25JZCI6IjVlMD",
    "activityName": "my-activity",
    "parameters": "Hello from the workflow",
    "tokenExpiry": "9999-12-31T23:59:59.9999999"
}

• token An exclusive token is issued to the worker to use in future requests for this activity.
• parameters The input data that the workflow attached to this actvity. Can be any object.
• tokenExpiry When the token expires and the activity will be made available to other workers.

To submit a successful response to an activity and pass some response data back to the workflow in the body of the request

POST /api/activity/success/eyJTdWJzY3JpcHRpb25JZCI6IjVlMD

"Hello from the worker"

The workflow will now continue and the "Hello from the worker" string will be mapped to the ActivityResult field on the workflow data, and then logged in the final step. The response data can be any object, not just scalar values.

Conductor 0.3.alpha

Conductor 0.3.alpha

Authentication

Conductor supports integrated authentication using the OpenID Connect protocol.

By default, authentication is disabled. To enable it,

• Set the auth environment variable to 'true'
• Set the alg environment variable to the signing algorithm (RS256 or ES256)
• Set the publickey variable to a Base64 encoded public key.

If authentication is enabled then you need to include a signed JWT bearer token along with every request. The is done by adding the Authorization: Bearer <<token>> header to each request.
The token should be a valid JWT token that was signed with the corresponding private key to the public one in the environment variable.

The token must also include a scope claim that indicate the level of access. The following scopes are used within Conductor.

• conductor:admin - Adminstrative tasks.
• conductor:author - Authoring of workflow definitions and steps.
• conductor:controller - Starting, stopping, suspending and resuming workflows.
• conductor:viewer - Querying the status of a workflow.

A minimal JWT payload the include all the scopes would look as follows

{
  "scope": "conductor:admin conductor:author conductor:controller conductor:viewer"
}

Some authentication servers that support OpenID Connect include

• Auth0 - A cloud service
• Okta - A cloud service
• Keycloak - Open source auth server
• Identity Server - Open source auth server
• Dex - Open source auth server