Octopus Workflow Builder
Organizations are under increasing pressure to deliver new digital experiences to their customers. To meet this demand, DevOps teams are adopting a number of practices such as Infrastructure as Code (IaC), microservices, GitOps, and Kubernetes, which all lend themselves to writing and maintaining more code than ever before.
An analysis of industry reports helps us understand the paradigm shifts towards an "everything as code" approach to DevOps.
IaC adoption is now mainstream, with most organizations having at least started the process of implementing IaC tools.
The Flexera State of the Cloud Report 2022, available from https://oc.to/bEY0yE, reports:
Native cloud tools are most commonly used today, including AWS CloudFormation templates (50 percent) and Azure Resource Manager templates (48 percent)
The State of Developer Ecosystem 2021, available from https://oc.to/bNVuU2, shows that only 37% of organizations use no infrastructure provisioning tools:
Synk Infrastructure as Code Security Insights 2021, available from https://oc.to/h2zyKh, reports most companies have started (if only just) their IaC journey:
We found that many companies are only starting out on their IaC journey, with 63% just beginning to explore the technology and only 7% stating they’ve implemented IaC to the best of current industry capabilities.
Microservice architecture describes a practice for designing software as independently deployable services. It allows teams to develop, test, release, and maintain individual services written with the best language and framework for the problem they solve.
There is a clear trend towards transitioning to microservices. The JRebel Java Developer Productivity Report 2021, available from https://oc.to/lDACFO, shows only 13% of respondents have no microservice adoption plans:
NGINX State of Modern App Delivery 2020, available from https://oc.to/VE9mbt, reports 60% of applications are built as microservices:
Kong Digital Innovation Benchmark 2021, available from https://oc.to/DKo3ah, shows a majority of respondents have fully or partially adopted microservices:
Eighty-seven percent of respondents say their organizations have either already fully transitioned to entirely distributed architectures (microservices, serverless, etc.) or are currently using a mix of monolithic architectures and microservices.
One of the benefits of microservices is that it allows teams to write code in the best language for the job. The DataDog State of Serverless, available from https://oc.to/rL9vbo, shows the majority of large organizations deploy serverless functions in three or more languages:
Kubernetes has done much to popularize declarative resource management with its rich and extensible resource model (often expressed as YAML) describing the desired state of a cluster.
The JetBrains DevOps The State of Developer Ecosystem in 2021, available from https://oc.to/OKQzSJ, report shows 37% of respondents use Kubernetes in production, and 7% using EKS, which is Amazon's managed Kubernetes service:
The GitLab DevSecOps 2021 survey, available from https://oc.to/OE41eQ, shows 46% of organizations have adopted Kubernetes:
The Red Hat State of Enterprise Open Source 2021, available from https://oc.to/luuCdF, shows 66% of respondents rate Kubernetes as extremely or very important to cloud-native application strategies:
The InfoQ DevOps and Cloud InfoQ Trends Report June 2022, available from https://oc.to/cY97Dx, lists yet more trends relating to code centric DevOps practices, with GitOps being adopted by an early majority and both policy as code and documentation as code implemented by early adopters:
The reports above highlight a clear trend towards processes expressed as code. DevOps teams are now expected to not only write application code (often using multiple languages), but also write code for infrastructure, networking, policy, and documentation.
This explosion of code provides many benefits in terms of repeatability, auditing, and reuse. But the flexibility of code and complex configuration files presents challenges for teams looking to enforce standards and promote consistent solutions.
This is not a new problem though. Engineering tools have long provided the ability to scaffold new projects from predefined templates. Whether using Maven archetypes, the dotnet new command, or template generators like Yeoman, development teams are able to quickly create new projects, removing the need to rewrite boilerplate code, and embedding best practices from the outset.
In a world where everything is code, all members of DevOps teams require the same ability to quickly create new projects from templates embedding best practices and business knowledge.
Platform engineering tools provide a solution by allowing DevOps teams to self service such templates. However, traditional scaffolding tools have some limitations, especially when scaled up to support large DevOps teams, which we'll cover in the next section.
In the previous section we looked at statistics from a multitude of reports indicating a clear trend towards defining everything as code, often written in multiple languages.
Scaffolding tools provide a way to bootstrap these template projects, as described by Wikipedia:
Complicated software projects often share certain conventions on project structure and requirements. For example, they often have separate folders for source code, binaries and code tests, as well as files containing license agreements, release notes and contact information. To simplify the creation of projects following those conventions, "scaffolding" tools can automatically generate them at the beginning of each project. Such tools include Yeoman, Cargo and Ritchie CLI.
Some of these tools have enjoyed immense popularity. Yeoman has close to ten thousand generators written by the community, with the JHipster template alone being downloaded over one hundred thousand times each month.
However, there are limitations to the traditional Command Line Interface (CLI) based tooling usually provided by tools like Yeoman.
Most scaffolding tools are CLI driven, forcing each member of the DevOps team to install the tools, download any templates, and generate the template project locally.
While this is a great solution for developers looking to bootstrap a project, it does have limitations when exposing standardized templates across a larger team.
To begin with, not all consumers of scaffolding tools have an engineering background. The "everything as code" movement extends just as easily to common documents like markdown or AsciiDoc. Installing tools like npm, installing scaffolding template dependencies, and then executing CLI tooling from the command line will be unfamiliar processes for many non-technical folk.
Managing dependencies becomes an increasing challenge as the size of a team grows. Each new version of a scaffolding template dependency must either be distributed to each user of the scaffolding tooling, or the users must take responsibility for ensuring they have the appropriate version installed locally before creating any template projects. In essence, each dependency has to be treated like a regular desktop application, with new versions pushed to end users machines. Unfortunately, dependency management tools were never designed to support this use case.
Tracking usage is also a challenge when every user of the scaffolding tool executes it locally on their own machine. Once you reach the point in your DevOps journey where scaffolding tools become part of your toolbox, you'll also have reached a point where everything is measured and reported on.
Finally, when using third party scaffolding templates, it is quite likely that only a small subset of the available options are appropriate for your organization. These values need to be documented in a disconnected system, such as a wiki, forcing your DevOps team to reference the documentation each time they wish to deploy a new template project.
The Octopus Workflow Builder overcomes these limitations by exposing Yeoman generators (the term used by Yeoman for scaffolding templates) via a web based interface. This allows anyone with a web browser to generate template projects, offloads dependency management to a centrally managed backend service, and allows rich UIs to be build with no-code JSON documents.
The Octopus Workflow Builder achieves this by implementing Yeoman as a service, providing point and click template generation to anyone with a web browser.
In the next section we'll get started with the Octopus Workflow Builder to generate template projects through the web based interface.
The Octopus Workflow Builder is provided as Docker images launched by Docker Compose.
To get started, you require Git and Docker. Installation instructions for Git are found at https://oc.to/jxG7mx, and the instructions for installing Docker are found at https://oc.to/bchkl4.
Note that Linux users may prefer to install docker.io instead of docker-ce. The StackOverflow question at https://oc.to/t1IZpm provides a discussion on the difference between the two packages.
-
Clone the Git repo containing the Octopus Workflow Builder source code with the command:
git clone https://github.com/OctopusSamples/content-team-apps.git -
Enter the directory containing the Docker Compose files with the command:
cd content-team-apps/docker/customizable-workflow-builderWindows users may need to use backslashes instead:
cd content-team-apps\docker\customizable-workflow-builder -
Start the containers with the command:
docker compose upLinux users who installed
docker.iomay need to use a slightly different syntax to start the containers:docker-compose up -
Open the web UI at http://localhost:5000.
You are presented with an initial list of generators, which shows the single option Spring Boot. Click the button, which presents the options associated with the generator. You can leave the default values as they are for now and click the Download button.
After a minute or so a ZIP file called template.zip is downloaded which includes the template project.
And with that you have successfully run the Octopus Workflow Builder and generated your first template project.
It is worth taking a moment to appreciate the benefits of the process you completed:
- Accessing the Octopus Workflow Builder required nothing more than a web browser.
- You did not have to know the name and specific version of a generator, as the web UI exposed a curated list for you.
- The rich user interface provides all the information you need to complete the process of creating a template project.
In the next section we'll look at how the user interface is built and how it can be customized to suit your organizations particular requirements.
The Octopus Workflow Builder is designed to be easily customized, allowing teams to define their own list of generators and the options passed to them.
The user interface is built with Adaptive Cards, described at https://oc.to/PoMD04. Adaptive Cards author user interfaces in JSON, providing the ability to fully customize the Octopus Workflow Builder without writing any code.
The first file loaded by the Octopus Workflow Builder is located at cards/index.json. The contents of this file is shown below:
{
"type": "AdaptiveCard",
"version": "1.0",
"body": [
{
"type": "TextBlock",
"text": "Select your generator"
},
{
"type": "ActionSet",
"actions": [
{
"type": "Action.Execute",
"title": "Spring Boot",
"verb": "openCard",
"data": {
"filename": "springboot.json"
}
}
]
}
]
}Let's break this file down.
The type property must be set to AdaptiveCard:
"type": "AdaptiveCard",The version property defines the schema version required for this card.
At the time of writing the latest version is 1.3, however most examples you'll find online are written for version 1.0, and so the examples shown here also use version 1.0:
"version": "1.0",The body array contains the elements shown in the primary card region:
"body": [The first element is a text block. The type property defines the element type, which is a required property for all elements and containers. The text property defines the text to display for a TextBlock element.
This element is used to display a heading:
{
"type": "TextBlock",
"text": "Select your generator"
},The next element is an action set container. Actions are displayed as a button, and an action set groups actions together:
{
"type": "ActionSet",The actions array defines the actions held by the action set:
"actions": [
Adaptive Cards define a number of actions to perform tasks like opening URLs, toggling the visibility of other elements, submitting data to the client (the client in this case is the JavaScript code that displays the Adaptive Cards UI), and executing custom actions.
Setting the type property to Action.Execute indicates that we are defining an action to execute a custom command:
{
"type": "Action.Execute",The name of the custom command is defined in the verb property. The openCard verb is recognized by the client as an instruction to load a new card JSON file and display it on the web page:
"verb": "openCard",The data property defines key/value pairs with data required to support the verb. Here we have defined the filename with the name of a new card JSON file to display:
"data": {
"filename": "springboot.json"
}
}
]
}
]
}Let's now look at the springboot.json file.
The springboot.json file contains the inputs required to create a template project. The contents of this file is shown below:
{
"type": "AdaptiveCard",
"version": "1.0",
"body": [
{
"type": "Input.Text",
"id": "generator",
"value": "generator-springboot",
"isVisible": false
},
{
"type": "TextBlock",
"text": "What is the application name?"
},
{
"type": "Input.Text",
"id": "answer.string.appName",
"value": "myservice"
},
{
"type": "TextBlock",
"text": "What is the default package name?"
},
{
"type": "Input.Text",
"id": "answer.string.packageName",
"value": "com.mycompany.myservice"
},
{
"type": "TextBlock",
"text": "Which type of database you want to use?"
},
{
"type": "Input.ChoiceSet",
"id": "answer.string.databaseType",
"isMultiSelect": false,
"value": "postgresql",
"choices": [
{
"title": "Postgresql",
"value": "postgresql"
},
{
"title": "MySQL",
"value": "mysql"
},
{
"title": "MariaDB",
"value": "mariadb"
}
]
},
{
"type": "TextBlock",
"text": "Which type of database migration tool you want to use?"
},
{
"type": "Input.ChoiceSet",
"id": "answer.string.dbMigrationTool",
"isMultiSelect": false,
"value": "flywaydb",
"choices": [
{
"title": "FlywayDB",
"value": "flywaydb"
},
{
"title": "Liquibase",
"value": "liquibase"
},
{
"title": "None",
"value": "none"
}
]
},
{
"type": "TextBlock",
"text": "Select the features you want?"
},
{
"type": "Input.ChoiceSet",
"id": "answer.list.features",
"isMultiSelect": true,
"choices": [
{
"title": "ELK Docker configuration",
"value": "elk"
},
{
"title": "Prometheus, Grafana Docker configuration",
"value": "monitoring"
},
{
"title": "Localstack Docker configuration",
"value": "localstack"
}
]
},
{
"type": "TextBlock",
"text": "Which build tool do you want to use?"
},
{
"type": "Input.ChoiceSet",
"id": "answer.string.buildTool",
"isMultiSelect": false,
"value": "maven",
"choices": [
{
"title": "Maven",
"value": "maven"
},
{
"title": "Gradle",
"value": "gradle"
}
]
}
],
"actions": [
{
"type": "Action.Execute",
"title": "< Back",
"verb": "openCard",
"data": {
"filename": "index.json"
}
},
{
"type": "Action.Execute",
"title": "Download",
"verb": "downloadTemplate"
}
]
}Many of the properties in the springboot.json file are identical to the index.json file, so we'll focus on the important differences here.
Each card used to generate a template project includes a field with the id of generator and value set to the name of the full Yeoman NPM generator package to be used.
Yeoman NPM generator packages all start with the prefix generator, for example generator-springboot.
Note that unlike the Yeoman yo command line tool, which drops the generator prefix from package names passed as arguments, the value defined here is the full NPM package name.
Because this is a fixed value that does not need to be displayed to the end user, the element's isVisible property is set to false:
{
"type": "Input.Text",
"id": "generator",
"value": "generator-springboot",
"isVisible": false
}We use a Input.Text element to provide a text field for inputting a custom answer to a question exposed by the generator.
The answer to each question exposed by the generator is captured by elements with id properties with the format answer.[type].[name].
The type component is set to one of the following values:
-
string- for string answers. -
number- for numeric answers. -
boolean- for true/false answers. -
char- for single character answers. -
list- for answers with multiple values.
The name component is set to the answer name as defined in the generator code.
Note that it is not immediately obvious what type and name to use without inspecting the source code of a generator. A tool called yeoman-inspector has been provided to facilitate the construction of Adaptive Cards used to generate template projects. This tool will be covered in later sections. For now, it is enough to understand how the id of an element is constructed.
The value property defines the default value for this element:
{
"type": "Input.Text",
"id": "answer.string.appName",
"value": "myservice"
},A Input.ChoiceSet element can be used where one option from a fixed list is required for an answer. Setting the isMultiSelect property to false ensures only one value can be selected:
{
"type": "Input.ChoiceSet",
"id": "answer.string.databaseType",
"isMultiSelect": false,
"value": "postgresql",
"choices": [
{
"title": "Postgresql",
"value": "postgresql"
},
{
"title": "MySQL",
"value": "mysql"
},
{
"title": "MariaDB",
"value": "mariadb"
}
]
}Where multiple options can be selected, a Input.ChoiceSet with isMultiSelect set to true is used.
Note the id of the element is set to answer.list.[name] to indicate that this answer accepts a list of values:
{
"type": "Input.ChoiceSet",
"id": "answer.list.features",
"isMultiSelect": true,
"choices": [
{
"title": "ELK Docker configuration",
"value": "elk"
},
{
"title": "Prometheus, Grafana Docker configuration",
"value": "monitoring"
},
{
"title": "Localstack Docker configuration",
"value": "localstack"
}
]
}All other elements are variations of the ones shown above, typically using a Input.Text for questions with a custom answer, a single-select Input.ChoiceSet for questions accepting a single answer from a fixed list, and a multi-select Input.ChoiceSet for questions accepting multiple answers from a fixed list.
The actions array defines the list of actions to perform. Unlike the ActionSet used in the index.json file, which displays action buttons in the primary card region, the actions array always displays action buttons in the card's action bar displayed at the bottom of the card:
"actions": [The first action returns the user back to the card defined in the index.json file:
{
"type": "Action.Execute",
"title": "< Back",
"verb": "openCard",
"data": {
"filename": "index.json"
}
},The second action sets the verb property to downloadTemplate to instruct the client to pass all the answers captured in the card's elements to the template generator service, which in turn executes Yeoman to generate the template project zip file. The resulting file is then downloaded by the browser:
{
"type": "Action.Execute",
"title": "Download",
"verb": "downloadTemplate"
}While it is possible to build these JSON files by hand, the only way to know the names, types, and default options of questions is to inspect the source code of the Yeoman generators. This is not a trivial task, so a tool called yeoman-inspector provides a convenient method of quickly building card JSON files from any generator.
The yeoman-inspector tool provides a convenient way to generate card JSON files by capturing the questions asked by a generator and mapping them to the appropriate card elements. The steps below detail how to install and use the yeoman-inspector tool.
-
Install the
yeoman-inspectortool with the following command:npm install -g @octopus-content-team/yeoman-input-inspector -
The generator you wish to build a card for must then be installed. In this example we'll install the
springbootgenerator:npm install -g generator-springboot -
Inspect the generator with the command:
yeoman-inspector springboot
The output displayed by the tool includes detailed information on the arguments, options, and questions exposed by the generator. These are the three different types of inputs used to configure a generator, although in practice the vast majority of generators are fully configured via questions, and arguments and options are secondary and optional if they are used at all. You can find more information on arguments, options, and questions at https://oc.to/8KmpnC.
The final section of the output, under the ADAPTIVE CARD EXAMPLE heading, is an Adaptive Card JSON file ready to be saved in the cards directory and referenced by the index.json file:
The yeoman-inspector tool makes it trivial to create a basic Adaptive Card for generating template projects. The resulting JSON files can then be tweaked to further restrict the options presented to the end user, to add additional validation rules (see https://oc.to/K2Ze7i for documentation on Adaptive Card input validation), or to make the card more visually appealing.
The yeoman-inspector tool attempts to execute a generator to the point where all the questions are asked, but does not go on to write any files to disk.
It does this by only executing the initial stages of the Yeoman run loop. You can find more information on the run loop at https://oc.to/UO6buP.
However, there are times when generators don't adhere to the Yeoman run loop priorities, meaning questions may be asked in priorities like writing that yeoman-inspect does not execute by default.
Setting the ALLOW_FULL_INSTALL environment variable to true forces yeoman-inspector to execute the full run loop. This results in template files being generated, but also ensures any questions asked outside of their typical run loop priorities will be captured.
There are configuration options to be aware of when running the Octopus Workflow Builder. These settings are configured as environment variables, and are defined in the the Docker Compose compose.yml file, which is shown below:
version: "3.9"
services:
frontend:
image: "octopussamples/customizableworkflowbuilderfrontend"
pull_policy: always
ports:
- "127.0.0.1:5000:5000"
volumes:
- ${PWD}/cards:/workspace/dist/cards
- ${PWD}/config.json:/workspace/dist/config.json
templategenerator:
image: "octopussamples/workflowbuildertemplategenerator"
pull_policy: always
ports:
- "127.0.0.1:4000:4000"
expose:
- "4000"
environment:
# UNSAFE_ENABLE_NPM_INSTALL is unsafe because it allows any random generator to be downloaded and run.
# Generators are just JavaScript code, which can do literally anything. UNSAFE_ENABLE_NPM_INSTALL should
# only be enabled for testing.
# The preferred solution to including new generators is to install them directly into the Docker image
# using "npm -i --no-save generator-<generatorname>".
- UNSAFE_ENABLE_NPM_INSTALL=trueAs the comments indicate, setting the UNSAFE_ENABLE_NPM_INSTALL environment variable for the octopussamples/workflowbuildertemplategenerator image has security implications.
Setting UNSAFE_ENABLE_NPM_INSTALL to true means the service that executes the Yeoman generators will attempt to download any generator that is not preinstalled in the Docker image. This is convenient for testing, but is not recommended from production use.
This is because Yeoman generators are just Node.js applications, with full access to all Node.js features like disk and network access. The ability to install and execute any random generator allows bad actors to essentially run any code they like within your Docker container.
A better solution is to set UNSAFE_ENABLE_NPM_INSTALL to false (which is the default value), and set the NPM_INSTALL_SAFELIST environment variable to a comma separated list of trusted generator NPM package names. An example compose.yml file with this setting is shown below:
version: "3.9"
services:
frontend:
image: "octopussamples/customizableworkflowbuilderfrontend"
pull_policy: always
ports:
- "127.0.0.1:5000:5000"
volumes:
- ${PWD}/cards:/workspace/dist/cards
- ${PWD}/config.json:/workspace/dist/config.json
templategenerator:
image: "octopussamples/workflowbuildertemplategenerator"
ports:
- "127.0.0.1:4000:4000"
expose:
- "4000"
environment:
- UNSAFE_ENABLE_NPM_INSTALL=false
- NPM_INSTALL_SAFELIST=generator-springbootAnother option is to create a custom image based on octopussamples/workflowbuildertemplategenerator and install trusted generators into the image directly.
Create a file called Dockerfile with the following contents. Replace the npm install generator-springboot command with the name of the generator packages you wish to install:
FROM octopussamples/workflowbuildertemplategenerator
USER root
RUN apt-get update; apt-get install npm -y
USER heroku
RUN cd /app; npm install generator-springboot
Build a new Docker image with the command:
docker build . -t myworkflowbuildertemplategenerator
Then use the new Docker image in the compose.yml file:
version: "3.9"
services:
frontend:
image: "octopussamples/customizableworkflowbuilderfrontend"
pull_policy: always
ports:
- "127.0.0.1:5000:5000"
volumes:
- ${PWD}/cards:/workspace/dist/cards
- ${PWD}/config.json:/workspace/dist/config.json
templategenerator:
image: "myworkflowbuildertemplategenerator"
ports:
- "127.0.0.1:4000:4000"
expose:
- "4000"
environment:
- UNSAFE_ENABLE_NPM_INSTALL=falseSetting the UNSAFE_ENABLE_NPM_INSTALL environment variable to false and either setting the NPM_INSTALL_SAFELIST environment variable or preinstalling trusted generators into the base image are required when running the Octopus Workflow Builder in production.
We hope the Octopus Workflow Builder provides your team with a convenient solution to create template projects in a code first DevOps world.
This is a free and open source project hosted at https://github.com/OctopusSamples/content-team-apps.
If you have any feedback, feel free to open a new GitHub issue.
You may also wish to provide a PR for additional features or fixes. The source code to the frontend project is found at https://github.com/OctopusSamples/content-team-apps/tree/main/js/template-customizable-frontend, and the backend template generator source code is found at https://github.com/OctopusSamples/content-team-apps/tree/main/js/octopus-template-generator.
Happy deployments!