Skip to content

Latest commit

 

History

History

runner

README.md

Runner

This provides a general enough, containerised and ephemeral execution environment: Pipeline lifecycle and streaming of logs. Each pipeline begins with an initial container image and the steps are applied in order on that image to reach the final state. Multiple isolated pipeline runs could be done concurrently on a single runner.

This is guaranteed to be rootless; ideal for usage in cloud native environments.

How does this work

  • This is implemented in Clojure/JVM
  • Uses RabbitMQ to receive messages and perform the necessary effects as well as producing events via stream
  • Uses XTDB backed by PostgreSQL for temporal persistence
  • Uses contajners to talk to podman to implement step executions.

Configuration

Aero is used and therefore several variables can be set by specifying them as environment variables. Possible variables are:

Environment variables defaults
BOB_STORAGE_URL jdbc:postgresql://localhost:5432/bob
BOB_STORAGE_USER bob
BOB_STORAGE_PASSWORD bob
BOB_QUEUE_URL amqp://localhost:5672
BOB_QUEUE_USER guest
BOB_QUEUE_PASSWORD guest
BOB_CONNECTION_RETRY_ATTEMPTS 10
BOB_CONNECTION_RETRY_DELAY 2000
BOB_STREAM_NAME bob.event-stream
BOB_STREAM_URL rabbitmq-stream://guest:guest@localhost:5552/%2f
BOB_STREAM_RETENTION_DAYS 7
CONTAINER_ENGINE_URL http://localhost:8080

Building and Running

Requirements, min versions, latest recommended.

Using Docker to easily boot up a local cluster

  • Install Docker 18+ and start it up
  • Run docker run -it --name bob-queue -p 5672:5672 -p 15672:15672 -p 5552:5552 -e RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS='-rabbitmq_stream advertised_host localhost' --entrypoint sh rabbitmq:management-alpine -c 'rabbitmq-plugins enable --offline rabbitmq_stream && rabbitmq-server' to run the latest management enabled RabbitMQ instance on port 5672, the streams interface on port 5552 and the admin control on port 15672. The default credentials are guest:guest.
  • Run docker exec bob-queue rabbitmq-plugins enable rabbitmq_stream to enable the stream plugin on the RabbitMQ instance.
  • Run docker run --rm -it --name bob-storage -p 5432:5432 -e POSTGRES_DB=bob -e POSTGRES_USER=bob -e POSTGRES_PASSWORD=bob postgres:alpine to run the latest PostgreSQL instance on port 5432.

Ways of connecting Runner to the cluster

  • To build an uberjar run bb compile to obtain a runner.jar. Running java --enable-preview -jar runner.jar should connect to it.
  • To run directly without building a JAR, run clojure -J--enable-preview -M -m runner.main from this dir.

Setting up the dev environment with the REPL

  • This uses Integrant to manage state across the app.
  • When loaded into the editor/REPL, find the reset fn in this namespace. Eval this when there is change to reload the state cleanly.

Running integration tests

Run bb test from this dir. (needs docker)