Skip to content

Latest commit

 

History

History
265 lines (188 loc) · 11.2 KB

README.md

File metadata and controls

265 lines (188 loc) · 11.2 KB
Raw

shell-operator logo

docker pull flant/shell-operator GH Discussions Telegram chat RU

Shell-operator is a tool for running event-driven scripts in a Kubernetes cluster.

This operator is not an operator for a particular software product such as prometheus-operator or kafka-operator. Shell-operator provides an integration layer between Kubernetes cluster events and shell scripts by treating scripts as hooks triggered by events. Think of it as an operator-sdk but for scripts.

Shell-operator is used as a base for more advanced addon-operator that supports Helm charts and value storages.

Shell-operator provides:

  • Ease of management of a Kubernetes cluster: use the tools that Ops are familiar with. It can be bash, python, kubectl, etc.
  • Kubernetes object events: hook can be triggered by add, update or delete events. Learn more about hooks.
  • Object selector and properties filter: shell-operator can monitor a particular set of objects and detect changes in their properties.
  • Simple configuration: hook binding definition is a JSON or YAML document on script's stdout.
  • Validating webhook machinery: hook can handle validating for Kubernetes resources.
  • Conversion webhook machinery: hook can handle version conversion for Kubernetes resources.

Contents:

Quickstart

You need to have a Kubernetes cluster, and the kubectl must be configured to communicate with your cluster.

The simplest setup of shell-operator in your cluster consists of these steps:

  • build an image with your hooks (scripts)
  • create necessary RBAC objects (for kubernetes bindings)
  • run Pod or Deployment with the built image

For more configuration options see RUNNING.

Build an image with your hooks

A hook is a script that, when executed with --config option, outputs configuration to stdout in YAML or JSON format. Learn more about hooks.

Let's create a small operator that will watch for all Pods in all Namespaces and simply log the name of a new Pod.

kubernetes binding is used to tell shell-operator about objects that we want to watch. Create the pods-hook.sh file with the following content:

#!/usr/bin/env bash

if [[ $1 == "--config" ]] ; then
  cat <<EOF
configVersion: v1
kubernetes:
- apiVersion: v1
  kind: Pod
  executeHookOnEvent: ["Added"]
EOF
else
  podName=$(jq -r .[0].object.metadata.name $BINDING_CONTEXT_PATH)
  echo "Pod '${podName}' added"
fi

Make the pods-hook.sh executable:

chmod +x pods-hook.sh

You can use a prebuilt image flant/shell-operator:latest with bash, kubectl, jq and shell-operator binaries to build you own image. You just need to ADD your hook into /hooks directory in the Dockerfile.

Create the following Dockerfile in the directory where you created the pods-hook.sh file:

FROM flant/shell-operator:latest
ADD pods-hook.sh /hooks

Build an image (change image tag according to your Docker registry):

docker build -t "registry.mycompany.com/shell-operator:monitor-pods" .

Push image to the Docker registry accessible by the Kubernetes cluster:

docker push registry.mycompany.com/shell-operator:monitor-pods

Create RBAC objects

We need to watch for Pods in all Namespaces. That means that we need specific RBAC definitions for shell-operator:

kubectl create namespace example-monitor-pods
kubectl create serviceaccount monitor-pods-acc --namespace example-monitor-pods
kubectl create clusterrole monitor-pods --verb=get,watch,list --resource=pods
kubectl create clusterrolebinding monitor-pods --clusterrole=monitor-pods --serviceaccount=example-monitor-pods:monitor-pods-acc

Install shell-operator in a cluster

Shell-operator can be deployed as a Pod. Put this manifest into the shell-operator-pod.yaml file:

apiVersion: v1
kind: Pod
metadata:
  name: shell-operator
spec:
  containers:
  - name: shell-operator
    image: registry.mycompany.com/shell-operator:monitor-pods
    imagePullPolicy: Always
  serviceAccountName: monitor-pods-acc

Start shell-operator by applying a shell-operator-pod.yaml file:

kubectl -n example-monitor-pods apply -f shell-operator-pod.yaml

It all comes together

Let's deploy a kubernetes-dashboard to trigger kubernetes binding defined in our hook:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/recommended.yaml

Now run kubectl -n example-monitor-pods logs po/shell-operator and observe that the hook will print dashboard pod name:

...
INFO[0027] queue task HookRun:main                       operator.component=handleEvents queue=main
INFO[0030] Execute hook                                  binding=kubernetes hook=pods-hook.sh operator.component=taskRunner queue=main task=HookRun
INFO[0030] Pod 'kubernetes-dashboard-775dd7f59c-hr7kj' added  binding=kubernetes hook=pods-hook.sh output=stdout queue=main task=HookRun
INFO[0030] Hook executed successfully                    binding=kubernetes hook=pods-hook.sh operator.component=taskRunner queue=main task=HookRun
...

Note: hook output is logged with output=stdout label.

To clean up a cluster, delete namespace and RBAC objects:

kubectl delete ns example-monitor-pods
kubectl delete clusterrole monitor-pods
kubectl delete clusterrolebinding monitor-pods

This example is also available in /examples: monitor-pods.

Hook binding types

Every hook should respond with JSON or YAML configuration of bindings when executed with --config flag.

kubernetes

This binding defines a subset of Kubernetes objects that shell-operator will monitor and a jq expression to filter their properties. Read more about onKubernetesEvent bindings here.

Example of YAML output from hook --config:

configVersion: v1
kubernetes:
- name: execute_on_changes_of_namespace_labels
  kind: Namespace
  executeHookOnEvent: ["Modified"]
  jqFilter: ".metadata.labels"

Note: it is possible to watch Custom Defined Resources, just use proper values for apiVersion and kind fields.

Note: return configuration as JSON is also possible as JSON is a subset of YAML.

onStartup

This binding has only one parameter: order of execution. Hooks are loaded at the start and then hooks with onStartup binding are executed in the order defined by parameter. Read more about onStartup bindings here.

Example hook --config:

configVersion: v1
onStartup: 10

schedule

This binding is used to execute hooks periodically. A schedule can be defined with a granularity of seconds. Read more about schedule bindings here.

Example hook --config with 2 schedules:

configVersion: v1
schedule:
- name: "every 10 min"
  crontab: "*/10 * * * *"
  allowFailure: true
- name: "Every Monday at 8:05"
  crontab: "5 8 * * 1"
  queue: mondays

Prometheus target

Shell-operator provides a /metrics endpoint. More on this in METRICS document.

Examples and notable users

More examples of how you can use shell-operator are available in the examples directory.

Prominent shell-operator use cases include:

  • Deckhouse Kubernetes platform where both projects, shell-operator and addon-operator, are used as the core technology to configure & extend K8s features;
  • KubeSphere Kubernetes platform's installer;
  • Kafka DevOps solution from Confluent.

Please find out & share more examples in Show & tell discussions.

Articles & talks

Shell-operator has been presented during KubeCon + CloudNativeCon Europe 2020 Virtual (Aug'20). Here is the talk called "Go? Bash! Meet the shell-operator":

Official publications on shell-operator:

Other languages:

Community

Please feel free to reach developers/maintainers and users via GitHub Discussions for any questions regarding shell-operator.

You're also welcome to follow @flant_com to stay informed about all our Open Source initiatives.

License

Apache License 2.0, see LICENSE.