Skip to content

Latest commit

 

History

History
340 lines (254 loc) · 9.86 KB

day27.md

File metadata and controls

340 lines (254 loc) · 9.86 KB
Raw

Day 27: 90DaysofDevOps

From Automated to Automatic - Event-Driven Infrastructure Management with Ansible

Daniel Bodky

Overview

A universal truth and recurring theme in the DevOps world is automation. From providing infrastructure to testing code to deploying to production, many parts of the DevOps lifecycle get automated already. One popular technology for managing infrastructure and configuration in an automated way is Ansible, but are we fully utilizing its capabilities yet?

This presentation will give a broad overview of Ansible and its architecture and use-cases, before exploring a relatively new feature, Event-driven Ansible (EDA). Analzying applications of event-driven Ansible, participants will see that automated management is nice, but automatic management is awesome, not just regarding DevOps principles, but also in terms of reaction times, the human tendency for minor mistakes, and toil for operators.

Participants will get first-hand insights into Ansible, its strengths, weaknesses, and the potential of event-driven automation within the DevOps world.

Note

The below content is a copy of the lab repository's README for convenience.

Event-Driven Ansible Lab

This is a lab designed to demonstrate Ansible and how Event-Driven Ansible (EDA) builds on top of its capabilities.

The setup is done with Ansible, too. It will install Ansible, EDA, Prometheus, and Alertmanager on a VM to demonstrate some of the capabilities of EDA.

Prerequisites

To follow along with this lab in its entirety, you will need three VMs:

Note

If you want to skip Ansible basics and go straight to EDA, you'll need just the eda-controller.example.com VM and can skip the others.

VM name OS
eda-controller.example.com CentOS/Rocky 8.9
company.example.com CentOS/Rocky 8.9
webshop.example.com Ubuntu 22.04

You'll need to be able to SSH to each of these VMs as root using SSH keys.

Lab Setup

Clone the repository and create a Python virtual environment

git clone https://github.com/mocdaniel/lab-event-driven-ansible.git
cd lab-event-driven-ansible
python3 -m venv .venv
source .venv/bin/activate

Install Ansible and other dependencies

pip install -r requirements.txt

Create the inventory file

---
# hosts.yml
webservers:
  hosts:
    webshop.example.com:
      ansible_host: <ip-address>
      webserver: apache2
    company.example.com:
      ansible_host: <ip-address>
      webserver: httpd
eda_controller:
  hosts:
    eda-controller.example.com:
      ansible_host: <ip-address>

Install Needed Roles and Collections

ansible-galaxy install -r requirements.yml

Run the Setup Playbook

After you created the inventory file and filled in the IP addresses, you can run the setup playbook:

ansible-playbook playbooks/setup.yml

Caution

Due to a known bug with Python on MacOS, you need to run export NO_PROXY="*" on MacOS before running the playbook

Demos

Lab 1: Ansible Basics

Ansible from the CLI via ansible

Ansible from the CLI via ansible

The first example installs a webserver on all hosts in the webservers group. The installed webserver is defined as a host variable in the inventory file hosts.yml (see above).

ansible \
   webservers  \
  -m package   \
  -a 'name="{{ webserver }}"' \
  --one-line

Afterwards, we can start the webserver on all hosts in the webservers group.

ansible \
   webservers  \
  -m service   \
  -a 'name="{{ webserver }}" state=started' \
  --one-line

Go on and check if the web servers are running on the respective hosts.

[!HINT] Ansible is idempotent - try running the commands again and see how the output differs.

Ansible from the CLI via ansible-playbook

Ansible from the CLI via ansible-playbook

The second example utilizes the following playbook to gather and display information for all hosts in the webservers group, utilizing the example role from the lab repository.

---
- name: Example role
  hosts: webservers
  gather_facts: false
  vars:
    greeting: "Hello World!"
  pre_tasks:
    - name: Say Hello
      ansible.builtin.debug:
        msg: "{{ greeting }}"
  roles:
    - role: example
  post_tasks:
    - name: Say goodbye
      ansible.builtin.debug:
        msg: Goodbye!
ansible-playbook \
    playbooks/example.yml

Lab 2: Event-Driven Ansible

Receive Generic Events via Webhook

Receive Generic Events via Webhook

If you followed the setup instructions for the EDA lab, you should already have a running EDA instance on the eda-controller.example.com VM.

If you navigate to /etc/edacontroller/rulebook.yml on the VM, you'll see the following rulebook:

---
- name: Listen to webhook events
  hosts: all
  sources:
    - ansible.eda.webhook:
        host: 0.0.0.0
        port: 5000
  rules:
    - name: Debug event output
      condition: 1 == 1
      action:
        debug:
          msg: "{{ event }}"

- name: Listen to Alertmanager alerts
  hosts: all
  sources:
    - ansible.eda.alertmanager:
        host: 0.0.0.0
        port: 9000
        data_alerts_path: alerts
        data_host_path: labels.instance
        data_path_separator: .
  rules:
    - name: Restart MySQL server
      condition: event.alert.labels.alertname == 'MySQL not running' and event.alert.status == 'firing'
      action:
        run_module:
          name: ansible.builtin.service
          module_args:
            name: mysql
            state: restarted
    - name: Debug event output
      condition: 1 == 1
      action:
        debug:
          msg: "{{ event }}"

For this part of the lab, the first rule is the one we're interested in: It listens to a generic webhook on port 5000 and prints the event's metadata to its logs.

To test this, we can use the curl command to send a POST request to the webhook /endpoint from the VM itself:

curl \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"foo": "bar"}' \
  http://localhost:5000/endpoint

If you now check the logs of the EDA controller, you should see the following output:

journalctl -fu eda-controller

Jan 11 16:35:29 eda-controller ansible-rulebook[56882]: {'payload': {'foo': 'bar'}, 'meta': {'endpoint': 'endpoint',
'headers': {'Host': 'localhost:5000', 'User-Agent': 'curl/7.76.1', 'Accept': '*/*', 'Content-Length': '21',
'Content-Type': 'application/x-www-form-urlencoded'}, 'source': {'name': 'ansible.eda.webhook', 'type': 'ansible.eda.webhook'},
'received_at': '2024-01-11T15:35:29.798401Z', 'uuid': '6ebf8dd2-60a2-455a-9383-97b81f535366'}}

A rule that always evaluates to true is not very useful, so let's change the rule to only print the the value of foo if the foo key is present in the event's payload, and no foo :( otherwise:

---
- name: Listen to webhook events
  hosts: all
  sources:
    - ansible.eda.webhook:
        host: 0.0.0.0
        port: 5000
  rules:
    - name: Foo
      condition: event.payload.foo is defined
      action:
        debug:
          msg: "{{ event.payload.foo }}"
    - name: No foo
      condition: 1 == 1
      action:
        debug:
          msg: "no foo :("

Send the same curl request again and check the logs, you should see a line saying bar now.

Let's also try a curl request with a different payload:

curl \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"bar": "baz"}' \
  http://localhost:5000/endpoint

This time, the output should be no foo :(.

Restarting Services Automatically with EDA

Restarting Services Automatically with EDA

The last lab is more of a demo - it shows how you can use EDA to automatically react on events observed by Prometheus and Alertmanager.

For this demo, the second ruleset in our rulebook is the one we're interested in:

- name: Listen to Alertmanager alerts
  hosts: all
  sources:
    - ansible.eda.alertmanager:
        host: 0.0.0.0
        port: 9000
        data_alerts_path: alerts
        data_host_path: labels.instance
        data_path_separator: .
  rules:
    - name: Restart MySQL server
      condition: event.alert.labels.alertname == 'MySQL not running' and event.alert.status == 'firing'
      action:
        run_playbook:
          name: ./playbook.yml
    - name: Debug event output
      condition: 1 == 1
      action:
        debug:
          msg: "{{ event }}"

With this rule, we can restart our MySQL server if it's not running! But how do we get the event to trigger? With Prometheus and Alertmanager!

When you ran the setup playbook, it installed Prometheus and Alertmanager on the eda-controller.example.com VM. You can access the Prometheus UI at http://<eda-controller-ip>:9090 and the Alertmanager UI at http://<eda-controller-ip>:9093.

It also installed a Prometheus exporter for the MySQL database that runs on the server.

With this setup, we can now shut down our MySQL server and see what happens - make sure to watch the output of the EDA controller's logs:

systemctl stop mysql
journalctl -fu edacontroller

Within 30-90 seconds, you should see EDA running our playbook and restarting the MySQL server. You can track that process by watching the Prometheus/Alertmanager UIs for firing alerts.

Once you see the playbook being executed in the logs, you can check the MySQL state once more:

systemctl status mysql

MySQL should be up and running again!