CI CD Pipeline Configuration for the Python Application

Overview

This project uses GitHub Actions for Continuous Integration (CI) and Continuous Deployment (CD). The CI workflow is defined in ci.yml, and the CD workflows are defined in cd.dev.yml, cd.staging.yml, and cd.prod.yml. These workflows ensure that the code is tested and deployed automatically whenever changes are pushed to the respective branches.

Continuous Integration (CI).

This guide explains the ci.yml configuration for setting up a continuous integration (CI) pipeline using GitHub Actions for the deployment of a python application. The pipeline is designed to build, run migrations and test the application with PostgreSQL as the database.

Triggering the Workflow.

The workflow is triggered by push and pull_request events on the main, staging, and dev branches.

name: CI

on:
  push:
    branches: [main, staging, dev]
  pull_request:
    branches: [main, staging, dev]

Jobs Configuration.

The workflow consists of a single job, build-and-test, which runs on the ubuntu-latest environment.

jobs:
  build-and-test:
    runs-on: ubuntu-latest

Services.

The job sets up a PostgreSQL service within the ubuntu-latest environment for the database.

       services:
      postgres:
        image: postgres:latest
        env:
          POSTGRES_USER: ${{ secrets.POSTGRES_USER }}
          POSTGRES_PASSWORD: ${{ secrets.POSTGRES_PASSWORD }}
          POSTGRES_DB: ${{ secrets.POSTGRES_DB }}
        ports:
          - 5432:5432

Attach the variables POSTGRES_USER, POSTGRES_PASSWORD and POSTGRES_DB to secrets and input the secret variables.

Steps:

1. Checkout Code

Checks out the repository code.

steps:
   - name: Checkout code
     uses: actions/checkout@v2

The checkout@v2 has other versions like the v1 which is the initial release version and v3 which is the latest version.

2. Set Up Python

Sets up a Python environment.

   - name: Set up Python
     uses: actions/setup-python@v4
     with:
       python-version: "3.9"
       virtual-environment: venv

This sets up a python virtual environment in the ubuntu-latest environment where the dependencies will be installed, .env.sample file copied to .env and the migration and test occurs as well.

3. Install Dependencies

Installs the required Python dependencies.

   - name: Install dependencies
     run: |
       pip install -r requirements.txt

This step uses pip, the Python package installer, to install all the packages listed in requirements.txt. This ensures a consistent and reproducible environment for running the application and tests.

4. Copy Environment Files

Copies environment files from .env.sample to .env

   - name: Copy env file
     run: cp .env.sample .env

In the CI pipeline, environment files (.env files) contain configuration settings like database credentials, API keys, and other environment-specific variables.

5. Run Migrations

Runs database migrations using Alembic.

   - name: Run migrations
     run: |
       activate
       alembic upgrade head

Running database migrations using Alembic involves applying changes to your database schema. Alembic is a lightweight database migration tool for use with SQLAlchemy and Python SQL toolkit. Migrations can include creating, modifying, or deleting tables and columns, as well as other database schema changes. This step ensures that the database schema is up to date with the latest changes in the application code.

6. Run Tests

Executes the test suite using pytest.

   - name: Run tests
     run: |
       activate
       pytest

Pytest is a popular testing framework for Python that makes it easy to write scalable test cases for your Python code. In this CI pipeline, running tests with Pytest ensures the code is working as expected and helps catch any bugs or issues before the code is deployed to the dev, staging, or main branches.

Continuous Deployment (CD)

Workflow Files: .github/workflows/cd.dev.yml, .github/workflows/cd.staging.yml, .github/workflows/cd.prod.yml
The CD workflows are triggered when the CI workflow completes for the respective branches (dev, staging, main). They deploy the code to remote servers using SSH.

Let's take the CD Workflow Code (Example: cd.dev.yml). Note that the same process follows for staging and prod respectively.

name: Dev Branch Deployment

on:
  workflow_run:
    workflows: ["CI"]
    types:
      - completed
    branches: [dev]

jobs:
  on-success:
    runs-on: ubuntu-latest
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Use SSH Action
        uses: appleboy/[email protected]
        with:
          host: ${{ secrets.HOST }}
          username: ${{ secrets.USERNAME }}
          password: ${{ secrets.PASSWORD }}
          script: |
            cd python/dev_source_code/
            git pull origin dev
            source .venv/bin/activate
            pip install -r requirements.txt

  on-failure:
    runs-on: ubuntu-latest
    if: ${{ github.event.workflow_run.conclusion == 'failure' }}
    steps:
      - run: echo 'The triggering workflow failed'

Let's dissect the cd.dev.yml file and examine each section in detail:

1. Workflow Name

name: Dev Branch Deployment

This line sets the name of the workflow. In this case, it is named "Dev Branch Deployment".

2. Trigger Events

on:
  workflow_run:
    workflows: ["CI"]
    types:
      - completed
    branches: [dev]

This section specifies the events that trigger the workflow. Here, the workflow is triggered when the CI workflow completes for the dev branch.

• workflows: Specifies the name of the workflow that triggers this deployment workflow.
• types: Specifies the type of event that triggers the workflow. In this case, it is triggered when the CI workflow is completed.
• branches: Specifies the branch that triggers the workflow. Here, it is the dev branch.

3. Jobs The jobs section defines the tasks that will be executed when the workflow is triggered. There are two jobs defined: on-success and on-failure.

   3.1. On-Success Job

jobs:
  on-success:
    runs-on: ubuntu-latest
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Use SSH Action
        uses: appleboy/[email protected]
        with:
          host: ${{ secrets.HOST }}
          username: ${{ secrets.USERNAME }}
          password: ${{ secrets.PASSWORD }}
          script: |
            cd python/dev_source_code/
            git pull origin dev
            source .venv/bin/activate
            pip install -r requirements.txt

• runs-on: Specifies the type of runner to use. Here, it uses ubuntu-latest.

• if: Conditional statement that checks if the CI workflow concluded successfully.

• steps: Defines the steps to be executed in this job.

  • Checkout code: Uses the actions/checkout@v3 action to check out the repository.
  • Use SSH Action: Uses the appleboy/[email protected] action to SSH into the remote server and deploy the code.
    • host: The hostname of the remote server, stored in the GitHub secret HOST.
    • username: The SSH username, stored in the GitHub secret USERNAME.
    • password: The SSH password, stored in the GitHub secret PASSWORD.
    • script: The script to execute on the remote server:
      • cd python/dev_source_code/: Change directory to the source code location.
      • git pull origin dev: Pull the latest code from the dev branch.
      • source .venv/bin/activate: Activate the virtual environment.
      • pip install -r requirements.txt: Install the required dependencies.

  3.2. On-Failure Job

  on-failure:
    runs-on: ubuntu-latest
    if: ${{ github.event.workflow_run.conclusion == 'failure' }}
    steps:
      - run: echo 'The triggering workflow failed'

• runs-on: Specifies the type of runner to use. Here, it uses ubuntu-latest.

• if: Conditional statement that checks if the CI workflow concluded with a failure.

• steps: Defines the steps to be executed in this job.

  • Run: Prints a message indicating that the triggering workflow failed.

Summary

The CI pipeline ensures that code changes are automatically tested and validated, maintaining the integrity and reliability of the application. The CD pipeline is designed to deploy the branches to a remote server when the CI workflow completes successfully. It uses SSH to connect to the server, pull the latest code, activate the virtual environment, and install dependencies. If the CI workflow fails, it prints a failure message.

Walkthrough of the CI/CD Pipeline

1. Code changes: Developers make changes to the code and push them to the repository.
2. CI workflow triggers: The ci.yml workflow is triggered on push and pull requests to the main, staging, and dev branches.
3. CI workflow runs: The ci.yml workflow executes the tasks outlined above.
4. CD workflow triggers: On completion of the ci.yml workflow, the corresponding CD workflow (cd.dev.yml, cd.staging.yml, or cd.prod.yml) is triggered.
5. CD workflow runs: The CD workflow executes the tasks outlined above, deploying the code to the specified environment if the CI workflow is successful and exits if the CI workflow fails.