Skip to content

Commit

Permalink
Improve CI tests runtime (#457)
Browse files Browse the repository at this point in the history
Before, Cosmos CI took approximately [18m
7s](https://github.com/astronomer/astronomer-cosmos/actions/runs/5823294146/usage)
to run its tests since it did not cache the dependencies.

Configure cache so the CI workers do not have to install all pip
dependencies at each CI job run.

Reduces test runtime to approximately
[12m-14m](https://github.com/astronomer/astronomer-cosmos/actions/runs/5854886292),
resulting in ~ 20-30% time savings to run tests in the CI.
  • Loading branch information
tatiana authored Aug 14, 2023
1 parent 5c68ed7 commit 3be1b27
Show file tree
Hide file tree
Showing 2 changed files with 38 additions and 4 deletions.
38 changes: 36 additions & 2 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,13 @@ jobs:
with:
ref: ${{ github.event.pull_request.head.sha || github.ref }}

- uses: actions/cache@v3
with:
path: |
~/.cache/pip
.nox
key: unit-${{ runner.os }}-${{ matrix.python-version }}-${{ matrix.airflow-version }}-${{ hashFiles('pyproject.toml') }}-${{ hashFiles('cosmos/__init__.py') }}

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
Expand Down Expand Up @@ -62,11 +69,9 @@ jobs:
airflow-version: ["2.3", "2.4", "2.5", "2.6"]
services:
postgres:
# Docker Hub image
image: postgres
env:
POSTGRES_PASSWORD: postgres
# Set health checks to wait until postgres has started
options: >-
--health-cmd pg_isready
--health-interval 10s
Expand All @@ -78,6 +83,12 @@ jobs:
- uses: actions/checkout@v3
with:
ref: ${{ github.event.pull_request.head.sha || github.ref }}
- uses: actions/cache@v3
with:
path: |
~/.cache/pip
.nox
key: integration-${{ runner.os }}-${{ matrix.python-version }}-${{ matrix.airflow-version }}-${{ hashFiles('pyproject.toml') }}-${{ hashFiles('cosmos/__init__.py') }}

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
Expand Down Expand Up @@ -118,10 +129,33 @@ jobs:
matrix:
python-version: ["3.10"]
airflow-version: ["2.6"]

services:
postgres:
image: postgres
env:
POSTGRES_PASSWORD: postgres
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432

steps:
- uses: actions/checkout@v3
with:
ref: ${{ github.event.pull_request.head.sha || github.ref }}
- uses: actions/cache@v3
with:
path: |
~/.cache/pip
.nox
key: integration-expensive-${{ runner.os }}-${{ matrix.python-version }}-${{ matrix.airflow-version }}-${{ hashFiles('pyproject.toml') }}-${{ hashFiles('cosmos/__init__.py') }}

- name: Checkout pull/${{ github.event.number }}
uses: actions/checkout@v3

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
Expand Down
4 changes: 2 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -162,15 +162,15 @@ test-integration = """pytest -vv \
--cov-report=xml \
--durations=0 \
-m integration \
-k 'not test_example_dag[example_cosmos_python_models]'
-k 'not (test_example_dag[example_cosmos_python_models] or test_example_dag[example_virtualenv])'
"""
test-integration-expensive = """pytest -vv \
--cov=cosmos \
--cov-report=term-missing \
--cov-report=xml \
--durations=0 \
-m integration \
-k 'test_example_dag[example_cosmos_python_models]'"""
-k 'test_example_dag[example_cosmos_python_models] or test_example_dag[example_virtualenv]'"""

[tool.pytest.ini_options]
filterwarnings = [
Expand Down

0 comments on commit 3be1b27

Please sign in to comment.