Skip to content

Commit

Permalink
[dev] Standardize e2e tests (#915)
Browse files Browse the repository at this point in the history
* Standardize e2e tests

* Ensure dss and monitoring images are current

* Fix container name in local log collection
  • Loading branch information
BenjaminPelletier authored Dec 6, 2022
1 parent 72b8b27 commit 37fbff5
Show file tree
Hide file tree
Showing 12 changed files with 190 additions and 243 deletions.
9 changes: 5 additions & 4 deletions .github/workflows/CI.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,13 @@ Before a pull request can be merged into the master branch, it must pass all aut

### Build `monitoring` image (`make build-monitoring`)

### End-to-end test (`make test-e2e`)
### Tear down any pre-existing local DSS instance (`make down-locally`)

Steps:
### Start local DSS instance (`make start-locally`)

### Probe local DSS instance (`make probe-locally`)

* `make start-locally` (build/dev/run_locally.sh)
* Run pytest in monitoring/prober (in `monitoring` container)
### Bring down local DSS instance (`make down-locally`)

## `monitoring` tests (`make check-monitoring`)

Expand Down
10 changes: 8 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,8 +58,14 @@ jobs:
run: make build-dss
- name: Build monitoring image
run: make build-monitoring
- name: End-to-end test
run: make test-e2e
- name: Tear down any pre-existing local DSS instance
run: make down-locally
- name: Start local DSS instance
run: make start-locally
- name: Probe local DSS instance
run: make probe-locally
- name: Bring down local DSS instance
run: make down-locally

monitoring-tests:
name: monitoring tests
Expand Down
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,6 @@ coverage.xml
.pytest_cache/
e2e_test_result
test_result
probe_local_instance_test_result.xml

# Translations
*.mo
Expand Down
36 changes: 33 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,10 @@ shell-lint:
go-lint:
echo "===== Checking Go lint (except for *.gen.go files) =====" && docker run --rm -v $(CURDIR):/dss -w /dss golangci/golangci-lint:v1.50.1 golangci-lint run --timeout 5m --skip-dirs /dss/build/workspace --skip-files '.*\.gen\.go' -v -E gofmt,bodyclose,rowserrcheck,misspell,golint,staticcheck,vet

# This mirrors the hygiene-tests continuous integration workflow job (.github/workflows/ci.yml)
.PHONY: hygiene-tests
hygiene-tests: python-lint check-hygiene validate-uss-qualifier-docs shell-lint go-lint

# --- Targets to autogenerate Go code for OpenAPI-defined interfaces ---
.PHONY: apis
apis: example_apis dummy_oauth_api dss_apis
Expand Down Expand Up @@ -130,18 +134,36 @@ build-monitoring:
cd monitoring && make build

.PHONY: test-e2e
test-e2e:
test/docker_e2e.sh
test-e2e: down-locally build-dss build-monitoring start-locally probe-locally collect-local-logs down-locally

tag:
scripts/tag.sh $(UPSTREAM_OWNER)/dss/v$(VERSION)

.PHONY: start-locally
start-locally:
build/dev/run_locally.sh
build/dev/run_locally.sh up -d
build/dev/wait_for_local_dss.sh

.PHONY: probe-locally
probe-locally:
monitoring/prober/run_locally.sh

.PHONY: collect-local-logs
collect-local-logs:
docker logs dss_sandbox_local-dss-core-service_1 2> core-service-for-testing.log

.PHONY: stop-locally
stop-locally:
build/dev/run_locally.sh stop

.PHONY: down-locally
down-locally:
build/dev/run_locally.sh down

# This mirrors the dss-tests continuous integration workflow job (.github/workflows/ci.yml)
.PHONY: dss-tests
dss-tests: evaluate-tanka test-go-units test-go-units-crdb build-dss build-monitoring down-locally start-locally probe-locally collect-local-logs down-locally

.PHONY: check-monitoring
check-monitoring:
cd monitoring && make test
Expand All @@ -150,3 +172,11 @@ check-monitoring:
evaluate-tanka:
docker container run -v $(CURDIR)/build/jsonnetfile.json:/build/jsonnetfile.json -v $(CURDIR)/build/deploy:/build/deploy grafana/tanka show --dangerous-allow-redirect /build/deploy/examples/minimum
docker container run -v $(CURDIR)/build/jsonnetfile.json:/build/jsonnetfile.json -v $(CURDIR)/build/deploy:/build/deploy grafana/tanka show --dangerous-allow-redirect /build/deploy/examples/schema_manager

# This mirrors the monitoring-tests continuous integration workflow job (.github/workflows/ci.yml)
.PHONY: monitoring-tests
monitoring-tests: check-monitoring

# This reproduces the entire continuous integration workflow (.github/workflows/ci.yml)
.PHONY: presubmit
presubmit: hygiene-tests dss-tests monitoring-tests
1 change: 0 additions & 1 deletion build/deploy/db_schemas/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,5 +19,4 @@ places:
* [DSS main.jsonnet](../examples/minimum/main.jsonnet)
* [Schema manager main.jsonnet](../examples/schema_manager/main.jsonnet)
* scd_ or rid_ bootstrapper.sh in [dev/startup](../../dev/startup)
* [docker_e2e.sh](../../../test/docker_e2e.sh)
* /pkg/{rid|scd}/store/cockroach/store.go
53 changes: 0 additions & 53 deletions build/dev/probe_local_instance.sh

This file was deleted.

5 changes: 1 addition & 4 deletions build/dev/standalone_instance.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,10 +130,7 @@ target version parameter to `migrate_local_db.sh`.

### Prober

* `probe_local_instance.sh` runs the end-to-end [prober](../../monitoring/prober)
integration test, similar to [docker_e2e.sh](../../test/docker_e2e.sh), but using
the DSS instance already deployed locally instead of also deploying a local
instance as docker_e2e.sh does
* `monitoring/prober/run_locally.sh` runs the [prober](../../monitoring/prober) integration tests using the DSS instance already deployed locally via [`run_locally.sh`](run_locally.sh)

### RID

Expand Down
47 changes: 47 additions & 0 deletions build/dev/wait_for_local_dss.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
#!/bin/bash

OAUTH_CONTAINER="dss_sandbox_local-dss-dummy-oauth_1"
CORE_SERVICE_CONTAINER="dss_sandbox_local-dss-core-service_1"
declare -a localhost_containers=("$OAUTH_CONTAINER" "$CORE_SERVICE_CONTAINER")

for container_name in "${localhost_containers[@]}"; do
last_message=""
while true; do
if [ "$( docker container inspect -f '{{.State.Status}}' "${container_name}" 2>/dev/null)" = "running" ]; then
break
fi
new_message="Waiting for ${container_name} container to start..."
if [ "${new_message}" = "${last_message}" ]; then
printf "."
else
printf '..%s..' "${new_message}"
last_message="${new_message}"
fi
sleep 3
done
if [ -n "${last_message}" ]; then
echo ""
fi
done

last_message=""
while true; do
health_status="$( docker container inspect -f '{{.State.Health.Status}}' "${CORE_SERVICE_CONTAINER}" )"
if [ "${health_status}" = "healthy" ]; then
break
else
new_message="Waiting for ${CORE_SERVICE_CONTAINER} to be available (currently ${health_status})..."
if [ "${new_message}" = "${last_message}" ]; then
printf "."
else
printf '..%s..' "${new_message}"
last_message="${new_message}"
fi
sleep 3
fi
done
if [ -n "${last_message}" ]; then
echo ""
fi

echo "Local DSS instance is now available."
24 changes: 15 additions & 9 deletions monitoring/prober/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

This directory contains integration tests that can be run against a live DSS
instance. These integration tests are also run as part of
[docker_e2e.sh](../../test/docker_e2e.sh) which is a self-contained system
[`make test-e2e`](../../Makefile) which is a self-contained system
integration test suite.

## Authorization
Expand All @@ -27,19 +27,19 @@ the tests that depend on that authorization will be skipped.
Example: `--rid-auth "UsernamePassword(https://example.com/token, username=uss1,
password=uss1, client_id=uss1)"`

## Running pytest via Docker
## Running prober via Docker
This approach takes slightly longer to execute due to building the prober image,
but it requires no setup and generally obtains more reproducible results than
running locally. From the root of this repo:

First, build the monitoring image:

(from [`monitoring`](../) working directory)
(from the repo root)
```shell script
docker image build . -t interuss/monitoring
monitoring/build.sh
```

...then run it:
...then run prober:

```shell script
docker run --rm interuss/monitoring \
Expand All @@ -51,7 +51,7 @@ docker run --rm interuss/monitoring \
[--scd-auth2 <SPEC>]
```

## Running pytest locally
## Running prober locally
This approach will save the time of building the prober Docker container, but
requires more setup and may not be fully-reproducible due to system differences
in Python, etc.
Expand All @@ -75,19 +75,25 @@ pytest \
-vv .
```

## Running prober on a local DSS deployment

A local DSS ecosystem can be deployed with `make start-locally` from the repo
root. Then, prober can be run (from the repo root) with
`monitoring/prober/run_locally.sh`

## Specifying specific tests

### All the tests in a folder

E.g.: `test/docker_e2e.sh scd/`
E.g.: `monitoring/prober/run_locally.sh scd/`

### All the tests in a file

`test/docker_e2e.sh scd/test_constraint_simple.py`
`monitoring/prober/run_locally.sh scd/test_constraint_simple.py`

### A specific test

`test/docker_e2e.sh scd/test_constraint_simple.py::test_ensure_clean_workspace`
`monitoring/prober/run_locally.sh scd/test_constraint_simple.py::test_ensure_clean_workspace`

## Writing prober tests
This project strives to make prober tests as easy as possible to create in order
Expand Down
60 changes: 60 additions & 0 deletions monitoring/prober/run_locally.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
#!/usr/bin/env bash

set -eo pipefail
set -x

# Find and change to repo root directory
OS=$(uname)
if [[ "$OS" == "Darwin" ]]; then
# OSX uses BSD readlink
BASEDIR="$(dirname "$0")"
else
BASEDIR=$(readlink -e "$(dirname "$0")")
fi
cd "${BASEDIR}/../.." || exit 1

CORE_SERVICE_CONTAINER="dss_sandbox_local-dss-core-service_1"
OAUTH_CONTAINER="dss_sandbox_local-dss-dummy-oauth_1"
declare -a localhost_containers=("$CORE_SERVICE_CONTAINER" "$OAUTH_CONTAINER")

for container_name in "${localhost_containers[@]}"; do
if [ "$( docker container inspect -f '{{.State.Status}}' "$container_name" )" == "running" ]; then
echo "$container_name available!"
else
echo '#########################################################################'
echo '## Prerequisite to run this command is: ##'
echo '## Local DSS instance + Dummy OAuth server (/build/dev/run_locally.sh) ##'
echo '#########################################################################'
echo "Error: $container_name not running. Execute 'build/dev/run_locally.sh up' before running monitoring/prober/run_locally.sh";
exit 1;
fi
done

echo "Re/Create e2e_test_result file"
RESULTFILE="$(pwd)/e2e_test_result"
touch "${RESULTFILE}"
cat /dev/null > "${RESULTFILE}"

if ! docker run --link "$OAUTH_CONTAINER":oauth \
--link "$CORE_SERVICE_CONTAINER":core-service \
--network dss_sandbox_default \
-v "${RESULTFILE}:/app/test_result" \
-w /app/monitoring/prober \
interuss/monitoring \
pytest \
"${1:-.}" \
-rsx \
--junitxml=/app/test_result \
--dss-endpoint http://core-service:8082 \
--rid-auth "DummyOAuth(http://oauth:8085/token,sub=fake_uss)" \
--rid-v2-auth "DummyOAuth(http://oauth:8085/token,sub=fake_uss)" \
--scd-auth1 "DummyOAuth(http://oauth:8085/token,sub=fake_uss)" \
--scd-auth2 "DummyOAuth(http://oauth:8085/token,sub=fake_uss2)" \
--scd-api-version 1.0.0; then

if [ "$CI" == "true" ]; then
echo "=== END OF TEST RESULTS ==="
echo "Dumping core-service logs"
docker logs "$CORE_SERVICE_CONTAINER"
fi
fi
Loading

0 comments on commit 37fbff5

Please sign in to comment.