Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NGINX Unit #992

Open
wants to merge 41 commits into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
5e86b32
Crude beginnings
gsfr Nov 14, 2017
a3d1637
Make test scripts more POSIX-friendly
ambrussimon Nov 17, 2017
1989c67
Requirements overhaul
ambrussimon Nov 17, 2017
5da4994
WIP - test script reorg
ambrussimon Nov 17, 2017
9949a27
Implement lessons learned in reaper infra
ambrussimon Nov 17, 2017
1ba996b
Fix URL for integration tests
ambrussimon Nov 20, 2017
f2c7376
Fix dependency_links
ambrussimon Nov 20, 2017
8b4ae1f
Fix abao
ambrussimon Nov 20, 2017
6e80c04
Fix scr/src + coverage WIP
ambrussimon Nov 20, 2017
ec2c7e5
Speed up pylint
gsfr Nov 20, 2017
e6f54e5
Fix __pycache__ and coverage
gsfr Nov 21, 2017
f6ad22a
Reinstate Travis CI
gsfr Nov 21, 2017
bff12f2
Fix abao tests
ambrussimon Nov 21, 2017
13eb2ae
Coverage fix attempt
ambrussimon Nov 21, 2017
d2fac9c
Add author and maintainer info
gsfr Nov 22, 2017
c0942c4
Send CI image to Docker Hub
gsfr Nov 22, 2017
d50a82b
Declare docker as the only way to run core
gsfr Nov 22, 2017
e2f99ad
Streamline container and network naming
gsfr Nov 22, 2017
4593041
Remove unused files
gsfr Nov 22, 2017
3873957
Add coverage save handler
ambrussimon Nov 29, 2017
313667b
Remove abao
gsfr Dec 14, 2017
105c54f
Rebase, uwsgi cleanup, py-openssl fix, re-add inject_build_info
ambrussimon Dec 14, 2017
0548493
Fix build info
gsfr Dec 14, 2017
6ac0f7e
Purging py-openssl (core will be non-SSL)
gsfr Dec 15, 2017
e870763
Have Travis push to Docker Hub for this branch
gsfr Jan 30, 2018
4feab76
Run app as unprivileged user
gsfr Jan 11, 2018
35a0678
Substantial cleanup
gsfr Jan 11, 2018
9f4f367
Pin to nginx/unit/tag/0.4
gsfr Jan 15, 2018
a477bcc
Force remove of .coverage and htmlcov
gsfr Jan 15, 2018
0379847
Fix runtime coverage save (use /tmp for write permissions)
ambrussimon Jan 17, 2018
a5bb1d8
Add pre-test cleanup - fix pyc problems for good
ambrussimon Jan 22, 2018
8558808
Simplify testing and update test docs
ambrussimon Jan 25, 2018
4222879
Coverage save atexit trial
ambrussimon Jan 25, 2018
384c727
Cleanup
gsfr Jan 30, 2018
4348175
Bump base image version
gsfr Jan 30, 2018
bb73a8e
Only chown if needed
gsfr Jan 30, 2018
28ca260
Clean up bin
gsfr Jan 31, 2018
3462a9d
Rename database.py
gsfr Jan 31, 2018
8ac2791
Remove entrypoint
gsfr Jan 31, 2018
4030d21
Add nginx for dev
gsfr Mar 9, 2018
fe5dbfc
Add upgrade_schema to dev
gsfr Mar 10, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 6 additions & 13 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -1,15 +1,8 @@
.git
**/.DS_Store
**/.*sw[op]
**/*.py[co]
**/*.egg-info

# --- Below Should mirror .gitignore, without leading '/' --- #
.DS_Store
.*sw[op]
*.py[co]
*.egg-info
persistent
runtime
bootstrap.json
.cache
.coverage*
coverage.xml
htmlcov
node_modules
/.coverage*
/htmlcov
File renamed without changes.
File renamed without changes.
8 changes: 1 addition & 7 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -2,12 +2,6 @@
.*sw[op]
*.py[co]
*.egg-info
/persistent
/runtime
bootstrap.json
.cache

/.coverage*
coverage.xml
/htmlcov
node_modules/
/bin/accesslog.csv
48 changes: 31 additions & 17 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,26 +1,40 @@
sudo: required
dist: trusty
sudo: required

services:
- mongodb
- docker

env:
global:
secure: HELJx6WPr+W2S0FV47KkRdlS9NCqlMcdRMK8xWgRqqrEPv24KEvNnHxCy0tRbzITqadYtyvI1MtqtmpG04uty8Gpkc7w6L6LMJ/OuLG0gVX7AnaUovYTlY04m1/L9oyzOrTDXk5J/BKbcyiz7uJtkTc/A8MMZAFfZh7hmhLID78= # BUILD_TRIGGER_URL
global:
- DOCKER_DIR="$HOME/.cache/docker"
- secure: "HhT1TdJcpqys8juVMw/DIZeK7oD4595TEKH5KlowH7MvwwFAUyQFb5W63F8dgk7elvRG+3fmga/m1JfXO+Iu7PVD912eiNDagW9aB3CEl3Z8zg+JUL8IjpMCkyKQDyJMnfOkrzdxdaqfOK+WmF+13f2qBu9Kc7wdXuzgHQrg4+0=" # CI_REGISTRY_USER
- secure: "hh7VDZnkxgl/vqHtS4IpXfIAckKpVQvoCzNW7fstr5Mcu8KNiCWIPgObBRm+m13aqpcFTMWQ6lT2kzORz2wWRbDeVhI1eGWOJswGNHPHZLO0Jaei6yfY2nY2mpxZbl+vdg00jkN64mi1ab3e++QgeLFruW0gyNefXX7E5L/mHTs=" # CI_REGISTRY_PASS

cache:
directories:
- $DOCKER_DIR

before_install:
- sudo apt-get update
- sudo apt-get -y install docker-ce realpath
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASS

install:
- bin/install-ubuntu.sh
- tests/bin/setup-integration-tests-ubuntu.sh
install: true

script:
- SCITRAN_PERSISTENT_DB_PORT=27017 tests/bin/run-tests-ubuntu.sh
- test -f "$DOCKER_DIR/image.tar" && docker load -i "$DOCKER_DIR/image.tar" || true
- docker build -t core:build --target build .
- docker build -t core:dist --target dist --build-arg VCS_BRANCH="$TRAVIS_BRANCH" --build-arg VCS_COMMIT="$TRAVIS_COMMIT" .
- docker build -t core:dev --target dev .
- docker save -o "$DOCKER_DIR/image.tar" $(docker history -q core:build | grep -v '<missing>') $(docker history -q core:dist | grep -v '<missing>')
- ./tests/bin/docker-tests.sh --image core:dev

after_success:
- if [ "$TRAVIS_BRANCH" == "master" -o "$TRAVIS_EVENT_TYPE" == "pull_request" ]; then
bash <(curl -s https://codecov.io/bash) -cF python;
fi
- if [ "$TRAVIS_TAG" ]; then
./docker/build-trigger.sh Tag "$TRAVIS_TAG" "$BUILD_TRIGGER_URL";
fi
- if [ "$TRAVIS_EVENT_TYPE" == "push" -a "$TRAVIS_BRANCH" == "master" ]; then
./docker/build-trigger.sh Branch "$TRAVIS_BRANCH" "$BUILD_TRIGGER_URL";
fi
- if [ "$TRAVIS_TAG" ]; then
docker tag core:dist scitran/core:$TRAVIS_TAG;
docker push scitran/core:$TRAVIS_TAG;
fi
- if [ "$TRAVIS_EVENT_TYPE" == "push" -a "$TRAVIS_BRANCH" == "nginx-unit" ]; then
docker tag core:dev scitran/core:nginx-unit;
docker push scitran/core:nginx-unit;
fi
154 changes: 51 additions & 103 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,103 +1,51 @@
#
# Image used for hosting scitran core with uwsgi.
#
# Example usage is in README.md
#

FROM ubuntu:14.04


# Install pre-requisites
RUN apt-get update \
&& apt-get install -y \
build-essential \
ca-certificates curl \
libatlas3-base \
numactl \
python-dev \
python-pip \
libffi-dev \
libssl-dev \
libpcre3 \
libpcre3-dev \
git \
&& rm -rf /var/lib/apt/lists/* \
&& pip install -U pip


# Grab gosu for easy step-down from root in a docker-friendly manner
# https://github.com/tianon/gosu
#
# Alternate key servers are due to reliability issues with ha.pool.sks-keyservers.net
RUN curl -o /usr/local/bin/gosu -SL "https://github.com/tianon/gosu/releases/download/1.6/gosu-$(dpkg --print-architecture)" \
&& curl -o /tmp/gosu.asc -SL "https://github.com/tianon/gosu/releases/download/1.6/gosu-$(dpkg --print-architecture).asc" \
&& export GNUPGHOME="$(mktemp -d)" \
&& for server in $(shuf -e ha.pool.sks-keyservers.net \
hkp://p80.pool.sks-keyservers.net:80 \
keyserver.ubuntu.com \
hkp://keyserver.ubuntu.com:80 \
pgp.mit.edu) ; do \
gpg --keyserver "$server" --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4 && break || : ; \
done \
&& gpg --batch --verify /tmp/gosu.asc /usr/local/bin/gosu \
&& rm -r "$GNUPGHOME" /tmp/gosu.asc \
&& chmod +x /usr/local/bin/gosu


# Setup environment
WORKDIR /var/scitran

RUN mkdir -p \
/var/scitran/config \
/var/scitran/data \
/var/scitran/code/api \
/var/scitran/logs \
/var/scitran/keys

# Declaring a volume makes the intent to map externally explicit. This enables
# the contents to survive/persist across container versions, and easy access
# to the contents outside the container.
#
# Declaring the VOLUME in the Dockerfile guarantees the contents are empty
# for any new container that doesn't specify a volume map via 'docker run -v '
# or similar option.
#
VOLUME /var/scitran/keys
VOLUME /var/scitran/data
VOLUME /var/scitran/logs


# Install pip modules
#
# Split this out for better cache re-use.
#
COPY requirements.txt docker/requirements-docker.txt /var/scitran/code/api/

RUN pip install --upgrade pip wheel setuptools \
&& pip install -r /var/scitran/code/api/requirements-docker.txt \
&& pip install -r /var/scitran/code/api/requirements.txt

COPY tests /var/scitran/code/api/tests/
RUN bash -e -x /var/scitran/code/api/tests/bin/setup-integration-tests-ubuntu.sh


# Copy full repo
#
COPY . /var/scitran/code/api/

COPY docker/uwsgi-entrypoint.sh /var/scitran/
COPY docker/uwsgi-config.ini /var/scitran/config/



# Inject build information into image so the source of the container can be
# determined from within it.
ARG BRANCH_LABEL=NULL
ARG COMMIT_HASH=0
COPY docker/inject_build_info.sh /
RUN /inject_build_info.sh ${BRANCH_LABEL} ${COMMIT_HASH} \
&& rm /inject_build_info.sh


ENTRYPOINT ["/var/scitran/uwsgi-entrypoint.sh"]
CMD ["uwsgi", "--ini", "/var/scitran/config/uwsgi-config.ini", "--http", "0.0.0.0:8080", "--http-keepalive", "--so-keepalive", "--add-header", "Connection: Keep-Alive" ]
FROM python:2.7-alpine3.7 as build

RUN apk --no-cache add build-base curl

WORKDIR /src/nginx-unit

RUN curl -L https://github.com/nginx/unit/archive/0.4.tar.gz | tar xz --strip-components 1
RUN ./configure --prefix=/usr/local --modules=lib --state=/var/local/unit --pid=/var/unit.pid --log=/var/log/unit.log \
&& ./configure python \
&& make install


FROM python:2.7-alpine3.7 as dist

RUN apk --no-cache add git

COPY --from=build /usr/local/sbin/unitd /usr/local/sbin/unitd
COPY --from=build /usr/local/lib/python.unit.so /usr/local/lib/python.unit.so

EXPOSE 80 8088
VOLUME /data/db /data/persistent

WORKDIR /src/core
ENV SCITRAN_PERSISTENT_DATA_PATH=/data/persistent

COPY nginx-unit.json /var/local/unit/conf.json
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt

COPY . .
RUN pip install -e .

ARG VCS_BRANCH=NULL
ARG VCS_COMMIT=NULL
RUN ./bin/build_info.sh $VCS_BRANCH $VCS_COMMIT | tee /version.json

CMD ["unitd", "--control", "*:8088", "--no-daemon", "--log", "/dev/stdout"]


FROM dist as dev

EXPOSE 27017

RUN apk --no-cache add mongodb nginx
RUN mkdir /run/nginx

COPY nginx.conf /etc/nginx/nginx.conf

RUN pip install -r tests/requirements.txt

CMD ["./bin/dev+mongo.sh"]
19 changes: 1 addition & 18 deletions README.md
Original file line number Diff line number Diff line change
@@ -19,23 +19,6 @@ SciTran Core is a RESTful HTTP API, written in Python and backed by MongoDB. It


### Usage
**Currently Python 2 Only**

#### OSX
```
$ ./bin/run-dev-osx.sh --help
```

For the best experience, please upgrade to a recent version of bash.
```
brew install bash bash-completion
sudo dscl . -create /Users/$(whoami) UserShell /usr/local/bin/bash
```

#### Ubuntu
```
mkvirtualenv scitran-core
./bin/install-ubuntu.sh
uwsgi --http :8080 --master --wsgi-file bin/api.wsgi -H $VIRTUAL_ENV \
--env SCITRAN_PERSISTENT_DB_URI="mongodb://localhost:27017/scitran-core"
docker run -p 80:80 -e SCITRAN_CORE_DRONE_SECRET=secret scitran/core
```
44 changes: 0 additions & 44 deletions TESTING.md

This file was deleted.

3 changes: 3 additions & 0 deletions api/app.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
from .web import start

application = start.app_factory()
2 changes: 1 addition & 1 deletion api/web/start.py
Original file line number Diff line number Diff line change
@@ -16,7 +16,7 @@ def save_coverage(cov):
def start_coverage():
import coverage
print("Enabling code coverage")
cov = coverage.coverage(source=["api"], data_suffix="integration-tests")
cov = coverage.coverage(source=["api"], data_file="/tmp/.coverage.integration-tests")
cov.start()
atexit.register(save_coverage, cov)

4 changes: 0 additions & 4 deletions bin/api.wsgi

This file was deleted.

11 changes: 11 additions & 0 deletions bin/build_info.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
#!/usr/bin/env sh

set -eu

cat <<EOF
{
"branch": "$1",
"commit": "$2",
"timestamp": "$(date -u +"%Y-%m-%dT%H:%M:%SZ")"
}
EOF
2 changes: 2 additions & 0 deletions bin/oneoffs/cas_statistic.py → bin/cas_statistic.py
100644 → 100755
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
#!/usr/bin/env python

import os
import pymongo
from collections import Counter
File renamed without changes.
16 changes: 16 additions & 0 deletions bin/dev+mongo.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
#!/usr/bin/env sh

unset CDPATH
cd "$(dirname "$0")"

nginx

mongod &

if [ "$(stat -c %U:%G $SCITRAN_PERSISTENT_DATA_PATH)" != "nobody:nobody" ]; then
chown nobody:nobody -R $SCITRAN_PERSISTENT_DATA_PATH
fi

./db_upgrade.py upgrade_schema

exec unitd --control "*:8088" --no-daemon --log /dev/stdout
500 changes: 0 additions & 500 deletions bin/dicom_doctype.py

This file was deleted.

76 changes: 0 additions & 76 deletions bin/install-dev-osx.sh

This file was deleted.

24 changes: 0 additions & 24 deletions bin/install-ubuntu.sh

This file was deleted.

5 changes: 0 additions & 5 deletions bin/integrity_check.py
Original file line number Diff line number Diff line change
@@ -92,8 +92,3 @@ def session_length():
except Exception as e:
logging.exception('Main method failed...')
sys.exit(1)





145 changes: 0 additions & 145 deletions bin/load_users_drone_secret.py

This file was deleted.

4 changes: 3 additions & 1 deletion bin/log_csv.py → bin/log_export.py
100644 → 100755
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
#!/usr/bin/env python

# This implementation as of July 19 2017 has these resource utilizations of the mongodb container:
# - 2 million entries: 1.50 Gb
# - 3 million entries: 2.05 Gb
@@ -57,7 +59,7 @@ def download_large_csv(params):
csv_file.flush()
params['end_date'] = end_date


print "Encountered unicode errors and skipped {} entries".format(unicode_err_count)
csv_file.close()

79 changes: 0 additions & 79 deletions bin/oneoffs/load_external_data.py

This file was deleted.

194 changes: 0 additions & 194 deletions bin/run-dev-osx.sh

This file was deleted.

File renamed without changes.
29 changes: 0 additions & 29 deletions bootstrap.sample.json

This file was deleted.

61 changes: 0 additions & 61 deletions docker/README.md

This file was deleted.

28 changes: 0 additions & 28 deletions docker/bootstrap-accounts.sh

This file was deleted.

77 changes: 0 additions & 77 deletions docker/bootstrap-data.sh

This file was deleted.

31 changes: 0 additions & 31 deletions docker/build-trigger.sh

This file was deleted.

32 changes: 0 additions & 32 deletions docker/inject_build_info.sh

This file was deleted.

5 changes: 0 additions & 5 deletions docker/pymongo-cli.py

This file was deleted.

3 changes: 0 additions & 3 deletions docker/requirements-docker.txt

This file was deleted.

8 changes: 0 additions & 8 deletions docker/uwsgi-config.ini

This file was deleted.

48 changes: 0 additions & 48 deletions docker/uwsgi-entrypoint.sh

This file was deleted.

2 changes: 0 additions & 2 deletions logs/.gitignore

This file was deleted.

17 changes: 17 additions & 0 deletions nginx-unit.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"listeners": {
"*:8080": {
"application": "scitran-core"
}
},

"applications": {
"scitran-core": {
"type": "python",
"workers": 2,
"user": "nobody",
"path": "/src/core",
"module": "api.app"
}
}
}
15 changes: 15 additions & 0 deletions nginx.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
events {
}

http {
upstream unit_backend {
server 127.0.0.1:8080;
}

server {
location /api {
proxy_pass http://unit_backend;
proxy_set_header Host $host;
}
}
}
10 changes: 4 additions & 6 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -1,17 +1,15 @@
django>=1.11.0,<1.12.0
elasticsearch==5.3.0
enum==0.4.6
enum34==1.1.6
git+https://github.com/flywheel-io/gears.git@v0.1.4#egg=gears
jsonschema==2.6.0
Markdown==2.6.5
markdown==2.6.5
pymongo==3.2
pyOpenSSL >=17.1.0,<18.0
python-dateutil==2.4.2
pytz==2015.7
requests==2.9.1
rfc3987==1.3.4
strict-rfc3339==0.7
unicodecsv==0.9.0
uwsgi==2.0.13.1
webapp2==2.5.2
WebOb==1.5.1
git+https://github.com/flywheel-io/gears.git@v0.1.4#egg=gears
webob==1.5.1
22 changes: 22 additions & 0 deletions setup.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
from setuptools import setup, find_packages

requirements = open('requirements.txt').readlines()
install_requires = [r for r in requirements if not r.startswith('git+')]
dependency_links = [r for r in requirements if r.startswith('git+')]
tests_require = open('tests/requirements.txt').readlines()

setup(
name = 'core',
version = '1.0.0',
description = 'Scitran API Core',
author = 'Gunnar Schaefer',
author_email = 'gsfr@flywheel.io',
maintainer = 'Megan Henning, Ambrus Simon',
maintainer_email = 'meganhenning@flywheel.io, ambrussimon@invenshure.com',
url = 'https://github.com/scitran/core',
license = 'MIT',
packages = find_packages(),
dependency_links = dependency_links,
install_requires = install_requires,
tests_require = tests_require,
)
24 changes: 24 additions & 0 deletions tests/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
## Testing
Build scitran-core and run automated tests in a docker container:
```
./tests/bin/docker-tests.sh
```

* To skip building the image, use `--no-build` (`-B`)
* All tests (unit, integration and linting) are executed by default
* To pass any arguments to `py.test`, use `-- PYTEST_ARGS`
* To run only a subset of test, use the [keyword expression filter](https://docs.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests) `-k`
* To see `print` output during tests, increase verbosity with `-vvv`
* To get a debugger session on failures, use [`--pdb`](https://docs.pytest.org/en/latest/usage.html#dropping-to-pdb-python-debugger-on-failures)

See [py.test usage](https://docs.pytest.org/en/latest/usage.html) for more.

### Example
The most common use case is adding a new (still failing) test, and wanting to
* (re-)run it as fast as possible (`-B` and `-k foo`)
* see output from quick and dirty `print` statements in the test (`-vvv`)
* get into an interactive pdb session to inspect what went wrong (`--pdb`)

```
./tests/bin/docker-tests.sh -B -- -k foo -vvv --pdb
```
149 changes: 149 additions & 0 deletions tests/bin/docker-tests.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
#!/usr/bin/env sh

set -eu
unset CDPATH
cd "$( dirname "$0" )/../.."


USAGE="
Usage:
$0 [OPTION...] [-- PYTEST_ARGS...]
Build scitran/core image and run tests in a Docker container.
Also displays coverage report and saves HTML in htmlcov dir.
Options:
-h, --help Print this help and exit
-B, --no-build Skip rebuilding default Docker image
--image IMAGE Use custom Docker image
-- PYTEST_ARGS Arguments passed to py.test
"


main() {
local DOCKER_IMAGE=
local PYTEST_ARGS=

while [ $# -gt 0 ]; do
case "$1" in
-h|--help)
log "$USAGE"
exit 0
;;
-B|--no-build)
DOCKER_IMAGE="scitran/core:testing"
;;
--image)
DOCKER_IMAGE="$2"
shift
;;
--)
shift
PYTEST_ARGS="$@"
break
;;
*)
log "Invalid argument: $1"
log "$USAGE"
exit 1
;;
esac
shift
done

# Docker build
if [ -z "${DOCKER_IMAGE}" ]; then
log "Building scitran/core:testing ..."
docker build -t scitran/core:testing .
else
docker tag "$DOCKER_IMAGE" "scitran/core:testing"
fi

log "Cleaning pyc and previous coverage results ..."
# Run within container to avoid permission problems
docker run --rm \
--name core-test-cleanup \
--volume $(pwd):/src/core \
scitran/core:testing \
sh -c "
find . -type d -name __pycache__ -exec rm -rf {} \;;
find . -type f -name '*.pyc' -delete;
rm -rf .coverage htmlcov;
"

trap clean_up EXIT
docker network create core-test

# Launch core + mongo
docker run -d \
--name core-test-service \
--network core-test \
--volume $(pwd)/api:/src/core/api \
--volume $(pwd)/tests:/src/core/tests \
--env SCITRAN_CORE_DRONE_SECRET=secret \
--env SCITRAN_RUNTIME_COVERAGE=true \
--env SCITRAN_CORE_ACCESS_LOG_ENABLED=true \
scitran/core:testing

# Execute tests
docker run -it \
--name core-test-runner \
--network core-test \
--volume $(pwd)/api:/src/core/api \
--volume $(pwd)/tests:/src/core/tests \
--env SCITRAN_SITE_API_URL=http://core-test-service/api \
--env SCITRAN_CORE_DRONE_SECRET=secret \
--env SCITRAN_PERSISTENT_DB_URI=mongodb://core-test-service:27017/scitran \
--env SCITRAN_PERSISTENT_DB_LOG_URI=mongodb://core-test-service:27017/logs \
scitran/core:testing \
tests/bin/tests.sh -- $PYTEST_ARGS
}


clean_up() {
local TEST_RESULT_CODE=$?
set +e

log "INFO: Test return code = $TEST_RESULT_CODE"
if [ "${TEST_RESULT_CODE}" = "0" ]; then
log "INFO: Collecting coverage..."

# Copy unit test coverage
docker cp core-test-runner:/src/core/.coverage .coverage.unit-tests 2>/dev/null

# Save integration test coverage
docker wait $(docker stop core-test-service)
docker cp core-test-service:/tmp/.coverage.integration-tests .

# Combine unit/integ coverage and report/grenerate html
docker run --rm \
--name core-test-coverage \
--volume $(pwd):/src/core \
scitran/core:testing \
sh -c '
coverage combine;
coverage report --skip-covered --show-missing;
coverage html;
'
else
log "INFO: Printing container logs..."
docker logs core-test-service
log "ERROR: Test return code = $TEST_RESULT_CODE. Container logs printed above."
fi

# Spin down dependencies
docker rm --force --volumes core-test-runner
docker rm --force --volumes core-test-service
docker network rm core-test
exit $TEST_RESULT_CODE
}


log() {
printf "\n%s\n" "$@" >&2
}


main "$@"
82 changes: 0 additions & 82 deletions tests/bin/run-tests-docker.sh

This file was deleted.

191 changes: 0 additions & 191 deletions tests/bin/run-tests-ubuntu.sh

This file was deleted.

14 changes: 0 additions & 14 deletions tests/bin/setup-integration-tests-ubuntu.sh

This file was deleted.

75 changes: 75 additions & 0 deletions tests/bin/tests.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
#!/usr/bin/env sh

set -eu
unset CDPATH
cd "$( dirname "$0" )/../.."


USAGE="
Usage:
$0 [-- PYTEST_ARGS...]
Runs all tests (unit, integ and linting) if no options are provided.
Assumes running in a scitran/core:testing container or that core and all
of its dependencies are installed the same way as in the Dockerfile.
Options:
-h, --help Print this help and exit
-- PYTEST_ARGS Arguments passed to py.test
Envvars (required for integration tests):
SCITRAN_SITE_API_URL URI to a running core instance (including /api)
SCITRAN_CORE_DRONE_SECRET API shared secret
SCITRAN_PERSISTENT_DB_URI Mongo URI to the scitran DB
SCITRAN_PERSISTENT_DB_LOG_URI Mongo URI to the scitran log DB
"


main() {
export PYTHONDONTWRITEBYTECODE=1
local PYTEST_ARGS=

while [ $# -gt 0 ]; do
case "$1" in
-h|--help)
log "$USAGE"
exit 0
;;
--)
shift
PYTEST_ARGS="$@"
break
;;
*)
log "Invalid argument: $1"
log "$USAGE" >&2
exit 1
;;
esac
shift
done

log "Running unit tests ..."
py.test --cov=api --cov-report= tests/unit_tests/python $PYTEST_ARGS

log "Running integration tests ..."
py.test tests/integration_tests/python $PYTEST_ARGS

log "Running pylint ..."
# TODO Enable Refactor and Convention reports
# TODO Move --disable into rc
pylint --jobs=4 --reports=no --disable=C,R,W0312,W0141,W0110 api

# log "Running pep8 ..."
# pep8 --max-line-length=150 --ignore=E402 api
}


log() {
printf "\n%s\n" "$@" >&2
}


main "$@"
1 change: 0 additions & 1 deletion tests/integration_tests/.npmignore

This file was deleted.

1,463 changes: 0 additions & 1,463 deletions tests/integration_tests/abao/abao_test_hooks.js

This file was deleted.

433 changes: 0 additions & 433 deletions tests/integration_tests/abao/load_fixture.py

This file was deleted.

This file was deleted.

1 change: 0 additions & 1 deletion tests/integration_tests/abao/test_files/notes.txt

This file was deleted.

Binary file removed tests/integration_tests/abao/test_files/test-1.dcm
Binary file not shown.
12 changes: 0 additions & 12 deletions tests/integration_tests/package.json

This file was deleted.

4 changes: 2 additions & 2 deletions tests/integration_tests/python/test_upgrades.py
Original file line number Diff line number Diff line change
@@ -9,8 +9,8 @@
def database(mocker):
bin_path = os.path.join(os.getcwd(), 'bin')
mocker.patch('sys.path', [bin_path] + sys.path)
import database
return database
import db_upgrade
return db_upgrade


def test_42(data_builder, api_db, as_admin, database):
File renamed without changes.
32 changes: 16 additions & 16 deletions tests/unit_tests/python/test_db_upgrade.py
Original file line number Diff line number Diff line change
@@ -6,51 +6,51 @@

bin_path = os.path.join(os.getcwd(), "bin")
sys.path.insert(0, bin_path)
import database
import db_upgrade

from api import config

CDV = database.CURRENT_DATABASE_VERSION
CDV = db_upgrade.CURRENT_DATABASE_VERSION

def test_all_upgrade_scripts_exist():
for i in range(1, CDV):
script_name = 'upgrade_to_{}'.format(i)
assert hasattr(database, script_name)
assert hasattr(db_upgrade, script_name)

def test_CDV_was_bumped():
script_name = 'upgrade_to_{}'.format(CDV+1)
assert hasattr(database, script_name) is False
assert hasattr(db_upgrade, script_name) is False


@patch('api.config.get_version', Mock(return_value={'database': 5}))
def test_get_db_version_from_config():
assert database.get_db_version() == 5
assert db_upgrade.get_db_version() == 5


@pytest.fixture(scope='function')
def database_mock_setup():
def db_upgrade_mock_setup():
setattr(config.db.singletons, 'update_one', Mock())
for i in range(1, CDV):
script_name = 'upgrade_to_{}'.format(i)
setattr(database, script_name, Mock())
setattr(db_upgrade, script_name, Mock())

@patch('database.get_db_version', Mock(return_value=0))
def test_all_upgrade_scripts_ran(database_mock_setup):
@patch('db_upgrade.get_db_version', Mock(return_value=0))
def test_all_upgrade_scripts_ran(db_upgrade_mock_setup):
with pytest.raises(SystemExit):
database.upgrade_schema()
db_upgrade.upgrade_schema()
for i in range(1, CDV):
script_name = 'upgrade_to_{}'.format(i)
assert getattr(database, script_name).called
assert getattr(db_upgrade, script_name).called

@patch('database.get_db_version', Mock(return_value=CDV-4))
def test_necessary_upgrade_scripts_ran(database_mock_setup):
@patch('db_upgrade.get_db_version', Mock(return_value=CDV-4))
def test_necessary_upgrade_scripts_ran(db_upgrade_mock_setup):
with pytest.raises(SystemExit):
database.upgrade_schema()
db_upgrade.upgrade_schema()
# Assert the necessary scripts were called
for i in range(CDV-3, CDV):
script_name = 'upgrade_to_{}'.format(i)
assert getattr(database, script_name).called
assert getattr(db_upgrade, script_name).called
# But not the scripts before it
for i in range(1, CDV-4):
script_name = 'upgrade_to_{}'.format(i)
assert getattr(database, script_name).called is False
assert getattr(db_upgrade, script_name).called is False