SURF SamenwerkingsBeheerSysteem (SBS)

SURF Research Access Management (SRAM) Platform

Overview Requirements

• Python 3.9.x
• MySQL v5.7.x or MariaDB 10.x
• Redis v6.x
• Yarn 1.x
• node
• libxmlsec1 (pre-1.3.0 see xmlsec/python-xmlsec#254 and see xmlsec/python-xmlsec#254 (comment) for a workaround)

Getting started

Server

Create a virtual environment and install the required python packages:

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r ./server/requirements/test.txt

Connect to your local mysql database: mysql -uroot and create the SBS database and user:

DROP DATABASE IF EXISTS sbs;
CREATE DATABASE sbs CHARACTER SET utf8mb4 DEFAULT CHARACTER SET utf8mb4;
DROP DATABASE IF EXISTS sbs_test;
CREATE DATABASE sbs_test CHARACTER SET utf8mb4 DEFAULT CHARACTER SET utf8mb4;
CREATE USER 'sbs'@'localhost' IDENTIFIED BY 'sbs';
GRANT ALL PRIVILEGES ON *.* TO 'sbs'@'localhost' WITH GRANT OPTION;

Ensure MySQL is running and run the Python server with the correct local environment settings:

PROFILE=local ALLOW_MOCK_USER_API=1 CONFIG=config/test_config.yml python -m server

With TESTING=1 no mails will be sent. If you do want to validate the mails you can run a fake smtp server with:

python -m smtpd -n -c DebuggingServer localhost:1025

If you want the emails to be opened in the browser when developing add the OPEN_MAIL_IN_BROWSER=1 to your environment. Or even better, use https://mailpit.axllent.org/ and capture all emails send. You can see all mails delivered at http://0.0.0.0:8025/ when you have the following configuration in you config file:

mail:
    host: 0.0.0.0
    port: 1025

Client

First install all dependencies with:

yarn install

The GUI can be started with:

cd client
yarn start

To create a GUI production build:

yarn build

To analyze the bundle:

yarn analyze

API

See the Swagger for the API documentation and data model. Or for local development http://localhost:8080/apidocs/

You can use the Swagger interface for testing or cUrl using the command line:

curl -H "Accept: application/json" -H "Content-type: application/json" -H "Authorization: bearer {api_key}" "http://localhost:8080/api/organisations/v1" | jq .
curl -H "Accept: application/json" -H "Content-type: application/json" -H "Authorization: bearer {api_key}" "http://localhost:8080/api/collaborations/v1/{co_identifier}" | jq .

Performance

When lazy loading other relationship data, the general rule-of-thumb leverages the best results:

Load child collections with selectinload and load single element relationships with joinedload.

Routes

To see all routes:

source .venv/bin/activate
cd server
CONFIG='config/test_config.yml' FLASK_APP='__main__.py' flask routes

Testing

To run the tests, you need a functioning Redis server on localhost:6379.

docker run -p 6379:6379 redis

To run all Python tests and validate syntax / formatting:

source .venv/bin/activate
cd server
pytest test
flake8 ./

To generate the coverage reports:

source .venv/bin/activate
cd server
pytest --cov=server --cov-report html:htmlcov test
open htmlcov/index.html

Within PyCharm you must mark the SBS/server/test directory as Test sources root in order to execute conftest.py before tests are run. See https://intellij-support.jetbrains.com/hc/en-us/community/posts/12897247432338-PyCharm-unable-to-find-fixtures-in-conftest-py

If you are getting errors in Pycharm when debugging, then have a look at https://youtrack.jetbrains.com/issue/PY-51495/PyCharm-debug-fails-upon-import-asyncio

To run all JavaScript tests:

cd client
yarn test

Or to run all the tests and do not watch - like CI:

cd client
CI=true yarn test

With the environment variable CONFIG=config/test_config.yml the test database is used. After you ran one or all of the tests the database is left with the test data seed. If you want to skip the login process when developing local then add the following to your environment: OPEN_MAIL_IN_BROWSER=1;PROFILE=local;CONFIG=config/test_config.yml;ALLOW_MOCK_USER_API=1

Deployment

See the https://github.com/SURFscz/SCZ-deploy project

Upgrade

See https://github.com/simion/pip-upgrader for upgrading automatically

source .venv/bin/activate
pip install pip-upgrader
cd server
pip-upgrade requirements/test.txt --dry-run

SURFSecureID

See the /config/saml_test configuration and the https://github.com/SURFscz/SCZ-deploy project

flask

To open a Flask terminal session:

source .venv/bin/activate
cd server
CONFIG='config/test_config.yml' FLASK_APP='__main__.py' flask shell

PyCharm

Because of eventlet the debugger in PyCharm sometimes crashes. Use the following environment properties to resolve this:

ALLOW_MOCK_USER_API=1;
CONFIG=config/acc_config.yml;
EVENTLET_HUB=poll;
PROFILE=local;
PYDEVD_USE_CYTHON=NO;
PYDEVD_USE_FRAME_EVAL=NO;
PYTHONUNBUFFERED=1

MySQL

When you encounter the following error after upgrading mySQL

E   ImportError: dlopen(/Users/okkeharsta/projects/SBS/.venv/lib/python3.9/site-packages/MySQLdb/_mysql.cpython-39-darwin.so, 0x0002): Library not loaded: /usr/local/opt/mysql/lib/libmysqlclient.23.dylib

then re-install the mysqlclient library with force

source .venv/bin/activate
pip uninstall mysqlclient
export MYSQLCLIENT_LDFLAGS=$(pkg-config --libs mysqlclient)
export MYSQLCLIENT_CFLAGS=$(pkg-config --cflags mysqlclient)
pip install mysqlclient --no-cache-dir

docker

For localhost deployment you can make use of docker. You can take existing docker-compose.yml as your starting point. The docker-compose file makes use of environment variables that you can adjust via a local .env file. To create your own .env file, copy the provided .env.example file:

$ cp env.example .env

and edit the values to some sane values.

Now adjust the contents of this .env file to match your desired configuration.

Then build the docker images and launch the containers:

$ export DOCKER_DEFAULT_PLATFORM=linux/amd64
$ docker compose build
$ docker compose up -d

Now open your browser at: http://localhost:8080