Skip to content
This repository has been archived by the owner on Nov 30, 2022. It is now read-only.

Development Environment Setup

Jump to bottom
Alexander Dubrawski edited this page Nov 16, 2020 · 23 revisions

Table of content

You have two options to set up the development environment:

  1. Use the install_dependencies_development script
  2. Install everything that needed manually

Get the Cockpit Source Code

  1. Clone the repository: git clone https://github.com/hyrise/Cockpit.git
  2. Enter the Cockpit directory: cd Cockpit

Get the Hyrise Database ready

Please check out the following guide:

https://github.com/hyrise/Cockpit/wiki/Hyrise-Things

Install script

The install script will set up the development environment. Simply run:

./install_dependencies_development.sh

The script will install all packages (python, node), influxdb and set up a pipenv (https://github.com/pypa/pipenv).

Install manually

First of all, you need to set up an influx database:

macOS 
brew install influxdb
Ubuntu 
wget -qO- https://repos.influxdata.com/influxdb.key | sudo apt-key add -
source /etc/lsb-release
echo "deb https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
sudo apt-get update && sudo apt-get install -y influxdb
sudo service influxdb start

Backend setup

You can install all needed packages with the following commands:

macOS 
brew install python3.8 \
    && brew install libpq \
    && brew install postgresql
Ubuntu 
# Install dependencies
sudo apt-get update
sudo apt-get install -y \
        python3.8 \
        python3.8-dev \
        python3.8-distutils \
        python3-pip \
        wget \
        curl \
        git

We are using pipenv as a developer environment. To install pipenv run the following commands:

python3.8 -m pip install pipenv \
    && pipenv install --python 3.8 \

Developer setup

Run the following commands to bootstrap your developer environment.

pipenv sync --dev

# Please make sure you enable the pre-commit hooks:
pipenv run pre-commit install

Frontend setup

The frontend part uses the Vue Composition API (https://composition-api.vuejs.org/), TypeScript (https://www.typescriptlang.org/) and Vuetify as Material Design Framework (https://vuetifyjs.com/de-DE/).

Instal npm

To get the frontend running we first need to install npm:

macOS 
brew install node
Ubuntu 
curl -sL https://deb.nodesource.com/setup_current.x | sudo -E bash -
sudo apt-get install -y nodejs

Initial Setup

Before starting the server, you need to locally install all dependencies of the project by running:

cd hyrisecockpit/frontend && npm install && npm audit fix

To connect to a running backend, you need to create a .env file in the frontend top level folder with your custom configuration. An example structure can be found in .env.example. To reset your config to the default connection or create a .env with the default values just run npm run reset:config.

Using the development environment

Backend

Pipenv

For the backend development (all python code) we are using pipenv (https://github.com/pypa/pipenv). pipenv aims to combine Pipfile, pip and virtualenv into one command on the command-line. The virtualenv directory typically gets placed in ~/.local/share/virtualenvs/XXX, with XXX being a hash of the path of the project directory. This is different from virtualenv, where the directory is typically in the current working directory. pipenv is meant to be used when developing Python applications (as opposed to libraries).

Access virtual environment

You ca access the virtual environment created by pipenv via the command pipenv shell. In this environment you can run the console scripts from the setup.py file to start the components. You can run (by just typing in the command line):

  • cockpit-backend to start the WSGI component (Flask).
  • cockpit-manager to start the manager component.
  • cockpit-generator to start the generator.
  • cockpit to start all backend components (WSGI, manager, generator) managed by on process.

Environment variables

You can create a .env to set your own environment variables. For example you could change the Port of the backend. You can have a look at the .env.example. To see the default values have a look at hyrisecockpit.settings.py

Install packages

You can install a python package via pipenv install. Let's say you want to install psycopg2. Simply run:

pipenv install psycopg2

After that you need to rebuild the requirements files:

pipenv run pipenv_to_requirements -o requirements.txt -f \
    && pipenv run pipenv_to_requirements -d requirements-dev.txt -f

If you want to install a package for development, let's say HalloWorld please run:

pipenv install --dev HalloWorld

After that, you need to rebuild the requirements files.

If you want to install a hook, please set a fixed version. For example, if you want to install mypy please run:

pipenv install --dev mypy==0.782

After that, you need to rebuild the requirements files.

Update Packages

If a package was installed by another PR and this PR is now merged, please make sure to update your packages. For that please run:

pipenv sync --dev

If you are switching to a PR where new packages were installed (for example they are not installed in dev and you switch to branch blabla) you need to run pipenv sync --dev every time you switch to blabla or back to dev

Hooks

If you want to run the pre-commit hooks manually without commiting, simply run:

pipenv run pre-commit

The hooks will run on all files that you added via git add.

This will trigger all configured pre-commit hooks, most notably:

  • Python code formatter black. Blackened code looks the same regardless of the project you're reading. Formatting becomes transparent after a while and you can focus on the content instead.
  • Python utility flake8, with plugins pep8-naming and flake8-bugbear.
  • Python utility bandit, a tool designed to find common security issues in Python code.
Failing hooks

If some hooks fail, you will not be able to commit. You may bypass this by using the git commit --no-verify, but it's a good practice to avoid bypassing it. Instead, add the edits made by e.g. black to the staging area, and commit with the pre-commit hooks passing. Some hooks, however, require you to change files manually, e.g. flake8 warning you of an unused variable.

Running a specific hook

You can run a single hook by running:

pipenv run pre-commit hook-id

The id of the hook can be found in the .pre-commit-config.yaml file.

Running hooks on all files

You can run all the pre-commit hooks on all files by running:

pipenv run pre-commit run --all-files

Tests

You can run all tests with:

pipenv run pytest

System tests

For a detailed explanation of how to run system tests please have a look at https://github.com/hyrise/Cockpit/wiki/System-Tests .

Frontend

Development Server

To start the server for development with hot-reloads, run:

cd hyrisecockpit/frontend && npm run serve

Linting

To automatically lint all files, run:

npm run lint

Testing

The frontend uses Cypress (https://www.cypress.io/) for UI/E2E-Testing. To start all tests with UI, run:

npm run test:e2e

To disable UI and execute all tests in headless mode, run: npm run test:e2e:headless All of the tests are using a backend stub with mocked responses. This backend stub can also be used for development purposes by mocking all network requests in a local node server. To start this test server, execute the following steps:

cd Cockpit/hyrisecockpit/frontend/devServer

npm install

npm start

The server runs at http://127.0.0.1:3000/ per default. To connect to this server in your frontend app, replace the backendUrl in the top level config.ts. To run all tests against this server execute npm run test:e2e:stubless or test:e2e:headless:stubless. If you have some problems try to run npm run start:dev:ci & npm run test:e2e:headless:stubless to execute all tests, or npm run start:dev:ci & npm run test:e2e:spec:headless "tests/e2e/specs/workloadGeneration/workloadGeneration.spec.ts" to execute a specific test (in this case workloadGeneration.spec.ts).

Quick Fix

Backend

If you change something in the setup.py for example add a new console script please run:

  1. pipenv --rm
  2. pipenv --three install
  3. pipenv sync --dev
Clone this wiki locally