Skip to content

Latest commit

 

History

History
250 lines (186 loc) · 8.26 KB

development.md

File metadata and controls

250 lines (186 loc) · 8.26 KB

DiffKemp development

This document will guide you through DiffKemp development. It contains information about:

Development environment

For developing DiffKemp, you can use:

Nix

We provide Nix flakes for building and testing DiffKemp.

First, it is necessary to install Nix and enable flakes:

mkdir -p ~/.config/nix
echo "experimental-features = nix-command flakes" >> ~/.config/nix/nix.conf

Then, building DiffKemp is as simple as running one of the following commands:

nix build                    # <- uses latest LLVM
nix build .#diffkemp-llvm14

This will create a new folder result/ containing the pre-built DiffKemp which can be then executed by:

result/bin/diffkemp ...

Tip

Rather than building DiffKemp with Nix, it is better to use Nix development environment described below, because the DiffKemp build created by Nix has following drawbacks:

  • Currently it is not possible to run view subcommand (the result viewer).
  • You still need to have some dependencies installed on your system to be able to use it (clang, llvm, and dependencies specific to the project you want to analyse).

Nix as development environment

It is also possible to use Nix as a development environment:

nix develop                   # <- uses latest LLVM
nix develop .#diffkemp-llvm14

This will enter a development shell with all DiffKemp dependencies pre-installed. You can then follow the standard build instructions to build and install DiffKemp. The only difference is that it is not possible to run pip install inside Nix shell (because of the way Nix works) and it is necessary to use the built-in setuptoolsShellHook function instead.

We also provide a special Nix environment for retrieving and preparing kernel versions necessary for running regression tests (nix develop .#test-kernel-buildenv).

Local development environment

You can also develop DiffKemp directly. For this you need to install the necessary dependencies to your system and then build DiffKemp.

Docker

Caution

The docker container is not currently maintained and it is recommended to use Nix flake instead.

We also provide development container image prepared that can be retrieved from DockerHub: https://hub.docker.com/r/viktormalik/diffkemp-devel/

After that, the container can be run using

docker/diffkemp-devel/run-container.py [--llvm-version LLVM_VERSION]
                                       [--build-dir BUILD_DIR]
                                       [--diffkemp-dir DIFFKEMP_DIR]
                                       [--image IMAGE]

The script mounts the current directory as a volume inside the container. Then it automatically builds SimpLL in BUILD_DIR (default is "build") using LLVM_VERSION (default is the newest supported version) and installs DiffKemp.

If running multiple containers at the same time, you need to specify a unique BUILD_DIR for each one.

If running the container from a different directory than the root DiffKemp directory, you need to specify where DiffKemp is located using the --diffkemp-dir option.

By default, the DockerHub image is used, but a custom image may be set using the --image option.

Build

To build DiffKemp, use the following commands:

cmake -S . -B build -GNinja -DBUILD_VIEWER=On
ninja -C build
pip install -e . # In case of Nix use setuptoolsShellHook instead
  • -DBUILD_VIEWER=On: This flag will install packages and build the result viewer. It will try to install the packages on every cmake run, so you may want to use -DBUILD_VIEWER=Off instead.

If you make changes to the SimpLL library, you will need to rebuild it by running ninja -C build. We are using CFFI for accessing the library from the Python. The CFFI is not rebuilt by default, so even if you rebuild the library, the changes will not be visible when running diffkemp compare .... Currently, you have multiple options to overcome this:

  1. Use diffkemp compare --disable-simpll-ffi ... which will run SimpLL through a binary instead of the CFFI.
  2. On some OS distributions, it is enough to run pip install -e . again.
  3. Another option is to to run rm build/_simpll.abi3.so and then run cmake again with -DSIMPLL_REBUILD_BINDINGS=On flag. After this, when you make changes to the SimpLL library, running ninja -C build should be enough and it should rebuild the CFFI, and the changes should be visible when running diffkemp compare .... (Note: This does not seems to be working in the Nix development environment, so use option 1 instead.)

Coding style

We require that the code is according to a certain coding style, which can be checked with the following tools:

  • For Python code: flake8 diffkemp tests.
  • For C++ code: it is necessary to have clang-format installed and the coding style can be checked with tools/check-clang-format.sh -d (it can be also automatically fixed by running tools/check-clang-format.sh -di).
  • For JavaScript code: npm --prefix view run lint.

Tests

The project contains multiple tests:

Python tests

The tests use pytest and can be run by:

pytest tests

There are 2 types of tests:

  • Unit tests (located in tests/unit_tests/)
  • Regression tests (located in tests/regression/)

The tests require the sources of the following kernel versions to be stored and configured in kernel/linux-{version} directories:

  • 3.10 (upstream kernel)
  • 4.11 (upstream kernel)
  • 3.10.0-514.el7 (CentOS 7.3 kernel)
  • 3.10.0-693.el7 (CentOS 7.4 kernel)
  • 3.10.0-862.el7 (CentOS 7.5 kernel)
  • 3.10.0-957.el7 (CentOS 7.6 kernel)
  • 4.18.0-80.el8 (RHEL 8.0 kernel)
  • 4.18.0-147.el8 (RHEL 8.1 kernel)
  • 4.18.0-193.el8 (RHEL 8.2 kernel)
  • 4.18.0-240.el8 (RHEL 8.3 kernel)

The required configuration of each kernel can be done by running:

make prepare
make modules_prepare

The rhel-kernel-get script can also be used to download and configure the aforementioned kernels.

Tip

Since CentOS 7 kernels require a rather old GCC 7, the most convenient way to download the kernels is to use the prepared Nix environment by running

nix develop .#test-kernel-buildenv

and using rhel-kernel-get inside the environment to retrieve the above kernels.

Tests for the SimpLL library

Tests are located in tests/unit_tests/simpll/ directory and they can be run by:

ninja -C build test

Tests for the result viewer

The result viewer contains unit tests and integration tests located in view/src/tests/ directory and they can be run by:

npm --prefix view test -- --watchAll
npm --prefix view run cypress:run

Tools for performing experiments

We have some tools which can be handy when developing DiffKemp:

Useful links