Integration tests with Eoulsan

Integration tests with Eoulsan

Introduction

Publication of a new version of the application implies that the code must be checked and validated with unitary tests. Integration or functional tests are created to reinforce analysis with one or several data sets, but manual launching is slow and fastidious. We chose to automate this operation. The open source software Jenkins is installed on an Apache2 server, it can launch tests, compile results and save the history.

The integration test is implemented in the software Eoulsan 2.0 with test-ng library java. The principle is very simple: an application is running and the output files are compared to the expected files. Several comparisons are available:

• comparison of the contents, with a specific file comparator for fastq, sam, bam, or text and binary
• comparison of the size files: xls, pdf, html, ...
• check presence or absence of certain files
• possibility to exclude some comparison files.

Creating an integration test

To create an integration test, we should create a global configuration file and a directory per test. Eoulsan launches a set of integration tests and indicates an application (the last committed version for example) as following :

• with a command line Eoulsan (example to run a set of tests):

$EOULSAN_PATH/eoulsan.sh it -testconf /path/to/configuration/file/it/it_global.conf -exec /path/to/executable/to/test/eoulsan-1.3-SNAPSHOT -f  /path/to/file/with/list/testname.txt

• with Maven, like in job defined in Jenkins (example to run only one test):

mvn clean install -Dit.conf.path=/path/to/configuration/file/it/it_global.conf -Dit.application.path=./targetdist/eoulsan-1.3-SNAPSHOT -Dit.test.name=testname

The integration test works with a very simple tree view:

$DIR/it_global.conf
$DIR/test/000001_<test_name>/
$DIR/output/<application_version_timestamp>
$DIR/log/

The it_global.conf is a global configuration file which defines main path directories and common options for tests. Example of it_global.conf:

##########################
## Integration tests configuration for eoulsan analysis.

## General parameters
# input.directory=${DIR_IT_PATH}input

log.directory=${DIR_IT_PATH}/log

tests.directory=${DIR_IT_PATH}/tests

output.analysis.directory=${DIR_IT_PATH}/output

# Parameters can be override by test.conf
command.to.launch.application=${application.path}/eoulsan.sh -conf eoulsan.conf -log eoulsan.log -loglevel all exec param.xml design.txt
command.to.get.application.version=${application.path}/eoulsan.sh -version | cut -d ' ' -f 1,3 | tr ' ' '_'

# Parameters completed by test.conf
files.to.compare=**/*global.txt
excluded.files.to.compare=

# if IT succeeded, delete file matching on pattern(s)
success.it.delete.file=false
files.to.remove=**/*.sam **/*.fq **/expression*.txt

difference.length.percent.in.comparaison.accepted=0.02

# pre / post script execute for each test
# pretreatment script : before script from test.conf
pre.global.script=${SCRIPTS_TEST_PATH}/pre_global_script_example.sh

# posttreatment script : after script from test.conf
post.global.script=${SCRIPTS_TEST_PATH}/post_global_script_example.py

# Set environment variables
# syntax setenv.VARIABLE_NAME=VALUE
env.var.EOULSAN_VERSION=`${application.path}/eoulsan.sh -version | cut -d ' ' -f 1,3 | tr ' ' '_'`
env.var.DIR_IT_PATH=`pwd -P`
env.var.PROGRAMMES_PATH=`pwd -P`/bin/
env.var.SCRIPTS_TEST_PATH=`pwd -P`/scripts

The test folder includes:

• a mandatory test.conf configuration file, if the file name is different, the test is skipped.
• all input files needed to run the application,
• a directory expected_<version-number> with all expected files to compare, they can be generated with a referenced version on the application or the script (if this directory is missing, the execution stops).

For each execution of the test produce $DIR/output/<application_version_timestamp>and one log file per execution.

Note: It is possible to run additional shell scripts before and/or after the application. If they are set in the global configuration file, the scripts will relate to all executed tests.