This is test framework for executing automated and manual tests.
It allows to write the following set of tests
- Unit tests - automated short-time tests for checking validity of algorithms
- Performance tests - automated single-core tests for checking and comparing the performance of different implementations of the same algorithm or how different parameters do affect the execution of the same algorithm.
- Manual tests - tests that can not be automated but can be simly launched and performed 'by hand'.
The build and correct unit test execution has been confirmed for following platforms:
- FreeBSD
- GNU/Linux
- OpenBSD
- Windows 32-bit
- Windows 64-bit
The following packages need to be installed for building:
- gcc >= 4.9
- make >= 4.0
To build the library, perform the following commands:
make config # Configure the build
make fetch # Fetch dependencies from Git repository
make
sudo make install
To get more build options, run:
make help
To uninstall library, simply issue:
make uninstall
To clean all binary files, run:
make clean
To clean the whole project tree including configuration files, run:
make prune
To compile compile the object file for future linking, perform the following commands:
make config # Configure the build
make fetch # Fetch dependencies from Git repository
make compile
These commands will create the .build/lsp-test-fw/lsp-test-fw.o
file by default
which can be linked to your project.
To get more options, run:
make help
To build framework with demo tests, issue the following command:
make config TEST=1 # Configure the build
make fetch # Fetch dependencies from Git repository
make compile
To build application which will allow to execute tests, issue the following command:
make config TEST=1 # Configure the build
make fetch # Fetch dependencies from Git repository
make
These commands will create the .build/lsp-test-fw/lsp-test-fw-test
executable.
To get more options, run:
make help
Manual tests are semi-automated or not automated at all. You may launch these tests to perform manual interaction with system or for retrieving some text data for further processing.
Unit tests are completely automated and check that the produced result is correct for the specific input data. By default unit tests are launched in separate processes and utilize all CPU cores for maximum performance. The status of each unit test is collected during execution and list of all failed tests are printed after completion.
Performance tests measure performance of some function/module under different conditions, gather final statistics and output them in a table format. These tests are very useful for measuring single-core performance of different modules and functions and performing code optimizations.
After build, we can launch the test binary by issuing command:
.build/lsp-test-fw/lsp-test-fw-test
The executable file provides simple command-line interface, so here's the full usage:
USAGE: {utest|ptest|mtest} [args...] [test name...]
First argument:
utest Unit testing subsystem
ptest Performance testing subsystem
mtest Manual testing subsystem
Additional arguments:
-a, --args [args...] Pass arguments to test
-d, --debug Disable time restrictions for unit tests
for debugging purposes
-e, --execute Launch tests specified after this switch
-f, --fork Fork child processes (opposite to --nofork)
-h, --help Display help
-i, --ignore Ignore tests specified after this switch
-j, --jobs Set number of job workers for unit tests
-l, --list List all available tests
-mt, --mtrace Enable mtrace log
-nf, --nofork Do not fork child processes (for better
debugging capabilities)
-nt, --nomtrace Disable mtrace log
-nsi, --nosysinfo Do not output system information
-o, --outfile file Output performance test statistics to specified file
-r, --resource path Location of the resource directory used by tests,
default resource path is 'res/test'
-s, --silent Do not output additional information from tests
-ss, --suppress Suppress extra messages, output only necessary information
-si, --sysinfo Output system information
-t, --tracepath path Override default trace path with specified value
default trace path is '/tmp/lsp-test-trace'
-td, --tempdir path Override default temporary directory for tests,
default temporary directory is '/tmp/lsp-test-temp'
-v, --verbose Output additional information from tests
Each test has fully-qualified name separated by dot symbols, tests from different test spaces (utest, ptest, mtest) may have similar fully-qualified names. Test names are case-sensitive.
To obtain a list of all unit tests we can issue:
.build/lsp-test-fw/lsp-test-fw-test utest --list
And then it is possible to launch all demo tests:
.test/lsp-plugins-test utest demo.*
If there is no unit test specified in arguments, all available unit tests will be launched.
To start unit test in debug mode, additional parameters are required to be passed:
.test/lsp-plugins-test utest --nofork --debug --verbose
Because unit tests are short-time fully-automated tests, they are parallelized and
executed by default by number_of_cores*2 processes. To disable this, option --nofork
is used.
Also, unit test execution time is limited by 5 seconds by default, so when the test gets stuck on
a break point, it gets killed. To prevent this, additional option --debug
is required to be passed.
Option --verbose
allows to the test to output additional information and is turned off by default.
Performance tests can be used to obtain full performance profile of target machine.
Because performance tests in most cases take much time for gathering statistics,
the final statistics for each test can be saved in a separate file by specifying --outfile
option:
.test/lsp-plugins-test ptest -o performance-test.log
Manual tests are mostly designed for developers' purposes.