Unit test framework

[bennylp]

This PR contains new unit testing framework as part of the work to refactor the test codes to make them faster (see #4014).

Features

This PR adds new unit testing framework in <pj/unittest.h> and pj/unittest.c. Some features of the framework are:

• Familiarity with common architecture as described in https://en.wikipedia.org/wiki/XUnit
• Multithreaded/parallel execution
• Easy to port existing test functions (mostly without modifications)
• Convenient PJ_TEST_XXX() macros can be used in place of manual testing and error reporting
• Nice output:

07:34:32.878 Performing 20 features tests with 4 worker threads
[ 1/20] hash_test                        [OK] [0.000s]
[ 2/20] atomic_test                      [OK] [0.000s]
[ 3/20] rand_test                        [OK] [0.002s]

• Even nicer output, logging is captured and only displayed if the test fails (configurable)
• Test shuffling feature
• Basic framework (without pool, thread, and mutex) is available

Using the test macros

At minimum, the <pj/unittest.h> provides many useful testing macros that can be used directly to replace manual checking in test apps, without having to use the framework. Test macros currently implemented are:

• PJ_TEST_SUCCESS(expr, err_reason, err_action)
• PJ_TEST_NON_ZERO(expr, err_reason, err_action)
• PJ_TEST_TRUE(expr, err_reason, err_action)
• PJ_TEST_NOT_NULL(expr, err_reason, err_action)
• PJ_TEST_EQ(expr0, expr1, err_reason, err_action)
• PJ_TEST_NEQ(expr0, expr1, err_reason, err_action)
• PJ_TEST_LT(expr0, expr1, err_reason, err_action)
• PJ_TEST_LTE(expr0, expr1, err_reason, err_action)
• PJ_TEST_GT(expr0, expr1, err_reason, err_action)
• PJ_TEST_GTE(expr0, expr1, err_reason, err_action)
• PJ_TEST_STRCMP(ps0, ps1, res_op, exp_result, err_reason, err_action)
• PJ_TEST_STRICMP(ps0, ps1, res_op, exp_result, err_reason, err_action)
• PJ_TEST_STREQ(ps0, ps1, err_reason, err_action)
• PJ_TEST_STRNEQ(ps0, ps1, err_reason, err_action)

For an example:

   PJ_TEST_EQ(recv_count, COUNT, NULL, {rc=-1; goto on_error;});

when recv_count!=COUNT will write something like this to the log:

11:56:41 Test "recv_count" (value=99) == "COUNT" (value=100) fails in ../pjlib-test/sometest.c:75

as well as executing the code in err_action.

Introduction to unit test framework

As mentioned above, the framework uses common architecture as described in https://en.wikipedia.org/wiki/XUnit. Basically the architecture is as follows.

Individual test is called a test case, and basically it wraps a test function into pj_test_case structure, which you can set many options on it.

The test cases are then put into a test suite, pj_test_suite. A test application (such as pjsip test) usually only needs one test suite, although pjlib-test has two test suites, essential and features, because it cannot run more complex unit-testing before the unit-test framework itself has been unit-tested!

Then we need a runner to run the tests. The framework provides two types of runners, basic runner and text runner. Basic runner simply runs the tests sequentially. In fact, we don't need pool and OS features (such as threads, mutexes, TLS) to create test cases, test suites, and run the test with the basic runner. And this is basically the idea when splitting pjlib unit tests into essential and features. The essential tests is unit-testing using basic runner, to test all features that are needed by the full unit-test framework, such as threads, mutexes, TLS, fifobuf, etc.

The text runner is a multithreaded test runner. This is the only difference from basic runner. Other than this, both runners provides the same functionality and share the same API.

Using the framework

Below is a working sample:

#include <pj/os.h>
#include <pj/pool.h>
#include <pj/unittest.h>

#define THIS_FILE   "demotest.c"

static int func_to_test(void *arg)
{
    pj_time_val tv;

    PJ_TEST_SUCCESS(pj_gettimeofday(&tv), NULL, return -10);
    /* This intentionally will fail */
    PJ_TEST_GT(tv.msec, 1000, NULL, return -20);
    return 0;
}

int test(pj_pool_t *pool)
{
   pj_test_case tc;
   char logbuf[1024];
   unsigned flags = 0;
   pj_test_suite suite;
   pj_test_runner *runner;
   pj_test_stat stat;

   /* Init test suite */
   pj_test_suite_init(&suite);

   /* Init test cases */
   pj_test_case_init(&tc, "test case 0", flags, &func_to_test, NULL,
                     logbuf, sizeof(logbuf), NULL);
   pj_test_suite_add_case(&suite, &tc);

   /* Create runner and run the test suite */
   PJ_LOG(3,(THIS_FILE, "Running test.."));
   pj_test_create_text_runner(pool, NULL, &runner);
   pj_test_run(runner, &suite);

   /* We can safely destroy the runner now */
   pj_test_runner_destroy(runner);

   /* Get/show stats, display logs for failed tests */
   pj_test_display_log_messages(&suite, PJ_TEST_FAILED_TESTS);
   pj_test_get_stat(&suite, &stat);
   pj_test_display_stat(&stat, "demo test", THIS_FILE);

   return stat.nfailed ? -1 : 0;
}

When run, it should produce something like this:

07:35:37.328 Running tests..
[ 1/1] test case 0 (arg=0)               [OK] [0.000s]
14:24:08 ------------ Displaying failed test logs: ------------
14:24:08 ------------ Logs for func_to_test (arg: 0) [rc:-20]: ------------
14:24:08  Test "tv.msec" (value=123) > "1000" (value=1000) fails in demotest.c:13
14:24:08 --------------------------------------------------------
14:24:08 Unit test statistics for demo test:
14:24:08     Total number of tests: 1
14:24:08     Number of test run:    1
14:24:08     Number of failed test: 1
14:24:08     Total duration:        0m0.000s

fifobuf development

This PR also contains modifications to fifobuf.[hc], a feature that is part of pjsip initial commit but yet has not find any use until now! The fifobuf is used to save log messages.

Known issues

Wrong logging in multithreaded scenario

Logging is saved in buffer to be printed at the end of the test (controllable). The framework does this by saving the current test case to TLS (thread local storage), and retrieve it from the logging callback. This has couple of drawbacks.

First, and this is rather major one, in software architecture where there is a central event dispatcher (like in PJSIP), multiple threads can poll the event dispatcher and any threads can process event that "belongs" to other thread. This will cause logging to be sent to the wrong test case.

In the current development of pjsip test (on separate branch), the "workaround" is to print this message to the screen:

********************************************************************
**                        W A R N I N G                           **
********************************************************************
** Due to centralized event processing in PJSIP, events may be    **
** read by different thread than the test's thread. This may      **
** cause logs to be saved by the wrong test when multithreaded    **
** testing is used. The test results are correct, but the log     **
** may not be accurate.                                           **
** For debugging with correct logging, use "-w 0 --log-no-cache"  **
********************************************************************

The second drawback, not so major one, when logging is sent by a thread that is not in the context of executing a test case (such as loop transport's worker thread), the framework will not find associated test case with the thread, and hence will just display the log immediately (rather than save it). It will be difficult to associate which test case with the message and at the very least, output will be cluttered.

Note that the test results are correct regardless of the logging. When debugging a test failure, you can disable worker thread (pj_test_runner_param.nthreads=0) and log caching (PJ_TEST_LOG_NO_CACHE) in order to get the correct logging.

No nested unit testing

You cannot run a nested unit testing from inside a test case function. The main reason is because logging can only be owned by one test case for each thread.