Refactor test apps to use unit-test framework (#4014)

* Initial work on unittest framework, tested

* Finished reorganizing pjlib-test to use unit-test framework

* Fix big problem where performance improvement is not observed with the new framework (reason: because parallel flag is not set, doh!)

* Tidying up global vars in pjlib-test. Add test options: stop on error, skip essential tests, list tests

* Change the PJ_TEST_XX() signature to be more generic and does not force returning value

* Modifications to some existing tests to use unit-test test macros

* Updated VS projects with argparse.h and unittest.h

* Fix warnings on Windows

* Disable parallel unit-testing for ioqueue stress test on Windows because of errors when running on Windows virtual machine

* Non-parallel test case will now run exclusively (previously it did not wait the previous test to complete)

* Add pjlib-test/test_util.h for common utilities for parsing, configuring, and running unit test. This can be used by all test apps. pjlib-util-test has been ported to use this utilities

* Dirty hack to fix error message being displayed when user selected essential test because it does not exist in features test (in pjlib-test)

* Replace PJ_TEST_PARALLEL with PJ_TEST_EXCLUSIVE

* Ported pjnath-test to use unit-testing framework, with limited speed-up from 45m originally to 15m using 10 worker threads

* Large modifications in pjnath-test to speed-up test. It is fast now (4:30 minutes with 10 worker threads, from 45:42m originally)

* Ported pjmedia-test to use unit-test

* Porting of pjsip test to unit testing framework. Not much of speed improvements due to exclusive tests

* Modifications in pj_argparse API (changed get() to get_bool() and add automatic error reporting) hopefully make it easier to use

* Refactor tsx_uas_test() to allow parallel testing

* Parallelize tsx_uac_test()

* Further effort to parallize tests in pjsip-test. Discoverd major issue with unit-test logging (see unittest.md)

* Showing PJLIB config is optional with cmd line option

* Bug fixing failed pjsip tests when running in parallel mode

* Finished paralleizing all pjsip tests except one

* Continuing correcting errors

* Add test shuffle feature

* Modify CI workflows to use standard arguments: -w 3 --shuffle. These args are set in GitHub action variables

* Updated due to change in argparse API signature (swap arg order)

* Removed unittest.md (it was draft for the PR)

* Fix the use of GitHub repository vars

* Tidying up minor conflicting merge in VC projects and pjlib.h

* Attempt to fix assertion failure when tdata is being accessed after reference counter goes to zero in regc_test

* Fixing failure in transport_loop_test(), first in loop_resolve_error(), presumably because there are other loop transport around when the test is run. Solution are: 1) make the test exclusive for now, 2) fix the cleanup of loop transport in other tests. Then there is failure in transport_rt_test(), because it gets loop transport from other test that is being shutdown. Sneakily add pjsip_tpmgr_get_transport_count_by_type() that's useful for debugging.

* Attempt to fix unresolved winmm.lib API (e.g. timeKillEvent, timeGetTime) called from BaseClasses

* Split exclusive part of transport_loop_test

* CI mods: split steps, envs, shorten job names

* Fix failed tsx_basic_test

* Restore previously shortened job names

* Replace assertion with PJ_TEST_XX in resolver_test

* Add --stdout-buf and --stderr-buf options in test apps to control stdout/stderr buffering

* Fix missing sipp exe on Windows CI

* Fix wrong CI args on Mac. Use -j for faster make

* Replace pj_rand() with own rand in unittest because rand() yields different sequence on different platform even with the same srand, making it hard to reproduce test sequence

* Fix swig make error on Linux and runall.py error reading log file on Windows

* Attempt to fix test repeatability by 1) delete all calls to pj_srand() except in pj_test_suite_shuffle(), 2) unit test PRNG explicitly uses pj_uint32_t instead of int. Also disable windows python tests since it is unreliable

* Fixed inexistant function

* Relaxing the strictness of the test since sometimes it raises error

* Set regc_test() exclusive because it crashes sometimes, probably concurrency issue

* Modified thread counter to unsigned long (from pj_uint32_t) since the counter value is 2*1e9 and is overflow during diff calculation

* Fixed port double destruction in mips_test() and include benchmark tests in pj/config_site_test.h

* Use any port since sometimes test fails with address in use error

* Fix conflicted return value in udp_ioqueue_test() and let it immediately exit on error so that we can see correlated error log

* Restore sleep(0) in thread test since without it the test may occasionally fail on Linux

* Various attempt to fix fluke error in resolve_test.c: 1) servers use random port numbers, 2) increase delay waiting for various DNS timers, 3) reset global vars to zero because test may be repeated for IPv6

* More relaxed packet count tests in resolver_test

* Use any port instead of hardcoded one in udp ioqueue unregister_test since binding fails occasionally

* Rollback previous changes in resolver_test that relaxed packet count test

* Protect access to pool from worker thread with mutex in resolver_test

* Faster resolver_test time by reducing timeout

* Use high number port to make it less prone to bind error

* Remove hardcoded port number, replace with bind to any

* Fix SSL to continue decrypting data after renego completes

* Fixed race condition when registering SIP module in transport test

* Disable loading TLS cert in TURN sock test if SChannel is used

* Add (missing!) pjnath-test in Windows CI

* Fix syntax error in ci-win.yml and renamed lin->ubuntu in ci-linux

* Temporary workaround for MacOS rwmutex deadlock issue

* More jobs/tests in CI Mac, and changed names

* Fix silly typo in config_site_test.h

* Prettyfy CI job names, added more test steps

* Fix CI: ffmpeg lib path, faster git clone

* CI: simplify job names, add some audio checking

* CI: Disable SDL check because vid renderer is not available on CI machine

* Fixed missing ffmpeg shared lib when running pjmedia test

* CI Mac: install gnutls

* Lookup renderer in video codec test

* CI MacOS: attempt to fix failed OpenSSL test

* CI Mac: better test split to make duration more uniform across jobs

* Replace deprecated egrep with grep -E

---------

Co-authored-by: Nanang Izzuddin <[email protected]>
Co-authored-by: sauwming <[email protected]>

.github/workflows/ci-linux.yml

@@ -1,25 +1,34 @@
-name: CI Linux
+name: CI Ubuntu
 on:
   push:
     branches:
       - 'master'
   pull_request:
     types: [opened, synchronize, reopened]
+env:
+  CI_ARGS: ${{ vars.CI_UBUNTU_ARGS }}
+  CI_MODE: ${{ vars.CI_MODE }}
+  MAKE_FAST: make -j 3
 jobs:
-  build-ubuntu-default:
-  # checking pure lib source distribution with plain configure & make
+  default-build:
+    # checking pure lib source distribution with plain configure & make
     runs-on: ubuntu-latest
+    name: Default / build only
     steps:
     - uses: actions/checkout@v2
     - name: configure
       run: ./configure
     - name: make
-      run: make
+      run: $MAKE_FAST
+    - name: get SSL info
+      run: pjlib/bin/pjlib-test-`make infotarget` --config --list | grep SSL
+    - name: verify oepnssl is used
+      run: pjlib/bin/pjlib-test-`make infotarget` --config --list | grep -E 'PJ_SSL_SOCK_IMP\s+:\s+1'
 
-  ubuntu-default-full-bundle-1:
-  # full bundle: enable all codecs + AEC + DTLS
-  # full bundle 1: running pjlib, pjlib-util, pjmedia, and pjsua tests
+  default-full-bundle-1:
+    # full bundle: enable all codecs + AEC + DTLS
     runs-on: ubuntu-latest
+    name: Default / pjmedia,pjsua
     steps:
     - uses: actions/checkout@v2
     - name: install dependencies
@@ -29,19 +38,33 @@ jobs:
     - name: configure
       run: CFLAGS="-g -fPIC" CXXFLAGS="-g -fPIC" LDFLAGS="-rdynamic" ./configure
     - name: make
-      run: make
+      run: $MAKE_FAST
     - name: swig bindings
       run: cd pjsip-apps/src/swig && make
     - name: set up Python 3.10 for pjsua test
       uses: actions/setup-python@v2
       with:
         python-version: '3.10'
-    - name: unit tests
-      run: make pjlib-test-ci pjlib-util-test pjmedia-test pjsua-test
+    - name: capture pjsua capabilities
+      run: |
+        cat << EOF | pjsip-apps/bin/pjsua-`make infotarget` --log-level 3 > pjsua-caps
+        Cp
+        xx
+        vid dev list
+        vid codec list
+        q
+        EOF
+        cat pjsua-caps
+    - name: ensure AMR codec is installed
+      run: cat pjsua-caps | grep -E 'AMR/8000'
+    - name: pjmedia-test
+      run: make pjmedia-test
+    - name: pjsua-test
+      run: make pjsua-test
 
-  ubuntu-default-full-bundle-2:
-  # full bundle 2: running pjnath test
+  default-full-bundle-2:
     runs-on: ubuntu-latest
+    name: Default / pjlib,util,pjnath
     steps:
     - uses: actions/checkout@v2
     - name: install dependencies
@@ -51,13 +74,17 @@ jobs:
     - name: configure
       run: CFLAGS="-g" LDFLAGS="-rdynamic" ./configure
     - name: make
-      run: make
-    - name: unit tests
+      run: $MAKE_FAST
+    - name: pjlib-test
+      run: make pjlib-test
+    - name: pjlib-util-test
+      run: make pjlib-util-test
+    - name: pjnath-test
       run: make pjnath-test
 
-  ubuntu-default-full-bundle-3:
-  # full bundle 3: running pjsip test
+  default-full-bundle-3:
     runs-on: ubuntu-latest
+    name: Default / pjsip
     steps:
     - uses: actions/checkout@v2
     - name: install dependencies
@@ -67,126 +94,184 @@ jobs:
     - name: configure
       run: CFLAGS="-g" LDFLAGS="-rdynamic" ./configure
     - name: make
-      run: make
-    - name: unit tests
+      run: $MAKE_FAST
+    - name: pjsip-test
       run: make pjsip-test
 
-  build-ubuntu-no-tls:
-  # no TLS
+  no-tls:
     runs-on: ubuntu-latest
+    name: No SSL / pjlib,pjsip
     steps:
     - uses: actions/checkout@v2
     - name: install dependencies
       run: sudo apt-get install -y swig
     - name: configure
       run: CFLAGS="-fPIC" CXXFLAGS="-fPIC" ./configure --disable-ssl
     - name: make
-      run: make
+      run: $MAKE_FAST
     - name: swig bindings
       run: cd pjsip-apps/src/swig && make
+    - name: pjlib-test
+      run: make pjlib-test
+    - name: pjsip-test
+      run: make pjsip-test
 
   # build-ubuntu-openssl
   # TLS: with OpenSSL (same as build-ubuntu-default)
 
-  build-ubuntu-gnu-tls:
-  # TLS: with GnuTLS
+  gnu-tls:
     runs-on: ubuntu-latest
+    name: GnuTLS / pjlib,pjnath,pjsip
     steps:
     - uses: actions/checkout@v2
     - name: install dependencies
       run: sudo apt-get update && sudo apt-get install -y --fix-missing swig libgnutls28-dev
     - name: configure
       run: CFLAGS="-fPIC" CXXFLAGS="-fPIC" ./configure --with-gnutls=/usr/
     - name: make
-      run: make
+      run: $MAKE_FAST
     - name: swig bindings
       run: cd pjsip-apps/src/swig && make
+    - name: get SSL info
+      run: pjlib/bin/pjlib-test-`make infotarget` --config --list | grep SSL
+    - name: verify gnu tls is used
+      run: pjlib/bin/pjlib-test-`make infotarget` --config --list | grep -E 'PJ_SSL_SOCK_IMP\s+:\s+2'
+    - name: pjlib-test
+      run: make pjlib-test
+    - name: pjnath-test
+      run: make pjnath-test
+    - name: pjsip-test
+      run: make pjsip-test
 
-  ubuntu-video-openh264-1:
-  # video: video enabled with vpx and openh264
-  # video 1: running pjlib, pjlib-util, pjmedia, and pjsua tests
+  vid-openh264-1:
     runs-on: ubuntu-latest
+    name: OpenH264+VPX / pjmedia,pjsua
     steps:
     - uses: actions/checkout@v2
     - name: install dependencies
-      run: sudo apt-get install -y swig nasm sip-tester libvpx-dev libopencore-amrnb-dev
+      run: sudo apt-get install -y swig nasm sip-tester libvpx-dev libopencore-amrnb-dev libsdl2-dev
     - name: get openh264
-      run: git clone --single-branch --branch openh264v2.1.0 https://github.com/cisco/openh264.git
+      run: git clone --depth 1 --single-branch --branch openh264v2.1.0 https://github.com/cisco/openh264.git
     - name: build openh264
-      run: cd openh264 && make && sudo make install && sudo ldconfig
+      run: cd openh264 && $MAKE_FAST && sudo make install && sudo ldconfig
     - name: config site
       run: cd pjlib/include/pj && cp config_site_test.h config_site.h && echo "#define PJMEDIA_HAS_VIDEO 1" >> config_site.h
     - name: configure
       run: CFLAGS="-g -fPIC -DHAS_VID_CODEC_TEST=0" CXXFLAGS="-g -fPIC" LDFLAGS="-rdynamic" ./configure
     - name: make
-      run: make
+      run: $MAKE_FAST
     - name: swig bindings
       run: cd pjsip-apps/src/swig && make
     - name: set up Python 3.10 for pjsua test
       uses: actions/setup-python@v4
       with:
         python-version: '3.10'
-    - name: unit tests
-      run: make pjlib-test-ci pjlib-util-test pjmedia-test pjsua-test
+    - name: capture pjsua capabilities
+      run: |
+        cat << EOF | pjsip-apps/bin/pjsua-`make infotarget` --log-level 3 > pjsua-caps
+        Cp
+        xx
+        vid dev list
+        vid codec list
+        q
+        EOF
+        cat pjsua-caps
+    - name: ensure AMR codec is installed
+      run: cat pjsua-caps | grep -E 'AMR/8000'
+    - name: ensure H264 codec is installed
+      run: cat pjsua-caps | grep -E 'H264/'
+    - name: ensure VP8 codec is installed
+      run: cat pjsua-caps | grep -E 'VP8/'
+    # SDL error: no available vid dev
+    #- name: ensure SDL is installed
+    #  run: cat pjsua-caps | grep -E '\[SDL\]\[render\]'
+    - name: pjmedia-test
+      run: make pjmedia-test
+    - name: pjsua-test
+      run: make pjsua-test
 
-  ubuntu-video-openh264-2:
-  # video 2: running pjnath test
+  vid-openh264-2:
     runs-on: ubuntu-latest
+    name: OpenH264+VPX / pjlib,util,pjnath
     steps:
     - uses: actions/checkout@v2
     - name: install dependencies
-      run: sudo apt-get install -y nasm libvpx-dev libopencore-amrnb-dev
+      run: sudo apt-get install -y nasm libvpx-dev libopencore-amrnb-dev libsdl2-dev
     - name: get openh264
-      run: git clone --single-branch --branch openh264v2.1.0 https://github.com/cisco/openh264.git
+      run: git clone --depth 1 --single-branch --branch openh264v2.1.0 https://github.com/cisco/openh264.git
     - name: build openh264
-      run: cd openh264 && make && sudo make install && sudo ldconfig
+      run: cd openh264 && $MAKE_FAST && sudo make install && sudo ldconfig
     - name: config site
       run: cd pjlib/include/pj && cp config_site_test.h config_site.h && echo "#define PJMEDIA_HAS_VIDEO 1" >> config_site.h
     - name: configure
       run: CFLAGS="-g" LDFLAGS="-rdynamic" ./configure
     - name: make
-      run: make
-    - name: unit tests
+      run: $MAKE_FAST
+    - name: pjlib-test
+      run: make pjlib-test
+    - name: pjlib-util-test
+      run: make pjlib-util-test
+    - name: pjnath-test
       run: make pjnath-test
 
-  ubuntu-video-openh264-3:
-  # video: 3: running pjsip test
+  vid-openh264-3:
     runs-on: ubuntu-latest
+    name: OpenH264+VPX / pjsip
     steps:
     - uses: actions/checkout@v2
     - name: install dependencies
-      run: sudo apt-get install -y nasm libvpx-dev libopencore-amrnb-dev
+      run: sudo apt-get install -y nasm libvpx-dev libopencore-amrnb-dev libsdl2-dev
     - name: get openh264
-      run: git clone --single-branch --branch openh264v2.1.0 https://github.com/cisco/openh264.git
+      run: git clone --depth 1 --single-branch --branch openh264v2.1.0 https://github.com/cisco/openh264.git
     - name: build openh264
-      run: cd openh264 && make && sudo make install && sudo ldconfig
+      run: cd openh264 && $MAKE_FAST && sudo make install && sudo ldconfig
     - name: config site
       run: cd pjlib/include/pj && cp config_site_test.h config_site.h && echo "#define PJMEDIA_HAS_VIDEO 1" >> config_site.h
     - name: configure
       run: CFLAGS="-g" LDFLAGS="-rdynamic" ./configure
     - name: make
-      run: make
-    - name: unit tests
+      run: $MAKE_FAST
+    - name: pjsip-test
       run: make pjsip-test
 
-  build-ubuntu-video-ffmpeg:
-  # video enabled with vpx and ffmpeg and x264
+  vid-ffmpeg:
     runs-on: ubuntu-latest
+    name: FFMPEG+x264 / pjmedia
     steps:
     - uses: actions/checkout@v2
     - name: install dependencies
-      run: sudo apt-get install -y swig nasm libx264-dev libvpx-dev
+      run: sudo apt-get install -y swig nasm libx264-dev libvpx-dev libsdl2-dev
     - name: get ffmpeg
-      run: git clone --single-branch --branch release/4.2 https://github.com/FFmpeg/FFmpeg.git
+      run: git clone --depth 1 --single-branch --branch release/4.2 https://github.com/FFmpeg/FFmpeg.git
     - name: configure ffmpeg
       run: cd FFmpeg && ./configure --enable-shared --disable-static --enable-gpl --enable-libx264
     - name: build ffmpeg
-      run: cd FFmpeg && make -j10 && sudo make install
+      run: cd FFmpeg && $MAKE_FAST && sudo make install
     - name: config site
       run: echo -e "#define PJMEDIA_HAS_VIDEO 1\n" > pjlib/include/pj/config_site.h
     - name: configure
       run: CFLAGS="-fPIC" CXXFLAGS="-fPIC" ./configure
     - name: make
-      run: make
+      run: $MAKE_FAST
     - name: swig bindings
       run: cd pjsip-apps/src/swig && make
+    - name: capture pjsua capabilities
+      run: |
+        export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
+        cat << EOF | pjsip-apps/bin/pjsua-`make infotarget` --log-level 3 > pjsua-caps
+        Cp
+        xx
+        vid dev list
+        vid codec list
+        q
+        EOF
+        cat pjsua-caps
+    - name: ensure H264 codec is installed
+      run: cat pjsua-caps | grep -E 'H264/'
+    - name: ensure VP8 codec is installed
+      run: cat pjsua-caps | grep -E 'VP8/'
+    # SDL error: no available vid dev
+    #- name: ensure SDL is installed
+    #  run: cat pjsua-caps | grep -E '\[SDL\]\[render\]'
+    - name: pjmedia-test
+      run: export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH && make pjmedia-test