diff --git a/.ci/docker/rockylinux.Dockerfile b/.ci/docker/rockylinux.Dockerfile index 9821de2..5d9a2b4 100644 --- a/.ci/docker/rockylinux.Dockerfile +++ b/.ci/docker/rockylinux.Dockerfile @@ -11,7 +11,8 @@ RUN dnf install -y \ clang \ g++ \ ninja-build \ - cmake + cmake \ + python3.11-venv RUN dnf clean all # Copy code diff --git a/.ci/docker/ubuntu.Dockerfile b/.ci/docker/ubuntu.Dockerfile index fbfe4ed..51af65a 100644 --- a/.ci/docker/ubuntu.Dockerfile +++ b/.ci/docker/ubuntu.Dockerfile @@ -8,7 +8,8 @@ RUN apt-get install -y \ clang-tidy \ g++ \ ninja-build \ - cmake + cmake \ + python3.11-venv RUN apt-get clean WORKDIR /workarea diff --git a/.github/workflows/cxx.yml b/.github/workflows/cxx.yml index 2ac2e49..5ee8cf7 100644 --- a/.github/workflows/cxx.yml +++ b/.github/workflows/cxx.yml @@ -80,4 +80,4 @@ jobs: cache-to: type=gha,mode=max - name: Run tests run: | - docker run ${{ matrix.cfg.id }} ctest --test-dir build + docker run ${{ matrix.cfg.id }} ctest --test-dir build --output-on-failure diff --git a/README.md b/README.md index 69fcafa..6072fa0 100644 --- a/README.md +++ b/README.md @@ -102,6 +102,61 @@ cmake -B /some/build/dir -S . -DCMAKE_CXX_CLANG_TIDY="clang-tidy;-checks=-*,cppc Otherwise follow the Basic Build workflow as described above. +### Testing + +Tests are written using `lit`, the [LLVM Integrated Tester][]. Each test is an +independent executable with a `main` function. Tests are written in a pretty +basic way, using `assert`. If you want, you can use an arbitrary testing framework +on top of that. However, we recommend against doing so because keeping the tests +basic makes them very portable, and in particular it makes them compatible with +the LLVM C++ Standard Library conformance test suite developed as part of libc++, +which is also reused by other implementations like the MSVC STL and libstdc++. + +#### Writing tests + +Tests are written as standalone executables with a `main` function. Tested conditions +are asserted using simple `assert(...)` statements. Various test flavors are supported: + +- `.pass.cpp` + These tests are built and run. The test succeeds if the program terminates normally + with a `0` exit code. The test fails if it doesn't compile, link or if it exits with + an error. +- `.verify.cpp` + These tests run using `clang-verify`. This allows checking that specific diagnostics + are being emitted at compile-time. + + Clang-verify supports various directives like `expected-error`, `expected-warning`, etc. + The full set of directives supported and how to use them is documented in [the Clang + documentation](https://clang.llvm.org/docs/InternalsManual.html#specifying-diagnostics). +- `.compile.pass.cpp` + These tests don't run, they only compile. The test passes if the program compiles, and + fails otherwise. This can be used to test things like `aliases` or concept conformance, + which don't need to actually run anything. +- `.sh.pass.cpp` + These tests run whatever shell commands are specified in the `RUN` commands specified + in the test. This provides a lot of flexibility for controlling how the test gets built + and run, and can be used to check things that are otherwise difficult to test (e.g. + compatibility between TUs built with different Standard modes). + +#### Running tests + +To run all the tests in the project, use: + +```shell +$ ctest --test-dir /some/build/dir +``` + +You can run a single test in the project by using `lit` directly: + +```shell +$ /some/build/dir/test/venv/bin/lit /some/build/dir/test/example-paper/test1.pass.cpp +``` + +That's not a typo, you must use the `/some/build/dir` directory as a prefix to the +path of the test you want to run. This lets `lit` find the testing configuration +that was generated by CMake. + + ## Usage ### From C++ @@ -141,6 +196,8 @@ Please do! Issues and pull requests are appreciated. Note that adding more C++ code will be out of scope for this project. Changes that further improve or simplify this project given that goal are appreciated. Enhancements to better support packaging ecosystems would also make sense. +[LLVM Integrated Tester]: https://llvm.org/docs/CommandGuide/lit.html +