Fuzzing is a technique to find potential bugs by providing randomly generated invalid inputs. To detect potential bugs such as programming errors we use fuzzing in combination with ASan (Address Sanitizer), MSan (Memory Sanitizer), UBSan (Undefined Behavior Sanitizer) and asserts in the code. An invalid input will likely produce a decoding error (some API function returning error), which is absolutely not a problem, but what it should not do is access memory out of bounds, use uninitialized memory or hit a false assert condition.
libjxl fuzzing is integrated into oss-fuzz
as the project libjxl. oss-fuzz regularly runs the fuzzers on the main
branch and reports bugs into their bug tracker which remains private until the
bugs are fixed in main.
There are several fuzzer executable targets defined in the tools/ directory
to fuzz different parts of the code. The main one is djxl_fuzzer, which uses
the public C decoder API to attempt to decode an image. The fuzzer input is not
directly the .jxl file, the last few bytes of the fuzzer input are used to
decide how will the API be used (if preview is requested, the pixel format
requested, if the .jxl input data is provided altogether, etc) and the rest of
the fuzzer input is provided as the .jxl file to the decoder. Some bugs might
reproduce only if the .jxl input is decoded in certain way.
The remaining fuzzer targets execute a specific portion the codec that might be easier to fuzz independently from the whole codec.
A fuzzer target, like djxl_fuzzer accepts as a parameter one or more files
that will be used as inputs. This runs the fuzzer program in test-only mode
where no new inputs are generated and only the provided files are tested. This
is the easiest way to reproduce a bug found by the fuzzer using the generated
test case from the bug report.
oss-fuzz uses a specific compiler version and flags, and it is built using
Docker. Different compiler versions will have different support for detecting
certain actions as errors, so we want to reproduce the build from oss-fuzz as
close as possible. To reproduce the build as generated by oss-fuzz there are a
few helper commands in ci.sh as explained below.
First you need the ossfuzz libjxl builder image. This is the base oss-fuzz builder image with a few dependencies installed. To generate it you need to check out the oss-fuzz project and build it:
git clone https://github.com/google/oss-fuzz.git ~/oss-fuzz
cd ~/oss-fuzz
sudo infra/helper.py build_image libjxlThis will create the gcr.io/oss-fuzz/libjxl docker image. You can check if it
was created verifying that it is listed in the output of the sudo docker image ls command.
To build the fuzzer targets from the current libjxl source checkout, use the
./ci.sh ossfuzz_msan command for MSan, ./ci.sh ossfuzz_asan command for ASan
or ./ci.sh ossfuzz_ubsan command for UBSan. All the JXL_ASSERT and
JXL_DASSERT calls are enabled in all the three modes. These ci.sh helpers will
reproduce the oss-fuzz docker call to build libjxl mounting the current source
directory into the Docker container. Ideally you will run this command in a
different build directory separated from your regular builds.
For example, for MSan builds run:
BUILD_DIR=build-fuzzmsan ./ci.sh ossfuzz_msanAfter this, the fuzzer program will be generated in the build directory like
for other build modes: build-fuzzmsan/tools/djxl_fuzzer.
After modifying the source code to fix the fuzzer-found bug, or to include more
debug information, you can rebuild only a specific fuzzer target to save on
rebuilding time and immediately run the test case again. For example, for
rebuilding and testing only djxl_fuzzer in MSan mode we can run:
BUILD_DIR=build-fuzzmsan ./ci.sh ossfuzz_msan djxl_fuzzer && build-fuzzmsan/tools/djxl_fuzzer path/to/testcase.binWhen MSan and ASan fuzzers fail they will print a stack trace at the point where
the error occurred, and some related information. To make these these stack
traces useful we need to convert the addresses to function names and source file
names and lines, which is done with the "symbolizer". For UBSan to print a stack
trace we need to set the UBSAN_OPTIONS environment variables when running the
fuzzer.
Set the following environment variables when testing the fuzzer binaries. Here
clang should match the compiler version used by the container, you can pass a
different compiler version in the following example by first installing the
clang package for that version outside the container and using clang-NN
(for example clang-11) instead of clang in the following commands:
symbolizer=$($(realpath "$(which clang)") -print-prog-name=llvm-symbolizer)
export MSAN_SYMBOLIZER_PATH="${symbolizer}"
export UBSAN_SYMBOLIZER_PATH="${symbolizer}"
export ASAN_SYMBOLIZER_PATH="${symbolizer}"
export ASAN_OPTIONS=detect_leaks=1
export UBSAN_OPTIONS=print_stacktrace=1Note: The symbolizer binary must be a program called llvm-symbolizer, any
other file name will fail. There are normally symlinks already installed with
the right name which the -print-prog-name would print.
Running the fuzzer targets in fuzzing mode can be achieved by running them with no parameters, or better with a parameter with the path to a directory containing a seed of files to use as a starting point. Note that passing a directory is considered a corpus to use for fuzzing while passing a file is considered an input to evaluate. Multi-process fuzzing is also supported. For details about all the fuzzing options run:
build-fuzzmsan/tools/djxl_fuzzer -help=1Fuzzing on itself can't find programming bugs unless an input makes the program perform an invalid operation (read/write out of bounds, perform an undefined behavior operation, etc). You can help the fuzzer find invalid situations by adding asserts:
-
JXL_DASSERTis only enabled in Debug builds, which includes all the ASan, MSan and UBSan builds. Performance of these checks is not an issue if kept within reasonable limits (automated msan/asan test should finish within 1 hour for example). Fuzzing is more effective when the given input runs faster, so keep that in mind when adding a complex DASSERT that runs multiple times per output pixel. -
For MSan builds it is also possible to specify that certain values must be initialized. This is automatic for values that are used to make decisions (like when used in an
ifstatement or in the ternary operator condition) but those checks can be made explicit for image data using theJXL_CHECK_IMAGE_INITIALIZED(image, rect)macro. This helps document and check (only in MSan builds) that a given portion of the image is expected to be initialized, allowing to catch errors earlier in the process.
In MSan builds it is considered an error to use uninitialized memory. Using
the memory normally requires something like a decision / branch based on the
uninitialized value, just running memcpy() or simple arithmetic over
uninitialized memory is not a problem. Notably, computing DemoteTo(),
NearestInt() or similar expressions that create a branch based on the value of
the uninitialized memory will trigger an MSan error.
In libjxl we often run vectorized operations over a series of values, rounding
up to the next multiple of a vector size, thus operating over uninitialized
values past the end of the requested region. These values are part of the image
padding but are not initialized. This behavior would not create an MSan error
unless the processing includes operations like NearestInt(). For such cases
the preferred solution is to use msan::UnpoisonMemory over the portion of
memory of the last SIMD vector before processing, and then running
msan::PoisonMemory over the corresponding value in the output side. A note
including why this is safe to do must be added, for example if the processing
doesn't involve any cross-lane computation.
Initializing padding memory in MSan builds is discouraged because it may hide bugs in functions that weren't supposed to read from the padding. Initializing padding memory in all builds, including Release builds, would mitigate the MSan potential security issue but it would hide the logic bug for a longer time and potentially incur in a performance hit.