Skip to content

Update fuzz testing documentation #9125

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alexreinking merged 1 commit into from
May 7, 2026
Merged

Update fuzz testing documentation #9125

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
189 changes: 127 additions & 62 deletions doc/FuzzTesting.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,110 +1,175 @@
# Fuzz testing

Halide has a set of fuzz-testing harnesses that can be used to find those tricky
to find, edge cases and bugs that would otherwise not be caught by a regular
unit-testing suite. At the moment these fuzz-tests are housed in the `test/fuzz`
directory. The fuzz testing suite use the common,
[libfuzzer](https://www.llvm.org/docs/LibFuzzer.html) interface for fuzz-tests.
Halide has a set of fuzz-testing harnesses in `test/fuzz/` that can find tricky
edge cases and bugs that are hard to catch with a regular unit-testing suite.
The fuzz tests are built on a small in-tree framework (`fuzz_helpers.h`,
`halide_fuzz_main.h`) that abstracts over two backends:

- **stdlib backend** — uses `std::mt19937_64` seeded from `std::random_device`.
Works with any standard C++ toolchain; no special compiler flags or external
runtime libraries required. Each run prints its seed so failures are
reproducible by re-running with that seed.
- **libfuzzer backend** — uses
[libFuzzer](https://www.llvm.org/docs/LibFuzzer.html) for coverage-guided
fuzzing. Requires a Clang toolchain built with `-fsanitize=fuzzer` support.

The stdlib backend is the default for regular development builds. The libfuzzer
backend is enabled automatically when the build system detects
`-fsanitize=fuzzer` (or equivalent) linker flags.

## Building fuzz tests

Fuzz testing requires specific instrumentation across the entire build; to do
this we make use of a fuzzing-specific-toolchain/preset. e.g.
### Standard build (stdlib backend)

No special flags are needed. The fuzz tests build as part of any normal CMake
configuration that has `WITH_TEST_FUZZ=YES`:

```
cmake -B build --preset linux-x64-fuzzer -DLLVM_ROOT=/path/to/llvminstall
cmake --build ./build -j$(nproc)
cmake -B build <your-usual-options> -DWITH_TEST_FUZZ=YES
cmake --build build -j$(nproc) --target test_fuzz
```

Note that the LLVM install that you use must be built with
`-D LLVM_ENABLE_RUNTIMES="compiler-rt"` set if you want to build the fuzzer
tests (failing to do so will fail at configure time); not all prebuilt LLVM
installs include this, so you may need to build LLVM from source to run the fuzz
tests locally.
### libfuzzer backend (coverage-guided fuzzing)

## Using the fuzz-harnesses
Use one of the fuzzing CMake presets, which set the necessary
`-fsanitize=fuzzer[-no-link]` flags across the entire build:

Fuzz-testing harnesses are a little different to a more traditional unit-test
and don't have a definitive end of test. In other words, a fuzz test will run:
**Linux:**

- for an infinite amount of time (the default),
- for a user specified maximum amount of time,
- until the fuzzer finds a bug and crashes,
- you manually kill the process e.g. (ctrl-C).
```
cmake -B build --preset linux-x64-fuzzer -DHalide_LLVM_ROOT=/path/to/llvm-install
cmake --build build -j$(nproc)
```

Once you have built the fuzz testing suite using the commands listed above you
can list the fuzz testing harnesses using the command:
**macOS (Homebrew LLVM):**

```
ls ./build/test/fuzz/fuzz_*
cmake -B build --preset macOS-fuzz
cmake --build build -j$(nproc)
```

To run a fuzzer simply run the fuzz-testing harness with no arguments. e.g.
The LLVM install used for libfuzzer builds must include the `compiler-rt`
runtime (i.e. built with `-DLLVM_ENABLE_RUNTIMES="compiler-rt"`). Not all
prebuilt LLVM installs include this; you may need to build LLVM from source or
use Homebrew's LLVM package on macOS.

`./build/test/fuzz/fuzz_simplify`
## Running fuzz tests

By default this will run the fuzz test on a single core and discard whatever.
temporary corpus is created.
### stdlib backend

To reuse a given corpus (recommended) create a new directory to store the corpus
generated by your fuzz testing harness and pass that directory into your fuzzer
e.g.
Run a fuzz harness directly:

```
mkdir fuzz_simplify_corpus -p
./build/test/fuzz/fuzz_simplify fuzz_simplify_corpus
./build/test/fuzz/fuzz_simplify
```

By default this runs 10,000 iterations, printing the seed before each one:

```
Seed: 12345678901234567
Seed: 98765432109876543
...
```

Control the number of iterations with `-runs=N`:

```
./build/test/fuzz/fuzz_simplify -runs=100000
```

Run all fuzz tests via CTest (1,000 iterations each, exit-code–based pass/fail):

```
ctest --test-dir build -L fuzz
```

### libfuzzer backend

After building with a fuzzing preset, run the harness with no arguments to start
coverage-guided fuzzing on a single core:

```
./build/test/fuzz/fuzz_simplify
```

To persist the corpus between runs (recommended):

This will save the state of the fuzzer between runs, this way any progress that
your fuzzer makes improving code-coverage will remain persistent on your disk.
```
mkdir -p fuzz_simplify_corpus
./build/test/fuzz/fuzz_simplify fuzz_simplify_corpus
```

Up until this point the fuzzer has only been running on a single core. To speed
things up a little, let's run the fuzzer in parallel across all available cores
on our machine.
To fuzz in parallel across all available cores:

```
./build/test/fuzz/fuzz_simplify fuzz_simplify_corpus -fork=$(nproc)
```

## Reproducing crashes
## Reproducing failures

### stdlib backend

When a run fails, rerun with the seed that was printed just before the crash:

```
./build/test/fuzz/fuzz_simplify 12345678901234567
```

This performs a single deterministic iteration with that seed.

An important part of fuzz testing is reproducing the crashing input. To handle
this, a libfuzzer-based fuzz harness will create a crash file whenever the
fuzzer exits unexpectedly. This will look something like:
### libfuzzer backend

`crash-<some_random_hash>`
libFuzzer writes a crash-input file on failure:

To reproduce a crash we simply rerun our fuzz harness with our crash file as the
first argument.
```
crash-<some_random_hash>
```

`./build/test/fuzz/fuzz_simplify crash-<some_random_hash>`
Replay it by passing it as the first argument:

So long as your fuzz harness and library are deterministic this should reproduce
the original crash.
```
./build/test/fuzz/fuzz_simplify crash-<some_random_hash>
```

## Adding new fuzz tests

A bare-bones fuzzer will look something like the following:
All fuzz tests use the `FUZZ_TEST` macro defined in `fuzz_helpers.h`. This macro
generates the correct entry point for whichever backend is active —
`LLVMFuzzerTestOneInput` for libfuzzer or a `main` that calls
`Halide::fuzz_main` for the stdlib backend.

A minimal fuzz test looks like this:

```cpp
#include <stdint.h>
#include <stddef.h>
#include <my_library.h>
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
// Randomly throw data at our function and hope it doesn't crash.
foo(data, size);
return 0;
#include "fuzz_helpers.h"

FUZZ_TEST(my_test, Halide::FuzzingContext &fuzz) {
int x = fuzz.ConsumeIntegralInRange<int>(0, 100);
bool b = fuzz.ConsumeBool();
my_function(x, b);
return 0;
}
```

This assumes that our function foo takes in a buffer and the size of said
buffer. But in many cases we would like to make use of more structured data.
e.g. a string or a vector of integers etc. Thankfully libfuzzer provides a handy
helper to convert a raw buffer into common structured data types, the
[FuzzedDataProvider class](https://github.com/llvm/llvm-project/blob/main/compiler-rt/include/fuzzer/FuzzedDataProvider.h).
For examples on how to use this class see `test/fuzz/simplify.cpp`.
`FuzzingContext` wraps `FuzzedDataProvider` (from libfuzzer's
`compiler-rt/include/fuzzer/FuzzedDataProvider.h`) and re-implements its
interface on top of `std::mt19937_64` for the stdlib backend, so the same API
works with both backends. Key methods:

- `ConsumeIntegral<T>()` — random value of type `T`
- `ConsumeIntegralInRange<T>(min, max)` — random value in `[min, max]`
- `ConsumeBool()` — random boolean
- `PickValueInArray(arr)` — random element from an array or initializer list
- `PickValueInVector(vec)` — random element from a `std::vector`

For richer examples, see `test/fuzz/simplify.cpp` and
`test/fuzz/random_expr_generator.h`.

To register a new fuzz test with CMake, add it to the `SOURCES` list in
`test/fuzz/CMakeLists.txt`.

## Other useful materials

- [The official libfuzzer docs](https://www.llvm.org/docs/LibFuzzer.html)
- [The libfuzzer tutorial](https://github.com/google/fuzzing/blob/master/tutorial/libFuzzerTutorial.md)
- [FuzzedDataProvider reference](https://github.com/llvm/llvm-project/blob/main/compiler-rt/include/fuzzer/FuzzedDataProvider.h)
Loading