Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/getting-started/adding_a_new_fuzzer.md
Original file line number Diff line number Diff line change
Expand Up @@ -308,6 +308,9 @@ you are adding): add an experiment request to
[service/experiment-requests.yaml](https://github.com/google/fuzzbench/blob/master/service/experiment-requests.yaml).
`service/experiment-requests.yaml` explains how to do this.

For detailed instructions on submitting experiments via pull requests, refer to the [Submitting Experiments via Pull Requests](../running-a-cloud-experiment/running_an_experiment.md) section in the Running an Experiment documentation.


At the end of the experiment, FuzzBench will generate a report comparing your
fuzzer to the latest versions of other fuzzers, so you only need to include
fuzzers that you've modified in a meaningful way (i.e. fuzzers whose results are
Expand Down
69 changes: 69 additions & 0 deletions docs/running-a-cloud-experiment/running_an_experiment.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,15 @@ FuzzBench.

# run_experiment.py

The `run_experiment.py` script is the core script used to start experiments in FuzzBench. It is also the script that the parameters passed to `gcbrun` are fed into. This script creates a dispatcher instance on Google Compute Engine which runs the experiment, including:

1. Building desired fuzzer-benchmark combinations.
1. Starting instances to run fuzzing trials with the fuzzer-benchmark builds and stopping them when they are done.
1. Measuring the coverage from these trials.
1. Generating reports based on these measurements.

The rest of this page will assume all commands are run from the root of FuzzBench.

## Experiment configuration file

You need to create an experiment configuration yaml file.
Expand Down Expand Up @@ -163,3 +172,63 @@ env:

Currently values in `env` are only set before running the fuzzer, not before
building the benchmarks or the fuzzer itself.

# Submitting Experiments via Pull Requests
This section complements the existing instructions for running experiments locally or on your own cloud infrastructure, providing additional guidance for contributing experiments via pull requests.
When contributing an experiment through a GitHub Pull Request (PR), note that the workflow and naming rules may differ from running experiments on your own infrastructure.

## Maintainer Workflow

FuzzBench maintainers may run your experiment manually using an internal tool (`/gcbrun`) instead of the `run_experiment.py` script you’d use on your own project. For example:

```bash
/gcbrun run_experiment.py -a \
--experiment-config /opt/fuzzbench/service/experiment-config.yaml \
--experiment-name 2025-01-22-freetype2-mts \
--fuzzers afl aflplusplus libfuzzer \
--benchmarks freetype2_ftfuzzer
```

Therefore, you **should** provide all needed artifacts (e.g., benchmarks, seed corpora) in the PR.

## Experiment Name Requirements

Your experiment name **must**:

- Use only lowercase letters (`a-z`), numbers (`0–9`), and hyphens (`-`).
- Be ≤ 30 characters long.
- Avoid underscores (`_`).

Invalid:
```bash
2025-01-22-freetype2_ftfuzzer-mutants
```

Valid:
```bash
2025-01-22-freetype2-mts
```

If the name is invalid, maintainers will rename it before running.

## Seeds and Custom Inputs

If your experiment relies on a custom seed corpus:

- Place it in a `.tar.gz` file.
- Link to it in your PR (GitHub repo, GCS URL, etc.).
- Make sure your benchmark supports loading it correctly.

## Logs and Debugging

- Fuzzer output logs are available. As outlined above.
- **Cloud logs are not publicly accessible**.
- To debug deeper issues, you can:
- Log directly to `stdout`/`stderr` in your benchmark or fuzzer script.
- Include lightweight instrumentation in the fuzzer (e.g., seed counts, coverage summaries).

## Configuration Files For Cloud Experiments

Fuzzer configuration files (e.g., `--fuzzer-configs`) are still supported for **offline/local experiments**. However, these have been deprecated for Google Cloud-based experiments run by the FuzzBench service. For cloud experiments, provide configuration parameters via your benchmark or fuzzer scripts directly.