Skip to content

👷 ci(fuzz): hide crashers, add a malloc pass - #970

Merged
gaborbernat merged 5 commits into
tox-dev:mainfrom
gaborbernat:fuzz/harness-hygiene
Oct 2, 2026
Merged

gaborbernat merged 5 commits into
tox-dev:mainfrom
gaborbernat:fuzz/harness-hygiene

Conversation

@gaborbernat

@gaborbernat gaborbernat commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

The deep fuzz job printed up to 400 bytes of each crashing input into its CI log, and the 📤 Upload crash corpus step added in #959 published crash files as a plain artifact. Logs and artifacts on a public repository are readable by anyone, so the crasher of an unfixed bug went public before its fix. The in-process driver also ran every input under pymalloc only, whose pools hide an over-read that stays inside a pool from AddressSanitizer, and tox handed the fuzz environments a random PYTHONHASHSEED per run, so a find on the hash-salted XML duplicate-attribute check might not replay.

The driver now logs a crasher only by SHA-256, length and harness, and keeps the bytes as .fuzz-crashes/crash-<sha256>, the libFuzzer artifact naming. The seed and mutation index that regenerate it go to crash-<sha256>.replay, and a deep run writes its sanitizer reports to crash-sanitizer.<pid>, because a report names the faulting function and line. The workflow uploads crashers only after encrypting each one with age v1.3.2, a checksum-pinned release binary, to the public key in the FUZZ_AGE_RECIPIENT repository variable. Without the variable it uploads nothing. Maintainers holding the identity key become the only readers, the audience OSS-Fuzz gives its reports until the fix ships. The fuzzing page of the development docs walks through the one-time key setup.

Each in-process run now executes twice, under pymalloc and under PYTHONMALLOC=malloc, over the same mutation sequence. The deep budget splits across the two passes, so the weekly 20-minute slot stays inside the six-hour job limit. fuzz and fuzz-smoke pin PYTHONHASHSEED=0, and --rng-seed now reaches the worker. The scheduled job draws a random seed per run and passes it only through the environment, since tox echoes its command line and the seed regenerates every crasher. The deep run also replays the JS minifier fixtures in tests/js/_corpus; it pointed at a directory that does not exist and never replayed them.

The roundtrip and sanitize idempotence checks skip two documented cases and nothing else: script text that breaks the script content restrictions by opening <!-- and then <script before any -->, whose serialization the standard does not promise to reparse to the same tree, and trees that reached the 512 open-element cap, past which the builder stops nesting.

Past finds (#521, #752, #949, #952) move to tests/fuzz_regressions/, and every run replays them first through every harness, through the standalone harnesses' file-argument replay and the in-process seed loop. jsoup's FuzzFixesTest replays its own finds the same way.

The malloc pass catches the class of #949, a one-past-the-end read that pymalloc hid and only the system allocator exposed. A regression of any listed fix now fails the per-PR smoke gate, and a scheduled find stays private while it waits for a fix.

@codspeed

codspeed Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 5.58%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 1 regressed benchmark
✅ 580 untouched benchmarks
⏩ 32 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ test_feature[transform-number-count-last] 552.1 µs 584.7 µs -5.58%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing gaborbernat:fuzz/harness-hygiene (2001f89) with main (c795023)

Open in CodSpeed

Footnotes

  1. 32 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@gaborbernat
gaborbernat force-pushed the fuzz/harness-hygiene branch 2 times, most recently from e391edf to 2b9c7e9 Compare October 1, 2026 23:49
The deep fuzz job printed up to 400 bytes of each crashing input into
its CI log and uploaded crash files as a plain artifact. Both are public
on this repository, so the crasher of an unfixed bug went public before
its fix. The driver now logs a crasher only by SHA-256, length, harness
and seeds, and the workflow uploads crashers only after encrypting them
with age to the key in the FUZZ_AGE_RECIPIENT repository variable.
Without that variable it uploads nothing.

The in-process driver now runs a second pass under PYTHONMALLOC=malloc,
because pymalloc pools hide small over-reads from AddressSanitizer. The
deep budget splits across the two passes, so the weekly job stays inside
the six-hour limit. The fuzz environments pin PYTHONHASHSEED, which tox
otherwise randomizes, so a find on a hash-dependent path replays.

Past finds now live in tests/fuzz_regressions, and every run replays
them first through every harness via the existing seed replay.
Every scheduled deep run used rng seed 0, so each one replayed the same
time-bounded mutation sequence and explored nothing new. The job now
passes its run number as the seed. The run number appears in the log,
so passing it back to --rng-seed reproduces a run.
The deep run pointed the JS harness at tests/serialize/js/_corpus, which
does not exist, so it never replayed a corpus file. The fixtures live in
tests/js/_corpus as JSON rows, so the driver writes each row's input to
its own file, the same extraction tools/js_sanitize.py uses.
The roundtrip and sanitize idempotence checks flagged two cases the
library handles as designed. Script text that opens "<!--" and then
"<script" before any "-->" breaks the HTML script content restrictions,
and its serialization reparses in the double-escaped state; the standard
warns such parser-made trees need not survive serialize and reparse.
Past 512 open elements the tree builder stops nesting, so reparsing the
flattened output nests it differently.

Each exemption applies only after a mismatch and only to those inputs:
script text matching the restriction's own grammar, or a tree that
reached the depth cap. Any other non-idempotent output still fails.
@gaborbernat
gaborbernat force-pushed the fuzz/harness-hygiene branch from 2b9c7e9 to 6238b23 Compare October 2, 2026 00:24
The scheduled run seeded its mutations with the public run number and
printed the mutation index and the full sanitizer report, so anyone could
regenerate an unfixed crasher or read where it faults. The seed is now
random per run and reaches fuzz.py only through the environment, the
log keeps the hash, length and harness, and the origin and sanitizer
reports go to the crash directory the workflow encrypts.
@gaborbernat
gaborbernat merged commit 7b2e38b into tox-dev:main Oct 2, 2026
51 of 52 checks passed
@gaborbernat
gaborbernat deleted the fuzz/harness-hygiene branch October 2, 2026 05:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant