Add script to generate fuzz coverage #3718
Conversation
|
👋 I see @jkczyz was un-assigned. |
|
If needed I can also push my script to generate corpus from |
There was a problem hiding this comment.
I wonder if we should include this coverage data in our codecov output in PRs...On the one hand its not "reliable" coverage data in that we aren't actually testing the behavior, but on the other hand we are at least hitting the code lines and presumable would find crashes. WDYT?
contrib/generate_fuzz_coverage.sh
Outdated
|
|
||
| if [ -d "$TARGET_CORPUS_DIR" ]; then | ||
| echo "Running cargo llvm-cov for target: $TARGET_NAME with corpus $TARGET_CORPUS_DIR" | ||
| if ! CARGO_TARGET_DIR=./target cargo +${TOOLCHAIN} llvm-cov run --no-report --manifest-path "$FUZZ_DIR/Cargo.toml" --target "$HOST_TRIPLE" --features "$COMBINED_FEATURES" --bin "$TARGET_NAME" -- "$TARGET_CORPUS_DIR" -runs=1; then |
There was a problem hiding this comment.
Rather than running the fuzz target specifically, can we just use cargo llvm-cov to run the tests? For each binary in the fuzz tests we already support running cargo test and it'll pull all the files from fuzz/test_cases/X and run it through the fuzz processor. Seems easier than relying on the libfuzzer logic.
There was a problem hiding this comment.
Not sure about that. Running on test_cases with cargo test would remove the fuzziness since now it will be run test_cases and wont run the fuzzer but since its coverage report we are basically doing the same thing. I would have to check on this.
There was a problem hiding this comment.
Someone doing fuzzing can presumably copy their corpus from wherever it is to test_cases. In the comment you note that we currently expect a corpus in corpus, but it could well be any folder name?
There was a problem hiding this comment.
I have changed the script to use cargo test instance now. Surprisingly reduced the code by 99%. Although I don't see the test_cases in any branch . Maybe you can try running the script and provide some feedback since I dont know the format on how test_cases directory has the inputs.
There was a problem hiding this comment.
Right, test_cases isn't automatically created anywhere, its just read by the tests if it exists. Not sure what to do about that wrt the script here, maybe we should have it look for the llvm-libfuzzer, afl, and honggfuzz dirs, copy them to test (if its empty) and then run cargo test?
|
👋 The first review has been submitted! Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer. |
I would prefer to not add it in codecov until we have a good public corpora of fuzz inputs to run the report on. |
Signed-off-by: Prabhat Verma <[email protected]>
Prabhat1308
force-pushed
the
pv/add_cov_script
branch
from
c45d613 to
8f005a0
Compare
This script generates the fuzz coverage in a html file when run .
Considerations
corpusdirectory infuzzdirectory.base32_target.rswould be undercorpus/base32_target.libFuzzerit may/may not work with corpus generated from other fuzzers.How to run
# from the root directory chmod +x contrib/generate_fuzz_coverage.sh contrib/generate_fuzz_coverage.sh