test(corpus): make the CORPUS_DIR gates report when they skip - #2
Merged
Conversation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
h4x0r
marked this pull request as ready for review
August 5, 2026 20:27
core/tests/corpus.rs stacked two silent skips in every corpus test:
let Some(dir) = corpus_dir() else { return }; // CORPUS_DIR unset
let path = dir.join("dynamic.vhd");
if !path.exists() { return; } // fixture absent
Neither says anything. `cargo test` prints ok whether the VHD reader was validated
against the corpus or never opened a file, and those two outcomes are not the same
claim.
Both halves now go through forensic-testgate::gated_file, which emits the notice
from inside the resolver — there is no quiet variant to reach for, so the notice
cannot be dropped in a later edit. A CORPUS_DIR pointing at a path that does not
exist now FAILS rather than skipping: setting the variable is the operator saying
"run this test", so a missing corpus is a misconfiguration.
This `corpus_dir()` helper is cloned verbatim across qcow2/vhd/vhdx/vmdk; this is
the vhd instance.
55 tests pass; clippy --all-targets --all-features -D warnings clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The `forensic-testgate` dev-dependency pointed at a machine-local scratch directory. `cargo metadata` failed on every other machine, so every CI job died at manifest load before running a single check. `forensic-testgate 0.1.0` published to crates.io on 2026-08-02, so the registry form the TODO was waiting on is now available and the TODO is removed rather than reworded — its blocker no longer exists. A git-pinned revision was the alternative, and was rejected: a git dep carries no publisher record, so cargo-vet would need an `[[exemptions]]` entry where a `trust` entry applies, and the dep would have to be swapped a second time once the registry version landed. Supply chain: the crate is ours and published by h4x0r, which is ADR-0018 mechanism (2), so it gets `[[trusted.forensic-testgate]]` scoped to `safe-to-run` (it is a dev-dependency) rather than the weaker exemption. The loud-skip contract is unchanged — with the gating variable unset the tests still announce the skip by name, and `FORENSIC_TESTGATE_STRICT=1` still turns each skip into a failure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
h4x0r
force-pushed
the
testgate/loud-skips
branch
from
August 5, 2026 22:40
9f4241c to
b726e63
Compare
forensic-testgate is the crate this branch adds to make the CORPUS_DIR gates report when they skip, so it is new to the vet store. It is ours, published to crates.io by h4x0r, which is ADR-0018 mechanism (2) — a trust entry, not an exemption pinned to 0.1.0 that would go stale on its next release. Verified by control: removing the trust entry makes cargo vet --locked fail naming forensic-testgate, restoring it passes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dependency form — resolved
The
forensic-testgatedev-dependency was apathdep pointing at a machine-local scratchdirectory, so
cargo metadatafailed on every other machine and every CI job died at manifestload before running a single check.
forensic-testgate0.1.0 published to crates.io on2026-08-02, so the dep is now the registry form this PR originally promised:
A git-pinned revision was the alternative, and was rejected. A git dep carries no publisher record,
so cargo-vet would need an
[[exemptions]]entry where atrustentry applies — the weakermechanism, which ADR-0018 treats as a defect when a stronger one is available — and the dep would
have had to be swapped a second time once the registry version landed.
Supply chain: the crate is ours and published by
h4x0r, which is ADR-0018 mechanism (2), so itgets
[[trusted.forensic-testgate]]scoped tosafe-to-run(it is a dev-dependency).The loud-skip contract is unchanged — with
CORPUS_DIRunset the tests still announce the skip byname, and
FORENSIC_TESTGATE_STRICT=1still turns each skip into a failure.Why
core/tests/corpus.rsstacked two silent skips in every corpus test:cargo testprintsokwhether the VHD reader was validated against the real corpus or never opened a file. Those are not the same claim, and the suite could not tell you which one you got.This is one instance of a fleet-wide pattern: a sweep found 216 env-gated test skip sites that produce no output at all. This exact
corpus_dir()helper is cloned verbatim inqcow2-forensic,vhdx-forensic, andvmdk-forensic; this is the vhd instance. Companion PR: SecurityRonin/qcow2-forensic#4.What changed
Both halves now resolve through
forensic_testgate::gated_file. The notice is emitted by the resolver, on the library side of the seam, so there is no quiet variant to reach for and a later edit cannot drop it.CORPUS_DIRThe third is the deliberate design call: setting the variable is the operator saying run this test, so a path that is not there is a misconfiguration, not a skip. The escape hatch is leaving the variable unset.
Verification
Full suite: 55 passed.
clippy --all-targets --all-features -D warningsclean.Separate finding — a gate that has never fired in CI
Not fixed here, to keep this diff reviewable.
core/tests/qemu_differential.rs:19and:98hardcode the oracle path:That is the macOS-ARM Homebrew path only, so on the Linux CI runner it never exists and the qemu differential tests silently skip on every CI run — they have never validated anything in CI.
qcow2-forensic/core/tests/real_images.rsalready fixed this exact bug by searching/opt/homebrew,/usr/localand/usr/bin, with a comment noting the hardcoded path "made the oracle tests skip on the Linux CI runner, dropping coverage." The same defect is live invhdx-forensicandqcow2-forensic'sforensic/crate.Worth its own PR, along with a loud tool-resolution gate so the next occurrence announces itself.
🤖 Generated with Claude Code