fix(measurement): purge __pycache__, so a run measures the tree it just rewrote

[lywinged]

Two commits. #59 took the import-guard half of this branch; what remains is the cache purge, plus tests for the guards themselves — which nothing was checking, including the one #59 landed.

---

1. Purge __pycache__, so a run measures the tree it just rewrote

Status.FAIL and Status.PASS are the same length, so rewriting a site leaves the file size unchanged and a run can end up measuring the previous iteration's bytecode. The failures that iteration caused are then attributed to the site currently under mutation: margins come out larger than they are, and a check held by exactly one test is reported as comfortably covered.

Two fresh clones, one at 424c2f8 and one at this commit, each run starting from a purged cache, nothing else touching either directory:

424c2f8       margin-1 watch list:  0, 0, 0, 2
this commit   margin-1 watch list:  8, 8, 8, 8

The list has eight entries — REPORT.md names them and calls them "the number to watch". On main that watch list came out empty three times in four. That is the wrong direction to be wrong in: the run reports a healthier suite than the one that exists.

The second symptom needs no numbers. After four runs on 424c2f8:

$ pytest -q
14 failed, 147 passed, 5 xpassed

$ git status --porcelain -- src/
                                  # nothing

Every mutated source file was restored, and the restoration is verified. The interpreter was still loading bytecode compiled from a mutated one. REPORT.md said the checkout was "restored and re-verified green after every mutation" — true of the files, not of what Python then executed. Corrected here.

2. Tests for the guards

This came out of an audit: revert each fix landed in these two repositories and count what fails. Ten of eleven have a test that notices. The one that does not is #59 — delete assert_suite_imports_this_checkout() from main() and the suite stays green at 161 passed.

measurement/ had no tests at all, so every guard in the script was in exactly the state the script exists to detect.

Thirteen tests, one broken precondition each, against a synthetic checkout — the smallest tree the script accepts, carrying one guarded check and one unguarded one so the reporting path is covered and not only the aborts.

Load-bearing, measured rather than claimed. Removing each guard and re-running:

Guard removed	Tests that fail
checkout not found	3
no FAIL sites	2
refusing to guess the line	1
import path (#59)	2
green baseline	1
__pycache__ purge	2
restore after each site	3

Every one of the thirteen fails under at least one removal. The purge accounts for two, one of them the reporting control — the synthetic tree reproduces the cache contamination rather than only asserting the directory is gone.

Two things this turned up while writing it

A test named for the "refusing to guess" guard was reaching "no FAIL sites" instead: its fixture bound Status.FAIL to a name, so the walk never collected the site. Renamed, given a fixture that reaches the guard, and the original behaviour kept as its own test — a Finding whose status arrives through a name is not collected, and the run reports on what it found without mentioning the omission. Nothing in this repository writes them that way and the site count is checkable against grep -c Status.FAIL, so it is recorded rather than fixed, and the test fails if that stops being true.

Run as a subprocess rather than imported, because the script resolves TRACE_TESTS at module scope and importing it would bind that to this repository. Exit codes are not asserted alone: a refusal exits 1, and so does a completed run that found an unverified check.

---

Also in this diff

• REPORT.md's guard list now covers both the guard measurement: refuse to report when the suite imports a different tree #59 added and the purge; it had neither.
• The module docstring says five guards and lists five. measurement: refuse to report when the suite imports a different tree #59 added a fourth without updating the count, so main currently reads "Three things this script guards against" over four guards.
• The reproduction block said # expect 118 passed, 5 xpassed. This tree produces 174.

Verification

Fresh clone, in a directory that has never held the repository: 174 pass, 5 xpassed, identical across two consecutive runs. ruff clean on the new file, and the same 68 pre-existing errors either side of this change.

The A/B measurement above was re-run from scratch after an earlier attempt was invalidated — I had run an unrelated git checkout inside the directory the experiment was using, which swapped the script under test mid-run. Given what this PR is about, that seemed worth saying rather than quietly re-reporting.

[github-actions]

❔ Contributor Check: UNKNOWN

Check	Result
Profile	UNKNOWN
Credential	LOW
Overall	UNKNOWN

Automated check by AgenTrust Contributor Check.

[imran-siddique]

The cache purge is the important half of this and I want it. Eight runs on one unchanged tree returning 8, 5, 2, 0, 5, 2, 5, 5 is a harness that cannot be cited, and the reason is exactly as you say: Status.FAIL and Status.PASS are the same length, so a rewrite leaves the file size untouched and the interpreter is free to load the previous iteration's bytecode.

The part that matters most is your point that it fails upward. A guard that inflates margins reports unguarded checks as verified, and this repo's whole output is a conformance claim, so that is the one direction it must never fail in. The REPORT.md note about restoration is the same insight and worth keeping: git status clean while pytest fails against stale bytecode is a genuinely confusing state to leave behind.

One thing to change. I merged #59 a few minutes ago, which adds the same import guard as assert_suite_imports_this_checkout. You wrote the two independently and they collide, so this branch now conflicts.

Could you rebase onto main and drop verify_import_path? That leaves _purge_caches and the REPORT.md updates, which is the right scope. Keep your REPORT.md wording on the cache guard, it is better than what #59 carried, and the 161 passed correction should come along too.

One difference worth knowing: #59 tests expected == imported.parent or expected in imported.parents, where yours tests only the second. Both are right for a normal layout.

[lywinged]

Force-pushed onto 424c2f8 and narrowed. #59 took the import-path guard, so this is now only the cache purge, and the title and description are updated to match.

Worth flagging one thing that changed with it: assert_suite_imports_this_checkout as merged is better than what this branch had, because it checks the probe's return code. The version here was silent if the interpreter could not import trace_tests at all. I dropped mine and kept yours rather than reconciling them.

The A/B numbers in the description are freshly measured on two clones at 424c2f8 and at this commit. They are not the numbers that were here before — the earlier run was taken on the pre-#59 tree, and a second attempt was invalidated when I ran an unrelated git checkout inside the directory the experiment was using, which swapped the script under test mid-run. Given what this PR is about, that seemed worth saying rather than quietly re-reporting.

[imran-siddique]

The cache purge occurs before every subprocess suite run, directly addressing same-size source rewrites and pytest bytecode caches. The synthetic harness covers the positive reporting path, each refusal guard, restoration, and cache removal; the earlier import-guard conflict is resolved.