Two checks run on every pull request that changes backend code. Each answers a question the other cannot:
| Check | Question | Blind spot |
|---|---|---|
| Diff coverage | Is every changed line run by some test? | Code that runs but whose result no test checks |
| verify-red | Do the PR's new tests notice the change as a whole? | Which part of the change each test notices (per-hunk revert, planned for 0.27) |
Both checks are advisory: neither is in gate's needs, so neither blocks a
merge. verify-red reports a verdict table and a summary line on every backend
PR, and CI does not pass it --enforce-fix (a flag for local use that makes the
script exit 1 for a fix-typed PR with no test, 0 red or an unexpected pass).
Gating waits for #1104's model, because the claim rule below is not yet sound
for every way a test gets its cases. What verify-red does do on its own (#1074):
- It grades against the merge commit's first parent (
HEAD^1), so a PR merged since yours was opened is not graded as yours. - It refuses a
@pytest.mark.controlthat the test did not declare on itself or one of its parameters (the job goes red; nothing is blocked). - Each PR's numbers go into a
test-evidenceartifact.
scripts/verify-red.sh # against origin/dev (else dev)
scripts/verify-red.sh --json e.json # also write the evidence fileThe runner (scripts/verify_red_ci.py) never modifies your checkout:
- It checks out the merge base with the base branch into a throwaway
git worktree --detachunder$TMPDIR. - It copies in your test-side files: everything under
tests/,extensions/*/tests/, and anyconftest.py, committed or not. - It runs only the tests you added or edited. That is the run without the fix. A test counts as edited when its AST changed, so reformatting a test does not count.
- It copies in the rest of your change, so the throwaway tree now matches your working tree, and runs the same tests again. That is the run with the fix.
- It removes the throwaway tree.
Nothing is ever restored, because nothing you own was changed. If a run is
killed outright, it can leave a stale directory under $TMPDIR. Once that
directory is gone, the next run prunes its worktree registration.
Both runs put the throwaway tree first on PYTHONPATH, and the runner refuses
to run (exit 2) if a package in that tree imports from somewhere else, such as
another checkout's editable install (#189). A top-level package or extension
that only the PR adds is stubbed out in the run without the fix, so the head's
editable install cannot serve it there. Both runs also set
PYTHONDONTWRITEBYTECODE, so the second run can't import stale bytecode from
the first. Both get a scratch HOME, PYRITE_CONFIG_DIR, PYRITE_DATA_DIR
and XDG_* directories, so a new test running against buggy code cannot
write to your real config. HF_HOME still points at your model cache.
A PR whose only non-test changes are changelog fragments, kb/ entries or
Markdown has nothing to verify. Those files are still left out of the run
without the fix, since a test may read them. A pytest run with the fix that
collects no test file at all (a conftest or plugin that breaks before
collection) is the check failing: exit 2, with pytest's last lines in the
summary. A file that collects but runs nothing, such as one that skips itself
when an optional dependency is missing, makes its tests n/a instead.
The output is one line, then a table of every test that is not red:
verify-red: 3 red · 1 import-only · 0 unexpected pass · 0 n/a (12 pre-existing tests in these files not run)
- red: the test passes with the fix and fails without it. This is the evidence a review wants.
- import-only: the test fails without the fix only because the code it needs
is missing. Either its file does not collect, or it raises
ImportError,AttributeErrororNameErrornaming an identifier the PR adds. The runner finds those identifiers by diffing the ASTs of the changed code. It reads the exception object itself, never the traceback, so the repo's--tb=shorthas no effect. This verdict is weak: it shows the test needs your code, not that it checks what the code does. - unexpected pass: the test passes without the fix, so it does not test the
change. In CI it gets a warning annotation. For a test you mean to pass
either way, such as a "this still works" guard, add
@pytest.mark.control(reason="..."), or the marker plus a docstring that says why. It then counts as control. A bare marker is rejected and the test stays an unexpected pass. The marker is read from what pytest records for each test cell; see "What acontrolmarker may be" under Known limits. - inherited: the test passes without the fix and its control marker was already on that cell -- the same function and parameter values -- at the merge base (the runner does a collect-only run of the base's test files). Not a failure, and not evidence: the PR still needs a red test of its own.
- n/a: no claim. The test failed or skipped with the fix, skipped without it, was not collected, or its run timed out.
Tests the PR did not add or edit are counted, not listed. A test is also
selected when a module-level table or helper its decorators use changed (a new
parametrize row); then only the cells whose parameter values the base did not
have are graded. Workers run
scripts/verify-red.sh once before reporting and paste the summary line. A
reviewer reads the CI job's line instead of running it again.
These are not fixed; weigh the line with them in mind.
-
Red through a wrapper is weaker evidence. A
TypeErrorfrom a keyword argument the PR adds, a failure wrapped in anExceptionGroup, or a failure that aCliRunner,TestClientor MCP wrapper turns into a status code or an output string all count as red, not import-only. The runner only reads the exception that reaches pytest. -
A conftest that imports a name the fix adds stops pytest before any test runs without the fix, so every selected test in that run is n/a.
-
A SIGKILLed or SIGTERMed driver orphans its pytest child, which keeps running in the throwaway tree until it finishes. Your checkout is still untouched.
-
Any Markdown counts as not code. A PR that changes only a skill,
CONTRIBUTING.mdor other.mdfile, plus a test that pins its text, has nothing to verify. -
What a
controlmarker may be, and what is caught. It counts only when written on the test function or on one of its parameters (pytest.param(marks=...)), however the marker object was built (a helper, a module constant,_grid()). The runner reads each cell's markers from pytest and refuses, with an error naming the file, the origin and the test, acontrolthat came from anywhere else: a class decorator, a module or classpytestmark(a constant too), apytest_collection_modifyitemshook, an autouse fixture'srequest.applymarker, orapplymarker/add_markerin the test body or a teardown. A refused marker is no exemption either: the cell is an unexpected pass. One the merge base already had on the same cell is inherited, not an error. Not caught: a hook that rewrites the function's ownpytestmarklist before pytest builds the items, which looks exactly like a decorator; and a refusable mark whose base copy was not collectable (the file then counts as new). -
The claim rule is sound only for module-level tables. A cell is a test function and its parameter values, not a node id: the runner records each cell's
callspec.paramsat collection, at the base and at the head, and a cell is the PR's claim when that pair is absent at the base (whatever itsid, wherever it sits; a reordered table claims nothing). That is what it does for a module-level table named in the test function's own decorators (CASES = [...],@pytest.mark.parametrize("a, b", CASES)), including a helper that builds one in the same module. What it does not handle:- Inline
parametrizelists (the common shape here) change the test function, so the whole function is graded, old cells included. - Class-level parametrization, fixture-params
(
@pytest.fixture(params=...)), generate-hook parametrization (pytest_generate_tests) and imported tables change no test function in the file, so a changed row selects nothing and is reported as no test added or edited.
Gating waits for #1104's model. The base is collected from its own copy of the test files (capped at 60 s); a file that does not collect there is named in the summary and all its cells count as new. A value whose
reprcarries a memory address is compared without it; a value whosereprdiffers between runs reads as a new row. - Inline
-
A fix that changes only test data (a golden, a fixture file) is graded by the test files whose source names that file: every test in them is claimed. If no test names it, the summary says
no test reads <file>(and--enforce-fixwould fail it). -
verify-redruns on pull requests only. It is skipped in the merge queue (merge_group). It is not ingateat all (advisory), and it does not re-run on a title or body edit: thepull_requestevent is not asked foredited, since that cancels and restarts all of CI. -
CI's environment decides. The
verify-redjob has no Postgres service, so every[postgres]cell is n/a there. On a machine withPYRITE_TEST_PG_URLset those cells run and can pass without the fix, so a local run can show more unexpected pass than CI does (#1053 showed 2 locally and none in CI). What CI prints is the line a reviewer reads, and your local line is evidence, not the verdict; to see CI's line, unsetPYRITE_TEST_PG_URL. -
Any docstring satisfies
control. The runner can't tell a docstring that says why the test passes without the fix from one that only says what it tests. -
Only packages with an
__init__.pyare stubbed. A top-level module, a namespace package or an extension without asrc/layout that only the PR adds can still be imported in the run without the fix. That produces a false unexpected pass, never a false red. A new extension reached through the plugin registry is red, not import-only, since the registry wraps the import error. -
One scratch
HOMEserves both runs, so the run without the fix can leave files the run with the fix sees. It also hides~/.gitconfigand the Playwright browser cache, so tests that need them are n/a. -
A conftest or plugin hook that crashes after collection reads as n/a, not exit 2, and so does a new test deselected by the default
-mfilter (slow,e2e). Both show "not collected", which isn't the real reason. Either way the line claims nothing, so afix:PR with it still goes back for having no red test (#409). -
Stale-tree cleanup assumes absolute
gitdirpaths. Withworktree.useRelativePathsset, a live run's registration could be removed. The cleanup also ignoresgit worktree lock.
On a pull request, the 3.12 leg of the test job runs the suite under
pytest --cov, with 3.12's low-overhead sysmon tracer. Then
diff-cover coverage.xml --compare-branch=<base> --fail-under=80 writes the
changed lines that no test runs to that job's step summary. A result below 80%
produces a warning annotation, never a failure. When the PR changes no measured
line, diff-cover reports 100%; the evidence file records that as null, and
the summary prints n/a.
The verify-red job waits for test, whatever its result, and writes
test-evidence.json (artifact test-evidence, kept 90 days):
{
"pr": 393,
"head": "<the PR's head sha>",
"merge_commit": "<the merge commit CI checked out>",
"merge_base": "<sha>",
"fix_commits": 1,
"verify_red": {"red": 3, "import-only": 1, "unexpected pass": 0, "n/a": 0,
"control": 0, "inherited": 0, "pre-existing": 12},
"diff_coverage": {"percent": 87.5, "lines": 40, "uncovered": 5}
}verify_red is null when there was nothing to verify. That happens when the
PR changes only test-side or inert files, has an empty diff, or has code but
no new or edited test. The summary names the kind of change and the file counts.
diff_coverage is null
when the test job produced no report. To collect the file:
gh run download <run-id> -n test-evidence.
A third CI job, experimental, is not about the PR's own tests: it keeps the
tests of experimental features (#657) running without letting them block a
merge.
- What is experimental: the surfaces
kb/designs/alpha-supported-surface.mdmarks experimental.tests/experimental_surface.pymaps them to test paths and node ids, and the rootconftest.pyapplies theexperimentalmarker from it. Security properties are never experimental; that file lists them andtests/test_experimental_surface.pychecks both lists against a real collection. - The gate: the
testjob (3.12 on a pull request, 3.13 in the merge queue, 3.11 and 3.12 on the push todevthat follows) runs-m "not slow and not e2e and not experimental".gateneedstest, notexperimental. - The ratchet:
experimentalruns-m "experimental and not slow and not e2e"on Python 3.12 with Postgres, on every backend PR and every push todev, and hands the JUnit report toscripts/experimental_ratchet.py check. The job is red only for news: a failure in neithertests/experimental_known_failures.txtnor an openexperimental-brokenissue (the job reads them withissues: read); a PR that adds a line to the list, which can only shrink; or a run that did not complete (no report, or a pytest exit status other than 0 or 1; the run step times out before the job so the ratchet still runs). A listed test that passes or no longer exists is named for removal, and a filed test that passes again is named so its issue can be closed. - Never silently: on a push to
dev, wheneverexperimentaldid not succeed (failed or cancelled), theexperimental-issuesjob (the only job withissues: write, using the workflow's token) opens or comments on anexperimental-brokenissue: per test file with a new failure, for a grown known-failures list, and for a run that did not complete, including one that left no result at all. On a PR the news is in the job summary only. - Collection errors: a file that fails to import stops the core run unless every test in it is experimental by the mapping; then the core run warns and continues, and the experimental job reports it as a failure of that file.
Run either set locally:
scripts/heavy_lock.py -- .venv/bin/pytest tests/ extensions/ -n 4 -m "not slow and not e2e and not experimental"
scripts/heavy_lock.py -- .venv/bin/pytest tests/ extensions/ -n 4 -m "experimental and not slow and not e2e"refresh-test-durations.yml checks the latest successful dev merge-group
CI run weekly and on manual dispatch. It downloads that run's merged
test-durations and all three junit shard reports. Above the same 25% spread
threshold used by the shard report, it uploads a proposed .test_durations.gz
and its evidence as the proposed-test-durations artifact. Balanced,
unchanged or missing evidence never silently rewrites the baseline.
A separate job opens or updates one issue, ci: Test duration refresh evidence, with the source run, old/new greedy least-duration estimates and the artifact link. A contributor can download and review the proposed baseline before opening a normal PR and running CI. No branch is pushed and Actions does not need permission to create pull requests. Artifact retention follows the repository setting; the issue links the latest proposal.
These are estimates, not a collection replay or a wall-clock promise. The
refresh never changes test selection. The parsing job has read-only contents
and Actions permissions; only the notification job has issues: write.
Only trusted dev code runs; downloaded timing/report artifacts are data.