Skip to content

The pre-push selection runs the tests that read the source tree a change touches #1650

Open

Description

@markramm

From the gate evaluation, kb/designs/gate-evaluation-2026-11.md (#1638), "The three changes", ticket 2. Filed at the maintainer's word (2026-11-17, the efficiency plan); written so a contributor can start from this issue alone.

Groom 2026-11-17

Automated groom (an AI agent working for the maintainer), copied from the evaluation as merged on dev 0eb1bcf. The doc's tables and grep commands are the evidence.

Observation. Ten red PR runs came from source-reading tests. Seven were
loop PRs, whose authors run the pre-push hook. Reproduced:

scripts/test-affected --list --files extensions/software-kb/src/pyrite_software_kb/board.py

selects 86 files and none of test_kb_reads_are_one_function,
test_link_reads_pass_a_scope, test_response_text_is_pyrites. Those are
the three that failed on #1553 in CI (runs 43985141356, 43988113479). For
pyrite/services/qa_service.py it misses three of the twelve checked.

Principle 8. The selection is the one place the rule "run what your
change can break" lives. Today the gap is covered by a row in the worker
skill (#1554), which is a rule each worker must remember, and outside
authors never read it.

User's model. "If my push passes locally, CI will not surprise me with
a rule about the source." CONTRIBUTING already promises this for guards.
Implementation model. Import reach cannot see a test that reads files.
A marker says which tree a test reads; a changed path under that tree
selects it.

Acceptance:   A change to any file under pyrite/, extensions/*/src/ or scripts/
              selects every test function that scans that tree. `--explain`
              says so. A test function that scans a source tree and carries no
              marker fails a completeness test. Fails today: a parametrised
              test over (changed file, scanning test) pairs, starting with the
              three above.
Regimes:      a deleted source file; a change only under tests/ (guards, as
              today); a full-suite trigger (unchanged); the stamp must still
              cover the selection.
Shapes:       56 files call ast.parse, rglob or inspect.getsource (grep below).
              Classify every scanning function: marked | not a scanner (the
              call parses test input) | out of scope and why. Two layouts: a
              file that is all scanners (test_layer_boundaries.py) and a
              behaviour file with one scanner (test_index_schema_drift.py: 39
              tests). Mark functions, not modules, as `guard` does. Trees read:
              pyrite/, extensions/*/src, scripts/, docs/, tests/.
Touches:      existing: scripts/test-affected, tests/test_test_affected.py,
              pyproject.toml (marker), the scanning test files (one decorator
              each), CONTRIBUTING.md (Guards), .claude/skills/pyrite-dev
              (the row becomes a pointer)   new: none
Sequence:     independent of ticket 1. After #1614 if both edit
              tests/test_cli_numeric_options_declare_a_range.py.
Model:        sonnet
heavy:        no
Cold read:    no (it only adds tests to a selection)
Out of scope: selection by execution reach (tests-q4). Registry-walking tests
              that import what they walk; import reach already finds them
              (checked: test_cli_numeric_options and
              test_every_entry_point_passes_the_policy are selected for a
              pyrite/ change).

Open question for the worker: one marker with an argument
(guard(reads="pyrite")) or a second marker. Predicted footprint: about 40
files, most of them one line; scripts/test-affected about 60 lines.
Changing a script widens that PR's matrix to three interpreters.

Related: #1554 (the worker-skill row this replaces with a pointer), #1553 (the PR whose CI runs showed the gap), #1614 (shares one test file).

Activity

  1. added this to the 0.26.4 milestone on Nov 17, 2026
  2. added
    qualityRefactoring, test refactoring, code health
    on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    qualityRefactoring, test refactoring, code health

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions