Skip to content

fix: a checker sees the item's own fields in a gate and in QA (#1587) - #1660

Draft
markramm wants to merge 1 commit into
devfrom
claude/gate-checker-property-failures-ddtum2
Draft

markramm wants to merge 1 commit into
devfrom
claude/gate-checker-property-failures-ddtum2

Conversation

@markramm

Copy link
Copy Markdown
Collaborator

Fixes #1587

Branch: this session was given its own branch name, claude/gate-checker-property-failures-ddtum2, which is used here instead of fix/1587-a-checker-sees-the-items-own-fields.

The plan is in the first comment below. The report will be appended here under "## Report".

Theme block (from the groom, 2026-11-17)

Acceptance:   A checker's verdict is the file's, in a gate and in QA, for the same item.
              tests/test_a_checker_sees_the_items_own_fields.py:
              - test_a_checker_gives_the_files_verdict[caller x checker x holds|fails]: parametrised over
                NAMED_CHECKERS.keys() x {gate via sw_check_ready, QAService.validate_entry, QAService.validate_kb};
                a holding item passes, a failing one fails, the same in each caller. Fails today
                (priority_present, has_field priority/status/assignee, has_any_field, has_tags, has_outlinks in the gate;
                priority_present, has_field priority/status/assignee, not_oversized, orphan_backlog_item in QA).
              - test_every_named_checker_has_a_case: the case table's keys == NAMED_CHECKERS.keys(), so a new checker
                cannot be added without a holding and a failing item.
              - test_has_field_reads_a_field_wherever_the_index_stores_it[location]: a metadata field (effort), each
                column the item type can carry (priority, status, assignee, a date), tags, and a GenericEntry
                protocol field; holds in gate and QA.
              - test_no_caller_builds_a_checker_entry_itself: AST/grep over pyrite/ and extensions/*/src. Only the
                builder module assigns _tag_count, _outlink_count or _links (and _DRY_RUN_ENTRY is the builder's).
              Still green: tests/test_named_rubric_checkers.py, test_rubric_evaluation.py, test_gates.py,
              test_board_config_refuses.py, test_board_config_is_read_by_one_function.py,
              test_gate_lookup_failure_keeps_its_error.py.
              Docs: kb/components/rubric-checkers.md states the entry a checker receives; docs/plugins.md one line.
              Changelog: changelog.d/1587-a-checker-sees-the-items-own-fields.fixed.md, naming both effects. Gates
              now pass these checkers for items that hold them. `pyrite qa validate` stops reporting missing
              priority/status/assignee, a missing parent epic, or an XL item with subtasks, for items that have them.
Regimes:      an index older than the file (a hand edit, no sync): the indexed verdict, as for every index read;
              no tags/links: counts 0, _links []; the builder's lookup fails in a gate: its own error, not board.yaml's (#1585);
              body_has_pattern timeout: gate and QA differ by design (#1538); the test uses patterns that do not time out.
Shapes:       12 checkers; 4 field locations (bag, 14 columns, tag table, both); 4 link shapes; 3 callers + dry run.
Touches:      existing: pyrite/services/qa_service.py, extensions/software-kb/src/pyrite_software_kb/plugin.py,
              extensions/software-kb/src/pyrite_software_kb/board.py, kb/components/rubric-checkers.md, docs/plugins.md
              new: pyrite/services/checker_entry.py, tests/test_a_checker_sees_the_items_own_fields.py, changelog.d fragment
Sequence:     independent of #1633, #1647 and #1614 (no shared file).
              PR #1628 (#1503, draft) rewrites rubric_checkers.py. Stay out of rubric_checkers.py (builder in a new
              module, no checker body edits) and the two land in either order.
Out of scope: no_open_blockers (#1585, merged). Plugin checkers in gates. QA's model-judged rubric. The
              GenericEntry double-write of protocol fields into metadata: the builder makes it harmless.
Decision:     none open.

Generated by Claude Code

Copy link
Copy Markdown
Collaborator Author

Plan (#1587)

1. The groom re-run on today's dev (0d04f49)

The groom's reproduction, re-run with a throwaway HOME, PYRITE_CONFIG_DIR and PYRITE_DATA_DIR: pyrite init --template software, then item-full (priority high, effort S, assignee, tags alpha/beta, a body [[link]]), epic-a (kind epic) and child (effort XL, priority low, tags alpha, subtask_of + has_subtask links to epic-a). One board.yaml gate holds 16 criteria, and the same 16 are a kb.yaml evaluation_rubric. Then index sync, sw check-ready <id> --format json, qa validate proj --entry <id> --format json and the whole-KB qa validate proj. ✗ = the checker says the criterion does not hold although it does. Δ = differs from the groom or was not run in it.

criterion item-full gate item-full QA child gate child QA
priority_present ✗ ✗ ✗ ✗
has_field priority ✗ ✗ ✗ ✗
has_field status ✗ ✗ ✗ ✗
has_field assignee ✗ (holds) ✗ (holds) fails, correct (none) fails, correct
has_field effort ok ok ok ok
has_field tags ✗ ✗ ✗ ✗
has_any_field [priority, assignee] ✗ ✗ Δ (QA not run in the groom) ✗ ✗ Δ
has_tags ✗ ok ✗ ok
has_outlinks ✗ ok ✗ ok
status_present, descriptive_title ok ok ok ok
not_oversized ok (S) ok ok ✗ "consider decomposing"
orphan_backlog_item fails, correct (no subtask_of) fails, correct ok ✗ "no parent epic"
body_has_section / _pattern / _code_block correct correct correct correct

The whole-KB sweep gives the same findings as --entry for both items Δ (the groom assumed this and did not run it). Every other cell matches the groom. Nothing in the groom is wrong on today's dev; #1553, #1585, #1611 and #1480 changed none of these paths.

2. Every place that calls a checker or builds its entry

Found with grep -rn "rubric_checkers|get_rubric_checkers|get_all_rubric_checkers|gate_checker|NAMED_CHECKERS|_DRY_RUN_ENTRY|_tag_count|_outlink_count|_links" over pyrite/ and extensions/*/src.

# place what it does class
1 plugin.py _evaluate_gate builds the gate's entry (dict(row) + meta + _links) fixed here: calls the builder once, outside evaluating. Guarded by test_a_checker_gives_the_files_verdict[gate-*] and by a new cell in test_gate_lookup_failure_keeps_its_error.py
2-6 _mcp_check_ready, _mcp_refine, _mcp_claim, _mcp_transition, _mcp_review SELECT * the row and hand it to _evaluate_gate; they build no entry guarded: test_every_gate_tool_gives_the_files_verdict[tool], parametrised from GATE_OF_TOOL (so a sixth tool needs a cell). Their bodies are not reshaped
7 board.py check_criterion dry run (_DRY_RUN_ENTRY) the load-time entry fixed here: it becomes the builder's empty_checker_entry(). Guarded by the structural test, plus a DEFAULT_BOARD_CONFIG control cell
8 qa_service.py _check_rubric_evaluation builds QA's entry (_tag_count, _outlink_count; no _links, narrow columns) fixed here: calls the builder. Guarded by the qa-entry and qa-kb cells
9 qa_service.py validate_entry SELECT narrow columns fixed here: SELECT * (see 4)
10 qa_service.py _check_named_rubric_all SELECT narrow columns fixed here: SELECT *
11 registry.get_all_rubric_checkers returns the functions; calls none safe by reading
12 cli/qa_commands.py _show_all_checkers, _show_kb_coverage list names and docstrings; call none safe by reading
13 plugin rubric checkers (get_rubric_checkers) run only in QA (8), so they get the built entry safe by reading: the same call site as 8; no shipped plugin has one
14 no_open_blockers its own lookup (#1585) out of scope
15 QA tier-2 model rubric (_collect_judgment_items, _evaluate_llm_rubric) skips every item with a checker safe by reading: no named checker runs there
16 QA surfaces: CLI qa validate and ci, MCP kb_qa_validate, REST qa.py/admin.py all go through QAService safe by reading: the same service as 8

test_no_caller_builds_a_checker_entry_itself (an AST scan of pyrite/ and extensions/*/src) fails if any module other than pyrite/services/checker_entry.py writes _tag_count, _outlink_count or _links into a dict, or defines a literal dict carrying them.

3. The builder

pyrite/services/checker_entry.py (new):

  • checker_entry(db, row) -> dict. row is a full entry row (SELECT *; a mapping or a sqlite3.Row). It reads:
    • the row's columns, copied as they are (title, body and status stay where the checkers read them today);
    • metadata: the parsed bag (a new dict, never the stored one), with each field column filled in where the bag has no non-empty value. The field columns are date importance status location assignee assigned_at priority due_date start_date end_date coordinates fips state lifecycle, plus title and summary. A test pins this list against the entry table's columns, so a new column needs a decision;
    • tags: the tag names from entry_tag JOIN tag, put into metadata["tags"] when there are any. _tag_count is their number;
    • _links: [{target, relation}] from link (the gate's shape today). _outlink_count is len(_links), the rows QA counts today.
    • Two queries through db.execute_sql. QA's two COUNT(*) become these two, so QA runs no extra query per entry.
  • empty_checker_entry() -> dict: the shape with no row, used by the dry run.
  • Projection order: the bag's value first, the column where the bag has none. That is the reverse of declared_over_bag. For a kb.yaml protocols: type, the bag holds the file's value as written and the column holds a lossy copy (location: [Paris, Lyon] becomes "Paris, Lyon"). For a plugin class there is no bag copy, so the column is the only reading. Truthiness is the same either way, so both kinds give one answer. This is in "Unsure".

4. QA's SELECTs

Both widen to SELECT * FROM entry WHERE .... _check_entry_fields and _validation_fields also read entry in validate_entry. _validation_fields copies every non-None row key into the fields it validates and then overrides _TYPED_COLUMNS from the file source, so the extra columns could change schema findings. To keep schema validation byte-identical, validate_entry keeps its narrow SELECT for those checks, and _check_rubric_evaluation gets the full row: one SELECT * by id, run only when the type has a rubric. The sweep's _check_named_rubric_all feeds only the rubric, so it simply widens to SELECT *. If a test shows the narrow row is not needed in validate_entry, I'll use one SELECT * there instead and say so.

5. Files, against the groom's Touches line

The groom's 8, and no others: pyrite/services/checker_entry.py (new), pyrite/services/qa_service.py (_check_rubric_evaluation and the two SELECTs only), plugin.py (_evaluate_gate only), board.py (_DRY_RUN_ENTRY only), kb/components/rubric-checkers.md, docs/plugins.md, tests/test_a_checker_sees_the_items_own_fields.py (new), and the changelog fragment. Also tests/test_gate_lookup_failure_keeps_its_error.py (a cell, as the brief asks) and the backlog item.

6. Findings, for the maintainer to read twice

  • A column can hold the model's default, not something the file wrote. A backlog_item with no priority in its file has priority = "medium" in its column (and status = "proposed"; a task gets priority = 5). pyrite get shows the same priority: medium. After this change, priority_present and has_field priority therefore hold for every backlog item in a gate, as status_present already does today for status. On dev, these checkers fail for every backlog item, those with a priority included. I read this as the item's verdict as Pyrite reads it (the same answer as get), not as a checker-logic or index change, so it is not a stop. If you read "the file's verdict" as "only what the file wrote", that answer can come only from a storage change (the index would have to record which fields the file wrote), which is on the stop list. The test states it with a control cell.
  • has_field will see tags, title and summary. It will not see aliases, sources or links (the link checkers read _links). Truthiness is unchanged (priority: 0 is missing). All of this goes in kb/components/rubric-checkers.md.
  • _evaluate_gate keeps its meta parameter so the five callers keep their shape, but no longer reads it: the builder parses the row's own metadata, which is what every caller passed.

7. Tests (red first)

tests/test_a_checker_sees_the_items_own_fields.py runs on a real KB (tests/_world.py), with real index sync, the real MCP sw_check_ready and the other gate tools, and the real QAService. Expected verdicts are written by hand. A whole-KB qa validate timing on a 2,000-entry throwaway KB, before and after, goes in the report.


Generated by Claude Code

@markramm

Copy link
Copy Markdown
Collaborator Author

Conductor answer to the plan on #1660 (#1587), 2026-11-17. Go ahead as planned. No stop.

  • §6 finding 1 (a column holds the model's default): I agree with your reading. A checker's verdict is the item as Pyrite reads it, the same value pyrite get shows, and status_present already works that way today. "Only what the file wrote" would need the index to record which fields the file wrote. That is a storage change, outside this theme. Do these four things:
    1. Pin it with the control cell you describe: a backlog_item with no priority in its file passes priority_present and has_field priority in the gate and in QA, with the reason in the test's docstring.
    2. Say it in kb/components/rubric-checkers.md and docs/plugins.md in one sentence each: for a type whose model defaults a field, has_field and priority_present hold when the file leaves the field out, because the item has that value as Pyrite reads it.
    3. In the changelog fragment, list the checkers that now pass for every item of a type with a defaulted field (priority_present and has_field priority on backlog_item; task's priority). Do not claim that "checkers now see only what the file says".
    4. Keep it in your report's "Unsure". It names a design point, so an Opus cold read of the branch follows your report. I will file the follow-up at landing ("a checker cannot tell a defaulted field from a written one"; backlog, no milestone).
  • §4: Keep the narrow SELECT for schema validation in validate_entry, as planned. If you widen it, show from a test that the schema findings are byte-identical.
  • §2: The AST guard and the GATE_OF_TOOL parametrisation are what this theme needs.
  • Commits: every commit must pass the commit hooks (never --no-verify on commit). A fix: commit must touch tests/. Otherwise use test:, refactor: or docs:.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

In a gate, has_tags, has_outlinks, priority_present and has_field priority fail for an item that has them

2 participants