operational-rigor §4: a check's name is not its coverage - #57
Merged
Conversation
A named check, test, or CI job earns evidentiary weight only from its assertion body. Incident shape: a check whose name implied it gated a model integration turned out to exercise only a regex pre-filter where the model's name was a routing label — the session had to correct a safety claim already given to the user. Rule: read what the check actually asserts before citing it as evidence a change is safe. Ships unprobed with Provenance per the covenant.
PR F-e-u-e-r#57 round-1 gate (grok-4.5 high: PROCEED; gpt-5.6-sol max: FIX 1). codex F1 reproduced and addressed: evidentiary weight now comes from the full trace to the pass/fail oracle - assertions (rule set + inputs for assertion-less job types), invocation path and setup, and whether that path executed in the cited run; identical assertions with different drive paths named as the discriminating case; uninspectable trace = coverage unverified, said so. Examples updated to the trace form. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhLu6NCGzM9HxnqNb87MAx
… pinned to the trace PR F-e-u-e-r#57 round-2 gate (grok-4.5 high: FIX 1; codex r2 pending). grok F1 reproduced: a costume-complying executor could trace, cite, and still over-claim the name-implied property. Fixed: assert only what the trace established; name-implied remainder is unverified and said so; third neg example pins the post-trace over-claim. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhLu6NCGzM9HxnqNb87MAx
PR F-e-u-e-r#57 round-2 codex FIX 1 reproduced: the pos example now names the cited run's execution observation; a run-skipped near-miss neg example added (static coverage is not the cited run's coverage). Body diffstat to be corrected (+27, was claimed +20). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhLu6NCGzM9HxnqNb87MAx
PR F-e-u-e-r#57 round-3 gate (grok-4.5 high: PROCEED; gpt-5.6-sol max: FIX 1,2). Both reproduced and addressed: - the trace is inspected at the revision the cited run actually used; a current definition plus an old run's log no longer combines into coverage (codex F1) - the trace must show the assertions PASSED in that run with their failure controlling the check's final status, tied to the file's own runs/passes/correct line (codex F2); 'said so' -> 'say so' (codex n3) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhLu6NCGzM9HxnqNb87MAx
…-e-u-e-r#57 r4 grok F1) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhLu6NCGzM9HxnqNb87MAx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds one rule to operational-rigor §4 (verify by observation): a check's name is not its coverage — before citing a named check, test, or CI job as evidence a change is safe, read its assertion body and cite what it actually asserts.
Why
A named gate reads as coverage of whatever its name implies. Incident shape: a session presented a CI check as gating a model integration's behavior; reading the check's source showed it exercised only a regex pre-filter in which the model's name was merely a routing label — and the session had to correct a safety claim it had already given the user. "There is a check called X" is a claim about naming, not behavior; the rule ships with a ✅/❌ pair making the citation form concrete.
Covenant status
unprobed, private incident cited as shape per the README covenant's second branch; Provenance names the unrun probe (sample named checks, diff name-implied vs actual assertion coverage) as debt.Checks
Single-file diff (grown across review rounds; see trail comment). Added-lines unicode sweep: only house-style chars (§ — ✅ ❌).
python3 .github/checks.py: all checks passed.