From 396c2ae0b5c26d1e5ccd586d8f37c6247ad5f9a9 Mon Sep 17 00:00:00 2001 From: firaen22 Date: Wed, 22 Jul 2026 10:09:47 +0800 Subject: [PATCH 1/6] =?UTF-8?q?operational-rigor=20=C2=A74:=20a=20check's?= =?UTF-8?q?=20name=20is=20not=20its=20coverage?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A named check, test, or CI job earns evidentiary weight only from its assertion body. Incident shape: a check whose name implied it gated a model integration turned out to exercise only a regex pre-filter where the model's name was a routing label — the session had to correct a safety claim already given to the user. Rule: read what the check actually asserts before citing it as evidence a change is safe. Ships unprobed with Provenance per the covenant. --- skills/operational-rigor/SKILL.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/skills/operational-rigor/SKILL.md b/skills/operational-rigor/SKILL.md index 82300cc..d840747 100644 --- a/skills/operational-rigor/SKILL.md +++ b/skills/operational-rigor/SKILL.md @@ -255,6 +255,17 @@ When rigor conflicts with finishing sooner, rigor wins. - Between failed fixes, return to a clean state; stacked half-fixes hide causes. - Reproduce reported bugs before fixing. Fix the observed failure, not the implied one. Refutation is valid: report confirmed non-bugs and ship nothing. +- **A check's name is not its coverage** (`unprobed` — private incident as + shape; see Provenance). A named gate earns evidentiary weight only from + its assertion body: one session cited a check whose name implied it gated + a model integration's behavior, then read its source and found it + exercised only a regex pre-filter in which the model's name was a routing + label — and had to correct a safety claim already given to the user. + Before citing a check, test, or CI job as evidence a change is safe, read + what it actually asserts and cite that; "there is a check called X" is a + claim about naming, not behavior. + ✅ "read check X: it asserts A and B but never drives C — C is unverified." + ❌ "the change is safe, check X covers it" (named, never read). - **A failing check has two suspects: the code and the check itself.** Before editing either, open the statement of intended behavior (spec, README, docstring, type) and confirm which side it backs; a disagreement is the @@ -538,6 +549,15 @@ the deployed path (contributor-reported shape; the private repo is verifiable by the contributor, not linkable here). It ships `unprobed` — the pack's private fixtures have no interactive arm to drive it (cf. the grill-pass note above); the marker records that debt, not an exemption. +The §4 check-name rule (2026-07-22) comes from a private incident: a +session presented a named CI check as gating a model integration's +behavior, then read the check's source and found it exercised only a +regex pre-filter in which the model's name was merely a routing label, +and had to correct the safety framing it had already given the user. +Private evidence, cited as shape per the README covenant's second branch; +the executable probe — sample a repo's named checks and diff name-implied +vs actual assertion coverage — has not been run; the in-body `unprobed` +marker records that debt. Stable behavioral rules; the environment-specific facts to re-verify now travel with the rules that cite them — the external-systems set in `references/external-systems.md`, plus §2's mount-check commands From 6bb98582f9729ac2188471a523b48e5f3232f428 Mon Sep 17 00:00:00 2001 From: F-e-u-e-r <189464303+F-e-u-e-r@users.noreply.github.com> Date: Wed, 22 Jul 2026 18:40:58 +0800 Subject: [PATCH 2/6] operational-rigor: r1 refinement of the check-name rule PR #57 round-1 gate (grok-4.5 high: PROCEED; gpt-5.6-sol max: FIX 1). codex F1 reproduced and addressed: evidentiary weight now comes from the full trace to the pass/fail oracle - assertions (rule set + inputs for assertion-less job types), invocation path and setup, and whether that path executed in the cited run; identical assertions with different drive paths named as the discriminating case; uninspectable trace = coverage unverified, said so. Examples updated to the trace form. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GhLu6NCGzM9HxnqNb87MAx --- skills/operational-rigor/SKILL.md | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/skills/operational-rigor/SKILL.md b/skills/operational-rigor/SKILL.md index d840747..e8480a0 100644 --- a/skills/operational-rigor/SKILL.md +++ b/skills/operational-rigor/SKILL.md @@ -256,15 +256,22 @@ When rigor conflicts with finishing sooner, rigor wins. - Reproduce reported bugs before fixing. Fix the observed failure, not the implied one. Refutation is valid: report confirmed non-bugs and ship nothing. - **A check's name is not its coverage** (`unprobed` — private incident as - shape; see Provenance). A named gate earns evidentiary weight only from - its assertion body: one session cited a check whose name implied it gated - a model integration's behavior, then read its source and found it - exercised only a regex pre-filter in which the model's name was a routing - label — and had to correct a safety claim already given to the user. - Before citing a check, test, or CI job as evidence a change is safe, read - what it actually asserts and cite that; "there is a check called X" is a - claim about naming, not behavior. - ✅ "read check X: it asserts A and B but never drives C — C is unverified." + shape; see Provenance). A named gate earns evidentiary weight from what + it asserts AND what it actually drives: one session cited a check whose + name implied it gated a model integration's behavior, then read its + source and found it exercised only a regex pre-filter in which the + model's name was a routing label — and had to correct a safety claim + already given to the user. Before citing a check, test, or CI job as + evidence a change is safe, trace it through to its pass/fail oracle — + the assertions (or, for a linter or build job, its rule set and + inputs), the invocation path and setup that feed them, and whether + that path executed in the cited run — and cite what the trace showed; + two checks with identical assertions differ when one drives the real + integration and the other a pre-filter. A trace you cannot inspect + leaves that coverage unverified — say so. "There is a check called X" + is a claim about naming, not behavior. + ✅ "traced check X: it asserts A and B against the real adapter, but + nothing in its path drives C — C is unverified." ❌ "the change is safe, check X covers it" (named, never read). - **A failing check has two suspects: the code and the check itself.** Before editing either, open the statement of intended behavior (spec, README, From 24ca35076cd65c912e6883a140c1bc15de32a210 Mon Sep 17 00:00:00 2001 From: F-e-u-e-r <189464303+F-e-u-e-r@users.noreply.github.com> Date: Wed, 22 Jul 2026 18:45:25 +0800 Subject: [PATCH 3/6] operational-rigor: r2 refinement of the check-name rule - claim scope pinned to the trace PR #57 round-2 gate (grok-4.5 high: FIX 1; codex r2 pending). grok F1 reproduced: a costume-complying executor could trace, cite, and still over-claim the name-implied property. Fixed: assert only what the trace established; name-implied remainder is unverified and said so; third neg example pins the post-trace over-claim. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GhLu6NCGzM9HxnqNb87MAx --- skills/operational-rigor/SKILL.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/skills/operational-rigor/SKILL.md b/skills/operational-rigor/SKILL.md index e8480a0..f06a347 100644 --- a/skills/operational-rigor/SKILL.md +++ b/skills/operational-rigor/SKILL.md @@ -265,14 +265,19 @@ When rigor conflicts with finishing sooner, rigor wins. evidence a change is safe, trace it through to its pass/fail oracle — the assertions (or, for a linter or build job, its rule set and inputs), the invocation path and setup that feed them, and whether - that path executed in the cited run — and cite what the trace showed; - two checks with identical assertions differ when one drives the real - integration and the other a pre-filter. A trace you cannot inspect - leaves that coverage unverified — say so. "There is a check called X" - is a claim about naming, not behavior. + that path executed in the cited run — and assert only the properties + that trace established: whatever the check's NAME implies but the + trace did not show stays unverified, said so; two checks with + identical assertions differ when one drives the real integration and + the other a pre-filter. A trace you cannot inspect leaves that + coverage unverified — say so. "There is a check called X" is a claim + about naming, not behavior. ✅ "traced check X: it asserts A and B against the real adapter, but nothing in its path drives C — C is unverified." ❌ "the change is safe, check X covers it" (named, never read). + ❌ "read it — it's a regex pre-filter, but the name says integration, + so the integration is covered" — a trace read and then overridden by + the name. - **A failing check has two suspects: the code and the check itself.** Before editing either, open the statement of intended behavior (spec, README, docstring, type) and confirm which side it backs; a disagreement is the From ddf62af733bae452a02839ffdac7051633c8b85f Mon Sep 17 00:00:00 2001 From: F-e-u-e-r <189464303+F-e-u-e-r@users.noreply.github.com> Date: Wed, 22 Jul 2026 18:50:51 +0800 Subject: [PATCH 4/6] operational-rigor: r2 codex refinement of the check-name rule examples PR #57 round-2 codex FIX 1 reproduced: the pos example now names the cited run's execution observation; a run-skipped near-miss neg example added (static coverage is not the cited run's coverage). Body diffstat to be corrected (+27, was claimed +20). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GhLu6NCGzM9HxnqNb87MAx --- skills/operational-rigor/SKILL.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/skills/operational-rigor/SKILL.md b/skills/operational-rigor/SKILL.md index f06a347..d7c5a6d 100644 --- a/skills/operational-rigor/SKILL.md +++ b/skills/operational-rigor/SKILL.md @@ -272,9 +272,13 @@ When rigor conflicts with finishing sooner, rigor wins. the other a pre-filter. A trace you cannot inspect leaves that coverage unverified — say so. "There is a check called X" is a claim about naming, not behavior. - ✅ "traced check X: it asserts A and B against the real adapter, but - nothing in its path drives C — C is unverified." + ✅ "traced check X: it asserts A and B against the real adapter, and + run 1234's log shows that path executed; nothing in its path drives + C — C is unverified." ❌ "the change is safe, check X covers it" (named, never read). + ❌ "read the source — it asserts A — so the cited run covers A" (the + run had that test conditionally skipped; static coverage is not the + cited run's coverage). ❌ "read it — it's a regex pre-filter, but the name says integration, so the integration is covered" — a trace read and then overridden by the name. From d3af0fdb0668924fb6812545736491996a8833a9 Mon Sep 17 00:00:00 2001 From: F-e-u-e-r <189464303+F-e-u-e-r@users.noreply.github.com> Date: Wed, 22 Jul 2026 18:57:35 +0800 Subject: [PATCH 5/6] operational-rigor: r3 refinements to the check-name rule PR #57 round-3 gate (grok-4.5 high: PROCEED; gpt-5.6-sol max: FIX 1,2). Both reproduced and addressed: - the trace is inspected at the revision the cited run actually used; a current definition plus an old run's log no longer combines into coverage (codex F1) - the trace must show the assertions PASSED in that run with their failure controlling the check's final status, tied to the file's own runs/passes/correct line (codex F2); 'said so' -> 'say so' (codex n3) Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GhLu6NCGzM9HxnqNb87MAx --- skills/operational-rigor/SKILL.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/skills/operational-rigor/SKILL.md b/skills/operational-rigor/SKILL.md index d7c5a6d..0d48cc2 100644 --- a/skills/operational-rigor/SKILL.md +++ b/skills/operational-rigor/SKILL.md @@ -264,17 +264,21 @@ When rigor conflicts with finishing sooner, rigor wins. already given to the user. Before citing a check, test, or CI job as evidence a change is safe, trace it through to its pass/fail oracle — the assertions (or, for a linter or build job, its rule set and - inputs), the invocation path and setup that feed them, and whether - that path executed in the cited run — and assert only the properties + inputs) inspected at the revision the cited run actually used, the + invocation path and setup that feed them, whether that path executed + in the cited run, and whether its assertions PASSED there with their + failure controlling the check's final status (a run is not a pass — + the runs/passes/correct line above) — and assert only the properties that trace established: whatever the check's NAME implies but the - trace did not show stays unverified, said so; two checks with + trace did not show stays unverified — say so; two checks with identical assertions differ when one drives the real integration and the other a pre-filter. A trace you cannot inspect leaves that coverage unverified — say so. "There is a check called X" is a claim about naming, not behavior. - ✅ "traced check X: it asserts A and B against the real adapter, and - run 1234's log shows that path executed; nothing in its path drives - C — C is unverified." + ✅ "traced check X at run 1234's revision: it asserts A and B against + the real adapter; the run's log shows that path executed and A, B + passed with failures propagating to the job status; nothing in its + path drives C — C is unverified." ❌ "the change is safe, check X covers it" (named, never read). ❌ "read the source — it asserts A — so the cited run covers A" (the run had that test conditionally skipped; static coverage is not the From ed3bad47772b92cd75a3d5ed86b6a575bd3e2638 Mon Sep 17 00:00:00 2001 From: F-e-u-e-r <189464303+F-e-u-e-r@users.noreply.github.com> Date: Wed, 22 Jul 2026 19:00:59 +0800 Subject: [PATCH 6/6] operational-rigor: fix the runs/passes/correct pointer direction (PR #57 r4 grok F1) Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GhLu6NCGzM9HxnqNb87MAx --- skills/operational-rigor/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/skills/operational-rigor/SKILL.md b/skills/operational-rigor/SKILL.md index 0d48cc2..03019a9 100644 --- a/skills/operational-rigor/SKILL.md +++ b/skills/operational-rigor/SKILL.md @@ -268,7 +268,7 @@ When rigor conflicts with finishing sooner, rigor wins. invocation path and setup that feed them, whether that path executed in the cited run, and whether its assertions PASSED there with their failure controlling the check's final status (a run is not a pass — - the runs/passes/correct line above) — and assert only the properties + the runs/passes/correct line later in this section) — and assert only the properties that trace established: whatever the check's NAME implies but the trace did not show stays unverified — say so; two checks with identical assertions differ when one drives the real integration and