|
| 1 | +The efficacy review that `docs/agent_failure_modes.md` §9 committed to when |
| 2 | +mitigation 6 shipped (PyAutoBrain#140, live 2026-07-17): after the ship series, |
| 3 | +has the falsified-by checkpoint stage gone rote? **Verdict: not proven rote — |
| 4 | +proven unobservable, which is its own finding. Keep the stage, vocabulary |
| 5 | +unchanged; make the reviewer's engagement with each lifted claim part of the |
| 6 | +recorded verdict so a rote pass becomes ledger-visible.** |
| 7 | + |
| 8 | +- completed: 2026-08-18 |
| 9 | +- origin: spun out of PyAutoBrain#130 at its 2026-08-15 close (the one §9 item |
| 10 | + still open); executed as a dashboard-work research session on branch |
| 11 | + `claude/automind-falsified-by-checkpoint-cmsqsi` (PyAutoBrain + PyAutoMind) |
| 12 | +- deliverables: Outcome block under mitigation 6 + §9 close-out in |
| 13 | + PyAutoBrain `docs/agent_failure_modes.md`; follow-up prompt |
| 14 | + `draft/feature/pyautobrain/review_claim_dispositions.md`; this record |
| 15 | + |
| 16 | +## Instrument validation first (the method note's D1 guard) |
| 17 | + |
| 18 | +Before trusting any firing-rate number, the claim-lifter was re-exercised on |
| 19 | +2026-08-18: a probe text with three known load-bearing claims ("proven |
| 20 | +byte-identical", "no-op", "does not affect") lifts 3/3 through the live |
| 21 | +`load_bearing_claims()`, and the 5 pinning tests in |
| 22 | +`PyAutoBrain/tests/test_review_claims.py` pass (8/8 with the in-place-resolve |
| 23 | +tests). A low firing rate below is therefore a fact about the inputs, not a |
| 24 | +dead instrument. |
| 25 | + |
| 26 | +## The evidence base |
| 27 | + |
| 28 | +Two sources, honestly scoped: |
| 29 | + |
| 30 | +1. **The autonomy log ship series, 2026-07-17 → 2026-08-01** — every `--auto` |
| 31 | + ship gate since go-live: 21 distinct tasks ran a review leg (jax-compile-time-research, |
| 32 | + inject-keck, inject-alma-simobserve, cold-compile-reduction, |
| 33 | + autotune-off-default, pix-nonfinite-localisation, |
| 34 | + interpolator-aggregator-test-mode, multistart-gradient-auto-convergence |
| 35 | + ph1+ph2, multistart-cadence-int-cast, delaunay-nan-callback, |
| 36 | + multistart-cadence-followups, python-312-floor 1A/1B/1C/1D/1E/4A/4B/4C/4D), |
| 37 | + plus one August cloud-session faculty run recorded in |
| 38 | + [[autohands-firewall-allowlist]] — **22 review-leg gates**, past the ~10-ship |
| 39 | + trial window. August ships after 08-01 were largely interactive/cloud |
| 40 | + sessions the autonomy log does not row; they are not counted either way. |
| 41 | +2. **Retro-measurement of the claim matcher over real shipped history** — the |
| 42 | + ReviewSurface is ephemeral (nothing persists what it lifted per ship), so |
| 43 | + firing rate was re-derived by running the live matcher over the squash-merge |
| 44 | + messages on `origin/main` since 2026-07-17 in the two repos available to the |
| 45 | + session. This is a proxy for the branch-message surface (squash messages |
| 46 | + carry the PR body, not always the full branch log) and covers organism |
| 47 | + repos only, not the library ships — disclosed, not hidden. |
| 48 | + |
| 49 | +## 1. Firing rate: neither empty nor saturated |
| 50 | + |
| 51 | +| repo | main commits since 2026-07-17 | with ≥1 lifted claim | |
| 52 | +|---|---:|---:| |
| 53 | +| PyAutoBrain | 50 | 13 (26%) | |
| 54 | +| PyAutoMind | 66 | 3 (5%; registry moves dominate) | |
| 55 | + |
| 56 | +Trigger distribution over the 26 lifted claim-lines: `verified` 17, |
| 57 | +`unchanged` 5, `identical` 3, `byte-identical` 2, `no-op` 1. The vocabulary is |
| 58 | +not too narrow (it fires regularly) and not too broad (74–95% of ships surface |
| 59 | +nothing). |
| 60 | + |
| 61 | +The interesting shape: `verified` dominates, and it mostly lifts the author's |
| 62 | +**evidence sentence** ("Verified they actually bite: reverting only |
| 63 | +_feature.py fails 7 of the new tests"), not a bare unsupported claim. Shipped |
| 64 | +commit messages in this window conspicuously carry "Verified by/against |
| 65 | +<probe>" inline — the claim culture mitigation 6 wanted. A zero finding rate |
| 66 | +is therefore at least partly deterrence, not only non-engagement. |
| 67 | + |
| 68 | +## 2. Finding rate: zero — and the record cannot say why |
| 69 | + |
| 70 | +Across the 22 review-leg gates: `unverified-claim` FINDINGS raised: **0**. |
| 71 | +Grep across the whole Mind (completion records, autonomy log, active/) finds |
| 72 | +the category name only in the stage's own shipping records and this prompt. |
| 73 | + |
| 74 | +FINDINGS the review leg did raise in the window were generic-correctness |
| 75 | +(2026-07-27 multistart-cadence-int-cast: `int()` truncation → infinite loop, |
| 76 | +caught by the run's own review pass 1) or came from **external** adversarial |
| 77 | +reviews (Codex gpt-5.6-sol on 07-27, Claude Opus 5 on the 07-29 |
| 78 | +python-312-floor merges) — not from step 2a. |
| 79 | + |
| 80 | +Distinguishing "unnecessary" from "rubber-stamped": the honest answer is the |
| 81 | +ledger cannot distinguish them, and that is the review's central finding (§5). |
| 82 | + |
| 83 | +## 3. Were any load-bearing? One documented exercise; it held |
| 84 | + |
| 85 | +Exactly one autonomy-log row records the adversarial claim pass operating on a |
| 86 | +lifted claim: 2026-07-21 interpolator-aggregator-test-mode — *"adversarial |
| 87 | +pass on 'no-op outside test mode' claim — gated by is_test_mode()+.exists(), |
| 88 | +proven by off-switch test"*. The claim had a falsified-by basis; correctly no |
| 89 | +finding; the outcome was unchanged. No ship was held and no correction was |
| 90 | +produced by the stage in the window. Its per-ship cost is also ≈ zero (a |
| 91 | +stdlib regex plus a few surface lines), so "earning its cost" is a low bar — |
| 92 | +but the earning is currently invisible. |
| 93 | + |
| 94 | +Sharpest counter-datum: the one confirmed-wrong load-bearing claim of the |
| 95 | +window — the 2026-07-27 "5 siblings affected" count, falsified to 2 by the |
| 96 | +external Codex review — lived in an **issue comment**, outside the |
| 97 | +commit-message surface the stage reads. The stage could not have caught the |
| 98 | +one escape that actually happened. (Widening the scanned surface to issue |
| 99 | +text was considered and not recommended: issue prose is where hedged |
| 100 | +discussion belongs; the boundary-crossing record the stage guards is the |
| 101 | +branch itself.) |
| 102 | + |
| 103 | +## 4. Idle-phrasing exclusion: holding, with two cheap residuals |
| 104 | + |
| 105 | +No changelog/rename chatter is lifted (the pinning test's contract holds on |
| 106 | +real data). Residual false positives observed in the retro-measurement: |
| 107 | +mid-sentence fragments of wrapped prose (line-based matching lifts "…row, so |
| 108 | +the yardstick is real data from the same machine. Verified by"), and narrative |
| 109 | +`identical`/`unchanged` describing the *bug*, not the change ("a night the |
| 110 | +gate correctly stopped looked identical"). Cost is seconds of reviewer |
| 111 | +attention per ship with **no bypass pressure** — unlike the F5 refusal class, |
| 112 | +an over-lifted line cannot train bypass-by-default because nothing blocks. |
| 113 | +Within budget; no vocabulary change recommended. |
| 114 | + |
| 115 | +## 5. Rubber-stamping: the signature is undetectable as instrumented |
| 116 | + |
| 117 | +The rote signature — claims lifted, reviewed CLEAN, no evidence cited — was |
| 118 | +looked for and **cannot be confirmed or refuted**: the ReviewSurface is |
| 119 | +ephemeral, and 20 of the 22 gates recorded only "review CLEAN" / "review |
| 120 | +self-CLEAN". A healthy pass and a rote one write the identical ledger row. |
| 121 | +Two gates only (07-21 above; the August cloud run) left evidence the surface |
| 122 | +was engaged at all. Additionally, on autonomous ships the "reader" that |
| 123 | +enforcement was delegated to is the branch's own author (`review self-CLEAN`) |
| 124 | +— the design's reader-enforcement premise is diluted exactly where the stage |
| 125 | +matters most. |
| 126 | + |
| 127 | +## Verdict |
| 128 | + |
| 129 | +**Keep, vocabulary unchanged; close the observability gap.** The instrument |
| 130 | +is live and calibrated, the one documented exercise worked as designed, the |
| 131 | +false-positive cost is negligible, and the claim culture it targets has |
| 132 | +visibly moved toward evidence-inline commit messages. But a stage whose |
| 133 | +engagement leaves no trace will go rote silently if it has not already; per |
| 134 | +the campaign's own ranking (deleting beats detecting beats reminding), the |
| 135 | +fix is to move it from remind-shaped to detect-shaped: the reviewing agent's |
| 136 | +verdict gains a **one-line disposition per lifted claim** — `claim → |
| 137 | +basis-cited <what> | idle | FINDING` — carried into the ship evidence |
| 138 | +(autonomy-log review cell / PR body). A bare CLEAN over a non-empty claims |
| 139 | +surface then reads as drift in the ledger, which is the reader-enforcement |
| 140 | +the original design promised. Filed as |
| 141 | +`draft/feature/pyautobrain/review_claim_dispositions.md` (small, safe). |
| 142 | + |
| 143 | +## Original prompt |
| 144 | + |
| 145 | +# Has the falsified-by checkpoint stage gone rote after ten ships |
| 146 | + |
| 147 | +Type: research |
| 148 | +Target: PyAutoBrain |
| 149 | +Repos: |
| 150 | +- PyAutoBrain |
| 151 | +Difficulty: small |
| 152 | +Autonomy: safe |
| 153 | +Priority: normal |
| 154 | +Status: formalised |
| 155 | + |
| 156 | +## What this is |
| 157 | + |
| 158 | +The efficacy review that `docs/agent_failure_modes.md` §9 committed to when |
| 159 | +mitigation 6 shipped: *"trial on the next ship series, review whether it went |
| 160 | +rote after ~10 ships."* Spun out of PyAutoBrain#130 when that issue was closed |
| 161 | +(2026-08-15) — it was the one §9 item still genuinely open, and nothing tracked |
| 162 | +it. |
| 163 | + |
| 164 | +This is an investigation producing a written verdict from evidence. No code |
| 165 | +change is committed up front; a fix may follow from the finding. |
| 166 | + |
| 167 | +## Background |
| 168 | + |
| 169 | +Mitigation 6 (PyAutoBrain#140, merged 2026-07-17, live) made the review faculty |
| 170 | +lift **load-bearing empirical claims** out of a branch's commit messages into |
| 171 | +the `ReviewSurface` as `claims to falsify` — the trigger vocabulary is `no-op`, |
| 172 | +`byte-identical`, `does-not-affect`, `proven`, `behaviour-preserving`. `AGENTS.md` |
| 173 | +step 2a then makes an unsupported one a FINDING of kind `unverified-claim`. |
| 174 | + |
| 175 | +Its design was deliberately reader-enforced rather than an author checklist, and |
| 176 | +scoped to load-bearing phrasing only, precisely so it could not decay into the |
| 177 | +"remember-to-run checklist" the campaign's own constraints ban. It targets the |
| 178 | +A5/F3 failure class — confident-wrong effect-claims. |
| 179 | + |
| 180 | +## Why it needs reviewing |
| 181 | + |
| 182 | +The doc's constraint list bans checklists as a mechanism, and a routine |
| 183 | +adversarial pass is the single mechanism most likely to decay into one. The |
| 184 | +worry is explicit in the shipping comment: *"the one that needs care not to |
| 185 | +become the banned checklist."* A stage that fires on every ship and is waved |
| 186 | +through every time is worse than no stage, because it also carries false |
| 187 | +assurance. |
| 188 | + |
| 189 | +## What to investigate |
| 190 | + |
| 191 | +Over the real ship history since 2026-07-17: |
| 192 | + |
| 193 | +1. **Firing rate** — on how many ships did `claims to falsify` populate at all, |
| 194 | + versus come back empty? An always-empty surface means the vocabulary is too |
| 195 | + narrow; an always-full one means it is too broad. |
| 196 | +2. **Finding rate** — how many `unverified-claim` FINDINGS were actually raised, |
| 197 | + and what happened to each? A stage that never produces a finding across ~10 |
| 198 | + ships is either unnecessary or being rubber-stamped; distinguish those two. |
| 199 | +3. **Were any load-bearing?** For each finding, did falsifying the claim change |
| 200 | + the outcome — a correction, a held ship — or was it cosmetic? This is the |
| 201 | + measure of whether the stage is earning its per-ship cost. |
| 202 | +4. **Idle-phrasing exclusion** — is the load-bearing-only scoping holding, or has |
| 203 | + the matcher started lifting incidental prose? Check for false positives of |
| 204 | + the kind that trained bypass-by-default in the guard's first hour (the F5 |
| 205 | + cost column). |
| 206 | +5. **Rubber-stamping** — look for the signature: claims lifted, reviewed CLEAN, |
| 207 | + no evidence cited in the review. That is the rote failure, and it looks |
| 208 | + identical to a healthy pass unless you read what the reviewer actually did. |
| 209 | + |
| 210 | +## Deliverable |
| 211 | + |
| 212 | +A verdict with the numbers behind it, and one of: keep as-is, narrow/broaden the |
| 213 | +trigger vocabulary, or retire the stage. If the finding is "it went rote", say |
| 214 | +what would fire instead — per the campaign's own ranking, deleting the |
| 215 | +possibility beats detecting it, and detecting beats reminding. |
| 216 | + |
| 217 | +## Method note |
| 218 | + |
| 219 | +Validate the instrument before trusting it: check that the review faculty's |
| 220 | +claim-lifting still runs on a branch with known load-bearing claims before |
| 221 | +concluding anything from a low firing rate. A null result that looks like a |
| 222 | +finding (D1) is the exact failure this campaign catalogued. |
| 223 | + |
| 224 | +<!-- formalised by the Intake (Conception) Agent on 2026-08-15 from user-intake --> |
0 commit comments