Commit 643f322
committed
fix(ai): refuse to refresh the duel-suite baseline from a failing run
`--refresh-baseline` ran the suite and wrote the result to the baseline path
without ever looking at the result's own verdicts. Because the baseline is what
every later run is compared against, one refresh from a red run blesses that
failure permanently: the next run compares equal to the blessed report, the
comparison reports no drift, and the gate exits 0 forever while the matchup is
still broken. Nothing in the file inspected `SuiteStatus` on the refresh path.
This is the other half of the policy phase-rs#7026 left open. That PR made an existing
matchup going non-Fail -> Fail fail the comparison, and reported a still-failing
matchup on every run rather than passing quietly — but it deliberately Warned
rather than Failed on the already-blessed case, on the grounds that the exit code
answers "did this change make things worse" and a baseline, however it got that
way, already sanctions its own contents. The question that PR filed rather than
smuggled in is whether a baseline may bless a failure at all. It may not.
`SuiteReport::failing_matchups` reports the matchups that failed their own
`Expected` check, judged with no reference to any baseline. That is a different
question from `CompareReport::any_fail`, which asks whether a change made things
worse than the baseline; this asks whether a run is fit to *become* the baseline.
It returns the matchups rather than a bool because the refusal is only actionable
if it can name which matchup failed and quote its `fail_reason` — the call site
would otherwise have to re-filter to say anything useful.
Only `SuiteStatus::Fail` disqualifies a run. `Open` does not, and the distinction
is load-bearing rather than incidental: `Expected::Open` is how a matchup declares
it has no verdict yet, and that declaration belongs in the suite definition where
it is visible and reviewable, not smuggled in by committing a red baseline. An
implementation keyed on `!= Pass` would conflate the two and make `Expected::Open`
unusable, which is why there is a test whose only job is to fail against it.
`Open` in fact has two producers — `grep`ed to confirm exactly two construction
sites outside tests — and review caught the doc comment claiming one:
`classify` returns `Open` for any matchup with zero games before it ever inspects
`Expected`. That exposed a second way to write an unfit baseline, from the
opposite side — a run that measured nothing. Comparison pairs by seed, so a
gameless baseline scores zero on every axis forever and the drift signal dies as
quietly as a blessed-red one. `SuiteReport::recorded_games` disqualifies it, and
deliberately counts games rather than testing for all-`Open`: a suite whose
matchups are all declared `Expected::Open` still plays real games, and that report
IS a usable baseline, because the paired-comparison arm decides on `games`, not on
`status` — stated that way rather than "never reads `status`", which is true here
but false on phase-rs#7026, cited two paragraphs above, where the paired arm gains status
tiers. Zero games is what makes a baseline inert; all-`Open` is not.
The routes are enumerated without claiming the enumeration is complete, because an
earlier draft said "two routes" and review found a third. `--games 0` is rejected at
parse time, since the existing error string already promised a positive integer and
`usize` alone does not. A `--suite-filter` selecting no matchups is caught on the
report, because `run_suite` does not reject an empty selection. `failed_result`
yields an empty `games` vector alongside `SuiteStatus::Fail`, which the failure
guard reports first because it names the actual setup error. And `SuiteOptions::new`
does not validate `games_per_matchup`, so a library caller can build a zero-game run
without touching the CLI at all.
No override flag. The escape hatch already exists one layer up and is the correct
layer, so adding a second one at the baseline would only let a caller bypass the
more visible mechanism. Verified non-bricking before choosing absolute refusal:
the committed baseline is 3/3 `Pass`, so no workflow depends on a blessed failure.
CI blast radius is nil, measured rather than assumed: `refresh-baseline` appears in
no workflow. CI runs `cargo ai-gate --games 10` and `--full-suite --games 100`,
both compare-only. Refresh is a local human operation, so this cannot break a
pipeline — it can only stop a person committing a blessed-red baseline.
The failing case is pinned to the run that motivated it rather than to invented
data: `the_recorded_failing_run_is_disqualified_as_a_baseline` transcribes the
recorded gate run (`.ab/noC-1.json`, the A+B+D leg of phase-rs#6969) — red-mirror and
affinity-mirror `Pass`, enchantress-mirror `Fail` carrying its verbatim reason
`mirror imbalance: p0=0.10, Wilson 95% CI [0.02, 0.40] excludes 0.50`. That is the
exact report a refresh would have blessed. It is transcribed rather than loaded
because the artifact is untracked and a test that read it would fail in CI.
Evidence. Nine mutants, tree restored byte-identical after each. Kill counts are
transcribed from the runs, not summarised — an earlier draft of this paragraph
asserted three of them from memory and review measured all three wrong.
On `failing_matchups`: dropping the filter is killed by 5 tests; returning nothing
by 1 (`the_recorded_failing_run_is_disqualified_as_a_baseline`); and the plausible
`!= SuiteStatus::Pass` by 3 — every test that asserts an `Open` matchup is not a
failure. That last one had been written up as killing only
`an_open_matchup_is_not_a_failure` "surviving the other two entirely", which was
simply false: the gameless and all-`Open` fixtures kill it too. The named test is
still the one that states the intent, but it is not the only thing standing between
that mutant and green, and claiming otherwise oversold a single test.
On `recorded_games`, with the constant stated because it decides the answer:
replacing the body with `0` is killed by 1 test, with `1` by 3, with `2` by 3.
The earlier draft said "making it constant fails all three of its tests" — true
only for `1`, and false for `0`, which is precisely the value the guard tests.
Counting non-`Open` matchups instead of games is killed by 1, the fixture written
for that conflation.
Two further mutants were found SURVIVING and are the reason
`an_all_open_run_that_played_games_is_still_a_usable_baseline` now carries an
uneven fixture. Counting matchups-with-games, and summing `games.len().min(1)`,
both returned the right answer for every fixture in the suite, because each
matchup carried exactly one game — so the total always equalled the matchup count
and "sum of games" was never distinguished from "number of matchups that played".
Against the real committed baseline (3 matchups x 10 games) those mutants return 3
where the contract says 30. The fixture now plays two games in one matchup and one
in the other, and asserts 3; both mutants die.
One further mutation is reported precisely rather than counted, because review
caught an earlier draft overstating it: dropping the `fail_reason` passthrough is
NOT a mutation of the predicate. `failing_matchups` yields `&MatchupResult`, so no
change to it can drop that field; the only reachable site is the test helper. It
shows the assertion genuinely reads the field, and pins the iterator's item type
against narrowing to `&str`, but it is not predicate coverage and is not counted as
such.
Because no unit test can reach a binary's `main`, the wiring was proven end to end,
two-sided, against a temporary baseline — the committed baseline was never written.
Positive control: a clean run still refreshes (red-mirror `PASS`, exit 0, baseline
written at sha256 `42701dbeac46b015…`), so the guard does not false-positive. True
positive: with `classify`'s mirror arm forced to Fail and the binary rebuilt, the
same command printed `refusing to refresh …: 1 matchup(s) failed their own suite
check` followed by the matchup and its reason, exited 1, and left that baseline at
sha256 `42701dbeac46b015…` — byte-identical, the overwrite prevented.
Both new refusals were then exercised through the CLI alone, with no source
mutation, which is stronger evidence than the forced-Fail arm: `--games 0` is
rejected at parse time without running the suite, a `--suite-filter` matching no
matchups is refused with nothing written, and a real two-game run still refreshes
as the positive control. The binary under test was verified to contain both refusal
strings first, so no arm can pass against a stale build.
The two refusals are ordered failures-first, and the order is load-bearing rather
than cosmetic: the conditions are not exclusive. `failed_result` builds a matchup
with an empty `games` vector AND `SuiteStatus::Fail`, so a run whose deck payloads
all fail to load satisfies both, and checking gamelessness first would replace each
matchup's `setup error: …` with a sentence about seeds. Nothing is lost by the
chosen order, because a merely gameless run — a `--suite-filter` matching nothing —
has no failing matchups to report.
One surface is deliberately left uncovered and is stated rather than implied: no
test executes the binary, so the refusal block itself — as opposed to the two
predicates behind it — can be deleted or inverted with the whole suite green. That
also means the ordering above is argued from the code path rather than pinned by a
test; producing a broken deck-payload tree to exercise it end to end was judged not
worth the fixture. The
end-to-end runs above were performed against this tree but are not committed as
tests. A process-spawning integration test would have to run a real suite to reach
the guard, which is minutes of CI for a local-only human command, so the trade is
made knowingly rather than overlooked.
Assisted-by: ClaudeCode:claude-opus-51 parent f7c4469 commit 643f322
2 files changed
Lines changed: 222 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
72 | 72 | | |
73 | 73 | | |
74 | 74 | | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
75 | 122 | | |
76 | 123 | | |
77 | 124 | | |
| |||
137 | 184 | | |
138 | 185 | | |
139 | 186 | | |
140 | | - | |
141 | | - | |
142 | | - | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
143 | 195 | | |
144 | 196 | | |
145 | 197 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
146 | 146 | | |
147 | 147 | | |
148 | 148 | | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
149 | 194 | | |
150 | 195 | | |
151 | 196 | | |
| |||
1082 | 1127 | | |
1083 | 1128 | | |
1084 | 1129 | | |
| 1130 | + | |
| 1131 | + | |
| 1132 | + | |
| 1133 | + | |
| 1134 | + | |
| 1135 | + | |
| 1136 | + | |
| 1137 | + | |
| 1138 | + | |
| 1139 | + | |
| 1140 | + | |
| 1141 | + | |
| 1142 | + | |
| 1143 | + | |
| 1144 | + | |
| 1145 | + | |
| 1146 | + | |
| 1147 | + | |
| 1148 | + | |
| 1149 | + | |
| 1150 | + | |
| 1151 | + | |
| 1152 | + | |
| 1153 | + | |
| 1154 | + | |
| 1155 | + | |
| 1156 | + | |
| 1157 | + | |
| 1158 | + | |
| 1159 | + | |
| 1160 | + | |
| 1161 | + | |
| 1162 | + | |
| 1163 | + | |
| 1164 | + | |
| 1165 | + | |
| 1166 | + | |
| 1167 | + | |
| 1168 | + | |
| 1169 | + | |
| 1170 | + | |
| 1171 | + | |
| 1172 | + | |
| 1173 | + | |
| 1174 | + | |
| 1175 | + | |
| 1176 | + | |
| 1177 | + | |
| 1178 | + | |
| 1179 | + | |
| 1180 | + | |
| 1181 | + | |
| 1182 | + | |
| 1183 | + | |
| 1184 | + | |
| 1185 | + | |
| 1186 | + | |
| 1187 | + | |
| 1188 | + | |
| 1189 | + | |
| 1190 | + | |
| 1191 | + | |
| 1192 | + | |
| 1193 | + | |
| 1194 | + | |
| 1195 | + | |
| 1196 | + | |
| 1197 | + | |
| 1198 | + | |
| 1199 | + | |
| 1200 | + | |
| 1201 | + | |
| 1202 | + | |
| 1203 | + | |
| 1204 | + | |
| 1205 | + | |
| 1206 | + | |
| 1207 | + | |
| 1208 | + | |
| 1209 | + | |
| 1210 | + | |
| 1211 | + | |
| 1212 | + | |
| 1213 | + | |
| 1214 | + | |
| 1215 | + | |
| 1216 | + | |
| 1217 | + | |
| 1218 | + | |
| 1219 | + | |
| 1220 | + | |
| 1221 | + | |
| 1222 | + | |
| 1223 | + | |
| 1224 | + | |
| 1225 | + | |
| 1226 | + | |
| 1227 | + | |
| 1228 | + | |
| 1229 | + | |
| 1230 | + | |
| 1231 | + | |
| 1232 | + | |
| 1233 | + | |
| 1234 | + | |
| 1235 | + | |
| 1236 | + | |
| 1237 | + | |
| 1238 | + | |
| 1239 | + | |
| 1240 | + | |
| 1241 | + | |
| 1242 | + | |
| 1243 | + | |
| 1244 | + | |
| 1245 | + | |
| 1246 | + | |
| 1247 | + | |
| 1248 | + | |
| 1249 | + | |
| 1250 | + | |
| 1251 | + | |
1085 | 1252 | | |
1086 | 1253 | | |
1087 | 1254 | | |
| |||
0 commit comments