Three gaps that let the perf gate report green through a regression

[mparrett]

Three items in the performance-gating machinery, grouped because they share a failure mode: the gate reports green while the thing it guards has moved. Each was raised as a non-blocking follow-up during review and never filed.

Related to #663, which asks for a measurement-status discipline. This one is narrower — three specific holes in the instruments #663 would report from.

Checked against main @ 8a8222d0.

1. The 5% budget has never been calibrated against a measured noise floor

bench-ratchet gates at 5%. Nothing establishes what the noise floor is per machine tier, so whether 5% sits above the noise or inside it is unknown. Two datapoints suggest it is closer than it should be.

Byte-identical binaries measured 3.7% and 1.1% apart on two benchmark packages, and a reproducible 4.2% gap came from compile-time code placement alone.

Separately, measuring a candidate one-line change in pkg/vm on an M2 (hyperfine, 3 warmup / 20 runs) gave order-dependent results: main 1.03 ± 0.03x faster with A first, the change 1.07 ± 0.09x faster with B first. The pkg/vm micros ran at ±20–30% σ with no family reaching significance. The change could not be judged at that noise level.

A comment-only edit pushed through the existing dispatch, per tier, would measure the floor directly. Until that number exists, 5% is a guess, and every regression call made against it inherits the guess.

2. jobsSelect truncates to the benchmark family before matching

cmd/bench-ratchet/main.go truncates a benchmark's name to its family — everything before the first / — before applying the -filter regexp. For a slash-specific filter such as -filter '^BenchmarkMapAssoc/HAMT-Assoc/100$', go test does run that exact sub-benchmark, but jobsSelect tests the whole regexp against BenchmarkMapAssoc and returns false.

The consequence inverts the intent. If that targeted sub-benchmark disappears, its baseline row is classified as out of scope and the MISSING signal — the reason to run a filter at all — is suppressed.

3. try-lower-go's fallback stays non-strict

In scripts/ir-stress.lg, try-lower-go's cond ends in :else :ok. Today every status compile-form returns is handled explicitly, so the branch is unreachable and the classification is correct. If compile-form grows a new failure-status map, the census counts it as coverage and the ratchet locks that in.

The sibling path, try-lower-bytecode, was made fail-loud. This one was left as-is deliberately, because changing it needs its own rebaseline. Filing it so the asymmetry is tracked rather than remembered.

Parented under #464.

[mparrett]

Item 1 says the 5% budget has never been checked against a measured floor. Here is a first measurement, which arrived by accident.

#708 changes one workflow file and no Go source, so the repeat A/B lane built base and head from identical code — an A/A run. Same machine tier and settings the lane always uses: AMD EPYC 7763, 4 CPU, go1.26.5, N=7 interleaved ABBA, profile pr-fast, 53 comparable families, base 8a8222d0 head fe764ec0.

Result

statistic over 53 families	value
max |median-of-7|	1.69%
range of medians	−1.41% to +1.69%
95th percentile |median|	1.36%
would-gate at 6 / 8 / 10%	clean / clean / clean
max |single-cycle worst|	39.77%

The last row is the interesting one next to the first. With code held identical, one cycle in the run swung 40% on a family, while the median of seven on that same family stayed inside 2%.

What it says about the budget

For the repeat lane, 5% looks defensible on this tier: the floor is about a third of it, with headroom. That is a narrower claim than item 1 asks for — one sample, one tier, one profile, and the amd64 runner draw varies — but it is the first number rather than an assumption, and it points the opposite way from what I expected when I filed it.

For single-shot A/B it confirms the opposite, and quantifies #445: a single cycle on identical code can report 40%. No budget in the 5–10% range survives that. The gap between the two rows is what interleaving buys, and it is large.

Suggested next step for this item

Two more A/A runs are queued (#709 is a deliberate comment-only control in pkg/vm; #708 has a second run from a follow-up push). Three samples on one tier would make the amd64 number reasonably solid. The arm64 macos tier has no A/A sample at all, and per #564 it is the noisier of the two, so it is the one that would most change the picture.

This cost nothing to obtain: any PR that touches no Go source produces an A/A. If the lane recorded its own A/A results somewhere over time, the floor would maintain itself instead of needing a calibration project.

[mparrett]

Correcting my comment above. I reported a 1.69% floor from one A/A run and said 5% looked defensible on this tier. That was one draw, and two more A/A runs since then show it was a lucky one. The second run of the very same PR — identical Go code both times — produced a family at +13.43%, gating at 6, 8 and 10%.

So there is no single floor. There is a per-family floor, and the spread across families is the finding.

Three A/A runs, same tier, same settings

All EPYC 7763, N=7, profile pr-fast, 53 families, code identical in each run (two workflow-only PRs and one comment-only change in pkg/vm).

Noisiest under identical code:

family	A/A medians	max
VectorConj/ArrayVector/1000	−0.0, −5.4, +13.4	13.4%
VectorCreation/ArrayVector/10000	−0.6, −12.9, +7.5	12.9%
VectorConj/PersistentVector/1000	+0.3, −0.9, +11.9	11.9%
VectorCreation/PersistentVector/10000	+0.2, +0.8, +5.7	5.7%

Stable under identical code:

family	A/A medians	max
FrameDispatch	−0.3, +0.2, +0.3	0.3%
VectorConj/ArrayVector/10	−0.8, +0.2, −0.4	0.8%
SeqIteration/List/01000	+0.1, +0.9, +1.0	1.0%
SeqIteration/List/100	−0.0, −1.1, −0.9	1.1%

Across all 53: median per-family max 1.86%, p90 4.56%, max 13.43%. 29 of 53 families stay under 2%.

Why one budget cannot fit

A single 5% number is wrong in both directions at once, which is more interesting than being merely too tight or too loose.

Too loose for the stable majority. FrameDispatch moves 0.3% on identical code. A genuine 4% regression there is far outside its own noise and sails through a 5% gate unnoticed.

Too tight for the memory-bound tail. VectorConj/ArrayVector/1000 produced 13.4% with nothing changed. Under a 5% gate that family is a false alarm generator, and the cost of that is not just noise — it is the credibility of every future red result.

The shape suggests the split is not arbitrary: the noisy families are the large-allocation ones (sizes 1000 and 10000), and the stable ones are the dispatch and small-collection benchmarks. Whatever the runner does to memory behavior between draws, it hits allocation-heavy work and leaves compute-bound work alone.

What I would do with this

Derive a per-family budget from accumulated A/A samples rather than picking one global number — something like max observed A/A times a factor, floored at a few percent. The stable families then get a gate that can actually catch something, and the allocation-heavy ones stop crying wolf.

The samples are free. Any PR touching no Go source is an A/A, and this repo produces those regularly; all three above were accidents. If the lane appended its A/A results to a file on the perf-data branch, the per-family floor would accumulate on its own and this issue would answer itself over a few weeks.

Caveat that matters: three samples, one tier, one profile. The arm64 macos tier has none, and #564 suggests it is the noisier of the two.