perf(bench-ratchet): seed the baseline from a window, reduced in ratio space

[mparrett]

Follow-up to #564, which I said I'd open. #564 has landed and this is now rebased onto it: one commit, five files. baseline.json is no longer in the diff, since main already carries the regenerated file from #564.

seed-baseline currently takes the newest snapshot per machine key. This takes a window of five and medians it, reduces in ratio space rather than raw ns/op, and gates on whether a snapshot agrees with its window instead of on how far its anchor drifted.

Correction to my #564 review

I said there to skip a candidate whose anchor is an outlier against its own tier. I built that, ran it against perf-data, and the corpus refutes it.

Snapshot a588a69d2759 (EPYC 9V74) sits 22.4% off its window's anchor. It is also uniformly 22.4% fast in raw ns/op across all 162 of its benchmarks, and agrees with its window on every ratio_to_anchor to within 0.1%. The host was fast that day and the anchor divided it back out, which is what the anchor is for. Anchor-deviation gating would have discarded it and two more like it, two of which are snapshots the current baseline is seeded from.

Over the 24 most recent amd64 snapshots:

	range / worst
anchor deviation from tier median	−22.4% … +3.1%
ratio_to_anchor offset from tier median	median 0.03%, worst 1.75%

So the gate is on ratio coherence: a snapshot is rejected when its ratios sit more than -seed-coherence-tolerance (default 5%) off the rest of the window. That catches the case that damages a baseline — the mixed capture, where the anchor caught the slow tail and the benchmarks did not, so every ratio is uniformly wrong while the raw numbers look ordinary. Nothing in the current corpus trips it, which is the expected result and not evidence the check is idle.

Method

A re-derivation over snapshots already committed on perf-data — no new measurement runs, reproducible with -seed-window 1 against -seed-window 5 and a diff of ratio_to_anchor. Provenance for the tail figures behind the windowing argument is in the commit message.

Why a window

One snapshot is one CI run, and one CI run is one sample. Seeding from the newest versus a median of five, same corpus, same day:

	share of 758 (tier, benchmark) floors
differ by more than 5% (the gate budget)	22.6%
differ by more than 10%	11.3%
differ by more than 25%	5.8%

Part of that spread is real code movement across the window and part is sampling; I can't separate them without a null control. Either way, seeding from one snapshot sets a fifth of the gate's thresholds from a single observation of it.

Median rather than min: update already takes a min over history when it ratchets, and seeding with a second minimum stacks two of them into a floor no clean run reaches.

Why ratio space

Raw ns_per_op carries host speed; ratio_to_anchor does not — see the table above. Reducing the quantity that carries host speed and then dividing imports that speed into the stored floor. The two approaches agree whenever a window comes from one host and diverge exactly when it does not, which a shared runner pool guarantees. ns_per_op is derived back from the reduced ratio and the window's anchor, so ratio_to_anchor == ns_per_op / anchor.ns_per_op holds by construction.

Additional fixes

• The arch filter now applies to the file's content, not only its name, and warns when the two disagree. That divergence put an EPYC profile under an Intel key once already, and the log line printed the filename.
• Snapshot names parse through an anchored regexp. A positional split on - yields a plausible-looking wrong SHA, and the machine slug is full of dashes. Unparseable names are skipped and reported — 120 of them in the current corpus, all predating the machine-slug naming.
• A benchmark present in under half the window is skipped rather than seeded from one observation. check reports it as NEW, which is honest.
• b.N movement across the window is reported. b.N is an output of the timing loop, so it moves when per-op cost moves, and ns/op is N-dependent wherever iterations share state. Reported, not acted on: excluding on it would shrink the gate silently.
• Every matching exclusion prefix is marked, not just the first. BenchmarkClojureTestSuite is a prefix of BenchmarkClojureTestSuiteCompileAndRun, so the longer entry was reported as stale on every run while it was in fact filtering.

Verification

• go test ./cmd/bench-ratchet — 21/21, 11 of them new, including the negative result above as TestSeedBaselineKeepsUniformlyFastSnapshot and the mixed capture it does reject.
• Run against the real perf-data timeline: 5 amd64 tiers seeded, 134–156 benchmarks each, no snapshot rejected, M3 preserved.
• -seed-window 1 reproduces the current behaviour, which is how the comparison table above was produced.

Not addressed here

The 5% budget still has no measured floor under it. Nothing in this PR tells you whether 5% is above or below the gap between two builds that cannot differ on these tiers — a comment-only-edit control run through the existing dispatch would. Worth doing before the budget is tightened.

[nnunley]

#564 landed on main as c9bf9335d100aa00b548b70eae6215d4486828b5 (squash merge, 2026-08-07).

This PR still lists the pre-squash #564 commits (6a825ba8c4a7 … 7f8d298bd120) plus its own tip 5601f1587de6, so it needs the rebase you described in the body before the four-file window/ratio-space delta is reviewable on its own.

No rush and no action needed from my side — flagging it since the blocker you were waiting on is cleared.

[nnunley]

Thanks for the rebase. The argument for ratio-space reduction convinced me, and the
negative result — keeping the uniformly-fast host that anchor-deviation gating would have
discarded — is the part that makes the coherence gate credible rather than decorative.
Deriving ns_per_op back from the reduced ratio so ratio == ns/anchor holds by
construction is the right call given what check relies on. Marking every matching
exclusion prefix rather than the first is a real fix: …TestSuite is a prefix of
…TestSuiteCompileAndRun, so first-match would permanently mis-report a live exclusion.

The open question: this lands the method but not the result. docs/perf/baseline.json
on main still comes from the old newest-snapshot, raw-ns seeding. By your own table 22.6% of
the 758 floors move by more than the 5% gate budget under the new method, so until someone
re-runs the seed, the gate keeps enforcing the old floors. Is the re-seed a follow-up, or
should it ride along here?

One nit: M3 preservation now keys only on CPUModel containing "M3" and overwrites a freshly
seeded entry. The old code also required the machine key. Nothing in the current baseline
trips it, but an amd64 model string containing "M3" would silently override its own fresh
seed; also requiring Arch != archPrefix closes it.

I ran the unit tests, not a seed against the real perf-data timeline, so the corpus figures
in the body are yours, not independently reproduced by me.