perf-pr A/B is ungateable as a single shot here too; median-of-N fixes it

[mparrett]

Summary

This is the paserati half of a finding first measured on let-go
(nooga/let-go#445): the two projects share a near-identical bench-ratchet and
perf-pr.yml, including a BenchmarkRatchetAnchor that is the same register-only
PCG loop. I measured whether the same blind spot is present here by running
perf-pr on a no-op PR (a comment-only change, so base and head are
perf-identical and every reported delta is pure noise), repeated on
ubuntu-latest.

It reproduces cleanly on paserati's own benchmarks:

• Across five same-code runs the anchor never moved more than 0.1%, yet 3
  families exceeded the current 10% budget and 9 exceeded 5%.
• The families that blow out are the memory-bound ones, and the noise tracks
  memory footprint with the code held fixed (details below).

So perf-pr here can't be gated as a single shot for the same reason it can't on
let-go, and the fix carries over: median-of-N interleaved A/B, plus a
memory-aware second anchor.

Evidence

Five no-op A/B runs, worst per-family delta (same code, so all noise):

Benchmark	worst |Δ|	deltas across 5 runs
PrototypeMethodAccess/Baseline/ArrayPrototypeMethod	26.8%	−16.6, +0.0, +2.7, +26.8, −5.6
GetOwn/n=64/last	21.2%	+21.2, +0.0, +0.0, −0.0, −0.1
PrototypeMethodAccess/Baseline/ObjectPrototypeChain	14.0%	−14.0, +4.7, −4.4, +1.0, −8.5
PrototypeMethodAccess/WithPrototypeCache/StringPrototypeMethod	9.9%	+2.0, +1.9, +9.9, +5.6, −9.4

Anchor Δ per run: +0.0, −0.1, −0.1, +0.1, +0.0. The blowouts are mixed-sign and
localized (a different family spikes each run) while the anchor stays flat, so
they slip past anchor normalization entirely. One run tripped the 10% budget on a
change that touches no code the benchmarks exercise; a gate turned on today would
fail that PR.

Root cause — the anchor measures compute, not memory

BenchmarkRatchetAnchor is a register-only loop with no loads or stores, so its
throughput tracks ALU/clock speed and is invariant to memory-subsystem pressure.
The benchmarks that blow out are memory-bound. When a co-tenant VM saturates
memory bandwidth between the base pass and the head pass, the memory-bound
benchmarks shift together and the compute-only anchor doesn't register it.

paserati's suite makes this unusually legible, because GetOwn sweeps object
size:

• GetOwn/n=1, n=4, n=8 (small objects, little memory) stay under 0.5%.
• GetOwn/n=64/last (largest object, most memory) hits 21.2%.
• PrototypeMethodAccess/Baseline (uncached prototype-chain walk, i.e. pointer
  chasing) blows out ±14–27%, while WithPrototypeCache (same operation, less
  memory traffic) stays tight.

The noise scales with memory footprint at fixed code. That is the signature of
bandwidth contention the register anchor can't see.

The min-of-count reducer helps, but can't reach this

These runs were measured on a branch that already reduces each benchmark's
-count repetitions by the minimum rather than the mean (interference is
one-directional, so the fastest of N repetitions is the least-contaminated
estimate). That change is proposed on its own in #22 and is worth landing first —
it stabilized the small GetOwn/n=1..8 variants completely (all under 0.5%
above), the ones that used to flip-flop phantom regressions run to run.

It does not reach the blowouts here, and my data shows why. Min-of-count only
helps when at least one of a benchmark's N repetitions is clean. When contention
spans a benchmark's entire measurement window in one of the two passes — which
is what happens to the larger memory-bound families — every repetition in that
window is slow, so the min is slow too. GetOwn/n=64/last reading +21% means all
of its head-pass repetitions were contended. Min operates within a capture;
this contention is across the base and head passes, minutes apart. The two
layers compose: min for intra-capture jitter, median-of-N for the cross-pass
window-spanning case. (Upstream still averages the repetitions, so it sees more of
this than the numbers above, not less.)

Proposal

Both steps mirror nooga/let-go#445; they apply here unchanged, and layer on top
of the min-of-count reducer (#22) rather than replacing it.

1. Make perf-pr a median-of-N interleaved A/B and turn on the gate. Build
   base and head once, run N interleaved base/head snapshots, gate on the
   per-family median (each snapshot still min-reduced internally). On let-go, N=5
   still false-fired ~1 run in 4 and N=7 held at an 8% budget. paserati's suite is
   small, so tuning N and the budget here is cheap — the noise-floor runs above
   already give a starting point.

2. Add a memory-aware second anchor (the deeper fix). Every family that
   defeats the median is memory-bound. A second calibration bench that is itself
   memory-bound (a strided or pointer-chasing sweep over a cache-exceeding
   buffer) would track the bandwidth contention the register anchor misses;
   normalizing memory-bound families to it would let the whole suite gate tight.

The full let-go writeup, the interleaved-A/B prototype, and the raw data are in
nooga/let-go#445 and the fork PR it links. I'm happy to port the median-of-N
change to paserati and open a PR here. Step 2 is worth discussing first, in
particular which memory-bound pattern best matches the contention these
benchmarks see.

[mparrett]

Follow-up: the memory-aware second anchor doesn't work — median-of-N alone is the fix.

I prototyped the memory-aware second anchor I suggested above and validated it on CI here. It doesn't hold up, and the reason is structural, so I want to record it rather than leave the suggestion standing.

The idea was a companion DRAM-bound benchmark (a 128 MiB random pointer-chase, ~98 ns/op against the register anchor's ~1.4) whose ratio would let memory-bound families (GetOwn deep-object, PrototypeMethodAccess chain walks) divide out common-mode contention. On paper it works: a synthetic same-code A/B with the head snapshot held under +20% memory pressure normalizes from +20% back to 0%.

On real CI it doesn't. Across 5 contended runs the per-cycle relationship between how much a family swings and how much the anchor swings is pure scatter (−0.58 to +2.66), with no co-variation. Dividing by the anchor injects the anchor's own volatility instead of cancelling the family's: on the run where the anchor swung ±14%, the worst family's |median| went from 4.09% raw to 13.23% normalized.

The cause is that go test -bench runs benchmarks sequentially, so the anchor and each family are measured seconds apart. Noisy-neighbor contention on a shared runner is non-stationary at that timescale, so the two never share the instantaneous contention the normalization assumes. The synthetic test passed only because it baked in same-instant co-variation. A working version would have to measure the anchor and family concurrently, a much larger change to the capture loop with uncertain payoff.

One caveat so nobody reruns this expecting a different answer: all 5 runs happened to be quiet, so I never caught the anchor normalizing a live phantom. That's moot given the structural reason — sequential measurement can't share instantaneous contention whether the runner is busy or idle.

So the recommendation narrows to median-of-N interleaved A/B with counterbalanced order (odd cycles base-first, even head-first, so position drift can't masquerade as a head regression). At N=7 and a 10% budget it holds 0 false positives across 10 no-op runs, worst |median| 7.63%. This composes with the min-of-count reducer (#22) as before — min within a capture, median across cycles. The memory-aware anchor is a dead end short of rebuilding the harness for concurrent measurement.