perf: b.N movement correlates with round-to-round spread; the causal direction is unresolved

[mparrett]

The tests package benchmarks run at very low iteration counts, so the reported ns/op is coarsely quantised and moves when b.N moves. At perf-pr.yml's -benchtime 500ms the worst of them runs one iteration.

Measured on a dedicated c7a.2xlarge (AMD EPYC 9R14, go1.26.0), 16 commits × 5 rounds at -count 3 -benchtime 1s — the perf-timeline.yml settings:

benchmark	b.N observed
BenchmarkArith	4–8
BenchmarkAdd	2–5
BenchmarkSetIndex	2
BenchmarkFibPlaceholderRun	1–2
BenchmarkMatrixMult	150–260 (fine)

perf-pr.yml uses -benchtime 500ms, so those counts roughly halve again.

Why it matters beyond coarseness

ns/op here is not simply duration / N with a constant per-op cost. The benchmarks create one interpreter outside the loop and reuse it:

func BenchmarkArith(b *testing.B) {
	chunk := compileFile(b, "scripts/bench_arith.ts")
	paserati := driver.NewPaserati()
	...
	for i := 0; i < b.N; i++ {
		_, runtimeErrs := paserati.InterpretChunk(chunk)
	}
}

So per-iteration cost falls as caches and IC state warm, and the amortised ns/op is genuinely N-dependent. Going from N=5 to N=8 changes the number without anything in the engine changing.

In our corpus this was visible as: commits where b.N moved had a median round-to-round spread of 32.0%, against 0.8% for commits where it held. Same source, same machine, same session.

What this is not

I initially thought this caused an instability we were chasing. It does not — pinning the count with -benchtime Nx and relaunching 20 times changed little, and made one control case worse. b.N is an output of the timing loop (N ≈ benchtime / per-op cost), so it moves with the measurement rather than driving it.

The claim here is narrower and I think solid: a metric quantised to 1-in-8 (or 1-in-1) cannot honestly report the 1–5% changes the perf lanes exist to detect.

Suggested fix

Raise -benchtime for the tests package until b.N is comfortably above ~20, or pin it per benchmark with Nx. MatrixMult already sits in a healthy range, so this is mostly about the four cheap-script benchmarks.

Costs wall clock. Buys a number whose resolution matches what it is being asked to resolve.

Provenance

Full data, the eight diagnostic experiments, and a corrections index (this investigation produced three wrong explanations before this one) are in my notes rather than upstream; happy to attach raw samples[] output if useful. iterations is already recorded in every sample, so this is checkable against any existing snapshot without re-running anything.

Tracker: #23. Adjacent but independent of #22, which scopes the reducer rather than the sampling underneath it.

[mparrett]

Correcting this issue: the mechanism I gave is wrong, and the fix I suggested follows from it.

The observation that prompted it stands. What I said it meant does not.

The quantisation claim is withdrawn. I wrote that ns/op is "quantised to 1-in-N" and that a metric quantised to 1-in-8 cannot report a 1–5% change. That is wrong. Go computes ns/op as elapsed nanoseconds divided by N, so the quantum is (1 ns)/N — an absolute quantity, not a fraction of the result. On BenchmarkFibPlaceholderRun at b.N=1, which runs at ~780 ms/op, that is 1.3 × 10⁻⁷ % of the reported value. Not 100%.

What low N actually costs is averaging: ns/op is the mean over N iterations, so per-iteration variance falls only as √N, and at N=1 a single slow iteration is the entire reading. Real, but a different claim, and much weaker than the one I made.

What survives. Measured on a dedicated c7a.2xlarge, 16 commits × 5 rounds at -count 3 -benchtime 1s:

benchmark	b.N observed	moved between commits
BenchmarkArith	4–8	yes
BenchmarkAdd	2–5	yes
BenchmarkSetIndex	2	no
BenchmarkFibPlaceholderRun	1–2	yes
BenchmarkMatrixMult	139–261	yes, in all 16

Commits where b.N moved had a median round-to-round spread of 32.0% against 0.8% where it held. That effect is real. Its cause is that these benchmarks build one interpreter outside the loop and reuse it, so per-iteration cost is not constant and ns/op is genuinely N-dependent. When N differs between two commits, the two numbers were produced under different protocols.

Four separate things were conflated in the original report, and only the second is this issue:

1. Low N — limited averaging (√N), not coarse quantisation.
2. N changing between commits — an inconsistent measurement protocol wherever per-op cost depends on N. This is the confound.
3. Slow-tail launch variance — not removed by pinning N; a fixed-N experiment found pinning helped some cases and made others worse.
4. The anchor's own resolution — added since, and it is the one I would have flagged first had I seen it. BenchmarkRatchetAnchor returned exactly two values across all 240 readings of that session, 1.084 and 1.085 ns/op. Go prints four significant digits, so one ulp is 0.0923% of the value, and the anchor's entire observed range is one ulp. Every delta on that page is a ratio against it, so nothing finer than ~0.09% is resolvable no matter how quiet the benchmark — and roughly half the measured cells have a round-to-round spread tighter than that, where normalising against the anchor can only add its quantum. The "anchor drift 0.031%" I quoted was that two-level quantum, not drift.

The suggested fix is withdrawn too. I proposed raising -benchtime until N > ~20. That assumes every benchmark converges upward to a steady state. Sweeping N from 1 to 256 shows the direction is not common to the suite:

MatrixMult   −33% by N=16, then flat     warms up
Arith        −38% by N=256               warms up
Add          +73% by N=256               degrades
Fib          +17% by N=256, monotone     degrades

A uniform raise walks each benchmark a different distance along its own curve, in its own direction, and changes what each one measures.

What I would suggest instead: pin b.N per benchmark. The confound is that the blend varies between commits, not that it is small, so any fixed N removes it — including one near where each benchmark already runs, which costs almost nothing. SetIndex runs at b.N=2 and is the tightest benchmark in the suite (0.30% MAD/median); MatrixMult runs at 139–261 and is both noisier and the only one whose N moved in every commit. Size is not the signal.

Happy to attach the raw sweep and the per-commit b.N data if useful. iterations is already recorded in every sample, so any existing snapshot can be checked against this without re-running anything.

(Correction prompted by an external review of our measurement report; the arithmetic error was mine.)

[mparrett]

Second correction, and this one goes at the sweep I posted a few hours ago. The b.N table in my last comment came from a screening run on a MacBook under ordinary desktop load. Re-run on the measurement host, none of its four shapes reproduce.

I should not have posted it. The document it came from opens by saying so: "Screening experiment, not a result … absolute numbers here are not usable; only the shape of each curve is … Arith in particular swings ±30% between adjacent N on this host and its curve should be re-run on the measurement host before anything is concluded from it." I lifted the shapes out and dropped the caveat.

The host-measured curves. Dedicated c7a.2xlarge (EPYC 9R14), go1.26.0, -count 3, min, b.N swept 1 → 256:

benchmark	N=1	N=256	end-to-end	scatter over all 9 points	previously posted
Add	316.0 ms	335.6 ms	+6.2%	6.8%	+73%
Arith	161.2 ms	161.4 ms	+0.1%	1.9%	−38%
FibPlaceholderRun	916.5 ms	912.5 ms	−0.4%	5.3%	+17%
MatrixMult	5.80 ms	5.85 ms	+0.9%	2.5%	−33%
SetIndex	885.9 ms	902.7 ms	+1.9%	2.4%	−19%

The end-to-end change is the same size as each curve's own scatter. These are flat curves with noise on them, not trends. The laptop was measuring its own load.

What that costs this issue. The mechanism I gave — these benchmarks build one interpreter outside the loop and reuse it, so ns/op is a function of N — is not supported on the machine we actually measure on. If per-op cost does not vary with N, then two commits measured at different N were not measured under different protocols, and the framing in the title does not follow either.

What still stands. The observation: across 16 commits × 5 rounds, commits where b.N moved had a median round-to-round spread of 32.0% against 0.8% where it held. That correlation is in the data and I have not been able to make it go away.

What I no longer have is a mechanism for it. The parsimonious reading now runs the other way: go test chooses b.N to fill -benchtime, so a benchmark whose timing is unstable gets an unstable N. Moving N would then be a symptom of noise rather than a cause of it. That also fits the fixed-N experiment reported earlier in this issue, which found pinning did not consistently reduce launch instability — under the causal story, pinning should have fixed it.

I am not asserting the reverse direction either. A session running right now pins every ./tests benchmark at a fixed N across 10 commits × 4 counterbalanced rounds; if the spread survives at fixed N, the causal story is dead. I will post the result whichever way it falls.

One thing the sweep did find, on the host. The BenchmarkPrototypeMethodAccess / BenchmarkPrototypeCacheHitRate group is strongly N-dependent — a median of −37.8% from N=1 to N=256. Those benchmarks call driver.NewPaserati() and RunString() inside the timed loop, so each iteration builds an engine and compiles the snippet, and larger N amortises the first, coldest one. Relatedly, all ten of them land within 8% of each other at ~1 ms regardless of workload or cache setting, which suggests they are dominated by engine startup rather than by the prototype access they are named for. That is a separate problem and I will file it separately.

Also withdrawn: a claim I made in the previous comment, that the calibration anchor cannot resolve below ~0.09%. The resolution arithmetic is right — four significant digits on a ~1.08–1.38 ns value — but I attributed a reading difference to b.N that was actually a difference between two sessions on different physical machines. On one box the anchor is flat across a 7000× range in N (3.6% total); between two sessions on the same instance type it moved +26.7% (1.084 → 1.373). So the anchor is not N-sensitive; it is machine-sensitive, which is what it is there for, and absolute ns/op simply does not carry across sessions.

Retitling again to stop the headline asserting a mechanism that is now in doubt. Sorry for the churn — I would rather correct this twice than leave it standing.

[mparrett]

The session I promised the result of has run, and it does not settle this issue. Reporting it anyway, since I said I would either way.

Setup. Dedicated c7a.2xlarge (EPYC 9R14), go1.26.0, 10 commits × 4 counterbalanced rounds, -count 3, reducer min. Every ./tests benchmark pinned at b.N=8, so the moving-N confound is removed by construction rather than argued away.

The spread did not collapse. Median per-commit MAD across the four rounds:

family	median	p90	max
./tests (pinned b.N)	2.70%	5.16%	6.80%
pkg/vm (unpinned)	0.41%	1.77%	2.41%

./tests remains the noisier family by roughly 6× with N held constant. So pinning did not buy the stability this issue predicted.

That is consistent with N being a symptom of instability rather than a cause of it — which is what I suggested in the previous comment — but it does not establish it. The pinned run also changed how much work each measurement does, and I cannot separate those two effects from this data. Inconclusive, and I would rather say so than pick the reading that flatters the last thing I wrote.

What the session did settle, which bears on the framing here. Null controls measured in the same run — pairs of commits compiling to byte-identical binaries, so there is no code for any difference to be attributable to — give a floor of 3.7% on ./tests and 1.1% on pkg/vm. A layout control (real code added but never executed) comes out no worse than the byte-identical one.

Against a 3.7% floor, much of the round-to-round spread this issue is about is simply the floor. The 32%-vs-0.8% correlation is still far outside it and still unexplained, but the ordinary case is smaller than the apparatus can resolve.

And the calibration anchor. It survived a real test: the same commit measured in July and again now, on boxes whose anchors read 1.084 and 1.378 ns/op — a 27.1% hardware difference — agrees on ratio_to_anchor to a median of 0.30% across 35 benchmarks. The normalisation works.

Its failure mode is transient rather than systematic: multi-launch excursions of about 4%, which a median across rounds does not fully absorb when two of four rounds are hit. That happened to one commit in this session and cost it several points of measured improvement.

Raw data, the run log, and the analysis script are in evidence/2026-08-03-remeasure-session/.