perf-timeline: test262.total is single-shot and poorly runner-normalized (~6% scatter swamps engine signal)

[mparrett]

Extends #26. #26 fixed the metric (sum → mean per test), which killed the pass-count inflation. Now that the test262.total series is dense (22 points), the remaining run-to-run scatter is visible — and it's dominated by measurement noise, not engine change. Two distinct causes, one quick fix and one deeper.

Evidence (the 22 backfilled points)

• Mean-per-test CoV 6.4%, min-to-max spread 1.43×.
• Passing set is essentially constant across the window: 40141–40194 (0.1%). So the perf-timeline: test262.total sum conflates per-test speed with pass-count (conformance inflates the trend) #26 composition confound is dormant here — the scatter is not pass-count.
• Anchor-normalized ratio CoV 6.3% — normalization removes almost none of the variance.
• corr(ratio, anchor) = −0.56. If the anchor cancelled runner effects, ratio would be independent of it. The anti-correlation means the anchor is injecting noise, not removing it.
• The anchor itself swings 1.41× across runs (0.997–1.41 ns), ~13% even on the same CPU model (EPYC 7763).

Root causes

1. Single-shot. The macro runs the suite once (count=1); the total is one summed-ns measurement with no min/median reduction — unlike the micro benches, which get b.N + -count. One-directional runner noise (GC, CPU migration, co-tenancy) lands directly on the point.
2. Anchor mismatch. BenchmarkRatchetAnchor is a pure-ALU loop; test262 is parse + VM + GC + allocation. They don't co-vary under runner conditions, so the ratio doesn't cancel — the "anchor drifts vs the mixed/memory workload" caveat already noted in docs/perf/index.html, but biting within measurement noise here, not just across CPU tiers.

Proposed

1. Quick, high-ROI: run the macro -count N (default 3) and reduce the total by min (least-contaminated run), matching the reducer rationale from perf(bench-ratchet): scope the -count reducer — min for the informational A/B, mean for the ratchet #22. Draft below. ~N× the macro wall-time per point.
2. Deeper: stop normalizing this series against the ALU anchor. Either normalize against a representative mixed micro-benchmark, or make it self-relative to a fixed reference commit (the "fixed reference set", perf-timeline: test262.total sum conflates per-test speed with pass-count (conformance inflates the trend) #26 option 1) — the only path to a real per-test-speed signal.

Two smaller notes surfaced while backfilling

• The page's modal-CPU filter keys off snapshot.machine.cpu_model — the micro run's CPU. Backfilled test262.total points were measured on a different (later) runner, so the filter doesn't correctly apply to this series.
• set_hash is soft-absent until bench-test262: fingerprint the contributing test set so test262.total is comparable #24's tooling reaches origin/main; attach it + segment the line across set changes (Epic: perf benchmarking infrastructure (tracker) #23) when it lands.

Refs: #26 #24 #23 #16

[mparrett]

Cross-checked the two fixes (and some adjacent tuning ideas) against the sibling let-go, which runs the same anchor-relative perf stack. Refined priority:

1. -count repetition → clear near-term fix (sibling-confirmed). let-go's perf-timeline already runs -count 3 -benchtime 2s, and its microbench harness does warm-then-3-reps. Running the macro -count N and keeping the min total just brings this series in line with established practice. This is the high-ROI change.

2. Anchor/normalization → real, but a shared greenfield limitation. let-go's docs/perf/ratchet.md uses the identical BenchmarkRatchetAnchor and documents it as only "roughly stable across machines within a CPU family." So the residual runner leak (the −0.56 above) is the shared design's acknowledged weak spot, not a paserati-specific bug. Redesigning it (mixed-workload normalizer, or self-relative to a fixed reference commit) improves both projects — worth doing, but deeper and not urgent.

Architectural direction (not a drop-in): let-go runs its whole-suite timing as a real Go benchmark (BenchmarkClojureTestSuite), so it inherits b.N + -count + the ratchet reducer for free, rather than a bespoke shell/jsonl macro. Unifying test262.total onto that machinery is attractive — but let-go's suite is ~379 ms while test262 is minutes across ~40k tests, so a b.N-scaling bench would blow the budget (would need fixed b.N=1/tight benchtime). The shell -count loop is the pragmatic fit at this scale.

GC / runner tuning → experimental, deliberately not recommending yet. GOGC/GOMEMLIMIT (to cut GC-pause noise on this allocation-heavy workload) and dedicated/larger runners (to remove the heterogeneous-pool variance) are textbook levers, but neither project sets them — both run plain ubuntu-latest with default GC. So they're unproven relative to house practice. If pursued, measure before/after on a handful of commits first, and note each changes absolute timings → a metric discontinuity requiring a fresh re-backfill for coherence.