Epic: perf benchmarking infrastructure (tracker)

[mparrett]

Master tracker for the perf-adjacent work across all three lanes — the benchmarking stack (A/B PR gate, Test262 macro-bench, trend page), the compiler correctness fixes surfaced alongside it, and the VM micro-optimization series. Ties together the merged foundation, the in-flight PRs, and the review findings that gate them.

Refreshed 2026-07-21: all bench findings fixed and pushed; the VM series (below) is now open as #29–#36 with its own guide in #37.

Whole-board status

Several of these read as "needs work" in the PR list but are further along — GitHub doesn't clear a review decision when a push addresses it, so the labels lag reality. Everything below is now awaiting your review; nothing is blocked on me.

Benchmarking infrastructure

• perf(bench): VM hot-path benchmark harness #20 — VM hot-path benchmark harness — approved, ready to merge
• perf-test262: pin corpus to head for both A/B halves; drop the paths filter #25 — perf-test262 corpus pin + drop paths filter (F4+F5) — approved, ready to merge
• bench-test262: fingerprint the contributing test set so test262.total is comparable #24 — bench-test262 set_hash fingerprint (F3) — approved; a reviewer-suggested test was added after, so it shows re-review-pending
• perf(bench-ratchet): scope the -count reducer — min for the informational A/B, mean for the ratchet #22 — bench-ratchet min-vs-mean reducer — all three review blockers addressed, awaiting re-review
• perf-timeline: opt-in test262.total macro point #16 — perf-timeline opt-in test262.total macro point — addressed (set_hash-tagged), awaiting re-review; gate bench-test262: fingerprint the contributing test set so test262.total is comparable #24 first (consumes set_hash)
• perf page: family-grouped breakdown + cross-tier CPU filter #15 — perf page grouping + CPU filter (F1+F2) — approved, then a follow-up push dismissed it; re-review requested, awaiting nooga

Compiler correctness (independent of the bench lane)

• compiler: report oversized-function jumps as a compile error, not a panic #17 — oversized-function jumps → compile error, not panic — approved, ready to merge
• compiler: anonymous arrows capturing >255 variables emit OpClosure16 #18 — arrows capturing >255 vars emit OpClosure16 — approved, ready to merge
• compiler: fix spilled let/const bindings losing their value #19 — spilled let/const bindings losing their value — fix extended to var/class/no-init paths (a superset of the review ask), awaiting re-review

VM micro-optimization series — 8 PRs, full map + reviewer's guide in #37

• perf(vm): iterator fast path — step built-in iterators without a call or result object #29 C · perf(vm): array-destructuring fast path for pristine arrays #35 D(←perf(vm): iterator fast path — step built-in iterators without a call or result object #29) · perf(vm): store primitive strings inline in Value #30 E · refactor(vm): fold the heap initialized[] flag into a value sentinel #31 F · perf(vm): allocation-free Map/Set keys via a comparable struct #32 G · perf(vm): pool the native-reentry sentinel register slice #33 H · perf(vm): pool callback args slices (CallArgs2/3/4) #36 I(←perf(vm): pool the native-reentry sentinel register slice #33) · perf(vm): allocation-free string relational comparison for BMP strings #34 J
• All fresh REVIEW_REQUIRED. Also available as one stacked branch (mparrett:perf/vm-microopt-all) for a merge-once path — PR on request.

Merged foundation: #4 (A/B gate) · #5 (trend page) · #10 (Test262 macro-bench).

Merge order

• The approved four (compiler: report oversized-function jumps as a compile error, not a panic #17, compiler: anonymous arrows capturing >255 variables emit OpClosure16 #18, perf(bench): VM hot-path benchmark harness #20, perf-test262: pin corpus to head for both A/B halves; drop the paths filter #25) land anytime.
• bench-test262: fingerprint the contributing test set so test262.total is comparable #24 before perf-timeline: opt-in test262.total macro point #16 (perf-timeline: opt-in test262.total macro point #16 consumes set_hash). bench-test262: fingerprint the contributing test set so test262.total is comparable #24 ↔ perf(bench-ratchet): scope the -count reducer — min for the informational A/B, mean for the ratchet #22 touch the same 3 lines of aggregateFromFile — either order; whoever lands second applies the 3-line resolution (keep perf(bench-ratchet): scope the -count reducer — min for the informational A/B, mean for the ratchet #22's reduce(...) and bench-test262: fingerprint the contributing test set so test262.total is comparable #24's SetHash:).
• compiler: report oversized-function jumps as a compile error, not a panic #17 / compiler: anonymous arrows capturing >255 variables emit OpClosure16 #18 / compiler: fix spilled let/const bindings losing their value #19 any order (independent correctness).
• VM series: independent of the infra lane. Within it, perf(vm): iterator fast path — step built-in iterators without a call or result object #29 before perf(vm): array-destructuring fast path for pristine arrays #35, perf(vm): pool the native-reentry sentinel register slice #33 before perf(vm): pool callback args slices (CallArgs2/3/4) #36; the other six are parallel. #20 (bench harness) is the natural companion for measuring the series but isn't a merge dependency.

Review findings — all resolved

From the retrospective review of #10 and the review of #15 (2026-07-12); fixes now pushed to their PRs.

• F1 · perf page: family-grouped breakdown + cross-tier CPU filter #15 — off-tier baseline. Ratios divided by an off-tier baseline.json (Overall ~1.12–1.22× in-browser). → Rebase onto the first modal snapshot when baseline.cpu_model !== modalCpu. Fixed (983d9c3): Overall 1.13–1.21× → 0.93–1.00× across 22 real snapshots.
• F2 · perf page: family-grouped breakdown + cross-tier CPU filter #15 — Test262 pollutes Overall. Overall averaged every non-null series; a sparse Test262 series shifts composition where it appears. → Restrict Overall to the continuous micro suite; Test262 in its own tab. Fixed (c41e7ab): composition-stable.
• F3 · perf: Test262 macro-benchmark (opt-in same-runner A/B) #10 — aggregate loses test identity. cmd/bench-test262 emitted only sum+count. → Per-record set_hash over the contributing set, threaded onto the snapshot. Fixed (bench-test262: fingerprint the contributing test set so test262.total is comparable #24). perf-timeline: opt-in test262.total macro point #16 use: segment the line where set_hash changes; keep the hash as a diagnostic, not an exact-match gate (the passing set drifts almost every point). The metric confound the hash can't fix is perf-timeline: test262.total sum conflates per-test speed with pass-count (conformance inflates the trend) #26.
• F4 · perf: Test262 macro-benchmark (opt-in same-runner A/B) #10 — unpinned corpus across A/B halves. Each half ran setup-test262.sh at its own .test262-rev. → Force the base half to head's rev. Fixed (perf-test262: pin corpus to head for both A/B halves; drop the paths filter #25, cd897e7).
• F5 · perf: Test262 macro-benchmark (opt-in same-runner A/B) #10 — paths filter blocks harness-only opt-in. The pull_request.paths filter ran before the label gate. → Dropped it; the label is the sole gate. Fixed (perf-test262: pin corpus to head for both A/B halves; drop the paths filter #25, cd897e7).

Composition validation

• 2026-07-13: all then-open PRs composed onto clean nooga/main (integration/all-prs) and validated end-to-end — one conflict across the set (the bench-test262: fingerprint the contributing test set so test262.total is comparable #24↔perf(bench-ratchet): scope the -count reducer — min for the informational A/B, mean for the ratchet #22 3-liner), everything else clean; build/vet/TestScripts green; composed perf-pr A/B green on real CI.
• 2026-07-21: re-verified integration/all-prs still builds green (build + vet + TestScripts + cmd/bench-ratchet + all cmd/pkg units). Note it's a 07-13 merge snapshot — refresh the merges before leaning on it for a new round. The VM series was verified separately per-slice (Test262 language differential, 0 new failures each; built-ins per-dir sweep, 0 regressions).

Follow-on (not findings here)

• perf-timeline: test262.total sum conflates per-test speed with pass-count (conformance inflates the trend) #26 — test262.total sums per-test times over the passing set, so improving conformance inflates the sum and reads as a slowdown; set_hash detects the set changed but can't correct it. Decide fixed-reference-set vs per-test-mean before the series is trusted as a speed trend.
• perf-timeline: test262.total is single-shot and poorly runner-normalized (~6% scatter swamps engine signal) #28 — test262.total is single-shot and poorly runner-normalized (~6% scatter swamps engine signal).
• perf-pr A/B is ungateable as a single shot here too; median-of-N fixes it #21 — perf-pr A/B is ungateable as a single shot; median-of-N fixes it (prototype staged on the fork).

[mparrett]

Two additions to the benchmarking-infrastructure lane, both found while using the same-host session tooling on the 2026-07-25 VM micro-opt stack:

• perf: b.N movement correlates with round-to-round spread; the causal direction is unresolved #48 — the tests benchmarks run at b.N ≤ 8 (N=1 for FibPlaceholderRun at perf-pr's 500ms), so ns/op is quantised well above the 1–5% changes the lane is meant to detect.
• perf: the calibration anchor is captured in a different process and iteration regime from what it normalizes #49 — the calibration anchor is captured in a separate go test process and at b.N=1e9, so it can't observe the regime it normalizes. It certified 0.031% drift across a session where a benchmark's reported value moved 70%.

Both are independent of #22, which scopes the reducer rather than the sampling underneath it. Neither is a gate change; #48 is a benchtime bump and #49 is a job-layout change.

Worth stating plainly since it bears on how much weight to give the above: this came out of an investigation that produced three wrong explanations before landing on these two. Bimodal builds, -count process sharing, and preceding CPU load were all proposed and all refuted by experiment. The root cause of what I was originally chasing turned out to be my own summary statistic — I was reporting max/min − 1 over 5 samples, which is set by the single worst observation and grows with sample size. Measured robustly the same data gives MAD/median 0.15% where that statistic said 3.3%.

The two issues above survived that process and are verified against current main; the discarded hypotheses are not in them.