perf-page: capture the release reference per machine tier so "% vs v1.8.0" is comparable across CI runners

[mparrett]

Problem

The "Are we fast yet?" timeline reports every suite run as a percentage against the v1.8.0 release line, but that release line was captured on a single machine while the timeline points land on a rotating set of CI CPUs. The comparison mixes a real code delta with cross-runner microarchitecture differences, so the headline number is a hardware lottery rather than a measurement.

Two facts drive this:

1. The v1.8.0 reference is single-machine. docs/perf/historical/v1.8.0.json is schema v1, captured on arm64/Apple M3. Every timeline point — all of them amd64 EPYC/Xeon GitHub runners — is compared to that one arm64 capture through ratio_to_anchor. That is cross-architecture, not just cross-microarchitecture, and ratio_to_anchor does not normalize across it (the anchor is a tight ALU loop; the suite is memory/branch/GC-bound, and different CPUs scale the two by different factors, which is the same reason ci(perf): same-runner A/B microbenchmark check on PRs #328 and perf-page: CPU-tier filter and selectable baseline; release-baseline recapture #329 exist).

2. The badge is the single latest point. cmd/perf-page/main.go sets the status pill to the latest plotted value vs the reference. It is not a median or a best-of; it is whichever snapshot ran last, on whatever CPU that job happened to draw.

The result on the deployed page: the "End-to-end suite" badge reads "~39% under v1.8.0" not because of a code change but because the most recent snapshot landed on a faster runner tier. The last five suite snapshots, in the chart's own metric (ratio_to_anchor vs v1.8.0):

time (UTC)	CPU	% vs v1.8.0
07-19 01:24	EPYC 7763	−27.8%
07-19 01:25	EPYC 9V74	−38.9%
07-20 16:42	EPYC 7763	−28.4%
07-20 17:27	EPYC 9V74	−39.2%
07-20 17:29	EPYC 9V74	−41.4% ← current badge

Consecutive runs of the same code swing 28% ↔ 41% on runner assignment alone. Pooled across all 228 timeline snapshots the two dominant tiers sit at distinct medians: EPYC 7763 at −31% (n=137) and EPYC 9V74 at −43% (n=66). That split is the "bimodal" band structure the chart shows.

Why the anchor work already merged does not close this

#362 already made the anchor per-machine (schema v2 partitions
baseline.json by arch/cpu_model, each tier carrying its own anchor), and #329 added perf-page -cpu <substr> to filter the timeline to one tier. Those help, but:

• The deploy does not pass -cpu (wasm/Makefile runs perf-page -out … only), so the live chart still pools every tier.
• More fundamentally, a per-machine anchor is not enough. The v1.8.0 reference — the thing every percentage is measured against — is still one arm64 M3 capture. Per-tier normalization is impossible while the denominator lives on a single machine.

Proposal

Make the release reference multi-machine, then compute and present the comparison per tier.

1. Capture the v1.8.0 reference on each CI machine tier. v1.8.0 is a fixed tag, so this is a backfill rather than a fresh measurement: the workflow_dispatch snapshot path (ci(perf-timeline): add workflow_dispatch for historical backfill #236/ci(perf-timeline): resolve dispatch ref in a script step (support short SHAs) #238) and the release-tag backfill (ci(perf): arm64 timeline leg + auto-backfill of release tags #363) already exist. Dispatch a v1.8.0 capture that lands on each runner tier a few times and store the results as per-tier references (schema v2, keyed the same as baseline.json).

2. Compare tip-on-X against v1.8.0-on-X. Once the reference is per-tier, perf-page computes each snapshot's percentage against the matching tier's reference. tip@EPYC7763 / v1.8.0@EPYC7763 is a real number; tip@EPYC7763 / v1.8.0@M3 is not. This also removes the arm64-vs-amd64 mismatch.

3. Badge = aggregate over tiers, with the per-tier breakdown shown. Replace the "latest point" pill with a summary across tiers, e.g. mean(−31%, −43%, −30%, …), and render the per-tier rows alongside it. Two caveats worth encoding in the design:

   • Weight it explicitly. A plain mean treats an n=2 tier equal to an n=137 tier. Decide between equal-weight-per-tier ("typical across hardware types") and sample-weighted ("typical across our actual fleet"); they give different numbers. Show which.
   • The aggregate still drifts with GitHub's fleet. If Actions shifts its CPU mix, the aggregate moves with no code change. So the per-tier breakdown is the ground truth and the aggregate is a convenience line, so the page should show both, not the aggregate alone.

Relationship to existing work

• perf(ratchet): machine-profile baseline + machine-independent alloc gate + interactive multi-machine dashboard #362 — per-machine anchor + schema v2 baseline. This issue extends the same partitioning to the historical reference.
• perf-page: CPU-tier filter and selectable baseline; release-baseline recapture #329 — -cpu filter and selectable baseline. A per-tier reference is the capture-side complement to that render-side filter.
• ci(perf): same-runner A/B microbenchmark check on PRs #328 — same-runner A/B on PRs. Same principle (compare on identical hardware), applied to the release line instead of PR base-vs-head.
• perf(bench-ratchet): robust sampling — warmup-discarding median + anchor-drift warning #561 / perf: recapture baseline with warmup-discarding median sampling (stacks on #561) #564 — robust sampling and baseline recapture. The v1.8.0 reference recapture should use the same warmup-discarding median sampling so the reference is not itself a single cold sample.

The -cpu-in-the-deploy change is a valid one-line stopgap to stop the page from pooling tiers today, but it discards data from the other tiers. The multi-machine reference is the durable fix.