Performance measurement status: ratchet is not consulted; startup regressed 5ms -> 9ms unnoticed

[nnunley]

Problem

Performance regressions land silently. The bench-ratchet machinery exists (make bench-ratchet, docs/perf/baseline.json, #651) but nothing in the development loop requires consulting it, so work that costs performance merges without anyone deciding to spend it.

Concrete symptom: startup time regressed from ~5ms to ~9ms wallclock (lg -e '(+ 1 1)', measured 2026-07-31 on main @ c809e33). No one chose that cost; it accumulated.

What this issue asks for

A general performance-measurement status discipline, not a one-off fix:

• A visible answer to "are we at par with baseline?" — a status surface (CI lane, make target summary, or perf page) that reports current vs docs/perf/baseline.json including startup time.
• Startup time tracked as a first-class ratchet metric (it is the user-facing number; bootprobe/lgbstat from feat(cmd): bootprobe + lgbstat — core boot and bundle diagnostics #623 give the decomposition).
• Root-cause and recovery of the lost ~4ms startup, or an explicit recorded decision accepting it.

Non-goals

Specific optimizations (those belong under #464 and its children). This is about measurement + status, so future losses are chosen, not discovered.

Related: #464 (runtime performance epic), #651 (baseline seeding from CI), #623 (bootprobe/lgbstat diagnostics).

[mparrett]

Two more instances of the shape you describe here, both found while reviewing
#648. Neither needs its own issue — they are the same problem this one names,
so I am parking them here rather than spawning tickets.

The bytecode census gate runs in no workflow, and its baseline has already
drifted. ir-stress-bytecode-gate (Makefile) checks lowering coverage
against docs/perf/ir-stress-bytecode-baseline.edn, captured 2026-07-22 by
#580. It appears in no file under .github/workflows/. I ran it on main
@ cb99d7c5 as a control while reviewing #648, and main fails its own
committed baseline — 719 failures against 719 baselined, with
:lower/not-on-stack/call +1 and :lower/not-on-stack/eq +1. Small, and not
anyone's regression in particular, which is the point: the file says "only
tightens, never loosen without investigating" and nothing has been enforcing
that for nine days. Same shape as baseline.json in your description, one
directory over.

The anchor sits in a different process from most of what it normalizes. anchorPackage is pkg/vm (cmd/bench-ratchet/main.go:90), and the anchor is described in-tree as "the divisor for EVERY normalized ratio" (:1489). But both the full profile (:747) and the fast gate (:781) build a job list in which the anchor is its own captureJob — a separate go test invocation — while the jobs it normalizes run elsewhere: suitePackage (:97), irCompilePackage, and initPackage = pkg/compiler (:118), which is the startup measurement this issue is about.

pr-fast is the exception, and deliberately: it returns a single job with the anchor folded into the filter (:704), and the comment above prFastFilter says why — "the whole set is one pkg/vm job. It includes the anchor so normalization works from that one job." So the property was understood and engineered for one profile of three.

Why that matters for a status surface. The ±15% anchor-drift warning (:1496, :1731) catches the noisy failure — it was added after a bimodal ~1.06 vs ~1.85 ns anchor on an M3 manufactured +38% phantom regressions. It cannot catch the quiet one. An anchor that never shared a process with what it scales is structurally unable to observe that variance, so it sits inside the threshold and certifies the session as clean.

The same design was ported to paserati, where the anchor is captured in a separate process and at b.N=1e9, so it cannot observe the regime it normalizes — filed as nooga/paserati#49, which cites pr-fast here as the pattern that gets it right. Their session's headline variance numbers were later withdrawn as an artifact of a max/min − 1 summary over five samples, so it is the structural point that stands, not the magnitudes. A status surface publishing anchor-normalized numbers as settled inherits the structural problem regardless.

Two suggestions for the surface this issue asks for, both cheap:

• List every ratchet and gate that exists alongside when it last ran. The census gate would have shown up as "never, in CI" immediately.
• Mark which normalized numbers came from the same process as the anchor. Today that is pr-fast and nothing else.

The reducer half is already handled: #561 replaced min with a warmup-discarding median (reduceSamples, :961). One caveat worth recording: the in-process climb its comment describes did not reproduce on paserati, where varying only -count gave 0.3% spread at count 3 against 3.2% at count 1 — more stable, not less. So the climb is benchmark-dependent rather than universal, and warmup-discard is cheap insurance either way. This is the sampling half.

[mparrett]

This is the connective tissue this issue is asking for. Writing it here rather than as a fifth issue; if the shape is right I'll turn it into docs/design/perf-lanes.md.

The problem

Answering "which numbers am I looking at?" currently requires a person. Four committed baselines gate or inform performance work, and they disagree about what they even record:

baseline	machine identity	capture provenance	enforced by
docs/perf/baseline.json	machines{arch/CPU}	per-entry stamps	make bench-ratchet, perf-pr.yml
docs/perf/ir-stress-baseline.edn	none	:captured "2026-07-21"	make ir-stress-gate
docs/perf/ir-stress-bytecode-baseline.edn	none	:captured "2026-07-22"	nothing
docs/perf/fanout-baseline.edn	none	none	make fanout-ratchet

One of the four knows what machine it came from. One is enforced by no workflow at all, and it has already drifted: main at cb99d7c5 fails its own committed baseline by two failures against the 22 July capture, measured 2026-08-01. Nothing in CI is positioned to notice: no workflow references ir-stress or fanout at all; both live only as Makefile targets.

That is not a coincidence. A baseline with no lane identity has nowhere to record where it came from, so nothing can notice when it stops matching.

What already exists

The lane key is not a new idea here: it is already computed, three times, in three places that do not talk to each other:

• machineKey() in cmd/bench-ratchet/main.go:1172 — "a filesystem-safe token identifying the machine CLASS".
• perfdata.MachineKey — the partition key for Baseline.Machines, e.g. arm64/Apple M3.
• perf-timeline.yml:109 — tags each snapshot filename with the canonical <arch>-<cpumodel>, explicitly "the same <arch>/<CPUModel> the timeline partitions on".

So a per-machine capture already runs in CI on a matrix of amd64 ubuntu and arm64 macos-14, already lands on the perf-data branch, and is already in the perfdata.Baseline shape. #651 makes exactly this point: the formats are already identical.

The axis is built. Its population is not. Across perf-data's timeline/, the amd64 leg has produced five distinct machine keys, because the ubuntu runner pool rotates CPU models: EPYC 7763 (88 snapshots), EPYC 9V74 (35), Xeon Platinum 8370C (7), Xeon Platinum 8573C (5), Xeon 6973P-C (4). Only the macos key is stable. These are distinct lanes rather than noise: per-benchmark medians between 9V74 and 7763 span 0.63x to 1.38x. So a correctly computed lane identity still leaves a lane set that nothing declares, populates deliberately, or tracks. Numbers and method are on #651.

And it stops at baseline.json. The other three baselines have no machine field at all.

The proposal

A lane is the identity of a measurement, carried by the artifact rather than reconstructed by whoever reads it. Four fields, the same envelope on every committed baseline:

• machine: arch/CPUModel, the key that already exists.
• profile: what was measured. bench-ratchet already has this concept internally (pr-fast, the fast gate, -full) but does not stamp it into the artifact; the census baselines have an equivalent notion in their corpus selection and record it only as English inside :how.
• target: what the numbers are compared against. Three distinct answers are in flight today and none is machine-readable: ratchet-over-all-history (baseline.json), a pinned release (bench: compare against a pinned-release baseline #655's benchmark/BASELINE_REF), and the previous capture (the census .edn files).
• provenance: commit, timestamp, command, and whether the capture was gated. Today this is a :captured date plus a freeform :how string; fanout-baseline.edn has neither.

Nothing about the numbers changes. This is an envelope, not a new metric.

What it buys

A quiet-machine or cloud run stops being a threat. Re-measuring on a dedicated box currently raises "does this invalidate the existing numbers?" for every artifact except baseline.json, where the answer is already no: an unseen key seeds a new profile and prints that it is doing so (main.go:403-406). With lanes that answer generalizes: a new machine is a new lane everywhere, not an overwrite anywhere. benchmark/results.md is the clearest current casualty: it records its machine as one line of prose, so regenerating it elsewhere rewrites every row under a changed heading and a row-wise diff reads as performance movement.

This issue's status surface becomes a table rather than a project. The ask here is "a visible answer to 'are we at par with baseline?'". Over lanes that is a listing: each lane, when it last ran, on what, against what, and whether anything enforces it. The census gate would have shown up as never-in-CI the first time anyone looked.

Anchor scope becomes expressible. bench-ratchet normalizes every ratio by an anchor benchmark, and in the full and fast-gate profiles that anchor is captured in its own go test process, separate from the jobs it scales (main.go:747, :781). pr-fast is the exception and deliberately so: it folds the anchor into a single job precisely so "normalization works from that one job" (:704). That distinction is real, understood in-tree, and currently invisible in the output. It is a lane property: same-process or not.

It also has a measured consequence. On the amd64 timeline leg the anchor is effectively fixed, 1.245 to 1.250 ns across nine captures, so normalization there is inert. On macos-14 it moves 1.25x across captures and up to 1.118x within a single capture, injecting variance into every ratio on that leg, and the ±15% drift warning from #561 sits above that leg's routine behavior. Same tool, same code, opposite normalization behavior, and nothing in the artifact says which one you are reading. The structural half of this was filed independently against the port of this design in paserati (nooga/paserati#49), which cites pr-fast here as the pattern that gets it right.

#651 becomes the machine-axis implementation rather than a one-off import path, and #655's BASELINE_REF becomes the target axis rather than a second private convention.

Resolution, the fifth field

I left this out of the four at first and then measured it, and the measurement argues for including it.

paserati#48 found its tests benchmarks running at b.N <= 8, one case at b.N = 1, quantizing ns/op more coarsely than the 1-5% its gate exists to detect. pr-fast here does not have that problem: it is the sub-microsecond pkg/vm families at healthy iteration counts. -full does, in six places: BenchmarkClojureTestSuite and BenchmarkClojureTestSuiteCompileAndRun, each under bytecode, ir_bytecode, and aot_native, all capturing at one iteration on every runner, with capture-to-capture spread of 1.09x–1.79x on amd64 and 1.36x–2.13x on arm64.

So resolution is not a property of the repo or of the tool. It differs between two profiles of the same binary, and a reader of baseline.json cannot tell which profile produced the row in front of them. That is the case for carrying it in the artifact.

Non-goals

• No threshold or gate-semantics changes. A lane says what a number is, not whether it passes.
• No format unification. The census baselines can stay .edn and baseline.json can stay JSON; the envelope is four fields, not a schema merge.
• No new CI cost. Everything proposed here describes captures that already run.