perf: the calibration anchor is captured in a different process and iteration regime from what it normalizes

[mparrett]

ratio_to_anchor divides every benchmark by BenchmarkRatchetAnchor, but the anchor is measured in a different go test process from most of what it normalizes, and in a very different iteration regime. The practical consequence is that anchor drift is a weaker soundness signal than it reads as.

The structure

defaultPackages is pkg/vm then github.com/nooga/paserati/tests (cmd/bench-ratchet/main.go:93). buildJobs makes one captureJob per package (:504), and captureOnePackage runs one exec.Command("go", "test", …) each. The anchor lives in pkg/vm (:79-80) — job 1.

So every tests benchmark is normalized by a number captured in a separate process, minutes earlier, with a compile in between.

The regime mismatch, which I think matters more

Across 80 measurements in one session the anchor recorded iterations: 1000000000 — Go's b.N cap — at 1.084 ns/op, identical to four significant figures every time. It is a fixed-iteration ~1.08 s register loop.

It is normalizing benchmarks that run at b.N ≤ 8 (see #48).

Those are different measurements of different things. The anchor cannot observe a tail event in a benchmark that executes eight iterations, because it isn't running then and wouldn't be sensitive to it if it were.

What that cost us

A same-host session held anchor drift to 0.031% across five rounds while tests.BenchmarkArith's reported value moved 70% between rounds. The drift gate passed, correctly by its own terms, and told us nothing. We treated 0.031% as "this session is clean" for some time before noticing.

let-go ships a ±15% anchor-drift warning (added after a bimodal M3 anchor manufactured +38% phantom regressions) and has the same blind spot for the quiet case — a drift gate catches a noisy anchor, not an insensitive one.

Suggested fix

Two options, not exclusive:

1. Capture the anchor once per package job, so it shares a process and time window with what it normalizes. Cheap, and it's the pattern let-go already uses in one of its three profiles — pr-fast returns a single job with the anchor folded into the benchmark filter, with a comment saying "It includes the anchor so normalization works from that one job." The property is understood there; it just wasn't carried into the other profiles.
2. Put the canary in the same duration/b.N regime as the benchmarks it certifies. A 1e9-iteration register loop and an 8-iteration script run are not comparable instruments.

One thing to avoid

An allocation-aware second anchor sounds like the obvious complement and has already been tried and shelved as a negative result: dividing by it inflated worst |median| from 4.09% to 13.23% with near-zero β, because sequential benchmarking can't share instantaneous contention. Worth reading that before re-attempting — a probe measured at a different moment cannot see a transient, which is the same objection that applies to the current anchor.

Scope

Verified against current main (cmd/bench-ratchet/main.go, 1178 lines). Independent of #22 — that changes which reducer runs over the samples, this is about what the samples are divided by.

Tracker: #23. Related: #48 (the b.N side of the same mismatch), #21.