perf: the boxed variadic call ABI keeps being the wall — name it, instrument it, give it fixed-arity lanes

[mparrett]

What this is

An advisory synthesis, not a bug. While working the #700 recovery we went back through our perf notes from the last two months, and one pattern accounts for essentially every large program-level result — in both directions. Filing it so the pattern has a name and an issue, because to date every encounter with it has been rediscovered from scratch.

The pattern

Four investigations hit the same contract from four directions:

1. A static/profile audit of the generated native tree (July). Roughly 9,200 []vm.Value argument constructions and ~7,000 cached-var call sites; in a sample of 8,000 generated argument slices, 91.4% had arity 1–4. Escape analysis confirms representative slices heap-allocate. A whole-core native self-host ran ~26% faster than interpreted but allocated ~50% more objects — less interpreter machinery, more boundary tax.
2. The vm: bytecode invoke is ~1.7x costlier since #645 (explicit frame chain) #700 regression hunt (this month). The entire ~1.5× on callback-heavy shapes is the host→VM entry; pure in-VM invocation measured 1.00× vs v1.12.2. perf(vm): keep defer scaffolding out of the dispatch hot path #719 (defer scaffolding) and vm: PreparedCall — resolve once, reuse one frame for per-element callback invokes #720's PreparedCall (resolve-once + frame reuse) recover it and land below the v1.12.2 baseline — the first direct attack on the boundary that shipped.
3. bench/aot-vs-vm. AOT is 21–26× on typed kernels and neutral-or-slower on collection-heavy fixtures — fastest exactly where code stays inside one layer, slowest where it crosses the boundary per element. AOT's coverage problem and the VM's callback problem are the same wall.
4. The value-level version. A CPU profile of the seq-callback workload shows runtime.convT64 at ~8%: every Int above 255 heap-boxes on each arithmetic op because the operand stack is []Value.

The negative space matches: in-layer micro-optimizations have repeatedly shown strong micros and flat program results. And #700 itself was a composition effect no single change caused: the //lg:native hoist series (#639/#640) moved per-element loops to the native side, and #645 then made the native→VM edge the expensive one — each sound alone, and their interaction inverted the cost model with no instrument in place that could have shown it.

Proposal (three pieces, all incremental)

1. Fixed-arity call lanes as the internal contract. PreparedCall (vm: PreparedCall — resolve once, reuse one frame for per-element callback invokes #720) plus Invoke1..4-shaped paths for codegen and natives whenever arity is statically known, with the variadic ABI kept as the semantic fallback. The 91.4% number says the fast path covers most call sites.
2. A boundary-density report in codegen: per-function reason codes for typed-direct / guarded-native / same-package direct / cross-package direct / cached-var trampoline / generic invoke / boxing adapter. One instrument that serves three consumers: a data-driven selective-AOT policy (instead of optimizing direct-call coverage by anecdote), an admission gate for future //lg:native migrations of callback-taking fns (Retire the twelve hand registrations the generated registrar shadows #696/Drain the remaining hand registrations in pkg/rt onto //lg:native #697), and a fail-loud story for lowering downgrades.
3. A per-element callback workload in the perf gate (some/reduce with a closure over a large range) — the shape that multiplied vm: make direct bytecode calls non-recursive with an explicit frame chain #645's entry cost by N and shipped through a green micro gate. Complements the calibration work in Three gaps that let the perf gate report green through a regression #705.

Why file it as one issue

Each piece has been proposed or half-built separately. The claim here is that they are one architectural object: the call boundary. Treating it as the unit — rather than per-layer micro-work — is what would have prevented #700, explains the AOT collection results, and sets the criterion for future native hoists. If the shape looks right we can split out concrete sub-issues (the lanes, the report, the gate workload) and take the first cut on each.

Related: #700, #719, #720, #705, #696, #697, #620, #636, #258.