perf(rt): restore reduce's ArrayVector and Range fast paths - #686
Conversation
Perf (pkg/vm micros) — base vs head, same runnerBase bench-ratchet — 0 regression(s) > 12.0% budget, 0 missing, 0 new
Showing 0 of 53 benchmarks (past budget). Full table in the run summary → |
Might be nice to surface wins here, too. 🤷🏻 |
|
Single-shot (EPYC 9V74): 0 of 53 benchmarks past the 12% budget, anchor stable at +0.2%. Repeat A/B (EPYC 7763, N=7 interleaved ABBA): all 53 families comparable, 0 missing / 0 new / 0 flaky, would-gate empty at 6%, 8% and 10%. Largest median 2.70% (VectorConj/ArrayVector/10); median of all medians −0.42%. The two families that would have shown trouble are flat: FuncInvoke/Closure +1.80%, FrameDispatch +0.70%. So the fix is neutral on the VM micros, which is what it should be — it touches pkg/rt, not pkg/vm. And "0 new" confirms the prediction: no reduce family exists, so this run says nothing about the actual win. One thing worth noting from the raw cycles: single-cycle deltas reach ±10% while every median sits under 3%. Cycle 3 is visibly contaminated — its anchor dropped 1.246 → 1.177 ns and most families spike in that one cycle. The interleaving plus median absorbed it exactly as #445 predicted, which is a decent argument for the repeat lane over the single-shot one. |
2220c61 to
18919d4
Compare
170d8b0 to
af3f870
Compare
| var knownShadowedHandRegistrations = []string{ | ||
| "clojure.core/conj", | ||
| "clojure.core/deref", | ||
| "clojure.core/get", | ||
| "clojure.core/int", | ||
| "clojure.core/name", | ||
| "clojure.core/namespace", | ||
| "clojure.core/nth", | ||
| "clojure.core/pop-binding!", | ||
| "clojure.core/push-binding!", | ||
| "clojure.core/some", | ||
| "clojure.core/str", | ||
| "clojure.core/subs", | ||
| } |
There was a problem hiding this comment.
I think the issue is that the conversion to lg native isn't complete yet, so these weren't covered as needed. This should be tracked by #531
…dowed duplicate reduce had two implementations: a NewCtxNativeFn closure in lang.go carrying ArrayVector and Range fast paths plus a reused two-element argument buffer, and the //lg:native Reduce/Reduce3 decls delegating to reduceColl, which had none of them. Both predate #639. The generated registration used to land in a distinct "clojure.core" namespace rather than canonical "core", so it was invisible and the closure won; #639 fixed that alias, and because the generated registrar drains last it now takes the var root. reduce lost the fast paths silently. The naive body allocates a fresh []vm.Value{acc, elem} per element and an ArrayChunk per 32 elements via ChunkedFirst — 66.5 MB against 17.5 MB reducing (range 1000000), and 33 GC cycles against 6. Measured 1.88x slower end to end (#656). Port both fast paths and the reused buffer into reduceColl and delete the closure, so there is one implementation. The buffer is shared by the general seq path too, which never had it: ec.Invoke does not retain its argument slice, so one buffer per reduce call beats one per element. Measured on darwin/arm64, hyperfine 3 warmup / 14 runs, reversed-order confirmed, reducing (range 1000000): pre-#639 34.6 ms 6 GC cycles main 64.9 ms 33 GC cycles this 41.3 ms 9 GC cycles Every reduce shape improves, not just the two with fast paths: over a list 37.1 -> 30.1 ms, over a vector 29.8 -> 25.5 ms, over a map 97.1 -> 90.1 ms. The residual against pre-#639 is not reduce-specific — `some`, `get`, `nth`, `conj`, `deref` and `str` all sit 1.06-1.49x above pre-#639 on main with no fast-path story, so it is separate and still open. Refs #656 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ar shadows A name registered both by hand in installLangNS and by the generated //lg:native registrar is a silent hazard rather than a style problem: the generated registrar drains last, so the hand-written closure never runs, and nothing stops the two bodies drifting. reduce drifted for months and the whole suite stayed green — including a test named TestReduceRangeFastPath, which asserts results, not the path. Record the collision where it already gets detected (setPrimitiveRoot's existing-binding branch) and expose the set, so adding a //lg:native decl for an already-hand-registered name fails here instead of years later in a benchmark. Twelve names are in the accepted set today; each is a duplicate implementation awaiting the same port-then-delete treatment reduce just got. The two bodies agree for the ones spot-checked (get, some); the rest are unaudited. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…hadow recordShadowedHandRegistration treated any pre-existing var as evidence of a hand-written registration. Registration state is process-global, so the second generated registration of the same ns/name finds the first one's root already interned and records its own predecessor as a collision. That made the ratchet depend on how many times the suite ran: `go test -count=2 ./pkg/rt` failed with phantom collisions for bind.probe.ns/probe-fn and test.reapplyhybrid/prim, both bound by tests that call the registration path directly. The recorder now skips names already present in genPrimBindings — it runs before the current binding is recorded, so a hit there means an earlier generated registration owns the var. TestReRegistrationIsNotAHandShadow pins the distinction; it fails without the check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
af3f870 to
fa641f7
Compare
|
Filed both under #531:
Also rebased onto main, which cleared the merge conflict. Checks are green. |
reduceis 1.88x slower than v1.12.2 onmain(#656, measured 2026-08-05). The cause is a duplicate registration, not anything in the primitives #639 hoisted.Root cause
reducehas had two implementations for months:vm.NewCtxNativeFnclosure inpkg/rt/lang.go, carrying an ArrayVector fast path, a Range fast path (direct arithmetic, zero seq/chunk allocation), and a reused two-element argument buffer;//lg:nativeReduce/Reduce3decls inpkg/rt/native_prims.go, delegating toreduceColl, which has none of that.Both predate #639. The generated registration used to land in a distinct
"clojure.core"namespace instead of canonical"core"— the alias bug #639's first commit fixed — so it was invisible and the closure won the name. Once the alias resolved, the generated registrar took the var root, because it drains afterinstallLangNS.reduceswitched to the naive body with no diff toreduceitself and no failing test.Reducing
(range 1000000)through that body allocates 66.5 MB against 17.5 MB: a fresh[]vm.Value{acc, elem}per element, plus anArrayChunkper 32 elements viaChunkedFirst. 33 GC cycles against 6.TestReduceRangeFastPathstayed green throughout. It asserts results, not which path produced them.What changed
Both fast paths and the reused buffer move into
reduceColl, and the lang.go closure is deleted, leaving one implementation. The general seq path gets the buffer too, which it never had:ec.Invokedoes not retain its argument slice, so one buffer perreducecall beats one per element.The rest of the change adds the missing guard. A name registered in both places is a silent hazard: the closure never runs, and nothing stops the two bodies drifting. Every generated registration that lands on a var an earlier registration already interned is now recorded, and a test asserts that set against an accepted list, so adding a
//lg:nativedecl for an already-hand-registered name fails in CI.The recorder ignores names an earlier generated registration installed. Registration state is process-global, so without that check a repeat registration records its own predecessor as a collision, making the set depend on how many times the registrar ran:
go test -count=2 ./pkg/rtreported phantom collisions for the two probe primitives thatpkg/rt's own tests bind through the registration path.Twelve names are in the accepted set:
conj,deref,get,int,name,namespace,nth,pop-binding!,push-binding!,some,str,subs. Each is a duplicate implementation awaiting the same port-then-delete treatment. All twelve pairs were compared body for body: the//lg:nativedecls are faithful ports, several carrying the closure's comments verbatim, soreduceis the only one that had drifted. The duplication is still worth retiring — a second copy nothing executes will drift again — but no other name is silently losing behavior today.Verification
(reduce + 0 (range 1000000))on darwin/arm64, hyperfine 3 warmup / 14 runs, confirmed in reversed order (2026-08-05):862c7cec)mainEvery reduce shape improves, not only the two with fast paths: over a list 37.1 → 30.1 ms, over a vector 29.8 → 25.5 ms, over a map 97.1 → 90.1 ms (pre-#639 parity).
Semantics spot-checked across 2-arg and 3-arg forms, nil and empty collections, vectors, lists, maps, and
reducedshort-circuit;pkg/rt's existingTestReduceRangeFastPathandTestReduceArrayVectorFastPathcover the ported paths.Two new tests cover the guard.
TestNoNewShadowedHandRegistrationsratchets the accepted set.TestReRegistrationIsNotAHandShadowpins that a repeat generated registration is not a collision; it fails when that check is removed.go test -count=2 ./pkg/rtis green.make check-generatedpasses;pkg/rt/native_prims.gofeeds the manifest digest, hence thegenerated.sumsbump.Residual
This closes most of the gap to pre-#639, not all of it, and the remainder is shared with other primitives. On
main,some(1.49x),get(1.18x),nth(1.17x),str(1.11x),conjandderef(1.06x) all sit above pre-#639 with no fast-path story of their own.someis clean at #639 and bisects to #645, so at least one further regression is in flight. Measured separately, not in this PR.