bench: refresh results on post-#552 main; update README benchmark section - #578
Conversation
…tion Recapture after the 2026-07-18/19 perf batch (#558-#571, #560): transducers -24% on the VM leg (and its variance collapses with #560's lock-free realized LazySeq access), map-filter AOT now beats the VM leg, -3-4% broadly elsewhere. README benchmark table switches from the dropped joker/go-joker/gloat lineup to the VM/AOT/babashka/JVM matrix results.md actually measures.
mparrett
left a comment
There was a problem hiding this comment.
Data checks out — I recomputed the ratio columns and cross-checked the body's deltas against the tables; all correct, and fib/tak staying flat is a nice control for the attribution story. Three README prose nits, none blocking: (1) map/filter at 11.3ms vs Babashka's 19.2ms is a 1.7× win, not "neck and neck" (that's persistent-map); (2) "order of magnitude ahead of… warm JVM" — warm JVM fib is ~0.2s once you back out the ~360ms startup baked into the column, so AOT is ~2× there, not 10×; (3) "JVM dominates long compute" now only holds against the VM leg — the previous sentence says AOT beats it.
One question: babashka moved −26%/−34% on persistent-map/transducers between captures — same bb version both days? Worth a version note in results.md for future cross-capture reads.
map/filter is a 1.7x win over babashka, not neck-and-neck (that's persistent-map); AOT vs warm JVM compute is ~2x, not an order of magnitude (that figure is vs babashka); and the warm JVM only outruns the VM leg, not the AOT build.
|
All three prose nits fixed in 81618ab — map/filter now stated as a 1.7× win, AOT-vs-warm-JVM downgraded to ~2× (the order-of-magnitude figure is vs babashka only), and the closing sentence now says the warm JVM outruns the VM leg but not the AOT build. On the babashka movement: same bb version both days — both captures record v1.12.217 in the Runtimes table — so it's capture-day conditions, not a version bump. The #552 capture day was visibly noisy in exactly that workload family (let-go transducers carried σ ±21.3ms that day, and bb persistent-map ±6.1ms); yesterday's run is the tighter one across every runtime. Versions are already recorded per capture, but a capture-date line in the generated header would make these cross-capture reads easier — worth a small run.sh follow-up. |
The '2x ahead of warm JVM' figure came from subtracting measured startup from the one-shot JVM wall clock, but that remainder still contains JIT warmup and script compilation. Measured in-process on the same machine (12 iterations each): steady-state HotSpot fib(35) is ~76ms vs ~99ms for the AOT build (flat from iteration 0, as expected for native code). Warm JVM is ~1.3x ahead of AOT; what AOT wins is one-shot wall clock (~5x).
|
Correction on nit (2), with measurements this time: the ~2× figure (mine originally, and the derived version in the review) was still wrong, because the one-shot JVM remainder (wall − startup ≈ 200ms) contains JIT warmup and script compilation, not just warm compute. Measured in-process on the capture machine, 12 iterations each of the identical fib(35):
So fully-warm HotSpot is ~1.3× ahead of AOT on this workload; what AOT wins is one-shot wall clock (0.11s vs 0.57s, ~5×). README now says exactly that (bd69524). |
mparrett
left a comment
There was a problem hiding this comment.
Thanks for the follow-up. LGTM.
The Performance table presents one undifferentiated AOT column, which makes the reduce row (1.7x slower under AOT) read as an anomaly. The emitted Go splits the fixtures cleanly: fib/tak/loop-recur lower to native code over unboxed ints (the 20-26x wins); the four seq rows only get a native run wrapper around the same boxed runtime calls the VM makes, so they tie or lose by construction until #270/#358 reach those bodies. Name the split under the table so every row reads as expected. Framing only — numbers are unchanged from the #578 recapture. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Performance table presents one undifferentiated AOT column, which makes the reduce row (1.7x slower under AOT) read as an anomaly. The emitted Go splits the fixtures cleanly: fib/tak/loop-recur lower to native code over unboxed ints (the 20-26x wins); the four seq rows only get a native run wrapper around the same boxed runtime calls the VM makes, so they tie or lose by construction until #270/#358 reach those bodies. Name the split under the table so every row reads as expected. Framing only — numbers are unchanged from the #578 recapture. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Recapture of
benchmark/results.mdon main at 2e10c0f (same M1 Pro as the #552 capture), after the 2026-07-18/19 perf batch (#558, #559, #560, #563, #567, #568, #569, #571) merged.Deltas vs the #552 capture (VM leg)
AOT leg: map-filter −16% (now beats the VM leg, flipping the #552 result), transducers −13%, the rest ~flat. The transducers win (and its variance collapse) tracks #560's lock-free realized-LazySeq access plus #567's valueEquiv fast paths; fib/tak stay flat as expected — they're call+add/sub workloads that don't touch the changed paths.
Startup reads +2ms in the tables but a direct interleaved hyperfine A/B of the #552 binary vs main (30 runs each) shows 10.1ms vs 10.2ms — capture-day drift, not a regression.
README
The root README benchmark section still showed the pre-#552 lineup (joker/go-joker/gloat) that the suite dropped; it now shows the VM/AOT/babashka/JVM matrix
results.mdactually measures, with the fresh numbers, and the intro's size/cold-start claims are corrected to ~13MB / ~10ms.