Skip to content

bench: refresh results on post-#552 main; update README benchmark section - #578

Merged
nooga merged 3 commits into
mainfrom
bench/refresh-post-552-results
Jul 19, 2026
Merged

bench: refresh results on post-#552 main; update README benchmark section#578
nooga merged 3 commits into
mainfrom
bench/refresh-post-552-results

Conversation

@nooga

@nooga nooga commented Jul 19, 2026

Copy link
Copy Markdown
Owner

Recapture of benchmark/results.md on main at 2e10c0f (same M1 Pro as the #552 capture), after the 2026-07-18/19 perf batch (#558, #559, #560, #563, #567, #568, #569, #571) merged.

Deltas vs the #552 capture (VM leg)

benchmark #552 now delta
transducers 60.9ms ± 21.3 46.5ms ± 0.9 −24%, σ collapses
reduce 41.0ms 39.3ms −4%
persistent-map 23.0ms 22.2ms −3.5%
loop-recur 70.9ms 68.8ms −3%
tak 2.432s 2.397s −1.4%
fib 2.424s 2.416s ~flat
map-filter 10.7ms 11.3ms within σ

AOT leg: map-filter −16% (now beats the VM leg, flipping the #552 result), transducers −13%, the rest ~flat. The transducers win (and its variance collapse) tracks #560's lock-free realized-LazySeq access plus #567's valueEquiv fast paths; fib/tak stay flat as expected — they're call+add/sub workloads that don't touch the changed paths.

Startup reads +2ms in the tables but a direct interleaved hyperfine A/B of the #552 binary vs main (30 runs each) shows 10.1ms vs 10.2ms — capture-day drift, not a regression.

README

The root README benchmark section still showed the pre-#552 lineup (joker/go-joker/gloat) that the suite dropped; it now shows the VM/AOT/babashka/JVM matrix results.md actually measures, with the fresh numbers, and the intro's size/cold-start claims are corrected to ~13MB / ~10ms.

…tion

Recapture after the 2026-07-18/19 perf batch (#558-#571, #560): transducers
-24% on the VM leg (and its variance collapses with #560's lock-free realized
LazySeq access), map-filter AOT now beats the VM leg, -3-4% broadly elsewhere.
README benchmark table switches from the dropped joker/go-joker/gloat lineup
to the VM/AOT/babashka/JVM matrix results.md actually measures.

@mparrett mparrett left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Data checks out — I recomputed the ratio columns and cross-checked the body's deltas against the tables; all correct, and fib/tak staying flat is a nice control for the attribution story. Three README prose nits, none blocking: (1) map/filter at 11.3ms vs Babashka's 19.2ms is a 1.7× win, not "neck and neck" (that's persistent-map); (2) "order of magnitude ahead of… warm JVM" — warm JVM fib is ~0.2s once you back out the ~360ms startup baked into the column, so AOT is ~2× there, not 10×; (3) "JVM dominates long compute" now only holds against the VM leg — the previous sentence says AOT beats it.

One question: babashka moved −26%/−34% on persistent-map/transducers between captures — same bb version both days? Worth a version note in results.md for future cross-capture reads.

map/filter is a 1.7x win over babashka, not neck-and-neck (that's
persistent-map); AOT vs warm JVM compute is ~2x, not an order of magnitude
(that figure is vs babashka); and the warm JVM only outruns the VM leg,
not the AOT build.
@nooga

nooga commented Jul 19, 2026

Copy link
Copy Markdown
Owner Author

All three prose nits fixed in 81618ab — map/filter now stated as a 1.7× win, AOT-vs-warm-JVM downgraded to ~2× (the order-of-magnitude figure is vs babashka only), and the closing sentence now says the warm JVM outruns the VM leg but not the AOT build.

On the babashka movement: same bb version both days — both captures record v1.12.217 in the Runtimes table — so it's capture-day conditions, not a version bump. The #552 capture day was visibly noisy in exactly that workload family (let-go transducers carried σ ±21.3ms that day, and bb persistent-map ±6.1ms); yesterday's run is the tighter one across every runtime. Versions are already recorded per capture, but a capture-date line in the generated header would make these cross-capture reads easier — worth a small run.sh follow-up.

The '2x ahead of warm JVM' figure came from subtracting measured startup
from the one-shot JVM wall clock, but that remainder still contains JIT
warmup and script compilation. Measured in-process on the same machine
(12 iterations each): steady-state HotSpot fib(35) is ~76ms vs ~99ms for
the AOT build (flat from iteration 0, as expected for native code). Warm
JVM is ~1.3x ahead of AOT; what AOT wins is one-shot wall clock (~5x).
@nooga

nooga commented Jul 19, 2026

Copy link
Copy Markdown
Owner Author

Correction on nit (2), with measurements this time: the ~2× figure (mine originally, and the derived version in the review) was still wrong, because the one-shot JVM remainder (wall − startup ≈ 200ms) contains JIT warmup and script compilation, not just warm compute.

Measured in-process on the capture machine, 12 iterations each of the identical fib(35):

  • JVM (clj, System/nanoTime around each call): 100.6 → 93 → 81 → settles at ~76ms by iter 3
  • AOT (letgo-aot, same driver ns, same timing): ~99ms flat from iter 0 — native code, no warmup, and it corroborates the wall−startup derivation (110.4 − 10.7 ≈ 99.7ms)

So fully-warm HotSpot is ~1.3× ahead of AOT on this workload; what AOT wins is one-shot wall clock (0.11s vs 0.57s, ~5×). README now says exactly that (bd69524).

@mparrett
mparrett self-requested a review July 19, 2026 02:32

@mparrett mparrett left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the follow-up. LGTM.

@nooga
nooga merged commit 9dd835e into main Jul 19, 2026
12 checks passed
@nooga
nooga deleted the bench/refresh-post-552-results branch July 19, 2026 02:41
mparrett added a commit that referenced this pull request Jul 20, 2026
The Performance table presents one undifferentiated AOT column, which
makes the reduce row (1.7x slower under AOT) read as an anomaly. The
emitted Go splits the fixtures cleanly: fib/tak/loop-recur lower to
native code over unboxed ints (the 20-26x wins); the four seq rows only
get a native run wrapper around the same boxed runtime calls the VM
makes, so they tie or lose by construction until #270/#358 reach those
bodies. Name the split under the table so every row reads as expected.

Framing only — numbers are unchanged from the #578 recapture.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mparrett added a commit that referenced this pull request Jul 20, 2026
The Performance table presents one undifferentiated AOT column, which
makes the reduce row (1.7x slower under AOT) read as an anomaly. The
emitted Go splits the fixtures cleanly: fib/tak/loop-recur lower to
native code over unboxed ints (the 20-26x wins); the four seq rows only
get a native run wrapper around the same boxed runtime calls the VM
makes, so they tie or lose by construction until #270/#358 reach those
bodies. Name the split under the table so every row reads as expected.

Framing only — numbers are unchanged from the #578 recapture.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants