feat: add optional jemalloc global allocator (#402) - #417
Open
mwiewior wants to merge 1 commit into
Open
Conversation
Adds a `jemalloc` Cargo feature (off by default) that installs tikv-jemallocator as the global allocator on Linux/macOS. Gated to non-MSVC targets, so Windows keeps the system allocator. Default builds are byte-identical to before. On the single-thread interval-operation benchmark suite jemalloc was ~12% faster (geomean) with equal-or-lower peak RSS than the default system allocator. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mwiewior
force-pushed
the
feat/jemalloc-allocator
branch
from
July 1, 2026 08:38
933d18d to
0b07f2b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in
jemallocCargo feature (off by default) that installstikv-jemallocatoras the globalallocator. Addresses #402.
maturin build --features jemalloc(Linux/macOS).cfg(not(target_env = "msvc"))), so Windows keeps thesystem allocator (
tikv-jemallocatordoesn't support MSVC).the feature is enabled.
Why jemalloc (and not mimalloc)
I benchmarked default vs mimalloc vs jemalloc on the single-thread interval-operation
suite (databio, 8 operations,
POLARS_MAX_THREADS=1,-C target-cpu=native, macOSarm64). Two reasons jemalloc wins:
equal-or-lower peak RSS. mimalloc was faster too (−9.3%) but cost +36–40% peak RSS
on reduce ops (
merge/complement) because it retains freed segments.importon macOS arm64 — a load-timeEXC_BAD_ACCESSinmi_bbitmap_try_find_and_clear. It only works if pinned to thev2feature. jemalloc has no such issue.Runtime, vs default (geomean of per-case ratios)
Peak memory (whole-run RSS, identical workload)
Verification
RUSTFLAGS="-Dwarnings" cargo check(default) and--features jemallocboth clean.--features jemallocwheel and confirmedimport polars_bio+ anoverlaprun succeed on macOS arm64.
Follow-ups (not in this PR)
To ship jemalloc to end users, add
--features jemallocto the maturin args in theLinux/macOS jobs of
.github/workflows/publish_to_pypi.yml(leave Windows untouched).on in wheels — multithreaded allocation is where jemalloc typically helps most.
🤖 Generated with Claude Code