Skip to content

Commit c6a9979

Browse files
wan9chiclaude
andcommitted
feat(fspy): route preload allocations through a lock-free global allocator
Alternative to #596 for the same problem: the preload library runs inside libc calls that programs may make from a signal handler or from the child of fork() in a multithreaded process, where libc malloc's lock may be held by a thread that is paused or gone. Where #596 hands each intercepted call its own bump arena, and converts call sites one at a time, this installs one lock-free allocator as the preload cdylib's #[global_allocator]. Every Rust allocation in the library is covered at once, with no call-site changes: power-of-two size classes carve blocks out of 1 MiB mmap'd slabs, freed blocks recycle through per-class Treiber free lists made ABA-resistant by a 40-bit generation tag, and larger or over-aligned requests map directly. Includes the same access-relative benchmark suite as #596 so the two approaches can be compared on identical workloads. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 5ed7a5a commit c6a9979

15 files changed

Lines changed: 1574 additions & 30 deletions

File tree

Cargo.lock

Lines changed: 8 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Cargo.toml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,7 @@ materialized_artifact = { path = "crates/materialized_artifact" }
7373
materialized_artifact_build = { path = "crates/materialized_artifact_build" }
7474
flate2 = "1.0.35"
7575
fspy = { path = "crates/fspy" }
76+
fspy_alloc = { path = "crates/fspy_alloc" }
7677
fspy_benchmark_launcher = { path = "crates/fspy_benchmark_launcher", artifact = "bin" }
7778
fspy_benchmark_target = { path = "crates/fspy_benchmark_target", artifact = "bin" }
7879
fspy_detours_sys = { path = "crates/fspy_detours_sys" }
@@ -120,6 +121,7 @@ ref-cast = "1.0.24"
120121
regex = "1.11.3"
121122
rusqlite = "0.39.0"
122123
rustc-hash = "2.1.1"
124+
rustix = { version = "1", default-features = false, features = ["mm", "param", "use-libc-auxv"] }
123125
# SeccompAction::UserNotif (SECCOMP_RET_USER_NOTIF) was added after the latest published release (v0.5.0)
124126
seccompiler = { git = "https://github.com/rust-vmm/seccompiler", rev = "08587106340b8e3cb361c7561411510039436857" }
125127
serde = "1.0.219"

crates/fspy_alloc/Cargo.toml

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
[package]
2+
name = "fspy_alloc"
3+
edition = "2024"
4+
license.workspace = true
5+
publish = false
6+
7+
[lib]
8+
doctest = false
9+
10+
[target.'cfg(unix)'.dependencies]
11+
rustix = { workspace = true }
12+
13+
[lints]
14+
workspace = true

crates/fspy_alloc/src/class.rs

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
//! Size-class policy: which layouts the pool serves, and at what block size.
2+
3+
use core::alloc::Layout;
4+
5+
const MIN_CLASS_SHIFT: u32 = 4;
6+
const MAX_CLASS_SHIFT: u32 = 16;
7+
pub const CLASS_COUNT: usize = (MAX_CLASS_SHIFT - MIN_CLASS_SHIFT) as usize + 1;
8+
const MIN_BLOCK_SIZE: usize = 1 << MIN_CLASS_SHIFT;
9+
pub const MAX_BLOCK_SIZE: usize = 1 << MAX_CLASS_SHIFT;
10+
/// Block areas start at this alignment within a slab, making it the largest
11+
/// alignment the pool can serve; stricter layouts map directly.
12+
pub const MAX_POOL_ALIGN: usize = 4096;
13+
14+
pub const fn block_size(class: usize) -> usize {
15+
1 << (MIN_CLASS_SHIFT as usize + class)
16+
}
17+
18+
/// Returns the size class for `layout`, or `None` if the request must be
19+
/// mapped directly (too large or over-aligned).
20+
pub const fn class_of(layout: Layout) -> Option<usize> {
21+
if layout.align() > MAX_POOL_ALIGN {
22+
return None;
23+
}
24+
let mut size = layout.size();
25+
// A block of `size >= align` at a `min(block size, 4 KiB)` boundary is
26+
// aligned to `align` (both are powers of two and `align <= 4 KiB`).
27+
if size < layout.align() {
28+
size = layout.align();
29+
}
30+
if size < MIN_BLOCK_SIZE {
31+
size = MIN_BLOCK_SIZE;
32+
}
33+
if size > MAX_BLOCK_SIZE {
34+
return None;
35+
}
36+
let shift = size.next_power_of_two().trailing_zeros();
37+
Some((shift - MIN_CLASS_SHIFT) as usize)
38+
}

crates/fspy_alloc/src/lib.rs

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
//! Lock-free, async-signal-safe global allocator for the fspy preload library.
2+
//!
3+
//! The preload library interposes libc functions that POSIX declares
4+
//! async-signal-safe (`open`, `stat`, `execve`, ...). Programs may call these
5+
//! from signal handlers, and — more commonly — from the child of `fork()` in a
6+
//! multithreaded process, where only async-signal-safe calls are permitted:
7+
//! the libc allocator's locks may be held forever by threads that no longer
8+
//! exist after the fork. Routing the preload's Rust allocations through this
9+
//! allocator keeps them safe in both contexts:
10+
//!
11+
//! - **No locks.** Every state transition is a lock-free compare-and-swap
12+
//! loop: an attempt only retries because another running thread completed
13+
//! its operation, so nothing ever waits on state that a thread which
14+
//! vanished at `fork()` — or sits suspended under a signal handler — would
15+
//! have to release. (Lock-free, not wait-free: an individual operation has
16+
//! no fixed retry bound under active contention.)
17+
//! - **No thread-locals.** TLS first-touch allocates through libc malloc on
18+
//! some platforms (macOS thread-local variables), which would reintroduce
19+
//! the hazard this crate exists to remove.
20+
//! - **mmap-backed.** Memory comes straight from the kernel. On Linux the
21+
//! allocator relies on nothing from libc: mapping syscalls are issued
22+
//! directly (rustix's raw backend) and even the page size is discovered by
23+
//! probing with raw syscalls. On macOS, which has no stable raw-syscall
24+
//! ABI, calls go through the thin libSystem stubs. libc malloc is never
25+
//! called anywhere.
26+
//!
27+
//! Design: power-of-two size classes (16 B ..= 64 KiB) carve blocks out of
28+
//! 1 MiB slabs; freed blocks recycle through a per-class Treiber free list
29+
//! made ABA-safe by a generation tag. Requests larger than the biggest class
30+
//! (or over-aligned beyond 4 KiB) map and unmap directly. See the `pool`
31+
//! module for the details.
32+
//!
33+
//! Because the allocator is a `const`-initialized static with no lazy setup,
34+
//! it works from the very first allocation in the process — even before the
35+
//! preload library's constructor runs.
36+
37+
#![cfg_attr(not(test), no_std)]
38+
39+
// Compile as an empty crate on non-unix targets: the allocator backs the unix
40+
// preload library. A Windows backend can be added alongside `sys::Mmap` if
41+
// the Windows preload ever needs one.
42+
43+
#[cfg(unix)]
44+
mod class;
45+
#[cfg(unix)]
46+
mod mapping;
47+
#[cfg(unix)]
48+
mod mmap;
49+
#[cfg(unix)]
50+
mod pool;
51+
#[cfg(unix)]
52+
mod slab;
53+
#[cfg(unix)]
54+
mod sys;
55+
56+
#[cfg(unix)]
57+
use core::{
58+
alloc::{GlobalAlloc, Layout},
59+
ptr::{self, NonNull},
60+
};
61+
62+
#[cfg(unix)]
63+
use crate::{mmap::Mmap, pool::Pool};
64+
65+
/// A lock-free, async-signal-safe, fork-safe [`GlobalAlloc`] implementation.
66+
///
67+
/// Intended to be installed as the `#[global_allocator]` of the fspy preload
68+
/// library. All memory comes from anonymous mappings; libc malloc is never
69+
/// called, no locks are taken, and no thread-local state is used.
70+
///
71+
/// Capacity is bounded by design: each size class can hold at most 256 slabs
72+
/// of 1 MiB (roughly 200 MiB per class). Requests beyond that — far outside
73+
/// anything the preload library does — fail like any other out-of-memory
74+
/// condition (`alloc` returns null).
75+
#[cfg(unix)]
76+
pub struct FspyAlloc {
77+
pool: Pool<Mmap>,
78+
}
79+
80+
#[cfg(unix)]
81+
impl FspyAlloc {
82+
/// Creates the allocator. `const` so it can back a `static` with no
83+
/// runtime initialization.
84+
#[must_use]
85+
pub const fn new() -> Self {
86+
Self { pool: Pool::new() }
87+
}
88+
}
89+
90+
#[cfg(unix)]
91+
impl Default for FspyAlloc {
92+
fn default() -> Self {
93+
Self::new()
94+
}
95+
}
96+
97+
// SAFETY: `Pool` hands out blocks that are non-null, at least `layout.size()`
98+
// bytes large, aligned to at least `layout.align()`, and exclusively owned
99+
// until returned via `dealloc`. Allocation failure is reported as null, and
100+
// none of the methods unwind.
101+
#[cfg(unix)]
102+
unsafe impl GlobalAlloc for FspyAlloc {
103+
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
104+
self.pool.alloc(layout).map_or(ptr::null_mut(), NonNull::as_ptr)
105+
}
106+
107+
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
108+
let Some(ptr) = NonNull::new(ptr) else { return };
109+
// SAFETY: per the GlobalAlloc contract, `ptr` was returned by this
110+
// allocator for this `layout`.
111+
unsafe { self.pool.dealloc(ptr, layout) }
112+
}
113+
114+
unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
115+
self.pool.alloc_zeroed(layout).map_or(ptr::null_mut(), NonNull::as_ptr)
116+
}
117+
118+
unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
119+
let Some(ptr) = NonNull::new(ptr) else { return ptr::null_mut() };
120+
// SAFETY: per the GlobalAlloc contract, `ptr` was returned by this
121+
// allocator for this `layout`, and `new_size` is non-zero.
122+
unsafe { self.pool.realloc(ptr, layout, new_size) }.map_or(ptr::null_mut(), NonNull::as_ptr)
123+
}
124+
}

crates/fspy_alloc/src/mapping.rs

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
//! Owned memory regions obtained from a [`Sys`] provider.
2+
3+
use core::{marker::PhantomData, mem, ptr::NonNull};
4+
5+
use crate::sys::Sys;
6+
7+
/// An owned region obtained from `S`, released on drop.
8+
///
9+
/// This is the only place that calls [`Sys::unmap`]: pool code either lets a
10+
/// `Mapping` drop (probe scratch, install races, freed large allocations) or
11+
/// deliberately leaks it with [`Mapping::into_raw`] (published slabs, live
12+
/// large allocations). Reconstructing ownership from a raw pointer via
13+
/// [`Mapping::from_raw`] is the single unsafe step.
14+
pub struct Mapping<S: Sys> {
15+
ptr: NonNull<u8>,
16+
size: usize,
17+
align: usize,
18+
sys: PhantomData<fn() -> S>,
19+
}
20+
21+
impl<S: Sys> Mapping<S> {
22+
/// Maps `size` bytes of zero-initialized memory aligned to `align`
23+
/// (a power of two). Returns `None` when memory is exhausted.
24+
pub fn new(size: usize, align: usize) -> Option<Self> {
25+
let ptr = S::map(size, align)?;
26+
Some(Self { ptr, size, align, sys: PhantomData })
27+
}
28+
29+
/// Reclaims ownership of a mapping previously released with
30+
/// [`Mapping::into_raw`].
31+
///
32+
/// # Safety
33+
///
34+
/// `ptr` must have come from `Mapping::<S>::into_raw` (or `Sys::map`)
35+
/// with exactly this `size` and `align`, the region must not be in use,
36+
/// and ownership must not be reclaimed twice.
37+
pub unsafe fn from_raw(ptr: NonNull<u8>, size: usize, align: usize) -> Self {
38+
Self { ptr, size, align, sys: PhantomData }
39+
}
40+
41+
/// The mapped region's base address.
42+
pub const fn ptr(&self) -> NonNull<u8> {
43+
self.ptr
44+
}
45+
46+
/// Releases ownership without unmapping; the region lives until (unless)
47+
/// [`Mapping::from_raw`] reclaims it.
48+
pub const fn into_raw(self) -> NonNull<u8> {
49+
let ptr = self.ptr;
50+
mem::forget(self);
51+
ptr
52+
}
53+
}
54+
55+
impl<S: Sys> Drop for Mapping<S> {
56+
fn drop(&mut self) {
57+
// SAFETY: this type owns the mapping (constructed from `Sys::map`
58+
// directly or via the `from_raw` contract), and after drop nothing
59+
// can use it.
60+
unsafe { S::unmap(self.ptr, self.size, self.align) }
61+
}
62+
}

0 commit comments

Comments
 (0)