feat(pool/mem): Connection Pool Shard Memory Fragmentation Under High-Frequency Tenant Churn #141 - #159
Merged
JamesEjembi merged 8 commits intoAug 21, 2026
Conversation
Light clients tracking multiple finality gadgets with a single fixed sync cadence let slow chains fall behind and fast chains thrash, so a chain's cached sync-committee view drifts out of date and finality stalls or, worse, finalizes on a stale committee (issue VeriNode-Labs#136). This adds a cross_chain module that derives every timing bound per chain from its own block time. - types: ChainConfig with per-chain sync timeout = max(3 * block_time_ms, 60_000), sync interval = block_time_ms / 4, and clock-drift budget = 500 ms * finality_hops, plus operational constants for every VeriNode-Labs#136 invariant. - committee_sync: per-chain sync scheduling, exponential retry backoff (1s -> 2s -> 4s -> ... capped at 30s), and drift detection by sampling staleness or observed clock skew. - finality_verifier: 2/3+1 committee-weight threshold with a 1.5x sync-timeout grace period that withholds finalization while sync drift is detected, so a temporarily skewed committee view cannot finalize early. - header_cache: bounded cache of the 256 most recent headers per chain. - light_client: LightClientRegistry tying the above together and exporting a chain_finality_lag_ms gauge per connected chain. All arithmetic is integer-only and saturating and the module is dependency-free so it compiles under no_std (WASM) and is shared verbatim by off-chain relayers and monitoring agents. Adds an integration test simulating a 2s chain and a 15s chain sharing one light client with 800ms injected relay latency, asserting finality lag stays well under the 10s target on both (1.0s and 3.75s respectively), alongside unit tests across every submodule. Refs VeriNode-Labs#136
…rection (VeriNode-Labs#139) - Add src/pool/capacity/model_linear.rs: weighted-average linear capacity model used by the global coordinator (equal CPU/memory/bandwidth weights). - Add src/pool/capacity/model_nonlinear.rs: GC-pause and NUMA-aware non-linear model used by the local estimator. GC-pause penalty: reduces available capacity by gc_pause_ms/1000 for the next 10 s after each pause. NUMA penalty: +5% effective memory utilisation per extra NUMA node, capped at 35% (8 nodes). - Add src/pool/capacity/local_estimator.rs: per-node estimator (1 s interval) that samples raw measurements, runs both models, and forwards the raw measurements plus both estimates in a LocalEstimatorSnapshot to the coordinator. - Add src/pool/capacity/global_coordinator.rs: aggregator (5 s sync interval) that applies capacity_global = capacity_local * (1 - |diff|) as the correction factor. If divergence exceeds 10% for 3 consecutive cycles it emits a CapacityModelDivergence warning and switches to the conservative (lower) estimate. Emits ModelConverged when divergence drops back within tolerance. - Add src/pool/capacity/mod.rs and src/pool/mod.rs: module wiring and re-exports. - Register pub mod pool in src/lib.rs with a doc comment matching project style. - Add tests/capacity_planning_divergence_test.rs: integration test covering the GC-pressure simulation (100 ms pause every 5 s stays within 10% tolerance), divergence correction factor, three-consecutive-cycle warning, conservative estimate after warning, convergence clears conservative mode, NUMA penalty reduces local but not linear estimate, and overcommit-ratio constant. - Register [[test]] entry in Cargo.toml.
…nce-139 Resolved formatting-only conflicts in: - Cargo.toml: kept capacity_planning_divergence_test entry + accepted origin/main's [lints.clippy] section and removal of arbitrary dep - src/lib.rs: kept pub mod pool (issue VeriNode-Labs#139) alongside origin/main changes - src/cross_chain/{committee_sync,finality_verifier,header_cache,light_client}.rs: accepted origin/main rustfmt wrapping (no logic change) - tests/light_client_finality_skew_test.rs: accepted origin/main rustfmt wrapping including div_ceil refactor (no logic change)
- global_coordinator.rs: move extern crate alloc to top, replace alloc:: qualified paths with use imports, wrap sync_node signature to fit within 100-char max_width - local_estimator.rs: prefix unused idle_inputs parameter with _, fix double-space in comment alignment - model_nonlinear.rs: minor whitespace normalisation - mod.rs: sort NonLinearInputs re-export alphabetically within use block - tests/capacity_planning_divergence_test.rs: merge duplicate use groups from same crate, wrap long assert message with line continuation, use LocalEstimatorSnapshot directly (no full-path qualifier)
- committee_sync.rs: revert to origin/main exact bytes — rustfmt right-aligns the trailing comment in backoff_defers_the_next_sync test; our merge resolution broke that alignment - local_estimator.rs: remove double space before inline comment on secs_since_gc field (rustfmt normalises to single space) - tests/capacity_planning_divergence_test.rs: reorder use blocks so pool comes before pool::capacity, single space after max 1.2x comment
clippy::unnecessary_map_or fires on map_or(false, |s| s.field) — replace with is_some_and(|s| s.field) in GlobalCoordinator::is_conservative
…enant churn (VeriNode-Labs#141) - Replace free-list with BuddyAllocator (src/mem/buddy_allocator.rs) that tracks contiguous free regions and coalesces adjacent blocks on every free, eliminating pathological external fragmentation under >1000 allocs/sec churn. - Add ShardAllocator (src/pool/shard_allocator.rs) wrapping BuddyAllocator with single-slot allocate/free API, per-pool fragmentation_ratio gauge (PoolFragmentationGauge), and FRAGMENTATION_ALARM_RATIO = 0.30 threshold. - Add ShardDefragmenter (src/pool/shard_defragmenter.rs) implementing a background mark-sweep compaction pass that runs when fragmentation > 30% and pool has been idle for >= COALESCING_WINDOW_MS = 500ms. Emits ShardDefragStarted / ShardDefragComplete event pair to coordinate with tenant migration; ShardDefragComplete carries (old_slot, new_slot) pairs. - Add TenantRegistry (src/pool/tenant_registry.rs) wiring allocator and defragmenter together for tenant connect/disconnect lifecycle. Applies slot remappings atomically after defrag passes. - Declare pub mod mem in src/lib.rs; expose shard_allocator, shard_defragmenter, tenant_registry from src/pool/mod.rs. - Add shard_memory_fragmentation_test.rs: stress test simulating 50k connect/disconnect churn cycles verifying allocation success rate > 99.9%, correct event pair emission, gauge well-formedness, and slot consistency. - Fix two pre-existing clippy lints in buddy_allocator.rs (unused_mut, manual_is_multiple_of) that would have broken the -D warnings CI gate. Closes VeriNode-Labs#141
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #141
Fixes pathological external memory fragmentation in the shard allocator under high-frequency tenant churn (>1,000 connect/disconnect cycles per second).
Changes
\src/mem/buddy_allocator.rs\ + \src/mem/mod.rs\ (new)
\src/pool/shard_allocator.rs\ (new)
\src/pool/shard_defragmenter.rs\ (new)
\src/pool/tenant_registry.rs\ (new)
\src/pool/mod.rs\ (updated)
\src/lib.rs\ (updated)
\Cargo.toml\ (updated)
\ ests/shard_memory_fragmentation_test.rs\ (new)
Technical Invariants Satisfied (issue #141)
Testing / Validation