feat(pool/capacity): Capacity Planning Model Divergence Between Local Estimator and Global Coordinator #139 - #158
Merged
JamesEjembi merged 7 commits intoAug 22, 2026
Conversation
Light clients tracking multiple finality gadgets with a single fixed sync cadence let slow chains fall behind and fast chains thrash, so a chain's cached sync-committee view drifts out of date and finality stalls or, worse, finalizes on a stale committee (issue VeriNode-Labs#136). This adds a cross_chain module that derives every timing bound per chain from its own block time. - types: ChainConfig with per-chain sync timeout = max(3 * block_time_ms, 60_000), sync interval = block_time_ms / 4, and clock-drift budget = 500 ms * finality_hops, plus operational constants for every VeriNode-Labs#136 invariant. - committee_sync: per-chain sync scheduling, exponential retry backoff (1s -> 2s -> 4s -> ... capped at 30s), and drift detection by sampling staleness or observed clock skew. - finality_verifier: 2/3+1 committee-weight threshold with a 1.5x sync-timeout grace period that withholds finalization while sync drift is detected, so a temporarily skewed committee view cannot finalize early. - header_cache: bounded cache of the 256 most recent headers per chain. - light_client: LightClientRegistry tying the above together and exporting a chain_finality_lag_ms gauge per connected chain. All arithmetic is integer-only and saturating and the module is dependency-free so it compiles under no_std (WASM) and is shared verbatim by off-chain relayers and monitoring agents. Adds an integration test simulating a 2s chain and a 15s chain sharing one light client with 800ms injected relay latency, asserting finality lag stays well under the 10s target on both (1.0s and 3.75s respectively), alongside unit tests across every submodule. Refs VeriNode-Labs#136
…rection (VeriNode-Labs#139) - Add src/pool/capacity/model_linear.rs: weighted-average linear capacity model used by the global coordinator (equal CPU/memory/bandwidth weights). - Add src/pool/capacity/model_nonlinear.rs: GC-pause and NUMA-aware non-linear model used by the local estimator. GC-pause penalty: reduces available capacity by gc_pause_ms/1000 for the next 10 s after each pause. NUMA penalty: +5% effective memory utilisation per extra NUMA node, capped at 35% (8 nodes). - Add src/pool/capacity/local_estimator.rs: per-node estimator (1 s interval) that samples raw measurements, runs both models, and forwards the raw measurements plus both estimates in a LocalEstimatorSnapshot to the coordinator. - Add src/pool/capacity/global_coordinator.rs: aggregator (5 s sync interval) that applies capacity_global = capacity_local * (1 - |diff|) as the correction factor. If divergence exceeds 10% for 3 consecutive cycles it emits a CapacityModelDivergence warning and switches to the conservative (lower) estimate. Emits ModelConverged when divergence drops back within tolerance. - Add src/pool/capacity/mod.rs and src/pool/mod.rs: module wiring and re-exports. - Register pub mod pool in src/lib.rs with a doc comment matching project style. - Add tests/capacity_planning_divergence_test.rs: integration test covering the GC-pressure simulation (100 ms pause every 5 s stays within 10% tolerance), divergence correction factor, three-consecutive-cycle warning, conservative estimate after warning, convergence clears conservative mode, NUMA penalty reduces local but not linear estimate, and overcommit-ratio constant. - Register [[test]] entry in Cargo.toml.
…nce-139 Resolved formatting-only conflicts in: - Cargo.toml: kept capacity_planning_divergence_test entry + accepted origin/main's [lints.clippy] section and removal of arbitrary dep - src/lib.rs: kept pub mod pool (issue VeriNode-Labs#139) alongside origin/main changes - src/cross_chain/{committee_sync,finality_verifier,header_cache,light_client}.rs: accepted origin/main rustfmt wrapping (no logic change) - tests/light_client_finality_skew_test.rs: accepted origin/main rustfmt wrapping including div_ceil refactor (no logic change)
- global_coordinator.rs: move extern crate alloc to top, replace alloc:: qualified paths with use imports, wrap sync_node signature to fit within 100-char max_width - local_estimator.rs: prefix unused idle_inputs parameter with _, fix double-space in comment alignment - model_nonlinear.rs: minor whitespace normalisation - mod.rs: sort NonLinearInputs re-export alphabetically within use block - tests/capacity_planning_divergence_test.rs: merge duplicate use groups from same crate, wrap long assert message with line continuation, use LocalEstimatorSnapshot directly (no full-path qualifier)
- committee_sync.rs: revert to origin/main exact bytes — rustfmt right-aligns the trailing comment in backoff_defers_the_next_sync test; our merge resolution broke that alignment - local_estimator.rs: remove double space before inline comment on secs_since_gc field (rustfmt normalises to single space) - tests/capacity_planning_divergence_test.rs: reorder use blocks so pool comes before pool::capacity, single space after max 1.2x comment
clippy::unnecessary_map_or fires on map_or(false, |s| s.field) — replace with is_some_and(|s| s.field) in GlobalCoordinator::is_conservative
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Capacity Planning Model Divergence Between Local Estimator and Global Coordinator
closes #139
Summary of changes
This PR implements the full capacity planning model divergence fix described in issue #139. Each shard node's local estimator now sends both raw measurements and its locally-computed non-linear estimate to the global coordinator, letting the coordinator detect and correct for model divergence.
Files added
Technical invariants satisfied
Correction formula
\
capacity_global = capacity_local * (1 - |estimate_local - estimate_linear|)
\\
After 3 consecutive cycles with divergence > 10%, a \CapacityModelDivergenceWarning\ is logged and the coordinator uses \min(estimate_local, estimate_linear)\ until convergence.
GC-pause modeling
\model_nonlinear.rs\ reduces available capacity by \gc_pause_ms / 1000\ for the 10 s following each pause (\GC_PENALTY_WINDOW_S = 10).
Testing / validation performed