Improvement on Index handling and propogation on cluster - #114
Merged
Merged
Conversation
…rent_deposit_load`, introduce `load_peak`, and refactor test catalogue.
…ult `wait_level` in `load_latency.rs`.
… diagnostic latency tracking.
…g, update load scenarios, and reduce replication poll latency.
…eplication and Waiters - Introduced a single, non-blocking index hook for reactive replication and waiters. - Replaced polling mechanisms with notifier-driven index propagation for reduced latency. - Enhanced cluster replication and transaction wait paths with event-driven signaling. - Updated ADR documentation and added entry to ADR README.
- Introduced `IndexHook` for lock-free, reactive progress tracking of pipeline indices (Compute, Commit, Snapshot). - Added `set_index_hook` and `fire_index_hook` for stage notifications of index changes, enabling event-driven flows. - Updated `load` and `load_latency` binaries to demonstrate hook usage for mirroring indices in real-time. - Extended `Pipeline` and `Ledger` APIs for hook registration and propagation.
- Replaced polling mechanisms within `Waiter` with `IndexWatch` for reactive transaction-level waits. - Added granular index hooks in `LedgerSlot` for propagating ledger index updates to `Waiter` in real-time. - Incorporated `IndexWatch` for reactive waiting on `Compute`, `Commit`, and `Snapshot` indices. - Enhanced `ClusterNode` to integrate and manage reactive index propagation for single-node clusters. - Added in-process latency benchmarking tools (`cluster_load_latency`) to measure performance improvements.
…ive waiters from hanging - Ensure an index hook is registered before reseeding; return an error if missing. - Simplify hook re-application logic for rebuilt ledgers.
…nd Raft heartbeats - Replaced `replication_poll_ms` with reactive index propagation, eliminating 5 ms idle polling delays. - Integrated Raft heartbeats for safety and freshness, driven by `heartbeat_interval` instead of configurable delays. - Updated documentation and ADR-027 to reflect the removal of idle polling and the refinement of replication behavior.
- Eliminated unused `replication_poll_ms` setting from node configuration files (`node1.toml`, `node2.toml`, `node3.toml`). - Reflects the shift to reactive replication propagation and Raft heartbeats.
- Exposed `heartbeat_interval` from `Node` configuration for use in replication drivers. - Ensures heartbeat cadence aligns with election timeout constraints for safety and liveness.
…tion ID in WAL buffers - Introduced `last_tx_id_in` to efficiently iterate WAL records and extract the last transaction ID (`tx_id`) from metadata records. - Updated exports in `lib.rs` to include `last_tx_id_in`.
- Reflects transition to Raft-driven heartbeats for replication cadence.
- Eliminated unused `replication_poll_ms` references across process logic, tests, and configuration files. - Reflects the shift to reactive replication and Raft-driven heartbeats for replication cadence.
- Introduced `cluster_commit` as a reactive index in `Waiter`, propagated via Raft advances. - Replaced polling with `IndexWatch` for all transaction wait levels (Compute, Commit, Snapshot, ClusterCommit). - Simplified `Waiter::new` initialization process, removing `LedgerSlot` and `Consensus` dependencies. - Updated `wait_for_transaction_level` to use reactive `IndexWatch` for `ClusterCommit`.
- Simplified `Waiter` initialization with ledger seeded indices. - Shared `Waiter` into `Consensus` for unified reactive watches. - Removed unused `replication_poll_ms` and references across configuration, tests, and docs. - Improved cluster commit driver to ensure reactive index propagation.
- Added `Waiter` to `Consensus` for unified reactive index management. - Replaced polling-based mechanisms with Raft-driven heartbeats and reactive propagation. - Removed `replication_poll_ms` references across codebase, tests, and configuration. - Updated replication drivers for more efficient and accurate acknowledgment of index updates.
- Eliminated `replication_poll_ms` references across proto mappers, types, simulator mocks, and UI components. - Reflects full migration to Raft-driven heartbeats and reactive propagation.
- Removed the `testing` crate and its dependencies across the workspace. - Merged `scenario` primitives and `scenarios` catalogue into `control` as lightweight modules. - Updated `control` service, runner, and metrics to reference the new unified structure. - Refactored latency probes to improve sampling clarity (`ProbeSample`) and enhance server-measured latency reporting.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.