Repository navigation
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Critical correctness, concurrency, nonce-ordering, and crash risks remain unresolved.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Updates sequencer and mempool ordering to prioritize transaction fee tips while preserving signer nonce sequencing.
Changes:
- Adds fee-tip and signer-aware transaction prioritization.
- Tracks bounded mempool capacity.
- Integrates ordering logic and adds sequencing tests.
File summaries
| File | Summary | Findings |
|---|---|---|
lez/sequencer/core/src/tests.rs |
Adds tip-priority and nonce-sequencing tests. | No final comments. |
lez/sequencer/core/src/lib.rs |
Applies tip ordering and signer lanes during block production. | Critical duplicate-witness panic risk; private transactions omit witness-derived lanes. |
lez/mempool/src/lib.rs |
Implements priority ordering and capacity tracking. | Critical capacity race; arrival ordering can discard future-nonce transactions. |
Review details
Suppressed comments (2)
lez/mempool/src/lib.rs:108
- This selection loop rescans every lane for every picked item, and
prioritizefully drains and orders the entire pool before the builder can take its first block-limited transactions. With the deployedmempool_max_sizeof 10,000 and amax_num_tx_in_blockof 20, a full backlog of independent senders causes roughly O(n²) work (about 50 million lane-head visits) per turn, in addition to fee classification/serialization. This makes production latency scale with backlog rather than block size; use a ready-head priority queue or otherwise avoid sorting items that cannot fit this block.
let mut ordered = Vec::with_capacity(items.len());
loop {
// candidates: every lane front + the loose items
let heads = lanes.values().filter_map(|lane| lane.front().copied());
let Some(best) = heads
lez/mempool/src/lib.rs:169
- This check is not atomic with
try_send: two cloned handles can observe the sameheldand channel capacity, both pass the comparison, and both enqueue, allowing the front buffer plus channel to exceedmax_size. The handle is cloned across the executor/RPC and gossip submission paths, so this race is reachable in production; use an atomic reservation or serialize the capacity check with the send.
let max_size = self.sender.max_capacity();
let in_channel = max_size.saturating_sub(self.sender.capacity());
if self.held.load(Ordering::Relaxed).saturating_add(in_channel) >= max_size {
return Err(tokio::sync::mpsc::error::TrySendError::Full(item));
- Files reviewed: 3/3 changed files
- Comments generated: 4
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| // per lane, item indices in arrival order; the front is the only one eligible | ||
| let mut lanes: HashMap<G, VecDeque<usize>> = HashMap::new(); | ||
| // loose items are those in no lane: always eligible | ||
| let mut loose: Vec<usize> = Vec::new(); | ||
| for (arrival, item) in std::iter::from_fn(|| self.pop()).enumerate() { |
There was a problem hiding this comment.
This precedes the PR, and shall be visited on it's own.
There was a problem hiding this comment.
note that since we have a multi-signer setting this is slightly more advanced to handle...
consider two tx'es with Alice and Bob as signers and their nonces are as follows:
- tx1: {Alice: 0, Bob: 1, ...}
- tx2: {Alice: 1, Bob: 0, ...}basically tx1 and tx2 were constructed, and Alice signed one before the other, and Bob with vice-versa.
when these tx'es arrive together, which one do we apply first? that becomes a question once we go with nonce-order instead of arrival order
arrival is linear w.r.t time so we don't have such problems, and we can pop them in that order alone. of course, nonce mismatch will cause tx'es to be dropped, but at least the pool is progressing and nothing stays indefinitely within the pool
but with nonce-ordering, tx1 waits for tx2 and tx2 waits for tx1, indefinitely, they would stay in the pool; the solution there would be to introduce an eviction policy (e.g. tx times out in mempool)
88c1fb2 to
690dec3
Compare
170215b to
6dfffe0
Compare
| pub fn len(&self) -> usize { | ||
| self.front_buffer.len().saturating_add(self.receiver.len()) | ||
| self.front | ||
| .len() | ||
| .saturating_add(self.items.len()) | ||
| .saturating_add(self.receiver.len()) |
There was a problem hiding this comment.
The shared len counter is what admission checks. Should len() (and is_empty()) just read it?
Currently, the length is recounted with len() (front + items + receiver), so the record_mempool_size metric and the admission check are two separate counts of the same thing. Additionally, nothing checks that these do not diverge.
There was a problem hiding this comment.
yeah indeed, i guess that was where I left off hahah
fixed in ac0deb7
| /// Send an item to the mempool blocking if max size is reached. | ||
| /// Send an item to the mempool blocking if the channel is full. Bounded by | ||
| /// the channel alone, unlike [`Self::try_push`]. | ||
| pub async fn push(&self, item: T) -> Result<(), tokio::sync::mpsc::error::SendError<T>> { |
There was a problem hiding this comment.
What is the purpose of having both push and try_push? It appears push is only used in tests.
| /// Send an item to the mempool, failing _immediately_ if it is full: the | ||
| /// bound counts what the pool already holds, not just the channel. |
There was a problem hiding this comment.
Shouldn't a larger tip transaction successfully be added to the pool? A smaller tip transaction would fall out. This would guarantee that larger tip transactions can be processed.
There was a problem hiding this comment.
technically yes, the problem is what to do with fee-exempt tx'es (we have staking and cross-zone locks that enter that category)
- they are rather important, but fee exempt means tip = 0, so a larger tipped tx should not drop those (maliciously)
- they are also spammable (except the succesful stake tx, because that actually moves money), so should there be a cap per block on
k-many fee exempt tx'es?
im working on these now, will let you know!
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3d7a88d to
ac0deb7
Compare
Suggested reviewersThese code owners were requested because this pull request touches areas they know best (see
arjentixSequencer core (
Sequencer actors ( 10 files
Test fixtures ( schouhySequencer actors ( 6 filesSequencer core ( Test fixtures ( PravdyvyTest fixtures ( Files without another code owner (1) |
…, cycle check at admission
…e size, refuse bridge deposits at admission
Warning
This PR was rebased onto
devand reworked (ordering is now tip per declared gas,MemPoolHandle::pushis removed). The description below is outdated and will be updated soon.🎯 Purpose
The sequencer drains its mempool FIFO right now. We did introduce
tipin fees #801 but an ordering over that was not implemented.This PR makes the block builder order mempool transactions by tip, highest first, while keeping each signer's nonce sequence intact.
Geth's
TransactionsByPriceAndNonceis akin to a heap-based k-way merge. But there, every tx has only one nonce associated with it, unlike ours where there are multiple signers and their nonce's must be ordered with respect as well. For our case, we instead move towards a topological-sort because a signer can depend on multiple signers that come before it, forming a DAG. The ordering of how this DAG should be processed is given by a heap-based Kahn's Algorithm.Note
A topological-sort where in-degree and out-degree are bounded by 1 is kind of a k-way merge; so we are equivalent to a Geth-like ordering if all our tx'es have 1 associated nonce only.
⚙️ Approach
MemPooltakespriorityandlanes_ofat construction and keeps the ordering itself: per-signer lanes in arrival order, plus aBinaryHeapof ready items keyed on(priority, earlier arrival).popingests the channel, then returns the best ready item; an item is ready only once it heads every lane it is in. This is Kahn's algorithm kept across pops, so nothing is re-sorted or re-classified per turn. A pop costspush_frontgives a deferred tx back ahead of everything, whatever its tip, so it lands in the next block as ondev. Lane order holds because it was ahead of its lane when popped.The sequencer passes the declared tip (0 for exempt classes) and the witness-set signers, private txs included.
Bound the pool with one pool-wide atomic
leninstead of the channel alone:try_pushreserves a slot with a singlefetch_addand refuses atmax_size, so check and admit cannot interleave, andprioritizenever touches the count since nothing leaves the pool during a reorder.Store-drained system txs (deposits, dispatches, finalize-unstake, slash) are untouched: they never enter the mempool and still go first.
Warning
Behavior change: exempt classes (private txs, cross-zone locks, sequencer-stake ops) carry tip 0 and now sort behind every tipped public tx, where FIFO let them compete on arrival. Ties are still broken by arrival. Within a lane arrival order always wins, so a tipped public tx cannot jump a same-signer private tx; the pair simply moves at the head's tip. This is expected until private txs are charged.
🧪 How to Test
mempool: highest tip pops first with arrival tie-break; a lane is never reordered and a multi-lane item waits on every lane; an item shared by two lanes is enqueued once;try_pushcounts whatprioritizemoved out of the channel.sequencer_core: Bob's tipped transfer lands ahead of Alice's untipped head even though he arrived last, and Alice's tipped second transfer still lands behind her first.🔗 Dependencies
None
🔜 Future Work
Over-declared gas closes the block. A misfit still ends the block, and declared gas is free since billing is on used gas. One tx per block declaring the full cap at tip 1 keeps all zero-tip traffic out, and exempt classes cannot tip.
devhas the same root flaw in FIFO shape (throttles everyone). Fix is skip-and-keep-filling designed together with eviction, or pricing declared gas. Tracked in a separate issue.Private txs sort at tip 0. Their lanes are handled, so nonce order is safe, but they compete with untipped traffic only. Revisit
declared_tipwhen pptx fees land, nothing else in the ordering changes.Lanes are ordered by arrival, not by the nonce a tx carries, so a nonce
n+1that arrives before noncenstill heads its lane, fails the strict nonce check, and is dropped. This is a pre-existing FIFO behavior; thelanes_ofAPI is the natural place to fix it (return(signer, nonce), sort lanes by nonce).prioritizere-drains, re-classifies, and re-sorts the whole pool every turn. A persistent priority structure insideMemPool(incremental insert, heap-backedpop) is the end state; but this is a rather larger PR and a greater core change. It should also introduce eviction w.r.t tip (e.g. if zero-tips fill the pool, a larger tip should NOT be evicted in favor of those zero-tipped ones). This can go in a different PR since our mempool is not stressed yet.Duplicate witness keys are accepted by the state machine and bump the nonce twice on apply. The lane code handles the duplicate consistently, but the signer's next tx will fail strict equality. Belongs in a state-machine issue.Opened issue fix(lee): reject duplicate witness keys in public and private transactions #893 and fixed by fix(lee)!: reject duplicate witness keys #894📋 PR Completion Checklist