Skip to content

fix(sequencer): prioritize fee tips on tx ordering - #880

Open
erhant wants to merge 7 commits into
devfrom
erhant/prioritize-fee-tips
Open

erhant wants to merge 7 commits into
devfrom
erhant/prioritize-fee-tips

Conversation

@erhant

@erhant erhant commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

Warning

This PR was rebased onto dev and reworked (ordering is now tip per declared gas, MemPoolHandle::push is removed). The description below is outdated and will be updated soon.

🎯 Purpose

The sequencer drains its mempool FIFO right now. We did introduce tip in fees #801 but an ordering over that was not implemented.

This PR makes the block builder order mempool transactions by tip, highest first, while keeping each signer's nonce sequence intact.

Geth's TransactionsByPriceAndNonce is akin to a heap-based k-way merge. But there, every tx has only one nonce associated with it, unlike ours where there are multiple signers and their nonce's must be ordered with respect as well. For our case, we instead move towards a topological-sort because a signer can depend on multiple signers that come before it, forming a DAG. The ordering of how this DAG should be processed is given by a heap-based Kahn's Algorithm.

Note

A topological-sort where in-degree and out-degree are bounded by 1 is kind of a k-way merge; so we are equivalent to a Geth-like ordering if all our tx'es have 1 associated nonce only.

⚙️ Approach

  • MemPool takes priority and lanes_of at construction and keeps the ordering itself: per-signer lanes in arrival order, plus a BinaryHeap of ready items keyed on (priority, earlier arrival). pop ingests the channel, then returns the best ready item; an item is ready only once it heads every lane it is in. This is Kahn's algorithm kept across pops, so nothing is re-sorted or re-classified per turn. A pop costs $\mathcal{O}(k \cdot (k + \log n))$.

  • push_front gives a deferred tx back ahead of everything, whatever its tip, so it lands in the next block as on dev. Lane order holds because it was ahead of its lane when popped.

  • The sequencer passes the declared tip (0 for exempt classes) and the witness-set signers, private txs included.

  • Bound the pool with one pool-wide atomic len instead of the channel alone: try_push reserves a slot with a single fetch_add and refuses at max_size, so check and admit cannot interleave, and prioritize never touches the count since nothing leaves the pool during a reorder.

  • Store-drained system txs (deposits, dispatches, finalize-unstake, slash) are untouched: they never enter the mempool and still go first.

Warning

Behavior change: exempt classes (private txs, cross-zone locks, sequencer-stake ops) carry tip 0 and now sort behind every tipped public tx, where FIFO let them compete on arrival. Ties are still broken by arrival. Within a lane arrival order always wins, so a tipped public tx cannot jump a same-signer private tx; the pair simply moves at the head's tip. This is expected until private txs are charged.

🧪 How to Test

RISC0_DEV_MODE=1 cargo test -p mempool
RISC0_DEV_MODE=1 cargo test -p sequencer_core --all-features the_builder_orders_by_tip
  • mempool: highest tip pops first with arrival tie-break; a lane is never reordered and a multi-lane item waits on every lane; an item shared by two lanes is enqueued once; try_push counts what prioritize moved out of the channel.
  • sequencer_core: Bob's tipped transfer lands ahead of Alice's untipped head even though he arrived last, and Alice's tipped second transfer still lands behind her first.

🔗 Dependencies

None

🔜 Future Work

  • Over-declared gas closes the block. A misfit still ends the block, and declared gas is free since billing is on used gas. One tx per block declaring the full cap at tip 1 keeps all zero-tip traffic out, and exempt classes cannot tip. dev has the same root flaw in FIFO shape (throttles everyone). Fix is skip-and-keep-filling designed together with eviction, or pricing declared gas. Tracked in a separate issue.

  • Private txs sort at tip 0. Their lanes are handled, so nonce order is safe, but they compete with untipped traffic only. Revisit declared_tip when pptx fees land, nothing else in the ordering changes.

  • Lanes are ordered by arrival, not by the nonce a tx carries, so a nonce n+1 that arrives before nonce n still heads its lane, fails the strict nonce check, and is dropped. This is a pre-existing FIFO behavior; the lanes_of API is the natural place to fix it (return (signer, nonce), sort lanes by nonce).

  • prioritize re-drains, re-classifies, and re-sorts the whole pool every turn. A persistent priority structure inside MemPool (incremental insert, heap-backed pop) is the end state; but this is a rather larger PR and a greater core change. It should also introduce eviction w.r.t tip (e.g. if zero-tips fill the pool, a larger tip should NOT be evicted in favor of those zero-tipped ones). This can go in a different PR since our mempool is not stressed yet.

    • The item above is done in 170215b, but it leaves out tip-based eviction. A full pool still rejects every arrival regardless of tip. Producers only hold a channel sender, so eviction needs a decision on admission semantics (an accepted tx could later be dropped) and must evict from lane tails to avoid nonce gaps.
  • Duplicate witness keys are accepted by the state machine and bump the nonce twice on apply. The lane code handles the duplicate consistently, but the signer's next tx will fail strict equality. Belongs in a state-machine issue. Opened issue fix(lee): reject duplicate witness keys in public and private transactions #893 and fixed by fix(lee)!: reject duplicate witness keys #894

📋 PR Completion Checklist

  • Complete PR description
  • Implement the core functionality
  • Add/update tests
  • Add/update documentation and inline comments

@erhant erhant self-assigned this Sep 10, 2026
@erhant
erhant marked this pull request as draft September 10, 2026 18:42
@erhant
erhant requested a lite review from Copilot September 11, 2026 15:37

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical correctness, concurrency, nonce-ordering, and crash risks remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Updates sequencer and mempool ordering to prioritize transaction fee tips while preserving signer nonce sequencing.

Changes:

  • Adds fee-tip and signer-aware transaction prioritization.
  • Tracks bounded mempool capacity.
  • Integrates ordering logic and adds sequencing tests.
File summaries
File Summary Findings
lez/sequencer/core/src/tests.rs Adds tip-priority and nonce-sequencing tests. No final comments.
lez/sequencer/core/src/lib.rs Applies tip ordering and signer lanes during block production. Critical duplicate-witness panic risk; private transactions omit witness-derived lanes.
lez/mempool/src/lib.rs Implements priority ordering and capacity tracking. Critical capacity race; arrival ordering can discard future-nonce transactions.
Review details

Suppressed comments (2)

lez/mempool/src/lib.rs:108

  • This selection loop rescans every lane for every picked item, and prioritize fully drains and orders the entire pool before the builder can take its first block-limited transactions. With the deployed mempool_max_size of 10,000 and a max_num_tx_in_block of 20, a full backlog of independent senders causes roughly O(n²) work (about 50 million lane-head visits) per turn, in addition to fee classification/serialization. This makes production latency scale with backlog rather than block size; use a ready-head priority queue or otherwise avoid sorting items that cannot fit this block.
        let mut ordered = Vec::with_capacity(items.len());
        loop {
            // candidates: every lane front + the loose items
            let heads = lanes.values().filter_map(|lane| lane.front().copied());
            let Some(best) = heads

lez/mempool/src/lib.rs:169

  • This check is not atomic with try_send: two cloned handles can observe the same held and channel capacity, both pass the comparison, and both enqueue, allowing the front buffer plus channel to exceed max_size. The handle is cloned across the executor/RPC and gossip submission paths, so this race is reachable in production; use an atomic reservation or serialize the capacity check with the send.
        let max_size = self.sender.max_capacity();
        let in_channel = max_size.saturating_sub(self.sender.capacity());
        if self.held.load(Ordering::Relaxed).saturating_add(in_channel) >= max_size {
            return Err(tokio::sync::mpsc::error::TrySendError::Full(item));
  • Files reviewed: 3/3 changed files
  • Comments generated: 4
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread lez/mempool/src/lib.rs Outdated
Comment on lines +88 to +92
// per lane, item indices in arrival order; the front is the only one eligible
let mut lanes: HashMap<G, VecDeque<usize>> = HashMap::new();
// loose items are those in no lane: always eligible
let mut loose: Vec<usize> = Vec::new();
for (arrival, item) in std::iter::from_fn(|| self.pop()).enumerate() {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This precedes the PR, and shall be visited on it's own.

@erhant erhant Oct 1, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note that since we have a multi-signer setting this is slightly more advanced to handle...

consider two tx'es with Alice and Bob as signers and their nonces are as follows:

- tx1: {Alice: 0, Bob: 1, ...}
- tx2: {Alice: 1, Bob: 0, ...}

basically tx1 and tx2 were constructed, and Alice signed one before the other, and Bob with vice-versa.
when these tx'es arrive together, which one do we apply first? that becomes a question once we go with nonce-order instead of arrival order

arrival is linear w.r.t time so we don't have such problems, and we can pop them in that order alone. of course, nonce mismatch will cause tx'es to be dropped, but at least the pool is progressing and nothing stays indefinitely within the pool

but with nonce-ordering, tx1 waits for tx2 and tx2 waits for tx1, indefinitely, they would stay in the pool; the solution there would be to introduce an eviction policy (e.g. tx times out in mempool)

Comment thread lez/mempool/src/lib.rs Resolved
Comment thread lez/sequencer/core/src/lib.rs Resolved
Comment thread lez/sequencer/core/src/lib.rs Resolved
Comment thread lez/mempool/src/lib.rs Resolved
Comment thread lez/mempool/src/lib.rs Resolved
Comment thread lez/mempool/src/lib.rs Outdated
Comment on lines +91 to +95
pub fn len(&self) -> usize {
self.front_buffer.len().saturating_add(self.receiver.len())
self.front
.len()
.saturating_add(self.items.len())
.saturating_add(self.receiver.len())

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shared len counter is what admission checks. Should len() (and is_empty()) just read it?

Currently, the length is recounted with len() (front + items + receiver), so the record_mempool_size metric and the admission check are two separate counts of the same thing. Additionally, nothing checks that these do not diverge.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah indeed, i guess that was where I left off hahah
fixed in ac0deb7

Comment thread lez/mempool/src/lib.rs Outdated
/// Send an item to the mempool blocking if max size is reached.
/// Send an item to the mempool blocking if the channel is full. Bounded by
/// the channel alone, unlike [`Self::try_push`].
pub async fn push(&self, item: T) -> Result<(), tokio::sync::mpsc::error::SendError<T>> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the purpose of having both push and try_push? It appears push is only used in tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indeed! removed in ac0deb7

Comment thread lez/mempool/src/lib.rs Outdated
Comment on lines +243 to +244
/// Send an item to the mempool, failing _immediately_ if it is full: the
/// bound counts what the pool already holds, not just the channel.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't a larger tip transaction successfully be added to the pool? A smaller tip transaction would fall out. This would guarantee that larger tip transactions can be processed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

technically yes, the problem is what to do with fee-exempt tx'es (we have staking and cross-zone locks that enter that category)

  • they are rather important, but fee exempt means tip = 0, so a larger tipped tx should not drop those (maliciously)
  • they are also spammable (except the succesful stake tx, because that actually moves money), so should there be a cap per block on k-many fee exempt tx'es?
    im working on these now, will let you know!

@erhant
erhant force-pushed the erhant/prioritize-fee-tips branch from 3d7a88d to ac0deb7 Compare November 13, 2026 12:49
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Suggested reviewers

These code owners were requested because this pull request touches areas they know best (see CODEOWNERS). Each is asked to focus on the files listed under their name.

Reviewer Why Files
arjentix Sequencer core, Sequencer actors, Test fixtures 16
schouhy Sequencer actors, Sequencer core, Test fixtures 10
Pravdyvy Test fixtures 1

arjentix

Sequencer core (/lez/mempool/, /lez/sequencer/core/, /lez/sequencer/service/)

Sequencer actors (/lez/sequencer/actors/executor/, /lez/sequencer/actors/, /lez/sequencer/actors/rpc_server/)

10 files

Test fixtures (/test_fixtures/)

schouhy

Sequencer actors (/lez/sequencer/actors/executor/)

6 files

Sequencer core (/lez/sequencer/core/, /lez/sequencer/service/)

Test fixtures (/test_fixtures/)

Pravdyvy

Test fixtures (/test_fixtures/)

Files without another code owner (1)

@erhant erhant mentioned this pull request Nov 13, 2026
13 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority:medium Normal priority

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants