Skip to content

Commit ccd9cb5

Browse files
authored
feat: add parallel builder (#219)
## 📝 Summary This PR introduces a new building algorithm which parallelizes the process of building blocks. In particular, it moves the largely sequential process of processing orders, resolving conflicts, and building blocks into modular and parallel process. Each of these components now runs in parallel, allowing for continuous intake of orderflow, flexible resolution of conflicts, and building blocks with the latest flow and best conflict resolution results. It is a significant architecture change. See [run_parallel_builder](https://github.com/flashbots/rbuilder/blob/parallel-builder/crates/rbuilder/src/building/builders/parallel_builder/mod.rs#L180) for a good code entry point and the [readme.md](https://github.com/flashbots/rbuilder/blob/parallel-builder/crates/rbuilder/src/building/builders/parallel_builder/readme.md) for more architecture details. Key Changes: - Introduced a new parallel_builder module, replacing the merging_builder. This adds several components for orderflow intake, conflict identification, conflict resolution, and merging of resolved conflicts. - Added a simulation cache to optimize repeated simulations of similar order sequences. - Updated the builder configuration to use the new Parallel Builder instead of the Merging Builder. - Implemented backtest functionality for the Parallel Builder. ## 💡 Motivation and Context - Concurrency driven efficiency improvements: running significant processes in parallel is more efficient, and should lead to better blocks. Processes aren't sequential, so they aren't blocked on each other anymore. - Reuses previous work: Instead of repeating conflict resolution, we are able to reuse old resolved conflict results. - Better conflict management: We can prioritize "fast" conflict resolution tasks to get results for a conflict group to the building assembler fast, but then also queue "slow" tasks (like evolutionary algorithms) which might produce better results but are slower. ## Testing Finally, I've added a range of tests to the new algorithm. I further have tested it extensively on a local node using mempool txs. - --- ## ✅ I have completed the following steps: * [✅ ] Run `make lint` * [✅ ] Run `make test` * [ ✅] Added tests (if applicable)
1 parent 3159c3f commit ccd9cb5

23 files changed

+3486
-891
lines changed

Cargo.lock

Lines changed: 23 additions & 7 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

config-backtest-example.toml

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,7 @@ failed_order_retries = 1
2929
drop_failed_orders = true
3030

3131
[[builders]]
32-
name = "merging"
33-
algo = "merging-builder"
32+
name = "parallel"
33+
algo = "parallel-builder"
3434
discard_txs = true
35-
num_threads = 5
36-
merge_wait_time_ms = 100
35+
num_threads = 25

config-live-example.toml

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -58,8 +58,7 @@ failed_order_retries = 1
5858
drop_failed_orders = true
5959

6060
[[builders]]
61-
name = "merging"
62-
algo = "merging-builder"
61+
name = "parallel"
62+
algo = "parallel-builder"
6363
discard_txs = true
64-
num_threads = 5
65-
merge_wait_time_ms = 100
64+
num_threads = 25

crates/rbuilder/Cargo.toml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -125,6 +125,9 @@ shellexpand = "3.1.0"
125125
async-trait = "0.1.80"
126126

127127
eth-sparse-mpt = { git = "https://github.com/flashbots/eth-sparse-mpt", rev = "5d0da73" }
128+
crossbeam = "0.8.4"
129+
parking_lot = "0.12.3"
130+
dashmap = "6.1.0"
128131

129132
[build-dependencies]
130133
built = { version = "0.7.1", features = ["git2", "chrono"] }

crates/rbuilder/src/building/builders/merging_builder/combinator.rs

Lines changed: 0 additions & 174 deletions
This file was deleted.

0 commit comments

Comments
 (0)