Skip to content

Parallel block import #1662

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 67 commits into
base: develop
Choose a base branch
from
Open

Parallel block import #1662

wants to merge 67 commits into from
+1,235 −467

Conversation

@Raneet10
Copy link
Member

@Raneet10 Raneet10 commented Jul 25, 2025 •
edited
Loading

Description

This PR adds the feature to allow a node to import blocks in parallel in stateless mode.

goos: darwin
goarch: arm64
pkg: github.com/ethereum/go-ethereum/core
cpu: Apple M2 Pro
                                  │ bench/stateless_seq_100.txt │ bench/stateless_par_100.txt │
                                  │           sec/op            │        sec/op         vs base   │
InsertChainStatelessSequential-10                   56.51µ ± 0%
InsertChainStatelessParallel-10                                            42.52µ ± 1%
geomean                                             56.51µ                 42.52µ       ? ¹ ²
¹ benchmark set differs from baseline; geomeans may not be comparable
² ratios must be >0 to compute geomean

Parallel import currently is ~25% faster.

Changes

  • Bugfix (non-breaking change that solves an issue)
  • Hotfix (change that solves an urgent issue, and requires immediate attention)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (change that is not backwards-compatible and/or changes current functionality)
  • Changes only for a subset of nodes

Checklist

  • I have added at least 2 reviewer or the whole pos-v1 team
  • I have added sufficient documentation in code
  • I will be resolving comments - if any - by pushing each fix in a separate commit and linking the commit hash in the comment reply
  • Created a task in Jira and informed the team for implementation in Erigon client (if applicable)
  • Includes RPC methods changes, and the Notion documentation has been updated

Testing

  • I have added unit tests
  • I have added tests to CI
  • I have tested this code manually on local environment
  • I have tested this code manually on remote devnet using express-cli
  • I have tested this code manually on amoy
  • I have created new e2e tests into express-cli

@Raneet10 Raneet10 requested a review from cffls August 22, 2025 12:55
@cffls
Copy link
Contributor

cffls commented Aug 22, 2025

From the run log, it looks like one issue is that the node won't be able to process a block/header unless its parent is known.

Screenshot 2025-08-22 at 10 54 45 AM

@Raneet10 Raneet10 changed the base branch from stateless_sync to develop August 26, 2025 10:18
@Raneet10 Raneet10 force-pushed the raneet10/parallel-block-import branch from ba47b51 to 7ad379c Compare August 28, 2025 11:40
@Raneet10 Raneet10 marked this pull request as ready for review August 29, 2025 11:42
cffls
cffls reviewed Aug 29, 2025
View reviewed changes
core/blockchain.go Outdated
for i, block := range chain {
// Check if block is already known before attempting to process
wg.Add(1)
go func(block *types.Block, i int) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a very big function. Could you move it to a separate function and potentially break it down?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

core/blockchain.go Outdated
var processed atomic.Int32

bs := NewBlockState() // Block state tracker to manage parallel block processing
bc.parallelImport.Store(true)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a config to enable/disable parallel block processing.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

core/blockchain_test.go Outdated
}
*/
factoryBIN := common.Hex2Bytes("608060405234801561001057600080fd5b50610241806100206000396000f3fe608060405234801561001057600080fd5b506004361061002a5760003560e01c80627743601461002f575b600080fd5b610049600480360381019061004491906100d8565b61004b565b005b6000808251602084016000f59050803b61006457600080fd5b5050565b600061007b61007684610146565b610121565b905082815260208101848484011115610097576100966101eb565b5b6100a2848285610177565b509392505050565b600082601f8301126100bf576100be6101e6565b5b81356100cf848260208601610068565b91505092915050565b6000602082840312156100ee576100ed6101f5565b5b600082013567ffffffffffffffff81111561010c5761010b6101f0565b5b610118848285016100aa565b91505092915050565b600061012b61013c565b90506101378282610186565b919050565b6000604051905090565b600067ffffffffffffffff821115610161576101606101b7565b5b61016a826101fa565b9050602081019050919050565b82818337600083830152505050565b61018f826101fa565b810181811067ffffffffffffffff821117156101ae576101ad6101b7565b5b80604052505050565b7f4e487b7100000000000000000000000000000000000000000000000000000000600052604160045260246000fd5b600080fd5b600080fd5b600080fd5b600080fd5b6000601f19601f830116905091905056fea2646970667358221220ea8b35ed310d03b6b3deef166941140b4d9e90ea2c92f6b41eb441daf49a59c364736f6c63430008070033")
factoryBIN := common.Hex2Bytes("608060405234801561001057600080fd5b50610241806100206000396000f3fe608060405234801561001057600080fd5b506004361061002a5760003560e01c80627743601461002f575b600080fd5b610049600480360381019061004491906100d8565b61004b565b005b6000808251602084016000f59050803b61006457600080fd5b5050565b600061007b61007684610146565b610121565b905082815260208101848484011115610097576100966101eb565b5b6100a2848285610177565b509392505050565b600082601f8301126100bf576100be6101e6565b5b81356100cf848260208601610068565b91505092915050565b6000602082840312156100ee576100ed6101f5565b5b600082013567ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff191690815f1a90535050565b6000604051905090565b600067ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1115610161576101606101b7565b5b61016a826101fa565b9050602081019050919050565b82818337600083830152505050565b61018f826101fa565b810181811067ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff17156101ae576101ad6101b7565b5b80604052505050565b7f4e487b7100000000000000000000000000000000000000000000000000000000600052604160045260246000fd5b600080fd5b600080fd5b600080fd5b600080fd5b6000601f19601f830116905091905056fea2646970667358221220ea8b35ed310d03b6b3deef166941140b4d9e90ea2c92f6b41eb441daf49a59c364736f6c63430008070033")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it required to change this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ouch. Yeah reverted this.

@Raneet10 Raneet10 force-pushed the raneet10/parallel-block-import branch from 73f3270 to 02e53a9 Compare September 4, 2025 12:27
@pratikspatil024 pratikspatil024 changed the base branch from develop to v2.3.0-candidate September 8, 2025 07:09
@Raneet10 Raneet10 force-pushed the raneet10/parallel-block-import branch from 60e727d to cf63cc3 Compare October 13, 2025 12:16
Raneet10 added 17 commits October 13, 2025 21:20
@Raneet10
core: enable retry for first block in batch
54541b3
@Raneet10
Revert "core: add temp log when committing code"
3bd7172 
This reverts commit cf63cc3.
@Raneet10
core: cap workers spun
96367ee
@Raneet10
Revert "core: cap workers spun"
a1d857a 
This reverts commit 96367ee.
@Raneet10
core: ensure statedb cleanup
675bc24
@Raneet10
core: add temp debug logs
30a3179
@Raneet10
core: add temp debug logs
2c4f840
@Raneet10
Merge branch 'raneet10/parallel-block-import' of github.com:maticnetw…
3483398 
…ork/bor into raneet10/parallel-block-import
@Raneet10
core: add temp log to track sidechain
29b3366
@Raneet10
core: add temp log for contract code in commitAndFlush
174765f
@Raneet10
core: add retry for all processing and internal state db errors
3c1bfec
@Raneet10
core: add state db error validation during deferred exec
21443d9
@Raneet10
core: add TestParallelStateless_ContractDeployedThenCalled
f09e2eb
@Raneet10
Revert "core: add temp log for contract code in commitAndFlush"
390fea0 
This reverts commit 174765f.
@Raneet10
Revert "core: add temp log to track sidechain"
cd7f338 
This reverts commit 29b3366.
@Raneet10
Revert "core: add temp debug logs"
dbe1292 
This reverts commit 2c4f840.
@Raneet10
internal/cli: turn off parallel import by default
489c505
@sonarqubecloud
Copy link

Quality Gate Passed Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.3% Duplication on New Code

See analysis details on SonarQube Cloud

cffls
cffls reviewed Oct 22, 2025
View reviewed changes
internal/cli/server/config.go
ProduceWitnesses bool `hcl:"producewitnesses,optional" toml:"producewitnesses,optional"`

// Parallel stateless import (download path) toggle
EnableParallelStatelessImport bool `hcl:"parallelstatelessimport,optional" toml:"parallelstatelessimport,optional"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we put - between words for this flag?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pratikspatil024 pratikspatil024 pratikspatil024 left review comments

@cffls cffls cffls left review comments

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Raneet10 @cffls @pratikspatil024