Skip to content

Commit 201f0a9

Browse files
committed
Verkle tree-based state with overlay transition
Squash the main verkle PR ahead of rebase don't call Bytes() in GetTreeKey (#137) trie: avoid endianness conversion in GetTreeKey (#140) * trie/utils: add concrete expected value in trie key generation test Signed-off-by: Ignacio Hagopian <[email protected]> * mod: update to latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: avoid endianness conversions Signed-off-by: Ignacio Hagopian <[email protected]> * apply review changes & update to official go-verkle version Signed-off-by: Ignacio Hagopian <[email protected]> Signed-off-by: Ignacio Hagopian <[email protected]> upgrade go-verkle to CoW version and get TestProcessVerkle to build (#138) updating ci to use self-hosted machine (#143) fix: storage offset in non-header group + reuse of value buffer (#145) dedup call to ChunkifyCode, same as replay branch (#156) * dedup call to ChunkifyCode, same as replay branch * fix some linter issues fix code offset in tree update (#157) fix REVERT in state processor test execution (#158) * fix code offset in tree update * fix REVERT in test execution save on key hashing: lump code size update with first code chunk group (#159) fix code chunk key calculation and storage key calculation (#161) * fix codeKey calculation * Remove * fix storageOffset * fix the fix to the fix to the offset fix * Remove copy/pasted, unused code in test * fix linter --------- Co-authored-by: Guillaume Ballet <[email protected]> fix: infinite loop when calling extcodecopy on empty code (#151) upgrade to latest go-verkle fix: only update code in the tree if it's dirty (#174) fix: read-touch the code size and Keccak of the origin (#175) List of changes for converting a sepolia database (#182) * naive conversion rebased on top of beverly hills * changes for the sepolia shadow fork conversion * fixes to please the linter * fixes to please the linter Unified point cache (#180) * Unified point cache * Use cache for Try*Account * alter Trie interface to use caching for slots (#181) * alter Trie interface to use caching for slots * fix: use a lock to protect the point cache (#185) * use fastest non-master go-verkle version & pull trie/Verkle.go changes to use new api (#184) * mod: update to fastest go-verkle version today Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: use new batch serialization api Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> * fix: TryDelete signature in unit tests --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> trie/utils: fix potential overflow (#191) * trie/utils: fix potential overflow Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: receive storage key as a byte slice Signed-off-by: Ignacio Hagopian <[email protected]> * revert formatter changes Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: fix mod 256 Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> trie/utils: fix incorrect bigint assignment (#193) Signed-off-by: Ignacio Hagopian <[email protected]> upgrade precomp link to fix CI fix: add missing code size&keccak leaves in empty accounts (#192) fixes to use the latest go-verkle@master (#197) * fixes to use the latest go-verkle@master * linter fixes * linter fixes for tests * fix: use jsign's go-verkle fix refactor: remove unused (*StateDB).GetXLittleEndian methods (#204) fix gas accounting issue in state_processor_test.go (#207) update go-verkle not to use StatelessNode anymore (#206) * update go-verkle not to use StatelessNode anymore * update go-verkle to latest refactor: move verkle gas accounting to its own block in TransitionDB (#208) fix a panic in deserializeVerkleProof if GetProofItems returns a nil ProofElements use the cachingDB instead of a custom VerkleDB (#209) * use the cachingDB instead of a custom VerkleDB * fix stack trace in LES remove holiman from CODEOWNERS as he gets too many emails read from tree in state object if the snapshot is nil (#205) add missing error checks for the root node type (#214) implement OpenStorageTrie for verkle trees (#210) * implement OpenStorageTrie for verkle trees * add a few comments for future maintenance * fix linter issue fix: copy balance leaf to new buffer in TryGetAccount (#217) implement some heretofore unimplemented iterator methods (#219) params: move verkle params to their own file (#228) fix: proper number of chunk evals (#215) overlay transition (#244) * overlay transition Fix some bugs identified in the code review Co-authored-by: Ignacio Hagopian <[email protected]> Include base -> overlay key-values migration logic (#199) * mod: add go-verkle version with key-value migration new apis Signed-off-by: Ignacio Hagopian <[email protected]> * core/stateprocessor: use constant for max number of migrated key-values Signed-off-by: Ignacio Hagopian <[email protected]> * core: add base->overlay key-values migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * core: fix some compiler errors Signed-off-by: Ignacio Hagopian <[email protected]> * trie: consider removing transition trie api in the future Signed-off-by: Ignacio Hagopian <[email protected]> * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> fix some unit tests errors get convresion block from file fix compilation issues fix initialization issue in migrator fix: changes needed to run the first 28 blocks important sutff: fix the banner fix: use nonce instead of balance in nonce leaf (#202) fixes for performing the overlay transition (#203) * fixes for performing the overlay transition * fixes for the full replay * fix: deletion-and-recreation of EoA * fixes to replay 2M+ blocks * upgrade to go-verkle@master * fix: proper number of chunk evals * rewrite conversion loop to fix known issues changes to make replay work with the overlay method (#216) * fixes for performing the overlay transition fixes for the full replay fix: deletion-and-recreation of EoA fixes to replay 2M+ blocks upgrade to go-verkle@master fix: proper number of chunk evals rewrite conversion loop to fix known issues changes to make replay work with the overlay method fixes to replay 2M+ blocks update to latest go-verkle@master * use a PBSS-like scheme for internal nodes (#221) * use a PBSS-like scheme for internal nodes * a couple of fixes coming from debugging replay * fix: use an error to notify the transition tree that a deleted account was found in the overlay tree (#222) * fixes for pbss replay (#227) * fixes for pbss replay * trie/verkle: use capped batch size (#229) * trie/verkle: use capped batch size Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: avoid path variable allocation per db.Put Signed-off-by: Ignacio Hagopian <[email protected]> * don't keep more than 32 state root conversions in RAM (#230) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Guillaume Ballet <[email protected]> * cleanup some code * mod: update go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> * re-enable snapshot (#231) * re-enable cancun block / snapshot (#226) * clear storage conversion key upon translating account (#234) * clear storage conversion key upon translating account * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> * fix: self-deadlock with translated root map mutex (#236) * return compressed commitment as root commitment (#237) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> fix first panic in *TransitionTrie.Copy() upgrade go-verkle to latest master mod: update go-verkle (#239) Signed-off-by: Ignacio Hagopian <[email protected]> core: print state root every 100 blocks (#240) Signed-off-by: Ignacio Hagopian <[email protected]> fix: only Commit the account trie (#242) fixes to get TestProcessVerkle to work with the overlay branch (#238) * fixes to get TestProcessVerkle to work with the overlay branch * fix all panics in verkle state processor test * fix proof verification move transition management to cachingDB * fix: mark the verkle transition as started if it's ended without being started * fix the verkle state processing test * fix linter errors * Add a function to clear verkle params for replay * fix: handle TransitionTrie in OpenStorageTrie * fix linter issue * fix the deleted account error (#247) * code cleanup (#248) * fix: don't error on a missing conversion.txt (#249) * Overlay Tree preimages exporting and usage (#246) * export overlay preimages tool Signed-off-by: Ignacio Hagopian <[email protected]> * use preimages flat file in overlay tree migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * cmd/geth: add --roothash to overlay tree preimage exporting command Signed-off-by: Ignacio Hagopian <[email protected]> * cleanup Signed-off-by: Ignacio Hagopian <[email protected]> * review feedback Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> * fix: reduce the PR footprint (#250) * fix: don't fail when preimages.bin is missing (#251) * fix: don't fail when preimages.bin is missing * fix: don't open the preimages file when outside of transition --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> review changes remove replay-specific code
1 parent 7de748d commit 201f0a9

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

58 files changed

+4338
-56
lines changed

.github/CODEOWNERS

+1-1
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ accounts/scwallet @gballet
66
accounts/abi @gballet @MariusVanDerWijden
77
cmd/clef @holiman
88
consensus @karalabe
9-
core/ @karalabe @holiman @rjl493456442
9+
core/ @karalabe @rjl493456442
1010
eth/ @karalabe @holiman @rjl493456442
1111
eth/catalyst/ @gballet
1212
eth/tracers/ @s1na

.github/workflows/go.yml

+48
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
name: Go lint and test
2+
3+
on:
4+
push:
5+
branches: [ master ]
6+
pull_request:
7+
branches: [ master, verkle-trie-proof-in-block-rebased, verkle-trie-post-merge, beverly-hills-head, 'verkle/replay-change-with-tree-group-tryupdate' ]
8+
workflow_dispatch:
9+
10+
jobs:
11+
build:
12+
runs-on: self-hosted
13+
steps:
14+
- uses: actions/checkout@v2
15+
- name: Set up Go
16+
uses: actions/setup-go@v2
17+
with:
18+
go-version: 1.18
19+
- name: Build
20+
run: go build -v ./...
21+
22+
lint:
23+
runs-on: self-hosted
24+
steps:
25+
- uses: actions/checkout@v2
26+
- name: Set up Go
27+
uses: actions/setup-go@v2
28+
with:
29+
go-version: 1.18
30+
- name: Download golangci-lint
31+
run: wget -O- -nv https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s latest
32+
- name: Lint
33+
run: ./bin/golangci-lint run
34+
- name: Vet
35+
run: go vet
36+
37+
test:
38+
runs-on: self-hosted
39+
steps:
40+
- uses: actions/checkout@v2
41+
- name: Set up Go
42+
uses: actions/setup-go@v2
43+
with:
44+
go-version: 1.18
45+
- name: Download precomputed points
46+
run: wget -nv https://github.com/gballet/go-verkle/releases/download/banderwagonv3/precomp -Otrie/utils/precomp
47+
- name: Test
48+
run: go test ./...

cmd/geth/chaincmd.go

+38
Original file line numberDiff line numberDiff line change
@@ -144,6 +144,17 @@ It's deprecated, please use "geth db import" instead.
144144
Description: `
145145
The export-preimages command exports hash preimages to an RLP encoded stream.
146146
It's deprecated, please use "geth db export" instead.
147+
`,
148+
}
149+
exportOverlayPreimagesCommand = &cli.Command{
150+
Action: exportOverlayPreimages,
151+
Name: "export-overlay-preimages",
152+
Usage: "Export the preimage in overlay tree migration order",
153+
ArgsUsage: "<dumpfile>",
154+
Flags: flags.Merge([]cli.Flag{utils.TreeRootFlag}, utils.DatabasePathFlags),
155+
Description: `
156+
The export-overlay-preimages command exports hash preimages to a flat file, in exactly
157+
the expected order for the overlay tree migration.
147158
`,
148159
}
149160
dumpCommand = &cli.Command{
@@ -399,6 +410,33 @@ func exportPreimages(ctx *cli.Context) error {
399410
return nil
400411
}
401412

413+
// exportOverlayPreimages dumps the preimage data to a flat file.
414+
func exportOverlayPreimages(ctx *cli.Context) error {
415+
if ctx.Args().Len() < 1 {
416+
utils.Fatalf("This command requires an argument.")
417+
}
418+
stack, _ := makeConfigNode(ctx)
419+
defer stack.Close()
420+
421+
chain, _ := utils.MakeChain(ctx, stack, true)
422+
423+
var root common.Hash
424+
if ctx.String(utils.TreeRootFlag.Name) != "" {
425+
rootBytes := common.FromHex(ctx.String(utils.StartKeyFlag.Name))
426+
if len(rootBytes) != common.HashLength {
427+
return fmt.Errorf("invalid root hash length")
428+
}
429+
root = common.BytesToHash(rootBytes)
430+
}
431+
432+
start := time.Now()
433+
if err := utils.ExportOverlayPreimages(chain, ctx.Args().First(), root); err != nil {
434+
utils.Fatalf("Export error: %v\n", err)
435+
}
436+
fmt.Printf("Export done in %v\n", time.Since(start))
437+
return nil
438+
}
439+
402440
func parseDumpConfig(ctx *cli.Context, stack *node.Node) (*state.DumpConfig, ethdb.Database, common.Hash, error) {
403441
db := utils.MakeChainDatabase(ctx, stack, true)
404442
var header *types.Header

cmd/geth/main.go

+1
Original file line numberDiff line numberDiff line change
@@ -209,6 +209,7 @@ func init() {
209209
exportCommand,
210210
importPreimagesCommand,
211211
exportPreimagesCommand,
212+
exportOverlayPreimagesCommand,
212213
removedbCommand,
213214
dumpCommand,
214215
dumpGenesisCommand,

0 commit comments

Comments
 (0)