Code

Bonsai archive feature#7475

Code

Open

matthew1001 wants to merge 64 commits intohyperledger:mainhyperledger/besu:mainfrom matthew1001:multi-version-flat-db-rebasematthew1001/besu:multi-version-flat-db-rebaseCopy head branch name to clipboard

+3,404-203

Commits

Commits on Oct 8, 2024

• rebase of bonsai archive

  Show description for a5315af
  

  jframe

  authored and

  matthew1001

  committed
  a5315af

  Copy full SHA for a5315af

• Change BonsaiReferenceTestWorldStateStorage getContextSafeCopy() to use constructor that reuses worldStateStorage so that we don't lose values in the EvmToolSpecTests

  Show description for ec8a7ff
  

  jframe

  authored and

  matthew1001

  committed
  ec8a7ff

  Copy full SHA for ec8a7ff

• Force archive mode for testing

  Show description for b2d9196
  

  jframe

  authored and

  matthew1001

  committed
  b2d9196

  Copy full SHA for b2d9196

• Rebase on main

  Show description for fd300b0
  matthew1001

  committed
  fd300b0

  Copy full SHA for fd300b0

• Create new DiffBasedWorldStateConfig when copying bonsai archive world state, and freeze it

  Show description for b10b6ae
  matthew1001

  committed
  b10b6ae

  Copy full SHA for b10b6ae

• Add BONSAI_ARCHIVE storage format

  Show description for 6e786f3
  matthew1001

  committed
  6e786f3

  Copy full SHA for 6e786f3

• New ACCOUNT_FREEZER_STATE DB segment. New BonsaiArchiveFreezer to listen for blocks and move account state to new DB segment

  Show description for 32771d3
  matthew1001

  committed
  32771d3

  Copy full SHA for 32771d3

• Add freezer segment for account storage

  Show description for b1f448d
  matthew1001

  committed
  b1f448d

  Copy full SHA for b1f448d

• Find frozen storage slots correctly, refactor the DB segment names

  Show description for 06ba76e
  matthew1001

  committed
  06ba76e

  Copy full SHA for 06ba76e

• Check for blocks to freeze state for on startup. Store the most recent block state has been frozen for

  Show description for 1d930f5
  matthew1001

  committed
  1d930f5

  Copy full SHA for 1d930f5

• Make sure genesis world state is created in archive mode

  Show description for 605d590
  matthew1001

  committed
  605d590

  Copy full SHA for 605d590

• Fix incorrect logs

  Show description for c7c2ee5
  matthew1001

  committed
  c7c2ee5

  Copy full SHA for c7c2ee5

• Ensure deleted storage is returned from live DB segment, not old storage from the freezer segment

  Show description for 990bd87
  matthew1001

  committed
  990bd87

  Copy full SHA for 990bd87

• Adding tests

  Show description for 8518428
  matthew1001

  committed
  8518428

  Copy full SHA for 8518428

• Refactor DB TX with retries

  Show description for a1cb77d
  matthew1001

  committed
  a1cb77d

  Copy full SHA for a1cb77d

• Honour code hash or account hash config for storing code. Add tests for DB mode

  Show description for 660d449
  matthew1001

  committed
  660d449

  Copy full SHA for 660d449

• Refactor code to move one block's worth of state/storage changes at a time. Add more tests

  Show description for 4063128
  matthew1001

  committed
  4063128

  Copy full SHA for 4063128

• Refactoring, fix issues, ensure only 1 batch of blocks is archived at a time

  Show description for 829a98e
  matthew1001

  committed
  829a98e

  Copy full SHA for 829a98e

• Improve archive progress logs

  Show description for faa8da1
  matthew1001

  committed
  faa8da1

  Copy full SHA for faa8da1

• Add archive mode tests to bonsai key/value tests

  Show description for 0543017
  matthew1001

  committed
  0543017

  Copy full SHA for 0543017

• Implement snap-serving from a Bonsai Archive node and add tests

  Show description for bf3d171
  matthew1001

  committed
  bf3d171

  Copy full SHA for bf3d171

• Parameterize snap-server tests, fix bug in storage slot ranges

  Show description for d2e5a24
  matthew1001

  committed
  d2e5a24

  Copy full SHA for d2e5a24

• Remove todo comments

  Show description for ee7c2fc
  matthew1001

  committed
  ee7c2fc

  Copy full SHA for ee7c2fc

• Make BONSAI_ARCHIVE experimental for the first release. Add metrics. Use the term archive, not freezer

  Show description for 38cfdd9
  matthew1001

  committed
  38cfdd9

  Copy full SHA for 38cfdd9

• Add changelog entry

  Show description for 35d60dd
  matthew1001

  committed
  35d60dd

  Copy full SHA for 35d60dd

• Don't store every block that is waiting to be archived

  Show description for b8facef
  matthew1001

  committed
  b8facef

  Copy full SHA for b8facef

• Remove pre-startup catchup. Start on first block import instead

  Show description for 75e57cc
  matthew1001

  committed
  75e57cc

  Copy full SHA for 75e57cc

• Multiple deletes of the same address in a single block causes Bonsai to fail the block

  Show description for a6552df
  matthew1001

  committed
  a6552df

  Copy full SHA for a6552df

Commits on Oct 9, 2024

• Metric should be a gauge not a counter

  Show description for 329da00
  matthew1001

  committed
  329da00

  Copy full SHA for 329da00

• Refactor archive tests after code refactor

  Show description for e6364b2
  matthew1001

  committed
  e6364b2

  Copy full SHA for e6364b2

• Merge branch 'main' into multi-version-flat-db-rebase

  matthew1001

  committed
  8824b21

  Copy full SHA for 8824b21

• Remove commented out sections from tests

  Show description for 46a33ef
  matthew1001

  committed
  46a33ef

  Copy full SHA for 46a33ef

Commits on Oct 17, 2024

• Update the BFT soak test to include a Bonsai archive node

  Show description for 4051eb0
  matthew1001

  committed
  4051eb0

  Copy full SHA for 4051eb0

Commits on Oct 22, 2024

• Add archive-specific checks to the BFT soak test

  Show description for c28a383
  matthew1001

  committed
  c28a383

  Copy full SHA for c28a383

Commits on Nov 4, 2024

• Merge with main

  Show description for c1f239f
  matthew1001

  committed
  c1f239f

  Copy full SHA for c1f239f

• Update plugin API hash

  Show description for c226aa6
  matthew1001

  committed
  c226aa6

  Copy full SHA for c226aa6

Commits on Nov 5, 2024

• Add javadoc

  Show description for a6123bb
  matthew1001

  committed
  a6123bb

  Copy full SHA for a6123bb

• Merge fix

  Show description for c4a9516
  matthew1001

  committed
  c4a9516

  Copy full SHA for c4a9516

• Unit test fix

  Show description for 81a0907
  matthew1001

  committed
  81a0907

  Copy full SHA for 81a0907

• Merge branch 'main' into multi-version-flat-db-rebase

  matthew1001

  authored
  0036e61

  Copy full SHA for 0036e61

• More UT fixing

  Show description for 0b0bed4
  matthew1001

  committed
  0b0bed4

  Copy full SHA for 0b0bed4

Commits on Nov 6, 2024

• Tidy up

  Show description for 614e5c2
  matthew1001

  committed
  614e5c2

  Copy full SHA for 614e5c2

• Merge branch 'main' into multi-version-flat-db-rebase

  matthew1001

  authored
  5392ef4

  Copy full SHA for 5392ef4

Commits on Jan 9, 2025

• Update ethereum/core/src/main/java/org/hyperledger/besu/ethereum/trie/diffbased/bonsai/worldview/BonsaiArchiver.java

  Show description for 763e810
  

  matthew1001

  and

  macfarla

  authored
  763e810

  Copy full SHA for 763e810

• Update plugins/rocksdb/src/main/java/org/hyperledger/besu/plugin/services/storage/rocksdb/configuration/BaseVersionedStorageFormat.java

  Show description for 8caf063
  

  matthew1001

  and

  macfarla

  authored
  8caf063

  Copy full SHA for 8caf063

• Update plugins/rocksdb/src/main/java/org/hyperledger/besu/plugin/services/storage/rocksdb/configuration/BaseVersionedStorageFormat.java

  Show description for 4065698
  

  matthew1001

  and

  macfarla

  authored
  4065698

  Copy full SHA for 4065698

• Remove debug file

  Show description for 602e18e
  matthew1001

  committed
  602e18e

  Copy full SHA for 602e18e

Commits on Jan 14, 2025

• Various refactoring to manage archive flat DB access more cleanly

  Show description for f298255
  matthew1001

  committed
  f298255

  Copy full SHA for f298255

• Merge branch 'main' into multi-version-flat-db-rebase

  Show description for f0a4ce1
  matthew1001

  authored
  f0a4ce1

  Copy full SHA for f0a4ce1

• Merge fix

  Show description for 88bd33e
  matthew1001

  committed
  88bd33e

  Copy full SHA for 88bd33e

• Merge tests

  Show description for 2537ce2
  matthew1001

  committed
  2537ce2

  Copy full SHA for 2537ce2

• Use alternative clearStorage()

  Show description for f2954f5
  matthew1001

  committed
  f2954f5

  Copy full SHA for f2954f5

• Merge branch 'main' into multi-version-flat-db-rebase

  matthew1001

  authored
  c002f4b

  Copy full SHA for c002f4b

• Fix metric name

  Show description for b65fdaa
  matthew1001

  committed
  b65fdaa

  Copy full SHA for b65fdaa

• Don't override rollMutableStateToBlockHash, create archive-specific function name

  Show description for b5d3c1a
  matthew1001

  committed
  b5d3c1a

  Copy full SHA for b5d3c1a

Commits on Jan 16, 2025

• Update world state with block number being persisted

  Show description for 3873f8e
  matthew1001

  committed
  3873f8e

  Copy full SHA for 3873f8e

• Merge branch 'main' into multi-version-flat-db-rebase

  matthew1001

  authored
  63db389

  Copy full SHA for 63db389

• Update changelog

  Show description for 25c5bc6
  matthew1001

  committed
  25c5bc6

  Copy full SHA for 25c5bc6

Commits on Jan 20, 2025

• Move context management out of flat DB strategy

  Show description for 633fad4
  matthew1001

  committed
  633fad4

  Copy full SHA for 633fad4

Commits on Jan 21, 2025

• Add more test cases, tidy up code

  Show description for 3d8d8c7
  matthew1001

  committed
  3d8d8c7

  Copy full SHA for 3d8d8c7

• Merge branch 'main' into multi-version-flat-db-rebase

  Show description for 24fa310
  matthew1001

  authored
  24fa310

  Copy full SHA for 24fa310

• Update plugin hash

  Show description for 8f98f28
  matthew1001

  committed
  8f98f28

  Copy full SHA for 8f98f28

Commits on Jan 23, 2025

• Address review comments

  Show description for b832ea8
  matthew1001

  committed
  b832ea8

  Copy full SHA for b832ea8

• Merge branch 'main' into multi-version-flat-db-rebase

  matthew1001

  authored
  0b8c9c8

  Copy full SHA for 0b8c9c8