Deploy new nimbus-eth1 DB snapshot

[jakubgs]

We have received a new nimbus-eth1 database snapshot from @arnetheduck :

jakubgs@metal-01.ih-eu-mda1.nimbus.eth1:/nimbus-eth1-db % du -hs mainnet-static-vid-keyed 
109G    mainnet-static-vid-keyed

Which needs to be copied to all Mainnet nimbus-eth1 nodes in order for them to work after this PR is merged:

• Path-based VertexID nimbus-eth1#3251

Because once it is the old DB format will not work and nodes will crash at startup.

[arnetheduck]

Also need to update template db for https://github.com/status-im/nimbus-eth1-benchmarks, use same number

[jakubgs]

This PR also should be merged before we start the nodes up again with new DB:

• drop logsBloom support nimbus-eth1#3292

[jakubgs]

I've copied the mainnet-static-vid-keyed.tar.gz database to all mainnet nodes and updated them. And one of them is doing well while other is not syncing:

jakubgs@nec-01.ih-eu-mda1.nimbus.mainnet:/docker % for script in /docker/nimbus-eth1-mainnet-*/rpc.sh; do $script eth_syncing | jq -c; done
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":{"startingBlock":"0x157e135","currentBlock":"0x15780f6","highestBlock":"0x158aa14"}}

But there's issues with stable beacon nodes not being synced:

jakubgs@nec-01.ih-eu-mda1.nimbus.mainnet:/docker % for port in $(seq 9301 9304); do c 0:$port/eth/v1/node/syncing | jq -c; done            
{"data":{"head_slot":"11735295","sync_distance":"73605","is_syncing":true,"is_optimistic":true,"el_offline":false}}
{"data":{"head_slot":"11808900","sync_distance":"0","is_syncing":false,"is_optimistic":true,"el_offline":true}}
{"data":{"head_slot":"11808899","sync_distance":"1","is_syncing":false,"is_optimistic":true,"el_offline":false}}
{"data":{"head_slot":"11808900","sync_distance":"0","is_syncing":false,"is_optimistic":true,"el_offline":true}}

And the sync distance appears to be growing.

[jakubgs]

I have purged Hoodi nimbus-eth1 nodes to allow them to re-sync from scratch:

 > a nimbus-hoodi-nec -a 'for script in /docker/nimbus-eth1-*/rpc.sh; do $script eth_syncing | jq -c; done'
nec-01.ih-eu-mda1.nimbus.hoodi | CHANGED | rc=0 >>
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":{"startingBlock":"0x7a00","currentBlock":"0x1","highestBlock":"0x78ca5"}}
{"jsonrpc":"2.0","id":1,"result":{"startingBlock":"0x5480","currentBlock":"0x1","highestBlock":"0x78cc1"}}
{"jsonrpc":"2.0","id":1,"result":{"startingBlock":"0xa80","currentBlock":"0x1","highestBlock":"0x78d0f"}}
nec-05.ih-eu-mda1.nimbus.hoodi | CHANGED | rc=0 >>
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":{"startingBlock":"0x7860","currentBlock":"0x1","highestBlock":"0x78ca5"}}
{"jsonrpc":"2.0","id":1,"result":{"startingBlock":"0x5560","currentBlock":"0x1","highestBlock":"0x78cc1"}}
{"jsonrpc":"2.0","id":1,"result":false}
nec-03.ih-eu-mda1.nimbus.hoodi | CHANGED | rc=0 >>
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":{"startingBlock":"0x7920","currentBlock":"0x1","highestBlock":"0x78ca5"}}
{"jsonrpc":"2.0","id":1,"result":{"startingBlock":"0x51e0","currentBlock":"0x1","highestBlock":"0x78cc1"}}
{"jsonrpc":"2.0","id":1,"result":{"startingBlock":"0x640","currentBlock":"0x1","highestBlock":"0x78d0f"}}
nec-04.ih-eu-mda1.nimbus.hoodi | CHANGED | rc=0 >>
{"jsonrpc":"2.0","id":1,"result":{"startingBlock":"0x7760","currentBlock":"0x1","highestBlock":"0x78ca0"}}
{"jsonrpc":"2.0","id":1,"result":{"startingBlock":"0x7700","currentBlock":"0x1","highestBlock":"0x78ca5"}}
{"jsonrpc":"2.0","id":1,"result":{"startingBlock":"0x5220","currentBlock":"0x1","highestBlock":"0x78cc1"}}
{"jsonrpc":"2.0","id":1,"result":{"startingBlock":"0x1a20","currentBlock":"0x1","highestBlock":"0x78d0f"}}
nec-02.ih-eu-mda1.nimbus.hoodi | CHANGED | rc=0 >>
{"jsonrpc":"2.0","id":1,"result":{"startingBlock":"0x82a0","currentBlock":"0x1","highestBlock":"0x78c9c"}}
{"jsonrpc":"2.0","id":1,"result":{"startingBlock":"0x71e0","currentBlock":"0x1","highestBlock":"0x78ca5"}}
{"jsonrpc":"2.0","id":1,"result":{"startingBlock":"0x4ec0","currentBlock":"0x1","highestBlock":"0x78cc1"}}
{"jsonrpc":"2.0","id":1,"result":false}

But similar issues with stable beacon nodes exist on mainnet.

[jakubgs]

I was unable to import state from ERA files for Holesky nodes due to this error:

 > sudo -u nimbus repo/build/nimbus_execution_client import --network=holesky --era-dir=/docker/era --data-dir=data/shared_holesky_0
INF 2025-05-29 18:27:47.149+00:00 Launching execution client                 version=v0.1.0-09c620a1 conf="(dataDir: data/shared_holesky_0, era1DirOpt: none(OutDir), eraDirOpt: some(/docker/era), keyStore: data/shared_holesky_0/keystore, chainDbMode: Aristo, importKey: , evm: \"\", trustedSetupFile: none(string), extraData: \"nimbus-eth1/v0.1.0-09c620a1\", gasLimit: 36000000, network: \"holesky\", customNetwork: none(NetworkParams), networkId: 17000, networkParams: (config: ..., genesis: ...), logLevel: \"INFO\", logStdout: auto, logMetricsEnabled: false, logMetricsInterval: 10, metricsEnabled: false, metricsPort: 9093, metricsAddress: 127.0.0.1, bootstrapNodes: @[], bootstrapFile: , bootstrapEnrs: @[], staticPeers: @[], staticPeersFile: , staticPeersEnrs: @[], reconnectMaxRetry: 0, reconnectInterval: 15, listenAddress: ::, tcpPort: 30303, udpPort: 0, maxPeers: 25, nat: (hasExtIp: false, nat: NatAny), discovery: V4, netKey: \"random\", agentString: \"nimbus-eth1/v0.1.0-09c620a1/linux-amd64/Nim-2.0.14/nimvm\", numThreads: 0, beaconChunkSize: 0, rocksdbMaxOpenFiles: 2048, rocksdbWriteBufferSize: 67108864, rocksdbRowCacheSize: 0, rocksdbBlockCacheSize: 2684354560, rdbVtxCacheSize: 536870912, rdbKeyCacheSize: 268435456, rdbBranchCacheSize: 1073741824, rdbPrintStats: false, rewriteDatadirId: false, cmd: import, maxBlocks: 18446744073709551615, chunkSize: 8192, csvStats: none(string), fullValidation: false, noValidation: true, storeBodies: false, storeReceipts: false, storeSlotHashes: false)"
INF 2025-05-29 18:27:50.517+00:00 Writing data dir ID                        ID=0x252e151541c3f35941aa44b3a13f874b6d4435e4e56fb293bc8bc5c87816a90c
INF 2025-05-29 18:27:50.517+00:00 Threadpool started                         numThreads=8
INF 2025-05-29 18:27:50.525+00:00 Writing genesis to DB                      blockHash=0xb5f7f912443c940f21fd611f12828d75b534364ed9e95ca4e307729a4661bde4 stateRoot=0x69d8c9d72f6fa4ad42d4702b433707212f90db395eb54dc20bc85de253788783 difficulty=1 gasLimit=25000000 timestamp=1695902100 nonce=0x0000000000001234
INF 2025-05-29 18:27:50.525+00:00 Database initialized                       base="(0xb5f7f912443c940f21fd611f12828d75b534364ed9e95ca4e307729a4661bde4, 0)" finalized="(0xb5f7f912443c940f21fd611f12828d75b534364ed9e95ca4e307729a4661bde4, 0)" head="(0xb5f7f912443c940f21fd611f12828d75b534364ed9e95ca4e307729a4661bde4, 0)"
NTC 2025-05-29 18:27:50.525+00:00 Importing era archive                      blockNumber=1 dataDir=data/shared_holesky_0 eraDir=/docker/era
FAT 2025-05-29 18:27:50.917+00:00 Could not load historical summaries        eraDir=/docker/era error="Unable to read state: SSZ BeaconState: object dynamic portion starts at invalid offset"

Error:

Unable to read state: SSZ BeaconState: object dynamic portion starts at invalid offset

I've opted to just purge the data folder and let them sync from scratch.

[jakubgs]

Hoodi nodes are all synced except for two:

 > a nimbus-hoodi-nec -a 'for script in /docker/nimbus-eth1-*/rpc.sh; do $script eth_syncing | jq -c; done'
nec-04.ih-eu-mda1.nimbus.hoodi | CHANGED | rc=0 >>
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}
nec-03.ih-eu-mda1.nimbus.hoodi | CHANGED | rc=0 >>
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}
nec-02.ih-eu-mda1.nimbus.hoodi | CHANGED | rc=0 >>
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}
nec-05.ih-eu-mda1.nimbus.hoodi | CHANGED | rc=0 >>
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":{"startingBlock":"0x7bb8d","currentBlock":"0x7bb8d","highestBlock":"0x0"}}
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}
nec-01.ih-eu-mda1.nimbus.hoodi | CHANGED | rc=0 >>
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":{"startingBlock":"0x7ecd7","currentBlock":"0x0","highestBlock":"0x0"}}
{"jsonrpc":"2.0","id":1,"result":false}

They all appear about the same size on nec-05 so it might just be a matter of time before it finishes:

jakubgs@nec-05.ih-eu-mda1.nimbus.hoodi:~ % sudo du -hsc /docker/nimbus-*
11G	/docker/nimbus-eth1-hoodi-01
14G	/docker/nimbus-eth1-hoodi-02
14G	/docker/nimbus-eth1-hoodi-03
14G	/docker/nimbus-eth1-hoodi-04
52G	total

[jakubgs]

This is done.