Investigate possible crash during epoch ledger snapshot syncing

[cjjdespres]

(I'm still trying to reproduce this locally. I got the crash with --hardfork-mode auto enabled - which we still have to rename - but the code looks like it could be susceptible to the same bug without that enabled. I also did not save the exact crash message, unfortunately.).

I was periodically syncing a daemon to devnet, with and without --hardfork-mode auto, and the daemon crashed in this bit of the code while --hardfork-mode auto was enabled:

mina/src/lib/consensus/proof_of_stake.ml

Lines 2663 to 2672 in b0ab82e

	let root_ledger_of_snapshot snapshot snapshot_config =
	O1trace.sync_thread "root_ledger_of_snapshot" (fun () ->
	match snapshot.ledger with
	| Ledger_snapshot.Ledger_root ledger ->
	Ok ledger
	| Ledger_snapshot.Genesis_epoch_ledger packed ->
	Genesis_ledger.Packed.create_root packed
	~config:snapshot_config
	~depth:Context.constraint_constants.ledger_depth () )
	in

The create_root function threw an exception when trying to sync one of the epoch snapshots because the rocksdb checkpoint failed - the target directory of the checkpoint already existed. In other words, there was an epoch ledger snapshot already at the snapshot_config location while the daemon was still at the genesis epoch snapshot.

This was not failing in my local testing before - it may have started because of #17874. Before that PR, we'd do this in this situation:

mina/src/lib/consensus/proof_of_stake.ml

Lines 2668 to 2676 in 2c70f34

	| Ledger_snapshot.Genesis_epoch_ledger packed ->
	let fresh_root_ledger =
	Mina_ledger.Ledger.Root.create ~logger
	~config:snapshot_config
	~depth:Context.constraint_constants.ledger_depth
	()
	in
	Genesis_ledger.Packed.populate_root packed
	fresh_root_ledger )

That Leder.Root.create would open up whatever database is present at that config location. (The code before we made all these root ledger handling changes did the same thing). It would then overwrite the contents of the database with the genesis ledger, and then sync the ledger to the network. Thus, the daemon did not have to care about cleaning up an old epoch ledger database that was lying around.

I'm unsure of a few things:

• If this can be reproduced with --hardfork-mode auto, or if I can get this to show up without that enabled. (I'm still looking at it).
• If the daemon was correctly at the genesis epoch ledger snapshots at the moment it crashed.

We might want to add some code to delete any snapshot backing that might be present at the snapshot_config location before creating a new root from genesis. Though, if this only shows up with --hardfork-mode auto, then this kind of failure might be the result of a bug elsewhere.

[cjjdespres]

Okay, I think this is partially a manifestation of #17847, though the differences in behaviour with --hardfork-mode auto and without it are somewhat worrying.

What I did was this:

1. Start a daemon without --hardfork-mode auto and with an empty config directory
2. Let it run just to the point it starts to sync the staking ledger snapshot to the network, then stop the daemon. A mina client stop will suffice.
3. Start the daemon again

If you do (3) without --hardfork-mode auto, as in #17847, then you'll get a crash because the daemon tries to compute the delegatee table using a malformed database. If you do (3) with --hardfork-mode auto, then what happens is this (I omitted a lot of log lines):

2025-10-01 21:02:43 UTC [Warn] Persisted frontier failed to load
  
2025-10-01 21:02:43 UTC [Debug] Persistent frontier dropped
  
2025-10-01 21:02:43 UTC [Warn] Persistent database is out of sync with snarked_ledger
...
2025-10-01 21:04:22 UTC [Info] Synchronizing consensus local state
  
2025-10-01 21:04:22 UTC [Info] Syncing local state; requesting 2 snapshots from peers
  
2025-10-01 21:04:22 UTC [Info] Syncing epoch ledger with hash $target_ledger_hash
  target_ledger_hash: "jxSvzoyxvkvaVaFLGYtjkA5SznqQYwWeB5nRiUNz9un62bFcXp9"
2025-10-01 21:04:22 UTC [Error] Could not send error report: Node_error_service was not configured
  
2025-10-01 21:04:22 UTC [Error] Could not send error report: Node_error_service was not configured
  
2025-10-01 21:04:22 UTC [Error] Could not send error report: Node_error_service was not configured
  
2025-10-01 21:04:22 UTC [Fatal] Unhandled top-level exception: $exn
Generating crash report
  exn: {
  "sexp": [
    "monitor.ml.Error",
    [ "Failure", "Invalid argument: Directory exists" ],
    [
      "Raised at Stdlib.failwith in file \"stdlib.ml\", line 29, characters 17-33",
      "Called from Rocks__Rocks_common.finalize in file \"rocks_common.ml\", line 98, characters 8-12",
      "Re-raised at Rocks__Rocks_common.finalize in file \"rocks_common.ml\", line 102, characters 21-30",
      "Called from Rocksdb__Database.create_checkpoint in file \"src/lib/rocksdb/database.ml\", line 15, characters 2-63",
      "Called from Merkle_ledger__Database.Make.create_checkpoint in file \"src/lib/merkle_ledger/database.ml\", line 74, characters 15-59",
      "Called from Merkle_ledger__Converting_merkle_tree.With_database.create_checkpoint in file \"src/lib/merkle_ledger/converting_merkle_tree.ml\", line 315, characters 6-112",
      "Called from Mina_ledger__Root.Make.create_checkpoint in file \"src/lib/mina_ledger/root.ml\", line 197, characters 22-73",
      "Called from Genesis_ledger.Utils.create_root_from_backing_root in file \"src/lib/genesis_ledger/genesis_ledger.ml\", line 81, characters 4-49",
      "Called from Base__Result.try_with in file \"src/result.ml\", line 195, characters 9-15",
      "Caught by monitor coda"
    ]
  ],
  "backtrace": [
    "Raised at Stdlib.failwith in file \"stdlib.ml\", line 29, characters 17-33",
    "Called from O1trace.exec_thread in file \"src/lib/o1trace/o1trace.ml\", line 82, characters 6-27",
    "Called from Bootstrap_controller.run_cycle.(fun) in file \"src/lib/bootstrap_controller/bootstrap_controller.ml\", line 531, characters 16-268",
    "Called from Async_kernel__Deferred0.bind.(fun) in file \"src/deferred0.ml\", line 54, characters 64-69",
    "Called from Async_kernel__Job_queue.run_job in file \"src/job_queue.ml\" (inlined), line 128, characters 2-5",
    "Called from Async_kernel__Job_queue.run_jobs in file \"src/job_queue.ml\", line 169, characters 6-47"
  ]
}
2025-10-01 21:04:22 UTC [Fatal] Unhandled top-level exception: $exn
Generating crash report
  exn: {
  "sexp": [
    "monitor.ml.Error",
    [
      "Failure",
      "sync timing task `root_ledger_of_snapshot` failed, exception reported to parent monitor"
    ],
    [
      "Raised at Stdlib.failwith in file \"stdlib.ml\", line 29, characters 17-33",
      "Called from O1trace.exec_thread in file \"src/lib/o1trace/o1trace.ml\", line 82, characters 6-27",
      "Called from Consensus__Proof_of_stake.Make_str.Hooks.sync_local_state.(fun).sync in file \"src/lib/consensus/proof_of_stake.ml\", line 2676, characters 23-174",
      "Called from Consensus__Proof_of_stake.Make_str.Hooks.sync_local_state.(fun) in file \"src/lib/consensus/proof_of_stake.ml\", line 2735, characters 16-141",
      "Called from Base__Result.try_with in file \"src/result.ml\", line 195, characters 9-15",
      "Caught by monitor coda"
    ]
  ],
  "backtrace": [
    "Raised at Stdlib.failwith in file \"stdlib.ml\", line 29, characters 17-33",
    "Called from O1trace.exec_thread in file \"src/lib/o1trace/o1trace.ml\", line 82, characters 6-27",
    "Called from Bootstrap_controller.run_cycle.(fun) in file \"src/lib/bootstrap_controller/bootstrap_controller.ml\", line 531, characters 16-268",
    "Called from Async_kernel__Deferred0.bind.(fun) in file \"src/deferred0.ml\", line 54, characters 64-69",
    "Called from Async_kernel__Job_queue.run_job in file \"src/job_queue.ml\" (inlined), line 128, characters 2-5",
    "Called from Async_kernel__Job_queue.run_jobs in file \"src/job_queue.ml\", line 169, characters 6-47"
  ]
}

I suspect what's happening is that the converting merkle tree code is crashing due to the malformed database in a different place than where the non-converting-merkle-tree code is crashing. So, the daemon ends up getting significantly further along into the startup process before it eventually crashes. That's my guess, anyway.

[cjjdespres]

I think this crash could happen without --hardfork-mode auto, if outdated epoch ledgers happened to exist on disk. This wasn't the cause of the crash in my testing, because I was running the daemon well away from an epoch boundary. But, it seems likely that it could happen. So we need to make sure we delete the existing snapshots if we populate from genesis.

Also, this can also be triggered by the following incorrect code that needs to be fixed:

mina/src/lib/consensus/proof_of_stake.ml

Lines 464 to 468 in 8f958aa

	if Mina_ledger.Ledger.Root.Config.exists_backing config then (
	[%log info]
	~metadata:
	[ ("config", Mina_ledger.Ledger.Root.Config.to_yojson config) ]
	"Loading epoch ledger from disk: $config" ;

The exists_backing tests if there is a backing of the matching type in the config directory, which is wrong. What we should do is test if there is any suitable backing; otherwise, we might throw away an up-to-date epoch ledger!

[cjjdespres]

We may be okay, actually - if we've loaded an old epoch snapshot then we'll have a Ledger_root snapshot and not a Genesis_epoch_ledger snapshot. Assuming that logic is correct, we'll only try to create a new epoch snapshot from the genesis snapshots when there are no old epoch snapshot databases around.

The problem seems to be confined to the exists_backing bug I noted above in #17899 (comment), because that is a bug in the logic of loading/cleaning up old epoch ledger snapshot databases.