Add an initial method to generate a full reference hard fork config for testing #17816

cjjdespres · 2025-09-17T15:36:31Z

The client RPC command mina advanced test generate-hard-fork-config --hardfork-config-dir <DIR> can be used to request that a running daemon generate and save a complete hard fork config in the specified directory <DIR>. This will currently contain:

A genesis/ directory containing the tar.gz archives of the hard fork genesis ledgers

A daemon.json containing the updated genesis constants for the hard fork
A genesis_legacy/ directory containing the tar.gz archives of the hard fork genesis ledgers without any account migration applied
A daemon.legacy.json containing the updated genesis constants based on the genesis_legacy/ ledgers
An empty activated file to indicate that the hard fork config was generated fully without errors

This method is not quite complete - it needs:

A parameter to specify a specific slot or chain height to use for the hard fork block: it will only dump the last block produced before the configured slot_tx_end (or the best tip if that's not set) at the moment
To calculate the genesis timestamp properly: it should actually be the timestamp of the chain end slot plus a configured offset
To save the hard fork block itself as a precomputed block json file

cjjdespres · 2025-09-17T15:50:22Z

Related: #17752

glyh · 2025-09-18T12:07:14Z

I suggest changing the folder structure to something like

$ tree genesis/
genesis/
├── berkeley
│   ├── activated
│   ├── daemon.json
│   └── whatever.tar.gz
└── mesa
    ├── activated
    ├── daemon.json
    └── whatever.tar.gz

3 directories, 6 files

So to be forward compatible. Also this allow us to mark completeness of fork config separately.

Maybe berkeley and mesa are not proper names, then use hard fork mode to name these folders?

cjjdespres · 2025-09-19T15:48:01Z

I think the directory structure is something that @dkijania might have opinions on as well, since he thought having this command output the precomputed fork block and the legacy format config would be useful for testing.

I was also wondering if it would be useful to save the fork_config.json that the forkConfig endpoint would return (reusing that code and not reimplementing it, of course) so we could test that the runtime genesis executables give the same output as the automatic config dump.

cjjdespres · 2025-09-19T15:57:32Z

I hadn't intended this to match the output of the automatic config dump exactly, but maybe that's wise. In that case, the structure for the migrated (mesa) format would look like

<DIR>/daemon.json
<DIR>/chain-state/mesa-{network}/activated
<DIR>/chain-state/mesa-{network}/genesis/{ledger}.tar.gz

if we wanted the output in <DIR> to be a valid config directory structure.

This is slightly complicated by the fact that the current compatible daemon wants the config directory to look like this:

<DIR>/daemon.json
<DIR>/genesis/{ledger}.tar.gz

Maybe it would be simpler if the command accepted a few parameters to control where the config directories and precomputed block json file should go, separately.

cjjdespres · 2025-09-19T16:42:02Z

Or it could be this by default:

/daemon.json
/chain-state/mesa-{network}/activated
/chain-state/mesa-{network}/genesis/{ledger}.tar.gz
/chain-state/mesa-{network}/fork-validation/precomputed_fork_block.json
/chain-state/mesa-{network}/fork-validation/fork_config.json
/chain-state/mesa-{network}/fork-validation/legacy/daemon.json
/chain-state/mesa-{network}/fork-validation/legacy/genesis/{ledger}.tar.gz

and in the automatic config dump we'd omit the fork-validation/ directory. (Or still output it if a testing option was set).

dkijania · 2025-09-19T18:48:53Z

src/app/cli/src/init/client.ml

     Cli_lib.Exceptions.handle_nicely
       Test_genesis_creation.time_genesis_creation )

+let test_generate_hardfork_config =


Am i understand correctly that we are putting this command under test since its mocking some parameters, right?

That's a good point. I don't actually think it needs to be under test. It could just be an advanced command. It's mocking some parameters only because I haven't added them yet. The genesis timestamp delta in particular could be an optional parameter to the command, and the daemon could use its configured timestamp delta if that parameter isn't set. But, the runtime config for that delta hasn't been added yet either.

dkijania

Any unit, component tests for this feature?

glyh

It does seems this PR has some rough edges to be completed before called functional

glyh · 2025-09-18T00:30:50Z

src/app/cli/src/init/mina_run.ml

+              Mina_lib.Hardfork_config.dump_reference_config
+                ~breadcrumb_spec:`Stop_slot ~directory_name mina
+            in
+            match result with


Nit:

convert error to json with helpers in Error_json and attach them to metadata in logging.

this log level should be error, I think

I think it's also useful to log a info line on succeed

glyh · 2025-09-18T12:10:45Z

src/app/cli/src/init/client.ml

+             Daemon_rpcs.Generate_hardfork_config.rpc directory_name port
+         with
+         | Error e ->
+             eprintf "Failed to request hardfork config generation: %s\n"


Nit:

use logger

use Error_json to log error in json to logging metadata

glyh · 2025-09-18T12:10:59Z

src/app/cli/src/init/client.ml

+               (Error.to_string_hum e) ;
+             exit 17
+         | Ok () ->
+             printf "Hardfork configuration successfully requested in %s\n"


Nit: use logger. and log dir name in metadata.

glyh · 2025-09-18T12:12:00Z

src/app/cli/src/init/mina_run.ml

+                ()
+          in
+          don't_wait_for (dump ()) ;
+          return (Ok ()) )


I don't understand the fire-and-forget pattern here. Why can't this be a blocking operation?

Like we block for a good reason. Client side block indicates that server is working.

Good catch. I'll change this. I wrote it like this because initially I was migrating the ledgers all at once (before the ledger sync was fully merged) and that was causing the daemon to become unresponsive, which made the command fail due to the timeout.

In reality we should be making sure that the daemon remains responsive throughout the hard fork config generation.

Should hopefully be fixed in the latest update.

glyh · 2025-09-18T12:13:07Z

src/lib/daemon_rpcs/daemon_rpcs.ml

+module Generate_hardfork_config = struct
+  type query = string [@@deriving bin_io_unversioned]
+
+  type response = unit Or_error.t [@@deriving bin_io_unversioned]


I assume this is not the final RPC version since more param to be added.

glyh · 2025-09-22T01:49:23Z

src/lib/mina_lib/mina_lib.ml

+          , Account.Hardfork.of_stable acct ) )
+    in
+    (* should this be a parameter? *)
+    let chunk_size = 1 lsl 6 in


Probably, but for now could we lift it to top level so it's easier to spot? When we want them actually to be params we can always do a refactor.

glyh · 2025-09-22T12:46:47Z

src/lib/mina_lib/mina_lib.ml

+            Ledger.Hardfork_db.create ~directory_name ~depth ()
+          in
+          `Inputs
+            ( accounts


We don't need to pass accounts along with length of accounts, I think. the later is just length of the previous.

glyh · 2025-09-22T12:49:56Z

src/lib/mina_lib/mina_lib.ml

+    let genesis_next_epoch_ledger_opt =
+      match source_ledgers.next_epoch_ledger with
+      | `Genesis l ->
+          get_genesis_migration_inputs ~name:"staking_ledger" l


Not something that's very important, but it seems here we can also clash if current epoch staking ledger = next epoch staking ledger

Also I'm confused by the naming scheme of these ledgers.

Not something that's very important, but it seems here we can also clash if current epoch staking ledger = next epoch staking ledger

True. This is actually likely to be encountered in my proposed component test for this command (single node with simple genesis ledger) so I think I'll end up wanting to fix that issue in the runtime_genesis_ledger.exe and here.

Also I'm confused by the naming scheme of these ledgers.

The ~name? I just wanted a temporary name for the ledger databases that were being generated. I do see that I accidentally give ~name:"staking_ledger" to the genesis next epoch ledger. I'll fix that.

glyh · 2025-09-22T13:07:10Z

src/lib/mina_lib/mina_lib.ml

+      ~target_dir ~ledger_name_prefix root =
+    let open Deferred.Or_error.Let_syntax in
+    let root_hash = get_root_hash root in
+    let ledger_dirname = get_directory root |> Option.value_exn in


Nit: add an error message here.

glyh · 2025-09-22T13:20:16Z

src/lib/mina_lib/mina_lib.ml

+      | `Root l ->
+          Ledger.Root.create_stable_checkpoint l ~directory_name
+      | `Uncommitted _l ->
+          Ledger.Root.create_stable_checkpoint


I don't understand the necessity of breaking apply diffs and creating checkpoint into 2 steps. It make here fishy at a first glance.

I'm hoping the new code is a little easier to follow. I still do it in stages, but hopefully a little more cleanly:

Collect the diffs and copy the roots

Migrate the stable database if the migrated version isn't available

Apply the diffs to the copied roots

I'm doing it this way because I wanted the code to work with or without ledger sync enabled, so (2) might take a while. So, I'd like to gather all the inputs necessary for generating the hard fork config in one go, including copying the roots. Then I can do whatever slow, gradual things need to be done on the copies of everything.

I'm still slightly concerned about what might happen if, say, we try generating a hard fork config at the precise moment the daemon decides to commit to a new snarked root. We might have to move the code to gather the hard fork inputs into the transition controller, or at least add some kind of "fork input copying in progress" async flag(s) somewhere, so the controller knows not to mess with the underlying databases until we're done.

glyh · 2025-09-22T13:24:09Z

Do you mind left some instructions here using these configs to boot a new post-HF node instance? That'd be helpful

glyh · 2025-09-23T13:58:23Z

I hadn't intended this to match the output of the automatic config dump exactly, but maybe that's wise. In that case, the structure for the migrated (mesa) format would look like
<DIR>/daemon.json
<DIR>/chain-state/mesa-{network}/activated
<DIR>/chain-state/mesa-{network}/genesis/{ledger}.tar.gz
if we wanted the output in <DIR> to be a valid config directory structure.

This is slightly complicated by the fact that the current compatible daemon wants the config directory to look like this:
<DIR>/daemon.json
<DIR>/genesis/{ledger}.tar.gz
Maybe it would be simpler if the command accepted a few parameters to control where the config directories and precomputed block json file should go, separately.

I think you can use symlink? Although I personally think it's not critical. I'm fine keeping it as it currently is, just a nit idea of improvement.

cjjdespres · 2025-10-01T18:41:34Z

I've force-pushed to bring in the recent compatible fixes. I'll be getting to the review comments next.

cjjdespres · 2025-10-01T19:20:45Z

Any unit, component tests for this feature?

That is something I want to add next. I was hoping for feedback on the general design/initial implementation before I kept going. (Thanks for that, by the way).

The basic component test I was thinking of was starting a single daemon with a known genesis ledger, using this command to generate a compatible hard fork config, then restarting the daemon with the output config.

It does seems this PR has some rough edges to be completed before called functional

I agree.

cjjdespres · 2025-10-03T18:44:25Z

I have updated/rewritten the PR. It's hopefully a bit cleaner. I'll update the PR description on Monday, but just to have it down:

The command is now mina advanced generate-hardfork-config instead of mina advanced test
The fork config is generated according to how the daemon's current config directory is laid out. So it's currently this:

/activated
/daemon.json
/genesis/<ledger tar files>
/fork-validation/legacy/daemon.json
/fork-validation/legacy/genesis/<ledger tar files>

I think it will be easier for testing purposes to create the config directories according to what the compatible and develop daemons currently expect. When we eventually add #17762 and subsequently modify the develop daemon to expect its chain state to be stored in chain-state/mesa-{network}/, we can adjust how the fork config is generated with this code.

Still to do for this code (in subsequent PRs, unless you feel strongly about doing this now)

save the precomputed block json in /fork-validation
save the same fork_config.json that would have been generated by the graphql query and put it in /fork-validation
add an option to choose whether or not to save any of the /fork-validation items at all
handle the corner case where we try generating a hard fork config and the daemon is still at genesis

Incidentally, if --hardfork-mode auto is enabled then this config generation is practically instantaneous, except that it takes 40s to generate the tar files themselves. So we'll probably want an option to not generate the tar files and just save the unpacked genesis databases.

cjjdespres · 2025-10-03T19:11:41Z

Do you mind left some instructions here using these configs to boot a new post-HF node instance? That'd be helpful

All you should need to do is something like:

Boot up a compatible daemon incorporating this PR and sync it to a network
Run mina advanced generate-hardfork-config --hardfork-config-dir /some/path
Stop the daemon
Start up a daemon that knows about the zkapp state size increase, and use --config-dir /some/path and not --config-file. You should probably start it in demo mode as well, unless you have other hard fork daemons lying around. Alternatively, start up a compatible daemon and give it --config-dir /some/path/fork-validation/legacy/.

I haven't tested the migrated hard fork config again since I wrote the previous version of this PR two weeks ago, but I have tested the /fork-validation/legacy/ config with a compatible daemon and it does still seem to boot up. Of course, we're going to want to start adding tests to expose all the horrifying bugs that surely still remain, make sure this agrees with the manual procedure, etc.

cjjdespres requested a review from a team as a code owner September 17, 2025 15:36

cjjdespres force-pushed the cjjdespres/full-fork-config branch from 96a8de1 to 6a7b23f Compare September 17, 2025 16:16

dkijania reviewed Sep 19, 2025

View reviewed changes

glyh reviewed Sep 22, 2025

View reviewed changes

cjjdespres force-pushed the cjjdespres/full-fork-config branch from 6a7b23f to 371f953 Compare October 1, 2025 18:40

Return source ledgers in prepare inputs

1214fc4

cjjdespres mentioned this pull request Oct 3, 2025

Make initial root migration for converting ledgers gradual #17875

Open

cjjdespres added 2 commits October 3, 2025 14:29

Add more config-agnostic methods to Root ledger

fafaa0b

Add client method to generate hard fork config

2c885cc

cjjdespres force-pushed the cjjdespres/full-fork-config branch from 371f953 to 2c885cc Compare October 3, 2025 18:30

Add activated file to generated hard fork config

2e0b1ff

Add an initial method to generate a full reference hard fork config for testing #17816

Are you sure you want to change the base?

Add an initial method to generate a full reference hard fork config for testing #17816

Uh oh!

Conversation

cjjdespres commented Sep 17, 2025 • edited by glyh Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cjjdespres commented Sep 17, 2025

Uh oh!

glyh commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cjjdespres commented Sep 19, 2025

Uh oh!

cjjdespres commented Sep 19, 2025

Uh oh!

cjjdespres commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dkijania left a comment

Choose a reason for hiding this comment

Uh oh!

glyh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

glyh commented Sep 22, 2025

Uh oh!

glyh commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cjjdespres commented Oct 1, 2025

Uh oh!

cjjdespres commented Oct 1, 2025

Uh oh!

cjjdespres commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cjjdespres commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

cjjdespres commented Sep 17, 2025 •

edited by glyh

Loading

glyh commented Sep 18, 2025 •

edited

Loading

cjjdespres commented Sep 19, 2025 •

edited

Loading

glyh commented Sep 23, 2025 •

edited

Loading

cjjdespres commented Oct 3, 2025 •

edited

Loading

cjjdespres commented Oct 3, 2025 •

edited

Loading