feat: [2/4] integrate smtforest, avoid ser/de of full account/vault data in database #1394

drahnr · 2025-11-28T10:40:42Z

Part 2/4 of #1185 and lays the ground for #1354

~~⚠️ The PR is rather large, a lot of the changes affect large pieces of code across.~~

Intent

Provide the baseline for cheap querying partial storage maps, but not implement it
Prepare for potential deprecation of AccountInfo

core changes

Remove the vault / storage_map BLOB entries from the accounts table.
Add SmtForest and integrate into apply_block and State::load

significant changes in the following files:

crates/store/src/db/schema.rs introduces account_storage_headers and removes storage (the full serialized account storage) from accounts table
crates/store/src/state.rs / fn State::apply_block now updates the database, but also the lookup table of roots for the SmtForest and the entries in the forest itself, indirect lookup tables

out of scope

usage of the LargeSmtForest for partial storage maps - [devops] Deploy without nuking db #670
individual of partial storage map queries - Implement support for retrieving subsets of storage maps in GetAccountProof endpoint #1354
cleanup for both DB and (Large|)SmtForest - Add cleanup routine for outdated historical storage entries #1355

how to review

Take the existing TODOs into consideration, if they make sense. This will be the follow up PR.

crates/proto/src/domain/account.rs

crates/store/src/db/models/queries/accounts.rs

crates/store/src/state.rs

Mirko-von-Leipzig

Mostly questions; no real issues found as yet

crates/store/src/db/migrations/2025062000000_setup/down.sql

crates/store/src/state.rs

bobbinth

Thank you! I left some comments inline and pushed a few commits to try to organize the code a bit better. Overall, we are still quite far from being able to merge this PR as there are some bugs and logical inconsistencies.

Also, given the amount of remaining issues, I'm a bit concerned about our ability to wrap this PR up in a timely manner. Given the complexity of this PR, deep reviews take quite a bit of time. Maybe there is a way to split this up into smaller incremental PRs that we can review and merge much more quickly.

bobbinth · 2025-12-27T17:40:01Z

crates/store/src/state.rs

+    let public_account_ids: Vec<AccountId> = db
+        .select_all_account_commitments()
+        .await?
+        .iter()
+        .filter_map(|(id, _commitment)| if id.has_public_state() { Some(*id) } else { None })
+        .collect();


This is pretty inefficient because we just need to get a list of accounts with public state, but instead we are loading all account commitments. This is OK for now because we'll replace the initialization code as we migrate to RocksDB-based backends.

We could filter as part of the query using lt/gt given a certain prefix range if the layout is advantageous, LIKE statements are not supported by diesel after taking a brief look.
EDIT: looking at the layout of the exact bits in AccountId and the prefix, it seems the pub/priv/network types are encoding it at the lower octet of the prefix, range filtering won't work.

bobbinth · 2025-12-27T17:43:24Z

crates/store/src/state.rs

    /// Read-write lock used to prevent writing to a structure while it is being used.
    ///
    /// The lock is writer-preferring, meaning the writer won't be starved.
    inner: RwLock<InnerState>,

+    /// Forest-related state `(SmtForest, storage_roots, vault_roots)` with its own lock.
+    forest: RwLock<InnerForest>,


I'm not seeing a good reason to separate InnerForest from InnerState - so, let's combine them. This could look simply as:

struct InnerState<S = MemoryStorage> where S: SmtStorage, { blockchain: Blockchain, account_tree: AccountTreeWithHistory<S>, nullifier_tree: NullifierTree<LargeSmt<S>>, forest: InnerForest, }

In the future (when moving to a RocksDB backend), we should refactor this to move InnerState into a separate file and clean up the module hierarchy.

It takes care of encapsulating the indices required for lookup and hiding the internals, which will come in handy once LargeSmtForest lands, which has a largely different API.

That's not withstanding a separate module cleanup/InnerState move.

crates/store/src/state.rs

crates/store/src/inner_forest/mod.rs

crates/store/src/db/models/queries/accounts.rs

…forest

bobbinth

Looks good! Thank you! I reviewed most of the non-test code and left some small comments inline.

Also, my understanding is that in this PR, we are just updating the SMT forest, but actually getting data from it would be done in the next PR, right?

crates/store/src/lib.rs

crates/store/src/db/mod.rs

crates/store/src/accounts/mod.rs

crates/store/src/lib.rs

crates/store/src/inner_forest/mod.rs

drahnr · 2026-01-06T17:31:12Z

Also, my understanding is that in this PR, we are just updating the SMT forest, but actually getting data from it would be done in the next PR, right?

That is correct. Querying and populating partial queries/responses is done in 3/4

bobbinth

Looks good! Thank you! I left some more comments inline - all of them should be pretty straight-forward. Once they are addressed, we should be good to merge.

bobbinth · 2026-01-07T01:08:47Z

crates/store/src/inner_forest/mod.rs

+    /// Retrieves the most recent vault SMT root for an account.
+    ///
+    /// Returns the latest vault root entry regardless of block number.
+    /// Used when applying incremental deltas where we always want the previous state.
+    fn get_latest_vault_root(&self, account_id: AccountId) -> Word {
+        self.vault_roots
+            .range((account_id, BlockNumber::GENESIS)..)
+            .take_while(|((id, _), _)| *id == account_id)
+            .last()
+            .map_or_else(Self::empty_smt_root, |(_, root)| *root)
+    }


Let's mention in the doc comments that if a vault root for the specified account cannot be found, we return a root of an empty tree.

Also, looking at the implementation, I wonder if a better backing structure would be BTreeMap<AccountId, (BlockNumber, Word)>. This would make getting the last vault root much simpler.

Getting root for a specific block number should be pretty simple as well (we could use a binary search or even just a linear scan since we know that the number of entries will be pretty small per account ID).

bobbinth · 2026-01-07T01:09:04Z

crates/store/src/inner_forest/mod.rs

+    /// Retrieves the most recent storage map SMT root for an account slot.
+    ///
+    /// Returns the latest storage root entry regardless of block number.
+    /// Used when applying incremental deltas where we always want the previous state.
+    fn get_latest_storage_map_root(
+        &self,
+        account_id: AccountId,
+        slot_name: &StorageSlotName,
+    ) -> Word {
+        self.storage_roots
+            .range((account_id, slot_name.clone(), BlockNumber::GENESIS)..)
+            .take_while(|((id, name, _), _)| *id == account_id && name == slot_name)
+            .last()
+            .map_or_else(Self::empty_smt_root, |(_, root)| *root)
+    }


Same comments as above - though, here the data structure would be BTreeMap<(AccountId, StorageSlotName), (BlockNumber, Word)>.

bobbinth · 2026-01-07T01:09:46Z

crates/store/src/inner_forest/mod.rs

+        let prev_root = if is_full_state {
+            Self::empty_smt_root()
+        } else {
+            self.get_latest_vault_root(account_id)
+        };


This could be just:

let prev_root = self.get_latest_vault_root(account_id);

bobbinth · 2026-01-07T01:14:50Z

crates/store/src/inner_forest/mod.rs

+                    .map_or(0, |asset| asset.amount());
+
+                let new_balance = i128::from(prev_amount) + i128::from(*amount_delta);
+                u64::try_from(new_balance.max(0)).expect("balance fits in u64")


If new_balance somehow ends up negative, we should catch this. A panic could be fine for now, but ideally, it'd be an error.

bobbinth · 2026-01-07T01:19:06Z

crates/store/src/inner_forest/mod.rs

+            let prev_root = if is_full_state {
+                Self::empty_smt_root()
+            } else {
+                self.get_latest_storage_map_root(account_id, slot_name)
+            };


Similar to one of the above comments, this could probably be just:

let prev_root = self.get_latest_storage_map_root(account_id, slot_name);

drahnr mentioned this pull request Dec 1, 2025

Implement the history overlay mechanism 0xMiden/crypto#679

Merged

5 tasks

drahnr force-pushed the bernhard-integrate-smtforest branch 2 times, most recently from 7980bed to 771f6d7 Compare December 2, 2025 11:16

drahnr added 12 commits December 2, 2025 16:22

for account queries, now return partials too

4ff0970

drop all tables as part of migration

5ee1043

externalize storage and vault blobs to separate tables

8eca359

trailing .

e7f17ed

smt forest

c8b43ab

changset, should go away after rebase

19164be

improve

8eb49af

TODO and deprecation

6725461

account queries

ee65a88

yes

741df6f

why

6a64077

y

9d5806e

drahnr force-pushed the bernhard-integrate-smtforest branch from 771f6d7 to b2aa54b Compare December 2, 2025 15:22

y

2964a93

drahnr force-pushed the bernhard-integrate-smtforest branch from b2aa54b to 2964a93 Compare December 2, 2025 15:54

drahnr commented Dec 2, 2025

View reviewed changes

crates/proto/src/domain/account.rs Outdated Show resolved Hide resolved

drahnr commented Dec 2, 2025

View reviewed changes

crates/store/src/db/models/queries/accounts.rs Outdated Show resolved Hide resolved

drahnr commented Dec 2, 2025

View reviewed changes

crates/store/src/db/models/queries/accounts.rs Outdated Show resolved Hide resolved

drahnr commented Dec 2, 2025

View reviewed changes

crates/store/src/db/models/queries/accounts.rs Outdated Show resolved Hide resolved

drahnr commented Dec 2, 2025

View reviewed changes

crates/store/src/db/models/queries/accounts.rs Outdated Show resolved Hide resolved

drahnr commented Dec 2, 2025

View reviewed changes

crates/store/src/db/models/queries/accounts.rs Outdated Show resolved Hide resolved

review comments

9416a63

drahnr commented Dec 2, 2025

View reviewed changes

crates/store/src/state.rs Outdated Show resolved Hide resolved

sanitize comments

ccc2d63

This was referenced Dec 2, 2025

refactor: [4/4] unify get_account_details and get_account_proof[s] into get_account #1385

Open

fix: make get account proof retrieve latest known state #1422

Merged

Mirko-von-Leipzig reviewed Dec 4, 2025

View reviewed changes

bobbinth added 3 commits December 27, 2025 01:22

chore: move InnerForest module

354d586

chore: refactor SMT forest initialization

a96def0

chore: re-organize account queries

e8cdad1

bobbinth reviewed Dec 28, 2025

View reviewed changes

drahnr added 13 commits December 29, 2025 17:54

nope

3009bf7

fix inconsistency

3110962

faster

53cb5e8

undue changes

0cc0c61

move fn to innerforest

ac7b8f9

y

369db2f

another

6cd1033

test re-review

7613624

fuckup

e04ff10

update

9d8c220

sync docs

5ed1a4f

cleanup

3346d9f

fmt

dbbc1eb

drahnr mentioned this pull request Dec 29, 2025

refactor [1/4]: change accounts db schema #1481

Open

drahnr changed the base branch from next to bernhard-db-schema-queries December 29, 2025 23:53

drahnr changed the title ~~feat: [1/3] integrate smtforest, avoid ser/de of full account/vault data in database~~ feat: [2/4] integrate smtforest, avoid ser/de of full account/vault data in database Dec 29, 2025

Merge branch 'bernhard-db-schema-queries' into bernhard-integrate-smt…

c5a199a

…forest

drahnr force-pushed the bernhard-integrate-smtforest branch from 3bd5451 to c5a199a Compare December 30, 2025 00:53

Merge branch 'bernhard-db-schema-queries' into bernhard-integrate-smt…

c712ba4

…forest

bobbinth reviewed Jan 6, 2026

View reviewed changes

drahnr mentioned this pull request Jan 6, 2026

Implement PersistentSmtForest 0xMiden/crypto#670

Open

drahnr and others added 3 commits January 6, 2026 18:56

review

d9a666f

remove dead code

29b840c

chore: minor rename

1b22a34

bobbinth approved these changes Jan 7, 2026

View reviewed changes

feat: [2/4] integrate smtforest, avoid ser/de of full account/vault data in database #1394

Are you sure you want to change the base?

feat: [2/4] integrate smtforest, avoid ser/de of full account/vault data in database #1394

Conversation

drahnr commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Intent

core changes

significant changes in the following files:

out of scope

how to review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Mirko-von-Leipzig left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bobbinth left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

drahnr Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bobbinth left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

drahnr commented Jan 6, 2026

Uh oh!

bobbinth left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

drahnr commented Nov 28, 2025 •

edited

Loading

drahnr Dec 29, 2025 •

edited

Loading