Account storage schema validation

[mmagician]

Defining the component schema itself is rather straightforward, and is outlined in #2062 (comment).

Once the schema is defined, we need some way for apps (wallet, explorer, etc) to access this schema defined by contract developers, and apply it to the onchain data (storage slot keys, values, types) they retrieve to correctly display the metadata about the storage slots. Let’s assume that we have a registry that hosts the schemas.

This issue is concerned with the validation of the schema.

Challenge 1: Schema validation

One of the challenges discussed in the retreat workshop is how the registry can validate storage schemas. Specifically, multiple schema definitions can be valid for the same account component.

Here’s an example (assuming named storage slots), where the first schema is the original one defined by the contract developer, and the second one is malicious:

[[storage]]
name = "approvers"
description = "Map with approvers"
type = "map"

[[storage.approvers.key]]
type = "u8"
name = "index"
description = "Index of the approver"

[[storage.approvers.value]]
type = "auth::rpo_falcon512::pub_key"
name = "pubkey"
description = "Falcon public signing key of the approver" 

and

[[storage]]
name = "approvers"
description = "Map with approvers"
type = "map"

[[storage.approvers.key]]
type = "u8"
name = "index"
description = "Index of the approver"

[[storage.approvers.value]]
type = "word"
name = "approver metadata"
description = "hash of the approver contact details" 

Without schema validation, if the wallet displays the second schema to the user, the user could be fooled into signing the more innocent-looking approver metadata changes, where in fact they might be giving up control over the account.

---

One potential solution to this is storing the commitment to the schema in one of the account storage slots itself. With named storage slots, this can become a common standard to store the commitment e.g. in the 0x1234::schema_commitment slot.

The commitment would be computed by canonically serializing the contents of the TOML, and hashing it to obtain a single-word value.

Then, to validate the schema, the registry (and the consuming apps) can easily validate that when the developer uploads the schema for a specific account, it matches that account’s onchain commitment.

Challenge 2: Component-wide vs. account-wide schema commitments

The description above assumes we have a single commitment to account storage, of ALL components together.

This is in contrast to how we currently create accounts, which is via individual AccountComponents, passed in as Packages, that together make up one Account.

If we were to maintain only a single commitment, the question is: which AccountComponent is the storage slot named 0x1234::schema_commitment defined in?

• we have a special component just for this purpose (preference)
• it is part of the first component in the list
• other?

The natural alternative to a per-account schema, then, is having per-component schema, i.e. 0x1234::<component>::schema_commitment.

We discussed this approach as an alternative, but now that I think of this, I don’t think the explorer would have a way of making sense of per-component commitments, because the onchain representation of the account no longer recognizes the distinction between the various AccountComponents (TODO is this correct?):

pub struct Account {
    id: AccountId,
    vault: AssetVault,
    storage: AccountStorage,
    code: AccountCode,
    nonce: Felt,
    seed: Option<Word>,
}

So even if the some of the components were “known” (with scheme available in the registry), while the others were new, such as for ExtendedWallet that holds the BasicWallet component + some custom component, the application that only sees AccountStorage onchain:

pub struct AccountStorage {
    slots: Vec<StorageSlot>,
}

cannot distinguish which StorageSlots come from which component.

---

Summary

Given all of the above, I think the best way forward right now is to create a dedicated AccountComponent with a single storage slot that holds per-account schema.

This would not be enforced at the protocol level, and thus be fully optional - but we could enshrine this in the client's account creation process, such that it becomes a de facto standard that the registry, explorer etc. can follow.

This special component would be created at the time of Account creation. Looking at the account creation process in the CLI, for example, we could do this as part of [process_packages](https://github.com/0xMiden/miden-client/blob/8f3e8c806187363319b9907814418984528f3c05/bin/miden-cli/src/commands/new_account.rs#L457), because there we have all the data necessary (inside Vec<Package>) to create the new component:

fn process_packages(
    packages: Vec<Package>,
    init_storage_data: &InitStorageData,
) -> Result<Vec<AccountComponent>, CliError>;

---

Perhaps a bigger question is whether the modularity of AccountComponent s still makes sense, now that storage slots are no longer indexed.

It would be great to understand the motivation behind the concept of AccountComponents, each with their own storage.

---

cc @igamigo lmk if I missed something (feel free to edit the issue directly)

[igamigo]

I think this accurately describes the main points we discussed.

we have a special component just for this purpose (preference)

Faucets already have a reserved storage slot for issuance tracking (not sure how this is currently implemented with named storage slots BTW), so maybe it could be something similar (so, not really attached to components).

We discussed this approach as an alternative, but now that I think of this, I don’t think the explorer would have a way of making sense of per-component commitments, because the onchain representation of the account no longer recognizes the distinction between the various AccountComponents (TODO is this correct?)

I think you'd have to do something like output the list of commitments when building the account, which is where you do know the list of components that are involved. And then, it should get stored on-chain and retrievable via RPC (but this is also true for the single-commitment approach). Or is there something I'm missing? Then you'd be able to match commitments and show metadata for the correct named storage slots. I'm assuming that the explorer has visibility to named storage slots. If we assume that an account can hold up to 64 components (not sure if this is enforced anywhere of what a sensible limit would be), this would be 2 KB of commitments I believe, so maybe something to keep in mind.

---

I like the idea of adding this "StorageMetadata" component because it makes the feature optional and "native" by default. It also becomes very easy to check whether the account exposes storage metadata by inspecting its interface.

[PhilippGackstatter]

because the onchain representation of the account no longer recognizes the distinction between the various AccountComponents (TODO is this correct?):

Yes, account components are not a protocol-level concept, they only exist as reusable pieces of functionality (with required storage) to make building account code and storage very convenient. And so once an account is built, it is not immediately clear what components were used to build it.

With named storage slots, this changes somewhat, because the slot names should be globally unique, and a registry could enforce this on a first-come, first-served basis. Similar to how crates.io names are registered. If we make the assumption that slot names are unique, then we could reconstruct the individual components from a given account storage by resolving each slot name to its component.

Component Registry

I remember when we implemented account components we were discussing a more general component registry as a future idea, where users could host the account components they authored which consist of MASM code (in packaged form) and the corresponding storage schema. Best case scenario, this means users can pick-and-choose components from the registry and build their account that way. If we had such a component registry, we wouldn't need a schema commitment because the schema would already be part of the component in the registry.

There are probably more challenges with such a more general registry, than I'm thinking of now. One is that we often have accompanying Rust code for a component (think of encoding a string into a token symbol). Maybe this could be shipped as a conveniently runnable script compiled to WASM, but that's overall a separate challenge.

Faucets already have a reserved storage slot for issuance tracking (not sure how this is currently implemented with named storage slots BTW), so maybe it could be something similar (so, not really attached to components).

I think at some point we discussed making the issuance of a faucet a component-level concern instead of a protocol-level concern. Keeping with that theme, I'd prefer if that "schema commitment component" was a standard instead of a protocol feature, if we end up having it.

[bobbinth]

Given all of the above, I think the best way forward right now is to create a dedicated AccountComponent with a single storage slot that holds per-account schema.

Overall, I agree - though, I'm not sure this would be an AccountComponent because components are defined by their code - and AFAICT, this component would not have any code.

To summarize my thoughts:

1. Once an account is defined, we'd have a single "account-wide" schema for the account. This schema would be built form individual schemas of the components that make up the account.
2. We'd be able to serialize this schema in a deterministic way (a potentially tricky part here would come when we allow account code updates as these may require adding storage slots, and thus, modifying the schema).
3. The commitment to the schema's serialized representation would be stored in a well-named storage slot (e.g., account::schema).
   a. To specify that a given account should have this storage slot populated, we could add something like AccountBuilder::with_schema() method.
4. To display the account storage in the schema-aware way, the user would need to provide the full schema. This would be done differently in different contexts (e.g., would happen automatically in the wallet during account creation, but the schema would need to be uploaded to the block explorer manually).
   a. We could also try to "detect" schemas automatically for some simple components. For example, figuring out that a given account is a basic wallet with RpoFalcon512 auth component, shouldn't be too difficult.
5. One open question is whether the schema should be exported together with the account into account files (probably yes, but maybe this could be optional).

Let me know if I missed anything in the above.

Perhaps a bigger question is whether the modularity of AccountComponent s still makes sense, now that storage slots are no longer indexed.

I think of AccountComponents as a convenience for writing reusable code and constructing accounts. But I think we should use them relatively lightly - i.e., at the protocol level they don't need to exist. I can imagine building accounts in a very different way that doesn't even require the concept of components.

[igamigo]

Overall, I agree - though, I'm not sure this would be an AccountComponent because components are defined by their code - and AFAICT, this component would not have any code.

I was thinking this would have some sort of getter for the component metadata commitment, no? Having a pattern where storage-only components expose getters for their data fits nicely with named storage slots because these are absolute now instead of relative to their own storage segment (and so you can identify them more easily).

Let me know if I missed anything in the above.

One thing that's not clear to me is how/when we can get the "full schema" of the account. Currently, when building an account with multiple components you are aggregating those components into a single AccountCode. However, at that point there is no component metadata involved (eg, you already went from package/component template to just an AccountComponent).
So, assembling the full schema would have to be done separately. Here there would have to be some deterministic ordering so as to make an account schema built from [BasicWalletSchema, RpoFalcon512Schema] be equal to one that's built from [RpoFalcon512Schema, BasicWalletSchema] (although I still think providing per-component schema commitments could be better for UX, though maybe not by much).

Another thing that I'm not sure was discussed is having a way of showing specific types in the explorer in specific ways. For example, you'd probably want to show TokenSymbol as the string instead of the word that you encode it into. For this we could extend TemplateFelt and TemplateWord to have a way of going from Word to String, as currently only the other direction is provided/expected. Then there's the question of how a user can extend this type registry with their own (can it be implemented within a Package?), but that's a bit separate.

[bobbinth]

I was thinking this would have some sort of getter for the component metadata commitment, no? Having a pattern where storage-only components expose getters for their data fits nicely with named storage slots because these are absolute now instead of relative to their own storage segment (and so you can identify them more easily).

Ah yes - good point - having a single getter for such a component would make sense. Though, maybe I'd still treat this component as "special" and use AccountBuilder::with_schema() to indicate that this account should have this component added.

One thing that's not clear to me is how/when we can get the "full schema" of the account. Currently, when building an account with multiple components you are aggregating those components into a single AccountCode. However, at that point there is no component metadata involved (eg, you already went from package/component template to just an AccountComponent).
So, assembling the full schema would have to be done separately. Here there would have to be some deterministic ordering so as to make an account schema built from [BasicWalletSchema, RpoFalcon512Schema] be equal to one that's built from [RpoFalcon512Schema, BasicWalletSchema] (although I still think providing per-component schema commitments could be better for UX, though maybe not by much).

Another option could be to add the schema to AccountComponent - though, I haven't thought through all the pros/cons here.

Another thing that I'm not sure was discussed is having a way of showing specific types in the explorer in specific ways. For example, you'd probably want to show TokenSymbol as the string instead of the word that you encode it into. For this we could extend TemplateFelt and TemplateWord to have a way of going from Word to String, as currently only the other direction is provided/expected. Then there's the question of how a user can extend this type registry with their own (can it be implemented within a Package?), but that's a bit separate.

In the short term, the set of "types" that we can specify would have to be hard-coded. In the longer-term, we could include a WASM module with every package that contains "display" (and probably other) logic for "user-defined" types.

[igamigo]

Currently the AccountStorageSchema contains multiple StorageSlotSchema, each of which keeps default values. If we wanted to commit to the schema, we can either omit the defaults or do something where we take the defaults out of the StorageSlotSchema and we put it on the AccountComponentMetadata level. I think this makes most high-level structs clenaer, because now the data you commit to is well separated from everything else.
For doing the above, we could add an InitStorageData field to AccountComponentMetadata, that keeps all the defaults specified in the TOML. The downside of this is that it complicates (de)serialization because in the description the defaults are put alongside the schema, but there are a couple of ways to deal with this:

• We can add intermediate structures to deal with the ambiguity (this probably adds a bunch of boilerplate code).
• We can do two passes in deserialization where we first read the schema and then read the InitStorageData of every schema field based on their positioning in the TOML. I think this might be the best option as it will not add too many unnecessary structs and not too much serialization code either, but I'm not entirely sure yet.
• We can make it so that the component metadata TOML specifies default values on a different table, and remove them from the type specifications themselves. I think this is the simplest option as you can just leverage the InitStorageData::from_toml code for it, but it makes the TOML itself less readable (i.e., it's easier to see the defaults right next to the slots instead of linking them through their identifier separately).

I'm working on option 2 to see how it goes and should have a draft PR soon. Then, there's the question of how you actually commit to the type specification. For now, I'm just converting the types to bytes and hashing that but there's probably more that can be done. Lastly, a separate PR would add the storage commitment component.