Implement the history overlay mechanism

[iamrecursion]

This mechanism is intended to be used for the forest implementation with pluggable storage backends. This forest will eventually replace the existing in-memory forest implementation, but for now is being developed in a separate portion of the tree to avoid breakage. We are aiming to merge portions of it incrementally to avoid mammoth PRs and increase visibility, and this mechanism being relatively self-contained makes it a good candidate for this.

This history mechanism is based on the following simplifying assumptions:

• A forest is generally used to store multiple independent trees and versions of each tree, but does not need to share between those independent trees.
• Historical versions are not important to persist, and can hence be handled entirely in memory.

The commit contains the basic functionality of this history storage and overlay mechanism, and tests that it functions correctly. It is almost entirely internal code, with little change to the public interface, and as a result temporarily uses #[allow(dead_code)] to keep the build warnings clean. These will be removed in future commits.

The interface for the History in this commit is a best effort at the design, and will likely need to undergo some minor changes to support concurrency as the shape of said concurrency in the forest itself is established.

This is the first bit of work toward #670. I would argue that this does not require a changelog entry as it is not user-facing.

Checklist before requesting a review

• Repo forked and branch created from next according to naming convention.
• Commit messages and codestyle follow conventions.
• Relevant issues are linked in the PR description.
• Tests added for new functionality.
• Documentation/comments updated according to changes.

[bobbinth]

Looks good! Thank you! Not a full review, but I left some comments inline.

Overall, the structure looks good - but we may want to introduce the notion of a version where each historical delta is associated with a numerical version. This should simplify code a bit and would also align better with the use case we have in miden-node.

Also, it would be great if we could use History for AccountTreeWithHistory. This way, we'd reduce the amount of code duplication. I think the interfaces are pretty close, but we should try to make sure that they are indeed aligned.

[bobbinth]

A couple of potential changes to this API:

• We could take a MutationSet instead of root, nodes, and leaves as for our specific use case, updates will be generated via mutation sets.
• I would also pass in a version so that we could associate a given mutation set with a version. This should make pruning of history a bit easier in the future.

Combining the above, I think the signature could look something like this:

pub fn add_version(
    &mut self,
    version: u32,
    updates: MutationSet<SMT_DEPTH, Word, Word>,
) -> Option<Delta> {
    ...
}

[iamrecursion]

Okay, I have pushed a new commit that addresses all of the feedback except the one comment that I have left unresolved. What follows is a summary of this discussion on Slack.

It seems to me—and I may well be missing something—that we cannot construct the reverse delta information passed to add_version with the forward delta information in MutationSet. That type does not contain the previous value for any of the changes, which in the case of overwrites (of both nodes and leaves) makes it insufficient as a source to construct the reverse delta information.

Note that the History is a container purely for the overlays, and does not have any notion of the tree that it is overlaying.

If we want to make use of MutationSet there are three options in my mind:

1. Add the previous values to MutationSet at a memory cost, but this has good safety as the MutationSet is only constructed in one place and could store the prior values in the same traversal.
2. Require taking the MutationSet and an impl SparseMerkleTree. This is footgunny as you have to check at runtime whether the tree and the set actually correspond. I also have concerns whether implementing SparseMerkleTree on the individual trees in the forest's Storage will be able to be done performantly.
3. Require taking the MutationSet and some auxiliary data that boils down to the reversals only where necessary, but this is very similar to the existing API.

If all you meant was to provide a MutationSet API for the forest itself, this is a separate concern than the API to history, and is definitely planned. The history—and hence calls to things like add_version—are intended to be internal to the tree or forest employing the type, and not client-facing of the tree or forest.

[bobbinth]

It seems to me—and I may well be missing something—that we cannot construct the reverse delta information passed to add_version with the forward delta information in MutationSet. That type does not contain the previous value for any of the changes, which in the case of overwrites (of both nodes and leaves) makes it insufficient as a source to construct the reverse delta information.

On Smt and LargeSmt we have methods for this - specifically, apply_mutations_with_reversion(). This method applies the mutations to the underlying tree and returns a mutation set that can be reversed the applies mutations. Whatever data structure we end up using to manage the latest versions of the trees would need to provide similar functionality. Or do you think that could be a problem?

[iamrecursion]

Not at all. The tree or forest—in other words the user-facing construct—will be able to perform this. I just don't think it's something that should live in the history manager itself.

Do you still want the API to talk in terms of MutationSet and assume it is the 'reversed' set, or can we leave it as is for now and make this a future improvement?

[bobbinth]

Looks good! Than you! Not a full review but I left some comments inline.

[bobbinth]

Looks good! Thank you! I left a few small comments inline.

[drahnr]

Two questions, otherwise LGTM

Thank you!