feat: verify range proofs

[demosdemon]

Important

This is a work in progress and not yet complete.

Concepts

This section briefly covers some concepts related to Firewood, Ethereum, Key Proofs, and Range Proofs that provide additional context for this change. Some
concepts apply to Change Proofs as well, but this document isn't focused on Change Proofs.

Branching Factor

The branching factor for the trie determines how we split the path when constructing trie nodes as well as how many children are possible for each node.

The default in firewood is a branching factor of 16 and additionally supports a branching factor of 256 with the branch_factor_256 cargo feature. This means each node can have up to 16 or 256 children, respectively.

Path and Path Components

Generically, a trie path is represented by a sequence of path components in big-endian order where a path component is every $\log_2(F)$ bits where $F$ is the branching factor.

Branch Factor 16

With a branching factor of 16, each path component is a 4-bit nibble and a path is a big-endian sequence of 4-bit nibbles. E.g., the hex key 0x1234567890abcdef is broken down into a sequence of nibbles:

<1 2 3 4 5 6 7 8 9 0 a b c d e f>

Conveniently, hexadecimal is encoded the same way and makes it easy to visually follow a hex encoded key through the trie.

Branch Factor 256

With a branching factor of 256, each path component is a full 8-bit byte. E.g., the hex key 0x1234567890abcdef is broken down into a sequence of bytes:

<12 34 56 78 90 ab cd ef>

Trie Bitmap

A trie bitmap is the map of the child's next path component to the child. In rust, we have this represented with an array type alias:

type Children<T> = [Option<T>; MAX_CHILDREN]

Where Children is a type alias for the array of optional T values. MAX_CHILDREN is defined by the branching factor. The default branching factor of 16 contains 16 children and when branching factor of 256 is enabled, the children array is for 256 elements.

The array itself is generic over the contents. This is typically the child node (e.g., Box<Self>) which forms a very simple trie structure:

struct TrieNode<V> {
    partial_path: Path,
    value: Option<V>,
    children: Children<Box<Self>>,
}

However, the element in the children array can be more complex and form a graph edge with additional information. For example, we can encode the merklization of the trie by including the hash of the child:

struct TrieNode<V> {
    partial_path: Path,
    value: Option<V>,
    children: Children<MerkleEdge<Box<Self>>>,
}

struct MerkleEdge<N> {
    hash: HashType,
    child: N,
}

Key Proofs

Key proofs within the Firewood PMT are represented by a sequence of nodes that follow the path from the root to the target key. If the key is included in the trie, the path terminates on the respective node with a value. If the key is not included in the trie, the path terminates on the node that would be the parent of the node for the key, if it were present. Additionally, each node along the path includes its partial path, value ¹, and a trie bitmap of hashes corresponding to each child.

For example, an inclusion proof for the for the key 89d6aaba59f247bcd526ea4631d82646939dea4f2f39bcf57a1c5f60ec4ca03f in some theoretical trie would be the following sequence of nodes.

|----------------------------------------------------------------------------|
| FullPath- <>                                                               |
| PP      - <>                                                               |
| Value   - null                                                             |
| Child 0 - df67b6a4f29d1df6db91f94743219c00a698bdfaa0513cb10f4c79683dbed916 |
| Child 1 - 49489cb4f7ad7178593ec42da89833088ecde27358e21e1e709af46e2eacd543 |
| Child 4 - d9ca51d8a2591ffe36eb56090d16e00ac560244991bc032e3a52d9400d7ae5f1 |
| Child 8 - 2f97cc62c53a38b2340ec0d2f7e803e734e71bd9f7454f8f53a4bb9a9ea767f4 |
| Child 9 - 7c0de170e80158ef0a5e158da486ed01f67829ad5109ed3cedfc3b0a57354d11 |
| Child d - 6608247446c8bfc5eb8ccbc96141f94a0c9520e42f32f324b847270602071008 |
|----------------------------------------------------------------------------|

|----------------------------------------------------------------------------|
| FullPath- <89>                                                             |
| PP      - <9>                                                              |
| Value   - null                                                             |
| Child 3 - 32a21e08b17e5e97eac3f5f189102ff8d91b8ae485b1df71a6bd9bb992162708 |
| Child 4 - 7e92c8b0b81338f7919dbd195e795b1aa8d02c67d88969fb46008af4d4c6378d |
| Child 9 - 692b498f8f4f6dc57f87308220588d97e1a9223d16c2031117caa98f10b056b3 |
| Child d - cd7818078ee28dac92ecf47fbac68831bfc0e91e388d55cc53824c6edec4602b |
|----------------------------------------------------------------------------|

|------------------------------------------------------------------------------|
| FullPath- <89d6aaba59f247bcd526ea4631d82646939dea4f2f39bcf57a1c5f60ec4ca03f> |
| PP      - <6aaba59f247bcd526ea4631d82646939dea4f2f39bcf57a1c5f60ec4ca03f>    |
| Value   - <abc1234>                                                          |
|------------------------------------------------------------------------------|

Diagraming this trie looks like:

graph TD
    root --df67b6a...--o node-0(((0)))
    root --49489cb...--o node-1(((1)))
    root --d9ca51d...--o node-4(((4)))
    root --2f97cc6...--> node-8[8<br><b>Partial Path:</b> 9]
    root --7c0de17...--o node-9(((9)))
    root --6608247...--o node-d(((d)))

    node-8 --32a21e0...--o node-893(((3)))
    node-8 --7e92c8b...--o node-894(((4)))
    node-8 --692b498...--o node-899(((9)))
    node-8 --cd78180...--> node-89d[d<br><b>Partial Path:</b> 6aaba59f247bcd526ea4631d82646939dea4f2f39bcf57a1c5f60ec4ca03f<br><b>Value:</b> abc123]

Loading

Like inclusion proofs, an exclusion proof includes every node in the trie along the path to the disproven key. The absence of a node where there would be if the child existed is the necessary proof that the key does not exist (along with a valid hash of that node).

Using the same proof from above as an example, it includes all the necessary information to disprove the existance of any key in the trie beginning with the following prefixes (exhaustive) https://gist.github.com/demosdemon/f2593327d34075aa904c89e784b6f122#file-excluded_keys-txt.

A reference python script to generate an exhaustive list of excluded key prefixes is also included in the gist.

Non Exhaustive:

2
3
5
6
7
80
81
82
83
84
85
86
87
88
8a
8b
8c
8d
8e
8f
890
891
892
895
896
897
898
89a
89b
89c
... # omitted keys with the 89d prefix
89e
89f
a
b
c
e
f

Range Proof

A range proof includes the necessary information to prove the existence or absence of every key in the range from start to end. This currently consists of a key proof for the start and end of the range as well as every key and value ² within the range.

For the totally unbounded range ³, the range proof is the trie itself. In practice, a range proof over the total range is unlikely to yield the total trie due to enforced limits on payload sizes.

For bounded ranges, a key proof is provided for start and end range. As a special note, bounded ranges may have an unbounded start and bounded end, in which case no proof is provided for start. This is because the proof of an unbounded start is simply the proof of the empty key which is automatically included in every proof as it will always be a single proof node over the root node. The converse is true as well; a bounded range may consist of a bounded start and unbounded end where no proof is provided for the end.

For truncated ranges, the end proof is the final key in the truncated range and not the end of the desired range. A proof beyond the final provided key is only valid if there are no keys in the range from the final key to the proven key (in which case, the range was not truncated).

Footnotes

1. Or the digest of the value; depending on the trie hashing algorithm. The MerkleDB compatible hash includes only the sha256 digest of the value if the value is 32 bytes or longer. However, the Ethereum compatible hash includes the full value of the node. In other words, the proof node includes that value that is part of the preimage. ↩

2. Unlike proof nodes, range nodes include the complete value every time. ↩

3. Using total ordering, a totally unbounded range for the PMT is a range beginning with the empty key and ending with a key of infinite length. ↩