From dfb073f784670115adf9a1cdf205aab5aa0b5346 Mon Sep 17 00:00:00 2001 From: "Augusto F. Hack" Date: Wed, 22 Feb 2023 14:29:02 +0100 Subject: [PATCH 01/26] docs: mention tree form order of NodeIndex docs --- src/merkle/index.rs | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/merkle/index.rs b/src/merkle/index.rs index 95d7123..1b47aab 100644 --- a/src/merkle/index.rs +++ b/src/merkle/index.rs @@ -4,6 +4,9 @@ use super::{Felt, MerkleError, RpoDigest, StarkField}; // ================================================================================================ /// A Merkle tree address to an arbitrary node. +/// +/// The position is relative to a tree in level order, where for a given depth `d` elements are +/// numbered from $0..2^d$. #[derive(Debug, Default, Copy, Clone, Eq, PartialEq, PartialOrd, Ord, Hash)] pub struct NodeIndex { depth: u8, From efa39e5ce0fe437a7371b5df929b6e7d2225374f Mon Sep 17 00:00:00 2001 From: "Augusto F. Hack" Date: Wed, 1 Mar 2023 17:45:33 +0100 Subject: [PATCH 02/26] feat: added pre-commit hook config --- .pre-commit-config.yaml | 43 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 .pre-commit-config.yaml diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 0000000..d00cf26 --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,43 @@ +# See https://pre-commit.com for more information +# See https://pre-commit.com/hooks.html for more hooks +repos: +- repo: https://github.com/pre-commit/pre-commit-hooks + rev: v3.2.0 + hooks: + - id: trailing-whitespace + - id: end-of-file-fixer + - id: check-yaml + - id: check-json + - id: check-toml + - id: pretty-format-json + - id: check-added-large-files + - id: check-case-conflict + - id: check-executables-have-shebangs + - id: check-merge-conflict + - id: detect-private-key +- repo: https://github.com/hackaugusto/pre-commit-cargo + rev: v1.0.0 + hooks: + # Allows cargo fmt to modify the source code prior to the commit + - id: cargo + name: Cargo fmt + args: ["+stable", "fmt", "--all"] + stages: [commit] + # Requires code to be properly formatted prior to pushing upstream + - id: cargo + name: Cargo fmt --check + args: ["+stable", "fmt", "--all", "--check"] + stages: [push, manual] + - id: cargo + name: Cargo check --all-targets + args: ["+stable", "check", "--all-targets"] + - id: cargo + name: Cargo check --all-targets --no-default-features + args: ["+stable", "check", "--all-targets", "--no-default-features"] + - id: cargo + name: Cargo check --all-targets --all-features + args: ["+stable", "check", "--all-targets", "--all-features"] + # Unlike fmt, clippy will not be automatically applied + - id: cargo + name: Cargo clippy + args: ["+nightly", "clippy", "--workspace", "--", "--deny", "clippy::all", "--deny", "warnings"] From 956e4c6fad779ef15eaa27702b26f05f65d31494 Mon Sep 17 00:00:00 2001 From: "Augusto F. Hack" Date: Wed, 1 Mar 2023 17:45:57 +0100 Subject: [PATCH 03/26] chore: initial run pre-commit --- .github/pull_request_template.md | 2 +- CONTRIBUTING.md | 2 +- benches/README.md | 6 +++--- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 8dfedbe..ffd0f23 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -6,4 +6,4 @@ - Commit messages and codestyle follow [conventions](./CONTRIBUTING.md). - Relevant issues are linked in the PR description. - Tests added for new functionality. -- Documentation/comments updated according to changes. \ No newline at end of file +- Documentation/comments updated according to changes. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7c67357..47a46eb 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -17,7 +17,7 @@ We are using [Github Flow](https://docs.github.com/en/get-started/quickstart/git ### Branching - The current active branch is `next`. Every branch with a fix/feature must be forked from `next`. -- The branch name should contain a short issue/feature description separated with hyphens [(kebab-case)](https://en.wikipedia.org/wiki/Letter_case#Kebab_case). +- The branch name should contain a short issue/feature description separated with hyphens [(kebab-case)](https://en.wikipedia.org/wiki/Letter_case#Kebab_case). For example, if the issue title is `Fix functionality X in component Y` then the branch name will be something like: `fix-x-in-y`. diff --git a/benches/README.md b/benches/README.md index 059c392..9e14f78 100644 --- a/benches/README.md +++ b/benches/README.md @@ -1,4 +1,4 @@ -# Miden VM Hash Functions +# Miden VM Hash Functions In the Miden VM, we make use of different hash functions. Some of these are "traditional" hash functions, like `BLAKE3`, which are optimized for out-of-STARK performance, while others are algebraic hash functions, like `Rescue Prime`, and are more optimized for a better performance inside the STARK. In what follows, we benchmark several such hash functions and compare against other constructions that are used by other proving systems. More precisely, we benchmark: * **BLAKE3** as specified [here](https://github.com/BLAKE3-team/BLAKE3-specs/blob/master/blake3.pdf) and implemented [here](https://github.com/BLAKE3-team/BLAKE3) (with a wrapper exposed via this crate). @@ -13,7 +13,7 @@ In the Miden VM, we make use of different hash functions. Some of these are "tra We benchmark the above hash functions using two scenarios. The first is a 2-to-1 $(a,b)\mapsto h(a,b)$ hashing where both $a$, $b$ and $h(a,b)$ are the digests corresponding to each of the hash functions. The second scenario is that of sequential hashing where we take a sequence of length $100$ field elements and hash these to produce a single digest. The digests are $4$ field elements in a prime field with modulus $2^{64} - 2^{32} + 1$ (i.e., 32 bytes) for Poseidon, Rescue Prime and RPO, and an array `[u8; 32]` for SHA3 and BLAKE3. -#### Scenario 1: 2-to-1 hashing `h(a,b)` +#### Scenario 1: 2-to-1 hashing `h(a,b)` | Function | BLAKE3 | SHA3 | Poseidon | Rp64_256 | RPO_256 | | ------------------- | ------ | --------| --------- | --------- | ------- | @@ -46,4 +46,4 @@ To run the benchmarks for Rescue Prime, Poseidon and SHA3, clone the following [ ``` cargo bench hash -``` \ No newline at end of file +``` From bc12fcafe992e42727a41a8c45b2d6c5e96c1bc5 Mon Sep 17 00:00:00 2001 From: "Augusto F. Hack" Date: Wed, 1 Mar 2023 18:32:24 +0100 Subject: [PATCH 04/26] chore: ignore pre-commit rev --- .git-blame-ignore-revs | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 .git-blame-ignore-revs diff --git a/.git-blame-ignore-revs b/.git-blame-ignore-revs new file mode 100644 index 0000000..a0350b9 --- /dev/null +++ b/.git-blame-ignore-revs @@ -0,0 +1,2 @@ +# initial run of pre-commit +956e4c6fad779ef15eaa27702b26f05f65d31494 From 32d37f1591a1df809e9f0f06b4ea0cb9c15ac116 Mon Sep 17 00:00:00 2001 From: "Augusto F. Hack" Date: Sat, 18 Feb 2023 02:24:49 +0100 Subject: [PATCH 05/26] feat: merkle mountain range --- src/merkle/mmr/accumulator.rs | 44 ++++ src/merkle/mmr/bit.rs | 46 ++++ src/merkle/mmr/full.rs | 299 ++++++++++++++++++++++ src/merkle/mmr/mod.rs | 15 ++ src/merkle/mmr/proof.rs | 33 +++ src/merkle/mmr/tests.rs | 440 +++++++++++++++++++++++++++++++++ src/merkle/mod.rs | 5 + src/merkle/simple_smt/tests.rs | 7 +- 8 files changed, 883 insertions(+), 6 deletions(-) create mode 100644 src/merkle/mmr/accumulator.rs create mode 100644 src/merkle/mmr/bit.rs create mode 100644 src/merkle/mmr/full.rs create mode 100644 src/merkle/mmr/mod.rs create mode 100644 src/merkle/mmr/proof.rs create mode 100644 src/merkle/mmr/tests.rs diff --git a/src/merkle/mmr/accumulator.rs b/src/merkle/mmr/accumulator.rs new file mode 100644 index 0000000..f27355f --- /dev/null +++ b/src/merkle/mmr/accumulator.rs @@ -0,0 +1,44 @@ +use super::{super::Vec, MmrProof, Rpo256, Word}; + +#[derive(Debug, Clone, PartialEq)] +pub struct MmrPeaks { + /// The number of leaves is used to differentiate accumulators that have the same number of + /// peaks. This happens because the number of peaks goes up-and-down as the structure is used + /// causing existing trees to be merged and new ones to be created. As an example, every time + /// the MMR has a power-of-two number of leaves there is a single peak. + /// + /// Every tree in the MMR forest has a distinct power-of-two size, this means only the right + /// most tree can have an odd number of elements (1). Additionally this means that the bits in + /// `num_leaves` conveniently encode the size of each individual tree. + /// + /// Examples: + /// + /// Example 1: With 5 leaves, the binary 0b101. The number of set bits is equal the number + /// of peaks, in this case there are 2 peaks. The 0-indexed least-significant position of + /// the bit determines the number of elements of a tree, so the rightmost tree has 2**0 + /// elements and the left most has 2**2. + /// + /// Example 2: With 12 leaves, the binary is 0b1100, this case also has 2 peaks, the + /// leftmost tree has 2**3=8 elements, and the right most has 2**2=4 elements. + pub num_leaves: usize, + + /// All the peaks of every tree in the MMR forest. The peaks are always ordered by number of + /// leaves, starting from the peak with most children, to the one with least. + /// + /// Invariant: The length of `peaks` must be equal to the number of true bits in `num_leaves`. + pub peaks: Vec, +} + +impl MmrPeaks { + /// Hashes the peaks sequentially, compacting it to a single digest + pub fn hash_peaks(&self) -> Word { + Rpo256::hash_elements(&self.peaks.as_slice().concat()).into() + } + + pub fn verify(&self, value: Word, opening: MmrProof) -> bool { + let root = &self.peaks[opening.peak_index()]; + opening + .merkle_path + .verify(opening.relative_pos() as u64, value, root) + } +} diff --git a/src/merkle/mmr/bit.rs b/src/merkle/mmr/bit.rs new file mode 100644 index 0000000..85376ff --- /dev/null +++ b/src/merkle/mmr/bit.rs @@ -0,0 +1,46 @@ +/// Iterate over the bits of a `usize` and yields the bit positions for the true bits. +pub struct TrueBitPositionIterator { + value: usize, +} + +impl TrueBitPositionIterator { + pub fn new(value: usize) -> TrueBitPositionIterator { + TrueBitPositionIterator { value } + } +} + +impl Iterator for TrueBitPositionIterator { + type Item = u32; + + fn next(&mut self) -> Option<::Item> { + // trailing_zeros is computed with the intrinsic cttz. [Rust 1.67.0] x86 uses the `bsf` + // instruction. AArch64 uses the `rbit clz` instructions. + let zeros = self.value.trailing_zeros(); + + if zeros == usize::BITS { + None + } else { + let bit_position = zeros; + let mask = 1 << bit_position; + self.value ^= mask; + Some(bit_position) + } + } +} + +impl DoubleEndedIterator for TrueBitPositionIterator { + fn next_back(&mut self) -> Option<::Item> { + // trailing_zeros is computed with the intrinsic ctlz. [Rust 1.67.0] x86 uses the `bsr` + // instruction. AArch64 uses the `clz` instruction. + let zeros = self.value.leading_zeros(); + + if zeros == usize::BITS { + None + } else { + let bit_position = usize::BITS - zeros - 1; + let mask = 1 << bit_position; + self.value ^= mask; + Some(bit_position) + } + } +} diff --git a/src/merkle/mmr/full.rs b/src/merkle/mmr/full.rs new file mode 100644 index 0000000..92c350c --- /dev/null +++ b/src/merkle/mmr/full.rs @@ -0,0 +1,299 @@ +//! A fully materialized Merkle mountain range (MMR). +//! +//! A MMR is a forest structure, i.e. it is an ordered set of disjoint rooted trees. The trees are +//! ordered by size, from the most to least number of leaves. Every tree is a perfect binary tree, +//! meaning a tree has all its leaves at the same depth, and every inner node has a branch-factor +//! of 2 with both children set. +//! +//! Additionally the structure only supports adding leaves to the right-most tree, the one with the +//! least number of leaves. The structure preserves the invariant that each tree has different +//! depths, i.e. as part of adding adding a new element to the forest the trees with same depth are +//! merged, creating a new tree with depth d+1, this process is continued until the property is +//! restabilished. +use super::bit::TrueBitPositionIterator; +use super::{super::Vec, MmrPeaks, MmrProof, Rpo256, Word}; +use crate::merkle::MerklePath; +use core::fmt::{Display, Formatter}; + +#[cfg(feature = "std")] +use std::error::Error; + +// MMR +// =============================================================================================== + +/// A fully materialized Merkle Mountain Range, with every tree in the forest and all their +/// elements. +/// +/// Since this is a full representation of the MMR, elements are never removed and the MMR will +/// grow roughly `O(2n)` in number of leaf elements. +pub struct Mmr { + /// Refer to the `forest` method documentation for details of the semantics of this value. + pub(super) forest: usize, + + /// Contains every element of the forest. + /// + /// The trees are in postorder sequential representation. This representation allows for all + /// the elements of every tree in the forest to be stored in the same sequential buffer. It + /// also means new elements can be added to the forest, and merging of trees is very cheap with + /// no need to copy elements. + pub(super) nodes: Vec, +} + +#[derive(Debug, PartialEq, Eq, Copy, Clone)] +pub enum MmrError { + InvalidPosition(usize), +} + +impl Display for MmrError { + fn fmt(&self, fmt: &mut Formatter<'_>) -> Result<(), core::fmt::Error> { + match self { + MmrError::InvalidPosition(pos) => write!(fmt, "Mmr does not contain position {pos}"), + } + } +} + +#[cfg(feature = "std")] +impl Error for MmrError {} + +impl Default for Mmr { + fn default() -> Self { + Self::new() + } +} + +impl Mmr { + // CONSTRUCTORS + // ============================================================================================ + + /// Constructor for an empty `Mmr`. + pub fn new() -> Mmr { + Mmr { + forest: 0, + nodes: Vec::new(), + } + } + + // ACCESSORS + // ============================================================================================ + + /// Returns the MMR forest representation. + /// + /// The forest value has the following interpretations: + /// - its value is the number of elements in the forest + /// - bit count corresponds to the number of trees in the forest + /// - each true bit position determines the depth of a tree in the forest + pub const fn forest(&self) -> usize { + self.forest + } + + // FUNCTIONALITY + // ============================================================================================ + + /// Given a leaf position, returns the Merkle path to its corresponding peak. If the position + /// is greater-or-equal than the tree size an error is returned. + /// + /// Note: The leaf position is the 0-indexed number corresponding to the order the leaves were + /// added, this corresponds to the MMR size _prior_ to adding the element. So the 1st element + /// has position 0, the second position 1, and so on. + pub fn open(&self, pos: usize) -> Result { + // find the target tree responsible for the MMR position + let tree_bit = + leaf_to_corresponding_tree(pos, self.forest).ok_or(MmrError::InvalidPosition(pos))?; + let forest_target = 1usize << tree_bit; + + // isolate the trees before the target + let forest_before = self.forest & high_bitmask(tree_bit + 1); + let index_offset = nodes_in_forest(forest_before); + + // find the root + let index = nodes_in_forest(forest_target) - 1; + + // update the value position from global to the target tree + let relative_pos = pos - forest_before; + + // collect the path and the final index of the target value + let (_, path) = + self.collect_merkle_path_and_value(tree_bit, relative_pos, index_offset, index); + + Ok(MmrProof { + forest: self.forest, + position: pos, + merkle_path: MerklePath::new(path), + }) + } + + /// Returns the leaf value at position `pos`. + /// + /// Note: The leaf position is the 0-indexed number corresponding to the order the leaves were + /// added, this corresponds to the MMR size _prior_ to adding the element. So the 1st element + /// has position 0, the second position 1, and so on. + pub fn get(&self, pos: usize) -> Result { + // find the target tree responsible for the MMR position + let tree_bit = + leaf_to_corresponding_tree(pos, self.forest).ok_or(MmrError::InvalidPosition(pos))?; + let forest_target = 1usize << tree_bit; + + // isolate the trees before the target + let forest_before = self.forest & high_bitmask(tree_bit + 1); + let index_offset = nodes_in_forest(forest_before); + + // find the root + let index = nodes_in_forest(forest_target) - 1; + + // update the value position from global to the target tree + let relative_pos = pos - forest_before; + + // collect the path and the final index of the target value + let (value, _) = + self.collect_merkle_path_and_value(tree_bit, relative_pos, index_offset, index); + + Ok(value) + } + + /// Adds a new element to the MMR. + pub fn add(&mut self, el: Word) { + // Note: every node is also a tree of size 1, adding an element to the forest creates a new + // rooted-tree of size 1. This may temporarily break the invariant that every tree in the + // forest has different sizes, the loop below will eagerly merge trees of same size and + // restore the invariant. + self.nodes.push(el); + + let mut left_offset = self.nodes.len().saturating_sub(2); + let mut right = el; + let mut left_tree = 1; + while self.forest & left_tree != 0 { + right = *Rpo256::merge(&[self.nodes[left_offset].into(), right.into()]); + self.nodes.push(right); + + left_offset = left_offset.saturating_sub(nodes_in_forest(left_tree)); + left_tree <<= 1; + } + + self.forest += 1; + } + + /// Returns an accumulator representing the current state of the MMMR. + pub fn accumulator(&self) -> MmrPeaks { + let peaks: Vec = TrueBitPositionIterator::new(self.forest) + .rev() + .map(|bit| nodes_in_forest(1 << bit)) + .scan(0, |offset, el| { + *offset += el; + Some(*offset) + }) + .map(|offset| self.nodes[offset - 1]) + .collect(); + + MmrPeaks { + num_leaves: self.forest, + peaks, + } + } + + // UTILITIES + // ============================================================================================ + + /// Internal function used to collect the Merkle path of a value. + fn collect_merkle_path_and_value( + &self, + tree_bit: u32, + relative_pos: usize, + index_offset: usize, + mut index: usize, + ) -> (Word, Vec) { + // collect the Merkle path + let mut tree_depth = tree_bit as usize; + let mut path = Vec::with_capacity(tree_depth + 1); + while tree_depth > 0 { + let bit = relative_pos & tree_depth; + let right_offset = index - 1; + let left_offset = right_offset - nodes_in_forest(tree_depth); + + // Elements to the right have a higher position because they were + // added later. Therefore when the bit is true the node's path is + // to the right, and its sibling to the left. + let sibling = if bit != 0 { + index = right_offset; + self.nodes[index_offset + left_offset] + } else { + index = left_offset; + self.nodes[index_offset + right_offset] + }; + + tree_depth >>= 1; + path.push(sibling); + } + + // the rest of the codebase has the elements going from leaf to root, adjust it here for + // easy of use/consistency sake + path.reverse(); + + let value = self.nodes[index_offset + index]; + (value, path) + } +} + +impl From for Mmr +where + T: IntoIterator, +{ + fn from(values: T) -> Self { + let mut mmr = Mmr::new(); + for v in values { + mmr.add(v) + } + mmr + } +} + +// UTILITIES +// =============================================================================================== + +/// Given a 0-indexed leaf position and the current forest, return the tree number responsible for +/// the position. +/// +/// Note: +/// The result is a tree position `p`, it has the following interpretations. $p+1$ is the depth of +/// the tree, which corresponds to the size of a Merkle proof for that tree. $2^p$ is equal to the +/// number of leaves in this particular tree. and $2^(p+1)-1$ corresponds to size of the tree. +pub(crate) const fn leaf_to_corresponding_tree(pos: usize, forest: usize) -> Option { + if pos >= forest { + None + } else { + // - each bit in the forest is a unique tree and the bit position its power-of-two size + // - each tree owns a consecutive range of positions equal to its size from left-to-right + // - this means the first tree owns from `0` up to the `2^k_0` first positions, where `k_0` + // is the highest true bit position, the second tree from `2^k_0 + 1` up to `2^k_1` where + // `k_1` is the second higest bit, so on. + // - this means the highest bits work as a category marker, and the position is owned by + // the first tree which doesn't share a high bit with the position + let before = forest & pos; + let after = forest ^ before; + let tree = after.ilog2(); + + Some(tree) + } +} + +/// Return a bitmask for the bits including and above the given position. +pub(crate) const fn high_bitmask(bit: u32) -> usize { + if bit > usize::BITS - 1 { + 0 + } else { + usize::MAX << bit + } +} + +/// Return the total number of nodes of a given forest +/// +/// Panics: +/// +/// This will panic if the forest has size greater than `usize::MAX / 2` +pub(crate) const fn nodes_in_forest(forest: usize) -> usize { + // - the size of a perfect binary tree is $2^{k+1}-1$ or $2*2^k-1$ + // - the forest represents the sum of $2^k$ so a single multiplication is necessary + // - the number of `-1` is the same as the number of trees, which is the same as the number + // bits set + let tree_count = forest.count_ones() as usize; + forest * 2 - tree_count +} diff --git a/src/merkle/mmr/mod.rs b/src/merkle/mmr/mod.rs new file mode 100644 index 0000000..d8903ca --- /dev/null +++ b/src/merkle/mmr/mod.rs @@ -0,0 +1,15 @@ +mod accumulator; +mod bit; +mod full; +mod proof; + +#[cfg(test)] +mod tests; + +use super::{Rpo256, Word}; + +// REEXPORTS +// ================================================================================================ +pub use accumulator::MmrPeaks; +pub use full::Mmr; +pub use proof::MmrProof; diff --git a/src/merkle/mmr/proof.rs b/src/merkle/mmr/proof.rs new file mode 100644 index 0000000..0904b83 --- /dev/null +++ b/src/merkle/mmr/proof.rs @@ -0,0 +1,33 @@ +/// The representation of a single Merkle path. +use super::super::MerklePath; +use super::full::{high_bitmask, leaf_to_corresponding_tree}; + +#[derive(Debug, Clone, PartialEq)] +pub struct MmrProof { + /// The state of the MMR when the MmrProof was created. + pub forest: usize, + + /// The position of the leaf value on this MmrProof. + pub position: usize, + + /// The Merkle opening, starting from the value's sibling up to and excluding the root of the + /// responsible tree. + pub merkle_path: MerklePath, +} + +impl MmrProof { + /// Converts the leaf global position into a local position that can be used to verify the + /// merkle_path. + pub fn relative_pos(&self) -> usize { + let tree_bit = leaf_to_corresponding_tree(self.position, self.forest) + .expect("position must be part of the forest"); + let forest_before = self.forest & high_bitmask(tree_bit + 1); + self.position - forest_before + } + + pub fn peak_index(&self) -> usize { + let root = leaf_to_corresponding_tree(self.position, self.forest) + .expect("position must be part of the forest"); + (self.forest.count_ones() - root - 1) as usize + } +} diff --git a/src/merkle/mmr/tests.rs b/src/merkle/mmr/tests.rs new file mode 100644 index 0000000..577c6c4 --- /dev/null +++ b/src/merkle/mmr/tests.rs @@ -0,0 +1,440 @@ +use super::bit::TrueBitPositionIterator; +use super::full::{high_bitmask, leaf_to_corresponding_tree, nodes_in_forest}; +use super::{super::Vec, Mmr, Rpo256, Word}; +use crate::merkle::{int_to_node, MerklePath}; + +#[test] +fn test_position_equal_or_higher_than_leafs_is_never_contained() { + let empty_forest = 0; + for pos in 1..1024 { + // pos is index, 0 based + // tree is a length counter, 1 based + // so a valid pos is always smaller, not equal, to tree + assert_eq!(leaf_to_corresponding_tree(pos, pos), None); + assert_eq!(leaf_to_corresponding_tree(pos, pos - 1), None); + // and empty forest has no trees, so no position is valid + assert_eq!(leaf_to_corresponding_tree(pos, empty_forest), None); + } +} + +#[test] +fn test_position_zero_is_always_contained_by_the_highest_tree() { + for leaves in 1..1024usize { + let tree = leaves.ilog2(); + assert_eq!(leaf_to_corresponding_tree(0, leaves), Some(tree)); + } +} + +#[test] +fn test_leaf_to_corresponding_tree() { + assert_eq!(leaf_to_corresponding_tree(0, 0b0001), Some(0)); + assert_eq!(leaf_to_corresponding_tree(0, 0b0010), Some(1)); + assert_eq!(leaf_to_corresponding_tree(0, 0b0011), Some(1)); + assert_eq!(leaf_to_corresponding_tree(0, 0b1011), Some(3)); + + // position one is always owned by the left-most tree + assert_eq!(leaf_to_corresponding_tree(1, 0b0010), Some(1)); + assert_eq!(leaf_to_corresponding_tree(1, 0b0011), Some(1)); + assert_eq!(leaf_to_corresponding_tree(1, 0b1011), Some(3)); + + // position two starts as its own root, and then it is merged with the left-most tree + assert_eq!(leaf_to_corresponding_tree(2, 0b0011), Some(0)); + assert_eq!(leaf_to_corresponding_tree(2, 0b0100), Some(2)); + assert_eq!(leaf_to_corresponding_tree(2, 0b1011), Some(3)); + + // position tree is merged on the left-most tree + assert_eq!(leaf_to_corresponding_tree(3, 0b0011), None); + assert_eq!(leaf_to_corresponding_tree(3, 0b0100), Some(2)); + assert_eq!(leaf_to_corresponding_tree(3, 0b1011), Some(3)); + + assert_eq!(leaf_to_corresponding_tree(4, 0b0101), Some(0)); + assert_eq!(leaf_to_corresponding_tree(4, 0b0110), Some(1)); + assert_eq!(leaf_to_corresponding_tree(4, 0b0111), Some(1)); + assert_eq!(leaf_to_corresponding_tree(4, 0b1000), Some(3)); + + assert_eq!(leaf_to_corresponding_tree(12, 0b01101), Some(0)); + assert_eq!(leaf_to_corresponding_tree(12, 0b01110), Some(1)); + assert_eq!(leaf_to_corresponding_tree(12, 0b01111), Some(1)); + assert_eq!(leaf_to_corresponding_tree(12, 0b10000), Some(4)); +} + +#[test] +fn test_high_bitmask() { + assert_eq!(high_bitmask(0), usize::MAX); + assert_eq!(high_bitmask(1), usize::MAX << 1); + assert_eq!(high_bitmask(usize::BITS - 2), 0b11usize.rotate_right(2)); + assert_eq!(high_bitmask(usize::BITS - 1), 0b1usize.rotate_right(1)); + assert_eq!(high_bitmask(usize::BITS), 0, "overflow should be handled"); +} + +#[test] +fn test_nodes_in_forest() { + assert_eq!(nodes_in_forest(0b0000), 0); + assert_eq!(nodes_in_forest(0b0001), 1); + assert_eq!(nodes_in_forest(0b0010), 3); + assert_eq!(nodes_in_forest(0b0011), 4); + assert_eq!(nodes_in_forest(0b0100), 7); + assert_eq!(nodes_in_forest(0b0101), 8); + assert_eq!(nodes_in_forest(0b0110), 10); + assert_eq!(nodes_in_forest(0b0111), 11); + assert_eq!(nodes_in_forest(0b1000), 15); + assert_eq!(nodes_in_forest(0b1001), 16); + assert_eq!(nodes_in_forest(0b1010), 18); + assert_eq!(nodes_in_forest(0b1011), 19); +} + +#[test] +fn test_nodes_in_forest_single_bit() { + assert_eq!(nodes_in_forest(2usize.pow(0)), 2usize.pow(1) - 1); + assert_eq!(nodes_in_forest(2usize.pow(1)), 2usize.pow(2) - 1); + assert_eq!(nodes_in_forest(2usize.pow(2)), 2usize.pow(3) - 1); + assert_eq!(nodes_in_forest(2usize.pow(3)), 2usize.pow(4) - 1); + + for bit in 0..(usize::BITS - 1) { + let size = 2usize.pow(bit + 1) - 1; + assert_eq!(nodes_in_forest(1usize << bit), size); + } +} + +const LEAVES: [Word; 7] = [ + int_to_node(0), + int_to_node(1), + int_to_node(2), + int_to_node(3), + int_to_node(4), + int_to_node(5), + int_to_node(6), +]; + +#[test] +fn test_mmr_simple() { + let mut postorder = Vec::new(); + postorder.push(LEAVES[0]); + postorder.push(LEAVES[1]); + postorder.push(*Rpo256::hash_elements(&[LEAVES[0], LEAVES[1]].concat())); + postorder.push(LEAVES[2]); + postorder.push(LEAVES[3]); + postorder.push(*Rpo256::hash_elements(&[LEAVES[2], LEAVES[3]].concat())); + postorder.push(*Rpo256::hash_elements( + &[postorder[2], postorder[5]].concat(), + )); + postorder.push(LEAVES[4]); + postorder.push(LEAVES[5]); + postorder.push(*Rpo256::hash_elements(&[LEAVES[4], LEAVES[5]].concat())); + postorder.push(LEAVES[6]); + + let mut mmr = Mmr::new(); + assert_eq!(mmr.forest(), 0); + assert_eq!(mmr.nodes.len(), 0); + + mmr.add(LEAVES[0]); + assert_eq!(mmr.forest(), 1); + assert_eq!(mmr.nodes.len(), 1); + assert_eq!(mmr.nodes.as_slice(), &postorder[0..mmr.nodes.len()]); + + let acc = mmr.accumulator(); + assert_eq!(acc.num_leaves, 1); + assert_eq!(acc.peaks, &[postorder[0]]); + + mmr.add(LEAVES[1]); + assert_eq!(mmr.forest(), 2); + assert_eq!(mmr.nodes.len(), 3); + assert_eq!(mmr.nodes.as_slice(), &postorder[0..mmr.nodes.len()]); + + let acc = mmr.accumulator(); + assert_eq!(acc.num_leaves, 2); + assert_eq!(acc.peaks, &[postorder[2]]); + + mmr.add(LEAVES[2]); + assert_eq!(mmr.forest(), 3); + assert_eq!(mmr.nodes.len(), 4); + assert_eq!(mmr.nodes.as_slice(), &postorder[0..mmr.nodes.len()]); + + let acc = mmr.accumulator(); + assert_eq!(acc.num_leaves, 3); + assert_eq!(acc.peaks, &[postorder[2], postorder[3]]); + + mmr.add(LEAVES[3]); + assert_eq!(mmr.forest(), 4); + assert_eq!(mmr.nodes.len(), 7); + assert_eq!(mmr.nodes.as_slice(), &postorder[0..mmr.nodes.len()]); + + let acc = mmr.accumulator(); + assert_eq!(acc.num_leaves, 4); + assert_eq!(acc.peaks, &[postorder[6]]); + + mmr.add(LEAVES[4]); + assert_eq!(mmr.forest(), 5); + assert_eq!(mmr.nodes.len(), 8); + assert_eq!(mmr.nodes.as_slice(), &postorder[0..mmr.nodes.len()]); + + let acc = mmr.accumulator(); + assert_eq!(acc.num_leaves, 5); + assert_eq!(acc.peaks, &[postorder[6], postorder[7]]); + + mmr.add(LEAVES[5]); + assert_eq!(mmr.forest(), 6); + assert_eq!(mmr.nodes.len(), 10); + assert_eq!(mmr.nodes.as_slice(), &postorder[0..mmr.nodes.len()]); + + let acc = mmr.accumulator(); + assert_eq!(acc.num_leaves, 6); + assert_eq!(acc.peaks, &[postorder[6], postorder[9]]); + + mmr.add(LEAVES[6]); + assert_eq!(mmr.forest(), 7); + assert_eq!(mmr.nodes.len(), 11); + assert_eq!(mmr.nodes.as_slice(), &postorder[0..mmr.nodes.len()]); + + let acc = mmr.accumulator(); + assert_eq!(acc.num_leaves, 7); + assert_eq!(acc.peaks, &[postorder[6], postorder[9], postorder[10]]); +} + +#[test] +fn test_mmr_open() { + let mmr: Mmr = LEAVES.into(); + let h01: Word = Rpo256::hash_elements(&LEAVES[0..2].concat()).into(); + let h23: Word = Rpo256::hash_elements(&LEAVES[2..4].concat()).into(); + + // node at pos 7 is the root + assert!( + mmr.open(7).is_err(), + "Element 7 is not in the tree, result should be None" + ); + + // node at pos 6 is the root + let empty: MerklePath = MerklePath::new(vec![]); + let opening = mmr + .open(6) + .expect("Element 6 is contained in the tree, expected an opening result."); + assert_eq!(opening.merkle_path, empty); + assert_eq!(opening.forest, mmr.forest); + assert_eq!(opening.position, 6); + assert!( + mmr.accumulator().verify(LEAVES[6], opening), + "MmrProof should be valid for the current accumulator." + ); + + // nodes 4,5 are detph 1 + let root_to_path = MerklePath::new(vec![LEAVES[4]]); + let opening = mmr + .open(5) + .expect("Element 5 is contained in the tree, expected an opening result."); + assert_eq!(opening.merkle_path, root_to_path); + assert_eq!(opening.forest, mmr.forest); + assert_eq!(opening.position, 5); + assert!( + mmr.accumulator().verify(LEAVES[5], opening), + "MmrProof should be valid for the current accumulator." + ); + + let root_to_path = MerklePath::new(vec![LEAVES[5]]); + let opening = mmr + .open(4) + .expect("Element 4 is contained in the tree, expected an opening result."); + assert_eq!(opening.merkle_path, root_to_path); + assert_eq!(opening.forest, mmr.forest); + assert_eq!(opening.position, 4); + assert!( + mmr.accumulator().verify(LEAVES[4], opening), + "MmrProof should be valid for the current accumulator." + ); + + // nodes 0,1,2,3 are detph 2 + let root_to_path = MerklePath::new(vec![LEAVES[2], h01]); + let opening = mmr + .open(3) + .expect("Element 3 is contained in the tree, expected an opening result."); + assert_eq!(opening.merkle_path, root_to_path); + assert_eq!(opening.forest, mmr.forest); + assert_eq!(opening.position, 3); + assert!( + mmr.accumulator().verify(LEAVES[3], opening), + "MmrProof should be valid for the current accumulator." + ); + + let root_to_path = MerklePath::new(vec![LEAVES[3], h01]); + let opening = mmr + .open(2) + .expect("Element 2 is contained in the tree, expected an opening result."); + assert_eq!(opening.merkle_path, root_to_path); + assert_eq!(opening.forest, mmr.forest); + assert_eq!(opening.position, 2); + assert!( + mmr.accumulator().verify(LEAVES[2], opening), + "MmrProof should be valid for the current accumulator." + ); + + let root_to_path = MerklePath::new(vec![LEAVES[0], h23]); + let opening = mmr + .open(1) + .expect("Element 1 is contained in the tree, expected an opening result."); + assert_eq!(opening.merkle_path, root_to_path); + assert_eq!(opening.forest, mmr.forest); + assert_eq!(opening.position, 1); + assert!( + mmr.accumulator().verify(LEAVES[1], opening), + "MmrProof should be valid for the current accumulator." + ); + + let root_to_path = MerklePath::new(vec![LEAVES[1], h23]); + let opening = mmr + .open(0) + .expect("Element 0 is contained in the tree, expected an opening result."); + assert_eq!(opening.merkle_path, root_to_path); + assert_eq!(opening.forest, mmr.forest); + assert_eq!(opening.position, 0); + assert!( + mmr.accumulator().verify(LEAVES[0], opening), + "MmrProof should be valid for the current accumulator." + ); +} + +#[test] +fn test_mmr_get() { + let mmr: Mmr = LEAVES.into(); + assert_eq!( + mmr.get(0).unwrap(), + LEAVES[0], + "value at pos 0 must correspond" + ); + assert_eq!( + mmr.get(1).unwrap(), + LEAVES[1], + "value at pos 1 must correspond" + ); + assert_eq!( + mmr.get(2).unwrap(), + LEAVES[2], + "value at pos 2 must correspond" + ); + assert_eq!( + mmr.get(3).unwrap(), + LEAVES[3], + "value at pos 3 must correspond" + ); + assert_eq!( + mmr.get(4).unwrap(), + LEAVES[4], + "value at pos 4 must correspond" + ); + assert_eq!( + mmr.get(5).unwrap(), + LEAVES[5], + "value at pos 5 must correspond" + ); + assert_eq!( + mmr.get(6).unwrap(), + LEAVES[6], + "value at pos 6 must correspond" + ); + assert!(mmr.get(7).is_err()); +} + +#[test] +fn test_mmr_invariants() { + let mut mmr = Mmr::new(); + for v in 1..=1028 { + mmr.add(int_to_node(v)); + let accumulator = mmr.accumulator(); + assert_eq!( + v as usize, + mmr.forest(), + "MMR leaf count must increase by one on every add" + ); + assert_eq!( + v as usize, accumulator.num_leaves, + "MMR and its accumulator must match leaves count" + ); + assert_eq!( + accumulator.num_leaves.count_ones() as usize, + accumulator.peaks.len(), + "bits on leaves must match the number of peaks" + ); + + let expected_nodes: usize = TrueBitPositionIterator::new(mmr.forest()) + .map(|bit_pos| nodes_in_forest(1 << bit_pos)) + .sum(); + + assert_eq!( + expected_nodes, + mmr.nodes.len(), + "the sum of every tree size must be equal to the number of nodes in the MMR (forest: {:b})", + mmr.forest(), + ); + } +} + +#[test] +fn test_bit_position_iterator() { + assert_eq!(TrueBitPositionIterator::new(0).count(), 0); + assert_eq!(TrueBitPositionIterator::new(0).rev().count(), 0); + + assert_eq!( + TrueBitPositionIterator::new(1).collect::>(), + vec![0] + ); + assert_eq!( + TrueBitPositionIterator::new(1).rev().collect::>(), + vec![0], + ); + + assert_eq!( + TrueBitPositionIterator::new(2).collect::>(), + vec![1] + ); + assert_eq!( + TrueBitPositionIterator::new(2).rev().collect::>(), + vec![1], + ); + + assert_eq!( + TrueBitPositionIterator::new(3).collect::>(), + vec![0, 1], + ); + assert_eq!( + TrueBitPositionIterator::new(3).rev().collect::>(), + vec![1, 0], + ); + + assert_eq!( + TrueBitPositionIterator::new(0b11010101).collect::>(), + vec![0, 2, 4, 6, 7], + ); + assert_eq!( + TrueBitPositionIterator::new(0b11010101) + .rev() + .collect::>(), + vec![7, 6, 4, 2, 0], + ); +} + +mod property_tests { + use super::leaf_to_corresponding_tree; + use proptest::prelude::*; + + proptest! { + #[test] + fn test_last_position_is_always_contained_in_the_last_tree(leaves in any::().prop_filter("cant have an empty tree", |v| *v != 0)) { + let last_pos = leaves - 1; + let lowest_bit = leaves.trailing_zeros(); + + assert_eq!( + leaf_to_corresponding_tree(last_pos, leaves), + Some(lowest_bit), + ); + } + } + + proptest! { + #[test] + fn test_contained_tree_is_always_power_of_two((leaves, pos) in any::().prop_flat_map(|v| (Just(v), 0..v))) { + let tree = leaf_to_corresponding_tree(pos, leaves).expect("pos is smaller than leaves, there should always be a corresponding tree"); + let mask = 1usize << tree; + + assert!(tree < usize::BITS, "the result must be a bit in usize"); + assert!(mask & leaves != 0, "the result should be a tree in leaves"); + } + } +} diff --git a/src/merkle/mod.rs b/src/merkle/mod.rs index 0b82752..6d53a71 100644 --- a/src/merkle/mod.rs +++ b/src/merkle/mod.rs @@ -5,6 +5,8 @@ use super::{ }; use core::fmt; +// REEXPORTS +// ================================================================================================ mod index; pub use index::NodeIndex; @@ -20,6 +22,9 @@ pub use path_set::MerklePathSet; mod simple_smt; pub use simple_smt::SimpleSmt; +mod mmr; +pub use mmr::{Mmr, MmrPeaks}; + // ERRORS // ================================================================================================ diff --git a/src/merkle/simple_smt/tests.rs b/src/merkle/simple_smt/tests.rs index 2096fd1..e449014 100644 --- a/src/merkle/simple_smt/tests.rs +++ b/src/merkle/simple_smt/tests.rs @@ -1,8 +1,7 @@ use super::{ - super::{MerkleTree, RpoDigest, SimpleSmt}, + super::{int_to_node, MerkleTree, RpoDigest, SimpleSmt}, NodeIndex, Rpo256, Vec, Word, }; -use crate::{Felt, FieldElement}; use core::iter; use proptest::prelude::*; use rand_utils::prng_array; @@ -275,7 +274,3 @@ fn compute_internal_nodes() -> (Word, Word, Word) { (root.into(), node2.into(), node3.into()) } - -const fn int_to_node(value: u64) -> Word { - [Felt::new(value), Felt::ZERO, Felt::ZERO, Felt::ZERO] -} From 3a6a4fcce66cb1ff28cc9b737c81c378df039cdd Mon Sep 17 00:00:00 2001 From: Victor Lopez Date: Sun, 19 Feb 2023 18:30:57 +0100 Subject: [PATCH 06/26] feat: refactor simple smt to use empty subtree constants Prior to this commit, there was an internal procedure with the merkle trees to compute empty sub-tree for arbitrary depths. However, this isn't ideal as this code can be reused in any merkle implementation that uses RPO as backend. This commit introduces a structure that will generate these empty subtrees values. --- benches/smt.rs | 2 +- src/hash/rpo/digest.rs | 2 +- src/merkle/empty_roots.rs | 454 +++++++++++++++++++++++++++++++++ src/merkle/mod.rs | 6 +- src/merkle/simple_smt/mod.rs | 125 +++++---- src/merkle/simple_smt/tests.rs | 48 ++-- 6 files changed, 560 insertions(+), 77 deletions(-) create mode 100644 src/merkle/empty_roots.rs diff --git a/benches/smt.rs b/benches/smt.rs index c03431f..a19c4d2 100644 --- a/benches/smt.rs +++ b/benches/smt.rs @@ -18,7 +18,7 @@ fn smt_rpo(c: &mut Criterion) { (i, word) }) .collect(); - let tree = SimpleSmt::new(entries, depth).unwrap(); + let tree = SimpleSmt::new(depth).unwrap().with_leaves(entries).unwrap(); trees.push(tree); } } diff --git a/src/hash/rpo/digest.rs b/src/hash/rpo/digest.rs index 3f56f9c..4af30ec 100644 --- a/src/hash/rpo/digest.rs +++ b/src/hash/rpo/digest.rs @@ -11,7 +11,7 @@ use core::{cmp::Ordering, ops::Deref}; pub struct RpoDigest([Felt; DIGEST_SIZE]); impl RpoDigest { - pub fn new(value: [Felt; DIGEST_SIZE]) -> Self { + pub const fn new(value: [Felt; DIGEST_SIZE]) -> Self { Self(value) } diff --git a/src/merkle/empty_roots.rs b/src/merkle/empty_roots.rs new file mode 100644 index 0000000..8ba6757 --- /dev/null +++ b/src/merkle/empty_roots.rs @@ -0,0 +1,454 @@ +use super::{Felt, RpoDigest, WORD_SIZE, ZERO}; +use core::slice; + +// EMPTY NODES SUBTREES +// ================================================================================================ + +/// Contains precomputed roots of empty subtrees in a Merkle rtee of depth 64. +pub struct EmptySubtreeRoots; + +impl EmptySubtreeRoots { + /// Returns a static slice with roots of empty subtrees of a Merkle tree starting at the + /// specified depth. + /// + /// # Panics + /// + /// This function will panic if the provided `depth` is greater than `64`. + pub const fn empty_hashes(depth: u8) -> &'static [RpoDigest] { + assert!(depth < 65); + let ptr = &EMPTY_SUBTREES_64[64 - depth as usize] as *const RpoDigest; + // Safety: this is a static/constant array, so it will never be outlived. If we attempt to + // use regular slices, this wouldn't be a `const` function, meaning we won't be able to use + // the returned value for static/constant definitions. + unsafe { slice::from_raw_parts(ptr, depth as usize + 1) } + } +} + +const EMPTY_SUBTREES_64: [RpoDigest; 65] = [ + RpoDigest::new([ + Felt::new(15321474589252129342), + Felt::new(17373224439259377994), + Felt::new(15071539326562317628), + Felt::new(3312677166725950353), + ]), + RpoDigest::new([ + Felt::new(12146678323567200178), + Felt::new(14288630174929498478), + Felt::new(13374892366980833045), + Felt::new(11840636859983936891), + ]), + RpoDigest::new([ + Felt::new(15220380953028059006), + Felt::new(2981707349961006045), + Felt::new(7409523958661360004), + Felt::new(2816116826688969892), + ]), + RpoDigest::new([ + Felt::new(7829641133220670678), + Felt::new(6170216088031698405), + Felt::new(11814483661801576435), + Felt::new(1762887097744793975), + ]), + RpoDigest::new([ + Felt::new(1299421782687082884), + Felt::new(9938699043036414489), + Felt::new(10193025806762503939), + Felt::new(12073246492422971113), + ]), + RpoDigest::new([ + Felt::new(3774016405860870757), + Felt::new(2584714598467121158), + Felt::new(7418645462301488344), + Felt::new(1016804897028793820), + ]), + RpoDigest::new([ + Felt::new(13238072489118494737), + Felt::new(6917129315345826393), + Felt::new(13736362398490889690), + Felt::new(4929049375601714136), + ]), + RpoDigest::new([ + Felt::new(2433738165854950976), + Felt::new(6710644905925382197), + Felt::new(10571480102433401045), + Felt::new(16853295309134271298), + ]), + RpoDigest::new([ + Felt::new(3162775558610426184), + Felt::new(11944004899624546116), + Felt::new(55767976185223284), + Felt::new(5892480272697245897), + ]), + RpoDigest::new([ + Felt::new(12582634330812132159), + Felt::new(6886254574119140332), + Felt::new(4407453795368410417), + Felt::new(6959805977831121004), + ]), + RpoDigest::new([ + Felt::new(16001070406220863863), + Felt::new(4426773743735082930), + Felt::new(6860108527212616559), + Felt::new(3994703491288516722), + ]), + RpoDigest::new([ + Felt::new(9755907048710665826), + Felt::new(13697078808748604851), + Felt::new(17210321635283113095), + Felt::new(1203394006092675979), + ]), + RpoDigest::new([ + Felt::new(3332855817731547893), + Felt::new(1068928372599561798), + Felt::new(17119375903210334455), + Felt::new(8148601736624954416), + ]), + RpoDigest::new([ + Felt::new(17265634841675424144), + Felt::new(18322832739735580203), + Felt::new(17896992777163902308), + Felt::new(6189383326950297131), + ]), + RpoDigest::new([ + Felt::new(9329637674239983584), + Felt::new(2512861030579248721), + Felt::new(10833150484884266896), + Felt::new(7470498642428983444), + ]), + RpoDigest::new([ + Felt::new(3611140194800558886), + Felt::new(17185933650781587767), + Felt::new(7835232399818923215), + Felt::new(7974155618002781326), + ]), + RpoDigest::new([ + Felt::new(17483286922353768131), + Felt::new(353378057542380712), + Felt::new(1935183237414585408), + Felt::new(4820339620987989650), + ]), + RpoDigest::new([ + Felt::new(16172462385444809646), + Felt::new(3268597753131435459), + Felt::new(3481491333654579291), + Felt::new(16487779176137683725), + ]), + RpoDigest::new([ + Felt::new(16595012576192613315), + Felt::new(16028552537812484518), + Felt::new(13016887826405546773), + Felt::new(14649690775021494057), + ]), + RpoDigest::new([ + Felt::new(11300236651178143890), + Felt::new(15307634289168527196), + Felt::new(2834866419963148279), + Felt::new(7512874625395280090), + ]), + RpoDigest::new([ + Felt::new(1148273481270068529), + Felt::new(7411276436636897120), + Felt::new(14325955409748352141), + Felt::new(15577038614919538356), + ]), + RpoDigest::new([ + Felt::new(13911627859049081064), + Felt::new(13298542751859672529), + Felt::new(18341014824837028242), + Felt::new(5587966507704160144), + ]), + RpoDigest::new([ + Felt::new(10957185917743597702), + Felt::new(15815185767119166433), + Felt::new(17883994521792846784), + Felt::new(15958104556930886663), + ]), + RpoDigest::new([ + Felt::new(13148367538964199489), + Felt::new(7372139436485928380), + Felt::new(13408383191801051600), + Felt::new(2114382634401123096), + ]), + RpoDigest::new([ + Felt::new(14448157482521530067), + Felt::new(17865161921504959156), + Felt::new(10319385198642448897), + Felt::new(364163501511998552), + ]), + RpoDigest::new([ + Felt::new(9722640569118951143), + Felt::new(16371655672847089887), + Felt::new(12379452272155069993), + Felt::new(11605969747977185617), + ]), + RpoDigest::new([ + Felt::new(2782512273606877924), + Felt::new(3656296563981095117), + Felt::new(5947388149010135441), + Felt::new(1678144343036748885), + ]), + RpoDigest::new([ + Felt::new(10347491038074052866), + Felt::new(11061756013655443653), + Felt::new(8901792852813329415), + Felt::new(10002477867799577447), + ]), + RpoDigest::new([ + Felt::new(16688151588649906570), + Felt::new(12937054427339650762), + Felt::new(2125115528195796454), + Felt::new(4796610823085621719), + ]), + RpoDigest::new([ + Felt::new(3032620037225059051), + Felt::new(13522881885116127385), + Felt::new(6010511038055304264), + Felt::new(8199256447383686121), + ]), + RpoDigest::new([ + Felt::new(11250302734399433639), + Felt::new(4970037623163209776), + Felt::new(15776613712371118341), + Felt::new(5554382612311754837), + ]), + RpoDigest::new([ + Felt::new(5116523511540088640), + Felt::new(12381059245485642368), + Felt::new(2176361879916914688), + Felt::new(11209293198464735683), + ]), + RpoDigest::new([ + Felt::new(11677748883385181208), + Felt::new(15891398395707500576), + Felt::new(3790704659934033620), + Felt::new(2126099371106695189), + ]), + RpoDigest::new([ + Felt::new(13948603355603496603), + Felt::new(15902438544472945077), + Felt::new(1969361494026622497), + Felt::new(17326911676634210553), + ]), + RpoDigest::new([ + Felt::new(16081431322775411514), + Felt::new(13201312030265587002), + Felt::new(18283434127959076535), + Felt::new(9889802180847551599), + ]), + RpoDigest::new([ + Felt::new(8490051641633132830), + Felt::new(11985660456681176415), + Felt::new(12193381039977027251), + Felt::new(17563185381678568385), + ]), + RpoDigest::new([ + Felt::new(3870617340693651786), + Felt::new(2748490321246408799), + Felt::new(8501743976565218963), + Felt::new(1660720190266083389), + ]), + RpoDigest::new([ + Felt::new(2121119282758520982), + Felt::new(9042267662074029772), + Felt::new(15431993929052434204), + Felt::new(10659345458998811701), + ]), + RpoDigest::new([ + Felt::new(15206763021853065070), + Felt::new(15268692497656424421), + Felt::new(13335448435922172445), + Felt::new(3421340628484408379), + ]), + RpoDigest::new([ + Felt::new(5175159910654039438), + Felt::new(10258564296733764665), + Felt::new(235961379704359454), + Felt::new(18007006485615491006), + ]), + RpoDigest::new([ + Felt::new(9455184082727641653), + Felt::new(6634498452861935579), + Felt::new(18189776179964984407), + Felt::new(3546641211720870472), + ]), + RpoDigest::new([ + Felt::new(2566088177506289568), + Felt::new(7785941571143323572), + Felt::new(13948908169667863201), + Felt::new(8557252288425473395), + ]), + RpoDigest::new([ + Felt::new(8801845050152766755), + Felt::new(514652983374395586), + Felt::new(13975919271481418443), + Felt::new(17480955484347349170), + ]), + RpoDigest::new([ + Felt::new(7078477424334594989), + Felt::new(9975053207879493059), + Felt::new(5220656123503260168), + Felt::new(13795787984352794188), + ]), + RpoDigest::new([ + Felt::new(1478357986561897612), + Felt::new(3963701567400985039), + Felt::new(10269836564499521403), + Felt::new(11874873630603798755), + ]), + RpoDigest::new([ + Felt::new(936391814816943993), + Felt::new(6085855616346025677), + Felt::new(5782721339195502211), + Felt::new(10409491632083436908), + ]), + RpoDigest::new([ + Felt::new(11138475264090866271), + Felt::new(17799626597540451271), + Felt::new(17968790388406362807), + Felt::new(9539434947296310791), + ]), + RpoDigest::new([ + Felt::new(13051724588530357940), + Felt::new(8058102530250142518), + Felt::new(1861782711432586670), + Felt::new(2928050228215055187), + ]), + RpoDigest::new([ + Felt::new(10650694022550988030), + Felt::new(5634734408638476525), + Felt::new(9233115969432897632), + Felt::new(1437907447409278328), + ]), + RpoDigest::new([ + Felt::new(9720135276484706819), + Felt::new(9350120041401976641), + Felt::new(1348777594376050933), + Felt::new(13138246165242825648), + ]), + RpoDigest::new([ + Felt::new(10866643979409126085), + Felt::new(13790633638103642042), + Felt::new(6374461622011119670), + Felt::new(5702679962735491362), + ]), + RpoDigest::new([ + Felt::new(5257277882444261955), + Felt::new(8511211402794551302), + Felt::new(3294838877645533839), + Felt::new(4084864647832858048), + ]), + RpoDigest::new([ + Felt::new(7948776578097466250), + Felt::new(8630046431048474853), + Felt::new(11549811661672434609), + Felt::new(14329713552208961509), + ]), + RpoDigest::new([ + Felt::new(734617692582477804), + Felt::new(11871516935077749937), + Felt::new(12085935336918533812), + Felt::new(11028098016323141988), + ]), + RpoDigest::new([ + Felt::new(10937083382606895486), + Felt::new(12203867463821771187), + Felt::new(13369919265612777227), + Felt::new(2521482611471096233), + ]), + RpoDigest::new([ + Felt::new(1242037330294600071), + Felt::new(8643213198640797337), + Felt::new(14112360612081236212), + Felt::new(11296904697431650998), + ]), + RpoDigest::new([ + Felt::new(11958494925108187724), + Felt::new(6059642826232274823), + Felt::new(1563918267677757605), + Felt::new(266509853282035592), + ]), + RpoDigest::new([ + Felt::new(17288335252189973373), + Felt::new(3243363076395469373), + Felt::new(8880515798614590986), + Felt::new(10260780639137628077), + ]), + RpoDigest::new([ + Felt::new(1839714959437284152), + Felt::new(12088193186987715006), + Felt::new(10200898335013164008), + Felt::new(12768529781145127245), + ]), + RpoDigest::new([ + Felt::new(1537615626967151439), + Felt::new(11731506816677487155), + Felt::new(4748463589169553420), + Felt::new(17495851576537541106), + ]), + RpoDigest::new([ + Felt::new(957733314860117562), + Felt::new(15623410588944187169), + Felt::new(4321611031548662227), + Felt::new(12856104259650439278), + ]), + RpoDigest::new([ + Felt::new(14827447693720375746), + Felt::new(17296925942589213350), + Felt::new(13524332314559504765), + Felt::new(15663886706087995199), + ]), + RpoDigest::new([ + Felt::new(18185978518863914335), + Felt::new(936586966360019113), + Felt::new(497299419609993926), + Felt::new(1977881506773614749), + ]), + RpoDigest::new([ + Felt::new(8635338869442206704), + Felt::new(11671305615285950885), + Felt::new(15253023094703789604), + Felt::new(7398108415970215319), + ]), + RpoDigest::new([ZERO; WORD_SIZE]), +]; + +#[test] +fn all_depths_opens_to_zero() { + use super::Rpo256; + + for depth in 1..=64 { + // fetch the subtrees and reverse it so the path is leaf -> root + let mut subtree = EmptySubtreeRoots::empty_hashes(depth).to_vec(); + subtree.reverse(); + + // the length of the subtrees set must be equal to depth + 1 as we also + // include the root + assert_eq!(depth as usize + 1, subtree.len()); + + // assert the opening is zero + let initial = RpoDigest::new([ZERO; WORD_SIZE]); + assert_eq!(initial, subtree.remove(0)); + + // compute every node of the path manually and compare with the output + subtree + .into_iter() + .scan(initial, |state, x| { + *state = Rpo256::merge(&[*state; 2]); + Some((x, *state)) + }) + .for_each(|(x, computed)| assert_eq!(x, computed)); + } +} + +#[test] +fn arbitrary_inputs_will_generate_sound_slices() { + let min = &EMPTY_SUBTREES_64[0] as *const RpoDigest; + let max = unsafe { min.add(64) }; + for depth in 0..=64 { + let subtree = EmptySubtreeRoots::empty_hashes(depth); + let first = &subtree[0] as *const RpoDigest; + let last = &subtree[depth as usize] as *const RpoDigest; + assert!(min <= first && first <= max); + assert!(min <= last && last <= max); + } +} diff --git a/src/merkle/mod.rs b/src/merkle/mod.rs index 6d53a71..c819f6c 100644 --- a/src/merkle/mod.rs +++ b/src/merkle/mod.rs @@ -1,12 +1,16 @@ use super::{ hash::rpo::{Rpo256, RpoDigest}, utils::collections::{vec, BTreeMap, Vec}, - Felt, StarkField, Word, ZERO, + Felt, StarkField, Word, WORD_SIZE, ZERO, }; use core::fmt; // REEXPORTS // ================================================================================================ + +mod empty_roots; +pub use empty_roots::EmptySubtreeRoots; + mod index; pub use index::NodeIndex; diff --git a/src/merkle/simple_smt/mod.rs b/src/merkle/simple_smt/mod.rs index 186ac25..34f0a17 100644 --- a/src/merkle/simple_smt/mod.rs +++ b/src/merkle/simple_smt/mod.rs @@ -1,4 +1,6 @@ -use super::{BTreeMap, MerkleError, MerklePath, NodeIndex, Rpo256, RpoDigest, Vec, Word}; +use super::{ + BTreeMap, EmptySubtreeRoots, MerkleError, MerklePath, NodeIndex, Rpo256, RpoDigest, Vec, Word, +}; #[cfg(test)] mod tests; @@ -29,38 +31,55 @@ impl SimpleSmt { // CONSTRUCTORS // -------------------------------------------------------------------------------------------- - /// Creates a new simple SMT. - /// - /// The provided entries will be tuples of the leaves and their corresponding keys. + /// Creates a new simple SMT with the provided depth. + pub fn new(depth: u8) -> Result { + // validate the range of the depth. + if depth < Self::MIN_DEPTH { + return Err(MerkleError::DepthTooSmall(depth)); + } else if Self::MAX_DEPTH < depth { + return Err(MerkleError::DepthTooBig(depth as u64)); + } + + let (store, root) = Store::new(depth); + Ok(Self { root, depth, store }) + } + + /// Appends the provided entries as leaves of the tree. /// /// # Errors /// /// The function will fail if the provided entries count exceed the maximum tree capacity, that /// is `2^{depth}`. - pub fn new(entries: R, depth: u8) -> Result + pub fn with_leaves(mut self, entries: R) -> Result where R: IntoIterator, I: Iterator + ExactSizeIterator, { + // check if the leaves count will fit the depth setup let mut entries = entries.into_iter(); - - // validate the range of the depth. - let max = 1 << depth; - if depth < Self::MIN_DEPTH { - return Err(MerkleError::DepthTooSmall(depth)); - } else if Self::MAX_DEPTH < depth { - return Err(MerkleError::DepthTooBig(depth as u64)); - } else if entries.len() > max { + let max = 1 << self.depth; + if entries.len() > max { return Err(MerkleError::InvalidEntriesCount(max, entries.len())); } - let (store, root) = Store::new(depth); - let mut tree = Self { root, depth, store }; - entries.try_for_each(|(key, leaf)| tree.insert_leaf(key, leaf))?; - - Ok(tree) + // append leaves and return + entries.try_for_each(|(key, leaf)| self.insert_leaf(key, leaf))?; + Ok(self) } + /// Replaces the internal empty digests used when a given depth doesn't contain a node. + pub fn with_empty_subtrees(mut self, hashes: I) -> Self + where + I: IntoIterator, + { + self.store + .replace_empty_subtrees(hashes.into_iter().collect()); + self + } + + // PUBLIC ACCESSORS + // -------------------------------------------------------------------------------------------- + /// Returns the root of this Merkle tree. pub const fn root(&self) -> Word { self.root @@ -71,6 +90,9 @@ impl SimpleSmt { self.depth } + // PROVIDERS + // -------------------------------------------------------------------------------------------- + /// Returns the set count of the keys of the leaves. pub fn leaves_count(&self) -> usize { self.store.leaves_count() @@ -81,16 +103,24 @@ impl SimpleSmt { /// # Errors /// Returns an error if: /// * The specified depth is greater than the depth of the tree. - /// * The specified key does not exist pub fn get_node(&self, index: &NodeIndex) -> Result { if index.is_root() { Err(MerkleError::DepthTooSmall(index.depth())) } else if index.depth() > self.depth() { Err(MerkleError::DepthTooBig(index.depth() as u64)) } else if index.depth() == self.depth() { - self.store.get_leaf_node(index.value()) + self.store + .get_leaf_node(index.value()) + .or_else(|| { + self.store + .empty_hashes + .get(index.depth() as usize) + .copied() + .map(Word::from) + }) + .ok_or(MerkleError::InvalidIndex(*index)) } else { - let branch_node = self.store.get_branch_node(index)?; + let branch_node = self.store.get_branch_node(index); Ok(Rpo256::merge(&[branch_node.left, branch_node.right]).into()) } } @@ -100,23 +130,19 @@ impl SimpleSmt { /// /// # Errors /// Returns an error if: - /// * The specified key does not exist as a branch or leaf node /// * The specified depth is greater than the depth of the tree. pub fn get_path(&self, mut index: NodeIndex) -> Result { if index.is_root() { return Err(MerkleError::DepthTooSmall(index.depth())); } else if index.depth() > self.depth() { return Err(MerkleError::DepthTooBig(index.depth() as u64)); - } else if index.depth() == self.depth() && !self.store.check_leaf_node_exists(index.value()) - { - return Err(MerkleError::InvalidIndex(index.with_depth(self.depth()))); } let mut path = Vec::with_capacity(index.depth() as usize); for _ in 0..index.depth() { let is_right = index.is_value_odd(); index.move_up(); - let BranchNode { left, right } = self.store.get_branch_node(&index)?; + let BranchNode { left, right } = self.store.get_branch_node(&index); let value = if is_right { left } else { right }; path.push(*value); } @@ -133,6 +159,9 @@ impl SimpleSmt { self.get_path(NodeIndex::new(self.depth(), key)) } + // STATE MUTATORS + // -------------------------------------------------------------------------------------------- + /// Replaces the leaf located at the specified key, and recomputes hashes by walking up the tree /// /// # Errors @@ -156,10 +185,7 @@ impl SimpleSmt { for _ in 0..index.depth() { let is_right = index.is_value_odd(); index.move_up(); - let BranchNode { left, right } = self - .store - .get_branch_node(&index) - .unwrap_or_else(|_| self.store.get_empty_node(index.depth() as usize + 1)); + let BranchNode { left, right } = self.store.get_branch_node(&index); let (left, right) = if is_right { (left, value) } else { @@ -200,16 +226,7 @@ impl Store { let leaves = BTreeMap::new(); // Construct empty node digests for each layer of the tree - let empty_hashes: Vec = (0..depth + 1) - .scan(Word::default().into(), |state, _| { - let value = *state; - *state = Rpo256::merge(&[value, value]); - Some(value) - }) - .collect::>() - .into_iter() - .rev() - .collect(); + let empty_hashes = EmptySubtreeRoots::empty_hashes(depth).to_vec(); let root = empty_hashes[0].into(); let store = Self { @@ -222,34 +239,30 @@ impl Store { (store, root) } - fn get_empty_node(&self, depth: usize) -> BranchNode { - let digest = self.empty_hashes[depth]; - BranchNode { - left: digest, - right: digest, - } + fn replace_empty_subtrees(&mut self, hashes: Vec) { + self.empty_hashes = hashes; } fn check_leaf_node_exists(&self, key: u64) -> bool { self.leaves.contains_key(&key) } - fn get_leaf_node(&self, key: u64) -> Result { - self.leaves - .get(&key) - .cloned() - .ok_or(MerkleError::InvalidIndex(NodeIndex::new(self.depth, key))) + fn get_leaf_node(&self, key: u64) -> Option { + self.leaves.get(&key).copied() } fn insert_leaf_node(&mut self, key: u64, node: Word) { self.leaves.insert(key, node); } - fn get_branch_node(&self, index: &NodeIndex) -> Result { - self.branches - .get(index) - .cloned() - .ok_or(MerkleError::InvalidIndex(*index)) + fn get_branch_node(&self, index: &NodeIndex) -> BranchNode { + self.branches.get(index).cloned().unwrap_or_else(|| { + let node = self.empty_hashes[index.depth() as usize + 1]; + BranchNode { + left: node, + right: node, + } + }) } fn insert_branch_node(&mut self, index: NodeIndex, left: RpoDigest, right: RpoDigest) { diff --git a/src/merkle/simple_smt/tests.rs b/src/merkle/simple_smt/tests.rs index e449014..6abd343 100644 --- a/src/merkle/simple_smt/tests.rs +++ b/src/merkle/simple_smt/tests.rs @@ -2,7 +2,6 @@ use super::{ super::{int_to_node, MerkleTree, RpoDigest, SimpleSmt}, NodeIndex, Rpo256, Vec, Word, }; -use core::iter; use proptest::prelude::*; use rand_utils::prng_array; @@ -31,7 +30,7 @@ const ZERO_VALUES8: [Word; 8] = [int_to_node(0); 8]; #[test] fn build_empty_tree() { - let smt = SimpleSmt::new(iter::empty(), 3).unwrap(); + let smt = SimpleSmt::new(3).unwrap(); let mt = MerkleTree::new(ZERO_VALUES8.to_vec()).unwrap(); assert_eq!(mt.root(), smt.root()); } @@ -39,7 +38,7 @@ fn build_empty_tree() { #[test] fn empty_digests_are_consistent() { let depth = 5; - let root = SimpleSmt::new(iter::empty(), depth).unwrap().root(); + let root = SimpleSmt::new(depth).unwrap().root(); let computed: [RpoDigest; 2] = (0..depth).fold([Default::default(); 2], |state, _| { let digest = Rpo256::merge(&state); [digest; 2] @@ -50,7 +49,7 @@ fn empty_digests_are_consistent() { #[test] fn build_sparse_tree() { - let mut smt = SimpleSmt::new(iter::empty(), 3).unwrap(); + let mut smt = SimpleSmt::new(3).unwrap(); let mut values = ZERO_VALUES8.to_vec(); // insert single value @@ -82,7 +81,10 @@ fn build_sparse_tree() { #[test] fn build_full_tree() { - let tree = SimpleSmt::new(KEYS4.into_iter().zip(VALUES4.into_iter()), 2).unwrap(); + let tree = SimpleSmt::new(2) + .unwrap() + .with_leaves(KEYS4.into_iter().zip(VALUES4.into_iter())) + .unwrap(); let (root, node2, node3) = compute_internal_nodes(); assert_eq!(root, tree.root()); @@ -92,7 +94,10 @@ fn build_full_tree() { #[test] fn get_values() { - let tree = SimpleSmt::new(KEYS4.into_iter().zip(VALUES4.into_iter()), 2).unwrap(); + let tree = SimpleSmt::new(2) + .unwrap() + .with_leaves(KEYS4.into_iter().zip(VALUES4.into_iter())) + .unwrap(); // check depth 2 assert_eq!(VALUES4[0], tree.get_node(&NodeIndex::new(2, 0)).unwrap()); @@ -103,7 +108,10 @@ fn get_values() { #[test] fn get_path() { - let tree = SimpleSmt::new(KEYS4.into_iter().zip(VALUES4.into_iter()), 2).unwrap(); + let tree = SimpleSmt::new(2) + .unwrap() + .with_leaves(KEYS4.into_iter().zip(VALUES4.into_iter())) + .unwrap(); let (_, node2, node3) = compute_internal_nodes(); @@ -132,18 +140,20 @@ fn get_path() { #[test] fn update_leaf() { - let mut tree = SimpleSmt::new(KEYS8.into_iter().zip(VALUES8.into_iter()), 3).unwrap(); + let mut tree = SimpleSmt::new(3) + .unwrap() + .with_leaves(KEYS8.into_iter().zip(VALUES8.into_iter())) + .unwrap(); // update one value let key = 3; let new_node = int_to_node(9); let mut expected_values = VALUES8.to_vec(); expected_values[key] = new_node; - let expected_tree = SimpleSmt::new( - KEYS8.into_iter().zip(expected_values.clone().into_iter()), - 3, - ) - .unwrap(); + let expected_tree = SimpleSmt::new(3) + .unwrap() + .with_leaves(KEYS8.into_iter().zip(expected_values.clone().into_iter())) + .unwrap(); tree.update_leaf(key as u64, new_node).unwrap(); assert_eq!(expected_tree.root, tree.root); @@ -152,8 +162,10 @@ fn update_leaf() { let key = 6; let new_node = int_to_node(10); expected_values[key] = new_node; - let expected_tree = - SimpleSmt::new(KEYS8.into_iter().zip(expected_values.into_iter()), 3).unwrap(); + let expected_tree = SimpleSmt::new(3) + .unwrap() + .with_leaves(KEYS8.into_iter().zip(expected_values.into_iter())) + .unwrap(); tree.update_leaf(key as u64, new_node).unwrap(); assert_eq!(expected_tree.root, tree.root); @@ -188,7 +200,7 @@ fn small_tree_opening_is_consistent() { let depth = 3; let entries = vec![(0, a), (1, b), (4, c), (7, d)]; - let tree = SimpleSmt::new(entries, depth).unwrap(); + let tree = SimpleSmt::new(depth).unwrap().with_leaves(entries).unwrap(); assert_eq!(tree.root(), Word::from(k)); @@ -219,7 +231,7 @@ proptest! { key in prop::num::u64::ANY, leaf in prop::num::u64::ANY, ) { - let mut tree = SimpleSmt::new(iter::empty(), depth).unwrap(); + let mut tree = SimpleSmt::new(depth).unwrap(); let key = key % (1 << depth as u64); let leaf = int_to_node(leaf); @@ -240,7 +252,7 @@ proptest! { count in 2u8..10u8, ref seed in any::<[u8; 32]>() ) { - let mut tree = SimpleSmt::new(iter::empty(), depth).unwrap(); + let mut tree = SimpleSmt::new(depth).unwrap(); let mut seed = *seed; let leaves = (1 << depth) - 1; From 88a646031fdd08dbf10465f1dcab4bad4c0b92f1 Mon Sep 17 00:00:00 2001 From: "Augusto F. Hack" Date: Tue, 14 Mar 2023 00:10:54 +0100 Subject: [PATCH 07/26] feat: add merkle store --- src/bit.rs | 169 +++++++++++++ src/hash/rpo/digest.rs | 12 + src/lib.rs | 1 + src/merkle/index.rs | 14 ++ src/merkle/merkle_tree.rs | 4 +- src/merkle/mod.rs | 9 +- src/merkle/simple_smt/mod.rs | 14 +- src/merkle/store.rs | 461 +++++++++++++++++++++++++++++++++++ 8 files changed, 675 insertions(+), 9 deletions(-) create mode 100644 src/bit.rs create mode 100644 src/merkle/store.rs diff --git a/src/bit.rs b/src/bit.rs new file mode 100644 index 0000000..a58be2b --- /dev/null +++ b/src/bit.rs @@ -0,0 +1,169 @@ +/// Yields the bits of a `u64`. +pub struct BitIterator { + /// The value that is being iterated bit-wise + value: u64, + /// True bits in the `mask` are the bits that have been visited. + mask: u64, +} + +impl BitIterator { + pub fn new(value: u64) -> BitIterator { + BitIterator { value, mask: 0 } + } + + /// An efficient skip implementation. + /// + /// Note: The compiler is smart enough to translate a `skip(n)` into a single shift instruction + /// if the code is inlined, however inlining does not always happen. + pub fn skip_front(mut self, n: u32) -> Self { + let mask = bitmask(n); + let ones = self.mask.trailing_ones(); + let mask_position = ones; + self.mask ^= mask << mask_position; + self + } + + /// An efficient skip from the back. + /// + /// Note: The compiler is smart enough to translate a `skip(n)` into a single shift instruction + /// if the code is inlined, however inlining does not always happen. + pub fn skip_back(mut self, n: u32) -> Self { + let mask = bitmask(n); + let ones = self.mask.leading_ones(); + let mask_position = u64::BITS - ones - n; + self.mask ^= mask << mask_position; + self + } +} + +impl Iterator for BitIterator { + type Item = bool; + + fn next(&mut self) -> Option<::Item> { + // trailing_ones is implemented with trailing_zeros, and the zeros are computed with the + // intrinsic cttz. [Rust 1.67.0] x86 uses the `bsf` instruction. AArch64 uses the `rbit + // clz` instructions. + let ones = self.mask.trailing_ones(); + + if ones == u64::BITS { + None + } else { + let bit_position = ones; + let mask = 1 << bit_position; + self.mask ^= mask; + let bit = self.value & mask; + Some(bit != 0) + } + } +} + +impl DoubleEndedIterator for BitIterator { + fn next_back(&mut self) -> Option<::Item> { + // leading_ones is implemented with leading_zeros, and the zeros are computed with the + // intrinsic ctlz. [Rust 1.67.0] x86 uses the `bsr` instruction. AArch64 uses the `clz` + // instruction. + let ones = self.mask.leading_ones(); + + if ones == u64::BITS { + None + } else { + let bit_position = u64::BITS - ones - 1; + let mask = 1 << bit_position; + self.mask ^= mask; + let bit = self.value & mask; + Some(bit != 0) + } + } +} + +#[cfg(test)] +mod test { + use super::BitIterator; + + #[test] + fn test_bit_iterator() { + let v = 0b1; + let mut it = BitIterator::new(v); + assert!(it.next().unwrap(), "first bit is true"); + assert!(it.all(|v| v == false), "every other value is false"); + + let v = 0b10; + let mut it = BitIterator::new(v); + assert!(!it.next().unwrap(), "first bit is false"); + assert!(it.next().unwrap(), "first bit is true"); + assert!(it.all(|v| v == false), "every other value is false"); + + let v = 0b10; + let mut it = BitIterator::new(v); + assert!(!it.next_back().unwrap(), "last bit is false"); + assert!(!it.next().unwrap(), "first bit is false"); + assert!(it.next().unwrap(), "first bit is true"); + assert!(it.all(|v| v == false), "every other value is false"); + } + + #[test] + fn test_bit_iterator_skip() { + let v = 0b1; + let mut it = BitIterator::new(v).skip_front(1); + assert!(it.all(|v| v == false), "every other value is false"); + + let v = 0b10; + let mut it = BitIterator::new(v).skip_front(1); + assert!(it.next().unwrap(), "first bit is true"); + assert!(it.all(|v| v == false), "every other value is false"); + + let high_bit = 0b1 << (u64::BITS - 1); + let mut it = BitIterator::new(high_bit).skip_back(1); + assert!(it.all(|v| v == false), "every other value is false"); + + let v = 0b10; + let mut it = BitIterator::new(v).skip_back(1); + assert!(!it.next_back().unwrap(), "last bit is false"); + assert!(!it.next().unwrap(), "first bit is false"); + assert!(it.next().unwrap(), "first bit is true"); + assert!(it.all(|v| v == false), "every other value is false"); + } + + #[test] + fn test_skip_all() { + let v = 0b1; + let mut it = BitIterator::new(v).skip_front(u64::BITS); + assert!(it.next().is_none(), "iterator must be exhausted"); + + let v = 0b1; + let mut it = BitIterator::new(v).skip_back(u64::BITS); + assert!(it.next().is_none(), "iterator must be exhausted"); + } + + #[test] + fn test_bit_iterator_count_bits_after_skip() { + let any_value = 0b1; + for s in 0..u64::BITS { + let it = BitIterator::new(any_value).skip_front(s); + assert_eq!(it.count() as u32, u64::BITS - s) + } + + let any_value = 0b1; + for s in 1..u64::BITS { + let it = BitIterator::new(any_value).skip_back(s); + assert_eq!(it.count() as u32, u64::BITS - s) + } + } + + #[test] + fn test_bit_iterator_rev() { + let v = 0b1; + let mut it = BitIterator::new(v).rev(); + assert!(it.nth(63).unwrap(), "the last value is true"); + } +} + +// UTILITIES +// =============================================================================================== + +fn bitmask(s: u32) -> u64 { + match 1u64.checked_shl(s) { + Some(r) => r - 1, + None => u64::MAX, + } +} diff --git a/src/hash/rpo/digest.rs b/src/hash/rpo/digest.rs index 4af30ec..92f2a15 100644 --- a/src/hash/rpo/digest.rs +++ b/src/hash/rpo/digest.rs @@ -73,12 +73,24 @@ impl From<[Felt; DIGEST_SIZE]> for RpoDigest { } } +impl From<&RpoDigest> for [Felt; DIGEST_SIZE] { + fn from(value: &RpoDigest) -> Self { + value.0 + } +} + impl From for [Felt; DIGEST_SIZE] { fn from(value: RpoDigest) -> Self { value.0 } } +impl From<&RpoDigest> for [u8; 32] { + fn from(value: &RpoDigest) -> Self { + value.as_bytes() + } +} + impl From for [u8; 32] { fn from(value: RpoDigest) -> Self { value.as_bytes() diff --git a/src/lib.rs b/src/lib.rs index 0b0386b..d0285b2 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -4,6 +4,7 @@ #[cfg_attr(test, macro_use)] extern crate alloc; +mod bit; pub mod hash; pub mod merkle; diff --git a/src/merkle/index.rs b/src/merkle/index.rs index 1b47aab..988c79f 100644 --- a/src/merkle/index.rs +++ b/src/merkle/index.rs @@ -1,4 +1,5 @@ use super::{Felt, MerkleError, RpoDigest, StarkField}; +use crate::bit::BitIterator; // NODE INDEX // ================================================================================================ @@ -97,6 +98,19 @@ impl NodeIndex { self.depth == 0 } + /// Returns a bit iterator for the `value`. + /// + /// Bits read from left-to-right represent which internal node's child should be visited to + /// arrive at the leaf. From the right-to-left the bit represent the position the hash of the + /// current element should go. + /// + /// Additionally, the value that is not visisted are the sibling values necessary for a Merkle + /// opening. + pub fn bit_iterator(&self) -> BitIterator { + let depth: u32 = self.depth.into(); + BitIterator::new(self.value).skip_back(u64::BITS - depth) + } + // STATE MUTATORS // -------------------------------------------------------------------------------------------- diff --git a/src/merkle/merkle_tree.rs b/src/merkle/merkle_tree.rs index e9c53ea..a04a866 100644 --- a/src/merkle/merkle_tree.rs +++ b/src/merkle/merkle_tree.rs @@ -9,7 +9,7 @@ use winter_math::log2; /// A fully-balanced binary Merkle tree (i.e., a tree where the number of leaves is a power of two). #[derive(Debug, Clone, PartialEq, Eq)] pub struct MerkleTree { - nodes: Vec, + pub(crate) nodes: Vec, } impl MerkleTree { @@ -108,6 +108,8 @@ impl MerkleTree { index.move_up(); } + debug_assert!(index.is_root(), "the path must include the root"); + Ok(path.into()) } diff --git a/src/merkle/mod.rs b/src/merkle/mod.rs index c819f6c..3db75dd 100644 --- a/src/merkle/mod.rs +++ b/src/merkle/mod.rs @@ -1,6 +1,6 @@ use super::{ hash::rpo::{Rpo256, RpoDigest}, - utils::collections::{vec, BTreeMap, Vec}, + utils::collections::{vec, BTreeMap, BTreeSet, Vec}, Felt, StarkField, Word, WORD_SIZE, ZERO, }; use core::fmt; @@ -29,13 +29,18 @@ pub use simple_smt::SimpleSmt; mod mmr; pub use mmr::{Mmr, MmrPeaks}; +mod store; +pub use store::MerkleStore; + // ERRORS // ================================================================================================ #[derive(Clone, Debug)] pub enum MerkleError { + ConflictingRoots(Vec), DepthTooSmall(u8), DepthTooBig(u64), + NodeNotInStorage(Word, NodeIndex), NumLeavesNotPowerOfTwo(usize), InvalidIndex(NodeIndex), InvalidDepth { expected: u8, provided: u8 }, @@ -48,6 +53,7 @@ impl fmt::Display for MerkleError { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { use MerkleError::*; match self { + ConflictingRoots(roots) => write!(f, "the merkle paths roots do not match {roots:?}"), DepthTooSmall(depth) => write!(f, "the provided depth {depth} is too small"), DepthTooBig(depth) => write!(f, "the provided depth {depth} is too big"), NumLeavesNotPowerOfTwo(leaves) => { @@ -64,6 +70,7 @@ impl fmt::Display for MerkleError { InvalidPath(_path) => write!(f, "the provided path is not valid"), InvalidEntriesCount(max, provided) => write!(f, "the provided number of entries is {provided}, but the maximum for the given depth is {max}"), NodeNotInSet(index) => write!(f, "the node indexed by {index} is not in the set"), + NodeNotInStorage(hash, index) => write!(f, "the node {:?} indexed by {} and depth {} is not in the storage", hash, index.value(), index.depth(),), } } } diff --git a/src/merkle/simple_smt/mod.rs b/src/merkle/simple_smt/mod.rs index 34f0a17..e330a97 100644 --- a/src/merkle/simple_smt/mod.rs +++ b/src/merkle/simple_smt/mod.rs @@ -15,7 +15,7 @@ mod tests; pub struct SimpleSmt { root: Word, depth: u8, - store: Store, + pub(crate) store: Store, } impl SimpleSmt { @@ -207,17 +207,17 @@ impl SimpleSmt { /// respectively. Hashes for blank subtrees at each layer are stored in `empty_hashes`, beginning /// with the root hash of an empty tree, and ending with the zero value of a leaf node. #[derive(Debug, Clone, PartialEq, Eq)] -struct Store { - branches: BTreeMap, +pub(crate) struct Store { + pub(crate) branches: BTreeMap, leaves: BTreeMap, - empty_hashes: Vec, + pub(crate) empty_hashes: Vec, depth: u8, } #[derive(Debug, Default, Clone, PartialEq, Eq)] -struct BranchNode { - left: RpoDigest, - right: RpoDigest, +pub(crate) struct BranchNode { + pub(crate) left: RpoDigest, + pub(crate) right: RpoDigest, } impl Store { diff --git a/src/merkle/store.rs b/src/merkle/store.rs new file mode 100644 index 0000000..82001dc --- /dev/null +++ b/src/merkle/store.rs @@ -0,0 +1,461 @@ +//! An in-memory data store for Merkle-lized data +//! +//! This is a in memory data store for Merkle trees, this store allows all the nodes of a tree +//! (leaves or internal) to live as long as necessary and without duplication, this allows the +//! implementation of efficient persistent data structures +use super::{ + BTreeMap, BTreeSet, EmptySubtreeRoots, MerkleError, MerklePath, MerkleTree, NodeIndex, Rpo256, + RpoDigest, SimpleSmt, Vec, Word, +}; + +#[derive(Debug)] +pub struct Node { + left: RpoDigest, + right: RpoDigest, +} + +pub struct MerkleStore { + nodes: BTreeMap, +} + +impl Default for MerkleStore { + fn default() -> Self { + Self::new() + } +} + +impl MerkleStore { + // CONSTRUCTORS + // -------------------------------------------------------------------------------------------- + + /// Creates an empty `MerkleStore` instance. + pub fn new() -> MerkleStore { + let mut nodes = BTreeMap::new(); + + // pre-populate the store with the empty hashes + let subtrees = EmptySubtreeRoots::empty_hashes(64); + for (child, parent) in subtrees.iter().zip(subtrees.iter().skip(1)) { + nodes.insert( + *parent, + Node { + left: *child, + right: *child, + }, + ); + } + MerkleStore { nodes } + } + + /// Adds all the nodes of a Merkle tree represented by `leaves`. + /// + /// This will instantiate a Merkle tree using `leaves` and include all the nodes into the + /// storage. + /// + /// # Errors + /// + /// This method may return the following errors: + /// - `DepthTooSmall` if leaves is empty or contains only 1 element + /// - `NumLeavesNotPowerOfTwo` if the number of leaves is not a power-of-two + pub fn add_merkle_tree(&mut self, leaves: Vec) -> Result { + let layers = leaves.len().ilog2(); + let tree = MerkleTree::new(leaves)?; + + let mut depth = 0; + let mut parent_offset = 1; + let mut child_offset = 2; + while depth < layers { + let layer_size = 1usize << depth; + for _ in 0..layer_size { + // merkle tree is using level form representation, so left and right siblings are + // next to each other + let left = tree.nodes[child_offset]; + let right = tree.nodes[child_offset + 1]; + self.nodes.insert( + tree.nodes[parent_offset].into(), + Node { + left: left.into(), + right: right.into(), + }, + ); + parent_offset += 1; + child_offset += 2; + } + depth += 1; + } + + Ok(tree.nodes[1]) + } + + /// Adds all the nodes of a Sparse Merkle tree represented by `entries`. + /// + /// This will instantiate a Sparse Merkle tree using `entries` and include all the nodes into + /// the storage. + /// + /// # Errors + /// + /// This will return `InvalidEntriesCount` if the length of `entries` is not `63`. + pub fn add_sparse_merkle_tree(&mut self, entries: R) -> Result + where + R: IntoIterator, + I: Iterator + ExactSizeIterator, + { + let smt = SimpleSmt::new(SimpleSmt::MAX_DEPTH)?.with_leaves(entries)?; + for branch in smt.store.branches.values() { + let parent = Rpo256::merge(&[branch.left, branch.right]); + self.nodes.insert( + parent, + Node { + left: branch.left, + right: branch.right, + }, + ); + } + + Ok(smt.root()) + } + + /// Adds all the nodes of a Merkle path represented by `path`. + /// + /// This will compute the sibling elements determined by the Merkle `path` and `node`, and + /// include all the nodes into the storage. + pub fn add_merkle_path( + &mut self, + index_value: u64, + node: Word, + path: MerklePath, + ) -> Result { + let mut node = node; + let mut index = NodeIndex::new(self.nodes.len() as u8, index_value); + + for sibling in path { + let (left, right) = match index.is_value_odd() { + true => (sibling, node), + false => (node, sibling), + }; + let parent = Rpo256::merge(&[left.into(), right.into()]); + self.nodes.insert( + parent, + Node { + left: left.into(), + right: right.into(), + }, + ); + + index.move_up(); + node = parent.into(); + } + + Ok(node) + } + + /// Adds all the nodes of multiple Merkle paths into the store. + /// + /// This will compute the sibling elements for each Merkle `path` and include all the nodes + /// into the storage. + /// + /// # Errors + /// + /// Every path must resolve to the same root, otherwise this will return an `ConflictingRoots` + /// error. + pub fn add_merkle_paths(&mut self, paths: I) -> Result + where + I: IntoIterator, + { + let paths: Vec<(u64, Word, MerklePath)> = paths.into_iter().collect(); + + let roots: BTreeSet = paths + .iter() + .map(|(index, node, path)| path.compute_root(*index, *node).into()) + .collect(); + + if roots.len() != 1 { + return Err(MerkleError::ConflictingRoots( + roots.iter().map(|v| Word::from(*v)).collect(), + )); + } + + for (index_value, node, path) in paths { + self.add_merkle_path(index_value, node, path)?; + } + + // Returns the parent of the last paths (assumes all paths have the same parent) or empty + // The length of unique_roots is checked above, so this wont panic + Ok(roots.iter().next().unwrap().into()) + } + + // PUBLIC ACCESSORS + // -------------------------------------------------------------------------------------------- + + /// Returns the node at `index` rooted on the tree `root`. + /// + /// # Errors + /// + /// This will return `NodeNotInStorage` if the element is not present in the store. + pub fn get_node(&self, root: Word, index: NodeIndex) -> Result { + let mut hash: RpoDigest = root.into(); + for bit in index.bit_iterator().rev() { + let node = self + .nodes + .get(&hash) + .ok_or(MerkleError::NodeNotInStorage(hash.into(), index))?; + hash = if bit { node.right } else { node.left } + } + + Ok(hash.into()) + } + + /// Returns the path for the node at `index` rooted on the tree `root`. + /// + /// # Errors + /// + /// This will return `NodeNotInStorage` if the element is not present in the store. + pub fn get_path( + &self, + root: Word, + index: NodeIndex, + ) -> Result<(Word, MerklePath), MerkleError> { + let mut hash: RpoDigest = root.into(); + let mut path = Vec::new(); + let node = RpoDigest::default(); + for bit in index.bit_iterator() { + let node = self + .nodes + .get(&hash) + .ok_or(MerkleError::NodeNotInStorage(hash.into(), index))?; + + hash = if bit { + path.push(node.left.into()); + node.right + } else { + path.push(node.right.into()); + node.left + } + } + + Ok((node.into(), MerklePath::new(path))) + } + + // DATA MUTATORS + // -------------------------------------------------------------------------------------------- + + pub fn set_node( + &mut self, + root: Word, + index: NodeIndex, + value: Word, + ) -> Result { + let (current_node, path) = self.get_path(root, index)?; + if current_node != value { + self.add_merkle_path(index.value(), value, path) + } else { + Ok(root) + } + } + + pub fn merge_roots(&mut self, root1: Word, root2: Word) -> Result { + let root1: RpoDigest = root1.into(); + let root2: RpoDigest = root2.into(); + + if !self.nodes.contains_key(&root1) { + Err(MerkleError::NodeNotInStorage( + root1.into(), + NodeIndex::new(0, 0), + )) + } else if !self.nodes.contains_key(&root1) { + Err(MerkleError::NodeNotInStorage( + root2.into(), + NodeIndex::new(0, 0), + )) + } else { + let parent: Word = Rpo256::merge(&[root1, root2]).into(); + self.nodes.insert( + parent.into(), + Node { + left: root1, + right: root2, + }, + ); + + Ok(parent) + } + } +} + +#[cfg(test)] +mod test { + use super::{MerkleError, MerkleStore, MerkleTree, NodeIndex, SimpleSmt, Word}; + use crate::merkle::int_to_node; + use crate::merkle::MerklePathSet; + + const KEYS4: [u64; 4] = [0, 1, 2, 3]; + const LEAVES4: [Word; 4] = [ + int_to_node(1), + int_to_node(2), + int_to_node(3), + int_to_node(4), + ]; + + #[test] + fn test_add_merkle_tree() -> Result<(), MerkleError> { + let mut store = MerkleStore::default(); + + let mtree = MerkleTree::new(LEAVES4.to_vec())?; + store.add_merkle_tree(LEAVES4.to_vec())?; + + assert!( + store + .get_node(mtree.root(), NodeIndex::new(mtree.depth(), 0)) + .is_ok(), + "node 0 must be in the tree" + ); + assert!( + store + .get_node(mtree.root(), NodeIndex::new(mtree.depth(), 1)) + .is_ok(), + "node 1 must be in the tree" + ); + assert!( + store + .get_node(mtree.root(), NodeIndex::new(mtree.depth(), 2)) + .is_ok(), + "node 2 must be in the tree" + ); + assert!( + store + .get_node(mtree.root(), NodeIndex::new(mtree.depth(), 3)) + .is_ok(), + "node 3 must be in the tree" + ); + + store + .get_node(mtree.root(), NodeIndex::new(mtree.depth(), 0)) + .expect("node 0 must be in tree"); + store + .get_node(mtree.root(), NodeIndex::new(mtree.depth(), 1)) + .expect("node 1 must be in tree"); + store + .get_node(mtree.root(), NodeIndex::new(mtree.depth(), 2)) + .expect("node 2 must be in tree"); + store + .get_node(mtree.root(), NodeIndex::new(mtree.depth(), 3)) + .expect("node 3 must be in tree"); + + Ok(()) + } + + #[test] + fn test_get_node_returns_self_for_root() { + let store = MerkleStore::default(); + let root_idx = NodeIndex::new(0, 0); + + // the root does not need any lookups in the storage itself, so the value is just returned + assert_eq!(store.get_node(LEAVES4[0], root_idx).unwrap(), LEAVES4[0]); + assert_eq!(store.get_node(LEAVES4[1], root_idx).unwrap(), LEAVES4[1]); + assert_eq!(store.get_node(LEAVES4[2], root_idx).unwrap(), LEAVES4[2]); + assert_eq!(store.get_node(LEAVES4[3], root_idx).unwrap(), LEAVES4[3]); + } + + #[test] + fn test_get_invalid_node() { + let mut store = MerkleStore::default(); + let mtree = MerkleTree::new(LEAVES4.to_vec()).expect("creating a merkle tree must work"); + store + .add_merkle_tree(LEAVES4.to_vec()) + .expect("adding a merkle tree to the store must work"); + let _ = store.get_node(mtree.root(), NodeIndex::new(mtree.depth(), 3)); + } + + #[test] + fn test_add_sparse_merkle_tree_one_level() -> Result<(), MerkleError> { + let mut store = MerkleStore::default(); + let keys2: [u64; 2] = [0, 1]; + let leaves2: [Word; 2] = [int_to_node(1), int_to_node(2)]; + store.add_sparse_merkle_tree(keys2.into_iter().zip(leaves2.into_iter()))?; + let smt = SimpleSmt::new(SimpleSmt::MAX_DEPTH) + .unwrap() + .with_leaves(keys2.into_iter().zip(leaves2.into_iter())) + .unwrap(); + + let idx = NodeIndex::new(1, 0); + assert_eq!( + store.get_node(smt.root(), idx).unwrap(), + smt.get_node(&idx).unwrap() + ); + + Ok(()) + } + + #[test] + fn test_add_sparse_merkle_tree() -> Result<(), MerkleError> { + let mut store = MerkleStore::default(); + store.add_sparse_merkle_tree(KEYS4.into_iter().zip(LEAVES4.into_iter()))?; + + let smt = SimpleSmt::new(SimpleSmt::MAX_DEPTH) + .unwrap() + .with_leaves(KEYS4.into_iter().zip(LEAVES4.into_iter())) + .unwrap(); + + let idx = NodeIndex::new(1, 0); + assert_eq!( + store.get_node(smt.root(), idx).unwrap(), + smt.get_node(&idx).unwrap() + ); + let idx = NodeIndex::new(1, 1); + assert_eq!( + store.get_node(smt.root(), idx).unwrap(), + smt.get_node(&idx).unwrap() + ); + + Ok(()) + } + + #[test] + fn test_add_merkle_paths() -> Result<(), MerkleError> { + let mut store = MerkleStore::default(); + let mtree = MerkleTree::new(LEAVES4.to_vec())?; + + let i0 = 0; + let p0 = mtree.get_path(NodeIndex::new(2, i0)).unwrap(); + + let i1 = 1; + let p1 = mtree.get_path(NodeIndex::new(2, i1)).unwrap(); + + let i2 = 2; + let p2 = mtree.get_path(NodeIndex::new(2, i2)).unwrap(); + + let i3 = 3; + let p3 = mtree.get_path(NodeIndex::new(2, i3)).unwrap(); + + let paths = [ + (i0, LEAVES4[i0 as usize], p0), + (i1, LEAVES4[i1 as usize], p1), + (i2, LEAVES4[i2 as usize], p2), + (i3, LEAVES4[i3 as usize], p3), + ]; + + store + .add_merkle_paths(paths.clone()) + .expect("the valid paths must work"); + + let set = MerklePathSet::new(3).with_paths(paths).unwrap(); + + assert_eq!( + set.get_node(NodeIndex::new(3, 0)).unwrap(), + store.get_node(set.root(), NodeIndex::new(2, 0b00)).unwrap(), + ); + assert_eq!( + set.get_node(NodeIndex::new(3, 1)).unwrap(), + store.get_node(set.root(), NodeIndex::new(2, 0b01)).unwrap(), + ); + assert_eq!( + set.get_node(NodeIndex::new(3, 2)).unwrap(), + store.get_node(set.root(), NodeIndex::new(2, 0b10)).unwrap(), + ); + assert_eq!( + set.get_node(NodeIndex::new(3, 3)).unwrap(), + store.get_node(set.root(), NodeIndex::new(2, 0b11)).unwrap(), + ); + + Ok(()) + } +} From 91667fd7dec45c070cfc44eebcc10bf76928a7d8 Mon Sep 17 00:00:00 2001 From: Victor Lopez Date: Thu, 16 Mar 2023 10:08:26 +0100 Subject: [PATCH 08/26] refactor: add derive proc macros to merkle store This commit introduce common derive proc macros to MerkleStore. These are required downstream as the in-memory storage can be cloned. It also introduces constructors common to the other types of the crate that will help to build a merkle store, using a build pattern. --- src/merkle/store.rs | 68 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 55 insertions(+), 13 deletions(-) diff --git a/src/merkle/store.rs b/src/merkle/store.rs index 82001dc..ae2dec9 100644 --- a/src/merkle/store.rs +++ b/src/merkle/store.rs @@ -8,12 +8,13 @@ use super::{ RpoDigest, SimpleSmt, Vec, Word, }; -#[derive(Debug)] +#[derive(Debug, Default, Copy, Clone, Eq, PartialEq)] pub struct Node { left: RpoDigest, right: RpoDigest, } +#[derive(Debug, Clone, Eq, PartialEq)] pub struct MerkleStore { nodes: BTreeMap, } @@ -30,22 +31,59 @@ impl MerkleStore { /// Creates an empty `MerkleStore` instance. pub fn new() -> MerkleStore { - let mut nodes = BTreeMap::new(); - // pre-populate the store with the empty hashes let subtrees = EmptySubtreeRoots::empty_hashes(64); - for (child, parent) in subtrees.iter().zip(subtrees.iter().skip(1)) { - nodes.insert( - *parent, - Node { - left: *child, - right: *child, - }, - ); - } + let nodes = subtrees + .iter() + .copied() + .zip(subtrees.iter().skip(1).copied()) + .map(|(child, parent)| { + ( + parent, + Node { + left: child, + right: child, + }, + ) + }) + .collect(); + MerkleStore { nodes } } + /// Appends the provided merkle tree represented by its `leaves` to the set. + pub fn with_merkle_tree(mut self, leaves: I) -> Result + where + I: IntoIterator, + { + self.add_merkle_tree(leaves)?; + Ok(self) + } + + /// Appends the provided sparse merkle tree represented by its `entries` to the set. + pub fn with_sparse_merkle_tree(mut self, entries: R) -> Result + where + R: IntoIterator, + I: Iterator + ExactSizeIterator, + { + self.add_sparse_merkle_tree(entries)?; + Ok(self) + } + + /// Appends the provided merkle path set. + pub fn with_merkle_path( + mut self, + index_value: u64, + node: Word, + path: MerklePath, + ) -> Result { + self.add_merkle_path(index_value, node, path)?; + Ok(self) + } + + // STATE MUTATORS + // -------------------------------------------------------------------------------------------- + /// Adds all the nodes of a Merkle tree represented by `leaves`. /// /// This will instantiate a Merkle tree using `leaves` and include all the nodes into the @@ -56,7 +94,11 @@ impl MerkleStore { /// This method may return the following errors: /// - `DepthTooSmall` if leaves is empty or contains only 1 element /// - `NumLeavesNotPowerOfTwo` if the number of leaves is not a power-of-two - pub fn add_merkle_tree(&mut self, leaves: Vec) -> Result { + pub fn add_merkle_tree(&mut self, leaves: I) -> Result + where + I: IntoIterator, + { + let leaves: Vec<_> = leaves.into_iter().collect(); let layers = leaves.len().ilog2(); let tree = MerkleTree::new(leaves)?; From 669ebb49fb18724c5c22a5eb416e56cc767b3edc Mon Sep 17 00:00:00 2001 From: "Augusto F. Hack" Date: Thu, 16 Mar 2023 23:26:02 +0100 Subject: [PATCH 09/26] bugfix: check if the requested root is in the storage --- src/merkle/store.rs | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/src/merkle/store.rs b/src/merkle/store.rs index ae2dec9..5b5c88e 100644 --- a/src/merkle/store.rs +++ b/src/merkle/store.rs @@ -235,6 +235,12 @@ impl MerkleStore { /// This will return `NodeNotInStorage` if the element is not present in the store. pub fn get_node(&self, root: Word, index: NodeIndex) -> Result { let mut hash: RpoDigest = root.into(); + + // Check the root is in the storage when called with `NodeIndex::root()` + self.nodes + .get(&hash) + .ok_or(MerkleError::NodeNotInStorage(hash.into(), index))?; + for bit in index.bit_iterator().rev() { let node = self .nodes @@ -385,18 +391,6 @@ mod test { Ok(()) } - #[test] - fn test_get_node_returns_self_for_root() { - let store = MerkleStore::default(); - let root_idx = NodeIndex::new(0, 0); - - // the root does not need any lookups in the storage itself, so the value is just returned - assert_eq!(store.get_node(LEAVES4[0], root_idx).unwrap(), LEAVES4[0]); - assert_eq!(store.get_node(LEAVES4[1], root_idx).unwrap(), LEAVES4[1]); - assert_eq!(store.get_node(LEAVES4[2], root_idx).unwrap(), LEAVES4[2]); - assert_eq!(store.get_node(LEAVES4[3], root_idx).unwrap(), LEAVES4[3]); - } - #[test] fn test_get_invalid_node() { let mut store = MerkleStore::default(); From 867b772d9a3f894ece86c629a2adcb78f5d41b05 Mon Sep 17 00:00:00 2001 From: Victor Lopez Date: Thu, 16 Mar 2023 19:50:59 +0100 Subject: [PATCH 10/26] fix: merkle store panics on bounds Prior to this commit, the MerkleStore panicked under certain bounds. It will prevent such panics by using checked operations. ilog2, for instance, will panic when the operand is zero. However, there is a documentation rule enforcing the merkle tree to be size at least 2. If this rule is checked, then the panic is impossible. --- src/bit.rs | 4 +- src/merkle/mod.rs | 2 +- src/merkle/path_set.rs | 8 ++ src/merkle/store.rs | 181 ++++++++++++++++++++++++++--------------- 4 files changed, 127 insertions(+), 68 deletions(-) diff --git a/src/bit.rs b/src/bit.rs index a58be2b..5eb2577 100644 --- a/src/bit.rs +++ b/src/bit.rs @@ -19,7 +19,7 @@ impl BitIterator { let mask = bitmask(n); let ones = self.mask.trailing_ones(); let mask_position = ones; - self.mask ^= mask << mask_position; + self.mask ^= mask.checked_shl(mask_position).unwrap_or(0); self } @@ -31,7 +31,7 @@ impl BitIterator { let mask = bitmask(n); let ones = self.mask.leading_ones(); let mask_position = u64::BITS - ones - n; - self.mask ^= mask << mask_position; + self.mask ^= mask.checked_shl(mask_position).unwrap_or(0); self } } diff --git a/src/merkle/mod.rs b/src/merkle/mod.rs index 3db75dd..550eb0c 100644 --- a/src/merkle/mod.rs +++ b/src/merkle/mod.rs @@ -35,7 +35,7 @@ pub use store::MerkleStore; // ERRORS // ================================================================================================ -#[derive(Clone, Debug)] +#[derive(Clone, Debug, PartialEq, Eq)] pub enum MerkleError { ConflictingRoots(Vec), DepthTooSmall(u8), diff --git a/src/merkle/path_set.rs b/src/merkle/path_set.rs index 0b9d85c..b483949 100644 --- a/src/merkle/path_set.rs +++ b/src/merkle/path_set.rs @@ -57,6 +57,14 @@ impl MerklePathSet { self.total_depth } + /// Returns all the leaf indexes of this path set. + pub fn indexes(&self) -> impl Iterator + '_ { + self.paths + .keys() + .copied() + .map(|index| NodeIndex::new(self.total_depth, index)) + } + /// Returns a node at the specified index. /// /// # Errors diff --git a/src/merkle/store.rs b/src/merkle/store.rs index 5b5c88e..12c38c9 100644 --- a/src/merkle/store.rs +++ b/src/merkle/store.rs @@ -4,8 +4,8 @@ //! (leaves or internal) to live as long as necessary and without duplication, this allows the //! implementation of efficient persistent data structures use super::{ - BTreeMap, BTreeSet, EmptySubtreeRoots, MerkleError, MerklePath, MerkleTree, NodeIndex, Rpo256, - RpoDigest, SimpleSmt, Vec, Word, + BTreeMap, BTreeSet, EmptySubtreeRoots, MerkleError, MerklePath, MerklePathSet, MerkleTree, + NodeIndex, Rpo256, RpoDigest, SimpleSmt, Vec, Word, }; #[derive(Debug, Default, Copy, Clone, Eq, PartialEq)] @@ -81,6 +81,51 @@ impl MerkleStore { Ok(self) } + // PUBLIC ACCESSORS + // -------------------------------------------------------------------------------------------- + + /// Returns the node at `index` rooted on the tree `root`. + /// + /// # Errors + /// + /// This will return `NodeNotInStorage` if the element is not present in the store. + pub fn get_node(&self, root: Word, index: NodeIndex) -> Result { + let mut hash: RpoDigest = root.into(); + + // Check the root is in the storage when called with `NodeIndex::root()` + self.nodes + .get(&hash) + .ok_or(MerkleError::NodeNotInStorage(hash.into(), index))?; + + for bit in index.bit_iterator().rev() { + let node = self + .nodes + .get(&hash) + .ok_or(MerkleError::NodeNotInStorage(hash.into(), index))?; + hash = if bit { node.right } else { node.left } + } + + Ok(hash.into()) + } + + /// Returns the path for the node at `index` rooted on the tree `root`. + /// + /// The path starts at the sibling of the target leaf. + /// + /// # Errors + /// + /// This will return `NodeNotInStorage` if the element is not present in the store. + pub fn get_path(&self, root: Word, mut index: NodeIndex) -> Result { + let mut path = Vec::with_capacity(index.depth().saturating_sub(1) as usize); + while index.depth() > 0 { + let sibling = index.sibling(); + index.move_up(); + let node = self.get_node(root, sibling)?; + path.push(node); + } + Ok(MerklePath::new(path)) + } + // STATE MUTATORS // -------------------------------------------------------------------------------------------- @@ -99,6 +144,10 @@ impl MerkleStore { I: IntoIterator, { let leaves: Vec<_> = leaves.into_iter().collect(); + if leaves.len() < 2 { + return Err(MerkleError::DepthTooSmall(leaves.len() as u8)); + } + let layers = leaves.len().ilog2(); let tree = MerkleTree::new(leaves)?; @@ -225,74 +274,24 @@ impl MerkleStore { Ok(roots.iter().next().unwrap().into()) } - // PUBLIC ACCESSORS - // -------------------------------------------------------------------------------------------- - - /// Returns the node at `index` rooted on the tree `root`. - /// - /// # Errors - /// - /// This will return `NodeNotInStorage` if the element is not present in the store. - pub fn get_node(&self, root: Word, index: NodeIndex) -> Result { - let mut hash: RpoDigest = root.into(); - - // Check the root is in the storage when called with `NodeIndex::root()` - self.nodes - .get(&hash) - .ok_or(MerkleError::NodeNotInStorage(hash.into(), index))?; - - for bit in index.bit_iterator().rev() { - let node = self - .nodes - .get(&hash) - .ok_or(MerkleError::NodeNotInStorage(hash.into(), index))?; - hash = if bit { node.right } else { node.left } - } - - Ok(hash.into()) + /// Appends the provided [MerklePathSet] into the store. + pub fn add_merkle_path_set(&mut self, path_set: &MerklePathSet) -> Result { + let root = path_set.root(); + path_set.indexes().try_fold(root, |_, index| { + let node = path_set.get_node(index)?; + let path = path_set.get_path(index)?; + self.add_merkle_path(index.value(), node, path) + }) } - /// Returns the path for the node at `index` rooted on the tree `root`. - /// - /// # Errors - /// - /// This will return `NodeNotInStorage` if the element is not present in the store. - pub fn get_path( - &self, - root: Word, - index: NodeIndex, - ) -> Result<(Word, MerklePath), MerkleError> { - let mut hash: RpoDigest = root.into(); - let mut path = Vec::new(); - let node = RpoDigest::default(); - for bit in index.bit_iterator() { - let node = self - .nodes - .get(&hash) - .ok_or(MerkleError::NodeNotInStorage(hash.into(), index))?; - - hash = if bit { - path.push(node.left.into()); - node.right - } else { - path.push(node.right.into()); - node.left - } - } - - Ok((node.into(), MerklePath::new(path))) - } - - // DATA MUTATORS - // -------------------------------------------------------------------------------------------- - pub fn set_node( &mut self, root: Word, index: NodeIndex, value: Word, ) -> Result { - let (current_node, path) = self.get_path(root, index)?; + let current_node = self.get_node(root, index)?; + let path = self.get_path(root, index)?; if current_node != value { self.add_merkle_path(index.value(), value, path) } else { @@ -331,9 +330,12 @@ impl MerkleStore { #[cfg(test)] mod test { - use super::{MerkleError, MerkleStore, MerkleTree, NodeIndex, SimpleSmt, Word}; - use crate::merkle::int_to_node; - use crate::merkle::MerklePathSet; + use super::*; + use crate::{ + hash::rpo::Rpo256, + merkle::{int_to_node, MerklePathSet}, + Felt, Word, + }; const KEYS4: [u64; 4] = [0, 1, 2, 3]; const LEAVES4: [Word; 4] = [ @@ -494,4 +496,53 @@ mod test { Ok(()) } + + #[test] + fn wont_open_to_different_depth_root() { + let empty = EmptySubtreeRoots::empty_hashes(64); + let a = [Felt::new(1); 4]; + let b = [Felt::new(2); 4]; + + // compute the root for a different depth + let mut root = Rpo256::merge(&[a.into(), b.into()]); + for depth in (1..=63).rev() { + root = Rpo256::merge(&[root, empty[depth]]); + } + let root = Word::from(root); + + let store = MerkleStore::default().with_merkle_tree([a, b]).unwrap(); + let index = NodeIndex::root(); + let err = store.get_node(root, index).err().unwrap(); + assert_eq!(err, MerkleError::NodeNotInStorage(root, index)); + } + + #[test] + fn store_path_opens_from_leaf() { + let a = [Felt::new(1); 4]; + let b = [Felt::new(2); 4]; + let c = [Felt::new(3); 4]; + let d = [Felt::new(4); 4]; + let e = [Felt::new(5); 4]; + let f = [Felt::new(6); 4]; + let g = [Felt::new(7); 4]; + let h = [Felt::new(8); 4]; + + let i = Rpo256::merge(&[a.into(), b.into()]); + let j = Rpo256::merge(&[c.into(), d.into()]); + let k = Rpo256::merge(&[e.into(), f.into()]); + let l = Rpo256::merge(&[g.into(), h.into()]); + + let m = Rpo256::merge(&[i.into(), j.into()]); + let n = Rpo256::merge(&[k.into(), l.into()]); + + let root = Rpo256::merge(&[m.into(), n.into()]); + + let store = MerkleStore::default() + .with_merkle_tree([a, b, c, d, e, f, g, h]) + .unwrap(); + let path = store.get_path(root.into(), NodeIndex::new(3, 1)).unwrap(); + + let expected = MerklePath::new([a.into(), j.into(), n.into()].to_vec()); + assert_eq!(path, expected); + } } From 8cb245dc1f36ef1b1d6ce884bc130d191247f139 Mon Sep 17 00:00:00 2001 From: Victor Lopez Date: Thu, 16 Mar 2023 19:50:59 +0100 Subject: [PATCH 11/26] bugfix: reverse merkle path to match other structures The store builds the path from root to leaf, this updates the code to return a path from leaf to root, as it is done by the other structures. This also added custom error for missing root. --- src/merkle/mod.rs | 6 +- src/merkle/{store.rs => store/mod.rs} | 295 +++------------ src/merkle/store/tests.rs | 497 ++++++++++++++++++++++++++ 3 files changed, 555 insertions(+), 243 deletions(-) rename src/merkle/{store.rs => store/mod.rs} (53%) create mode 100644 src/merkle/store/tests.rs diff --git a/src/merkle/mod.rs b/src/merkle/mod.rs index 550eb0c..411dd7e 100644 --- a/src/merkle/mod.rs +++ b/src/merkle/mod.rs @@ -40,13 +40,14 @@ pub enum MerkleError { ConflictingRoots(Vec), DepthTooSmall(u8), DepthTooBig(u64), - NodeNotInStorage(Word, NodeIndex), + NodeNotInStore(Word, NodeIndex), NumLeavesNotPowerOfTwo(usize), InvalidIndex(NodeIndex), InvalidDepth { expected: u8, provided: u8 }, InvalidPath(MerklePath), InvalidEntriesCount(usize, usize), NodeNotInSet(u64), + RootNotInStore(Word), } impl fmt::Display for MerkleError { @@ -70,7 +71,8 @@ impl fmt::Display for MerkleError { InvalidPath(_path) => write!(f, "the provided path is not valid"), InvalidEntriesCount(max, provided) => write!(f, "the provided number of entries is {provided}, but the maximum for the given depth is {max}"), NodeNotInSet(index) => write!(f, "the node indexed by {index} is not in the set"), - NodeNotInStorage(hash, index) => write!(f, "the node {:?} indexed by {} and depth {} is not in the storage", hash, index.value(), index.depth(),), + NodeNotInStore(hash, index) => write!(f, "the node {:?} indexed by {} and depth {} is not in the storage", hash, index.value(), index.depth(),), + RootNotInStore(root) => write!(f, "the root {:?} is not in the storage", root), } } } diff --git a/src/merkle/store.rs b/src/merkle/store/mod.rs similarity index 53% rename from src/merkle/store.rs rename to src/merkle/store/mod.rs index 12c38c9..961d4b8 100644 --- a/src/merkle/store.rs +++ b/src/merkle/store/mod.rs @@ -8,6 +8,9 @@ use super::{ NodeIndex, Rpo256, RpoDigest, SimpleSmt, Vec, Word, }; +#[cfg(test)] +mod tests; + #[derive(Debug, Default, Copy, Clone, Eq, PartialEq)] pub struct Node { left: RpoDigest, @@ -88,42 +91,67 @@ impl MerkleStore { /// /// # Errors /// - /// This will return `NodeNotInStorage` if the element is not present in the store. + /// This method can return the following errors: + /// - `RootNotInStore` if the `root` is not present in the storage. + /// - `NodeNotInStore` if a node needed to traverse from `root` to `index` is not present in the store. pub fn get_node(&self, root: Word, index: NodeIndex) -> Result { let mut hash: RpoDigest = root.into(); - // Check the root is in the storage when called with `NodeIndex::root()` + // corner case: check the root is in the storage when called with index `NodeIndex::root()` self.nodes .get(&hash) - .ok_or(MerkleError::NodeNotInStorage(hash.into(), index))?; + .ok_or(MerkleError::RootNotInStore(hash.into()))?; for bit in index.bit_iterator().rev() { let node = self .nodes .get(&hash) - .ok_or(MerkleError::NodeNotInStorage(hash.into(), index))?; + .ok_or(MerkleError::NodeNotInStore(hash.into(), index))?; hash = if bit { node.right } else { node.left } } Ok(hash.into()) } - /// Returns the path for the node at `index` rooted on the tree `root`. + /// Returns the node at the specified `index` and its opening to the `root`. /// /// The path starts at the sibling of the target leaf. /// /// # Errors /// - /// This will return `NodeNotInStorage` if the element is not present in the store. - pub fn get_path(&self, root: Word, mut index: NodeIndex) -> Result { - let mut path = Vec::with_capacity(index.depth().saturating_sub(1) as usize); - while index.depth() > 0 { - let sibling = index.sibling(); - index.move_up(); - let node = self.get_node(root, sibling)?; - path.push(node); + /// This method can return the following errors: + /// - `RootNotInStore` if the `root` is not present in the storage. + /// - `NodeNotInStore` if a node needed to traverse from `root` to `index` is not present in the store. + pub fn get_path( + &self, + root: Word, + index: NodeIndex, + ) -> Result<(Word, MerklePath), MerkleError> { + let mut hash: RpoDigest = root.into(); + let mut path = Vec::with_capacity(index.depth().into()); + + // corner case: check the root is in the storage when called with index `NodeIndex::root()` + self.nodes + .get(&hash) + .ok_or(MerkleError::RootNotInStore(hash.into()))?; + + for bit in index.bit_iterator().rev() { + let node = self + .nodes + .get(&hash) + .ok_or(MerkleError::NodeNotInStore(hash.into(), index))?; + + hash = if bit { + path.push(node.left.into()); + node.right + } else { + path.push(node.right.into()); + node.left + } } - Ok(MerklePath::new(path)) + + path.reverse(); + Ok((hash.into(), MerklePath::new(path))) } // STATE MUTATORS @@ -269,8 +297,6 @@ impl MerkleStore { self.add_merkle_path(index_value, node, path)?; } - // Returns the parent of the last paths (assumes all paths have the same parent) or empty - // The length of unique_roots is checked above, so this wont panic Ok(roots.iter().next().unwrap().into()) } @@ -284,16 +310,22 @@ impl MerkleStore { }) } + /// Sets a node to `value`. + /// + /// # Errors + /// + /// This method can return the following errors: + /// - `RootNotInStore` if the `root` is not present in the storage. + /// - `NodeNotInStore` if a node needed to traverse from `root` to `index` is not present in the store. pub fn set_node( &mut self, root: Word, index: NodeIndex, value: Word, ) -> Result { - let current_node = self.get_node(root, index)?; - let path = self.get_path(root, index)?; - if current_node != value { - self.add_merkle_path(index.value(), value, path) + let result = self.get_path(root, index)?; + if result.0 != value { + self.add_merkle_path(index.value(), value, result.1) } else { Ok(root) } @@ -304,12 +336,12 @@ impl MerkleStore { let root2: RpoDigest = root2.into(); if !self.nodes.contains_key(&root1) { - Err(MerkleError::NodeNotInStorage( + Err(MerkleError::NodeNotInStore( root1.into(), NodeIndex::new(0, 0), )) } else if !self.nodes.contains_key(&root1) { - Err(MerkleError::NodeNotInStorage( + Err(MerkleError::NodeNotInStore( root2.into(), NodeIndex::new(0, 0), )) @@ -327,222 +359,3 @@ impl MerkleStore { } } } - -#[cfg(test)] -mod test { - use super::*; - use crate::{ - hash::rpo::Rpo256, - merkle::{int_to_node, MerklePathSet}, - Felt, Word, - }; - - const KEYS4: [u64; 4] = [0, 1, 2, 3]; - const LEAVES4: [Word; 4] = [ - int_to_node(1), - int_to_node(2), - int_to_node(3), - int_to_node(4), - ]; - - #[test] - fn test_add_merkle_tree() -> Result<(), MerkleError> { - let mut store = MerkleStore::default(); - - let mtree = MerkleTree::new(LEAVES4.to_vec())?; - store.add_merkle_tree(LEAVES4.to_vec())?; - - assert!( - store - .get_node(mtree.root(), NodeIndex::new(mtree.depth(), 0)) - .is_ok(), - "node 0 must be in the tree" - ); - assert!( - store - .get_node(mtree.root(), NodeIndex::new(mtree.depth(), 1)) - .is_ok(), - "node 1 must be in the tree" - ); - assert!( - store - .get_node(mtree.root(), NodeIndex::new(mtree.depth(), 2)) - .is_ok(), - "node 2 must be in the tree" - ); - assert!( - store - .get_node(mtree.root(), NodeIndex::new(mtree.depth(), 3)) - .is_ok(), - "node 3 must be in the tree" - ); - - store - .get_node(mtree.root(), NodeIndex::new(mtree.depth(), 0)) - .expect("node 0 must be in tree"); - store - .get_node(mtree.root(), NodeIndex::new(mtree.depth(), 1)) - .expect("node 1 must be in tree"); - store - .get_node(mtree.root(), NodeIndex::new(mtree.depth(), 2)) - .expect("node 2 must be in tree"); - store - .get_node(mtree.root(), NodeIndex::new(mtree.depth(), 3)) - .expect("node 3 must be in tree"); - - Ok(()) - } - - #[test] - fn test_get_invalid_node() { - let mut store = MerkleStore::default(); - let mtree = MerkleTree::new(LEAVES4.to_vec()).expect("creating a merkle tree must work"); - store - .add_merkle_tree(LEAVES4.to_vec()) - .expect("adding a merkle tree to the store must work"); - let _ = store.get_node(mtree.root(), NodeIndex::new(mtree.depth(), 3)); - } - - #[test] - fn test_add_sparse_merkle_tree_one_level() -> Result<(), MerkleError> { - let mut store = MerkleStore::default(); - let keys2: [u64; 2] = [0, 1]; - let leaves2: [Word; 2] = [int_to_node(1), int_to_node(2)]; - store.add_sparse_merkle_tree(keys2.into_iter().zip(leaves2.into_iter()))?; - let smt = SimpleSmt::new(SimpleSmt::MAX_DEPTH) - .unwrap() - .with_leaves(keys2.into_iter().zip(leaves2.into_iter())) - .unwrap(); - - let idx = NodeIndex::new(1, 0); - assert_eq!( - store.get_node(smt.root(), idx).unwrap(), - smt.get_node(&idx).unwrap() - ); - - Ok(()) - } - - #[test] - fn test_add_sparse_merkle_tree() -> Result<(), MerkleError> { - let mut store = MerkleStore::default(); - store.add_sparse_merkle_tree(KEYS4.into_iter().zip(LEAVES4.into_iter()))?; - - let smt = SimpleSmt::new(SimpleSmt::MAX_DEPTH) - .unwrap() - .with_leaves(KEYS4.into_iter().zip(LEAVES4.into_iter())) - .unwrap(); - - let idx = NodeIndex::new(1, 0); - assert_eq!( - store.get_node(smt.root(), idx).unwrap(), - smt.get_node(&idx).unwrap() - ); - let idx = NodeIndex::new(1, 1); - assert_eq!( - store.get_node(smt.root(), idx).unwrap(), - smt.get_node(&idx).unwrap() - ); - - Ok(()) - } - - #[test] - fn test_add_merkle_paths() -> Result<(), MerkleError> { - let mut store = MerkleStore::default(); - let mtree = MerkleTree::new(LEAVES4.to_vec())?; - - let i0 = 0; - let p0 = mtree.get_path(NodeIndex::new(2, i0)).unwrap(); - - let i1 = 1; - let p1 = mtree.get_path(NodeIndex::new(2, i1)).unwrap(); - - let i2 = 2; - let p2 = mtree.get_path(NodeIndex::new(2, i2)).unwrap(); - - let i3 = 3; - let p3 = mtree.get_path(NodeIndex::new(2, i3)).unwrap(); - - let paths = [ - (i0, LEAVES4[i0 as usize], p0), - (i1, LEAVES4[i1 as usize], p1), - (i2, LEAVES4[i2 as usize], p2), - (i3, LEAVES4[i3 as usize], p3), - ]; - - store - .add_merkle_paths(paths.clone()) - .expect("the valid paths must work"); - - let set = MerklePathSet::new(3).with_paths(paths).unwrap(); - - assert_eq!( - set.get_node(NodeIndex::new(3, 0)).unwrap(), - store.get_node(set.root(), NodeIndex::new(2, 0b00)).unwrap(), - ); - assert_eq!( - set.get_node(NodeIndex::new(3, 1)).unwrap(), - store.get_node(set.root(), NodeIndex::new(2, 0b01)).unwrap(), - ); - assert_eq!( - set.get_node(NodeIndex::new(3, 2)).unwrap(), - store.get_node(set.root(), NodeIndex::new(2, 0b10)).unwrap(), - ); - assert_eq!( - set.get_node(NodeIndex::new(3, 3)).unwrap(), - store.get_node(set.root(), NodeIndex::new(2, 0b11)).unwrap(), - ); - - Ok(()) - } - - #[test] - fn wont_open_to_different_depth_root() { - let empty = EmptySubtreeRoots::empty_hashes(64); - let a = [Felt::new(1); 4]; - let b = [Felt::new(2); 4]; - - // compute the root for a different depth - let mut root = Rpo256::merge(&[a.into(), b.into()]); - for depth in (1..=63).rev() { - root = Rpo256::merge(&[root, empty[depth]]); - } - let root = Word::from(root); - - let store = MerkleStore::default().with_merkle_tree([a, b]).unwrap(); - let index = NodeIndex::root(); - let err = store.get_node(root, index).err().unwrap(); - assert_eq!(err, MerkleError::NodeNotInStorage(root, index)); - } - - #[test] - fn store_path_opens_from_leaf() { - let a = [Felt::new(1); 4]; - let b = [Felt::new(2); 4]; - let c = [Felt::new(3); 4]; - let d = [Felt::new(4); 4]; - let e = [Felt::new(5); 4]; - let f = [Felt::new(6); 4]; - let g = [Felt::new(7); 4]; - let h = [Felt::new(8); 4]; - - let i = Rpo256::merge(&[a.into(), b.into()]); - let j = Rpo256::merge(&[c.into(), d.into()]); - let k = Rpo256::merge(&[e.into(), f.into()]); - let l = Rpo256::merge(&[g.into(), h.into()]); - - let m = Rpo256::merge(&[i.into(), j.into()]); - let n = Rpo256::merge(&[k.into(), l.into()]); - - let root = Rpo256::merge(&[m.into(), n.into()]); - - let store = MerkleStore::default() - .with_merkle_tree([a, b, c, d, e, f, g, h]) - .unwrap(); - let path = store.get_path(root.into(), NodeIndex::new(3, 1)).unwrap(); - - let expected = MerklePath::new([a.into(), j.into(), n.into()].to_vec()); - assert_eq!(path, expected); - } -} diff --git a/src/merkle/store/tests.rs b/src/merkle/store/tests.rs new file mode 100644 index 0000000..a60adb4 --- /dev/null +++ b/src/merkle/store/tests.rs @@ -0,0 +1,497 @@ +use super::*; +use crate::{ + hash::rpo::Rpo256, + merkle::{int_to_node, MerklePathSet}, + Felt, Word, +}; + +const KEYS4: [u64; 4] = [0, 1, 2, 3]; +const LEAVES4: [Word; 4] = [ + int_to_node(1), + int_to_node(2), + int_to_node(3), + int_to_node(4), +]; + +#[test] +fn test_root_not_in_storage() -> Result<(), MerkleError> { + let mtree = MerkleTree::new(LEAVES4.to_vec())?; + let store = MerkleStore::default().with_merkle_tree(LEAVES4)?; + assert_eq!( + store.get_node(LEAVES4[0], NodeIndex::new(mtree.depth(), 0)), + Err(MerkleError::RootNotInStore(LEAVES4[0])), + "Leaf 0 is not a root" + ); + assert_eq!( + store.get_path(LEAVES4[0], NodeIndex::new(mtree.depth(), 0)), + Err(MerkleError::RootNotInStore(LEAVES4[0])), + "Leaf 0 is not a root" + ); + + Ok(()) +} + +#[test] +fn test_merkle_tree() -> Result<(), MerkleError> { + let mut store = MerkleStore::default(); + + let mtree = MerkleTree::new(LEAVES4.to_vec())?; + store.add_merkle_tree(LEAVES4.to_vec())?; + + // STORE LEAVES ARE CORRECT ============================================================== + // checks the leaves in the store corresponds to the expected values + assert_eq!( + store.get_node(mtree.root(), NodeIndex::new(mtree.depth(), 0)), + Ok(LEAVES4[0]), + "node 0 must be in the tree" + ); + assert_eq!( + store.get_node(mtree.root(), NodeIndex::new(mtree.depth(), 1)), + Ok(LEAVES4[1]), + "node 1 must be in the tree" + ); + assert_eq!( + store.get_node(mtree.root(), NodeIndex::new(mtree.depth(), 2)), + Ok(LEAVES4[2]), + "node 2 must be in the tree" + ); + assert_eq!( + store.get_node(mtree.root(), NodeIndex::new(mtree.depth(), 3)), + Ok(LEAVES4[3]), + "node 3 must be in the tree" + ); + + // STORE LEAVES MATCH TREE =============================================================== + // sanity check the values returned by the store and the tree + assert_eq!( + mtree.get_node(NodeIndex::new(mtree.depth(), 0)), + store.get_node(mtree.root(), NodeIndex::new(mtree.depth(), 0)), + "node 0 must be the same for both MerkleTree and MerkleStore" + ); + assert_eq!( + mtree.get_node(NodeIndex::new(mtree.depth(), 1)), + store.get_node(mtree.root(), NodeIndex::new(mtree.depth(), 1)), + "node 1 must be the same for both MerkleTree and MerkleStore" + ); + assert_eq!( + mtree.get_node(NodeIndex::new(mtree.depth(), 2)), + store.get_node(mtree.root(), NodeIndex::new(mtree.depth(), 2)), + "node 2 must be the same for both MerkleTree and MerkleStore" + ); + assert_eq!( + mtree.get_node(NodeIndex::new(mtree.depth(), 3)), + store.get_node(mtree.root(), NodeIndex::new(mtree.depth(), 3)), + "node 3 must be the same for both MerkleTree and MerkleStore" + ); + + // STORE MERKLE PATH MATCHS ============================================================== + // assert the merkle path returned by the store is the same as the one in the tree + let result = store + .get_path(mtree.root(), NodeIndex::new(mtree.depth(), 0)) + .unwrap(); + assert_eq!( + LEAVES4[0], result.0, + "Value for merkle path at index 0 must match leaf value" + ); + assert_eq!( + mtree.get_path(NodeIndex::new(mtree.depth(), 0)), + Ok(result.1), + "merkle path for index 0 must be the same for the MerkleTree and MerkleStore" + ); + + let result = store + .get_path(mtree.root(), NodeIndex::new(mtree.depth(), 1)) + .unwrap(); + assert_eq!( + LEAVES4[1], result.0, + "Value for merkle path at index 0 must match leaf value" + ); + assert_eq!( + mtree.get_path(NodeIndex::new(mtree.depth(), 1)), + Ok(result.1), + "merkle path for index 1 must be the same for the MerkleTree and MerkleStore" + ); + + let result = store + .get_path(mtree.root(), NodeIndex::new(mtree.depth(), 2)) + .unwrap(); + assert_eq!( + LEAVES4[2], result.0, + "Value for merkle path at index 0 must match leaf value" + ); + assert_eq!( + mtree.get_path(NodeIndex::new(mtree.depth(), 2)), + Ok(result.1), + "merkle path for index 0 must be the same for the MerkleTree and MerkleStore" + ); + + let result = store + .get_path(mtree.root(), NodeIndex::new(mtree.depth(), 3)) + .unwrap(); + assert_eq!( + LEAVES4[3], result.0, + "Value for merkle path at index 0 must match leaf value" + ); + assert_eq!( + mtree.get_path(NodeIndex::new(mtree.depth(), 3)), + Ok(result.1), + "merkle path for index 0 must be the same for the MerkleTree and MerkleStore" + ); + + Ok(()) +} + +#[test] +fn test_get_invalid_node() { + let mut store = MerkleStore::default(); + let mtree = MerkleTree::new(LEAVES4.to_vec()).expect("creating a merkle tree must work"); + store + .add_merkle_tree(LEAVES4.to_vec()) + .expect("adding a merkle tree to the store must work"); + let _ = store.get_node(mtree.root(), NodeIndex::new(mtree.depth(), 3)); +} + +#[test] +fn test_add_sparse_merkle_tree_one_level() -> Result<(), MerkleError> { + let mut store = MerkleStore::default(); + let keys2: [u64; 2] = [0, 1]; + let leaves2: [Word; 2] = [int_to_node(1), int_to_node(2)]; + store.add_sparse_merkle_tree(keys2.into_iter().zip(leaves2.into_iter()))?; + let smt = SimpleSmt::new(1) + .unwrap() + .with_leaves(keys2.into_iter().zip(leaves2.into_iter())) + .unwrap(); + + let idx = NodeIndex::new(1, 0); + assert_eq!(smt.get_node(&idx).unwrap(), leaves2[0]); + assert_eq!( + store.get_node(smt.root(), idx).unwrap(), + smt.get_node(&idx).unwrap() + ); + + let idx = NodeIndex::new(1, 1); + assert_eq!(smt.get_node(&idx).unwrap(), leaves2[1]); + assert_eq!( + store.get_node(smt.root(), idx).unwrap(), + smt.get_node(&idx).unwrap() + ); + + Ok(()) +} + +#[test] +fn test_sparse_merkle_tree() -> Result<(), MerkleError> { + let mut store = MerkleStore::default(); + store.add_sparse_merkle_tree(KEYS4.into_iter().zip(LEAVES4.into_iter()))?; + + let smt = SimpleSmt::new(SimpleSmt::MAX_DEPTH) + .unwrap() + .with_leaves(KEYS4.into_iter().zip(LEAVES4.into_iter())) + .unwrap(); + + // STORE LEAVES ARE CORRECT ============================================================== + // checks the leaves in the store corresponds to the expected values + assert_eq!( + store.get_node(smt.root(), NodeIndex::new(smt.depth(), 0)), + Ok(LEAVES4[0]), + "node 0 must be in the tree" + ); + assert_eq!( + store.get_node(smt.root(), NodeIndex::new(smt.depth(), 1)), + Ok(LEAVES4[1]), + "node 1 must be in the tree" + ); + assert_eq!( + store.get_node(smt.root(), NodeIndex::new(smt.depth(), 2)), + Ok(LEAVES4[2]), + "node 2 must be in the tree" + ); + assert_eq!( + store.get_node(smt.root(), NodeIndex::new(smt.depth(), 3)), + Ok(LEAVES4[3]), + "node 3 must be in the tree" + ); + + // STORE LEAVES MATCH TREE =============================================================== + // sanity check the values returned by the store and the tree + assert_eq!( + smt.get_node(&NodeIndex::new(smt.depth(), 0)), + store.get_node(smt.root(), NodeIndex::new(smt.depth(), 0)), + "node 0 must be the same for both SparseMerkleTree and MerkleStore" + ); + assert_eq!( + smt.get_node(&NodeIndex::new(smt.depth(), 1)), + store.get_node(smt.root(), NodeIndex::new(smt.depth(), 1)), + "node 1 must be the same for both SparseMerkleTree and MerkleStore" + ); + assert_eq!( + smt.get_node(&NodeIndex::new(smt.depth(), 2)), + store.get_node(smt.root(), NodeIndex::new(smt.depth(), 2)), + "node 2 must be the same for both SparseMerkleTree and MerkleStore" + ); + assert_eq!( + smt.get_node(&NodeIndex::new(smt.depth(), 3)), + store.get_node(smt.root(), NodeIndex::new(smt.depth(), 3)), + "node 3 must be the same for both SparseMerkleTree and MerkleStore" + ); + + // STORE MERKLE PATH MATCHS ============================================================== + // assert the merkle path returned by the store is the same as the one in the tree + let result = store + .get_path(smt.root(), NodeIndex::new(smt.depth(), 0)) + .unwrap(); + assert_eq!( + LEAVES4[0], result.0, + "Value for merkle path at index 0 must match leaf value" + ); + assert_eq!( + smt.get_path(NodeIndex::new(smt.depth(), 0)), + Ok(result.1), + "merkle path for index 0 must be the same for the MerkleTree and MerkleStore" + ); + + let result = store + .get_path(smt.root(), NodeIndex::new(smt.depth(), 1)) + .unwrap(); + assert_eq!( + LEAVES4[1], result.0, + "Value for merkle path at index 0 must match leaf value" + ); + assert_eq!( + smt.get_path(NodeIndex::new(smt.depth(), 1)), + Ok(result.1), + "merkle path for index 1 must be the same for the MerkleTree and MerkleStore" + ); + + let result = store + .get_path(smt.root(), NodeIndex::new(smt.depth(), 2)) + .unwrap(); + assert_eq!( + LEAVES4[2], result.0, + "Value for merkle path at index 0 must match leaf value" + ); + assert_eq!( + smt.get_path(NodeIndex::new(smt.depth(), 2)), + Ok(result.1), + "merkle path for index 0 must be the same for the MerkleTree and MerkleStore" + ); + + let result = store + .get_path(smt.root(), NodeIndex::new(smt.depth(), 3)) + .unwrap(); + assert_eq!( + LEAVES4[3], result.0, + "Value for merkle path at index 0 must match leaf value" + ); + assert_eq!( + smt.get_path(NodeIndex::new(smt.depth(), 3)), + Ok(result.1), + "merkle path for index 0 must be the same for the MerkleTree and MerkleStore" + ); + + Ok(()) +} + +#[test] +fn test_add_merkle_paths() -> Result<(), MerkleError> { + let mtree = MerkleTree::new(LEAVES4.to_vec())?; + + let i0 = 0; + let p0 = mtree.get_path(NodeIndex::new(2, i0)).unwrap(); + + let i1 = 1; + let p1 = mtree.get_path(NodeIndex::new(2, i1)).unwrap(); + + let i2 = 2; + let p2 = mtree.get_path(NodeIndex::new(2, i2)).unwrap(); + + let i3 = 3; + let p3 = mtree.get_path(NodeIndex::new(2, i3)).unwrap(); + + let paths = [ + (i0, LEAVES4[i0 as usize], p0), + (i1, LEAVES4[i1 as usize], p1), + (i2, LEAVES4[i2 as usize], p2), + (i3, LEAVES4[i3 as usize], p3), + ]; + + let mut store = MerkleStore::default(); + store + .add_merkle_paths(paths.clone()) + .expect("the valid paths must work"); + + let depth = 3; + let set = MerklePathSet::new(depth).with_paths(paths).unwrap(); + + // STORE LEAVES ARE CORRECT ============================================================== + // checks the leaves in the store corresponds to the expected values + assert_eq!( + store.get_node(set.root(), NodeIndex::new(set.depth() - 1, 0)), + Ok(LEAVES4[0]), + "node 0 must be in the set" + ); + assert_eq!( + store.get_node(set.root(), NodeIndex::new(set.depth() - 1, 1)), + Ok(LEAVES4[1]), + "node 1 must be in the set" + ); + assert_eq!( + store.get_node(set.root(), NodeIndex::new(set.depth() - 1, 2)), + Ok(LEAVES4[2]), + "node 2 must be in the set" + ); + assert_eq!( + store.get_node(set.root(), NodeIndex::new(set.depth() - 1, 3)), + Ok(LEAVES4[3]), + "node 3 must be in the set" + ); + + // STORE LEAVES MATCH SET ================================================================ + // sanity check the values returned by the store and the set + assert_eq!( + set.get_node(NodeIndex::new(set.depth(), 0)), + store.get_node(set.root(), NodeIndex::new(set.depth() - 1, 0)), + "node 0 must be the same for both SparseMerkleTree and MerkleStore" + ); + assert_eq!( + set.get_node(NodeIndex::new(set.depth(), 1)), + store.get_node(set.root(), NodeIndex::new(set.depth() - 1, 1)), + "node 1 must be the same for both SparseMerkleTree and MerkleStore" + ); + assert_eq!( + set.get_node(NodeIndex::new(set.depth(), 2)), + store.get_node(set.root(), NodeIndex::new(set.depth() - 1, 2)), + "node 2 must be the same for both SparseMerkleTree and MerkleStore" + ); + assert_eq!( + set.get_node(NodeIndex::new(set.depth(), 3)), + store.get_node(set.root(), NodeIndex::new(set.depth() - 1, 3)), + "node 3 must be the same for both SparseMerkleTree and MerkleStore" + ); + + // STORE MERKLE PATH MATCHS ============================================================== + // assert the merkle path returned by the store is the same as the one in the set + let result = store + .get_path(set.root(), NodeIndex::new(set.depth() - 1, 0)) + .unwrap(); + assert_eq!( + LEAVES4[0], result.0, + "Value for merkle path at index 0 must match leaf value" + ); + assert_eq!( + set.get_path(NodeIndex::new(set.depth(), 0)), + Ok(result.1), + "merkle path for index 0 must be the same for the MerkleTree and MerkleStore" + ); + + let result = store + .get_path(set.root(), NodeIndex::new(set.depth() - 1, 1)) + .unwrap(); + assert_eq!( + LEAVES4[1], result.0, + "Value for merkle path at index 0 must match leaf value" + ); + assert_eq!( + set.get_path(NodeIndex::new(set.depth(), 1)), + Ok(result.1), + "merkle path for index 1 must be the same for the MerkleTree and MerkleStore" + ); + + let result = store + .get_path(set.root(), NodeIndex::new(set.depth() - 1, 2)) + .unwrap(); + assert_eq!( + LEAVES4[2], result.0, + "Value for merkle path at index 0 must match leaf value" + ); + assert_eq!( + set.get_path(NodeIndex::new(set.depth(), 2)), + Ok(result.1), + "merkle path for index 0 must be the same for the MerkleTree and MerkleStore" + ); + + let result = store + .get_path(set.root(), NodeIndex::new(set.depth() - 1, 3)) + .unwrap(); + assert_eq!( + LEAVES4[3], result.0, + "Value for merkle path at index 0 must match leaf value" + ); + assert_eq!( + set.get_path(NodeIndex::new(set.depth(), 3)), + Ok(result.1), + "merkle path for index 0 must be the same for the MerkleTree and MerkleStore" + ); + + Ok(()) +} + +#[test] +fn wont_open_to_different_depth_root() { + let empty = EmptySubtreeRoots::empty_hashes(64); + let a = [Felt::new(1); 4]; + let b = [Felt::new(2); 4]; + + // Compute the root for a different depth. We cherry-pick this specific depth to prevent a + // regression to a bug in the past that allowed the user to fetch a node at a depth lower than + // the inserted path of a Merkle tree. + let mut root = Rpo256::merge(&[a.into(), b.into()]); + for depth in (1..=63).rev() { + root = Rpo256::merge(&[root, empty[depth]]); + } + let root = Word::from(root); + + // For this example, the depth of the Merkle tree is 1, as we have only two leaves. Here we + // attempt to fetch a node on the maximum depth, and it should fail because the root shouldn't + // exist for the set. + let store = MerkleStore::default().with_merkle_tree([a, b]).unwrap(); + let index = NodeIndex::root(); + let err = store.get_node(root, index).err().unwrap(); + assert_eq!(err, MerkleError::RootNotInStore(root)); +} + +#[test] +fn store_path_opens_from_leaf() { + let a = [Felt::new(1); 4]; + let b = [Felt::new(2); 4]; + let c = [Felt::new(3); 4]; + let d = [Felt::new(4); 4]; + let e = [Felt::new(5); 4]; + let f = [Felt::new(6); 4]; + let g = [Felt::new(7); 4]; + let h = [Felt::new(8); 4]; + + let i = Rpo256::merge(&[a.into(), b.into()]); + let j = Rpo256::merge(&[c.into(), d.into()]); + let k = Rpo256::merge(&[e.into(), f.into()]); + let l = Rpo256::merge(&[g.into(), h.into()]); + + let m = Rpo256::merge(&[i.into(), j.into()]); + let n = Rpo256::merge(&[k.into(), l.into()]); + + let root = Rpo256::merge(&[m.into(), n.into()]); + + let store = MerkleStore::default() + .with_merkle_tree([a, b, c, d, e, f, g, h]) + .unwrap(); + let path = store.get_path(root.into(), NodeIndex::new(3, 1)).unwrap(); + + let expected = MerklePath::new([a.into(), j.into(), n.into()].to_vec()); + assert_eq!(path.1, expected); +} + +#[test] +fn test_set_node() -> Result<(), MerkleError> { + let mtree = MerkleTree::new(LEAVES4.to_vec())?; + let mut store = MerkleStore::default().with_merkle_tree(LEAVES4)?; + let value = int_to_node(42); + let index = NodeIndex::new(mtree.depth(), 0); + let new_root = store.set_node(mtree.root(), index, value)?; + assert_eq!( + store.get_node(new_root, index), + Ok(value), + "Value must have changed" + ); + + Ok(()) +} From 17eb8d78d3ec9d4fa5e501f2c104cdcff77fffe9 Mon Sep 17 00:00:00 2001 From: "Augusto F. Hack" Date: Mon, 20 Mar 2023 21:55:57 +0100 Subject: [PATCH 12/26] chore: storage -> store --- src/merkle/mod.rs | 4 ++-- src/merkle/store/mod.rs | 18 +++++++++--------- src/merkle/store/tests.rs | 2 +- 3 files changed, 12 insertions(+), 12 deletions(-) diff --git a/src/merkle/mod.rs b/src/merkle/mod.rs index 411dd7e..0afbce9 100644 --- a/src/merkle/mod.rs +++ b/src/merkle/mod.rs @@ -71,8 +71,8 @@ impl fmt::Display for MerkleError { InvalidPath(_path) => write!(f, "the provided path is not valid"), InvalidEntriesCount(max, provided) => write!(f, "the provided number of entries is {provided}, but the maximum for the given depth is {max}"), NodeNotInSet(index) => write!(f, "the node indexed by {index} is not in the set"), - NodeNotInStore(hash, index) => write!(f, "the node {:?} indexed by {} and depth {} is not in the storage", hash, index.value(), index.depth(),), - RootNotInStore(root) => write!(f, "the root {:?} is not in the storage", root), + NodeNotInStore(hash, index) => write!(f, "the node {:?} indexed by {} and depth {} is not in the store", hash, index.value(), index.depth(),), + RootNotInStore(root) => write!(f, "the root {:?} is not in the store", root), } } } diff --git a/src/merkle/store/mod.rs b/src/merkle/store/mod.rs index 961d4b8..5610429 100644 --- a/src/merkle/store/mod.rs +++ b/src/merkle/store/mod.rs @@ -92,12 +92,12 @@ impl MerkleStore { /// # Errors /// /// This method can return the following errors: - /// - `RootNotInStore` if the `root` is not present in the storage. + /// - `RootNotInStore` if the `root` is not present in the store. /// - `NodeNotInStore` if a node needed to traverse from `root` to `index` is not present in the store. pub fn get_node(&self, root: Word, index: NodeIndex) -> Result { let mut hash: RpoDigest = root.into(); - // corner case: check the root is in the storage when called with index `NodeIndex::root()` + // corner case: check the root is in the store when called with index `NodeIndex::root()` self.nodes .get(&hash) .ok_or(MerkleError::RootNotInStore(hash.into()))?; @@ -120,7 +120,7 @@ impl MerkleStore { /// # Errors /// /// This method can return the following errors: - /// - `RootNotInStore` if the `root` is not present in the storage. + /// - `RootNotInStore` if the `root` is not present in the store. /// - `NodeNotInStore` if a node needed to traverse from `root` to `index` is not present in the store. pub fn get_path( &self, @@ -130,7 +130,7 @@ impl MerkleStore { let mut hash: RpoDigest = root.into(); let mut path = Vec::with_capacity(index.depth().into()); - // corner case: check the root is in the storage when called with index `NodeIndex::root()` + // corner case: check the root is in the store when called with index `NodeIndex::root()` self.nodes .get(&hash) .ok_or(MerkleError::RootNotInStore(hash.into()))?; @@ -160,7 +160,7 @@ impl MerkleStore { /// Adds all the nodes of a Merkle tree represented by `leaves`. /// /// This will instantiate a Merkle tree using `leaves` and include all the nodes into the - /// storage. + /// store. /// /// # Errors /// @@ -208,7 +208,7 @@ impl MerkleStore { /// Adds all the nodes of a Sparse Merkle tree represented by `entries`. /// /// This will instantiate a Sparse Merkle tree using `entries` and include all the nodes into - /// the storage. + /// the store. /// /// # Errors /// @@ -236,7 +236,7 @@ impl MerkleStore { /// Adds all the nodes of a Merkle path represented by `path`. /// /// This will compute the sibling elements determined by the Merkle `path` and `node`, and - /// include all the nodes into the storage. + /// include all the nodes into the store. pub fn add_merkle_path( &mut self, index_value: u64, @@ -270,7 +270,7 @@ impl MerkleStore { /// Adds all the nodes of multiple Merkle paths into the store. /// /// This will compute the sibling elements for each Merkle `path` and include all the nodes - /// into the storage. + /// into the store. /// /// # Errors /// @@ -315,7 +315,7 @@ impl MerkleStore { /// # Errors /// /// This method can return the following errors: - /// - `RootNotInStore` if the `root` is not present in the storage. + /// - `RootNotInStore` if the `root` is not present in the store. /// - `NodeNotInStore` if a node needed to traverse from `root` to `index` is not present in the store. pub fn set_node( &mut self, diff --git a/src/merkle/store/tests.rs b/src/merkle/store/tests.rs index a60adb4..c469553 100644 --- a/src/merkle/store/tests.rs +++ b/src/merkle/store/tests.rs @@ -14,7 +14,7 @@ const LEAVES4: [Word; 4] = [ ]; #[test] -fn test_root_not_in_storage() -> Result<(), MerkleError> { +fn test_root_not_in_store() -> Result<(), MerkleError> { let mtree = MerkleTree::new(LEAVES4.to_vec())?; let store = MerkleStore::default().with_merkle_tree(LEAVES4)?; assert_eq!( From 78e82f2ee6f83ba7c463ef9bf98fa58fc497e8d2 Mon Sep 17 00:00:00 2001 From: "Augusto F. Hack" Date: Fri, 17 Mar 2023 01:19:55 +0100 Subject: [PATCH 13/26] feat: add benchmark for storages --- Cargo.toml | 4 + benches/store.rs | 504 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 508 insertions(+) create mode 100644 benches/store.rs diff --git a/Cargo.toml b/Cargo.toml index 5886cd0..c5901e3 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -20,6 +20,10 @@ harness = false name = "smt" harness = false +[[bench]] +name = "store" +harness = false + [features] default = ["blake3/default", "std", "winter_crypto/default", "winter_math/default", "winter_utils/default"] std = ["blake3/std", "winter_crypto/std", "winter_math/std", "winter_utils/std"] diff --git a/benches/store.rs b/benches/store.rs new file mode 100644 index 0000000..0e60b04 --- /dev/null +++ b/benches/store.rs @@ -0,0 +1,504 @@ +use criterion::{black_box, criterion_group, criterion_main, BatchSize, BenchmarkId, Criterion}; +use miden_crypto::merkle::{MerkleStore, MerkleTree, NodeIndex, SimpleSmt}; +use miden_crypto::Word; +use miden_crypto::{hash::rpo::RpoDigest, Felt}; +use rand_utils::{rand_array, rand_value}; + +/// Since MerkleTree can only be created when a power-of-two number of elements is used, the sample +/// sizes are limited to that. +static BATCH_SIZES: [usize; 3] = [2usize.pow(4), 2usize.pow(7), 2usize.pow(10)]; + +/// Generates a random `RpoDigest`. +fn random_rpo_digest() -> RpoDigest { + rand_array::().into() +} + +/// Generates a random `Word`. +fn random_word() -> Word { + rand_array::().into() +} + +/// Generates a u64 in `0..range`. +fn random_index(range: u64) -> u64 { + rand_value::() % range +} + +/// Benchmarks getting an empty leaf from the SMT and MerkleStore backends. +fn get_empty_leaf_simplesmt(c: &mut Criterion) { + let mut group = c.benchmark_group("get_empty_leaf_simplesmt"); + + let depth = 63u8; + let size = 2u64.pow(depth as u32); + + // both SMT and the store are pre-populated with empty hashes, accessing these values is what is + // being benchmarked here, so no values are inserted into the backends + let smt = SimpleSmt::new(depth).unwrap(); + let store = MerkleStore::new(); + let root = smt.root(); + + group.bench_function(BenchmarkId::new("SimpleSmt", depth), |b| { + b.iter_batched( + || random_index(size), + |value| black_box(smt.get_node(&NodeIndex::new(depth, value))), + BatchSize::SmallInput, + ) + }); + + group.bench_function(BenchmarkId::new("MerkleStore", depth), |b| { + b.iter_batched( + || random_index(size), + |value| black_box(store.get_node(root, NodeIndex::new(depth, value))), + BatchSize::SmallInput, + ) + }); +} + +/// Benchmarks getting a leaf on Merkle trees and Merkle stores of varying power-of-two sizes. +fn get_leaf_merkletree(c: &mut Criterion) { + let mut group = c.benchmark_group("get_leaf_merkletree"); + + let random_data_size = BATCH_SIZES.into_iter().max().unwrap(); + let random_data: Vec = (0..random_data_size).map(|_| random_rpo_digest()).collect(); + + for size in BATCH_SIZES { + let leaves = &random_data[..size]; + + let mtree_leaves: Vec = leaves.iter().map(|v| v.into()).collect(); + let mtree = MerkleTree::new(mtree_leaves.clone()).unwrap(); + let store = MerkleStore::new().with_merkle_tree(mtree_leaves).unwrap(); + let depth = mtree.depth(); + let root = mtree.root(); + let size_u64 = size as u64; + + group.bench_function(BenchmarkId::new("MerkleTree", size), |b| { + b.iter_batched( + || random_index(size_u64), + |value| black_box(mtree.get_node(NodeIndex::new(depth, value))), + BatchSize::SmallInput, + ) + }); + + group.bench_function(BenchmarkId::new("MerkleStore", size), |b| { + b.iter_batched( + || random_index(size_u64), + |value| black_box(store.get_node(root, NodeIndex::new(depth, value))), + BatchSize::SmallInput, + ) + }); + } +} + +/// Benchmarks getting a leaf on SMT and Merkle stores of varying power-of-two sizes. +fn get_leaf_simplesmt(c: &mut Criterion) { + let mut group = c.benchmark_group("get_leaf_simplesmt"); + + let random_data_size = BATCH_SIZES.into_iter().max().unwrap(); + let random_data: Vec = (0..random_data_size).map(|_| random_rpo_digest()).collect(); + + for size in BATCH_SIZES { + let leaves = &random_data[..size]; + + let smt_leaves = leaves + .iter() + .enumerate() + .map(|(c, v)| (c.try_into().unwrap(), v.into())) + .collect::>(); + let smt = SimpleSmt::new(63) + .unwrap() + .with_leaves(smt_leaves.clone()) + .unwrap(); + let store = MerkleStore::new() + .with_sparse_merkle_tree(smt_leaves) + .unwrap(); + let depth = smt.depth(); + let root = smt.root(); + let size_u64 = size as u64; + + group.bench_function(BenchmarkId::new("SimpleSmt", size), |b| { + b.iter_batched( + || random_index(size_u64), + |value| black_box(smt.get_node(&NodeIndex::new(depth, value))), + BatchSize::SmallInput, + ) + }); + + group.bench_function(BenchmarkId::new("MerkleStore", size), |b| { + b.iter_batched( + || random_index(size_u64), + |value| black_box(store.get_node(root, NodeIndex::new(depth, value))), + BatchSize::SmallInput, + ) + }); + } +} + +/// Benchmarks getting a node at half of the depth of an empty SMT and an empty Merkle store. +fn get_node_of_empty_simplesmt(c: &mut Criterion) { + let mut group = c.benchmark_group("get_node_of_empty_simplesmt"); + + let depth = 63u8; + let size = 2u64.pow(depth as u32); + + // both SMT and the store are pre-populated with the empty hashes, accessing the internal nodes + // of these values is what is being benchmarked here, so no values are inserted into the + // backends. + let smt = SimpleSmt::new(depth).unwrap(); + let store = MerkleStore::new(); + let root = smt.root(); + let half_depth = depth / 2; + + group.bench_function(BenchmarkId::new("SimpleSmt", depth), |b| { + b.iter_batched( + || random_index(size), + |value| black_box(smt.get_node(&NodeIndex::new(half_depth, value))), + BatchSize::SmallInput, + ) + }); + + group.bench_function(BenchmarkId::new("MerkleStore", depth), |b| { + b.iter_batched( + || random_index(size), + |value| black_box(store.get_node(root, NodeIndex::new(half_depth, value))), + BatchSize::SmallInput, + ) + }); +} + +/// Benchmarks getting a node at half of the depth of a Merkle tree and Merkle store of varying +/// power-of-two sizes. +fn get_node_merkletree(c: &mut Criterion) { + let mut group = c.benchmark_group("get_node_merkletree"); + + let random_data_size = BATCH_SIZES.into_iter().max().unwrap(); + let random_data: Vec = (0..random_data_size).map(|_| random_rpo_digest()).collect(); + + for size in BATCH_SIZES { + let leaves = &random_data[..size]; + + let mtree_leaves: Vec = leaves.iter().map(|v| v.into()).collect(); + let mtree = MerkleTree::new(mtree_leaves.clone()).unwrap(); + let store = MerkleStore::new().with_merkle_tree(mtree_leaves).unwrap(); + let half_depth = mtree.depth() / 2; + let root = mtree.root(); + let size_u64 = size as u64; + + group.bench_function(BenchmarkId::new("MerkleTree", size), |b| { + b.iter_batched( + || random_index(size_u64), + |value| black_box(mtree.get_node(NodeIndex::new(half_depth, value))), + BatchSize::SmallInput, + ) + }); + + group.bench_function(BenchmarkId::new("MerkleStore", size), |b| { + b.iter_batched( + || random_index(size_u64), + |value| black_box(store.get_node(root, NodeIndex::new(half_depth, value))), + BatchSize::SmallInput, + ) + }); + } +} + +/// Benchmarks getting a node at half the depth on SMT and Merkle stores of varying power-of-two +/// sizes. +fn get_node_simplesmt(c: &mut Criterion) { + let mut group = c.benchmark_group("get_node_simplesmt"); + + let random_data_size = BATCH_SIZES.into_iter().max().unwrap(); + let random_data: Vec = (0..random_data_size).map(|_| random_rpo_digest()).collect(); + + for size in BATCH_SIZES { + let leaves = &random_data[..size]; + + let smt_leaves = leaves + .iter() + .enumerate() + .map(|(c, v)| (c.try_into().unwrap(), v.into())) + .collect::>(); + let smt = SimpleSmt::new(63) + .unwrap() + .with_leaves(smt_leaves.clone()) + .unwrap(); + let store = MerkleStore::new() + .with_sparse_merkle_tree(smt_leaves) + .unwrap(); + let root = smt.root(); + let size_u64 = size as u64; + let half_depth = smt.depth() / 2; + + group.bench_function(BenchmarkId::new("SimpleSmt", size), |b| { + b.iter_batched( + || random_index(size_u64), + |value| black_box(smt.get_node(&NodeIndex::new(half_depth, value))), + BatchSize::SmallInput, + ) + }); + + group.bench_function(BenchmarkId::new("MerkleStore", size), |b| { + b.iter_batched( + || random_index(size_u64), + |value| black_box(store.get_node(root, NodeIndex::new(half_depth, value))), + BatchSize::SmallInput, + ) + }); + } +} + +/// Benchmarks getting a path of a leaf on the Merkle tree and Merkle store backends. +fn get_leaf_path_merkletree(c: &mut Criterion) { + let mut group = c.benchmark_group("get_leaf_path_merkletree"); + + let random_data_size = BATCH_SIZES.into_iter().max().unwrap(); + let random_data: Vec = (0..random_data_size).map(|_| random_rpo_digest()).collect(); + + for size in BATCH_SIZES { + let leaves = &random_data[..size]; + + let mtree_leaves: Vec = leaves.iter().map(|v| v.into()).collect(); + let mtree = MerkleTree::new(mtree_leaves.clone()).unwrap(); + let store = MerkleStore::new().with_merkle_tree(mtree_leaves).unwrap(); + let depth = mtree.depth(); + let root = mtree.root(); + let size_u64 = size as u64; + + group.bench_function(BenchmarkId::new("MerkleTree", size), |b| { + b.iter_batched( + || random_index(size_u64), + |value| black_box(mtree.get_path(NodeIndex::new(depth, value))), + BatchSize::SmallInput, + ) + }); + + group.bench_function(BenchmarkId::new("MerkleStore", size), |b| { + b.iter_batched( + || random_index(size_u64), + |value| black_box(store.get_path(root, NodeIndex::new(depth, value))), + BatchSize::SmallInput, + ) + }); + } +} + +/// Benchmarks getting a path of a leaf on the SMT and Merkle store backends. +fn get_leaf_path_simplesmt(c: &mut Criterion) { + let mut group = c.benchmark_group("get_leaf_path_simplesmt"); + + let random_data_size = BATCH_SIZES.into_iter().max().unwrap(); + let random_data: Vec = (0..random_data_size).map(|_| random_rpo_digest()).collect(); + + for size in BATCH_SIZES { + let leaves = &random_data[..size]; + + let smt_leaves = leaves + .iter() + .enumerate() + .map(|(c, v)| (c.try_into().unwrap(), v.into())) + .collect::>(); + let smt = SimpleSmt::new(63) + .unwrap() + .with_leaves(smt_leaves.clone()) + .unwrap(); + let store = MerkleStore::new() + .with_sparse_merkle_tree(smt_leaves) + .unwrap(); + let depth = smt.depth(); + let root = smt.root(); + let size_u64 = size as u64; + + group.bench_function(BenchmarkId::new("SimpleSmt", size), |b| { + b.iter_batched( + || random_index(size_u64), + |value| black_box(smt.get_path(NodeIndex::new(depth, value))), + BatchSize::SmallInput, + ) + }); + + group.bench_function(BenchmarkId::new("MerkleStore", size), |b| { + b.iter_batched( + || random_index(size_u64), + |value| black_box(store.get_path(root, NodeIndex::new(depth, value))), + BatchSize::SmallInput, + ) + }); + } +} + +/// Benchmarks creation of the different storage backends +fn new(c: &mut Criterion) { + let mut group = c.benchmark_group("new"); + + let random_data_size = BATCH_SIZES.into_iter().max().unwrap(); + let random_data: Vec = (0..random_data_size).map(|_| random_rpo_digest()).collect(); + + for size in BATCH_SIZES { + let leaves = &random_data[..size]; + + // MerkleTree constructor is optimized to work with vectors. Create a new copy of the data + // and pass it to the benchmark function + group.bench_function(BenchmarkId::new("MerkleTree::new", size), |b| { + b.iter_batched( + || leaves.iter().map(|v| v.into()).collect::>(), + |l| black_box(MerkleTree::new(l)), + BatchSize::SmallInput, + ) + }); + + // This could be done with `bench_with_input`, however to remove variables while comparing + // with MerkleTree it is using `iter_batched` + group.bench_function( + BenchmarkId::new("MerkleStore::with_merkle_tree", size), + |b| { + b.iter_batched( + || leaves.iter().map(|v| v.into()).collect::>(), + |l| black_box(MerkleStore::new().with_merkle_tree(l)), + BatchSize::SmallInput, + ) + }, + ); + + group.bench_function(BenchmarkId::new("SimpleSmt::new", size), |b| { + b.iter_batched( + || { + leaves + .iter() + .enumerate() + .map(|(c, v)| (c.try_into().unwrap(), v.into())) + .collect::>() + }, + |l| black_box(SimpleSmt::new(63).unwrap().with_leaves(l)), + BatchSize::SmallInput, + ) + }); + + group.bench_function( + BenchmarkId::new("MerkleStore::with_sparse_merkle_tree", size), + |b| { + b.iter_batched( + || { + leaves + .iter() + .enumerate() + .map(|(c, v)| (c.try_into().unwrap(), v.into())) + .collect::>() + }, + |l| black_box(MerkleStore::new().with_sparse_merkle_tree(l)), + BatchSize::SmallInput, + ) + }, + ); + } +} + +/// Benchmarks updating a leaf on MerkleTree and MerkleStore backends. +fn update_leaf_merkletree(c: &mut Criterion) { + let mut group = c.benchmark_group("update_leaf_merkletree"); + + let random_data_size = BATCH_SIZES.into_iter().max().unwrap(); + let random_data: Vec = (0..random_data_size).map(|_| random_rpo_digest()).collect(); + + for size in BATCH_SIZES { + let leaves = &random_data[..size]; + + let mtree_leaves: Vec = leaves.iter().map(|v| v.into()).collect(); + let mut mtree = MerkleTree::new(mtree_leaves.clone()).unwrap(); + let mut store = MerkleStore::new().with_merkle_tree(mtree_leaves).unwrap(); + let depth = mtree.depth(); + let root = mtree.root(); + let size_u64 = size as u64; + + group.bench_function(BenchmarkId::new("MerkleTree", size), |b| { + b.iter_batched( + || (random_index(size_u64), random_word()), + |(index, value)| black_box(mtree.update_leaf(index, value)), + BatchSize::SmallInput, + ) + }); + + let mut store_root = root; + group.bench_function(BenchmarkId::new("MerkleStore", size), |b| { + b.iter_batched( + || (random_index(size_u64), random_word()), + |(index, value)| { + // The MerkleTree automatically updates its internal root, the Store maintains + // the old root and adds the new one. Here we update the root to have a fair + // comparison + store_root = store + .set_node(root, NodeIndex::new(depth, index), value) + .unwrap(); + black_box(store_root) + }, + BatchSize::SmallInput, + ) + }); + } +} + +/// Benchmarks updating a leaf on SMT and MerkleStore backends. +fn update_leaf_simplesmt(c: &mut Criterion) { + let mut group = c.benchmark_group("update_leaf_simplesmt"); + + let random_data_size = BATCH_SIZES.into_iter().max().unwrap(); + let random_data: Vec = (0..random_data_size).map(|_| random_rpo_digest()).collect(); + + for size in BATCH_SIZES { + let leaves = &random_data[..size]; + + let smt_leaves = leaves + .iter() + .enumerate() + .map(|(c, v)| (c.try_into().unwrap(), v.into())) + .collect::>(); + let mut smt = SimpleSmt::new(63) + .unwrap() + .with_leaves(smt_leaves.clone()) + .unwrap(); + let mut store = MerkleStore::new() + .with_sparse_merkle_tree(smt_leaves) + .unwrap(); + let depth = smt.depth(); + let root = smt.root(); + let size_u64 = size as u64; + + group.bench_function(BenchmarkId::new("SimpleSMT", size), |b| { + b.iter_batched( + || (random_index(size_u64), random_word()), + |(index, value)| black_box(smt.update_leaf(index, value)), + BatchSize::SmallInput, + ) + }); + + let mut store_root = root; + group.bench_function(BenchmarkId::new("MerkleStore", size), |b| { + b.iter_batched( + || (random_index(size_u64), random_word()), + |(index, value)| { + // The MerkleTree automatically updates its internal root, the Store maintains + // the old root and adds the new one. Here we update the root to have a fair + // comparison + store_root = store + .set_node(root, NodeIndex::new(depth, index), value) + .unwrap(); + black_box(store_root) + }, + BatchSize::SmallInput, + ) + }); + } +} + +criterion_group!( + store_group, + get_empty_leaf_simplesmt, + get_leaf_merkletree, + get_leaf_path_merkletree, + get_leaf_path_simplesmt, + get_leaf_simplesmt, + get_node_merkletree, + get_node_of_empty_simplesmt, + get_node_simplesmt, + new, + update_leaf_merkletree, + update_leaf_simplesmt, +); +criterion_main!(store_group); From 84086bdb9520da89b321fe6f95250f5d8f0d50bf Mon Sep 17 00:00:00 2001 From: Victor Lopez Date: Tue, 21 Mar 2023 10:40:48 +0100 Subject: [PATCH 14/26] feat: add merkle path containers and return them on tree update Returning tuples is often confusing as they don't convey meaning and it should be used only when there is no possible ambiguity. For `MerkleStore`, we had a couple of tuples being returned, and reading the implementation was required in order to distinguish if they were leaf values or computed roots. This commit introduces two containers that will self-document these returns: `RootPath` and `ValuePath`. It will also update `set_node` to return both the new root & the new path, so we can prevent duplicated traversals downstream when updating a node (one to update, the second to fetch the new path/root). --- src/merkle/mod.rs | 2 +- src/merkle/path.rs | 24 +++++++++++++++++ src/merkle/store/mod.rs | 42 +++++++++++++++++------------ src/merkle/store/tests.rs | 57 ++++++++++++++++++++------------------- 4 files changed, 80 insertions(+), 45 deletions(-) diff --git a/src/merkle/mod.rs b/src/merkle/mod.rs index 0afbce9..5b204ac 100644 --- a/src/merkle/mod.rs +++ b/src/merkle/mod.rs @@ -18,7 +18,7 @@ mod merkle_tree; pub use merkle_tree::MerkleTree; mod path; -pub use path::MerklePath; +pub use path::{MerklePath, RootPath, ValuePath}; mod path_set; pub use path_set::MerklePathSet; diff --git a/src/merkle/path.rs b/src/merkle/path.rs index d7edd5d..9a4e46b 100644 --- a/src/merkle/path.rs +++ b/src/merkle/path.rs @@ -82,3 +82,27 @@ impl IntoIterator for MerklePath { self.nodes.into_iter() } } + +// MERKLE PATH CONTAINERS +// ================================================================================================ + +/// A container for a [Word] value and its [MerklePath] opening. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct ValuePath { + /// The node value opening for `path`. + pub value: Word, + /// The path from `value` to `root` (exclusive). + pub path: MerklePath, +} + +/// A container for a [MerklePath] and its [Word] root. +/// +/// This structure does not provide any guarantees regarding the correctness of the path to the +/// root. For more information, check [MerklePath::verify]. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct RootPath { + /// The node value opening for `path`. + pub root: Word, + /// The path from `value` to `root` (exclusive). + pub path: MerklePath, +} diff --git a/src/merkle/store/mod.rs b/src/merkle/store/mod.rs index 5610429..13ca5f9 100644 --- a/src/merkle/store/mod.rs +++ b/src/merkle/store/mod.rs @@ -5,7 +5,7 @@ //! implementation of efficient persistent data structures use super::{ BTreeMap, BTreeSet, EmptySubtreeRoots, MerkleError, MerklePath, MerklePathSet, MerkleTree, - NodeIndex, Rpo256, RpoDigest, SimpleSmt, Vec, Word, + NodeIndex, RootPath, Rpo256, RpoDigest, SimpleSmt, ValuePath, Vec, Word, }; #[cfg(test)] @@ -122,11 +122,7 @@ impl MerkleStore { /// This method can return the following errors: /// - `RootNotInStore` if the `root` is not present in the store. /// - `NodeNotInStore` if a node needed to traverse from `root` to `index` is not present in the store. - pub fn get_path( - &self, - root: Word, - index: NodeIndex, - ) -> Result<(Word, MerklePath), MerkleError> { + pub fn get_path(&self, root: Word, index: NodeIndex) -> Result { let mut hash: RpoDigest = root.into(); let mut path = Vec::with_capacity(index.depth().into()); @@ -150,8 +146,13 @@ impl MerkleStore { } } + // the path is computed from root to leaf, so it must be reversed path.reverse(); - Ok((hash.into(), MerklePath::new(path))) + + Ok(ValuePath { + value: hash.into(), + path: MerklePath::new(path), + }) } // STATE MUTATORS @@ -233,17 +234,17 @@ impl MerkleStore { Ok(smt.root()) } - /// Adds all the nodes of a Merkle path represented by `path`. + /// Adds all the nodes of a Merkle path represented by `path`, opening to `node`. Returns the + /// new root. /// /// This will compute the sibling elements determined by the Merkle `path` and `node`, and /// include all the nodes into the store. pub fn add_merkle_path( &mut self, index_value: u64, - node: Word, + mut node: Word, path: MerklePath, ) -> Result { - let mut node = node; let mut index = NodeIndex::new(self.nodes.len() as u8, index_value); for sibling in path { @@ -272,6 +273,8 @@ impl MerkleStore { /// This will compute the sibling elements for each Merkle `path` and include all the nodes /// into the store. /// + /// For further reference, check [MerkleStore::add_merkle_path]. + /// /// # Errors /// /// Every path must resolve to the same root, otherwise this will return an `ConflictingRoots` @@ -301,6 +304,8 @@ impl MerkleStore { } /// Appends the provided [MerklePathSet] into the store. + /// + /// For further reference, check [MerkleStore::add_merkle_path]. pub fn add_merkle_path_set(&mut self, path_set: &MerklePathSet) -> Result { let root = path_set.root(); path_set.indexes().try_fold(root, |_, index| { @@ -319,16 +324,19 @@ impl MerkleStore { /// - `NodeNotInStore` if a node needed to traverse from `root` to `index` is not present in the store. pub fn set_node( &mut self, - root: Word, + mut root: Word, index: NodeIndex, value: Word, - ) -> Result { - let result = self.get_path(root, index)?; - if result.0 != value { - self.add_merkle_path(index.value(), value, result.1) - } else { - Ok(root) + ) -> Result { + let node = value; + let ValuePath { value, path } = self.get_path(root, index)?; + + // performs the update only if the node value differs from the opening + if node != value { + root = self.add_merkle_path(index.value(), node, path.clone())?; } + + Ok(RootPath { root, path }) } pub fn merge_roots(&mut self, root1: Word, root2: Word) -> Result { diff --git a/src/merkle/store/tests.rs b/src/merkle/store/tests.rs index c469553..97e38a7 100644 --- a/src/merkle/store/tests.rs +++ b/src/merkle/store/tests.rs @@ -90,12 +90,12 @@ fn test_merkle_tree() -> Result<(), MerkleError> { .get_path(mtree.root(), NodeIndex::new(mtree.depth(), 0)) .unwrap(); assert_eq!( - LEAVES4[0], result.0, + LEAVES4[0], result.value, "Value for merkle path at index 0 must match leaf value" ); assert_eq!( mtree.get_path(NodeIndex::new(mtree.depth(), 0)), - Ok(result.1), + Ok(result.path), "merkle path for index 0 must be the same for the MerkleTree and MerkleStore" ); @@ -103,12 +103,12 @@ fn test_merkle_tree() -> Result<(), MerkleError> { .get_path(mtree.root(), NodeIndex::new(mtree.depth(), 1)) .unwrap(); assert_eq!( - LEAVES4[1], result.0, + LEAVES4[1], result.value, "Value for merkle path at index 0 must match leaf value" ); assert_eq!( mtree.get_path(NodeIndex::new(mtree.depth(), 1)), - Ok(result.1), + Ok(result.path), "merkle path for index 1 must be the same for the MerkleTree and MerkleStore" ); @@ -116,12 +116,12 @@ fn test_merkle_tree() -> Result<(), MerkleError> { .get_path(mtree.root(), NodeIndex::new(mtree.depth(), 2)) .unwrap(); assert_eq!( - LEAVES4[2], result.0, + LEAVES4[2], result.value, "Value for merkle path at index 0 must match leaf value" ); assert_eq!( mtree.get_path(NodeIndex::new(mtree.depth(), 2)), - Ok(result.1), + Ok(result.path), "merkle path for index 0 must be the same for the MerkleTree and MerkleStore" ); @@ -129,12 +129,12 @@ fn test_merkle_tree() -> Result<(), MerkleError> { .get_path(mtree.root(), NodeIndex::new(mtree.depth(), 3)) .unwrap(); assert_eq!( - LEAVES4[3], result.0, + LEAVES4[3], result.value, "Value for merkle path at index 0 must match leaf value" ); assert_eq!( mtree.get_path(NodeIndex::new(mtree.depth(), 3)), - Ok(result.1), + Ok(result.path), "merkle path for index 0 must be the same for the MerkleTree and MerkleStore" ); @@ -241,12 +241,12 @@ fn test_sparse_merkle_tree() -> Result<(), MerkleError> { .get_path(smt.root(), NodeIndex::new(smt.depth(), 0)) .unwrap(); assert_eq!( - LEAVES4[0], result.0, + LEAVES4[0], result.value, "Value for merkle path at index 0 must match leaf value" ); assert_eq!( smt.get_path(NodeIndex::new(smt.depth(), 0)), - Ok(result.1), + Ok(result.path), "merkle path for index 0 must be the same for the MerkleTree and MerkleStore" ); @@ -254,12 +254,12 @@ fn test_sparse_merkle_tree() -> Result<(), MerkleError> { .get_path(smt.root(), NodeIndex::new(smt.depth(), 1)) .unwrap(); assert_eq!( - LEAVES4[1], result.0, + LEAVES4[1], result.value, "Value for merkle path at index 0 must match leaf value" ); assert_eq!( smt.get_path(NodeIndex::new(smt.depth(), 1)), - Ok(result.1), + Ok(result.path), "merkle path for index 1 must be the same for the MerkleTree and MerkleStore" ); @@ -267,12 +267,12 @@ fn test_sparse_merkle_tree() -> Result<(), MerkleError> { .get_path(smt.root(), NodeIndex::new(smt.depth(), 2)) .unwrap(); assert_eq!( - LEAVES4[2], result.0, + LEAVES4[2], result.value, "Value for merkle path at index 0 must match leaf value" ); assert_eq!( smt.get_path(NodeIndex::new(smt.depth(), 2)), - Ok(result.1), + Ok(result.path), "merkle path for index 0 must be the same for the MerkleTree and MerkleStore" ); @@ -280,12 +280,12 @@ fn test_sparse_merkle_tree() -> Result<(), MerkleError> { .get_path(smt.root(), NodeIndex::new(smt.depth(), 3)) .unwrap(); assert_eq!( - LEAVES4[3], result.0, + LEAVES4[3], result.value, "Value for merkle path at index 0 must match leaf value" ); assert_eq!( smt.get_path(NodeIndex::new(smt.depth(), 3)), - Ok(result.1), + Ok(result.path), "merkle path for index 0 must be the same for the MerkleTree and MerkleStore" ); @@ -375,12 +375,12 @@ fn test_add_merkle_paths() -> Result<(), MerkleError> { .get_path(set.root(), NodeIndex::new(set.depth() - 1, 0)) .unwrap(); assert_eq!( - LEAVES4[0], result.0, + LEAVES4[0], result.value, "Value for merkle path at index 0 must match leaf value" ); assert_eq!( set.get_path(NodeIndex::new(set.depth(), 0)), - Ok(result.1), + Ok(result.path), "merkle path for index 0 must be the same for the MerkleTree and MerkleStore" ); @@ -388,12 +388,12 @@ fn test_add_merkle_paths() -> Result<(), MerkleError> { .get_path(set.root(), NodeIndex::new(set.depth() - 1, 1)) .unwrap(); assert_eq!( - LEAVES4[1], result.0, + LEAVES4[1], result.value, "Value for merkle path at index 0 must match leaf value" ); assert_eq!( set.get_path(NodeIndex::new(set.depth(), 1)), - Ok(result.1), + Ok(result.path), "merkle path for index 1 must be the same for the MerkleTree and MerkleStore" ); @@ -401,12 +401,12 @@ fn test_add_merkle_paths() -> Result<(), MerkleError> { .get_path(set.root(), NodeIndex::new(set.depth() - 1, 2)) .unwrap(); assert_eq!( - LEAVES4[2], result.0, + LEAVES4[2], result.value, "Value for merkle path at index 0 must match leaf value" ); assert_eq!( set.get_path(NodeIndex::new(set.depth(), 2)), - Ok(result.1), + Ok(result.path), "merkle path for index 0 must be the same for the MerkleTree and MerkleStore" ); @@ -414,12 +414,12 @@ fn test_add_merkle_paths() -> Result<(), MerkleError> { .get_path(set.root(), NodeIndex::new(set.depth() - 1, 3)) .unwrap(); assert_eq!( - LEAVES4[3], result.0, + LEAVES4[3], result.value, "Value for merkle path at index 0 must match leaf value" ); assert_eq!( set.get_path(NodeIndex::new(set.depth(), 3)), - Ok(result.1), + Ok(result.path), "merkle path for index 0 must be the same for the MerkleTree and MerkleStore" ); @@ -474,10 +474,13 @@ fn store_path_opens_from_leaf() { let store = MerkleStore::default() .with_merkle_tree([a, b, c, d, e, f, g, h]) .unwrap(); - let path = store.get_path(root.into(), NodeIndex::new(3, 1)).unwrap(); + let path = store + .get_path(root.into(), NodeIndex::new(3, 1)) + .unwrap() + .path; let expected = MerklePath::new([a.into(), j.into(), n.into()].to_vec()); - assert_eq!(path.1, expected); + assert_eq!(path, expected); } #[test] @@ -486,7 +489,7 @@ fn test_set_node() -> Result<(), MerkleError> { let mut store = MerkleStore::default().with_merkle_tree(LEAVES4)?; let value = int_to_node(42); let index = NodeIndex::new(mtree.depth(), 0); - let new_root = store.set_node(mtree.root(), index, value)?; + let new_root = store.set_node(mtree.root(), index, value)?.root; assert_eq!( store.get_node(new_root, index), Ok(value), From ef342cec2371a88fff440524c36dba848a362c42 Mon Sep 17 00:00:00 2001 From: "Augusto F. Hack" Date: Wed, 22 Mar 2023 10:53:12 +0100 Subject: [PATCH 15/26] bugfix: fix store benchmark --- benches/store.rs | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/benches/store.rs b/benches/store.rs index 0e60b04..793118a 100644 --- a/benches/store.rs +++ b/benches/store.rs @@ -425,7 +425,8 @@ fn update_leaf_merkletree(c: &mut Criterion) { // comparison store_root = store .set_node(root, NodeIndex::new(depth, index), value) - .unwrap(); + .unwrap() + .root; black_box(store_root) }, BatchSize::SmallInput, @@ -478,7 +479,8 @@ fn update_leaf_simplesmt(c: &mut Criterion) { // comparison store_root = store .set_node(root, NodeIndex::new(depth, index), value) - .unwrap(); + .unwrap() + .root; black_box(store_root) }, BatchSize::SmallInput, From d6cbd178e1e41a6d313d3e93596a3f0652eadda7 Mon Sep 17 00:00:00 2001 From: "Augusto F. Hack" Date: Wed, 22 Mar 2023 11:30:19 +0100 Subject: [PATCH 16/26] chore: clarified assert message --- src/merkle/merkle_tree.rs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/merkle/merkle_tree.rs b/src/merkle/merkle_tree.rs index a04a866..5f8b08e 100644 --- a/src/merkle/merkle_tree.rs +++ b/src/merkle/merkle_tree.rs @@ -108,7 +108,10 @@ impl MerkleTree { index.move_up(); } - debug_assert!(index.is_root(), "the path must include the root"); + debug_assert!( + index.is_root(), + "the path walk must go all the way to the root" + ); Ok(path.into()) } From b250752883235ade0a66fd8a8d41fc1005183fb7 Mon Sep 17 00:00:00 2001 From: "Augusto F. Hack" Date: Wed, 22 Mar 2023 14:17:09 +0100 Subject: [PATCH 17/26] store: added with_merkle_paths constructor And unit tests for each constructor type. --- src/merkle/store/mod.rs | 9 +++++++ src/merkle/store/tests.rs | 56 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 65 insertions(+) diff --git a/src/merkle/store/mod.rs b/src/merkle/store/mod.rs index 13ca5f9..a59bbf5 100644 --- a/src/merkle/store/mod.rs +++ b/src/merkle/store/mod.rs @@ -84,6 +84,15 @@ impl MerkleStore { Ok(self) } + /// Appends the provided merkle path set. + pub fn with_merkle_paths(mut self, paths: I) -> Result + where + I: IntoIterator, + { + self.add_merkle_paths(paths)?; + Ok(self) + } + // PUBLIC ACCESSORS // -------------------------------------------------------------------------------------------- diff --git a/src/merkle/store/tests.rs b/src/merkle/store/tests.rs index 97e38a7..1fa65e6 100644 --- a/src/merkle/store/tests.rs +++ b/src/merkle/store/tests.rs @@ -498,3 +498,59 @@ fn test_set_node() -> Result<(), MerkleError> { Ok(()) } + +#[test] +fn test_constructors() -> Result<(), MerkleError> { + let store = MerkleStore::new().with_merkle_tree(LEAVES4)?; + let mtree = MerkleTree::new(LEAVES4.to_vec())?; + + let depth = mtree.depth(); + let leaves = 2u64.pow(depth.into()); + for index in 0..leaves { + let index = NodeIndex::new(depth, index); + let value_path = store.get_path(mtree.root(), index)?; + assert_eq!(mtree.get_path(index)?, value_path.path); + } + + let store = MerkleStore::default() + .with_sparse_merkle_tree(KEYS4.into_iter().zip(LEAVES4.into_iter()))?; + let smt = SimpleSmt::new(SimpleSmt::MAX_DEPTH) + .unwrap() + .with_leaves(KEYS4.into_iter().zip(LEAVES4.into_iter())) + .unwrap(); + let depth = smt.depth(); + + for key in KEYS4 { + let index = NodeIndex::new(depth, key); + let value_path = store.get_path(smt.root(), index)?; + assert_eq!(smt.get_path(index)?, value_path.path); + } + + let d = 2; + let paths = [ + (0, LEAVES4[0], mtree.get_path(NodeIndex::new(d, 0)).unwrap()), + (1, LEAVES4[1], mtree.get_path(NodeIndex::new(d, 1)).unwrap()), + (2, LEAVES4[2], mtree.get_path(NodeIndex::new(d, 2)).unwrap()), + (3, LEAVES4[3], mtree.get_path(NodeIndex::new(d, 3)).unwrap()), + ]; + + let store1 = MerkleStore::default().with_merkle_paths(paths.clone())?; + let store2 = MerkleStore::default() + .with_merkle_path(0, LEAVES4[0], mtree.get_path(NodeIndex::new(d, 0))?)? + .with_merkle_path(1, LEAVES4[1], mtree.get_path(NodeIndex::new(d, 1))?)? + .with_merkle_path(2, LEAVES4[2], mtree.get_path(NodeIndex::new(d, 2))?)? + .with_merkle_path(3, LEAVES4[3], mtree.get_path(NodeIndex::new(d, 3))?)?; + let set = MerklePathSet::new(d + 1).with_paths(paths).unwrap(); + + for key in [0, 1, 2, 3] { + let index = NodeIndex::new(d, key); + let value_path1 = store1.get_path(set.root(), index)?; + let value_path2 = store2.get_path(set.root(), index)?; + assert_eq!(value_path1, value_path2); + + let index = NodeIndex::new(d + 1, key); + assert_eq!(set.get_path(index)?, value_path1.path); + } + + Ok(()) +} From 0375f31035b546c2aff7d60bb09af25d7c792736 Mon Sep 17 00:00:00 2001 From: "Augusto F. Hack" Date: Wed, 22 Mar 2023 12:06:02 +0100 Subject: [PATCH 18/26] feat: added utility to format MerkleTree and MerklePath to hex Example formatted MerkleTree: ``` 880abe452320966617646e7740b014954300f19a28780a0889d62ff33f4b0534 1ade1369091efa31201e9b60c9c28874d0ddce5362b335135a6bb4c917285983 3e60a9c843b4bb19f7a0572102e6507195f5240767a396335fd21981b048b807 0100000000000000000000000000000000000000000000000000000000000000 0200000000000000000000000000000000000000000000000000000000000000 0300000000000000000000000000000000000000000000000000000000000000 0400000000000000000000000000000000000000000000000000000000000000 ``` Example formatted MerklePath: ``` [0400000000000000000000000000000000000000000000000000000000000000, 1ade1369091efa31201e9b60c9c28874d0ddce5362b335135a6bb4c917285983] ``` --- src/lib.rs | 7 +----- src/merkle/merkle_tree.rs | 53 +++++++++++++++++++++++++++++++++++++-- src/merkle/mod.rs | 2 +- src/utils.rs | 21 ++++++++++++++++ 4 files changed, 74 insertions(+), 9 deletions(-) create mode 100644 src/utils.rs diff --git a/src/lib.rs b/src/lib.rs index d0285b2..7701d36 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -7,18 +7,13 @@ extern crate alloc; mod bit; pub mod hash; pub mod merkle; +pub mod utils; // RE-EXPORTS // ================================================================================================ pub use winter_crypto::{RandomCoin, RandomCoinError}; pub use winter_math::{fields::f64::BaseElement as Felt, FieldElement, StarkField}; -pub mod utils { - pub use winter_utils::{ - collections, string, uninit_vector, ByteReader, ByteWriter, Deserializable, - DeserializationError, Serializable, SliceReader, - }; -} // TYPE ALIASES // ================================================================================================ diff --git a/src/merkle/merkle_tree.rs b/src/merkle/merkle_tree.rs index 5f8b08e..434c8e7 100644 --- a/src/merkle/merkle_tree.rs +++ b/src/merkle/merkle_tree.rs @@ -1,6 +1,9 @@ use super::{Felt, MerkleError, MerklePath, NodeIndex, Rpo256, RpoDigest, Vec, Word}; -use crate::{utils::uninit_vector, FieldElement}; -use core::slice; +use crate::{ + utils::{string::String, uninit_vector, word_to_hex}, + FieldElement, +}; +use core::{fmt, slice}; use winter_math::log2; // MERKLE TREE @@ -157,6 +160,52 @@ impl MerkleTree { } } +/// Utility to vizualize a [MerkleTree] in text. +pub fn tree_to_text(tree: &MerkleTree) -> Result { + let indent = " "; + let mut s = String::new(); + s.push_str(&word_to_hex(&tree.root())?); + s.push('\n'); + for d in 1..=tree.depth() { + let entries = 2u64.pow(d.into()); + for i in 0..entries { + let index = NodeIndex::new(d, i); + + let node = tree + .get_node(index) + .expect("The index must always be valid"); + + for _ in 0..d { + s.push_str(indent); + } + s.push_str(&word_to_hex(&node)?); + s.push('\n'); + } + } + + Ok(s) +} + +/// Utility to vizualize a [MerklePath] in text. +pub fn path_to_text(path: &MerklePath) -> Result { + let mut s = String::new(); + s.push('['); + + for el in path.iter() { + s.push_str(&word_to_hex(el)?); + s.push_str(", "); + } + + // remove the last ", " + if path.len() != 0 { + s.pop(); + s.pop(); + } + s.push(']'); + + Ok(s) +} + // TESTS // ================================================================================================ diff --git a/src/merkle/mod.rs b/src/merkle/mod.rs index 5b204ac..b0c7b2d 100644 --- a/src/merkle/mod.rs +++ b/src/merkle/mod.rs @@ -15,7 +15,7 @@ mod index; pub use index::NodeIndex; mod merkle_tree; -pub use merkle_tree::MerkleTree; +pub use merkle_tree::{path_to_text, tree_to_text, MerkleTree}; mod path; pub use path::{MerklePath, RootPath, ValuePath}; diff --git a/src/utils.rs b/src/utils.rs new file mode 100644 index 0000000..522d9ea --- /dev/null +++ b/src/utils.rs @@ -0,0 +1,21 @@ +use super::Word; +use crate::utils::string::String; +use core::fmt::{self, Write}; + +// RE-EXPORTS +// ================================================================================================ +pub use winter_utils::{ + collections, string, uninit_vector, ByteReader, ByteWriter, Deserializable, + DeserializationError, Serializable, SliceReader, +}; + +/// Converts a [Word] into hex. +pub fn word_to_hex(w: &Word) -> Result { + let mut s = String::new(); + + for byte in w.iter().flat_map(|e| e.to_bytes()) { + write!(s, "{byte:02x}")?; + } + + Ok(s) +} From 9531d2bd3427b8d2787427111351b17ab570feca Mon Sep 17 00:00:00 2001 From: Bobbin Threadbare Date: Thu, 23 Mar 2023 01:03:58 -0700 Subject: [PATCH 19/26] fix: to paths reduction of MerklePathSet --- src/merkle/index.rs | 4 +- src/merkle/path_set.rs | 136 +++++++++++++++++++++++++++++--------- src/merkle/store/mod.rs | 9 ++- src/merkle/store/tests.rs | 30 ++++----- 4 files changed, 126 insertions(+), 53 deletions(-) diff --git a/src/merkle/index.rs b/src/merkle/index.rs index 988c79f..5729083 100644 --- a/src/merkle/index.rs +++ b/src/merkle/index.rs @@ -78,7 +78,7 @@ impl NodeIndex { self.depth } - /// Returns the value of the current depth. + /// Returns the value of this index. pub const fn value(&self) -> u64 { self.value } @@ -104,7 +104,7 @@ impl NodeIndex { /// arrive at the leaf. From the right-to-left the bit represent the position the hash of the /// current element should go. /// - /// Additionally, the value that is not visisted are the sibling values necessary for a Merkle + /// Additionally, the value that is not visited are the sibling values necessary for a Merkle /// opening. pub fn bit_iterator(&self) -> BitIterator { let depth: u32 = self.depth.into(); diff --git a/src/merkle/path_set.rs b/src/merkle/path_set.rs index b483949..9a90687 100644 --- a/src/merkle/path_set.rs +++ b/src/merkle/path_set.rs @@ -1,4 +1,4 @@ -use super::{BTreeMap, MerkleError, MerklePath, NodeIndex, Rpo256, Vec, Word, ZERO}; +use super::{BTreeMap, MerkleError, MerklePath, NodeIndex, Rpo256, ValuePath, Vec, Word, ZERO}; // MERKLE PATH SET // ================================================================================================ @@ -57,14 +57,6 @@ impl MerklePathSet { self.total_depth } - /// Returns all the leaf indexes of this path set. - pub fn indexes(&self) -> impl Iterator + '_ { - self.paths - .keys() - .copied() - .map(|index| NodeIndex::new(self.total_depth, index)) - } - /// Returns a node at the specified index. /// /// # Errors @@ -84,15 +76,23 @@ impl MerklePathSet { }); } - let index_value = index.to_scalar_index(); - let parity = index_value & 1; - let index_value = index_value / 2; + let parity = index.value() & 1; + let path_key = index.value() - parity; self.paths - .get(&index_value) - .ok_or(MerkleError::NodeNotInSet(index_value)) + .get(&path_key) + .ok_or(MerkleError::NodeNotInSet(path_key)) .map(|path| path[parity as usize]) } + /// Returns a leaf at the specified index. + /// + /// # Errors + /// * The specified index is not valid for the depth of the structure. + /// * Leaf with the requested path does not exist in the set. + pub fn get_leaf(&self, index: u64) -> Result { + self.get_node(NodeIndex::new(self.depth(), index)) + } + /// Returns a Merkle path to the node at the specified index. The node itself is /// not included in the path. /// @@ -111,18 +111,42 @@ impl MerklePathSet { }); } - let index_value = index.to_scalar_index(); - let index = index_value / 2; - let parity = index_value & 1; + let parity = index.value() & 1; + let path_key = index.value() - parity; let mut path = self .paths - .get(&index) + .get(&path_key) .cloned() - .ok_or(MerkleError::NodeNotInSet(index))?; + .ok_or(MerkleError::NodeNotInSet(index.value()))?; path.remove(parity as usize); Ok(path) } + /// Returns all paths in this path set together with their indexes. + pub fn to_paths(&self) -> Vec<(u64, ValuePath)> { + let mut result = Vec::with_capacity(self.paths.len() * 2); + + for (&index, path) in self.paths.iter() { + // push path for the even index into the result + let path1 = ValuePath { + value: path[0], + path: MerklePath::new(path[1..].to_vec()), + }; + result.push((index, path1)); + + // push path for the odd index into the result + let mut path2 = path.clone(); + let leaf2 = path2.remove(1); + let path2 = ValuePath { + value: leaf2, + path: path2, + }; + result.push((index + 1, path2)); + } + + result + } + // STATE MUTATORS // -------------------------------------------------------------------------------------------- @@ -141,7 +165,7 @@ impl MerklePathSet { value: Word, mut path: MerklePath, ) -> Result<(), MerkleError> { - let depth = (path.len() + 1) as u8; + let depth = path.len() as u8; let mut index = NodeIndex::new(depth, index_value); if index.depth() != self.total_depth { return Err(MerkleError::InvalidDepth { @@ -151,8 +175,6 @@ impl MerklePathSet { } // update the current path - let index_value = index.to_scalar_index(); - let upper_index_value = index_value / 2; let parity = index_value & 1; path.insert(parity as usize, value); @@ -172,7 +194,8 @@ impl MerklePathSet { } // finish updating the path - self.paths.insert(upper_index_value, path); + let path_key = index_value - parity; + self.paths.insert(path_key, path); Ok(()) } @@ -188,10 +211,9 @@ impl MerklePathSet { return Err(MerkleError::InvalidIndex(index)); } - let path = match self - .paths - .get_mut(&index.clone().move_up().to_scalar_index()) - { + let parity = index.value() & 1; + let path_key = index.value() - parity; + let path = match self.paths.get_mut(&path_key) { Some(path) => path, None => return Err(MerkleError::NodeNotInSet(base_index_value)), }; @@ -255,7 +277,7 @@ mod tests { let root_exp = calculate_parent_hash(parent0, 0, parent1); - let set = super::MerklePathSet::new(3) + let set = super::MerklePathSet::new(2) .with_paths([(0, leaf0, vec![leaf1, parent1].into())]) .unwrap(); @@ -267,7 +289,7 @@ mod tests { let path_6 = vec![int_to_node(7), int_to_node(45), int_to_node(123)]; let hash_6 = int_to_node(6); let index = 6_u64; - let depth = 4_u8; + let depth = 3_u8; let set = super::MerklePathSet::new(depth) .with_paths([(index, hash_6, path_6.clone().into())]) .unwrap(); @@ -282,7 +304,7 @@ mod tests { let path_6 = vec![int_to_node(7), int_to_node(45), int_to_node(123)]; let hash_6 = int_to_node(6); let index = 6_u64; - let depth = 4_u8; + let depth = 3_u8; let set = MerklePathSet::new(depth) .with_paths([(index, hash_6, path_6.into())]) .unwrap(); @@ -312,7 +334,7 @@ mod tests { let index_6 = 6_u64; let index_5 = 5_u64; let index_4 = 4_u64; - let depth = 4_u8; + let depth = 3_u8; let mut set = MerklePathSet::new(depth) .with_paths([ (index_6, hash_6, path_6.into()), @@ -337,6 +359,58 @@ mod tests { assert_eq!(new_hash_5, new_path_4[0]); } + #[test] + fn depth_3_is_correct() { + let a = int_to_node(1); + let b = int_to_node(2); + let c = int_to_node(3); + let d = int_to_node(4); + let e = int_to_node(5); + let f = int_to_node(6); + let g = int_to_node(7); + let h = int_to_node(8); + + let i = Rpo256::merge(&[a.into(), b.into()]); + let j = Rpo256::merge(&[c.into(), d.into()]); + let k = Rpo256::merge(&[e.into(), f.into()]); + let l = Rpo256::merge(&[g.into(), h.into()]); + + let m = Rpo256::merge(&[i.into(), j.into()]); + let n = Rpo256::merge(&[k.into(), l.into()]); + + let root = Rpo256::merge(&[m.into(), n.into()]); + + let mut set = MerklePathSet::new(3); + + let value = b; + let index = 1; + let path = MerklePath::new([a.into(), j.into(), n.into()].to_vec()); + set.add_path(index, value, path.clone()).unwrap(); + assert_eq!(value, set.get_leaf(index).unwrap()); + assert_eq!(Word::from(root), set.root()); + + let value = e; + let index = 4; + let path = MerklePath::new([f.into(), l.into(), m.into()].to_vec()); + set.add_path(index, value, path.clone()).unwrap(); + assert_eq!(value, set.get_leaf(index).unwrap()); + assert_eq!(Word::from(root), set.root()); + + let value = a; + let index = 0; + let path = MerklePath::new([b.into(), j.into(), n.into()].to_vec()); + set.add_path(index, value, path.clone()).unwrap(); + assert_eq!(value, set.get_leaf(index).unwrap()); + assert_eq!(Word::from(root), set.root()); + + let value = h; + let index = 7; + let path = MerklePath::new([g.into(), k.into(), m.into()].to_vec()); + set.add_path(index, value, path.clone()).unwrap(); + assert_eq!(value, set.get_leaf(index).unwrap()); + assert_eq!(Word::from(root), set.root()); + } + // HELPER FUNCTIONS // -------------------------------------------------------------------------------------------- diff --git a/src/merkle/store/mod.rs b/src/merkle/store/mod.rs index a59bbf5..ada5f40 100644 --- a/src/merkle/store/mod.rs +++ b/src/merkle/store/mod.rs @@ -317,11 +317,10 @@ impl MerkleStore { /// For further reference, check [MerkleStore::add_merkle_path]. pub fn add_merkle_path_set(&mut self, path_set: &MerklePathSet) -> Result { let root = path_set.root(); - path_set.indexes().try_fold(root, |_, index| { - let node = path_set.get_node(index)?; - let path = path_set.get_path(index)?; - self.add_merkle_path(index.value(), node, path) - }) + for (index, path) in path_set.to_paths() { + self.add_merkle_path(index, path.value, path.path)?; + } + Ok(root) } /// Sets a node to `value`. diff --git a/src/merkle/store/tests.rs b/src/merkle/store/tests.rs index 1fa65e6..e29f8e7 100644 --- a/src/merkle/store/tests.rs +++ b/src/merkle/store/tests.rs @@ -320,28 +320,28 @@ fn test_add_merkle_paths() -> Result<(), MerkleError> { .add_merkle_paths(paths.clone()) .expect("the valid paths must work"); - let depth = 3; + let depth = 2; let set = MerklePathSet::new(depth).with_paths(paths).unwrap(); // STORE LEAVES ARE CORRECT ============================================================== // checks the leaves in the store corresponds to the expected values assert_eq!( - store.get_node(set.root(), NodeIndex::new(set.depth() - 1, 0)), + store.get_node(set.root(), NodeIndex::new(set.depth(), 0)), Ok(LEAVES4[0]), "node 0 must be in the set" ); assert_eq!( - store.get_node(set.root(), NodeIndex::new(set.depth() - 1, 1)), + store.get_node(set.root(), NodeIndex::new(set.depth(), 1)), Ok(LEAVES4[1]), "node 1 must be in the set" ); assert_eq!( - store.get_node(set.root(), NodeIndex::new(set.depth() - 1, 2)), + store.get_node(set.root(), NodeIndex::new(set.depth(), 2)), Ok(LEAVES4[2]), "node 2 must be in the set" ); assert_eq!( - store.get_node(set.root(), NodeIndex::new(set.depth() - 1, 3)), + store.get_node(set.root(), NodeIndex::new(set.depth(), 3)), Ok(LEAVES4[3]), "node 3 must be in the set" ); @@ -350,29 +350,29 @@ fn test_add_merkle_paths() -> Result<(), MerkleError> { // sanity check the values returned by the store and the set assert_eq!( set.get_node(NodeIndex::new(set.depth(), 0)), - store.get_node(set.root(), NodeIndex::new(set.depth() - 1, 0)), + store.get_node(set.root(), NodeIndex::new(set.depth(), 0)), "node 0 must be the same for both SparseMerkleTree and MerkleStore" ); assert_eq!( set.get_node(NodeIndex::new(set.depth(), 1)), - store.get_node(set.root(), NodeIndex::new(set.depth() - 1, 1)), + store.get_node(set.root(), NodeIndex::new(set.depth(), 1)), "node 1 must be the same for both SparseMerkleTree and MerkleStore" ); assert_eq!( set.get_node(NodeIndex::new(set.depth(), 2)), - store.get_node(set.root(), NodeIndex::new(set.depth() - 1, 2)), + store.get_node(set.root(), NodeIndex::new(set.depth(), 2)), "node 2 must be the same for both SparseMerkleTree and MerkleStore" ); assert_eq!( set.get_node(NodeIndex::new(set.depth(), 3)), - store.get_node(set.root(), NodeIndex::new(set.depth() - 1, 3)), + store.get_node(set.root(), NodeIndex::new(set.depth(), 3)), "node 3 must be the same for both SparseMerkleTree and MerkleStore" ); // STORE MERKLE PATH MATCHS ============================================================== // assert the merkle path returned by the store is the same as the one in the set let result = store - .get_path(set.root(), NodeIndex::new(set.depth() - 1, 0)) + .get_path(set.root(), NodeIndex::new(set.depth(), 0)) .unwrap(); assert_eq!( LEAVES4[0], result.value, @@ -385,7 +385,7 @@ fn test_add_merkle_paths() -> Result<(), MerkleError> { ); let result = store - .get_path(set.root(), NodeIndex::new(set.depth() - 1, 1)) + .get_path(set.root(), NodeIndex::new(set.depth(), 1)) .unwrap(); assert_eq!( LEAVES4[1], result.value, @@ -398,7 +398,7 @@ fn test_add_merkle_paths() -> Result<(), MerkleError> { ); let result = store - .get_path(set.root(), NodeIndex::new(set.depth() - 1, 2)) + .get_path(set.root(), NodeIndex::new(set.depth(), 2)) .unwrap(); assert_eq!( LEAVES4[2], result.value, @@ -411,7 +411,7 @@ fn test_add_merkle_paths() -> Result<(), MerkleError> { ); let result = store - .get_path(set.root(), NodeIndex::new(set.depth() - 1, 3)) + .get_path(set.root(), NodeIndex::new(set.depth(), 3)) .unwrap(); assert_eq!( LEAVES4[3], result.value, @@ -540,7 +540,7 @@ fn test_constructors() -> Result<(), MerkleError> { .with_merkle_path(1, LEAVES4[1], mtree.get_path(NodeIndex::new(d, 1))?)? .with_merkle_path(2, LEAVES4[2], mtree.get_path(NodeIndex::new(d, 2))?)? .with_merkle_path(3, LEAVES4[3], mtree.get_path(NodeIndex::new(d, 3))?)?; - let set = MerklePathSet::new(d + 1).with_paths(paths).unwrap(); + let set = MerklePathSet::new(d).with_paths(paths).unwrap(); for key in [0, 1, 2, 3] { let index = NodeIndex::new(d, key); @@ -548,7 +548,7 @@ fn test_constructors() -> Result<(), MerkleError> { let value_path2 = store2.get_path(set.root(), index)?; assert_eq!(value_path1, value_path2); - let index = NodeIndex::new(d + 1, key); + let index = NodeIndex::new(d, key); assert_eq!(set.get_path(index)?, value_path1.path); } From 2ff96f40cba886635cd0a211aea5a3b55cdf2f1a Mon Sep 17 00:00:00 2001 From: Victor Lopez Date: Thu, 23 Mar 2023 12:54:28 +0100 Subject: [PATCH 20/26] feat: add empty subtree constants to cover u8::MAX depth Prior to this commit, we limited the constants count to 64 for the empty subtrees depth computation. This is a hard-assumption that every tree of Miden will have a depth up to 64 - and will cause undefined behavior if it doesn't. With the introduction of `MerkleStore::merge_roots` and the deprecation of `mtree_cwm` instruction from the VM, this assumption is broken and the user might end with trees with depth greater than 64. This broken assumption could lead to attack vectors. We can easily fix that by extending the pre-computed hashes list to the maximum of `u8` (i.e. 255). This will have zero impact on functionality, and will be completely safe to use without hard assumptions. --- src/merkle/empty_roots.rs | 1687 +++++++++++++++++++++++++++++++------ 1 file changed, 1409 insertions(+), 278 deletions(-) diff --git a/src/merkle/empty_roots.rs b/src/merkle/empty_roots.rs index 8ba6757..7f8c4a2 100644 --- a/src/merkle/empty_roots.rs +++ b/src/merkle/empty_roots.rs @@ -4,19 +4,14 @@ use core::slice; // EMPTY NODES SUBTREES // ================================================================================================ -/// Contains precomputed roots of empty subtrees in a Merkle rtee of depth 64. +/// Contains precomputed roots of empty subtrees in a Merkle tree. pub struct EmptySubtreeRoots; impl EmptySubtreeRoots { /// Returns a static slice with roots of empty subtrees of a Merkle tree starting at the /// specified depth. - /// - /// # Panics - /// - /// This function will panic if the provided `depth` is greater than `64`. pub const fn empty_hashes(depth: u8) -> &'static [RpoDigest] { - assert!(depth < 65); - let ptr = &EMPTY_SUBTREES_64[64 - depth as usize] as *const RpoDigest; + let ptr = &EMPTY_SUBTREES[255 - depth as usize] as *const RpoDigest; // Safety: this is a static/constant array, so it will never be outlived. If we attempt to // use regular slices, this wouldn't be a `const` function, meaning we won't be able to use // the returned value for static/constant definitions. @@ -24,390 +19,1536 @@ impl EmptySubtreeRoots { } } -const EMPTY_SUBTREES_64: [RpoDigest; 65] = [ +const EMPTY_SUBTREES: [RpoDigest; 256] = [ RpoDigest::new([ - Felt::new(15321474589252129342), - Felt::new(17373224439259377994), - Felt::new(15071539326562317628), - Felt::new(3312677166725950353), + Felt::new(0xee3d94db86d48dc4), + Felt::new(0x3d13166c7aba0368), + Felt::new(0x282e861f2936aa7), + Felt::new(0xf0328a1745537b4), ]), RpoDigest::new([ - Felt::new(12146678323567200178), - Felt::new(14288630174929498478), - Felt::new(13374892366980833045), - Felt::new(11840636859983936891), + Felt::new(0x33174b312b730760), + Felt::new(0x9d1e00c5b50352b2), + Felt::new(0x16bf9ec4acef2e42), + Felt::new(0x4263877e63c4cbe7), ]), RpoDigest::new([ - Felt::new(15220380953028059006), - Felt::new(2981707349961006045), - Felt::new(7409523958661360004), - Felt::new(2816116826688969892), + Felt::new(0xa8e039042672a8bc), + Felt::new(0x1010d951d941d9d4), + Felt::new(0xd3e8f0ecc866ac3b), + Felt::new(0xb2dbbbb80da232ba), ]), RpoDigest::new([ - Felt::new(7829641133220670678), - Felt::new(6170216088031698405), - Felt::new(11814483661801576435), - Felt::new(1762887097744793975), + Felt::new(0xa0d2c1a3e455f299), + Felt::new(0x648e8e13867dc8eb), + Felt::new(0xe79f94ea61189847), + Felt::new(0xb88a89e1f2765a31), ]), RpoDigest::new([ - Felt::new(1299421782687082884), - Felt::new(9938699043036414489), - Felt::new(10193025806762503939), - Felt::new(12073246492422971113), + Felt::new(0xe168133fd9ab570d), + Felt::new(0x6fc7d0295ac5a3a2), + Felt::new(0xc973ea026e9411c), + Felt::new(0x63c29ea04552b532), ]), RpoDigest::new([ - Felt::new(3774016405860870757), - Felt::new(2584714598467121158), - Felt::new(7418645462301488344), - Felt::new(1016804897028793820), + Felt::new(0x27e57ecc0f9f196), + Felt::new(0xc02f6e29fd19b059), + Felt::new(0x4a42fbc436efb0b6), + Felt::new(0xbff30574a98a1b29), ]), RpoDigest::new([ - Felt::new(13238072489118494737), - Felt::new(6917129315345826393), - Felt::new(13736362398490889690), - Felt::new(4929049375601714136), + Felt::new(0x18c32592a4f4d0b), + Felt::new(0x6b9a08797252d5d5), + Felt::new(0xdbfe48f00a088a2), + Felt::new(0x16b4c3e485b173e3), ]), RpoDigest::new([ - Felt::new(2433738165854950976), - Felt::new(6710644905925382197), - Felt::new(10571480102433401045), - Felt::new(16853295309134271298), + Felt::new(0x26c8902938b831a5), + Felt::new(0x66ee91b36943f92e), + Felt::new(0x4e8deeafef9f5725), + Felt::new(0xbb35751d5dfb0a33), ]), RpoDigest::new([ - Felt::new(3162775558610426184), - Felt::new(11944004899624546116), - Felt::new(55767976185223284), - Felt::new(5892480272697245897), + Felt::new(0x74af678f8e020ff4), + Felt::new(0xd4784cda0beed295), + Felt::new(0x4380949d841d793c), + Felt::new(0xdf587011d09d3bbb), ]), RpoDigest::new([ - Felt::new(12582634330812132159), - Felt::new(6886254574119140332), - Felt::new(4407453795368410417), - Felt::new(6959805977831121004), + Felt::new(0xa211d1da76aaef98), + Felt::new(0xd904ccc6435e268), + Felt::new(0x1c6f16a5d03b8416), + Felt::new(0x87800f7f5da9c93), ]), RpoDigest::new([ - Felt::new(16001070406220863863), - Felt::new(4426773743735082930), - Felt::new(6860108527212616559), - Felt::new(3994703491288516722), + Felt::new(0xa00bbad0a52adeff), + Felt::new(0xe22179c651da9d76), + Felt::new(0x474f10493a3723f4), + Felt::new(0x84397e6bd34a1f5b), ]), RpoDigest::new([ - Felt::new(9755907048710665826), - Felt::new(13697078808748604851), - Felt::new(17210321635283113095), - Felt::new(1203394006092675979), + Felt::new(0xe8f440afef4d082b), + Felt::new(0x14fff8e329613cc9), + Felt::new(0x78e984bc8b40f4f1), + Felt::new(0x6ed8f02e5be1bab2), ]), RpoDigest::new([ - Felt::new(3332855817731547893), - Felt::new(1068928372599561798), - Felt::new(17119375903210334455), - Felt::new(8148601736624954416), + Felt::new(0xda824edf085b5f9f), + Felt::new(0xc8a8f1c1b86d349e), + Felt::new(0xe1bf6975afb7b2de), + Felt::new(0xd7df51ea51028489), ]), RpoDigest::new([ - Felt::new(17265634841675424144), - Felt::new(18322832739735580203), - Felt::new(17896992777163902308), - Felt::new(6189383326950297131), + Felt::new(0xf64873d31456de99), + Felt::new(0x1fc9cb920b6c72b), + Felt::new(0x96613d9d71af4373), + Felt::new(0x61d607eb097e76c9), ]), RpoDigest::new([ - Felt::new(9329637674239983584), - Felt::new(2512861030579248721), - Felt::new(10833150484884266896), - Felt::new(7470498642428983444), + Felt::new(0xca304d2b3b778719), + Felt::new(0xa54d8602f37eed39), + Felt::new(0xb4574db6dc09bcf2), + Felt::new(0x5e42cd4f1de9587c), ]), RpoDigest::new([ - Felt::new(3611140194800558886), - Felt::new(17185933650781587767), - Felt::new(7835232399818923215), - Felt::new(7974155618002781326), + Felt::new(0x17575dfa689d8a07), + Felt::new(0x1db9d374d7436444), + Felt::new(0x21d1e8dca296f38d), + Felt::new(0xbc4aad43a9d93f54), ]), RpoDigest::new([ - Felt::new(17483286922353768131), - Felt::new(353378057542380712), - Felt::new(1935183237414585408), - Felt::new(4820339620987989650), + Felt::new(0x9fa0697330c054cd), + Felt::new(0xd5d57fbf059452e8), + Felt::new(0xe848fafb1c43414c), + Felt::new(0xacb7754fd77c9d52), ]), RpoDigest::new([ - Felt::new(16172462385444809646), - Felt::new(3268597753131435459), - Felt::new(3481491333654579291), - Felt::new(16487779176137683725), + Felt::new(0x406af89b918e596c), + Felt::new(0xb735a2c588ee87df), + Felt::new(0xb40ff1dd1c3c6599), + Felt::new(0x675a582b4c8a68ac), ]), RpoDigest::new([ - Felt::new(16595012576192613315), - Felt::new(16028552537812484518), - Felt::new(13016887826405546773), - Felt::new(14649690775021494057), + Felt::new(0x530ff6be0c86a2f6), + Felt::new(0x5541fabfefd34c91), + Felt::new(0x4af1579d212149ae), + Felt::new(0x23962b7df862f27c), ]), RpoDigest::new([ - Felt::new(11300236651178143890), - Felt::new(15307634289168527196), - Felt::new(2834866419963148279), - Felt::new(7512874625395280090), + Felt::new(0x1676b694f41cfc0d), + Felt::new(0x59b165ea5f354fd8), + Felt::new(0x5b45ee14e2501f08), + Felt::new(0xd0c8ca7bc2e01e18), ]), RpoDigest::new([ - Felt::new(1148273481270068529), - Felt::new(7411276436636897120), - Felt::new(14325955409748352141), - Felt::new(15577038614919538356), + Felt::new(0x2cff5d1e629ddc7b), + Felt::new(0x5062be34e4351fe), + Felt::new(0xfd76495b9d8ea67), + Felt::new(0xb96453b1c8060ca8), ]), RpoDigest::new([ - Felt::new(13911627859049081064), - Felt::new(13298542751859672529), - Felt::new(18341014824837028242), - Felt::new(5587966507704160144), + Felt::new(0x860b00517d3de1ef), + Felt::new(0xd609c82af07b9dad), + Felt::new(0xa54a528b8f1cbddc), + Felt::new(0xb4fff658ed97e635), ]), RpoDigest::new([ - Felt::new(10957185917743597702), - Felt::new(15815185767119166433), - Felt::new(17883994521792846784), - Felt::new(15958104556930886663), + Felt::new(0xd002cea8f347c347), + Felt::new(0xa135cebffdf3ec10), + Felt::new(0xb0200ea08da2cef4), + Felt::new(0x4e6f2f75d627b137), ]), RpoDigest::new([ - Felt::new(13148367538964199489), - Felt::new(7372139436485928380), - Felt::new(13408383191801051600), - Felt::new(2114382634401123096), + Felt::new(0xc1983ce677cabbf4), + Felt::new(0x58f0143480f44788), + Felt::new(0xf8c23e4f84b6c6c1), + Felt::new(0xc9ce41371c4900b8), ]), RpoDigest::new([ - Felt::new(14448157482521530067), - Felt::new(17865161921504959156), - Felt::new(10319385198642448897), - Felt::new(364163501511998552), + Felt::new(0x837d99979bc9a5e6), + Felt::new(0x7621559aa4af175a), + Felt::new(0x6986737347c799c2), + Felt::new(0x8cee99eb47c3e702), ]), RpoDigest::new([ - Felt::new(9722640569118951143), - Felt::new(16371655672847089887), - Felt::new(12379452272155069993), - Felt::new(11605969747977185617), + Felt::new(0x42e17ba02508a41f), + Felt::new(0xb95e349bd55ba61f), + Felt::new(0xcc2bfeb29c4c68b2), + Felt::new(0xf268f57860a446b1), ]), RpoDigest::new([ - Felt::new(2782512273606877924), - Felt::new(3656296563981095117), - Felt::new(5947388149010135441), - Felt::new(1678144343036748885), + Felt::new(0xd3ffd4ccc6dda508), + Felt::new(0x81db1910ef04ca07), + Felt::new(0x5c698ee6c3aeab97), + Felt::new(0x2ac1e2c2c5f237de), ]), RpoDigest::new([ - Felt::new(10347491038074052866), - Felt::new(11061756013655443653), - Felt::new(8901792852813329415), - Felt::new(10002477867799577447), + Felt::new(0x1f42a1ef25bd0aad), + Felt::new(0x81b0f63e2760b8db), + Felt::new(0xe9607c7061b018f9), + Felt::new(0xf02a88202294a700), ]), RpoDigest::new([ - Felt::new(16688151588649906570), - Felt::new(12937054427339650762), - Felt::new(2125115528195796454), - Felt::new(4796610823085621719), + Felt::new(0xea5da09b39b60468), + Felt::new(0xe48ea41d94fe91a7), + Felt::new(0x24dde954ce08b32b), + Felt::new(0xe1bb6e41bd0613e6), ]), RpoDigest::new([ - Felt::new(3032620037225059051), - Felt::new(13522881885116127385), - Felt::new(6010511038055304264), - Felt::new(8199256447383686121), + Felt::new(0xc5e9f7188b43a24f), + Felt::new(0x8d7132abc9d901e4), + Felt::new(0xdc09a33ff4d0eb03), + Felt::new(0xa119bb1db594b4cf), ]), RpoDigest::new([ - Felt::new(11250302734399433639), - Felt::new(4970037623163209776), - Felt::new(15776613712371118341), - Felt::new(5554382612311754837), + Felt::new(0x589002afcbd4a233), + Felt::new(0xe4eae44d3c2a308d), + Felt::new(0x8bc0bca14b6b4dde), + Felt::new(0x3716e0e86a7aaa6c), ]), RpoDigest::new([ - Felt::new(5116523511540088640), - Felt::new(12381059245485642368), - Felt::new(2176361879916914688), - Felt::new(11209293198464735683), + Felt::new(0xaa4ba9602230007e), + Felt::new(0x2b2c3e14b888a3d4), + Felt::new(0x90a36fb42ec2ba19), + Felt::new(0x2e07ef26b078c4a7), ]), RpoDigest::new([ - Felt::new(11677748883385181208), - Felt::new(15891398395707500576), - Felt::new(3790704659934033620), - Felt::new(2126099371106695189), + Felt::new(0x32307da7aad33113), + Felt::new(0x343ed87928b9ab0c), + Felt::new(0x1c01d79482c021f0), + Felt::new(0x6f866afccc595439), ]), RpoDigest::new([ - Felt::new(13948603355603496603), - Felt::new(15902438544472945077), - Felt::new(1969361494026622497), - Felt::new(17326911676634210553), + Felt::new(0x9780804b58b0d066), + Felt::new(0x1329929c6dc19c09), + Felt::new(0xc04add06dbaef6bf), + Felt::new(0xf494a28db17c5c4), ]), RpoDigest::new([ - Felt::new(16081431322775411514), - Felt::new(13201312030265587002), - Felt::new(18283434127959076535), - Felt::new(9889802180847551599), + Felt::new(0xe9dbb1c64d55571f), + Felt::new(0x663f0f716f28734), + Felt::new(0x7285fd4b8e87a78c), + Felt::new(0x2e152a4595b7597e), ]), RpoDigest::new([ - Felt::new(8490051641633132830), - Felt::new(11985660456681176415), - Felt::new(12193381039977027251), - Felt::new(17563185381678568385), + Felt::new(0x5531fabfa5960807), + Felt::new(0x8afe79be96d903a4), + Felt::new(0x24321cce4f1942f8), + Felt::new(0xb1829ec9d60aac8f), ]), RpoDigest::new([ - Felt::new(3870617340693651786), - Felt::new(2748490321246408799), - Felt::new(8501743976565218963), - Felt::new(1660720190266083389), + Felt::new(0x9f7afc6634a82d1), + Felt::new(0x496e26bc17af352b), + Felt::new(0x8216f090e1d13381), + Felt::new(0x610cf5a3b3e190f9), ]), RpoDigest::new([ - Felt::new(2121119282758520982), - Felt::new(9042267662074029772), - Felt::new(15431993929052434204), - Felt::new(10659345458998811701), + Felt::new(0xb5f8c141a9acd007), + Felt::new(0x4430345ace970576), + Felt::new(0x64d97e5533db3170), + Felt::new(0x95c016d769b0fc2d), ]), RpoDigest::new([ - Felt::new(15206763021853065070), - Felt::new(15268692497656424421), - Felt::new(13335448435922172445), - Felt::new(3421340628484408379), + Felt::new(0x88820d6a7ba5a94a), + Felt::new(0x27b614d79eb7b30b), + Felt::new(0xff2751e904085d5f), + Felt::new(0x752509a0860b37d), ]), RpoDigest::new([ - Felt::new(5175159910654039438), - Felt::new(10258564296733764665), - Felt::new(235961379704359454), - Felt::new(18007006485615491006), + Felt::new(0x1070bc84bb53a855), + Felt::new(0x1edad3d5da84e59b), + Felt::new(0x8efd48a13e4dfe0d), + Felt::new(0x3ab20af6203aba62), ]), RpoDigest::new([ - Felt::new(9455184082727641653), - Felt::new(6634498452861935579), - Felt::new(18189776179964984407), - Felt::new(3546641211720870472), + Felt::new(0xb4d6d3cc85438d08), + Felt::new(0x5592639fb2792724), + Felt::new(0x5939996ea4c52176), + Felt::new(0xaa83a79236367ee7), ]), RpoDigest::new([ - Felt::new(2566088177506289568), - Felt::new(7785941571143323572), - Felt::new(13948908169667863201), - Felt::new(8557252288425473395), + Felt::new(0x4c08ac735aa1925a), + Felt::new(0x84951e177ac84e86), + Felt::new(0xd5b2657778d3271a), + Felt::new(0x375f75333654a77c), ]), RpoDigest::new([ - Felt::new(8801845050152766755), - Felt::new(514652983374395586), - Felt::new(13975919271481418443), - Felt::new(17480955484347349170), + Felt::new(0x2fcbd8fcd125e5), + Felt::new(0xd8f711ed1b369d43), + Felt::new(0x9688301695b6bcd4), + Felt::new(0x52a010319401179), ]), RpoDigest::new([ - Felt::new(7078477424334594989), - Felt::new(9975053207879493059), - Felt::new(5220656123503260168), - Felt::new(13795787984352794188), + Felt::new(0x1c67f8fde4c9c070), + Felt::new(0x438ccdf9d82b3a3f), + Felt::new(0xb9324515d5547ff5), + Felt::new(0x85ff37504c8230f0), ]), RpoDigest::new([ - Felt::new(1478357986561897612), - Felt::new(3963701567400985039), - Felt::new(10269836564499521403), - Felt::new(11874873630603798755), + Felt::new(0xcf8b6fabda4621f3), + Felt::new(0x1df94bb4ea8aeb6d), + Felt::new(0x8efffb7e8996b9e5), + Felt::new(0xa9aef575e8a86c4d), ]), RpoDigest::new([ - Felt::new(936391814816943993), - Felt::new(6085855616346025677), - Felt::new(5782721339195502211), - Felt::new(10409491632083436908), + Felt::new(0x6e20862a64baaaef), + Felt::new(0xc54fbbfa034d6f1b), + Felt::new(0x16d9fd099f5bba71), + Felt::new(0xe4ac4cf3186fae83), ]), RpoDigest::new([ - Felt::new(11138475264090866271), - Felt::new(17799626597540451271), - Felt::new(17968790388406362807), - Felt::new(9539434947296310791), + Felt::new(0x12914625293d7f84), + Felt::new(0xd3b46add4f77be8), + Felt::new(0xaac8846e6eeb9acd), + Felt::new(0xab6a69452b4b167b), ]), RpoDigest::new([ - Felt::new(13051724588530357940), - Felt::new(8058102530250142518), - Felt::new(1861782711432586670), - Felt::new(2928050228215055187), + Felt::new(0x69652e812cdfe03d), + Felt::new(0x22731622b139de96), + Felt::new(0xd7226e9a887f368d), + Felt::new(0xe9bbf6ad8f51ee13), ]), RpoDigest::new([ - Felt::new(10650694022550988030), - Felt::new(5634734408638476525), - Felt::new(9233115969432897632), - Felt::new(1437907447409278328), + Felt::new(0xc39a01964af141d7), + Felt::new(0xb5ab2062263dcaa2), + Felt::new(0x1d7fbcd9204cbd34), + Felt::new(0xd48c517d5543c163), ]), RpoDigest::new([ - Felt::new(9720135276484706819), - Felt::new(9350120041401976641), - Felt::new(1348777594376050933), - Felt::new(13138246165242825648), + Felt::new(0x44118fda0c2b4af2), + Felt::new(0x487d307ce7444bb2), + Felt::new(0x171b7c6a17d734b2), + Felt::new(0xd9a737ddf65949d9), ]), RpoDigest::new([ - Felt::new(10866643979409126085), - Felt::new(13790633638103642042), - Felt::new(6374461622011119670), - Felt::new(5702679962735491362), + Felt::new(0xc2cdc1b940450fec), + Felt::new(0x29864b9632eff0cd), + Felt::new(0x9ae31f150850e78c), + Felt::new(0xf9f9d0ef1092be87), ]), RpoDigest::new([ - Felt::new(5257277882444261955), - Felt::new(8511211402794551302), - Felt::new(3294838877645533839), - Felt::new(4084864647832858048), + Felt::new(0x1703dd34002f3862), + Felt::new(0xf04b44446be81ea1), + Felt::new(0x8da51598849beb99), + Felt::new(0x8112e155f7f856a0), ]), RpoDigest::new([ - Felt::new(7948776578097466250), - Felt::new(8630046431048474853), - Felt::new(11549811661672434609), - Felt::new(14329713552208961509), + Felt::new(0x3d4da8351f41dc1c), + Felt::new(0x682e55817f56f30b), + Felt::new(0xf20cc7fe5b98b951), + Felt::new(0x8297d3de042785d4), ]), RpoDigest::new([ - Felt::new(734617692582477804), - Felt::new(11871516935077749937), - Felt::new(12085935336918533812), - Felt::new(11028098016323141988), + Felt::new(0x1f9d07a435a6d13e), + Felt::new(0x789a1330825c199a), + Felt::new(0x6e058e9dbc30f3a0), + Felt::new(0xb09be46b59290984), ]), RpoDigest::new([ - Felt::new(10937083382606895486), - Felt::new(12203867463821771187), - Felt::new(13369919265612777227), - Felt::new(2521482611471096233), + Felt::new(0xaf2d49c9a3975d21), + Felt::new(0xebd4d399fc30a751), + Felt::new(0x224a3884ca353e5d), + Felt::new(0xbebba344bbe055a7), ]), RpoDigest::new([ - Felt::new(1242037330294600071), - Felt::new(8643213198640797337), - Felt::new(14112360612081236212), - Felt::new(11296904697431650998), + Felt::new(0xdf576dc16b0abc3f), + Felt::new(0x40439af403c36338), + Felt::new(0x317b1f2308849c53), + Felt::new(0x91e5c9d14107cb04), ]), RpoDigest::new([ - Felt::new(11958494925108187724), - Felt::new(6059642826232274823), - Felt::new(1563918267677757605), - Felt::new(266509853282035592), + Felt::new(0x93af916aa15f97e2), + Felt::new(0x50d4aec3e408fba7), + Felt::new(0xd16bd5f71b6d6915), + Felt::new(0x27b96db871be03ef), ]), RpoDigest::new([ - Felt::new(17288335252189973373), - Felt::new(3243363076395469373), - Felt::new(8880515798614590986), - Felt::new(10260780639137628077), + Felt::new(0x72fce6dd7d54e348), + Felt::new(0x632a2e8b6177c670), + Felt::new(0xefd897bebdc4ec2b), + Felt::new(0xfe66bfe440033790), ]), RpoDigest::new([ - Felt::new(1839714959437284152), - Felt::new(12088193186987715006), - Felt::new(10200898335013164008), - Felt::new(12768529781145127245), + Felt::new(0xc581364aef408d6a), + Felt::new(0xfcc7efb35cccae32), + Felt::new(0xee0a97dded065fbf), + Felt::new(0x2b1eb2c45fd0e633), ]), RpoDigest::new([ - Felt::new(1537615626967151439), - Felt::new(11731506816677487155), - Felt::new(4748463589169553420), - Felt::new(17495851576537541106), + Felt::new(0x9e460e8159152a88), + Felt::new(0xcc5a2946f03bf507), + Felt::new(0x95535e9cf29e4ab9), + Felt::new(0x29b23d31ffe6df18), ]), RpoDigest::new([ - Felt::new(957733314860117562), - Felt::new(15623410588944187169), - Felt::new(4321611031548662227), - Felt::new(12856104259650439278), + Felt::new(0xbae2c405d8ba715d), + Felt::new(0xb886f0545ae16153), + Felt::new(0x728d5965a4cdfc0b), + Felt::new(0x86bd552048f3ebc4), ]), RpoDigest::new([ - Felt::new(14827447693720375746), - Felt::new(17296925942589213350), - Felt::new(13524332314559504765), - Felt::new(15663886706087995199), + Felt::new(0x3a4c6dbaa6feda93), + Felt::new(0x8a32917885a3f22c), + Felt::new(0xd6016ba7fc1a0717), + Felt::new(0x3bfd41569497b156), ]), RpoDigest::new([ - Felt::new(18185978518863914335), - Felt::new(936586966360019113), - Felt::new(497299419609993926), - Felt::new(1977881506773614749), + Felt::new(0xa907fad371653f15), + Felt::new(0x6be9ce6ac746f5bc), + Felt::new(0x1bee5ac8750d2444), + Felt::new(0x16050d83d4f7a90c), ]), RpoDigest::new([ - Felt::new(8635338869442206704), - Felt::new(11671305615285950885), - Felt::new(15253023094703789604), - Felt::new(7398108415970215319), + Felt::new(0x4b194182aa7e9324), + Felt::new(0x813af49c845cea5e), + Felt::new(0x6886f4d8628bab16), + Felt::new(0xe3b6ef1419e2432c), + ]), + RpoDigest::new([ + Felt::new(0x3edc103de28f1fac), + Felt::new(0xb6a05b8802d6ed5c), + Felt::new(0xf320c3f130a175c8), + Felt::new(0x326c8bb02f9a51f6), + ]), + RpoDigest::new([ + Felt::new(0x5b1ac27a49b5d1da), + Felt::new(0x9e1fa75b04da7545), + Felt::new(0x9a522396a1cd68af), + Felt::new(0x91a4d435f3fcd43f), + ]), + RpoDigest::new([ + Felt::new(0x318ac5d8f1e489ce), + Felt::new(0x339e7a0b2aec5843), + Felt::new(0x38f15bf9832a2c28), + Felt::new(0x5e3fef94216f72f1), + ]), + RpoDigest::new([ + Felt::new(0xc43e0723d2a7e79c), + Felt::new(0xa06167cc0ebdf1e5), + Felt::new(0xe62f10089af57ba6), + Felt::new(0x838c863d60b859a2), + ]), + RpoDigest::new([ + Felt::new(0xd10456af5f30e5d5), + Felt::new(0x235df7fe21fb912c), + Felt::new(0xe5acc29d13d80779), + Felt::new(0x580b83247a1f6524), + ]), + RpoDigest::new([ + Felt::new(0x2a8b1bf7e9bc5675), + Felt::new(0x9e523f2d659a3e30), + Felt::new(0x3ecfdb1615666b74), + Felt::new(0xf53746b86fedee7f), + ]), + RpoDigest::new([ + Felt::new(0xa12095b3b22680a9), + Felt::new(0x3010ad751585161d), + Felt::new(0xfb9c0ea33c7437b2), + Felt::new(0x9225d8151ec724a8), + ]), + RpoDigest::new([ + Felt::new(0x1b09eac8ad815107), + Felt::new(0x33cb241ad41b562d), + Felt::new(0xa04f457b4cd1ece9), + Felt::new(0x84f27a45985d700e), + ]), + RpoDigest::new([ + Felt::new(0xe5598d92d1507185), + Felt::new(0x84aa2bf7d87a26e8), + Felt::new(0x158f0e13550dec2a), + Felt::new(0x54d699e5eb65ee63), + ]), + RpoDigest::new([ + Felt::new(0x902e89f122f8f8f7), + Felt::new(0xc2da7127af8c699a), + Felt::new(0x75762e75b77a1662), + Felt::new(0x7e683b3c116af130), + ]), + RpoDigest::new([ + Felt::new(0xabc2aa2ecd2316dd), + Felt::new(0x44558fa721857f00), + Felt::new(0xf61dd475fdbc23d0), + Felt::new(0x22ba84332065a9e8), + ]), + RpoDigest::new([ + Felt::new(0x5aa94e045e4bb7ae), + Felt::new(0xf6ddadbdd8747728), + Felt::new(0xeeab65efab2a1d2), + Felt::new(0xd12cc579c49b9db5), + ]), + RpoDigest::new([ + Felt::new(0x71ea68262a73196a), + Felt::new(0x9612483af09f1bde), + Felt::new(0x7fe5fd69bbf241a4), + Felt::new(0x34de27c57b37975d), + ]), + RpoDigest::new([ + Felt::new(0xf29bc8ba140714f6), + Felt::new(0xf0b44caca4f6561e), + Felt::new(0x742695d702446774), + Felt::new(0x7e1437b52ee16c0c), + ]), + RpoDigest::new([ + Felt::new(0x13f6180493eaa129), + Felt::new(0x8fa2e77f499c911c), + Felt::new(0x1223e5ccda975bf), + Felt::new(0xc2a362e5449eac8b), + ]), + RpoDigest::new([ + Felt::new(0xcf1254ec733c8fb0), + Felt::new(0x34359ae1e2272fc9), + Felt::new(0xce928a65262d59d5), + Felt::new(0xc84e1f72e2e78101), + ]), + RpoDigest::new([ + Felt::new(0x8841b659676a2df5), + Felt::new(0x4c808c965135ff8f), + Felt::new(0x374d574fd96ee7d1), + Felt::new(0xa0ae0e5765bc8716), + ]), + RpoDigest::new([ + Felt::new(0xba3692cf34a6eb7a), + Felt::new(0x384dce8b1fd8fcd5), + Felt::new(0x248f1c83f6cf6055), + Felt::new(0xbf50ca14b3c5b022), + ]), + RpoDigest::new([ + Felt::new(0x18611824fa468341), + Felt::new(0xaab4187ff224ec04), + Felt::new(0x4ad742d8a070d084), + Felt::new(0xfa3bb42df7d86480), + ]), + RpoDigest::new([ + Felt::new(0x2ab25bf43fc462b5), + Felt::new(0x6ac0cc243f54b796), + Felt::new(0x2401eabf391a2199), + Felt::new(0x62a71dae211b983), + ]), + RpoDigest::new([ + Felt::new(0xbc5e568df9f18772), + Felt::new(0xee864850b75a99ba), + Felt::new(0x2a53e3e6776ae456), + Felt::new(0x8eb51bedbe483d7c), + ]), + RpoDigest::new([ + Felt::new(0xce8161f4c705bfbb), + Felt::new(0xf1071a4e343a37e9), + Felt::new(0xddc4878a9e5de00f), + Felt::new(0xee33d737cd3c5dc8), + ]), + RpoDigest::new([ + Felt::new(0x9eadd43aebfcd43d), + Felt::new(0xf35cec43429c0a95), + Felt::new(0xcad253fc16b63e5a), + Felt::new(0xea25dc9baaf21d38), + ]), + RpoDigest::new([ + Felt::new(0xa85a87fbf220f449), + Felt::new(0x1db1c09109882161), + Felt::new(0xab5139cb30eb2c88), + Felt::new(0xe62f2ade31d95b14), + ]), + RpoDigest::new([ + Felt::new(0xad3fae6f7f635376), + Felt::new(0x21e5dba9b8e21ac8), + Felt::new(0x86506eeeba6c7151), + Felt::new(0x6bf71fdffc8d9ae7), + ]), + RpoDigest::new([ + Felt::new(0x37ec52a9396f4574), + Felt::new(0xf19404a514aa9285), + Felt::new(0x3ed5ae669769c4e7), + Felt::new(0x2286b493b85c9481), + ]), + RpoDigest::new([ + Felt::new(0xc37fc37b83940bd2), + Felt::new(0xe3d67417540b620b), + Felt::new(0x1495f7a7848dde0a), + Felt::new(0xeaf4f9c053465ff), + ]), + RpoDigest::new([ + Felt::new(0x80131752569df8f0), + Felt::new(0x30720a862b82f732), + Felt::new(0xabed5fb95dbe678b), + Felt::new(0x6cf7da37075ad45e), + ]), + RpoDigest::new([ + Felt::new(0xa318ea66909473fe), + Felt::new(0x4a6c6ebc4bee8b3c), + Felt::new(0xf0d622f04ce1b02e), + Felt::new(0x92c2f8e192c000a1), + ]), + RpoDigest::new([ + Felt::new(0xb39d728756dca017), + Felt::new(0x4f66acee5bcd7d98), + Felt::new(0xf623331bed29e125), + Felt::new(0xbcfc777f0eb03793), + ]), + RpoDigest::new([ + Felt::new(0x6cdabd98e067b039), + Felt::new(0xd6356a27c3df3ddc), + Felt::new(0xd5afb88820db9d2f), + Felt::new(0x8203a7adfa667bfc), + ]), + RpoDigest::new([ + Felt::new(0x1ddef8e482da50e0), + Felt::new(0x7fa3c9c0865609ec), + Felt::new(0x6ca762886d4d6227), + Felt::new(0x9a95160f2a4fe5d9), + ]), + RpoDigest::new([ + Felt::new(0x607230c3b366dbd5), + Felt::new(0x5b996a7d876b7602), + Felt::new(0xf61df5d15469c8ea), + Felt::new(0x9bb4f5c06ac49403), + ]), + RpoDigest::new([ + Felt::new(0x6a27c9e7082595e7), + Felt::new(0xbf93eb89e2090438), + Felt::new(0xd2db18139bedc636), + Felt::new(0x79710c33a1f1f612), + ]), + RpoDigest::new([ + Felt::new(0xf54e4461aa09608b), + Felt::new(0x898a7b52804d88c9), + Felt::new(0xbc548fab0257ea25), + Felt::new(0xe783017a62b49474), + ]), + RpoDigest::new([ + Felt::new(0xf7efdb376a7734c9), + Felt::new(0x2d4ded56d9ef2076), + Felt::new(0xa17d90a509b879d0), + Felt::new(0xcf012a20045b29e1), + ]), + RpoDigest::new([ + Felt::new(0x37e40a30232a4f06), + Felt::new(0xfbd9877fb761052e), + Felt::new(0xc4c41f56a70377cd), + Felt::new(0x631e942f6680d4cc), + ]), + RpoDigest::new([ + Felt::new(0xcf868b6d54b515a5), + Felt::new(0xa522edf7c43f7aee), + Felt::new(0x66057652f34d479), + Felt::new(0x59f4a86223bc80bd), + ]), + RpoDigest::new([ + Felt::new(0xb7214ce5a0ba8dfd), + Felt::new(0x5c7a6e583e4e255e), + Felt::new(0xabc8369f8bf38a1c), + Felt::new(0xb5db79ae07f0689c), + ]), + RpoDigest::new([ + Felt::new(0x18c980169ef2d0bb), + Felt::new(0x6526b64df8eb4eac), + Felt::new(0xfe4d8327ca5bd91a), + Felt::new(0xe36d607069c7dd85), + ]), + RpoDigest::new([ + Felt::new(0x602a97209948e5cc), + Felt::new(0xb7d19db914da726), + Felt::new(0xe4e43672c24d376c), + Felt::new(0x8bb9f7465e019213), + ]), + RpoDigest::new([ + Felt::new(0x187bff077d393e3d), + Felt::new(0x17fb9a97c5055580), + Felt::new(0x618469c060eb2719), + Felt::new(0xfc7be4b58477e5ac), + ]), + RpoDigest::new([ + Felt::new(0x1d40fcbc7a25cc97), + Felt::new(0xaee142f7cebadbd5), + Felt::new(0x22dbaed94300ddf8), + Felt::new(0xe069c36278753a06), + ]), + RpoDigest::new([ + Felt::new(0xcd1e21c5f02ce44d), + Felt::new(0x3b0ddbaa04daff25), + Felt::new(0xbb55cd14f54818c7), + Felt::new(0xc57f1b84ed302102), + ]), + RpoDigest::new([ + Felt::new(0x5c8e1f56cbdb0f87), + Felt::new(0xeeeb31b4d317cf1d), + Felt::new(0x8bf45cd3659a6d1), + Felt::new(0x9e179aa20693175a), + ]), + RpoDigest::new([ + Felt::new(0x10f58975fbb0fca), + Felt::new(0x5f35c19eb0f615c1), + Felt::new(0x9870cdafe46a3d), + Felt::new(0xcec9d9f3925df88b), + ]), + RpoDigest::new([ + Felt::new(0x89e90b2f029b50c0), + Felt::new(0xd78a4223d0036c8a), + Felt::new(0x996b326a1d5cd76d), + Felt::new(0x5b314d29bb1694e3), + ]), + RpoDigest::new([ + Felt::new(0x1be6e6955ba0f3a8), + Felt::new(0xc7e07c49076315ef), + Felt::new(0x93e91de5c7849fb2), + Felt::new(0xe81bc86fc641596f), + ]), + RpoDigest::new([ + Felt::new(0x5320464735f18522), + Felt::new(0x1a741214432ca63d), + Felt::new(0xaf3ed59d324bdbe8), + Felt::new(0x2493eb414c91ac94), + ]), + RpoDigest::new([ + Felt::new(0x35897b61f231fa86), + Felt::new(0xb1531e954332f229), + Felt::new(0x92e950b1c1f874a), + Felt::new(0x469de0412ca52491), + ]), + RpoDigest::new([ + Felt::new(0x1ecea76deca59ec5), + Felt::new(0xe884b570f5d54e45), + Felt::new(0x58939f3a1b5bc7e1), + Felt::new(0xf14eab10f926958f), + ]), + RpoDigest::new([ + Felt::new(0x26251aa927a69723), + Felt::new(0xb1808fe0795ab008), + Felt::new(0xd195fe923d1944c9), + Felt::new(0x2334a61c28dc63c), + ]), + RpoDigest::new([ + Felt::new(0xe4b659081d9cf4e4), + Felt::new(0xf1174a5f72916819), + Felt::new(0x1de902b42b3b4054), + Felt::new(0xbe2bc215120367d0), + ]), + RpoDigest::new([ + Felt::new(0xfc87b8043d32428f), + Felt::new(0x8f8cb244e3ddf6da), + Felt::new(0xc7539186ece143a7), + Felt::new(0xf28008f902075229), + ]), + RpoDigest::new([ + Felt::new(0xf76c24c9f86c44d3), + Felt::new(0x97c7abcbb6d07d35), + Felt::new(0x9d8e37a1697a0d4), + Felt::new(0xa3f818e48770f5fa), + ]), + RpoDigest::new([ + Felt::new(0x885686c79c1cd95e), + Felt::new(0xcdebe76fd203c23e), + Felt::new(0xdf9b7cd5099673ed), + Felt::new(0xe60438536ad13270), + ]), + RpoDigest::new([ + Felt::new(0x7790809942b9389d), + Felt::new(0xa3d82432c31de99), + Felt::new(0xaea11fece88c7d27), + Felt::new(0x5cc764da96d0b2f0), + ]), + RpoDigest::new([ + Felt::new(0x80e555c41170427f), + Felt::new(0x87e68144276d79c8), + Felt::new(0xebdc63f28aa58a53), + Felt::new(0x168dd22672627819), + ]), + RpoDigest::new([ + Felt::new(0xea1dc59c29da5b6c), + Felt::new(0xa33188c0a077761), + Felt::new(0xabd3c84cddbe1477), + Felt::new(0xd28244bc92f36e0f), + ]), + RpoDigest::new([ + Felt::new(0xdadc2beb7ccfe3fa), + Felt::new(0x218532461f981fb4), + Felt::new(0xf0455f1d4e2f9732), + Felt::new(0xa7338b43d2b7e62d), + ]), + RpoDigest::new([ + Felt::new(0x195d8bc1cfe2711a), + Felt::new(0x44e392ba7e259f47), + Felt::new(0x480120d41e18ab3c), + Felt::new(0x2056ffb29c2d89d1), + ]), + RpoDigest::new([ + Felt::new(0x382e33ba5fe6ada3), + Felt::new(0x45402a8903efebc9), + Felt::new(0xb9b0d63a59c70da), + Felt::new(0x7afebd4726d8cfe5), + ]), + RpoDigest::new([ + Felt::new(0xbf60bf6b45a4c9d), + Felt::new(0xfb5b9b553646f19c), + Felt::new(0x9949b60ce7639da3), + Felt::new(0x9c62552c0d1868ff), + ]), + RpoDigest::new([ + Felt::new(0xdb2a0aba0fc5e4f8), + Felt::new(0x8ee4f01d4b0fa49e), + Felt::new(0xd70a17a77b5c4a03), + Felt::new(0x57aaaa5b48fea66e), + ]), + RpoDigest::new([ + Felt::new(0x6d635940443564cb), + Felt::new(0xc7fbf0e26b5e3ff6), + Felt::new(0xa45bce664368b65e), + Felt::new(0xd6c5c1a92be0c60d), + ]), + RpoDigest::new([ + Felt::new(0x6ea62d6033fb2dd3), + Felt::new(0x1a37910cf90ec6d8), + Felt::new(0x83d826e9933760b5), + Felt::new(0xf8387c90d9c6b5a9), + ]), + RpoDigest::new([ + Felt::new(0x134766f1da2fbc91), + Felt::new(0xcfaeea545df2c757), + Felt::new(0xd0accefaed1eaa0f), + Felt::new(0xec38d4053f84b163), + ]), + RpoDigest::new([ + Felt::new(0xb02ad1e757380aee), + Felt::new(0x4538b8ea13112d4), + Felt::new(0xb2d761fe842a2a85), + Felt::new(0x8e98d58adf5a1f29), + ]), + RpoDigest::new([ + Felt::new(0x44603d9549ddee64), + Felt::new(0x43de72d570967bbb), + Felt::new(0x4a3e71144e62d0fa), + Felt::new(0xffb2fdcb48965939), + ]), + RpoDigest::new([ + Felt::new(0x606f3ee12fe9ec0c), + Felt::new(0xe7d494ab8e483d87), + Felt::new(0x3b47f7c0d316cd4a), + Felt::new(0x86f941c7fa834581), + ]), + RpoDigest::new([ + Felt::new(0x30c2385facf08b86), + Felt::new(0x4446168e25ac2c21), + Felt::new(0x61c6db1c3f283b21), + Felt::new(0x2fdf6bc360bf803), + ]), + RpoDigest::new([ + Felt::new(0xeec8d9cc3e46d243), + Felt::new(0x65bcae511dcce39), + Felt::new(0xd3da5bbfdbd09cd3), + Felt::new(0xe7c35fc3d11216a5), + ]), + RpoDigest::new([ + Felt::new(0x841fb6fb35e7b49b), + Felt::new(0xfc4e2e1239caa7b8), + Felt::new(0x37cb93ec88f102e5), + Felt::new(0xa707a1556032152c), + ]), + RpoDigest::new([ + Felt::new(0x37c67bd7b7cef984), + Felt::new(0x75bbe46da2ee5c90), + Felt::new(0x3a5c568d1f71cab1), + Felt::new(0x36939cdca2dc0b55), + ]), + RpoDigest::new([ + Felt::new(0x4f76756a55f3a644), + Felt::new(0xd30f8fa45394aff4), + Felt::new(0x65c55096158b202f), + Felt::new(0x368a5fb0b0d475d0), + ]), + RpoDigest::new([ + Felt::new(0xa9b9acd256cabb0f), + Felt::new(0xd8b1170f301208c7), + Felt::new(0xab152f908d46bf8), + Felt::new(0x1b7a10556730ec16), + ]), + RpoDigest::new([ + Felt::new(0xd967a72076e3059c), + Felt::new(0xbd1015a08ffe8881), + Felt::new(0xf72f186dde0c6e78), + Felt::new(0xa58910205352895a), + ]), + RpoDigest::new([ + Felt::new(0x130333f2fd400a4d), + Felt::new(0xf20104837a118d6e), + Felt::new(0xda1e5d608fb9062c), + Felt::new(0xb8ac5c76d60950b8), + ]), + RpoDigest::new([ + Felt::new(0x65d0deae6fb0c6cb), + Felt::new(0x1b442ae344dcd9e7), + Felt::new(0x1eedabab8fc07fa4), + Felt::new(0xb0dc89b96f256189), + ]), + RpoDigest::new([ + Felt::new(0xef88de626968c17a), + Felt::new(0x569a01072cdbbc2b), + Felt::new(0xc99bbba6d083c68f), + Felt::new(0x9ed4a176fe341849), + ]), + RpoDigest::new([ + Felt::new(0x5d49d1e9d17448a6), + Felt::new(0x6974d510bc47ee66), + Felt::new(0xbcbea4dec0b68586), + Felt::new(0xdaa5457e0cfd3e61), + ]), + RpoDigest::new([ + Felt::new(0x9fceba739503cda0), + Felt::new(0xb9daf271ac42c8ba), + Felt::new(0x10fe3e8de8680d83), + Felt::new(0xd7e1dc73ced7730b), + ]), + RpoDigest::new([ + Felt::new(0x93ec6c422d4271ea), + Felt::new(0x73923813232b0e70), + Felt::new(0xbbe6a4441a900b65), + Felt::new(0x36b2164f37c9319b), + ]), + RpoDigest::new([ + Felt::new(0xce3ecb2eed624694), + Felt::new(0xb7e1d75fff7a454c), + Felt::new(0x86c24aa3a8d92d2b), + Felt::new(0xb1ba74cafa9ce649), + ]), + RpoDigest::new([ + Felt::new(0xb5fae724eb357479), + Felt::new(0x359532ddc4840cb9), + Felt::new(0x4b111251e037e9fa), + Felt::new(0xfcdab1cdd314c1d9), + ]), + RpoDigest::new([ + Felt::new(0xb3a89464d21c9ff1), + Felt::new(0x8136e1b457a59ca8), + Felt::new(0x88b0fa606b53c4d5), + Felt::new(0x89645f8a9dfe97a2), + ]), + RpoDigest::new([ + Felt::new(0xfe115ef35b814cbf), + Felt::new(0x63de467fb93b6851), + Felt::new(0x17c73b03c9f44ad8), + Felt::new(0x53742721f568b5be), + ]), + RpoDigest::new([ + Felt::new(0xd8110ea6e905cc2), + Felt::new(0xd67b3c7cea25100), + Felt::new(0x9e49b38ed51d1c60), + Felt::new(0xe9e24f9b597c9bfd), + ]), + RpoDigest::new([ + Felt::new(0xefe9086b5bb5a504), + Felt::new(0x991f92a90c9346a3), + Felt::new(0xe4fab215a20f453b), + Felt::new(0x4e4d4dde9146d61a), + ]), + RpoDigest::new([ + Felt::new(0xaa998c3b26497ffa), + Felt::new(0x985bd5cf4ccefb3c), + Felt::new(0xce44e80aa02424bb), + Felt::new(0x75158a37503aed75), + ]), + RpoDigest::new([ + Felt::new(0xdb61760c917116f1), + Felt::new(0xf378c9645174a832), + Felt::new(0x1216aa71b73e7fac), + Felt::new(0x8a4e7f0591a129fd), + ]), + RpoDigest::new([ + Felt::new(0xaf11a04daaf4ed67), + Felt::new(0xd3e59f0d7dad9064), + Felt::new(0x30c206089a2c294d), + Felt::new(0xe104db59761e8a99), + ]), + RpoDigest::new([ + Felt::new(0x70b545ba7a6d447), + Felt::new(0x6ac0e423ddf68913), + Felt::new(0xf9b50997257bb033), + Felt::new(0xdac37c7b1c18b48a), + ]), + RpoDigest::new([ + Felt::new(0xd182b9dff0fcd5c0), + Felt::new(0xf87619ae86b6eb02), + Felt::new(0x6838c1b612b17cb5), + Felt::new(0x9b705d5b6bcf92c), + ]), + RpoDigest::new([ + Felt::new(0xfba622b3026c6193), + Felt::new(0xdacde486f8129b96), + Felt::new(0xd5acd22a7c2cf6aa), + Felt::new(0xf5beb40535e6c0f2), + ]), + RpoDigest::new([ + Felt::new(0x59bde17b2d501969), + Felt::new(0xb4abe1389123d3b9), + Felt::new(0x683d8dd8635d9a67), + Felt::new(0x347e01da4c07833), + ]), + RpoDigest::new([ + Felt::new(0x4e28956ab7162a06), + Felt::new(0xccfcc7358f48c727), + Felt::new(0x7b3485f20c979144), + Felt::new(0xeeb27fa694f1c8fd), + ]), + RpoDigest::new([ + Felt::new(0x275b2c0ee883807b), + Felt::new(0x8f68f2016c1391cd), + Felt::new(0xb59fdccb20322765), + Felt::new(0xeb9b902c5351d5d4), + ]), + RpoDigest::new([ + Felt::new(0xb767d8cb8816cc8e), + Felt::new(0xbd29bb02cdcbc9af), + Felt::new(0xeb1dca9bfebee6f), + Felt::new(0x57597da8109c0354), + ]), + RpoDigest::new([ + Felt::new(0xeb32a8db8cf216), + Felt::new(0xeb5532ac68f304c1), + Felt::new(0x9bca72ffccb957ee), + Felt::new(0x33d4b152ebedb841), + ]), + RpoDigest::new([ + Felt::new(0x439b20dce9810169), + Felt::new(0x2b693e2530a1b88c), + Felt::new(0x36b8898f4e900c7a), + Felt::new(0x7bf5064dde3a0da1), + ]), + RpoDigest::new([ + Felt::new(0x8794201ce6158fe0), + Felt::new(0xfcc53644557471f3), + Felt::new(0xa66d87f6ae6f64d0), + Felt::new(0x4e876d9d933b2ad0), + ]), + RpoDigest::new([ + Felt::new(0x6ff8f4900e43bab6), + Felt::new(0x40014f298cb7b9a3), + Felt::new(0x9d6b252ff946ee3d), + Felt::new(0xb014d99ab8508072), + ]), + RpoDigest::new([ + Felt::new(0x9cdd5a4a37511cae), + Felt::new(0x684444122d770c18), + Felt::new(0x8982944b22a22577), + Felt::new(0x50a58d944629de54), + ]), + RpoDigest::new([ + Felt::new(0x853f5b8ad557fac3), + Felt::new(0xdab1743c03b8da56), + Felt::new(0xc70d6683d4f4c086), + Felt::new(0x2f1d0f67a5dfae4c), + ]), + RpoDigest::new([ + Felt::new(0xf3b6fe76eb11284), + Felt::new(0xbeb9e98b146c63a8), + Felt::new(0xc7e8824fce7777ad), + Felt::new(0x5229918b04410d6a), + ]), + RpoDigest::new([ + Felt::new(0xc170c46601ffc4f3), + Felt::new(0x1258e8e47103c39b), + Felt::new(0x612e99da984aac99), + Felt::new(0xc82fcfcf56d6dd94), + ]), + RpoDigest::new([ + Felt::new(0xf793819d04d5679d), + Felt::new(0xb738b97ec0a52dd3), + Felt::new(0x4df897389119a098), + Felt::new(0xa5af45eb0d007785), + ]), + RpoDigest::new([ + Felt::new(0xfcf59c6c9d7280e7), + Felt::new(0x662b993b320f3345), + Felt::new(0xeb8e04ba28f156fa), + Felt::new(0xe72233ee6a444749), + ]), + RpoDigest::new([ + Felt::new(0x7ad6b7badfc9e757), + Felt::new(0x3332f340184af6f5), + Felt::new(0xe92a736dcdf52022), + Felt::new(0xf1759f8041119245), + ]), + RpoDigest::new([ + Felt::new(0x166382d3c8ca3a95), + Felt::new(0x36c8c25f971d771a), + Felt::new(0xe82cc977ee1402cc), + Felt::new(0xe13f6dc2ab919177), + ]), + RpoDigest::new([ + Felt::new(0x41fb8b9d5a9ab5e8), + Felt::new(0xb2a608f5d6fbc37d), + Felt::new(0xe81719d9778c54b4), + Felt::new(0xcc0d4373ef2bb8e1), + ]), + RpoDigest::new([ + Felt::new(0x60b3788c45c1bc29), + Felt::new(0xcb38969a428c5423), + Felt::new(0xca11f6b9d957f57c), + Felt::new(0x7a166881648514cb), + ]), + RpoDigest::new([ + Felt::new(0x3548d7dcbe4af37f), + Felt::new(0xb199194f25a6f55a), + Felt::new(0x751017bda8f0d816), + Felt::new(0x80117260c6525c4a), + ]), + RpoDigest::new([ + Felt::new(0x1d1d34d842e95671), + Felt::new(0x38ab2833c4a2bded), + Felt::new(0x3a53caf5f6e20f5f), + Felt::new(0xfee946c5ebce837d), + ]), + RpoDigest::new([ + Felt::new(0x65dbb5d849e46f92), + Felt::new(0x753679d0d09d1250), + Felt::new(0xc539adf9062e7698), + Felt::new(0xd76f4b13d2635af3), + ]), + RpoDigest::new([ + Felt::new(0x9eae446581f7076d), + Felt::new(0x237138e3a1c55ee8), + Felt::new(0xfb54da0b58969484), + Felt::new(0x9a899375c4f10483), + ]), + RpoDigest::new([ + Felt::new(0x88b12578d3c36cc4), + Felt::new(0x9cd868d6a29df146), + Felt::new(0x5443d21524637821), + Felt::new(0xceb3692fad9f76b9), + ]), + RpoDigest::new([ + Felt::new(0x4e45d3e34ebfcb0d), + Felt::new(0xedca6a0f8d90eb2e), + Felt::new(0x1a182dad4a914731), + Felt::new(0xd89946f41f53f106), + ]), + RpoDigest::new([ + Felt::new(0x42a89e1abee8b12d), + Felt::new(0x3af360678475225a), + Felt::new(0x3e28a1cdede0c7a3), + Felt::new(0x25cc76ad1c8788e), + ]), + RpoDigest::new([ + Felt::new(0xa6e955e3b0721737), + Felt::new(0xbbd721b51cbc605f), + Felt::new(0xfa537854669f20a5), + Felt::new(0x2a1ab83bb24d43f7), + ]), + RpoDigest::new([ + Felt::new(0x1c289dc62c4e7907), + Felt::new(0x186437cf1f8b3abf), + Felt::new(0x929244addee54abf), + Felt::new(0x42ec87a9141b58fb), + ]), + RpoDigest::new([ + Felt::new(0xf5ea743b81e59fee), + Felt::new(0x32e2c826d0fc57d4), + Felt::new(0xbc411bca1b745388), + Felt::new(0xc9af8de5bc3b8692), + ]), + RpoDigest::new([ + Felt::new(0xf75129028d95ebab), + Felt::new(0xf9d02d7713784ff2), + Felt::new(0x944b5fea457c946b), + Felt::new(0xa6645d6f5389a91f), + ]), + RpoDigest::new([ + Felt::new(0x236ce9e32fb0131c), + Felt::new(0x92cac3a2b67600b8), + Felt::new(0x437914d1d9e409bb), + Felt::new(0x6bff102a4ea81e38), + ]), + RpoDigest::new([ + Felt::new(0x9906825ec6d45f25), + Felt::new(0x1245ccf4121950a2), + Felt::new(0xff82337f357949bb), + Felt::new(0xe1f59f6b824f97af), + ]), + RpoDigest::new([ + Felt::new(0xd6030ea8046f0d1e), + Felt::new(0x69fd3c732586f5db), + Felt::new(0x765cccb10c151f30), + Felt::new(0xd1bad2f1ba1bdc5b), + ]), + RpoDigest::new([ + Felt::new(0xd4a0cff6578c123e), + Felt::new(0xf11a17948930b14a), + Felt::new(0xd128dd2a4213b53c), + Felt::new(0x2df8fe54f23f6b91), + ]), + RpoDigest::new([ + Felt::new(0xa891ab94552907b2), + Felt::new(0xc64b69708f99b16e), + Felt::new(0xb99d29be9af9df15), + Felt::new(0xa452647d61bd3d7b), + ]), + RpoDigest::new([ + Felt::new(0xd339a7d5f35bdf7e), + Felt::new(0x296127097a98dbdd), + Felt::new(0x66d3eac876c1a984), + Felt::new(0x2714db555ccbdca4), + ]), + RpoDigest::new([ + Felt::new(0x6ca87920a067acd6), + Felt::new(0x55a102d742879de5), + Felt::new(0xa3f57a4b6a56d7f3), + Felt::new(0x18770879a832a177), + ]), + RpoDigest::new([ + Felt::new(0x1208793a0a77e984), + Felt::new(0x89ed5a1f45b34219), + Felt::new(0x8d74e6f75e77ef03), + Felt::new(0xa78cc9b7d29a1ee9), + ]), + RpoDigest::new([ + Felt::new(0x345fffecefd56a65), + Felt::new(0x23dec03b9f663006), + Felt::new(0x66f452be070b34d8), + Felt::new(0xe1c6aaaa54765dc), + ]), + RpoDigest::new([ + Felt::new(0xb7b714ca60c77011), + Felt::new(0x5ffe9476baa10259), + Felt::new(0xbea15cd2b4150dda), + Felt::new(0x4467804ab919e3d8), + ]), + RpoDigest::new([ + Felt::new(0x21c65ff7a4185640), + Felt::new(0x5d20fffcc7779035), + Felt::new(0x92b571249fa728d5), + Felt::new(0xe9e2eeca83b49342), + ]), + RpoDigest::new([ + Felt::new(0x2be46f9d0b431948), + Felt::new(0xa5c1a12c95a56f44), + Felt::new(0xc620adca5cd874), + Felt::new(0x51c64b969c1dc0c9), + ]), + RpoDigest::new([ + Felt::new(0xae9e7f421ab12b3f), + Felt::new(0x5f90e40cdb47bbec), + Felt::new(0x3d2a6be00ce19d31), + Felt::new(0x609632a98643a06c), + ]), + RpoDigest::new([ + Felt::new(0xde0f38c1d4cf5977), + Felt::new(0x3d6f0f44105b0fb2), + Felt::new(0x5f34005bed22db6f), + Felt::new(0x377009a91543c872), + ]), + RpoDigest::new([ + Felt::new(0x8763f1c59420f662), + Felt::new(0xbe15cc9b37fdbdb3), + Felt::new(0xeed75857d5451487), + Felt::new(0x10b3507901fcfb8b), + ]), + RpoDigest::new([ + Felt::new(0x2e40aeb53f7ba2f5), + Felt::new(0xed598b103221e46), + Felt::new(0xed943dae7f3be8f7), + Felt::new(0x7115a61b37a95c30), + ]), + RpoDigest::new([ + Felt::new(0xef9bdb6947abe190), + Felt::new(0xfe47c74c5ad62e2b), + Felt::new(0xf85ee41bc6960d64), + Felt::new(0x55e51b57d5990e2b), + ]), + RpoDigest::new([ + Felt::new(0x81798808c02cffe0), + Felt::new(0x22df79cb29991e51), + Felt::new(0x965714f831e09f90), + Felt::new(0x67ac8aef55047894), + ]), + RpoDigest::new([ + Felt::new(0x321d58dfe138e326), + Felt::new(0xee80b399b444c937), + Felt::new(0x6cbc565ad3d69ccf), + Felt::new(0x6ea9e44922bdc48e), + ]), + RpoDigest::new([ + Felt::new(0xf2a11cd35120f6c3), + Felt::new(0x4e77378973b2ca8), + Felt::new(0x1adb26e6c1630440), + Felt::new(0x42e5495895baee92), + ]), + RpoDigest::new([ + Felt::new(0xe07020daabcdebae), + Felt::new(0x2d5c6456b50a05c3), + Felt::new(0x3050bdec1715085b), + Felt::new(0xe4d05bce00c4f30d), + ]), + RpoDigest::new([ + Felt::new(0xe64d54038f19cbc3), + Felt::new(0xde70db9c37ffc1a6), + Felt::new(0xb4a54682fe025b15), + Felt::new(0xcb4e28247a791f29), + ]), + RpoDigest::new([ + Felt::new(0x9cd2815cef52b092), + Felt::new(0xd46fa4485d7aaf5c), + Felt::new(0x275777fd3b9637f7), + Felt::new(0x684317aab8bb48da), + ]), + RpoDigest::new([ + Felt::new(0xfef7ca5bc1a8531), + Felt::new(0x66da24a709f70f60), + Felt::new(0xc6d0048b0222dc8d), + Felt::new(0xd82cc216eed266b4), + ]), + RpoDigest::new([ + Felt::new(0xc11007d4b20c68e8), + Felt::new(0xb88dea2c92f105d1), + Felt::new(0xfe885fceee959992), + Felt::new(0x4d8c71eeb47c8f90), + ]), + RpoDigest::new([ + Felt::new(0x980fbe89dd050886), + Felt::new(0xdb7ad3a9a9f39be1), + Felt::new(0xf830b6439c7e9bc0), + Felt::new(0xdd76938b7c686c07), + ]), + RpoDigest::new([ + Felt::new(0xb678629d183cf841), + Felt::new(0x664f19c21a6465bc), + Felt::new(0xba142578e79eb9d0), + Felt::new(0x1d57cbcdb2bbcb18), + ]), + RpoDigest::new([ + Felt::new(0xc8822aae44990ed3), + Felt::new(0xf7edce1d2a235ab4), + Felt::new(0x8f35d229e981ca01), + Felt::new(0x50dc4c673c8fc58), + ]), + RpoDigest::new([ + Felt::new(0x86edc215b92306e7), + Felt::new(0xe333ce17fd0f40df), + Felt::new(0xabcca6409bb48e29), + Felt::new(0xa110affea61c2951), + ]), + RpoDigest::new([ + Felt::new(0x269d782ba380a2e4), + Felt::new(0x32bdc65a619310cd), + Felt::new(0x52895e02cb1df991), + Felt::new(0x1749f76388896455), + ]), + RpoDigest::new([ + Felt::new(0x8f99ac4698f12102), + Felt::new(0x9983407c99f948c5), + Felt::new(0x7b89856f980cb407), + Felt::new(0x8acff0a017e68367), + ]), + RpoDigest::new([ + Felt::new(0xe7983973c759318a), + Felt::new(0xb389a6754d7976ca), + Felt::new(0x1d7ded5044c6ede6), + Felt::new(0x4290fc214bd001d7), + ]), + RpoDigest::new([ + Felt::new(0x2a1607db398812eb), + Felt::new(0xbbaaed6d5de95899), + Felt::new(0x53699ff1add9cc48), + Felt::new(0x71c99c4c20a2cbe9), + ]), + RpoDigest::new([ + Felt::new(0x9c211abaf37beba7), + Felt::new(0x44f91ee2c8f84830), + Felt::new(0xdaf1ca95546ff105), + Felt::new(0x4d15218eb972c855), + ]), + RpoDigest::new([ + Felt::new(0x47018b067647ff40), + Felt::new(0xabd25bc93e3cd280), + Felt::new(0x1e33fd9953262800), + Felt::new(0x9b8f68c64db159c3), + ]), + RpoDigest::new([ + Felt::new(0xa20fb2bcf2636418), + Felt::new(0xdc8996a5085ba820), + Felt::new(0x349b49cddfc936d4), + Felt::new(0x1d816c1d0bfef815), + ]), + RpoDigest::new([ + Felt::new(0xc19364d95b916e9b), + Felt::new(0xdcb0cf99f9ca31b5), + Felt::new(0x1b5493d83dca1621), + Felt::new(0xf0758e5e3459a0f9), + ]), + RpoDigest::new([ + Felt::new(0xdf2cb895c1de5b3a), + Felt::new(0xb7347b591512353a), + Felt::new(0xfdbbce7837412eb7), + Felt::new(0x893fa2af604d446f), + ]), + RpoDigest::new([ + Felt::new(0x75d2b8fc6043d11e), + Felt::new(0xa6559eaebeed7d5f), + Felt::new(0xa9379773dad056b3), + Felt::new(0xf3bcf80d0748d3c1), + ]), + RpoDigest::new([ + Felt::new(0x35b731f6c583214a), + Felt::new(0x26249962927dd05f), + Felt::new(0x75fc431a3e7c9e93), + Felt::new(0x170c1036d421083d), + ]), + RpoDigest::new([ + Felt::new(0x1d6fbabfdf0d9096), + Felt::new(0x7d7c9687d9eca6cc), + Felt::new(0xd62974b75d27af1c), + Felt::new(0x93ed9a3825cc1c35), + ]), + RpoDigest::new([ + Felt::new(0xd30946664518ff6e), + Felt::new(0xd3e54aedfb7ba7e5), + Felt::new(0xb91107b2bfadfa1d), + Felt::new(0x2f7b0b2cb06cc43b), + ]), + RpoDigest::new([ + Felt::new(0x47d1dc850260558e), + Felt::new(0x8e5dbde02c522c39), + Felt::new(0x3464d9fe9fd4a1e), + Felt::new(0xf9e5bcfe534437be), + ]), + RpoDigest::new([ + Felt::new(0x83378fd3d8e8a235), + Felt::new(0x5c12793119b6dfdb), + Felt::new(0xfc6f110fe6acbc57), + Felt::new(0x3138336422622a48), + ]), + RpoDigest::new([ + Felt::new(0x239c939a0a9787a0), + Felt::new(0x6c0d389b75f1cfb4), + Felt::new(0xc1947a136998e2a1), + Felt::new(0x76c1779e38ffb573), + ]), + RpoDigest::new([ + Felt::new(0x7a266f705e62cd23), + Felt::new(0x7246a2960a038c2), + Felt::new(0xc1f47087a22fa2cb), + Felt::new(0xf298d4651af0b8b2), + ]), + RpoDigest::new([ + Felt::new(0x623bcdb6cc15e3ad), + Felt::new(0x8a6e820adf4725c3), + Felt::new(0x48737f1587e13208), + Felt::new(0xbf747c141bf3564c), + ]), + RpoDigest::new([ + Felt::new(0x14842ec18fe2348c), + Felt::new(0x3701e593309841cf), + Felt::new(0x8e85c9f1884fdf7b), + Felt::new(0xa4cc06a6b8cae8e3), + ]), + RpoDigest::new([ + Felt::new(0xcfebb64f4bf4f79), + Felt::new(0x54754d70569eeecd), + Felt::new(0x50405a6b79d1ae83), + Felt::new(0x9075f17ccebc056c), + ]), + RpoDigest::new([ + Felt::new(0x9a93d03d0f9c825f), + Felt::new(0xf704fa1580fe53c7), + Felt::new(0xf95df7a7d06a22b7), + Felt::new(0x8462e18adf8b3607), + ]), + RpoDigest::new([ + Felt::new(0xb5210a5cbd9556b4), + Felt::new(0x6fd4218ebf3c7336), + Felt::new(0x19d661830be725ae), + Felt::new(0x28a2862b49428353), + ]), + RpoDigest::new([ + Felt::new(0x93ceddc8a18724fe), + Felt::new(0x4e3299172e9130ed), + Felt::new(0x80229e0e1808c860), + Felt::new(0x13f479347eb7fd78), + ]), + RpoDigest::new([ + Felt::new(0x86e4db88d39d4203), + Felt::new(0x81c24ca37b068741), + Felt::new(0x12b7d21371ca58f5), + Felt::new(0xb6546d4722efcbb0), + ]), + RpoDigest::new([ + Felt::new(0x96ce1320155172c5), + Felt::new(0xbf622c3a5ab86fba), + Felt::new(0x5876a31266554836), + Felt::new(0x4f23fd3646963922), + ]), + RpoDigest::new([ + Felt::new(0x48f59a613c9baa43), + Felt::new(0x761de5ad2fd2c806), + Felt::new(0x2db99e809e35728f), + Felt::new(0x38b05ac131a0edc0), + ]), + RpoDigest::new([ + Felt::new(0x6e4fba2ef071cb8a), + Felt::new(0x77c415814dc5a0e5), + Felt::new(0xa0492c8344a64bb1), + Felt::new(0xc6dd5e8dbc841fe5), + ]), + RpoDigest::new([ + Felt::new(0xa31e2e634187fec), + Felt::new(0xa4c019c12457e4b1), + Felt::new(0xa7b9de27a75ec6b4), + Felt::new(0x990bacb76b218164), + ]), + RpoDigest::new([ + Felt::new(0x97c85363ab1c997e), + Felt::new(0xa95cd8ccab0b41b3), + Felt::new(0xb98b7ebc3680470b), + Felt::new(0x22fe1b13640f19a9), + ]), + RpoDigest::new([ + Felt::new(0x113c9a5f0dfbf187), + Felt::new(0x77f2dc9c46f3d699), + Felt::new(0xc3d92d34c950f8f4), + Felt::new(0x9cc6aaf7ffdaf2b6), + ]), + RpoDigest::new([ + Felt::new(0xa5f51bc68b62924c), + Felt::new(0x54182d0b8b519f87), + Felt::new(0x15b42760e2ad50a5), + Felt::new(0x3b2d54cae046788), + ]), + RpoDigest::new([ + Felt::new(0xefec815078a58b7d), + Felt::new(0x2d02bd88ae2b8a3d), + Felt::new(0x7b3dee115eaac60a), + Felt::new(0x8e659da09a00ffad), + ]), + RpoDigest::new([ + Felt::new(0x1987fb02253b4338), + Felt::new(0xa7c1e3a856b329be), + Felt::new(0x8d90defd305553e8), + Felt::new(0xb132ee2f7526c94d), + ]), + RpoDigest::new([ + Felt::new(0x1556b5441e77734f), + Felt::new(0xa2ceaf4b2822a233), + Felt::new(0x41e5ee79d553e00c), + Felt::new(0xf2cdc04fd67a69f2), + ]), + RpoDigest::new([ + Felt::new(0xd4a8d6025cb6e3a), + Felt::new(0xd8d181277e183f21), + Felt::new(0x3bf9725618de11d3), + Felt::new(0xb26a0eb5a7d9c86e), + ]), + RpoDigest::new([ + Felt::new(0xcdc5ad1e45e351c2), + Felt::new(0xf00b06807244caa6), + Felt::new(0xbbb01495c9a7e17d), + Felt::new(0xd9614df7e656a73f), + ]), + RpoDigest::new([ + Felt::new(0xfc61931c05c4a55f), + Felt::new(0xcff6ce2372650a9), + Felt::new(0x6e6c32ff4dc92c6), + Felt::new(0x1b72d8c04b0e409d), + ]), + RpoDigest::new([ + Felt::new(0x77d6e2f32439dbf0), + Felt::new(0xa1f8ce9eb02419a5), + Felt::new(0xd3ad9fb0cea61624), + Felt::new(0x66ab5c684fbb8597), ]), RpoDigest::new([ZERO; WORD_SIZE]), ]; @@ -416,7 +1557,10 @@ const EMPTY_SUBTREES_64: [RpoDigest; 65] = [ fn all_depths_opens_to_zero() { use super::Rpo256; - for depth in 1..=64 { + // assert the length of the static constants is correct and will cover every possible depth + assert!(EMPTY_SUBTREES.len() > u8::MAX as usize); + + for depth in 0..=u8::MAX { // fetch the subtrees and reverse it so the path is leaf -> root let mut subtree = EmptySubtreeRoots::empty_hashes(depth).to_vec(); subtree.reverse(); @@ -439,16 +1583,3 @@ fn all_depths_opens_to_zero() { .for_each(|(x, computed)| assert_eq!(x, computed)); } } - -#[test] -fn arbitrary_inputs_will_generate_sound_slices() { - let min = &EMPTY_SUBTREES_64[0] as *const RpoDigest; - let max = unsafe { min.add(64) }; - for depth in 0..=64 { - let subtree = EmptySubtreeRoots::empty_hashes(depth); - let first = &subtree[0] as *const RpoDigest; - let last = &subtree[depth as usize] as *const RpoDigest; - assert!(min <= first && first <= max); - assert!(min <= last && last <= max); - } -} From 2d1bc3ba34e0d46b756d9568245254003a1d161c Mon Sep 17 00:00:00 2001 From: "Augusto F. Hack" Date: Wed, 22 Mar 2023 12:56:09 +0100 Subject: [PATCH 21/26] store: added user documentation on usage and purpose --- src/merkle/store/mod.rs | 65 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 60 insertions(+), 5 deletions(-) diff --git a/src/merkle/store/mod.rs b/src/merkle/store/mod.rs index a59bbf5..7777369 100644 --- a/src/merkle/store/mod.rs +++ b/src/merkle/store/mod.rs @@ -1,8 +1,3 @@ -//! An in-memory data store for Merkle-lized data -//! -//! This is a in memory data store for Merkle trees, this store allows all the nodes of a tree -//! (leaves or internal) to live as long as necessary and without duplication, this allows the -//! implementation of efficient persistent data structures use super::{ BTreeMap, BTreeSet, EmptySubtreeRoots, MerkleError, MerklePath, MerklePathSet, MerkleTree, NodeIndex, RootPath, Rpo256, RpoDigest, SimpleSmt, ValuePath, Vec, Word, @@ -17,6 +12,61 @@ pub struct Node { right: RpoDigest, } +/// An in-memory data store for Merkle-lized data. +/// +/// This is a in memory data store for Merkle trees, this store allows all the nodes of multiple +/// trees to live as long as necessary and without duplication, this allows the implementation of +/// space efficient persistent data structures. +/// +/// Example usage: +/// +/// ```rust +/// # use miden_crypto::{ZERO, Felt, Word}; +/// # use miden_crypto::merkle::{NodeIndex, MerkleStore, MerkleTree}; +/// # use miden_crypto::hash::rpo::Rpo256; +/// # const fn int_to_node(value: u64) -> Word { +/// # [Felt::new(value), ZERO, ZERO, ZERO] +/// # } +/// # let A = int_to_node(1); +/// # let B = int_to_node(2); +/// # let C = int_to_node(3); +/// # let D = int_to_node(4); +/// # let E = int_to_node(5); +/// # let F = int_to_node(6); +/// # let G = int_to_node(7); +/// # let H0 = int_to_node(8); +/// # let H1 = int_to_node(9); +/// # let T0 = MerkleTree::new([A, B, C, D, E, F, G, H0].to_vec()).expect("even number of leaves provided"); +/// # let T1 = MerkleTree::new([A, B, C, D, E, F, G, H1].to_vec()).expect("even number of leaves provided"); +/// # let ROOT0 = T0.root(); +/// # let ROOT1 = T1.root(); +/// let mut store = MerkleStore::new(); +/// +/// // the store is initialized with the SMT empty nodes +/// assert_eq!(store.num_internal_nodes(), 64); +/// +/// // populates the store with two merkle trees, common nodes are shared +/// store.add_merkle_tree([A, B, C, D, E, F, G, H0]); +/// store.add_merkle_tree([A, B, C, D, E, F, G, H1]); +/// +/// // every leaf except the last are the same +/// for i in 0..7 { +/// let d0 = store.get_node(ROOT0, NodeIndex::new(3, i)).unwrap(); +/// let d1 = store.get_node(ROOT1, NodeIndex::new(3, i)).unwrap(); +/// assert_eq!(d0, d1, "Both trees have the same leaf at pos {i}"); +/// } +/// +/// // The leafs A-B-C-D are the same for both trees, so are their 2 immediate parents +/// for i in 0..4 { +/// let d0 = store.get_path(ROOT0, NodeIndex::new(3, i)).unwrap(); +/// let d1 = store.get_path(ROOT1, NodeIndex::new(3, i)).unwrap(); +/// assert_eq!(d0.path[0..2], d1.path[0..2], "Both sub-trees are equal up to two levels"); +/// } +/// +/// // Common internal nodes are shared, the two added trees have a total of 30, but the store has +/// // only 10 new entries, corresponding to the 10 unique internal nodes of these trees. +/// assert_eq!(store.num_internal_nodes() - 64, 10); +/// ``` #[derive(Debug, Clone, Eq, PartialEq)] pub struct MerkleStore { nodes: BTreeMap, @@ -96,6 +146,11 @@ impl MerkleStore { // PUBLIC ACCESSORS // -------------------------------------------------------------------------------------------- + /// Return a count of the non-leaf nodes in the store. + pub fn num_internal_nodes(&self) -> usize { + self.nodes.len() + } + /// Returns the node at `index` rooted on the tree `root`. /// /// # Errors From 7957cc929aa8964099038d26da6701a1b2d08f3d Mon Sep 17 00:00:00 2001 From: "Augusto F. Hack" Date: Fri, 24 Mar 2023 19:36:51 +0100 Subject: [PATCH 22/26] feat: added MerkleStore serde --- src/merkle/store/mod.rs | 45 +++++++++++++++++++++++++++++++++++++++ src/merkle/store/tests.rs | 12 +++++++++++ 2 files changed, 57 insertions(+) diff --git a/src/merkle/store/mod.rs b/src/merkle/store/mod.rs index 99fb1ce..c98d1b3 100644 --- a/src/merkle/store/mod.rs +++ b/src/merkle/store/mod.rs @@ -2,6 +2,7 @@ use super::{ BTreeMap, BTreeSet, EmptySubtreeRoots, MerkleError, MerklePath, MerklePathSet, MerkleTree, NodeIndex, RootPath, Rpo256, RpoDigest, SimpleSmt, ValuePath, Vec, Word, }; +use crate::utils::{ByteReader, ByteWriter, Deserializable, DeserializationError, Serializable}; #[cfg(test)] mod tests; @@ -430,3 +431,47 @@ impl MerkleStore { } } } + +// SERIALIZATION +// ================================================================================================ + +impl Serializable for Node { + fn write_into(&self, target: &mut W) { + self.left.write_into(target); + self.right.write_into(target); + } +} + +impl Deserializable for Node { + fn read_from(source: &mut R) -> Result { + let left = RpoDigest::read_from(source)?; + let right = RpoDigest::read_from(source)?; + Ok(Node { left, right }) + } +} + +impl Serializable for MerkleStore { + fn write_into(&self, target: &mut W) { + target.write_u64(self.nodes.len() as u64); + + for (k, v) in self.nodes.iter() { + k.write_into(target); + v.write_into(target); + } + } +} + +impl Deserializable for MerkleStore { + fn read_from(source: &mut R) -> Result { + let len = source.read_u64()?; + let mut nodes: BTreeMap = BTreeMap::new(); + + for _ in 0..len { + let key = RpoDigest::read_from(source)?; + let value = Node::read_from(source)?; + nodes.insert(key, value); + } + + Ok(MerkleStore { nodes }) + } +} diff --git a/src/merkle/store/tests.rs b/src/merkle/store/tests.rs index e29f8e7..ecfedb4 100644 --- a/src/merkle/store/tests.rs +++ b/src/merkle/store/tests.rs @@ -5,6 +5,9 @@ use crate::{ Felt, Word, }; +#[cfg(std)] +use std::error::Error; + const KEYS4: [u64; 4] = [0, 1, 2, 3]; const LEAVES4: [Word; 4] = [ int_to_node(1), @@ -554,3 +557,12 @@ fn test_constructors() -> Result<(), MerkleError> { Ok(()) } + +#[cfg(std)] +#[test] +fn test_serialization() -> Result<(), Box> { + let original = MerkleStore::new().with_merkle_tree(LEAVES4)?; + let decoded = MerkleStore::read_from_bytes(&original.to_bytes())?; + assert_eq!(original, decoded); + Ok(()) +} From a58922756afe2aca9330a56a1f6df77baec07806 Mon Sep 17 00:00:00 2001 From: Bobbin Threadbare Date: Fri, 24 Mar 2023 14:58:19 -0700 Subject: [PATCH 23/26] chore: update crate versions, dependencies, and CHANGELOG --- CHANGELOG.md | 7 +++++++ Cargo.toml | 12 ++++++------ LICENSE | 2 +- README.md | 1 + src/hash/blake/mod.rs | 2 +- src/hash/rpo/mod.rs | 2 +- 6 files changed, 17 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 767f151..5d3fc0a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,10 @@ +## 0.2.0 (2023-03-24) + +- Implemented `MerkleStore` (#93, #94, #95, #107 #112). +- Added benchmarks for `MerkleStore` vs. other structs (#97). +- Added Merkle path containers (#99). +- Fixed depth handling in `MerklePathSet` (#110). + ## 0.1.4 (2023-02-22) - Re-export winter-crypto Hasher, Digest & ElementHasher (#72) diff --git a/Cargo.toml b/Cargo.toml index c5901e3..069e286 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,12 +1,12 @@ [package] name = "miden-crypto" -version = "0.1.4" +version = "0.2.0" description = "Miden Cryptographic primitives" authors = ["miden contributors"] readme = "README.md" license = "MIT" repository = "https://github.com/0xPolygonMiden/crypto" -documentation = "https://docs.rs/miden-crypto/0.1.4" +documentation = "https://docs.rs/miden-crypto/0.2.0" categories = ["cryptography", "no-std"] keywords = ["miden", "crypto", "hash", "merkle"] edition = "2021" @@ -30,11 +30,11 @@ std = ["blake3/std", "winter_crypto/std", "winter_math/std", "winter_utils/std"] [dependencies] blake3 = { version = "1.3", default-features = false } -winter_crypto = { version = "0.5", package = "winter-crypto", default-features = false } -winter_math = { version = "0.5", package = "winter-math", default-features = false } -winter_utils = { version = "0.5", package = "winter-utils", default-features = false } +winter_crypto = { version = "0.6", package = "winter-crypto", default-features = false } +winter_math = { version = "0.6", package = "winter-math", default-features = false } +winter_utils = { version = "0.6", package = "winter-utils", default-features = false } [dev-dependencies] criterion = { version = "0.4", features = ["html_reports"] } proptest = "1.1.0" -rand_utils = { version = "0.5", package = "winter-rand-utils" } +rand_utils = { version = "0.6", package = "winter-rand-utils" } diff --git a/LICENSE b/LICENSE index b29a140..c6fe12c 100644 --- a/LICENSE +++ b/LICENSE @@ -1,6 +1,6 @@ MIT License -Copyright (c) 2022 Polygon Miden +Copyright (c) 2023 Polygon Miden Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal diff --git a/README.md b/README.md index e01e1f2..f595325 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,7 @@ For performance benchmarks of these hash functions and their comparison to other * `MerkleTree`: a regular fully-balanced binary Merkle tree. The depth of this tree can be at most 64. * `SimpleSmt`: a Sparse Merkle Tree, mapping 63-bit keys to 4-element leaf values. * `MerklePathSet`: a collection of Merkle authentication paths all resolving to the same root. The length of the paths can be at most 64. +* `MerkleStore`: a collection of Merkle trees of different heights designed to efficiently store trees with common subtrees. The module also contains additional supporting components such as `NodeIndex`, `MerklePath`, and `MerkleError` to assist with tree indexation, opening proofs, and reporting inconsistent arguments/state. diff --git a/src/hash/blake/mod.rs b/src/hash/blake/mod.rs index 5255228..eb07ad6 100644 --- a/src/hash/blake/mod.rs +++ b/src/hash/blake/mod.rs @@ -290,7 +290,7 @@ where let digest = if Felt::IS_CANONICAL { blake3::hash(E::elements_as_bytes(elements)) } else { - let base_elements = E::as_base_elements(elements); + let base_elements = E::slice_as_base_elements(elements); let blen = base_elements.len() << 3; let mut bytes = unsafe { uninit_vector(blen) }; diff --git a/src/hash/rpo/mod.rs b/src/hash/rpo/mod.rs index 1cde967..95f2c97 100644 --- a/src/hash/rpo/mod.rs +++ b/src/hash/rpo/mod.rs @@ -212,7 +212,7 @@ impl ElementHasher for Rpo256 { fn hash_elements>(elements: &[E]) -> Self::Digest { // convert the elements into a list of base field elements - let elements = E::as_base_elements(elements); + let elements = E::slice_as_base_elements(elements); // initialize state to all zeros, except for the first element of the capacity part, which // is set to 1 if the number of elements is not a multiple of RATE_WIDTH. From 9e6c8ff70030336940ac551988b36e3200d19941 Mon Sep 17 00:00:00 2001 From: "Augusto F. Hack" Date: Fri, 24 Mar 2023 20:22:55 +0100 Subject: [PATCH 24/26] bugfix: fix internal nodes of for empty leafs of a SMT The path returned by `EmptySubtreeRoots` starts at the root, and goes to the leaf. The MerkleStore constructor assumed the other direction, so the parent/child hashes were reversed. This fixes the bug and adds a test. --- src/merkle/store/mod.rs | 9 +++-- src/merkle/store/tests.rs | 81 ++++++++++++++++++++++++++++++++++++--- 2 files changed, 80 insertions(+), 10 deletions(-) diff --git a/src/merkle/store/mod.rs b/src/merkle/store/mod.rs index 99fb1ce..ffd7702 100644 --- a/src/merkle/store/mod.rs +++ b/src/merkle/store/mod.rs @@ -43,7 +43,7 @@ pub struct Node { /// let mut store = MerkleStore::new(); /// /// // the store is initialized with the SMT empty nodes -/// assert_eq!(store.num_internal_nodes(), 64); +/// assert_eq!(store.num_internal_nodes(), 255); /// /// // populates the store with two merkle trees, common nodes are shared /// store.add_merkle_tree([A, B, C, D, E, F, G, H0]); @@ -65,7 +65,7 @@ pub struct Node { /// /// // Common internal nodes are shared, the two added trees have a total of 30, but the store has /// // only 10 new entries, corresponding to the 10 unique internal nodes of these trees. -/// assert_eq!(store.num_internal_nodes() - 64, 10); +/// assert_eq!(store.num_internal_nodes() - 255, 10); /// ``` #[derive(Debug, Clone, Eq, PartialEq)] pub struct MerkleStore { @@ -85,11 +85,12 @@ impl MerkleStore { /// Creates an empty `MerkleStore` instance. pub fn new() -> MerkleStore { // pre-populate the store with the empty hashes - let subtrees = EmptySubtreeRoots::empty_hashes(64); + let subtrees = EmptySubtreeRoots::empty_hashes(255); let nodes = subtrees .iter() + .rev() .copied() - .zip(subtrees.iter().skip(1).copied()) + .zip(subtrees.iter().rev().skip(1).copied()) .map(|(child, parent)| { ( parent, diff --git a/src/merkle/store/tests.rs b/src/merkle/store/tests.rs index e29f8e7..52984dc 100644 --- a/src/merkle/store/tests.rs +++ b/src/merkle/store/tests.rs @@ -2,7 +2,7 @@ use super::*; use crate::{ hash::rpo::Rpo256, merkle::{int_to_node, MerklePathSet}, - Felt, Word, + Felt, Word, WORD_SIZE, ZERO, }; const KEYS4: [u64; 4] = [0, 1, 2, 3]; @@ -12,6 +12,7 @@ const LEAVES4: [Word; 4] = [ int_to_node(3), int_to_node(4), ]; +const EMPTY: Word = [ZERO; WORD_SIZE]; #[test] fn test_root_not_in_store() -> Result<(), MerkleError> { @@ -141,6 +142,51 @@ fn test_merkle_tree() -> Result<(), MerkleError> { Ok(()) } +#[test] +fn test_empty_roots() { + let store = MerkleStore::default(); + let mut root = RpoDigest::new(EMPTY); + + for depth in 0..255 { + root = Rpo256::merge(&[root; 2]); + assert!( + store.get_node(root.into(), NodeIndex::new(0, 0)).is_ok(), + "The root of the empty tree of depth {depth} must be registered" + ); + } +} + +#[test] +fn test_leaf_paths_for_empty_trees() -> Result<(), MerkleError> { + let store = MerkleStore::default(); + + // Starts at 1 because leafs are not included in the store. + // Ends at 64 because it is not possible to represent an index of a depth greater than 64, + // because a u64 is used to index the leaf. + for depth in 1..64 { + let smt = SimpleSmt::new(depth)?; + + let index = NodeIndex::new(depth, 0); + let store_path = store.get_path(smt.root(), index)?; + let smt_path = smt.get_path(index)?; + assert_eq!( + store_path.value, EMPTY, + "the leaf of an empty tree is always ZERO" + ); + assert_eq!( + store_path.path, smt_path, + "the returned merkle path does not match the computed values" + ); + assert_eq!( + store_path.path.compute_root(depth.into(), EMPTY), + smt.root(), + "computed root from the path must match the empty tree root" + ); + } + + Ok(()) +} + #[test] fn test_get_invalid_node() { let mut store = MerkleStore::default(); @@ -211,6 +257,11 @@ fn test_sparse_merkle_tree() -> Result<(), MerkleError> { Ok(LEAVES4[3]), "node 3 must be in the tree" ); + assert_eq!( + store.get_node(smt.root(), NodeIndex::new(smt.depth(), 4)), + Ok(EMPTY), + "unmodified node 4 must be ZERO" + ); // STORE LEAVES MATCH TREE =============================================================== // sanity check the values returned by the store and the tree @@ -234,6 +285,11 @@ fn test_sparse_merkle_tree() -> Result<(), MerkleError> { store.get_node(smt.root(), NodeIndex::new(smt.depth(), 3)), "node 3 must be the same for both SparseMerkleTree and MerkleStore" ); + assert_eq!( + smt.get_node(&NodeIndex::new(smt.depth(), 4)), + store.get_node(smt.root(), NodeIndex::new(smt.depth(), 4)), + "node 4 must be the same for both SparseMerkleTree and MerkleStore" + ); // STORE MERKLE PATH MATCHS ============================================================== // assert the merkle path returned by the store is the same as the one in the tree @@ -255,7 +311,7 @@ fn test_sparse_merkle_tree() -> Result<(), MerkleError> { .unwrap(); assert_eq!( LEAVES4[1], result.value, - "Value for merkle path at index 0 must match leaf value" + "Value for merkle path at index 1 must match leaf value" ); assert_eq!( smt.get_path(NodeIndex::new(smt.depth(), 1)), @@ -268,12 +324,12 @@ fn test_sparse_merkle_tree() -> Result<(), MerkleError> { .unwrap(); assert_eq!( LEAVES4[2], result.value, - "Value for merkle path at index 0 must match leaf value" + "Value for merkle path at index 2 must match leaf value" ); assert_eq!( smt.get_path(NodeIndex::new(smt.depth(), 2)), Ok(result.path), - "merkle path for index 0 must be the same for the MerkleTree and MerkleStore" + "merkle path for index 2 must be the same for the MerkleTree and MerkleStore" ); let result = store @@ -281,12 +337,25 @@ fn test_sparse_merkle_tree() -> Result<(), MerkleError> { .unwrap(); assert_eq!( LEAVES4[3], result.value, - "Value for merkle path at index 0 must match leaf value" + "Value for merkle path at index 3 must match leaf value" ); assert_eq!( smt.get_path(NodeIndex::new(smt.depth(), 3)), Ok(result.path), - "merkle path for index 0 must be the same for the MerkleTree and MerkleStore" + "merkle path for index 3 must be the same for the MerkleTree and MerkleStore" + ); + + let result = store + .get_path(smt.root(), NodeIndex::new(smt.depth(), 4)) + .unwrap(); + assert_eq!( + EMPTY, result.value, + "Value for merkle path at index 4 must match leaf value" + ); + assert_eq!( + smt.get_path(NodeIndex::new(smt.depth(), 4)), + Ok(result.path), + "merkle path for index 4 must be the same for the MerkleTree and MerkleStore" ); Ok(()) From d68be83bc474874cb8a36c4397a259b5f33b4d99 Mon Sep 17 00:00:00 2001 From: Bobbin Threadbare Date: Sat, 25 Mar 2023 00:00:24 -0700 Subject: [PATCH 25/26] chore: add Mmr to readme and changelog --- CHANGELOG.md | 1 + README.md | 1 + 2 files changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5d3fc0a..4d8f28c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,6 @@ ## 0.2.0 (2023-03-24) +- Implemented `Mmr` and related structs (#67) - Implemented `MerkleStore` (#93, #94, #95, #107 #112). - Added benchmarks for `MerkleStore` vs. other structs (#97). - Added Merkle path containers (#99). diff --git a/README.md b/README.md index f595325..365c7c7 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,7 @@ For performance benchmarks of these hash functions and their comparison to other * `SimpleSmt`: a Sparse Merkle Tree, mapping 63-bit keys to 4-element leaf values. * `MerklePathSet`: a collection of Merkle authentication paths all resolving to the same root. The length of the paths can be at most 64. * `MerkleStore`: a collection of Merkle trees of different heights designed to efficiently store trees with common subtrees. +* `Mmr`: a Merkle mountain range structure designed to function as an append-only log. The module also contains additional supporting components such as `NodeIndex`, `MerklePath`, and `MerkleError` to assist with tree indexation, opening proofs, and reporting inconsistent arguments/state. From 703692553d0c0363843115a7204efc33617201c0 Mon Sep 17 00:00:00 2001 From: Bobbin Threadbare Date: Sat, 25 Mar 2023 00:45:17 -0700 Subject: [PATCH 26/26] chore: add winterfell dependency update to changelog --- CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4d8f28c..aa67da3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,10 +1,11 @@ ## 0.2.0 (2023-03-24) -- Implemented `Mmr` and related structs (#67) +- Implemented `Mmr` and related structs (#67). - Implemented `MerkleStore` (#93, #94, #95, #107 #112). - Added benchmarks for `MerkleStore` vs. other structs (#97). - Added Merkle path containers (#99). - Fixed depth handling in `MerklePathSet` (#110). +- Updated Winterfell dependency to v0.6. ## 0.1.4 (2023-02-22)