Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IF: Create new efficient incremental Merkle tree #2333

Closed
arhag opened this issue Mar 22, 2024 · 0 comments · Fixed by #2361
Closed

IF: Create new efficient incremental Merkle tree #2333

arhag opened this issue Mar 22, 2024 · 0 comments · Fixed by #2361
Assignees

Comments

@arhag
Copy link
Member

arhag commented Mar 22, 2024

The existing incremental Merkle tree, even for the non-legacy version, is designed in a way that unnecessarily stores extra digests and does extra hashes. This version is consistent with calculate_merkle (which is the non-legacy version).

This behavior is also how the legacy version works except it also adds the complexity of modifying the MSB of the digests to indicate left and right pairs.

We do not want to touch the legacy versions at all for backwards compatibility. It would be good to rename the types to include legacy in them and also rename the file names to indicate legacy as well.

But the non-legacy versions need to be replaced to use a more efficient incremental Merkle tree structure that does not have the additional digests/hashes. This should be stored in a separate file to keep it clearly separated from the legacy versions.

Information on how the new efficient incremental Merkle tree should work:

The incremental Merkle tree stores at a minimum (std::popcount(digests.size()) + (std::has_single_bit(digests.size()) ? 0 : 1)) number of hashes.

And when you append a new digest to the incremental Merkle tree, that increments digests.size() which indicates through the the change in 1s and 0s of digests.size() how to aggregate the existing hashes together (and remove them from the structure) to form the new hash you need to store in the structure.

So when digests.size() == 3, you have 0b0011 which means you have three hashes:

  1. The root (included since std::has_single_bit(digests.size()) == false) is hash(hash(digests[0], digests[1]), digests[2]).
  2. The one represented by 0b0010 is hash(digests[0], digests[1]).
  3. The one represented by 0b0001 is digests[2].

When you add a new digest and get digests.size() == 4, you have 0b0100, which means the two digests represented by 0b0010 and 0b0001 are replaced by a single digest:

  1. The one represented by 0b0100 is hash(hash(digests[0],digests[1]), hash(digests[2],digests[3])).

Note that you can also store the root explicitly, but in this case, since std::has_single_bit(digests.size()) == true, it would be redundant.

The implementation is free to always explicitly store a root (even if redundant) or only store the minimally needed number of digests, whichever is easier to implement.

The implementation should try to use C++17 since we will need a version of this implementation in CDT as well.

This new incremental Merkle tree should be used for the Validation Trees.

There should also be a non-incremental version of calculating the Merkle tree as well that is consistent with this new more efficient way of computing the root. There should be tests to ensure they are consistent with each other.

This non-incremental version of calculating the Merkle root should be used to calculate the the roots used for action_mroot and transaction_mroot of Proper IF Blocks only. The incremental Merkle tree could be used as well, but since we already have the list of digests used to calculate the root, it is faster to use the non-incremental version on a separate thread (and potentially in parallel).

@enf-ci-bot enf-ci-bot moved this to Todo in Team Backlog Mar 22, 2024
@arhag arhag removed the triage label Mar 22, 2024
@greg7mdp greg7mdp moved this from Todo to In Progress in Team Backlog Mar 25, 2024
@greg7mdp greg7mdp moved this from In Progress to Awaiting Review in Team Backlog Mar 29, 2024
@greg7mdp greg7mdp linked a pull request Mar 29, 2024 that will close this issue
@greg7mdp greg7mdp moved this from Awaiting Review to Done in Team Backlog Apr 2, 2024
@greg7mdp greg7mdp closed this as completed Apr 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants