You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The existing incremental Merkle tree, even for the non-legacy version, is designed in a way that unnecessarily stores extra digests and does extra hashes. This version is consistent with calculate_merkle (which is the non-legacy version).
This behavior is also how the legacy version works except it also adds the complexity of modifying the MSB of the digests to indicate left and right pairs.
We do not want to touch the legacy versions at all for backwards compatibility. It would be good to rename the types to include legacy in them and also rename the file names to indicate legacy as well.
But the non-legacy versions need to be replaced to use a more efficient incremental Merkle tree structure that does not have the additional digests/hashes. This should be stored in a separate file to keep it clearly separated from the legacy versions.
Information on how the new efficient incremental Merkle tree should work:
The incremental Merkle tree stores at a minimum (std::popcount(digests.size()) + (std::has_single_bit(digests.size()) ? 0 : 1)) number of hashes.
And when you append a new digest to the incremental Merkle tree, that increments digests.size() which indicates through the the change in 1s and 0s of digests.size() how to aggregate the existing hashes together (and remove them from the structure) to form the new hash you need to store in the structure.
So when digests.size() == 3, you have 0b0011 which means you have three hashes:
The root (included since std::has_single_bit(digests.size()) == false) is hash(hash(digests[0], digests[1]), digests[2]).
The one represented by 0b0010 is hash(digests[0], digests[1]).
The one represented by 0b0001 is digests[2].
When you add a new digest and get digests.size() == 4, you have 0b0100, which means the two digests represented by 0b0010 and 0b0001 are replaced by a single digest:
The one represented by 0b0100 is hash(hash(digests[0],digests[1]), hash(digests[2],digests[3])).
Note that you can also store the root explicitly, but in this case, since std::has_single_bit(digests.size()) == true, it would be redundant.
The implementation is free to always explicitly store a root (even if redundant) or only store the minimally needed number of digests, whichever is easier to implement.
The implementation should try to use C++17 since we will need a version of this implementation in CDT as well.
This new incremental Merkle tree should be used for the Validation Trees.
There should also be a non-incremental version of calculating the Merkle tree as well that is consistent with this new more efficient way of computing the root. There should be tests to ensure they are consistent with each other.
This non-incremental version of calculating the Merkle root should be used to calculate the the roots used for action_mroot and transaction_mroot of Proper IF Blocks only. The incremental Merkle tree could be used as well, but since we already have the list of digests used to calculate the root, it is faster to use the non-incremental version on a separate thread (and potentially in parallel).
The text was updated successfully, but these errors were encountered:
The existing incremental Merkle tree, even for the non-legacy version, is designed in a way that unnecessarily stores extra digests and does extra hashes. This version is consistent with
calculate_merkle
(which is the non-legacy version).This behavior is also how the legacy version works except it also adds the complexity of modifying the MSB of the digests to indicate left and right pairs.
We do not want to touch the legacy versions at all for backwards compatibility. It would be good to rename the types to include legacy in them and also rename the file names to indicate legacy as well.
But the non-legacy versions need to be replaced to use a more efficient incremental Merkle tree structure that does not have the additional digests/hashes. This should be stored in a separate file to keep it clearly separated from the legacy versions.
Information on how the new efficient incremental Merkle tree should work:
The incremental Merkle tree stores at a minimum
(std::popcount(digests.size()) + (std::has_single_bit(digests.size()) ? 0 : 1))
number of hashes.And when you append a new digest to the incremental Merkle tree, that increments
digests.size()
which indicates through the the change in 1s and 0s ofdigests.size()
how to aggregate the existing hashes together (and remove them from the structure) to form the new hash you need to store in the structure.So when
digests.size() == 3
, you have0b0011
which means you have three hashes:std::has_single_bit(digests.size()) == false
) ishash(hash(digests[0], digests[1]), digests[2])
.0b0010
ishash(digests[0], digests[1])
.0b0001
isdigests[2]
.When you add a new digest and get
digests.size() == 4
, you have0b0100
, which means the two digests represented by0b0010
and0b0001
are replaced by a single digest:0b0100
ishash(hash(digests[0],digests[1]), hash(digests[2],digests[3]))
.Note that you can also store the root explicitly, but in this case, since
std::has_single_bit(digests.size()) == true
, it would be redundant.The implementation is free to always explicitly store a root (even if redundant) or only store the minimally needed number of digests, whichever is easier to implement.
The implementation should try to use C++17 since we will need a version of this implementation in CDT as well.
This new incremental Merkle tree should be used for the Validation Trees.
There should also be a non-incremental version of calculating the Merkle tree as well that is consistent with this new more efficient way of computing the root. There should be tests to ensure they are consistent with each other.
This non-incremental version of calculating the Merkle root should be used to calculate the the roots used for
action_mroot
andtransaction_mroot
of Proper IF Blocks only. The incremental Merkle tree could be used as well, but since we already have the list of digests used to calculate the root, it is faster to use the non-incremental version on a separate thread (and potentially in parallel).The text was updated successfully, but these errors were encountered: