-
Notifications
You must be signed in to change notification settings - Fork 86
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test(benchmarks): improve bechmarks to have complete performance data #326
Merged
sripwoud
merged 17 commits into
privacy-scaling-explorations:main
from
0xarcano:docs/merkel-tree-bechmarks
Sep 25, 2024
Merged
Changes from all commits
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
e589099
test(benchmarks): improve bechmarks to have complete performance data
0xarcano 81093d8
fix(benchmarks): remove package winston and to-hex libraries to fix v…
0xarcano 1c1e108
docs(benchmarks): added benchmarks summary
0xarcano 83deb40
Update benchmarks/README.md
0xarcano 5e76b88
Update benchmarks/benchmark-merkle-trees.ts
0xarcano 50054f3
Update benchmarks/README.md
0xarcano f42c8bd
Update benchmarks/benchmark-merkle-trees.ts
0xarcano 8b761ea
Update benchmarks/README.md
0xarcano 93efa6b
refactor(benchmarks): added winston to package.json and refactored in…
0xarcano 3fa14a7
Update benchmarks/benchmark-merkle-trees.ts
0xarcano 1585210
fix(benchmarks): added counter initialization mistake
0xarcano e4989dc
Merge remote-tracking branch 'refs/remotes/origin/docs/merkel-tree-be…
0xarcano 6af06df
Merge branch 'main' into docs/merkel-tree-bechmarks
cedoor f981010
docs(benchmarks): changed the usage instructions to point the exisiti…
0xarcano 6bd356e
Merge remote-tracking branch 'refs/remotes/origin/docs/merkel-tree-be…
0xarcano f761603
style(benchmarks): benchmark-merkle-trees.ts changes from prettier va…
0xarcano 7710579
fix(benchmarks): added the new version of yarn.lock
0xarcano File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,99 @@ | ||
# ZK-Kit: Merkle Tree Benchmarks | ||
|
||
This project is part of the `zk-kit` repository and includes benchmarking tests for three Merkle Tree implementations: | ||
|
||
- **Incremental Merkle Tree (IMT)** from `@zk-kit/imt` | ||
- **Lean Incremental Merkle Tree (LeanIMT)** from `@zk-kit/lean-imt` | ||
- **Sparse Merkle Tree (SMT)** from `@zk-kit/smt` | ||
|
||
The benchmark compares the performance of each Merkle Tree implementation for operations like adding leaves, generating proofs, and verifying proofs. | ||
|
||
## Table of Contents | ||
|
||
- [Installation](#installation) | ||
- [Usage](#usage) | ||
- [Benchmark](#benchmark) | ||
- [Results](#results) | ||
- [Dependencies](#dependencies) | ||
|
||
--- | ||
|
||
## Installation | ||
|
||
Before running the benchmark, ensure that you have Node.js version 20 installed and the required dependencies set up. | ||
|
||
1. **Clone the Repository**: | ||
```bash | ||
git clone https://github.com/privacy-scaling-explorations/zk-kit.git | ||
cd zk-kit | ||
yarn | ||
``` | ||
|
||
## Usage | ||
|
||
To run the benchmark, use the following script: | ||
|
||
```bash | ||
yarn run benchmarks | ||
``` | ||
|
||
## Benchmark | ||
|
||
The benchmark includes the following tests for each Merkle Tree implementation: | ||
|
||
1. Add Leaves: Measures the time it takes to insert a number of leaves into the tree. | ||
2. Generate Proofs: Measures the performance of generating Merkle proofs. | ||
3. Verify Proofs: Measures how quickly the generated proofs can be verified. | ||
4. Update Leaves: Measures how quickly the leaves can be updated. | ||
5. Delete Leaves: Measures how quickly the leaves can be deleted. The Lean Merkle tree is excluded because it does not have deleted method implemented. | ||
|
||
## Benchmark Structure. | ||
|
||
- The benchmarks are defined using the [benny](https://caderek.github.io/benny/) library. | ||
- Results are saved to .html files in the benchmarks/results directory for visualization and further analysis. | ||
|
||
## Results | ||
|
||
The results for the benchmarking process will be saved in the following formats: | ||
|
||
Charts: Performance charts for each operation are saved as .html files. | ||
Tables: Detailed benchmark results in tabular form are also saved. | ||
Results can be found in the benchmarks/results directory after running the benchmark. | ||
|
||
## Example of Benchmark Execution | ||
|
||
Here’s a typical output from running the benchmark: | ||
|
||
```bash | ||
|
||
Suite: add-merkle-trees-1000 | ||
IMT - Add 1000 leaves x 12,345 ops/sec ±2.31% (92 runs sampled) | ||
LeanIMT - Add 1000 leaves x 9,876 ops/sec ±1.87% (87 runs sampled) | ||
SparseMT - Add 1000 leaves x 7,654 ops/sec ±3.14% (75 runs sampled) | ||
Fastest is IMT - Add 1000 leaves | ||
|
||
Suite: proof-generation-merkle-trees-1000 | ||
IMT - Generated 500 proofs x 4,321 ops/sec ±2.02% (80 runs sampled) | ||
LeanIMT - Generated 500 proofs x 3,456 ops/sec ±1.75% (78 runs sampled) | ||
SparseMT - Generated 500 proofs x 2,789 ops/sec ±2.63% (68 runs sampled) | ||
Fastest is IMT - Generated 500 proofs | ||
``` | ||
|
||
The Benchmarks suggested in the index.ts are for **8, 128 and 1024 leafs** to see how each Merkle tree for different sizes of trees | ||
perform because their theorical expected behavior described [here](https://github.com/privacy-scaling-explorations/zk-kit?tab=readme-ov-file#i-need-to-use-a-merkle-tree-to-prove-the-inclusion-or-exclusion-of-data-elements-within-a-set-which-type-of-merkle-tree-should-i-use). | ||
|
||
## Dependencies | ||
|
||
This benchmark depends on the following packages: | ||
|
||
- @zk-kit/imt: Incremental Merkle Tree implementation. | ||
- @zk-kit/lean-imt: Lean Incremental Merkle Tree implementation. | ||
- @zk-kit/smt: Sparse Merkle Tree implementation. | ||
- benny: A benchmark library for testing performance. | ||
- ts-node: TypeScript execution environment for Node.js. | ||
- winston: Logger used for capturing errors and logs during execution. | ||
|
||
## Notes | ||
|
||
- Node.js Version: The benchmarks are tested and run with Node.js version 20. Ensure you have the correct version installed. | ||
- Logging: Logs, including any errors that occur during benchmarking, are captured using the winston logger and saved to error.log in the root directory. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit:
nice but unnecessary.
GitHub web ui automatically adds a button to see the table of contents for all
.md
files 😉https://github.blog/changelog/2021-04-13-table-of-contents-support-in-markdown-files/