From d2ee904e119500a93eaafb5581e585a63cdcbd0d Mon Sep 17 00:00:00 2001 From: DL Date: Fri, 11 Oct 2024 12:18:11 +0800 Subject: [PATCH 01/11] Update spec.md --- ethstorage/archiver/spec.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/ethstorage/archiver/spec.md b/ethstorage/archiver/spec.md index 54c831ee..cb49456d 100644 --- a/ethstorage/archiver/spec.md +++ b/ethstorage/archiver/spec.md @@ -149,12 +149,17 @@ When the API is queried, the content of blobs is loaded to generate `kzg_commitm One thing worth noting is how the blob's index is determined. We need to use the order of blob kzg commitments obtained in the beacon API `/eth/v2/beacon/blocks/{block_id}`, and use the corresponding relationship between `versionedash` and kzg commitment to obtain the index of the blob in the block. -### Considerations for Solution 2: +### Considerations for Solution 2: - It is assumed that the beacon API `/eth/v2/beacon/blocks/{block_id}` is always available for any blocks, and the event log data from the execution layer is always available for any blocks. As EIP-4444 comes and execution payload data may not be available, we may need a key-value db table to save the relationship between the beacon block and kv index such as: - ``` +``` BeaconRootHash->{"kv_index":[0,1,2]} - ``` +``` - If the performance is proven to be a problem, we may need to add a local cache/prefetch layer to save the metadata. - If the blob transaction hashes are required, they can also be retrieved through standard JSON-RPC API. - + +# Conclusion + +Considering the complexity of Solution 1, which requires modifications to multiple components of the es-node, including the downloader and sync processes, we have opted for Solution 2 as the final implementation. + + From 6fc3963c33cec25532a2b33932c0c3a690f57b9b Mon Sep 17 00:00:00 2001 From: syntrust Date: Fri, 11 Oct 2024 15:00:48 +0800 Subject: [PATCH 02/11] minor --- ethstorage/archiver/spec.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/ethstorage/archiver/spec.md b/ethstorage/archiver/spec.md index cb49456d..672c3a22 100644 --- a/ethstorage/archiver/spec.md +++ b/ethstorage/archiver/spec.md @@ -152,14 +152,16 @@ One thing worth noting is how the blob's index is determined. We need to use the ### Considerations for Solution 2: - It is assumed that the beacon API `/eth/v2/beacon/blocks/{block_id}` is always available for any blocks, and the event log data from the execution layer is always available for any blocks. As EIP-4444 comes and execution payload data may not be available, we may need a key-value db table to save the relationship between the beacon block and kv index such as: + ``` BeaconRootHash->{"kv_index":[0,1,2]} ``` + - If the performance is proven to be a problem, we may need to add a local cache/prefetch layer to save the metadata. - If the blob transaction hashes are required, they can also be retrieved through standard JSON-RPC API. # Conclusion -Considering the complexity of Solution 1, which requires modifications to multiple components of the es-node, including the downloader and sync processes, we have opted for Solution 2 as the final implementation. +Given the complexity of Solution 1, which requires modifications to some key components of es-node, including the downloader and sync processes, Solution 2 is simpler and more straightforward, making it the preferred choice for implementation. From dda168919b06d17ae50a5d1f1eab0cde94952e4e Mon Sep 17 00:00:00 2001 From: syntrust Date: Tue, 15 Oct 2024 11:06:02 +0800 Subject: [PATCH 03/11] test --- ethstorage/archiver/test.md | 76 +++++++++++++++++++++++++++++++++++++ run-rpc.sh | 3 +- 2 files changed, 78 insertions(+), 1 deletion(-) create mode 100644 ethstorage/archiver/test.md diff --git a/ethstorage/archiver/test.md b/ethstorage/archiver/test.md new file mode 100644 index 00000000..3e7cb6b3 --- /dev/null +++ b/ethstorage/archiver/test.md @@ -0,0 +1,76 @@ +# EthStorage Archive Service Testing Document + +## Introduction + +This document outlines the testing strategy and details for the blob archiver service of EthStorage. It aims to verify that the functionalities meet the specified requirements and is free of critical defects. + +## Test Scope + +The scope of testing includes: +- Functional Testing: make sure expired blobs can be retreived from EthStorage correctly. +- Integration Testing: make sure OP Stack derivation works correclty with the retrieved blobs from EthStorage. + +## Environment Setup + +The tests will be conducted in an environment with the following services up and running: +- An Ethereum L1 RPC of an execution client running in archive mode +- An Ethereum L1 Beacon URL +- An EthStorage node (es-node) with the archive service enabled +- A Proxy of Ethereum Beacon RPC to mock a short expiration time +- An OP node that uses EthStorage archive service + +### Running an EthStorage node with the archive service enabled + +The following steps download source code of es-node, build, init, and run an es-node as a RPC node with archive service enabled. + +``` +git clone https://github.com/ethstorage/es-node.git + +cd es-node + +make + +./init-rpc.sh + +./run-rpc.sh --archiver.enabled +``` + +Noted that an Ethereum EL RPC in archive mode and an Ethereum Beacon URL are used implicitly in the shell scripts in below command. + +### Running a Proxy of Ethereum Beacon URL + +``` +git clone https://github.com/ethstorage/beacon-api-wrapper.git + +cd beacon-api-wrapper + +go run cmd/main.go +``` + +For details and more options please refer to https://github.com/ethstorage/beacon-api-wrapper#beacon-api-wrapper. + +### Running an OP Node + +Following the [OP documentation](https://docs.optimism.io/builders/node-operators/tutorials/testnet) to run an op-node from source, and use the following options to make sure that +- the L1 Beacon URL points to the Proxy, and +- the archive service points to the es-node RPC. + +For example, + +``` +--l1.beacon http://65.108.236.27:3600 --l1.beacon-archiver http://65.108.236.27:9645 +``` + +## Test Cases + +### Test Case 1: Load an expired blob from EthStorage +- **Objective**: Verify that blobs over 4096 epochs age that is expired by the Beacon Chain can be retrieved from EthStorage archive service. +- **Preconditions**: +- **Inputs**: +- **Expected Result**: + +### Test Case 2: + + +## Conclusion +This document serves as the foundation for the software testing framework and process of the blob archiver service. It provides a clear structure for the testing activities to ensure quality outcomes. diff --git a/run-rpc.sh b/run-rpc.sh index 7deb74f2..712ff942 100755 --- a/run-rpc.sh +++ b/run-rpc.sh @@ -2,4 +2,5 @@ ./run.sh \ --rpc.addr 0.0.0.0 \ - --miner.enabled=false \ No newline at end of file + --miner.enabled=false \ + $@ \ No newline at end of file From 0b2a1b3989e31bb7b2aca62695f02108c48d35d7 Mon Sep 17 00:00:00 2001 From: syntrust Date: Tue, 15 Oct 2024 18:16:35 +0800 Subject: [PATCH 04/11] test case --- ethstorage/archiver/test.md | 46 ++++++++++++++++++++++++++++++++----- 1 file changed, 40 insertions(+), 6 deletions(-) diff --git a/ethstorage/archiver/test.md b/ethstorage/archiver/test.md index 3e7cb6b3..472ddc6b 100644 --- a/ethstorage/archiver/test.md +++ b/ethstorage/archiver/test.md @@ -16,7 +16,7 @@ The tests will be conducted in an environment with the following services up and - An Ethereum L1 RPC of an execution client running in archive mode - An Ethereum L1 Beacon URL - An EthStorage node (es-node) with the archive service enabled -- A Proxy of Ethereum Beacon RPC to mock a short expiration time +- A Proxy of Ethereum Beacon RPC to mock a short blob retention period - An OP node that uses EthStorage archive service ### Running an EthStorage node with the archive service enabled @@ -37,6 +37,9 @@ make Noted that an Ethereum EL RPC in archive mode and an Ethereum Beacon URL are used implicitly in the shell scripts in below command. + +For details and more options to run an es-node, please refer to https://docs.ethstorage.io/storage-provider-guide/tutorials. + ### Running a Proxy of Ethereum Beacon URL ``` @@ -47,6 +50,8 @@ cd beacon-api-wrapper go run cmd/main.go ``` +The Proxy works just like a normal Beacon API except the blobs retension period is much shorter, like 3 hours by default. This means when a `blob_sidecars` request (`/eth/v1/beacon/blob_sidecars/{block_id}`) asking for blobs of more than 900 slot ago, it will return an empty list: `{"data":[]}`. + For details and more options please refer to https://github.com/ethstorage/beacon-api-wrapper#beacon-api-wrapper. ### Running an OP Node @@ -56,7 +61,6 @@ Following the [OP documentation](https://docs.optimism.io/builders/node-operator - the archive service points to the es-node RPC. For example, - ``` --l1.beacon http://65.108.236.27:3600 --l1.beacon-archiver http://65.108.236.27:9645 ``` @@ -64,13 +68,43 @@ For example, ## Test Cases ### Test Case 1: Load an expired blob from EthStorage -- **Objective**: Verify that blobs over 4096 epochs age that is expired by the Beacon Chain can be retrieved from EthStorage archive service. -- **Preconditions**: -- **Inputs**: -- **Expected Result**: +- **Objective**: Verify that blobs that is expired by the Beacon Chain can be retrieved from EthStorage archive service. +- **Steps**: + 1. Upload a blob aimed to be archived by EthStorage. (You may need [eth-blob-uploader](#eth-blob-uploader) for this.) + 2. Record the slot as block_id of the blob for later use. + 3. Query `/eth/v1/beacon/blob_sidecars/{block_id}` from the Beacon Chain URL. For example, + ``` + curl -X 'GET' 'http://88.99.30.186:3500/eth/v1/beacon/blob_sidecars/6103987' -H 'accept: application/json' + + ``` + 4. Wait for the blob expires and the above query returns `{"data":[]}`. + 5. Query `/eth/v1/beacon/blob_sidecars/{block_id}` from the EthStorage archive service. For example, + ``` + curl -X 'GET' 'http://65.108.236.27:9645/eth/v1/beacon/blob_sidecars/6103987' -H 'accept: application/json' + ``` +- **Expected Result**: The result of step 3 and 5 are the same. ### Test Case 2: +## Tools + +### eth-blob-uploader + +`eth-blob-uploader` is a handy tool to upload EIP-4844 blobs to Ethereum. + +The following command demostrates how to use `eth-blob-uploader` to upload a local file `hello.txt` to EthStorage testnet as a blob by calling the function `putBlob(bytes32 key,uint256 blobIdx,uint256 length)` of the storage contract deployed on Sepolia. +``` +eth-blob-uploader -r http://88.99.30.186:8545 \ +-p \ +-f hello.txt \ +-t 0x804C520d3c084C805E37A35E90057Ac32831F96f \ +-v 1500000000000000 \ +-d 0x4581a9201c8aff950685c2ed4bc3174f3472287b56d9517b9c948127319a09a7a36deac800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000 +``` +Record the transaction hash so that you can find on [EtherScan](https://sepolia.etherscan.io/) which slot the blob is belong to. + +Refer to https://github.com/ethstorage/eth-blob-uploader for more details. + ## Conclusion This document serves as the foundation for the software testing framework and process of the blob archiver service. It provides a clear structure for the testing activities to ensure quality outcomes. From 1a660c4a9bea4953b9c5ff3d402f2e12e1e48e53 Mon Sep 17 00:00:00 2001 From: syntrust Date: Tue, 15 Oct 2024 18:16:49 +0800 Subject: [PATCH 05/11] update cases --- integration_tests/archiver_test.go | 27 +++++++-------------------- 1 file changed, 7 insertions(+), 20 deletions(-) diff --git a/integration_tests/archiver_test.go b/integration_tests/archiver_test.go index c3fffee1..f4d8e3f2 100644 --- a/integration_tests/archiver_test.go +++ b/integration_tests/archiver_test.go @@ -10,7 +10,6 @@ import ( "fmt" "io" "net/http" - "strconv" "strings" "testing" @@ -30,35 +29,23 @@ type test struct { msg string } -func (t *test) toUrl() string { - if len(t.archivedIndices) == 0 { - return fmt.Sprintf(urlPattern, archiverAddr, t.query) - } - var strArr []string - for _, val := range t.archivedIndices { - strArr = append(strArr, strconv.FormatUint(uint64(val), 10)) - } - query := fmt.Sprintf("%s?indices=%s", t.query, strings.Join(strArr, ",")) - return fmt.Sprintf(urlPattern, archiverAddr, query) -} - func TestArchiveAPI(t *testing.T) { tests := []test{ { - query: "4756895", - archivedIndices: []uint64{3}, + query: "6082444", // within 4096 epoch, can be retrieved from both beacon and archiver + archivedIndices: []uint64{0, 1}, }, { - query: "4756895?indices=3", - archivedIndices: []uint64{3}, + query: "6082444?indices=0,1", // specified all the indices + archivedIndices: []uint64{0, 1}, }, { - query: "4756895?indices=0,3", - archivedIndices: []uint64{3}, + query: "6082444?indices=1", // specified one of the indices + archivedIndices: []uint64{1}, }, { - query: "4756895?indices=1", + query: "6082444?indices=3", httpCode: 404, msg: "Blob not found in EthStorage", }, From 1f9cb968c46de8f13974e7f1bc9145eb06ece745 Mon Sep 17 00:00:00 2001 From: syntrust Date: Wed, 16 Oct 2024 12:03:33 +0800 Subject: [PATCH 06/11] updates --- ethstorage/archiver/test.md | 64 +++++++++++++++++++++---------------- 1 file changed, 37 insertions(+), 27 deletions(-) diff --git a/ethstorage/archiver/test.md b/ethstorage/archiver/test.md index 472ddc6b..74a7df50 100644 --- a/ethstorage/archiver/test.md +++ b/ethstorage/archiver/test.md @@ -2,13 +2,13 @@ ## Introduction -This document outlines the testing strategy and details for the blob archiver service of EthStorage. It aims to verify that the functionalities meet the specified requirements and is free of critical defects. +This document outlines the testing strategy and details for the blob archiver service of EthStorage. It aims to verify that the functionalities meet the specified requirements and are free of critical defects. ## Test Scope The scope of testing includes: -- Functional Testing: make sure expired blobs can be retreived from EthStorage correctly. -- Integration Testing: make sure OP Stack derivation works correclty with the retrieved blobs from EthStorage. +- **Functional Testing**: Ensure expired blobs can be retrieved from EthStorage correctly. +- **Integration Testing**: Ensure OP Stack derivation works correctly with the retrieved blobs from EthStorage. ## Environment Setup @@ -17,13 +17,13 @@ The tests will be conducted in an environment with the following services up and - An Ethereum L1 Beacon URL - An EthStorage node (es-node) with the archive service enabled - A Proxy of Ethereum Beacon RPC to mock a short blob retention period -- An OP node that uses EthStorage archive service +- An OP node that uses the EthStorage archive service -### Running an EthStorage node with the archive service enabled +### Running an EthStorage Node with the Archive Service Enabled -The following steps download source code of es-node, build, init, and run an es-node as a RPC node with archive service enabled. +The following steps download the source code of es-node, build, initialize, and run an es-node as an RPC node with the archive service enabled. -``` +```sh git clone https://github.com/ethstorage/es-node.git cd es-node @@ -35,14 +35,14 @@ make ./run-rpc.sh --archiver.enabled ``` -Noted that an Ethereum EL RPC in archive mode and an Ethereum Beacon URL are used implicitly in the shell scripts in below command. +Note that an Ethereum EL RPC in archive mode and an Ethereum Beacon URL are used implicitly in the shell scripts in the above commands. -For details and more options to run an es-node, please refer to https://docs.ethstorage.io/storage-provider-guide/tutorials. +For details and more options to run an es-node, please refer to the [EthStorage documentation](https://docs.ethstorage.io/storage-provider-guide/tutorials). ### Running a Proxy of Ethereum Beacon URL -``` +```sh git clone https://github.com/ethstorage/beacon-api-wrapper.git cd beacon-api-wrapper @@ -52,37 +52,46 @@ go run cmd/main.go The Proxy works just like a normal Beacon API except the blobs retension period is much shorter, like 3 hours by default. This means when a `blob_sidecars` request (`/eth/v1/beacon/blob_sidecars/{block_id}`) asking for blobs of more than 900 slot ago, it will return an empty list: `{"data":[]}`. -For details and more options please refer to https://github.com/ethstorage/beacon-api-wrapper#beacon-api-wrapper. +For details and more options please refer to this [repo](https://github.com/ethstorage/beacon-api-wrapper#beacon-api-wrapper). ### Running an OP Node -Following the [OP documentation](https://docs.optimism.io/builders/node-operators/tutorials/testnet) to run an op-node from source, and use the following options to make sure that -- the L1 Beacon URL points to the Proxy, and -- the archive service points to the es-node RPC. +Following the [OP documentation](https://docs.optimism.io/builders/node-operators/tutorials/testnet) to run an op-node from source, and use the following options to make sure that: +- The L1 Beacon URL points to the Proxy, and +- The archive service points to the es-node RPC. For example, -``` +```sh --l1.beacon http://65.108.236.27:3600 --l1.beacon-archiver http://65.108.236.27:9645 ``` ## Test Cases ### Test Case 1: Load an expired blob from EthStorage -- **Objective**: Verify that blobs that is expired by the Beacon Chain can be retrieved from EthStorage archive service. + +- **Objective**: + - Verify that the blobs expired by the Beacon Chain can be retrieved from EthStorage archive service. + - Verify that the blobs retrieved from EthStorage archive service are the same as from Beacon Chain in terms of blob data and some key properties. - **Steps**: 1. Upload a blob aimed to be archived by EthStorage. (You may need [eth-blob-uploader](#eth-blob-uploader) for this.) - 2. Record the slot as block_id of the blob for later use. - 3. Query `/eth/v1/beacon/blob_sidecars/{block_id}` from the Beacon Chain URL. For example, - ``` + 2. Find the slot number and `Versioned Hash` of the blob from [the block explorer](https://sepolia.etherscan.io/) for later use. + 3. Query blobs by `/eth/v1/beacon/blob_sidecars/{block_id}` from the Beacon Chain URL, replacing `{block_id}` with the slot in step 2. For example, + ```sh curl -X 'GET' 'http://88.99.30.186:3500/eth/v1/beacon/blob_sidecars/6103987' -H 'accept: application/json' - - ``` - 4. Wait for the blob expires and the above query returns `{"data":[]}`. - 5. Query `/eth/v1/beacon/blob_sidecars/{block_id}` from the EthStorage archive service. For example, ``` + 4. Wait for the blob expire and the above query to return `{"data":[]}`. + 5. Query blobs by `/eth/v1/beacon/blob_sidecars/{block_id}` from the EthStorage archive service, replacing `{block_id}` with the slot, just like in step 3. For example, + ```sh curl -X 'GET' 'http://65.108.236.27:9645/eth/v1/beacon/blob_sidecars/6103987' -H 'accept: application/json' ``` -- **Expected Result**: The result of step 3 and 5 are the same. +- **Expected Result**: + - The result of step 3 should contain the blob specifed by step 2. + - The result of step 5 should contain the blob specifed by step 2. + - The following properties have the same value between step 3 and 5: + - Blob data + - KZG Commitment + - KZG Proof + ### Test Case 2: @@ -94,7 +103,8 @@ For example, `eth-blob-uploader` is a handy tool to upload EIP-4844 blobs to Ethereum. The following command demostrates how to use `eth-blob-uploader` to upload a local file `hello.txt` to EthStorage testnet as a blob by calling the function `putBlob(bytes32 key,uint256 blobIdx,uint256 length)` of the storage contract deployed on Sepolia. -``` + +```sh eth-blob-uploader -r http://88.99.30.186:8545 \ -p \ -f hello.txt \ @@ -102,9 +112,9 @@ eth-blob-uploader -r http://88.99.30.186:8545 \ -v 1500000000000000 \ -d 0x4581a9201c8aff950685c2ed4bc3174f3472287b56d9517b9c948127319a09a7a36deac800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000 ``` -Record the transaction hash so that you can find on [EtherScan](https://sepolia.etherscan.io/) which slot the blob is belong to. +Record the transaction hash so that you can find the detail of the transaction on [the block explorer](https://sepolia.etherscan.io/) such as the `Versioned Hash` of the blob, and which slot the blob is belong to. Refer to https://github.com/ethstorage/eth-blob-uploader for more details. ## Conclusion -This document serves as the foundation for the software testing framework and process of the blob archiver service. It provides a clear structure for the testing activities to ensure quality outcomes. +This document serves as the foundation for the testing framework and process of the blob archiver service. From 4c738d98a37f0e61a348da5753fde1505b945b7a Mon Sep 17 00:00:00 2001 From: syntrust Date: Wed, 16 Oct 2024 12:04:45 +0800 Subject: [PATCH 07/11] green channel for rpc setup --- init.sh | 123 ++++++++++++++++++++++++++++++-------------------------- 1 file changed, 66 insertions(+), 57 deletions(-) diff --git a/init.sh b/init.sh index 9608865d..ccf9b8ed 100755 --- a/init.sh +++ b/init.sh @@ -17,6 +17,7 @@ data_dir="./es-data" remaining_args="" shards="--shard_index 0" +use_miner=1 while [ $# -gt 0 ]; do if [[ $1 == --miner.zk-prover-impl ]]; then @@ -28,10 +29,15 @@ while [ $# -gt 0 ]; do elif [[ $1 == --datadir ]]; then data_dir=$2 shift 2 + elif [[ $1 == --shard_index ]]; then + shards="" + remaining_args="$remaining_args $1" + shift + elif [[ $1 == --encoding_type=0 ]]; then # from init-rpc.sh + use_miner=0 + remaining_args="$remaining_args $1" + shift else - if [[ $1 == --shard_index ]]; then - shards="" - fi remaining_args="$remaining_args $1" shift fi @@ -39,70 +45,73 @@ done if [ -n "$zkp_mode" ] && [ "$zkp_mode" != 1 ] && [ "$zkp_mode" != 2 ]; then echo "Error: zk prover mode can only be 1 or 2." - exit 1 -fi - -# download zkey if not yet -zkey_name="blob_poseidon2.zkey" -zkey_size=560301223 -zkey_url="https://es-node-zkey.s3.us-west-1.amazonaws.com/blob_poseidon2_testnet1.zkey" -if [ "$zkp_mode" = 1 ]; then - zkey_name="blob_poseidon.zkey" - zkey_size=280151245 - zkey_url="https://drive.usercontent.google.com/download?id=1ZLfhYeCXMnbk6wUiBADRAn1mZ8MI_zg-&export=download&confirm=t&uuid=16ddcd58-2498-4d65-8931-934df3d0065c" + exit 1 fi -zkey_file="./build/bin/snark_lib/zkey/$zkey_name" -if [ ! -e ${zkey_file} ] || [ $(wc -c < ${zkey_file}) -ne ${zkey_size} ]; then - echo "Start downloading ${zkey_file}..." - curl $zkey_url -o ${zkey_file} - if [ ! -e ${zkey_file} ]; then - echo "Error: The zkey file was not downloaded. Please try again." - exit 1 + +if [ $use_miner = 1 ]; then + # download zkey if not yet + zkey_name="blob_poseidon2.zkey" + zkey_size=560301223 + zkey_url="https://es-node-zkey.s3.us-west-1.amazonaws.com/blob_poseidon2_testnet1.zkey" + if [ "$zkp_mode" = 1 ]; then + zkey_name="blob_poseidon.zkey" + zkey_size=280151245 + zkey_url="https://drive.usercontent.google.com/download?id=1ZLfhYeCXMnbk6wUiBADRAn1mZ8MI_zg-&export=download&confirm=t&uuid=16ddcd58-2498-4d65-8931-934df3d0065c" fi - if [ $(wc -c < ${zkey_file}) -ne ${zkey_size} ]; then - echo "Error: The zkey file was not downloaded correctly. You can check the file content for more information." - exit 1 + zkey_file="./build/bin/snark_lib/zkey/$zkey_name" + if [ ! -e ${zkey_file} ] || [ $(wc -c < ${zkey_file}) -ne ${zkey_size} ]; then + echo "Start downloading ${zkey_file}..." + curl $zkey_url -o ${zkey_file} + if [ ! -e ${zkey_file} ]; then + echo "Error: The zkey file was not downloaded. Please try again." + exit 1 + fi + if [ $(wc -c < ${zkey_file}) -ne ${zkey_size} ]; then + echo "Error: The zkey file was not downloaded correctly. You can check the file content for more information." + exit 1 + fi + else + echo "√ ${zkey_file} already exists." fi -else - echo "√ ${zkey_file} already exists." -fi - -if [ -n "$zkp_impl" ] && [ "$zkp_impl" != 1 ] && [ "$zkp_impl" != 2 ]; then - echo "Error: miner.zk-prover-impl can only be 1 or 2" - exit 1 -fi + if [ -n "$zkp_impl" ] && [ "$zkp_impl" != 1 ] && [ "$zkp_impl" != 2 ]; then + echo "Error: miner.zk-prover-impl can only be 1 or 2" + exit 1 + fi -if [ "$zkp_impl" = 1 ]; then - if ! [ -x "$(command -v node)" ]; then - echo 'Error: Node.js is not installed.' - exit 1 - fi + if [ "$zkp_impl" = 1 ]; then - # check node js version - node_version=$(node -v) - major_version=$(echo $node_version | cut -d'v' -f2 | cut -d'.' -f1) - if [ "$major_version" -lt 16 ]; then - echo "Error: Node.js version is too old: $node_version; must be 16 and above." + if ! [ -x "$(command -v node)" ]; then + echo 'Error: Node.js is not installed.' exit 1 - else - echo "√ Node.js version is compatible." - fi + fi - # install snarkjs if not - if ! [ "$(command -v snarkjs)" ]; then - echo "snarkjs not found, start installing..." - snarkjs_install=$(npm install -g snarkjs 2>&1) - if [ $? -eq 0 ]; then - echo "√ snarkjs installed successfully." - else - echo "Error: snarkjs install failed with the following error:" - echo "$snarkjs_install" + # check node js version + node_version=$(node -v) + major_version=$(echo $node_version | cut -d'v' -f2 | cut -d'.' -f1) + if [ "$major_version" -lt 16 ]; then + echo "Error: Node.js version is too old: $node_version; must be 16 and above." exit 1 - fi - else - echo "√ snarkjs is already installed." + else + echo "√ Node.js version is compatible." + fi + + # install snarkjs if not + if ! [ "$(command -v snarkjs)" ]; then + echo "snarkjs not found, start installing..." + snarkjs_install=$(npm install -g snarkjs 2>&1) + if [ $? -eq 0 ]; then + echo "√ snarkjs installed successfully." + else + echo "Error: snarkjs install failed with the following error:" + echo "$snarkjs_install" + exit 1 + fi + else + echo "√ snarkjs is already installed." + fi + fi fi From dfc434a8050430acaafd766caa83af47b4469876 Mon Sep 17 00:00:00 2001 From: syntrust Date: Tue, 22 Oct 2024 10:21:51 +0800 Subject: [PATCH 08/11] update --- ethstorage/archiver/test.md | 28 ++++++++++++++++++---------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/ethstorage/archiver/test.md b/ethstorage/archiver/test.md index 74a7df50..1382b6a4 100644 --- a/ethstorage/archiver/test.md +++ b/ethstorage/archiver/test.md @@ -17,7 +17,7 @@ The tests will be conducted in an environment with the following services up and - An Ethereum L1 Beacon URL - An EthStorage node (es-node) with the archive service enabled - A Proxy of Ethereum Beacon RPC to mock a short blob retention period -- An OP node that uses the EthStorage archive service +- An OP node that uses the EthStorage archive service as the backup source of blobs ### Running an EthStorage Node with the Archive Service Enabled @@ -54,16 +54,10 @@ The Proxy works just like a normal Beacon API except the blobs retension period For details and more options please refer to this [repo](https://github.com/ethstorage/beacon-api-wrapper#beacon-api-wrapper). -### Running an OP Node +### Running an OP Stack rollup Following the [OP documentation](https://docs.optimism.io/builders/node-operators/tutorials/testnet) to run an op-node from source, and use the following options to make sure that: -- The L1 Beacon URL points to the Proxy, and -- The archive service points to the es-node RPC. - -For example, -```sh ---l1.beacon http://65.108.236.27:3600 --l1.beacon-archiver http://65.108.236.27:9645 -``` +- The BatchInbox is deployed on L1 and used as batch receiver. ## Test Cases @@ -93,7 +87,21 @@ For example, - KZG Proof -### Test Case 2: +### Test Case 2: Sync an OP node using blobs from EthStorage +- **Objective**: +- **Steps**: +1. Following this [tutorial](https://github.com/ethstorage/pm/blob/main/L2/testnet_new_node.md) to set up an op-node, and try to sycn data from the Beacon Chain. + +2. Change configuation of the OP node, making sure that: + - The L1 Beacon URL points to the Proxy, and + - The archive service points to the es-node RPC. + +For example, +```sh +--l1.beacon http://65.108.236.27:3600 --l1.beacon-archiver http://65.108.236.27:9645 +``` +- **Expected Result**: + ## Tools From 2eb208ac1152fe5adf24dd04f91d400ad531d38f Mon Sep 17 00:00:00 2001 From: syntrust Date: Thu, 24 Oct 2024 15:45:50 +0800 Subject: [PATCH 09/11] rm doc --- ethstorage/archiver/test.md | 128 ------------------------------------ 1 file changed, 128 deletions(-) delete mode 100644 ethstorage/archiver/test.md diff --git a/ethstorage/archiver/test.md b/ethstorage/archiver/test.md deleted file mode 100644 index 1382b6a4..00000000 --- a/ethstorage/archiver/test.md +++ /dev/null @@ -1,128 +0,0 @@ -# EthStorage Archive Service Testing Document - -## Introduction - -This document outlines the testing strategy and details for the blob archiver service of EthStorage. It aims to verify that the functionalities meet the specified requirements and are free of critical defects. - -## Test Scope - -The scope of testing includes: -- **Functional Testing**: Ensure expired blobs can be retrieved from EthStorage correctly. -- **Integration Testing**: Ensure OP Stack derivation works correctly with the retrieved blobs from EthStorage. - -## Environment Setup - -The tests will be conducted in an environment with the following services up and running: -- An Ethereum L1 RPC of an execution client running in archive mode -- An Ethereum L1 Beacon URL -- An EthStorage node (es-node) with the archive service enabled -- A Proxy of Ethereum Beacon RPC to mock a short blob retention period -- An OP node that uses the EthStorage archive service as the backup source of blobs - -### Running an EthStorage Node with the Archive Service Enabled - -The following steps download the source code of es-node, build, initialize, and run an es-node as an RPC node with the archive service enabled. - -```sh -git clone https://github.com/ethstorage/es-node.git - -cd es-node - -make - -./init-rpc.sh - -./run-rpc.sh --archiver.enabled -``` - -Note that an Ethereum EL RPC in archive mode and an Ethereum Beacon URL are used implicitly in the shell scripts in the above commands. - - -For details and more options to run an es-node, please refer to the [EthStorage documentation](https://docs.ethstorage.io/storage-provider-guide/tutorials). - -### Running a Proxy of Ethereum Beacon URL - -```sh -git clone https://github.com/ethstorage/beacon-api-wrapper.git - -cd beacon-api-wrapper - -go run cmd/main.go -``` - -The Proxy works just like a normal Beacon API except the blobs retension period is much shorter, like 3 hours by default. This means when a `blob_sidecars` request (`/eth/v1/beacon/blob_sidecars/{block_id}`) asking for blobs of more than 900 slot ago, it will return an empty list: `{"data":[]}`. - -For details and more options please refer to this [repo](https://github.com/ethstorage/beacon-api-wrapper#beacon-api-wrapper). - -### Running an OP Stack rollup - -Following the [OP documentation](https://docs.optimism.io/builders/node-operators/tutorials/testnet) to run an op-node from source, and use the following options to make sure that: -- The BatchInbox is deployed on L1 and used as batch receiver. - -## Test Cases - -### Test Case 1: Load an expired blob from EthStorage - -- **Objective**: - - Verify that the blobs expired by the Beacon Chain can be retrieved from EthStorage archive service. - - Verify that the blobs retrieved from EthStorage archive service are the same as from Beacon Chain in terms of blob data and some key properties. -- **Steps**: - 1. Upload a blob aimed to be archived by EthStorage. (You may need [eth-blob-uploader](#eth-blob-uploader) for this.) - 2. Find the slot number and `Versioned Hash` of the blob from [the block explorer](https://sepolia.etherscan.io/) for later use. - 3. Query blobs by `/eth/v1/beacon/blob_sidecars/{block_id}` from the Beacon Chain URL, replacing `{block_id}` with the slot in step 2. For example, - ```sh - curl -X 'GET' 'http://88.99.30.186:3500/eth/v1/beacon/blob_sidecars/6103987' -H 'accept: application/json' - ``` - 4. Wait for the blob expire and the above query to return `{"data":[]}`. - 5. Query blobs by `/eth/v1/beacon/blob_sidecars/{block_id}` from the EthStorage archive service, replacing `{block_id}` with the slot, just like in step 3. For example, - ```sh - curl -X 'GET' 'http://65.108.236.27:9645/eth/v1/beacon/blob_sidecars/6103987' -H 'accept: application/json' - ``` -- **Expected Result**: - - The result of step 3 should contain the blob specifed by step 2. - - The result of step 5 should contain the blob specifed by step 2. - - The following properties have the same value between step 3 and 5: - - Blob data - - KZG Commitment - - KZG Proof - - -### Test Case 2: Sync an OP node using blobs from EthStorage -- **Objective**: -- **Steps**: -1. Following this [tutorial](https://github.com/ethstorage/pm/blob/main/L2/testnet_new_node.md) to set up an op-node, and try to sycn data from the Beacon Chain. - -2. Change configuation of the OP node, making sure that: - - The L1 Beacon URL points to the Proxy, and - - The archive service points to the es-node RPC. - -For example, -```sh ---l1.beacon http://65.108.236.27:3600 --l1.beacon-archiver http://65.108.236.27:9645 -``` -- **Expected Result**: - - - -## Tools - -### eth-blob-uploader - -`eth-blob-uploader` is a handy tool to upload EIP-4844 blobs to Ethereum. - -The following command demostrates how to use `eth-blob-uploader` to upload a local file `hello.txt` to EthStorage testnet as a blob by calling the function `putBlob(bytes32 key,uint256 blobIdx,uint256 length)` of the storage contract deployed on Sepolia. - -```sh -eth-blob-uploader -r http://88.99.30.186:8545 \ --p \ --f hello.txt \ --t 0x804C520d3c084C805E37A35E90057Ac32831F96f \ --v 1500000000000000 \ --d 0x4581a9201c8aff950685c2ed4bc3174f3472287b56d9517b9c948127319a09a7a36deac800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000 -``` -Record the transaction hash so that you can find the detail of the transaction on [the block explorer](https://sepolia.etherscan.io/) such as the `Versioned Hash` of the blob, and which slot the blob is belong to. - -Refer to https://github.com/ethstorage/eth-blob-uploader for more details. - -## Conclusion -This document serves as the foundation for the testing framework and process of the blob archiver service. From 36f870f3f8e66f63ce2c164fd70f21d46626feb1 Mon Sep 17 00:00:00 2001 From: syntrust Date: Thu, 24 Oct 2024 15:51:37 +0800 Subject: [PATCH 10/11] revert --- integration_tests/archiver_test.go | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/integration_tests/archiver_test.go b/integration_tests/archiver_test.go index f4d8e3f2..c3fffee1 100644 --- a/integration_tests/archiver_test.go +++ b/integration_tests/archiver_test.go @@ -10,6 +10,7 @@ import ( "fmt" "io" "net/http" + "strconv" "strings" "testing" @@ -29,23 +30,35 @@ type test struct { msg string } +func (t *test) toUrl() string { + if len(t.archivedIndices) == 0 { + return fmt.Sprintf(urlPattern, archiverAddr, t.query) + } + var strArr []string + for _, val := range t.archivedIndices { + strArr = append(strArr, strconv.FormatUint(uint64(val), 10)) + } + query := fmt.Sprintf("%s?indices=%s", t.query, strings.Join(strArr, ",")) + return fmt.Sprintf(urlPattern, archiverAddr, query) +} + func TestArchiveAPI(t *testing.T) { tests := []test{ { - query: "6082444", // within 4096 epoch, can be retrieved from both beacon and archiver - archivedIndices: []uint64{0, 1}, + query: "4756895", + archivedIndices: []uint64{3}, }, { - query: "6082444?indices=0,1", // specified all the indices - archivedIndices: []uint64{0, 1}, + query: "4756895?indices=3", + archivedIndices: []uint64{3}, }, { - query: "6082444?indices=1", // specified one of the indices - archivedIndices: []uint64{1}, + query: "4756895?indices=0,3", + archivedIndices: []uint64{3}, }, { - query: "6082444?indices=3", + query: "4756895?indices=1", httpCode: 404, msg: "Blob not found in EthStorage", }, From 7858aa46f155af2e0d2abf010ac4f51de6fd4b22 Mon Sep 17 00:00:00 2001 From: syntrust Date: Thu, 24 Oct 2024 16:00:15 +0800 Subject: [PATCH 11/11] revert --- ethstorage/archiver/spec.md | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/ethstorage/archiver/spec.md b/ethstorage/archiver/spec.md index 672c3a22..6eee68de 100644 --- a/ethstorage/archiver/spec.md +++ b/ethstorage/archiver/spec.md @@ -149,7 +149,7 @@ When the API is queried, the content of blobs is loaded to generate `kzg_commitm One thing worth noting is how the blob's index is determined. We need to use the order of blob kzg commitments obtained in the beacon API `/eth/v2/beacon/blocks/{block_id}`, and use the corresponding relationship between `versionedash` and kzg commitment to obtain the index of the blob in the block. -### Considerations for Solution 2: +### Considerations for Solution 2: - It is assumed that the beacon API `/eth/v2/beacon/blocks/{block_id}` is always available for any blocks, and the event log data from the execution layer is always available for any blocks. As EIP-4444 comes and execution payload data may not be available, we may need a key-value db table to save the relationship between the beacon block and kv index such as: @@ -159,9 +159,4 @@ BeaconRootHash->{"kv_index":[0,1,2]} - If the performance is proven to be a problem, we may need to add a local cache/prefetch layer to save the metadata. - If the blob transaction hashes are required, they can also be retrieved through standard JSON-RPC API. - -# Conclusion - -Given the complexity of Solution 1, which requires modifications to some key components of es-node, including the downloader and sync processes, Solution 2 is simpler and more straightforward, making it the preferred choice for implementation. - - +