Testnet Release (#1223)

* update package versions (#1119) * update release to install correct risc0 version (#1121) * make default rust version 1.79.0 in release (#1122) * make default rust version 1.79.0 * use home directory for rzup * fix rzup path (#1123) * Fix typo (#1129) * don't use rzup (#1139) * Update l1 fee overhead (#1137) Co-authored-by: eyusufatik <[email protected]> * Move sync blocks count (#1126) * Fullnode: bypass wallet check for bitcoin DA (#1128) * Add option to skip wallet check * Create bitcoin service based on wallet check requirement * Fix tests * Pass bool instead * E2E Add support for env (#1138) * E2E Add support for env * check if soft confirmation heights are sequential in zk (#1141) * change finality depth to 8 and change how depth is calculated (#1143) * change finality depth to 8 and change how depth is calculated * Fix prover test * fix test * fix release for macos * add DA_ID validation, release naming stadardization --------- Co-authored-by: jfldde <[email protected]> Co-authored-by: Cetin <[email protected]> * fix macos release ci (#1144) * Add make clean-all target (#1146) * Add clean-all target * Add clean-docker * add bitcoin da config for backup directory (#1148) * add bitcoin da config for backup directory * add new configs to bitcoin-regtest configs * use cargo manifest dir in tests to set tx backup dir * roman's comment * use get_tx_backup_dir in bitcoin da service tests * Change deposit script prefix (#1150) * Change deposit script prefix * Test fixes * Test bridge fix * Use bonsai retry macro (#1145) * include pending utxos in get_utxos (#1154) * Add cycle-diff script and perf CI (#1153) * Add cycle-diff script and perf CI * Set TARGET_PCT to 3 * Set cargo test package * Shellcheck * Rename to base_* * Temporarly run CI on this branch * Output generate to stdout as well * Remove redundant build step * Don't silence build output * Upload correct result file * Fix yml * Add debug * Use docker to run e2e tests * Remove debug * Remove branch trigger * Add NUM_RUNS and average * Don't check regression on Execution Time * Set bitcoincore-rev (#1155) * future proof da parsing in sequencer and prover (#1156) * parse seq comms future proof in zk (#1157) * Make trait verify parsed public (#1161) * full node fix l1 start height (#1160) * move prover last scanned l1 height table and functions to shared ledger ops * full node use last scan l1 table * Update crates/fullnode/src/runner.rs Co-authored-by: Erce Can Bektüre <[email protected]> --------- Co-authored-by: Erce Can Bektüre <[email protected]> * Remove redundant build (#1164) * Process L1 blocks in a different task. (#1163) * Process L1 blocks in a different task. This introduces handler for L1 blocks which is spawned as a separate task * Add DA block handler * Happy clippy * Move all L1 processing into L1BlockHandler * Remove uninformative log * Keep logic for retrying L1 blocks if L2 range is missing * Happy Clippy: 2 * Process L2 blocks in order * add some logs * Process blocks immediately and queue if failed * Import info * Fix logs * Process all L2 blocks in pending * more logs --------- Co-authored-by: eyusufatik <[email protected]> * Make l2 height of GetSoftConfirmationResponse public (#1165) * Further restrict eth_getLogs (#1166) * Further restrict eth_getLogs * Log test fix * Log test fix * Prover: improve L1 / L2 sync (#1168) * Separate L1 block sync * L2 worker to push to pending blocks * Enable prover simulation (#1171) * Enable prover simulation * Unneeded return * Use Mutex instead of RwLock --------- Co-authored-by: Esad Yusuf Atik <[email protected]> * Don't delete account code (#1179) * Don't delete account code * Don't charge l1 fee for delete acc.code * use BlockId in some evm queries (#1182) * use BlockId in some evm queries * Update checks.yml * Update checks.yml --------- Co-authored-by: Erce Can Bektüre <[email protected]> * Don't insert duplicate account codes (#1180) * Don't insert duplicate account codes * Don't charge l1 fee if code is already in Db * Fix lints * StorageInternalCache::exists -> contains_key * Revert exists/contains_key * Don't insert code on duplicates * Add validate_and_check_DA_ID job to checks.yml (#1177) * Add validate_and_check_DA_ID job * Try share build * Fix path * Use smaller instance * add locked flag to cargo-udeps (#1187) * add locked flag to cargo-udeps * change order of flags * fix usage of wrong block env (#1189) * Optimize rocksdb (#1127) * Optimize rocksdb * Set LRU cache * Remove unused ledger tables (#1131) * Remove unused ledger tables * record deposit data, why wouldn't we?< * lint --------- Co-authored-by: Esad Yusuf Atik <[email protected]> Co-authored-by: eyusufatik <[email protected]> * Complete test implementation (#1124) * Complete test implementation * Add comment * Wait for proof * Sort before checking range * Send shutdown message to da queue * Cleanup * Remove docker from prover test config * Restore prefixes * Update bin/citrea/tests/bitcoin_e2e/tests/prover_test.rs Co-authored-by: jfldde <[email protected]> * Restore inscribes_queue * Use finality depth to generate 1 finalized DA height * Unwrap or else * Update crates/prover/src/da_block_handler.rs * Mention L1 height in log * Remove unnecessary sort --------- Co-authored-by: jfldde <[email protected]> * Add l1 status to citrea_syncStatus (#1192) * add ledger_db to get_ethereum_rpc * add ledger_db to ethereum_rpc::Ethereum, get l1 block numbers * changed CitreaStatus and add err handling * fix test * format code, remove unused imports * add l1 sync test * rename CitreaStatus, use finalized l1 height * make requests async * switch from spawn to join * fix error names + use ledger db for head l2 * fix lint --------- Co-authored-by: eyusufatik <[email protected]> * Use bitcoincore-rpc reqwest transport (#1195) * Remove redundant install (#1196) * use tick interval in sequencer (#1197) * use tick interval in sequencer * fix rakan's comment * fix test_minimum_base_fee test * make ethers.js test less flaky * use bigger machine for build ci * try fixing ethers.js ci * log ethers.js error * fix ethers.js test * Process missed da blocks on start (#1201) * Process missed da blocks on start * Simulate with a test * Restore deleted by mistake code * Disable log * some improvements --------- Co-authored-by: eyusufatik <[email protected]> * Fix block number event bug in light client (#1203) * Fix block number event bug in light client * Test fixes * Further test fix * Avoid calling spawn_da_queue for full node (#1204) * dont spawn_da_queue for full nodes * format * edit comment * disable healthcheck (#1205) * CI: Checks cleanup (#1209) * SKIP_GUEST_BUILD in udeps * Remove Rust toolchain from web3 tests * Cleanup components and warning * Use SHORT_PREFIX env in CI (#1206) * Use SHORT_PREFIX env to reduce prefix length. Reduce nonce finding difficulty, speeds up bitcoin_e2e tests and addresses related flakiness. * Use SHORT_PREFIX for test and coverage only * Revet short backtraces * Revert redundant to_vec * Revert import in bitcoin-da * add command explaining prefix stuff * Merge comments --------- Co-authored-by: eyusufatik <[email protected]> * Customisable workflow dispatch name (#1207) * Check compressed state diff size for generating proofs (#1212) * Check compressed state diff size for generating proofs * Better naming * Clarify why we check against compressed state diff size * Use const in bitcoinda * Fmt * Move const to primitives * Dprint * Fix * Testnet Instructions (#1219) * create run-testnet.dm * add testnet full node config * add run-testnet.md * make initial doc * update instructions * fix curls * add testnet genesis * remove todo * Update readme for testnet (#1225) * Update readme for testnet * Better text --------- Co-authored-by: Rakan Al-Huneiti <[email protected]> Co-authored-by: Erce Can Bektüre <[email protected]> Co-authored-by: jfldde <[email protected]> Co-authored-by: Cetin <[email protected]> Co-authored-by: okkothejawa <[email protected]> Co-authored-by: Roman <[email protected]> Co-authored-by: Ege Okan Ünaldı <[email protected]> Co-authored-by: Ömer Talip Akalın <[email protected]>
chainwayxyz · Sep 24, 2024 · cc2dc72 · cc2dc72
1 parent 953e109
commit cc2dc72
Show file tree

Hide file tree

Showing 126 changed files with 4,258 additions and 2,385 deletions.
diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml
diff --git a/.github/workflows/docker_hive.yml b/.github/workflows/docker_hive.yml
@@ -15,7 +15,7 @@ concurrency:
   group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
   # Except in `nightly` and `stable` branches! Any cancelled job will cause the
   # CI run to fail, and we want to keep a clean history for major branches.
-  cancel-in-progress: ${{ (github.ref != 'refs/heads/nightly') && (github.ref != 'refs/heads/main') }}
+  cancel-in-progress: ${{ (github.ref != 'refs/heads/nightly') && (github.ref != 'refs/heads/devnet-freeze') && (github.ref != 'refs/heads/main')}}
 
 jobs:
   docker:

diff --git a/.github/workflows/hive.yml b/.github/workflows/hive.yml
@@ -16,7 +16,7 @@ concurrency:
   group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
   # Except in `nightly` and `stable` branches! Any cancelled job will cause the
   # CI run to fail, and we want to keep a clean history for major branches.
-  cancel-in-progress: ${{ (github.ref != 'refs/heads/nightly') && (github.ref != 'refs/heads/main') }}
+  cancel-in-progress: ${{ (github.ref != 'refs/heads/nightly') && (github.ref != 'refs/heads/devnet-freeze') && (github.ref != 'refs/heads/main')}}
 
 jobs:
   prepare:

diff --git a/.github/workflows/perf.yml b/.github/workflows/perf.yml
@@ -0,0 +1,61 @@
+name: Performance Comparison
+
+on:
+  workflow_dispatch:
+
+env:
+  CARGO_TERM_COLOR: always
+  FOUNDRY_PROFILE: ci
+  TARGET_PCT: 3
+  COMPARISON_FILE: comparison_results.log
+  USE_DOCKER: "true"
+
+jobs:
+  performance-comparison:
+    runs-on: ubicloud-standard-16
+    steps:
+      - uses: actions/checkout@v4
+      - name: Fetch latest nightly
+        run: |
+          git fetch origin nightly:nightly
+      - uses: rui314/setup-mold@v1
+      - name: Install Protoc
+        uses: arduino/setup-protoc@v2
+        with:
+          version: "23.2"
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
+      - name: Toolchain
+        uses: dtolnay/[email protected]
+        with:
+          override: true
+          components: rustfmt, clippy
+      - name: Rust Cache
+        uses: Swatinem/rust-cache@v2
+      - name: Install cargo-risczero
+        uses: taiki-e/install-action@v2
+        with:
+          tool: [email protected]
+      - name: Install risc0-zkvm toolchain
+        run: cargo risczero install --version r0.1.79.0-2
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      - name: Cache ethereum-tests
+        uses: actions/cache@v3
+        with:
+          key: "eth-tests-1c23e3c"
+          path: crates/evm/ethereum-tests
+
+      - name: Run Performance Comparison
+        run: |
+          chmod +x ./resources/scripts/cycle-diff.sh
+          ./resources/scripts/cycle-diff.sh generate
+
+      - name: Check Performance Regression
+        run: |
+          ./resources/scripts/cycle-diff.sh check
+
+      - name: Upload comparison results
+        uses: actions/upload-artifact@v4
+        with:
+          name: comparison-results
+          path: comparison_results.log
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -11,7 +11,84 @@ env:
 
 jobs:
 
-  x86_64_binary_extraction:
+  validate_DA_ID_format:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Validate EXPECTED_BITCOIN_DA_ID format
+        run: |
+          echo "Raw EXPECTED_BITCOIN_DA_ID value:"
+          echo "$EXPECTED_BITCOIN_DA_ID"
+          
+          echo "Length of EXPECTED_BITCOIN_DA_ID: ${#EXPECTED_BITCOIN_DA_ID}"
+          
+          if [ -z "${EXPECTED_BITCOIN_DA_ID// }" ]; then
+            echo "Error: EXPECTED_BITCOIN_DA_ID is not set, empty, or contains only spaces"
+            exit 1
+          fi
+          
+          # Remove any trailing newline or carriage return
+          EXPECTED_BITCOIN_DA_ID=$(echo "$EXPECTED_BITCOIN_DA_ID" | tr -d '\n\r')
+          
+          # Count commas and spaces
+          comma_count=$(echo "$EXPECTED_BITCOIN_DA_ID" | tr -cd ',' | wc -c)
+          space_count=$(echo "$EXPECTED_BITCOIN_DA_ID" | tr -cd ' ' | wc -c)
+          
+          echo "Number of commas: $comma_count"
+          echo "Number of spaces: $space_count"
+          
+          # Split the string into an array and trim each element
+          IFS=', ' read -ra raw_numbers <<< "$EXPECTED_BITCOIN_DA_ID"
+          numbers=()
+          for num in "${raw_numbers[@]}"; do
+            trimmed_num=$(echo "$num" | tr -d '[:space:]')  # Remove all whitespace
+            numbers+=("$trimmed_num")
+          done
+          
+          echo "Number of elements after splitting and trimming: ${#numbers[@]}"
+          
+          # Check if there are exactly 8 numbers
+          if [ ${#numbers[@]} -ne 8 ]; then
+            echo "Error: EXPECTED_BITCOIN_DA_ID should contain exactly 8 numbers"
+            echo "Actual number of elements: ${#numbers[@]}"
+            exit 1
+          fi
+          
+          # Check if all numbers are valid u32
+          for i in "${!numbers[@]}"; do
+            num=${numbers[$i]}
+            echo "Checking number $((i+1)): '$num'"
+            echo "Hex representation: $(echo -n "$num" | xxd -p)"
+            if ! [[ $num =~ ^[0-9]+$ ]]; then
+              echo "Error: '$num' is not composed of digits only"
+              exit 1
+            fi
+            if [ $num -gt 4294967295 ]; then
+              echo "Error: '$num' is greater than 4294967295"
+              exit 1
+            fi
+          done
+          
+          # Reconstruct the trimmed DA_ID
+          trimmed_da_id=$(IFS=', '; echo "${numbers[*]}")
+          
+          # Final check
+          if [ $comma_count -eq 7 ] && [ $space_count -eq 7 ] && [ ${#numbers[@]} -eq 8 ]; then
+            echo "EXPECTED_BITCOIN_DA_ID is valid:"
+            echo "- Contains 7 commas"
+            echo "- Contains 7 spaces"
+            echo "- Contains 8 valid u32 numbers"
+            echo "Original value: $EXPECTED_BITCOIN_DA_ID"
+            echo "Trimmed value: $trimmed_da_id"
+          else
+            echo "Error: EXPECTED_BITCOIN_DA_ID format is incorrect"
+            echo "- Comma count: $comma_count (should be 7)"
+            echo "- Space count: $space_count (should be 7)"
+            echo "- Number count: ${#numbers[@]} (should be 8)"
+            exit 1
+          fi
+
+  linux_amd64_binary_extraction:
+    needs: validate_DA_ID_format
     runs-on: ubicloud-standard-30
     steps:
       - name: Checkout
@@ -25,18 +102,22 @@ jobs:
       - name: Install Rust
         run: |
           curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
+          rustup install 1.79.0
+          rustup default 1.79.0
 
       - name: Install Cargo Binstall
         run: |
           source $HOME/.cargo/env
-          cargo install cargo-binstall
+          cargo install --version 1.7.0 cargo-binstall
 
-      - name: Install Cargo Risczero
+      - name: Install cargo-risczero
         run: |
-          cargo binstall -y cargo-risczero
-          cargo risczero install --version v2024-04-22.0
-          # TODO: do below on testnet launch
-          # curl -L https://risczero.com/install | bash; source /home/runner/.bashrc; rzup -v 1.0.5;
+          cargo binstall [email protected] --no-confirm
+
+      - name: Install risc0-zkvm toolchain
+        run: cargo risczero install --version r0.1.79.0-2
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Build Project
         env:
@@ -48,22 +129,26 @@ jobs:
         id: check-id
         run: |
           RESULT=$(grep -R "BITCOIN_DA_ID" target/ || echo "Grep failed")
-
-          if echo "$RESULT" | grep -q "${{ env.EXPECTED_BITCOIN_DA_ID }}"; then
+          EXPECTED_BITCOIN_DA_ID=$(echo "${{ env.EXPECTED_BITCOIN_DA_ID }}" | tr -d '\n\r')
+          if echo "$RESULT" | grep -q "$EXPECTED_BITCOIN_DA_ID"; then
             echo "Check passed successfully."
-          else
+            echo "Expected: BITCOIN_DA_ID ${{ env.EXPECTED_BITCOIN_DA_ID }} "
+            echo "Actual: $RESULT"
+              
+          else      
             echo "Check failed. Expected: BITCOIN_DA_ID ${{ env.EXPECTED_BITCOIN_DA_ID }} "
             echo "Actual: $RESULT"
             exit 1
           fi
 
-      - name: Upload x86_64 Binary
+      - name: Upload linux-amd64 Binary
         uses: actions/upload-artifact@v4
         with:
-          name: citrea_${{ github.ref_name }}_x86_64
+          name: citrea-${{ github.ref_name }}-linux-amd64
           path: target/release/citrea
 
   osx_arm64_binary_extraction:
+    needs: validate_DA_ID_format
     runs-on: self-hosted-citrea-osx-arm64
     steps:
       - name: Checkout
@@ -72,18 +157,20 @@ jobs:
       - name: Install Rust
         run: |
           curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
+          rustup install 1.79.0
+          rustup default 1.79.0
 
       - name: Install Cargo Binstall
         run: |
           source $HOME/.cargo/env
-          cargo install cargo-binstall
-
-      - name: Install Cargo Risczero
+          cargo install --version 1.7.0 cargo-binstall
+      - name: Install cargo-risczero
         run: |
-          source $HOME/.cargo/env
-          cargo binstall -y cargo-risczero
-          cargo risczero install --version v2024-04-22.0
-
+          cargo binstall [email protected] --no-confirm
+      - name: Install risc0-zkvm toolchain
+        run: cargo risczero install --version r0.1.79.0-2
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
       - name: Build Project
         env:
           REPR_GUEST_BUILD: 1
@@ -95,55 +182,58 @@ jobs:
         id: check-id
         run: |
           RESULT=$(grep -R "BITCOIN_DA_ID" target/ || echo "Grep failed")
-          
-          if echo "$RESULT" | grep -q "${{ env.EXPECTED_BITCOIN_DA_ID }}"; then
+          EXPECTED_BITCOIN_DA_ID=$(echo "${{ env.EXPECTED_BITCOIN_DA_ID }}" | tr -d '\n\r')
+          if echo "$RESULT" | grep -q "$EXPECTED_BITCOIN_DA_ID"; then
             echo "Check passed successfully."
+            echo "Expected: BITCOIN_DA_ID ${{ env.EXPECTED_BITCOIN_DA_ID }} "
+            echo "Actual: $RESULT"
+              
           else      
             echo "Check failed. Expected: BITCOIN_DA_ID ${{ env.EXPECTED_BITCOIN_DA_ID }} "
             echo "Actual: $RESULT"
             exit 1
           fi
 
-      - name: Upload osx_arm64 Binary
+      - name: Upload osx-arm64 Binary
         uses: actions/upload-artifact@v4
         with:
-          name: citrea_${{ github.ref_name }}_osx_arm64
+          name: citrea-${{ github.ref_name }}-osx-arm64
           path: target/release/citrea
 
   release:
-    needs: [ x86_64_binary_extraction, osx_arm64_binary_extraction ]
+    needs: [ linux_amd64_binary_extraction, osx_arm64_binary_extraction ]
     runs-on: ubuntu-latest
     steps:
-      - name: Download x86_64 Binary
+      - name: Download linux-amd64 Binary
         uses: actions/download-artifact@v4
         with:
-          name: citrea_${{ github.ref_name }}_x86_64
+          name: citrea-${{ github.ref_name }}-linux-amd64
           path: release
 
       - name: rename file
         run: |
-          mv release/citrea release/citrea_${{ github.ref_name }}_x86_64
+          mv release/citrea release/citrea-${{ github.ref_name }}-linux-amd64
 
       - name: Download OSX ARM64 Binary
         uses: actions/download-artifact@v4
         with:
-          name: citrea_${{ github.ref_name }}_osx_arm64
+          name: citrea-${{ github.ref_name }}-osx-arm64
           path: release
 
       - name: rename file
         run: |
-            mv release/citrea release/citrea_${{ github.ref_name }}_osx_arm64
+            mv release/citrea release/citrea-${{ github.ref_name }}-osx-arm64
 
       - name: Release
         uses: softprops/action-gh-release@v2
         with:
           files: |
-            release/citrea_${{ github.ref_name }}_osx_arm64
-            release/citrea_${{ github.ref_name }}_x86_64
+            release/citrea-${{ github.ref_name }}-osx-arm64
+            release/citrea-${{ github.ref_name }}-linux-amd64
           name: Release ${{ github.ref_name }}
           body: |
             This is the release for version ${{ github.ref_name }}.
             
             It includes:
-            - citrea_${{ github.ref_name }}_x86_64
-            - citrea_${{ github.ref_name }}_osx_arm64
+            - citrea-${{ github.ref_name }}-linux-amd64
+            - citrea-${{ github.ref_name }}-osx-arm64