Merge pull request #2238 from demergent-labs/parallel_benchmarking

Parallel benchmarking

.github/actions/README.md

@@ -0,0 +1,10 @@
+## Prerequisite
+
+These actions assume that the repository has already been checked out before
+calling the action, typically using `actions/checkout@v4`. If you have not
+checked out the code in a previous step, make sure to do so to avoid errors.
+
+Unless otherwise specified, these actions do **not** perform a checkout action
+themselves because it would be redundant. These actions are part of the
+repository's codebase, so if the code hasn't already been checked out, the
+actions themselves wouldn't even be available to call.

.github/actions/commit_and_push/README.md

@@ -0,0 +1,12 @@
+## Example Usage
+
+```yaml
+steps:
+    - uses: actions/checkout@v4
+
+    - uses: ./.github/actions/commit_and_push
+      with:
+          gpg_signing_key: ${{ secrets.GPG_SIGNING_KEY }}
+          branch-name: 'branch-name'
+          commit-message: 'commit message'
+```

.github/actions/commit_and_push/action.yml

@@ -0,0 +1,33 @@
+name: Commit and Push Changes
+description: 'Configures git, commits changes, and pushes to a new branch'
+inputs:
+    branch-name:
+        description: 'Name of the branch to create'
+        required: true
+    commit-message:
+        description: 'Commit message'
+        required: true
+    gpg_signing_key:
+        description: 'The GPG signing key to use for signing commits'
+        required: true
+runs:
+    using: composite
+    steps:
+        - uses: ./.github/actions/configure_git
+          with:
+              gpg_signing_key: ${{ inputs.gpg_signing_key }}
+
+        - name: Commit and push changes
+          shell: bash
+          run: |
+              # Create and switch to new branch
+              git switch -c "${{ inputs.branch-name }}"
+
+              # Show status of working directory
+              echo "Current git status:"
+              git status
+
+              # Add and commit changes if there are any
+              git add --all
+              git commit -m "${{ inputs.commit-message }}"
+              git push origin "${{ inputs.branch-name }}"

.github/actions/configure_git/README.md

@@ -1,14 +1,3 @@
-## Prerequisite
-
-This action assumes that the repository has already been checked out before
-calling the action, typically using `actions/checkout@v4`. If you have not
-checked out the code in a previous step, make sure to do so to avoid errors.
-
-This action does **not** perform a checkout action itself because it would be
-redundant. This action is part of the repository's codebase, so if the code
-hasn't already been checked out, the action itself wouldn't even be available to
-call.
-
 ## Example Usage
 
 ```yaml

.github/actions/create_branch_name/README.md

@@ -0,0 +1,13 @@
+## Example Usage
+
+```yaml
+steps:
+    - id: create-branch-name
+      uses: ./.github/actions/create_branch_name
+      with:
+          prefix: 'update--0.24.2-rc.89-'
+          path: 'examples/hello_world'
+```
+
+The action will convert the path to use hyphens instead of slashes and combine it with the prefix.
+For example, with the inputs above, it would output: `update--examples-hello_world`

.github/actions/create_branch_name/action.yml

@@ -0,0 +1,21 @@
+name: Create Branch Name
+description: 'Creates a branch name by combining a prefix with a path, converting slashes to hyphens'
+inputs:
+    prefix:
+        description: 'Prefix for the branch name'
+        required: true
+    path:
+        description: 'Path to convert into branch name'
+        required: true
+outputs:
+    branch-name:
+        description: 'The generated branch name'
+        value: ${{ steps.create-branch-name.outputs.branch-name }}
+runs:
+    using: composite
+    steps:
+        - id: create-branch-name
+          shell: bash
+          run: |
+              SAFE_PATH=$(echo '${{ inputs.path }}' | sed 's/\//-/g')
+              echo "branch-name=${{ inputs.prefix }}${SAFE_PATH}" >> $GITHUB_OUTPUT

.github/actions/get_exclude_dirs/README.md

@@ -0,0 +1,15 @@
+## Example Usage
+
+```yaml
+steps:
+    - uses: actions/checkout@v4
+
+    - id: get-exclude-dirs
+      uses: ./.github/actions/get_exclude_dirs
+      with:
+          exclude-slow: true
+          exclude-unstable: true
+          exclude-release: true
+
+    - run: echo "${{ steps.get-exclude-dirs.outputs.exclude-dirs }}"
+```

.github/actions/get_exclude_dirs/action.yml

@@ -0,0 +1,71 @@
+name: Get Exclude Directories
+description: 'Gets a list of directories to exclude based on input parameters'
+inputs:
+    exclude-slow:
+        description: 'Whether to exclude slow benchmarks'
+        required: true
+    exclude-unstable:
+        description: 'Whether to exclude unstable benchmarks'
+        required: true
+    exclude-release-only:
+        description: 'Whether to exclude release only benchmarks'
+        required: true
+outputs:
+    exclude-dirs:
+        description: 'Space-separated list of directories to exclude'
+        value: ${{ steps.set-exclude-dirs.outputs.exclude-dirs }}
+runs:
+    using: composite
+    steps:
+        - id: set-exclude-dirs
+          shell: bash
+          run: |
+              RELEASE_ONLY_EXCLUSIONS="${{ format('
+              tests/end_to_end/candid_rpc/class_syntax/new
+              tests/end_to_end/http_server/new
+              ') }}"
+
+              UNSTABLE_EXCLUSIONS="${{ format('
+              examples/basic_bitcoin
+              examples/bitcoin_psbt
+              examples/ckbtc
+              tests/end_to_end/http_server/ethers_base
+              tests/end_to_end/http_server/http_outcall_fetch
+              tests/end_to_end/http_server/ic_evm_rpc
+              tests/property/candid_rpc/class_api/stable_b_tree_map
+              tests/property/candid_rpc/functional_api/stable_b_tree_map
+              tests/property/ic_api/performance_counter
+              tests/property/ic_api/instruction_counter
+              ') }}"
+
+              SLOW_EXCLUSIONS="${{ format('
+              tests/end_to_end/candid_rpc/functional_syntax/ckbtc
+              tests/end_to_end/candid_rpc/class_syntax/bitcoin
+              tests/end_to_end/http_server/large_files
+              tests/end_to_end/http_server/open_value_sharing
+              tests/end_to_end/candid_rpc/class_syntax/stable_structures
+              tests/end_to_end/candid_rpc/functional_syntax/bitcoin
+              tests/end_to_end/candid_rpc/functional_syntax/composite_queries
+              tests/end_to_end/candid_rpc/functional_syntax/cross_canister_calls
+              tests/end_to_end/candid_rpc/functional_syntax/management_canister
+              tests/end_to_end/candid_rpc/functional_syntax/stable_structures
+              tests/end_to_end/http_server/autoreload
+              ') }}"
+
+              EXCLUDE_DIRS=""
+
+              if [[ "${{ inputs.exclude-release-only }}" == "true" ]]; then
+                  EXCLUDE_DIRS="$EXCLUDE_DIRS $RELEASE_ONLY_EXCLUSIONS"
+              fi
+
+              if [[ "${{ inputs.exclude-unstable }}" == "true" ]]; then
+                  EXCLUDE_DIRS="$EXCLUDE_DIRS $UNSTABLE_EXCLUSIONS"
+              fi
+
+              if [[ "${{ inputs.exclude-slow }}" == "true" ]]; then
+                  EXCLUDE_DIRS="$EXCLUDE_DIRS $SLOW_EXCLUSIONS"
+              fi
+
+              # Trim leading or trailing spaces and save the exclude-dirs in the environment
+              EXCLUDE_DIRS=$(echo $EXCLUDE_DIRS | xargs)
+              echo "exclude-dirs=$EXCLUDE_DIRS" >> $GITHUB_OUTPUT

.github/actions/get_test_infos/README.md

@@ -1,16 +1,3 @@
-## Prerequisite
-
-This action assumes that the repository has already been checked out before
-calling the action, typically using `actions/checkout@v4`. If you have not
-checked out the code in a previous step, make sure to do so to avoid errors.
-
-This action does **not** perform a checkout action itself because it would be
-redundant. This action is part of the repository’s codebase, so if the code
-hasn’t already been checked out, the action itself wouldn't even be available to
-call. Additionally, rerunning a checkout at this stage could potentially
-overwrite any earlier `actions/checkout` step with different parameters, such as
-checking out a specific branch.
-
 ## Example Usage
 
 ```yaml

.github/actions/set_run_conditions/README.md

@@ -1,16 +1,3 @@
-## Prerequisite
-
-This action assumes that the repository has already been checked out before
-calling the action, typically using `actions/checkout@v4`. If you have not
-checked out the code in a previous step, make sure to do so to avoid errors.
-
-This action does **not** perform a checkout action itself because it would be
-redundant. This action is part of the repository's codebase, so if the code
-hasn't already been checked out, the action itself wouldn't even be available to
-call. Additionally, rerunning a checkout at this stage could potentially
-overwrite any earlier `actions/checkout` step with different parameters, such as
-checking out a specific branch.
-
 ## Example Usage
 
 ```yaml

.github/actions/setup_dfx/README.md

@@ -1,16 +1,3 @@
-## Prerequisite
-
-This action assumes that the repository has already been checked out before
-calling the action, typically using `actions/checkout@v4`. If you have not
-checked out the code in a previous step, make sure to do so to avoid errors.
-
-This action does **not** perform a checkout action itself because it would be
-redundant. This action is part of the repository’s codebase, so if the code
-hasn’t already been checked out, the action itself wouldn't even be available to
-call. Additionally, rerunning a checkout at this stage could potentially
-overwrite any earlier `actions/checkout` step with different parameters, such as
-checking out a specific branch.
-
 ## Example Usage
 
 ```yaml

.github/actions/setup_node/README.md

@@ -1,16 +1,3 @@
-## Prerequisite
-
-This action assumes that the repository has already been checked out before
-calling the action, typically using `actions/checkout@v4`. If you have not
-checked out the code in a previous step, make sure to do so to avoid errors.
-
-This action does **not** perform a checkout action itself because it would be
-redundant. This action is part of the repository’s codebase, so if the code
-hasn’t already been checked out, the action itself wouldn't even be available to
-call. Additionally, rerunning a checkout at this stage could potentially
-overwrite any earlier `actions/checkout` step with different parameters, such as
-checking out a specific branch.
-
 ## Example Usage
 
 ```yaml