Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: Allow flexible Rusk binary building in workflow #3231

Merged
merged 3 commits into from
Dec 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 84 additions & 0 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# Workflows Overview

This directory contains the GitHub Actions workflows that automate various processes in our Rusk monorepo. Below is a detailed description of each workflow, its purpose, and its key components.

## Table of Contents
1. [General Notes](#general-notes)
2. [Workflow Files](#workflow-files)
3. [Conventions](#conventions)
4. [Adding or Modifying Workflows](#adding-or-modifying-workflows)
5. [Troubleshooting](#troubleshooting)
6. [Common Problems](#common-problems)

## General Notes
- These workflows handle tasks like CI, benchmarks, building binaries, and more.
- Workflows are triggered by various events, such as `push`, `pull_request`, or manually via `workflow_dispatch`.
- Reusable actions and patterns, such as `dorny/paths-filter` for change detection and `actions/checkout`, are heavily utilized to standardize processes.
- We heavily rely on self-hosted runners, available through `runs-on: core`. These runners are stateful. It's only recommended to use other runners if you need stronger consistenty guarantees, as they're slower.
- Workflows like `rusk_build.yml` and `ruskwallet_build.yml` use matrices for multi-OS and multi-feature builds. Thesse ensure compatibility across multiple operating systems and architectures.
- Outputs like binaries and Docker images are stored as artifacts for download and reuse.

## Workflow Files
### [benchmarks.yml](./binary_copy.yml)
**Purpose**: Runs benchmarks for `rusk` and `node` components, and uploads the results as an artifact.
**Trigger**: `push` to the `master` branch.

### [binary_copy.yml](./binary_copy.yml)
**Purpose**: Builds and copies the `rusk` binary to a host directory on the runner.
**Trigger**: `push` to the `master` branch.

### [docker_image_build.yml](./docker_image_build.yml)
**Purpose**: Builds a Docker image and uploads it as an artifact.
**Trigger**: `workflow_dispatch` (manual trigger).

### [explorer_ci.yml](./explorer_ci.yml)
**Purpose**: CI for the `explorer`. Lints, tests, typechecks and builds it.
**Trigger**: `pull_request` events.

### [profile_ci.yml](./profile_ci.yml)
**Purpose**: Generates proving keys using `make keys`.
**Trigger**: `workflow_dispatch`.

### [rusk_build.yml](./rusk_build.yml)
**Purpose**: Compiles `rusk` binaries for multiple operating systems and architectures. Packages binaries with their corresponding version and features.
**Trigger**: `workflow_dispatch`.

### [rusk_ci.yml](./rusk_ci.yml)
**Purpose**: CI for the `rusk` repository. Executes formatting, linting, and tests.
**Trigger**: `pull_request` events.

### [ruskwallet_build.yml](./ruskwallet_build.yml)
**Purpose**: Compiles `rusk-wallet` binaries for multiple OSes and architectures. Packages and uploads the artifacts
**Trigger**: `workflow_dispatch`.

### [ruskwallet_ci.yml](./ruskwallet_ci.yml)
**Purpose**: CI for the `rusk-wallet` repository, with specific nightly tests for multiple platforms.
**Trigger**: `pull_request` events.

### [w3sperjs_ci.yml](./w3sperjs_ci.yml)
**Purpose**: CI for `w3sper.js`, executing linting and test tasks.
**Trigger**: `pull_request` and `workflow_dispatch`.

### [webwallet_ci.yml](./webwallet_ci.yml)
**Purpose**: CI for the `web-wallet`, executing lints, tests, typechecks and builds the app.
**Trigger**: `pull_request` events.

## Adding or Modifying Workflows
1. Create a new `.yml` file in this directory.
2. Use a descriptive `name` for the workflow.
3. Document the workflow in this README.
4. Follow existing patterns for consistency.
5. Test the workflow thoroughly before merging.

## Troubleshooting
### General Debugging
Use the GitHub Actions logs to investigate failures. Checking the jobs and collapsing the runs often provide a lot of output information on versions or filters used. Add `set -x` or debug specific commands to problematic steps to gather more information.

### Change Detection Issues
Verify the path patterns in `dorny/paths-filter`. Make sure the `filters` section includes all relevant paths. You can check the `changes` job in a workflow run and check the `dorny/paths-filter` run.

### Matrix Build Failures
Check compatibility for the target platform or flags. Make sure the appropriate Rust targets, Node or Deno versions are installed. For Rust, you can check the `dsherret/rust-toolchain-file` run in a workflow run for the installer release.

## Common Problems
It often occurs that CI reports `action.yml`/`action.yaml`/`Dockerfile` are not found. This is often a false-positive where the post-run fails due to a prior failure. Investigate the workflow run by check if the earlier steps report any other issue.
70 changes: 60 additions & 10 deletions .github/workflows/rusk_build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,44 @@ on:
description: "Git branch, ref, or SHA to checkout"
required: true
default: "master"
runner:
description: "Choose runner target to build against (JSON array)"
required: true
default: "[\"ubuntu-24.04\", \"macos-15\", \"arm-linux\"]"
features:
description: "Choose features to build (JSON array)"
required: true
default: "[\"default\", \"archive\", \"prover\"]"

jobs:
config:
runs-on: ubuntu-latest
name: Show configuration
outputs:
archs: ${{steps.final.outputs.archs}}
steps:
- name: get archs
run: |
echo "# Parameters" | tee -a $GITHUB_STEP_SUMMARY
echo | tee -a $GITHUB_STEP_SUMMARY
echo "features = $FEATURES" | tee -a $GITHUB_STEP_SUMMARY
echo | tee -a $GITHUB_STEP_SUMMARY
echo "runner = $RUNNER" | tee -a $GITHUB_STEP_SUMMARY
env:
FEATURES: ${{github.event.inputs.features}}
RUNNER: ${{github.event.inputs.runner}}

build_and_publish:
name: Build Rusk binaries for ${{ matrix.os }} (${{ matrix.features }})
name: Build Rusk binaries for ${{ matrix.os }} (${{ matrix.feature }})
runs-on: ${{ matrix.os }}
needs:
- config
continue-on-error: ${{ !contains(fromJson(github.event.inputs.runner), matrix.os) }}
strategy:
matrix:
os: [ubuntu-24.04, macos-15, arm-linux]
feature: ${{ fromJson(github.event.inputs.features) }}
compiler: [cargo]
features: [default, archive]
os: [ubuntu-24.04, macos-15, arm-linux]
include:
- os: ubuntu-24.04
target: linux-x64
Expand All @@ -26,8 +54,17 @@ jobs:
- os: arm-linux
target: linux-arm64
flags: --target=aarch64-unknown-linux-gnu
fail-fast: false

steps:
- name: Skip Non-matching Configurations
if: |
!contains(fromJson(github.event.inputs.runner), matrix.os) ||
!contains(fromJson(github.event.inputs.features), matrix.feature)
run: |
echo "Skipping build for ${{ matrix.os }} - ${{ matrix.feature }}"
exit 1

- name: Checkout Repository
uses: actions/checkout@v4
with:
Expand All @@ -40,18 +77,31 @@ jobs:
run: rustup target add aarch64-apple-darwin
if: ${{ matrix.os == 'macos-15' }}

- name: Check for Prover Features
id: config
run: |
echo "Checking feature requirements..."
if [[ "${{ matrix.feature }}" == *"prover"* ]]; then
echo "NO_DEFAULT_FEATURES=--no-default-features" >> $GITHUB_ENV
echo "SKIP_WASM=true" >> $GITHUB_ENV
else
echo "NO_DEFAULT_FEATURES=" >> $GITHUB_ENV
echo "SKIP_WASM=false" >> $GITHUB_ENV
fi

- name: Compile keys
shell: bash
run: make keys

- name: Compile WASM Contracts
if: ${{ env.SKIP_WASM != 'true' }}
shell: bash
run: make wasm

- name: Build Rusk binary
shell: bash
working-directory: ./rusk
run: cargo build --release --features "${{ matrix.features }}" ${{ matrix.flags }}
run: cargo build --release ${{ env.NO_DEFAULT_FEATURES }} --features "${{ matrix.feature }}" ${{ matrix.flags }}

- name: Extract Version
run: |
Expand All @@ -60,14 +110,14 @@ jobs:

- name: Package Binaries
run: |
mkdir rusk-${{ env.SEMVER }}-${{ matrix.target }}-${{ matrix.features }}
mv target/release/rusk rusk-${{ env.SEMVER }}-${{ matrix.target }}-${{ matrix.features }}
tar -czvf rusk-${{ env.SEMVER }}-${{ matrix.target }}-${{ matrix.features }}.tar.gz \
rusk-${{ env.SEMVER }}-${{ matrix.target }}-${{ matrix.features }}
mkdir rusk-${{ env.SEMVER }}-${{ matrix.target }}-${{ matrix.feature }}
mv target/release/rusk rusk-${{ env.SEMVER }}-${{ matrix.target }}-${{ matrix.feature }}
tar -czvf rusk-${{ env.SEMVER }}-${{ matrix.target }}-${{ matrix.feature }}.tar.gz \
rusk-${{ env.SEMVER }}-${{ matrix.target }}-${{ matrix.feature }}

- name: Upload Binaries as Artifacts
uses: actions/upload-artifact@v4
with:
name: rusk-${{ env.SEMVER }}-${{ matrix.target }}-${{ matrix.features }}
path: ./rusk-${{ env.SEMVER }}-${{ matrix.target }}-${{ matrix.features }}.tar.gz
name: rusk-${{ env.SEMVER }}-${{ matrix.target }}-${{ matrix.feature }}
path: ./rusk-${{ env.SEMVER }}-${{ matrix.target }}-${{ matrix.feature }}.tar.gz
retention-days: 5
Loading