Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sanity check #2091

Closed
wants to merge 72 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
5ecc634
put service tests in the right place
bdemann Aug 26, 2024
c30880a
try setting name
bdemann Aug 26, 2024
b683952
first pass
bdemann Aug 27, 2024
a7f3854
limit to npm packages
bdemann Aug 27, 2024
4ae2ebd
restrict to npm packages
bdemann Aug 27, 2024
b7e5ed1
add exclusions
bdemann Aug 27, 2024
6657880
alphabetize
bdemann Aug 27, 2024
181eabc
update to json
bdemann Aug 27, 2024
0a48161
limit to only packages that have tests
bdemann Aug 27, 2024
9322394
test new auto generated tests
bdemann Aug 27, 2024
7e2adee
hook into real tests
bdemann Aug 28, 2024
f78eccb
update publish-github-actions
bdemann Aug 28, 2024
174bb65
move should run tests to it's own job
bdemann Aug 28, 2024
27dce50
clean up
bdemann Aug 28, 2024
21748da
make num runs a variable
bdemann Aug 28, 2024
fd4b9c0
join similar command into one job
bdemann Aug 28, 2024
5f1de50
move generate-tests script
bdemann Aug 28, 2024
57031d1
update to variables
bdemann Aug 28, 2024
e03f5ac
add report of full path of tests just incase it isn't obvious from th…
bdemann Aug 28, 2024
45273a6
update yml to use new outputs
bdemann Aug 28, 2024
8127bd4
move out conditionals to top levels
bdemann Aug 28, 2024
1b24bb6
get rid of artificial delay for feature prs
bdemann Aug 29, 2024
2c51bba
move versions to env variables
bdemann Aug 29, 2024
e76c4fe
create custom action for finding test dirs
bdemann Aug 29, 2024
2434e2a
first pass at build job
bdemann Aug 29, 2024
6a1993c
move build to it's own workflow
bdemann Aug 29, 2024
35954b9
file clean up
bdemann Aug 29, 2024
45c87d7
try object filtering
bdemann Aug 29, 2024
3593238
pr fixes
bdemann Aug 29, 2024
2a2d8c0
pr fixes
bdemann Aug 29, 2024
006a615
better wording in action
bdemann Aug 29, 2024
96a37f5
naming updates
bdemann Aug 29, 2024
d95051f
get rid of unused var
bdemann Aug 29, 2024
fe2c5a6
pr fixes
bdemann Aug 29, 2024
3ff7ef1
make sure empty all_directories leads to empty result
bdemann Aug 30, 2024
c410511
fix capitalization
bdemann Aug 30, 2024
c304651
rename generate_tests
bdemann Aug 30, 2024
b4e9f2c
variable clean up
bdemann Aug 30, 2024
8e3dac1
test commit message
bdemann Aug 30, 2024
c64698c
Merge branch 'main' into update_github_workflows
bdemann Aug 30, 2024
e86a157
pr fixes
bdemann Aug 30, 2024
3772934
rename build workflow
bdemann Aug 30, 2024
e758108
rename to release candidate deploy
bdemann Aug 30, 2024
a98b9ca
move should release to it's own action
bdemann Aug 30, 2024
4fc4061
clean up and stabilize should run action
bdemann Aug 30, 2024
e86715e
pr fixes
bdemann Aug 30, 2024
bc5b666
simplify logic for num runs
bdemann Aug 30, 2024
0e319a4
move configurable options to the top of the env
bdemann Aug 30, 2024
50f898c
update env logic
bdemann Aug 30, 2024
34c93b7
fix logic
bdemann Aug 29, 2024
0378e9b
Merge branch 'main' into update_github_workflows
bdemann Sep 5, 2024
ffdbf70
add ref to checkout
bdemann Sep 5, 2024
660984d
checkout outputs
bdemann Sep 5, 2024
7a6828f
use javascript for json generation
bdemann Sep 6, 2024
6346ba4
only run get_tests if the tests or release should run
bdemann Sep 6, 2024
fa7a066
Merge branch 'main' into update_github_workflows
bdemann Sep 6, 2024
6ab44e4
Merge branch 'main' into update_github_workflows
bdemann Sep 11, 2024
a248d31
build the binaries on release
bdemann Sep 11, 2024
a18a0a1
Merge branch 'main' into update_github_workflows
bdemann Sep 12, 2024
4d8c1b9
make sure rust is installed for release
bdemann Sep 12, 2024
facf4f5
update path to install rust script
bdemann Sep 12, 2024
4124ad0
add installation for was2ic
bdemann Sep 12, 2024
bf92722
add wasm32-wasi target
bdemann Sep 12, 2024
99b7216
better logging
bdemann Sep 12, 2024
b5cf3bc
rename install script
bdemann Sep 12, 2024
a1a7cab
update to be more immutable
bdemann Sep 12, 2024
338c6ab
update logic
bdemann Sep 12, 2024
ef1c12f
pr fixes
bdemann Sep 12, 2024
29221ef
change contains to startsWith were appropriate
bdemann Sep 12, 2024
2115e21
a fix for jordan
bdemann Sep 12, 2024
f48d26d
pr fix
bdemann Sep 12, 2024
ecae95f
this seems a little sus
bdemann Sep 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions .github/actions/get_tests/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
name: Get tests
description: Gets a list of directories of npm projects with an npm test script
inputs:
directories:
description: List of directories to search for npm projects with an npm test script
required: true
exclude_dirs:
description: List of directories to exclude from the search
required: false
default: ''
node_version:
description: The version of Node.js to use
required: true
default: '20.x'
outputs:
tests:
description: All of the tests found by this action
value: ${{ steps.get_tests.outputs.tests }}
runs:
using: composite
steps:
- uses: actions/checkout@v4

- uses: actions/setup-node@v4
with:
node-version: ${{ inputs.node_version }}

- name: Get tests
id: get_tests
run: |
# Export the input variables to make them available to the script
export INPUT_DIRECTORIES="${{ inputs.directories }}"
export INPUT_EXCLUDE_DIRS="${{ inputs.exclude_dirs }}"
# Run the script
tests=$(./.github/actions/get_tests/get_tests.sh | base64 -d)
echo "tests=${tests}" >> "$GITHUB_OUTPUT"
shell: bash
56 changes: 56 additions & 0 deletions .github/actions/get_tests/generate_json.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
import { basename } from 'path';

// Function to check if a directory is excluded
function isExcluded(dir, excludeDirs) {
return excludeDirs.some((exclude) => dir.includes(exclude));
}

// Function to generate JSON object for each directory
function generateJson(dir) {
const name = basename(dir);
const type = getType(dir);
const syntax = getSyntax(dir);
const api = getApi(dir);

return {
path: dir,
name: name,
type: type,
...(syntax != '' && { syntax }),
...(api != '' && { api })
};
}
// Helper functions to determine type, syntax, and API
function getType(dir) {
if (dir.includes('/examples/')) return 'ex';
if (dir.includes('/end_to_end/')) return 'e2e';
if (dir.includes('/property/')) return 'prop';
return '';
}

function getSyntax(dir) {
if (dir.includes('/candid_rpc/')) return 'crpc';
if (dir.includes('/http_server/')) return 'http';
if (dir.includes('/ic_api/')) return 'ic_api';
return '';
}

function getApi(dir) {
const functionalPattern = /\/functional(?:_api|_syntax)?\//;
const classPattern = /\/class(?:_api|_syntax)?\//;
if (functionalPattern.test(dir)) return 'functional';
if (classPattern.test(dir)) return 'class';
return '';
}

// Get input from the command-line arguments
const directories = process.argv[2].split(' ').filter((path) => path !== '');
const excludeDirs = process.argv[3].split(',').filter((path) => path !== '');

// Filter directories and generate JSON
const jsonArray = directories
.filter((dir) => dir && !isExcluded(dir, excludeDirs))
.map((dir) => generateJson(dir));

// Output the final JSON string
console.log(JSON.stringify(jsonArray));
41 changes: 41 additions & 0 deletions .github/actions/get_tests/get_tests.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
#!/bin/bash

# Inputs from action.yml or environment variables
DIRECTORIES=($INPUT_DIRECTORIES)
EXCLUDE_DIRS=($INPUT_EXCLUDE_DIRS)

# Function to discover test directories
discover_directories() {
local dir=$1
find "$dir" -type d -not -path "*/node_modules/*" -exec sh -c '
for pkg in "$@"; do
if [ -f "$pkg/package.json" ]; then
if jq -e ".scripts.test" "$pkg/package.json" > /dev/null; then
echo "$pkg"
fi
fi
done
' sh {} +
}

# Discover directories in the provided directories, excluding specified directories
all_directories=""
for dir in "${DIRECTORIES[@]}"; do
all_directories+=$(discover_directories "$dir")
all_directories+=$'\n'
done

# Prepare the data to be passed into the JavaScript script
sorted_directories=$(echo "$all_directories" | sort | tr '\n' ' ') # Convert into a space-separated string
exclude_dirs=$(IFS=, ; echo "${EXCLUDE_DIRS[*]}") # Convert array to comma-separated

# Call the JavaScript script with input directories and excluded directories
json_result=$(node .github/actions/get_tests/generate_json.js "$sorted_directories" "$exclude_dirs")

# Format the result
result="${json_result//'%'/'%25'}"
result="${result//$'\n'/'%0A'}"
result="${result//$'\r'/'%0D'}"

# Output the final result as base64
echo "$result" | base64
27 changes: 27 additions & 0 deletions .github/actions/should_release/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
name: Should release
description: Determines if the current pull request is for testing or for starting a release
outputs:
should_release:
description: Returns true if this branch should start a release, otherwise false
value: ${{ steps.determine_should_release.outputs.should_release }}
runs:
using: composite
steps:
- uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.ref }} # This is necessary for this job to be able to get the correct commit message from `git log`

- id: determine_should_release
run: |
BRANCH_NAME="${{ github.head_ref }}"
RELEASE_VERSION="${BRANCH_NAME:9}"
COMMIT_MESSAGE=$(git log -1 --pretty=format:"%s")
IS_RELEASE_BRANCH_PR="${{ startsWith(github.head_ref, 'release--') }}"

SHOULD_RELEASE="false"
if [[ $IS_RELEASE_BRANCH_PR == "true" && "$COMMIT_MESSAGE" == "release--$RELEASE_VERSION" ]]; then
SHOULD_RELEASE="true"
fi

echo "should_release=${SHOULD_RELEASE}" >> "$GITHUB_OUTPUT"
shell: bash
42 changes: 42 additions & 0 deletions .github/scripts/install_global_dependencies.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
#!/bin/bash

# File path for the global dependencies
GLOBAL_DEPENDENCIES_FILE="$PWD/global_dependencies"

# Extract the rustc version from the file
RUST_VERSION=$(grep 'rustc version:' "$GLOBAL_DEPENDENCIES_FILE" | awk '{print $4}' | cut -d' ' -f1)

if [[ -z "$RUST_VERSION" ]]; then
echo "Rust version not found in $GLOBAL_DEPENDENCIES_FILE"
exit 1
fi

# Install Rust using rustup with the extracted version
echo "Installing Rust version $RUST_VERSION"
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain "$RUST_VERSION"
source $HOME/.cargo/env

rustup target add wasm32-wasi

# Extract the wasi2ic version and repository URL
WASI2IC_LINE=$(grep 'wasi2ic version:' "$GLOBAL_DEPENDENCIES_FILE")
WASI2IC_VERSION=$(echo "$WASI2IC_LINE" | awk -F' ' '{print $4}')
WASI2IC_URL=$(echo "$WASI2IC_LINE" | grep -oP '(?<=\().+?(?=\))')

if [[ -z "$WASI2IC_VERSION" || -z "$WASI2IC_URL" ]]; then
echo "wasi2ic version or URL not found in $GLOBAL_DEPENDENCIES_FILE"
exit 1
fi

# Install wasi2ic
echo "Installing wasi2ic version $WASI2IC_VERSION from $WASI2IC_URL"
cargo install --git "$WASI2IC_URL"

# Confirm installation
echo "\nThe following global dependencies were installed"
rustc --version
cargo --version

rustup target list --installed | grep wasm32-wasi

cargo install --list | grep wasi2ic
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,17 @@ VERSION=$1

directories_json_string_with_linebreaks=$2
directories_json_string="${directories_json_string_with_linebreaks//$'\\n'/''}"
directories=$(echo "$directories_json_string" | jq -c -r '.[]')
directories=$(echo "$directories_json_string" | jq -c -r '.[] | .path')

sed -E -i "s/(\"version\": \")(.*)(\")/\1$VERSION\3/" package.json
sed -E -i "s/(\"version\": \")(.*)(\")/\1$VERSION\3/" dfx_extension/extension.json
# TODO we need to keep the dependencies.json file up-to-date as well
npm install

# Build the binary templates
npx azle template
npx azle template --experimental

if [[ "$VERSION" == *"-rc."* ]];
then
npm publish --tag next
Expand Down
89 changes: 89 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
name: Release
on:
push:
branches:
- main
pull_request: # Runs on pull requests to any branch
env:
DFX_VERSION: 0.21.0
NODE_VERSION: 20
jobs:
determine-should-release:
if: ${{ startsWith(github.head_ref, 'release--') }}
name: Determine if this branch should release
runs-on: ubuntu-latest
outputs:
should_release: ${{ steps.determine_should_release.outputs.should_release }}
steps:
- uses: actions/checkout@v4

- id: determine_should_release
uses: ./.github/actions/should_release

get-tests:
needs: determine-should-release
if: ${{ startsWith(github.head_ref, 'release--') && needs.determine-should-release.outputs.should_release }}
name: Get tests
runs-on: ubuntu-latest
outputs:
tests: ${{ steps.get_tests.outputs.tests }}
steps:
- uses: actions/checkout@v4

- name: Get all test dirs
id: get_tests
uses: ./.github/actions/get_tests
with:
directories: |
./examples
./tests

release:
name: Deploy release
# Only run this job if it's a release branch. This job will run instead of run-tests and will automatically publish another commit which will be tested
if: ${{ needs.determine-should-release.outputs.should_release == 'true' }}
needs:
- determine-should-release
- get-tests
runs-on: ubuntu-latest
env:
GPG_SIGNING_KEY: ${{ secrets.GPG_SIGNING_KEY }} # All commits must be verified
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.ref || github.ref }}
token: ${{ secrets.LASTMJS_GITHUB_TOKEN || github.token }}

- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
registry-url: https://registry.npmjs.org
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

- name: Install curl (if necessary)
run: sudo apt-get install curl -y

- name: Install Rust from global_dependencies
run: ./.github/scripts/install_global_dependencies.sh

- name: Install dfx
run: |
DFXVM_INIT_YES=true DFX_VERSION=${{ env.DFX_VERSION }} sh -ci "$(curl -fsSL https://sdk.dfinity.org/install.sh)"
echo "$HOME/.local/share/dfx/bin" >> $GITHUB_PATH

# TODO we should use some Action-specific bot account
- name: Configure git for publishing release
run: |
git config --global user.name 'Jordan Last'
git config --global user.email '[email protected]'
git config --global commit.gpgsign true
echo -n "$GPG_SIGNING_KEY" | base64 --decode | gpg --import
git config --global user.signingkey C8B77BCBE16CD2B94B43F9C8757397B82D4ED7B0

- name: Publish release
run: |
BRANCH_NAME="${{ github.head_ref }}"
RELEASE_VERSION="${BRANCH_NAME:9}"
./.github/scripts/publish_github_action.sh $RELEASE_VERSION ${{ toJSON(needs.get-tests.outputs.tests) }}
Loading
Loading