Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wasm binary deploy process #1866

Merged
merged 40 commits into from
Jul 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
b094de1
first successful manual wasm binary manipulation!
lastmjs Jul 5, 2024
675b4c2
fix linters and prettiers
lastmjs Jul 5, 2024
3dd4d22
fix linters and prettiers
lastmjs Jul 5, 2024
fd30a6f
start working towards cleaning up unnecessary compilation code
lastmjs Jul 8, 2024
6a022d0
major cleanup of podman/docker
lastmjs Jul 8, 2024
21c12dd
manual binary manipulation working very well for all canister method …
lastmjs Jul 8, 2024
64aa180
add environment vars, get rid of regular assets option
lastmjs Jul 8, 2024
df12971
Merge branch 'main' of github.com:demergent-labs/azle into wasm_binar…
lastmjs Jul 9, 2024
6837c91
Merge branch 'main' of github.com:demergent-labs/azle into wasm_binar…
lastmjs Jul 9, 2024
b70f439
let us see if this fixes a lot of things
lastmjs Jul 10, 2024
259c06b
add return types
lastmjs Jul 10, 2024
b44b8e4
replace all occurrences of ts-node with tsx
lastmjs Jul 10, 2024
6c376b1
remove --skip-project
lastmjs Jul 10, 2024
667cdbe
add http canister optimization back in
lastmjs Jul 10, 2024
a2741e3
make many things asynchronous
lastmjs Jul 10, 2024
e3b8969
fix large files
lastmjs Jul 10, 2024
add4317
fix whoami example
lastmjs Jul 10, 2024
f209000
fix tfjs
lastmjs Jul 10, 2024
52bd8cb
implement indexes for class-based syntax
lastmjs Jul 10, 2024
11aa9c9
simplify the test.yml
lastmjs Jul 10, 2024
9fbd164
fix autoreload, compile reload_js into the binary
lastmjs Jul 10, 2024
bd29296
introduce concept of WasmData, get open value sharing to work
lastmjs Jul 11, 2024
e632060
add management canister did automatically
lastmjs Jul 11, 2024
3e555d8
manually upload the icrc files
lastmjs Jul 11, 2024
5415d8f
add better error handling to heartbeat, inspect_message, and pre_upgr…
lastmjs Jul 11, 2024
5bc965e
add panic hooks to init and post upgrade
lastmjs Jul 11, 2024
8c2a908
ckbtc fixed
lastmjs Jul 11, 2024
d7966e0
rip out all guard function stuff
lastmjs Jul 11, 2024
dae659e
fix the robust imports example
lastmjs Jul 12, 2024
98040d0
update binary
lastmjs Jul 12, 2024
d5916b3
get rid of trailing comma
lastmjs Jul 12, 2024
fb190d1
get rid of canister_methods
lastmjs Jul 12, 2024
39563c2
lots of cleanup
lastmjs Jul 12, 2024
c7ef94d
clean up fix candid
lastmjs Jul 12, 2024
6bb02ca
fix path issues
lastmjs Jul 12, 2024
4cc5def
add .npmignore
lastmjs Jul 12, 2024
efbc0f2
get rid of thirdparty.yml
lastmjs Jul 12, 2024
f53d603
misc cleanup
lastmjs Jul 12, 2024
3e41d42
add init/post_upgrade comment
lastmjs Jul 12, 2024
df7002a
cleanup
lastmjs Jul 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
1 change: 1 addition & 0 deletions .eslintignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,4 @@ dfx_generated
declarations
property_tests/tests/**/src/*
dist
static_canister_template_licenses.yml
File renamed without changes.
51 changes: 3 additions & 48 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ jobs:
with:
ref: ${{ contains(github.head_ref, 'release--') && github.event.pull_request.head.ref || github.ref }} # This is necessary for this job to be able to commit and push to the origin remote properly
token: ${{ secrets.LASTMJS_GITHUB_TOKEN || github.token }} # A personal GitHub token is setup as a secret so that committing and pushing to GitHub from the Action will trigger another workflow
- uses: actions/setup-node@v3
- uses: actions/setup-node@v4
with:
node-version: 20
registry-url: https://registry.npmjs.org
Expand All @@ -63,8 +63,6 @@ jobs:
run: echo -n "$GPG_SIGNING_KEY" | base64 --decode | gpg --import
- if: contains(github.head_ref, 'release--')
run: git config --global user.signingkey C8B77BCBE16CD2B94B43F9C8757397B82D4ED7B0
- if: contains(github.head_ref, 'release--')
run: sudo apt-get install -y podman
- id: example_directories
# TODO to improve this further we might be able to create an environment variable that grabs the example directories with a glob
# TODO we want to be able to easily include and exclude examples though
Expand Down Expand Up @@ -103,7 +101,6 @@ jobs:
"examples/file_protocol",
"examples/fs",
"examples/func_types",
"examples/guard_functions",
"examples/heartbeat",
"examples/hello_world",
"examples/http_outcall_fetch",
Expand Down Expand Up @@ -218,7 +215,6 @@ jobs:
"tests/end_to_end/candid_rpc/class_syntax/date",
"tests/end_to_end/candid_rpc/class_syntax/ethereum_json_rpc",
"tests/end_to_end/candid_rpc/class_syntax/func_types",
"tests/end_to_end/candid_rpc/class_syntax/guard_functions",
"tests/end_to_end/candid_rpc/class_syntax/heartbeat",
"tests/end_to_end/candid_rpc/class_syntax/ic_api",
"tests/end_to_end/candid_rpc/class_syntax/icrc",
Expand Down Expand Up @@ -263,7 +259,7 @@ jobs:
"tests/end_to_end/candid_rpc/class_syntax/stable_structures",
"tests/end_to_end/candid_rpc/class_syntax/timers",
"tests/end_to_end/candid_rpc/class_syntax/tuple_types",
"tests/end_to_end/candid_rpc/class_syntax/update",
"tests/end_to_end/candid_rpc/class_syntax/update"
]
END
)
Expand Down Expand Up @@ -310,40 +306,18 @@ jobs:
- if: ${{ needs.release-candidate-deploy.outputs.should_run_tests }}
uses: actions/checkout@v4
- if: ${{ needs.release-candidate-deploy.outputs.should_run_tests }}
uses: actions/setup-node@v3
uses: actions/setup-node@v4
with:
node-version: 20
- if: ${{ needs.release-candidate-deploy.outputs.should_run_tests }}
name: Install dfx
run: |
DFXVM_INIT_YES=true DFX_VERSION=0.20.1 sh -ci "$(curl -fsSL https://sdk.dfinity.org/install.sh)"
echo "$HOME/.local/share/dfx/bin" >> $GITHUB_PATH
- if: ${{ needs.release-candidate-deploy.outputs.should_run_tests }}
uses: actions/cache@v3
with:
path: /home/runner/.config/azle
# path: $HOME/.config/azle
key: config-azle-${{ hashFiles('src/compiler/Dockerfile', 'src/compiler/rust/**') }}
# This is for the --native-compilation tests
- if: ${{ needs.release-candidate-deploy.outputs.should_run_tests }}
name: Install Rust
shell: bash -l {0}
run: |
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain=1.73.0 --profile=minimal
rustup target add wasm32-wasi
# This is for the --native-compilation tests
- if: ${{ needs.release-candidate-deploy.outputs.should_run_tests }}
name: Install wasi2ic
shell: bash -l {0}
run: cargo install --git https://github.com/wasm-forge/wasi2ic --rev 806c3558aad24224852a9582f018178402cb3679
- if: ${{ needs.release-candidate-deploy.outputs.should_run_tests && matrix.os == 'ubuntu-latest' }}
shell: bash -l {0}
run: sudo apt-get install -y podman
- if: ${{ needs.release-candidate-deploy.outputs.should_run_tests && matrix.os == 'macos-latest' }}
shell: bash -l {0}
# The DNS server stuff is because of this: https://github.com/actions/runner-images/issues/6383
run: |
brew install podman
sudo networksetup -setdnsservers Ethernet 9.9.9.9
- if: ${{ needs.release-candidate-deploy.outputs.should_run_tests }}
shell: bash -l {0} # TODO figure out why this is here and comment about it
Expand All @@ -353,16 +327,6 @@ jobs:
run: npm link
- if: ${{ needs.release-candidate-deploy.outputs.should_run_tests && (matrix.azle_source == 'npm' || matrix.os == 'macos-latest') }}
run: npx azle clean # We want to test the full curl download when testing from a clean install or mac
# This is for the --native-compilation tests
- if: ${{ needs.release-candidate-deploy.outputs.should_run_tests }}
name: Install wasmedge-quickjs
shell: bash -l {0}
run: |
mkdir -p /home/runner/.config/azle
cd /home/runner/.config/azle
git clone https://github.com/demergent-labs/wasmedge-quickjs
cd wasmedge-quickjs
git checkout c21ff69f442998e4cda4619166e23a9bc91418be
- if: ${{ needs.release-candidate-deploy.outputs.should_run_tests }}
shell: bash -l {0}
run: npm run lint
Expand All @@ -380,11 +344,6 @@ jobs:
- if: ${{ needs.release-candidate-deploy.outputs.should_run_tests }}
working-directory: ${{ matrix.example_directories }}
run: dfx start --clean --background --host 127.0.0.1:8000
# This is to put wasmedge-quickjs in the correct location for the --native-compilation tests
- if: ${{ needs.release-candidate-deploy.outputs.should_run_tests }}
shell: bash -l {0}
working-directory: ${{ matrix.example_directories }}
run: mv /home/runner/.config/azle/wasmedge-quickjs "/home/runner/.config/azle/wasmedge-quickjs_$(npx azle dockerfile-hash)"
- if: ${{ needs.release-candidate-deploy.outputs.should_run_tests && !contains(github.head_ref, 'release--') && !(github.ref == 'refs/heads/main' && contains(github.event.head_commit.message, 'Merge pull request') && contains(github.event.head_commit.message, 'demergent-labs/release--')) }}
shell: bash -l {0}
working-directory: ${{ matrix.example_directories }}
Expand All @@ -397,10 +356,6 @@ jobs:
shell: bash -l {0}
working-directory: ${{ matrix.example_directories }}
run: AZLE_PROPTEST_NUM_RUNS=100 AZLE_PROPTEST_VERBOSE=true npm test
- if: ${{ needs.release-candidate-deploy.outputs.should_run_tests && matrix.example_directories != 'examples/new' }}
shell: bash -l {0}
working-directory: ${{ matrix.example_directories }}
run: $GITHUB_WORKSPACE/node_modules/.bin/tsc --noEmit --skipLibCheck --target es2020 --strict --moduleResolution node --allowJs

check-basic-integration-tests-success:
needs: basic-integration-tests
Expand Down
25 changes: 25 additions & 0 deletions .npmignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Ignore everything
*

# Whitelist
!canisters/**
!dfx/**
!dfx_extension/**
!experimental/**
!http_client/**
!scripts/**
!src/**
!test/**
!type_tests/**
!.openvaluesharing.json
!bin.mjs
!install_script.sh
!LICENSE
!LICENSE_EXTENSION.md
!NOTICE
!package-lock.json
!package.json
!README.md
!server.did
!static_canister_template_licenses.yml
!static_canister_template.wasm
1 change: 1 addition & 0 deletions .prettierignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,4 @@ node_modules
target
/the_azle_book/book
/docs
static_canister_template_licenses.yml
32 changes: 0 additions & 32 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 1 addition & 5 deletions Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,2 @@
[workspace]
members = [
"src/compiler/rust/canister",
"src/compiler/rust/canister_methods",
"src/compiler/rust/open_value_sharing",
]
members = ["src/compiler/rust/canister", "src/compiler/rust/open_value_sharing"]
38 changes: 0 additions & 38 deletions Dockerfile

This file was deleted.

12 changes: 0 additions & 12 deletions bin.js

This file was deleted.

4 changes: 4 additions & 0 deletions bin.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
#!/usr/bin/env node
lastmjs marked this conversation as resolved.
Show resolved Hide resolved

import 'tsx';
import('./src/compiler/index.ts');
9 changes: 9 additions & 0 deletions contributing/static_canister_template_licensing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
Because we are now distributing `static_canister_template.wasm` as a compiled binary, we must adhere to stricter licensing requirements, as we are now distributing a lot of software in binary form.

The first attempt to comply sits at `static_canister_template_licenses.yml`. You can generate the licenses anew with:

```bash
cargo bundle-licenses --format yaml --output static_canister_template_licenses.yml --previous static_canister_template_licenses.yml --check-previous
```

Search the file for `NOT FOUND`, and manually copy any licenses in from their repositories if necessary.
2 changes: 1 addition & 1 deletion examples/apollo_server/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"scripts": {
"pretest": "ts-node --transpile-only --ignore=false test/pretest.ts",
"pretest": "tsx test/pretest.ts",
"test": "jest"
},
"dependencies": {
Expand Down
2 changes: 1 addition & 1 deletion examples/async_await/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "async_await_end_to_end_test_functional_syntax",
"scripts": {
"pre_tests": "ts-node --transpile-only --ignore=false test/pretest.ts",
"pre_tests": "tsx test/pretest.ts",
"tests": "npm run pre_tests && jest",
"test": "AZLE_TEST_FETCH=false npm run tests && AZLE_TEST_FETCH=true npm run tests"
},
Expand Down
2 changes: 1 addition & 1 deletion examples/audio_and_video/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"scripts": {
"build": "cd src/frontend && vite build",
"pretest": "ts-node --transpile-only --ignore=false --skip-project test/pretest.ts",
"pretest": "tsx test/pretest.ts",
"test": "jest"
},
"dependencies": {
Expand Down
2 changes: 1 addition & 1 deletion examples/audio_recorder/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "audio_recorder_end_to_end_test_functional_syntax",
"scripts": {
"pretest": "ts-node --transpile-only --ignore=false test/pretest.ts",
"pretest": "tsx test/pretest.ts",
"test": "jest"
},
"dependencies": {
Expand Down
2 changes: 1 addition & 1 deletion examples/autoreload/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"scripts": {
"pretest": "ts-node --transpile-only --ignore=false test/pretest.ts",
"pretest": "tsx test/pretest.ts",
"test": "jest"
},
"dependencies": {
Expand Down
2 changes: 1 addition & 1 deletion examples/basic_bitcoin/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"name": "basic_bitcoin",
"scripts": {
"install": "./scripts/install.sh",
"pretest": "ts-node --transpile-only --ignore=false test/pretest.ts",
"pretest": "tsx test/pretest.ts",
"test": "jest"
},
"dependencies": {
Expand Down
2 changes: 1 addition & 1 deletion examples/bitcoin/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"bitcoin-cli": "bitcoin-cli -conf=$(pwd)/.bitcoin.conf -datadir=$(pwd)/.bitcoin/data -rpcport=18443",
"deploy": "./scripts/deploy.sh bitcoin",
"install": "./scripts/install.sh",
"pretest": "ts-node --transpile-only --ignore=false test/pretest.ts",
"pretest": "tsx test/pretest.ts",
"test": "jest"
},
"dependencies": {
Expand Down
4 changes: 2 additions & 2 deletions examples/bitcoin_psbt/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,10 @@
"getrawmempool": ".bitcoin/bin/bitcoin-cli -conf=$(pwd)/.bitcoin.conf getrawmempool true",
"getrawtransaction": ".bitcoin/bin/bitcoin-cli -conf=$(pwd)/.bitcoin.conf getrawtransaction $npm_config_txid",
"help": ".bitcoin/bin/bitcoin-cli -conf=$(pwd)/.bitcoin.conf help",
"premanualtest": "ts-node --transpile-only --ignore=false test/pretest.ts",
"premanualtest": "tsx test/pretest.ts",
"manualtest": "jest test/manual_test.ts test/manual.test.ts",
"install": "./scripts/install.sh",
"pretest": "ts-node --transpile-only --ignore=false test/pretest.ts",
"pretest": "tsx test/pretest.ts",
"test": "jest test/test.ts"
},
"dependencies": {
Expand Down
2 changes: 1 addition & 1 deletion examples/bitcoinjs_lib/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"scripts": {
"pretest": "ts-node --transpile-only --ignore=false test/pretest.ts",
"pretest": "tsx test/pretest.ts",
"test": "jest"
},
"dependencies": {
Expand Down
2 changes: 1 addition & 1 deletion examples/bitcore_lib/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "bitcore_lib_example",
"scripts": {
"pretest": "ts-node --transpile-only --ignore=false test/pretest.ts",
"pretest": "tsx test/pretest.ts",
"test": "jest"
},
"dependencies": {
Expand Down
2 changes: 1 addition & 1 deletion examples/blob_array/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "blob_array_end_to_end_test_functional_syntax",
"scripts": {
"pretest": "ts-node --transpile-only --ignore=false test/pretest.ts",
"pretest": "tsx test/pretest.ts",
"test": "jest"
},
"dependencies": {
Expand Down
Loading
Loading