Use tinygo to build wasm

[rolsonquadras]

Currently, the wasm size is around 8.3MB and its built with go bin. We can look to use tinygo inorder to reduce the wasm size.

agent-sdk/cmd/agent-js-worker/scripts/build_assets.sh

Line 8 in ee6d0d4

GOOS=js GOARCH=wasm go build -o src/agent-js-worker.wasm main.go

[rolsonquadras]

1. document the compile errors with tinygo
2. try to find work around (build tinygo locally with reflect packcage changes)

[victro]

This is the initial issue with the official tinygo:

$ tinygo build -o src/agent-js-worker.wasm -target wasm
# google.golang.org/protobuf/internal/descfmt
../../../../../../go/pkg/mod/google.golang.org/[email protected]/internal/descfmt/stringer.go:44:31: reflect.ValueOf(vs).MethodByName undefined (type reflect.Value has no field or method MethodByName)
../../../../../../go/pkg/mod/google.golang.org/[email protected]/internal/descfmt/stringer.go:93:29: reflect.ValueOf(vs).MethodByName undefined (type reflect.Value has no field or method MethodByName)
../../../../../../go/pkg/mod/google.golang.org/[email protected]/internal/descfmt/stringer.go:124:11: rv.MethodByName undefined (type reflect.Value has no field or method MethodByName)
../../../../../../go/pkg/mod/google.golang.org/[email protected]/internal/descfmt/stringer.go:194:9: rv.MethodByName undefined (type reflect.Value has no field or method MethodByName)
../../../../../../go/pkg/mod/google.golang.org/[email protected]/internal/descfmt/stringer.go:209:13: v.MethodByName undefined (type reflect.Value has no field or method MethodByName)
../../../../../../go/pkg/mod/google.golang.org/[email protected]/internal/descfmt/stringer.go:219:12: rv.MethodByName undefined (type reflect.Value has no field or method MethodByName)
$ tinygo version
tinygo version 0.22.0 darwin/amd64 (using go version go1.17.6 and LLVM version 13.0.0)

[victro]

Trying with the source code for tinygo I made some changes to make those errors disappear, but those didin't work just changing the tinygo build because some new errors appeared, since tinygo doesn't support os/user as shown here https://tinygo.org/docs/reference/lang-support/stdlib/.

~/development/securekey/fork/tinygo/build/tinygo version 
tinygo version 0.23.0-dev-a7a69d38 darwin/amd64 (using go version go1.17.6 and LLVM version 13.0.0)
$ ~/development/securekey/fork/tinygo/build/tinygo build -o src/agent-js-worker.wasm -target wasm 
# net
~/development/securekey/fork/tinygo/src/net/tcpsock.go:28:16: undeclared name: DefaultResolver
~/development/securekey/fork/tinygo/src/net/tcpsock.go:28:49: undeclared name: context

[victro]

Adding -ldflags "-s -w" reduced dist directory from 8.3MB to 8MB

[rolsonquadras]

Adding -ldflags "-s -w" reduced dist directory from 8.3MB to 8MB

@victro this is the compressed file size. Can you add uncompressed wasm size as well ?

[victro]

I am attaching some documentation of the steps that I did and the current state of the errors for this ticket
Steps.pdf

[victro]

<style> </style>

Tinygo

The first step was to add placeholders for functions not implemented in the following files:

 

modified:   loader/goroot.go

    modified:   src/net/ipsock.go

    modified:   src/net/net.go

    modified:   src/net/tcpsock.go

    modified:   src/os/errors.go

    modified:   src/os/file_unix.go

    modified:   src/os/file_windows.go

    modified:   src/reflect/type.go

    modified:   src/reflect/value.go

    modified:   src/runtime/extern.go

    modified:   src/runtime/symtab.go

 

And add a new package “user” for with the following content:

 

package user

 

func lookupUser(username string) (*User, error) {

    panic("unimplemented: User.lookupUser()")

}

 

type User struct {

    // Uid is the user ID.

    // On POSIX systems, this is a decimal number representing the uid.

    // On Windows, this is a security identifier (SID) in a string format.

    // On Plan 9, this is the contents of /dev/user.

    Uid string

    // Gid is the primary group ID.

    // On POSIX systems, this is a decimal number representing the gid.

    // On Windows, this is a SID in a string format.

    // On Plan 9, this is the contents of /dev/user.

    Gid string

    // Username is the login name.

    Username string

    // Name is the user's real or display name.

    // It might be blank.

    // On POSIX systems, this is the first (or only) entry in the GECOS field

    // list.

    // On Windows, this is the user's display name.

    // On Plan 9, this is the contents of /dev/user.

    Name string

    // HomeDir is the path to the user's home directory (if they have one).

    HomeDir string

}

 

func Current() (*User, error) {

    panic("unimplemented: user.Current()")

}

 

I had to change a readonly file just to avoid having the issue of a int conversion error and the file is:

 

Changes in ../../../../../go/pkg/mod/github.com/google/tink/[email protected]/aead/subtle/aes_gcm.go

 

 

And had some issues with os/user that were bypassed by changing the following content in

tinygo/loader/goroot.go

// The boolean indicates whether to merge the subdirs. True means merge, false

// means use the TinyGo version.

func pathsToOverride(needsSyscallPackage bool) map[string]bool {

    paths := map[string]bool{

        "":                      true,

        "crypto/":               true,

        "crypto/rand/":          false,

        "device/":               false,

        "examples/":             false,

        "internal/":             true,

        "internal/bytealg/":     false,

        "internal/reflectlite/": false,

        "internal/task/":        false,

        "machine/":              false,

        "net/":                  true,

        "os/":                   true,

        "os/user/":              true,

        "reflect/":              false,

        "runtime/":              false,

        "sync/":                 true,

        "testing/":              true,

    }

    if needsSyscallPackage {

        paths["syscall/"] = true // include syscall/js

    }

    return paths

}

 

 

After that make again and the final operation I am working on right now is passing the following issue:

 

../../../../../../go/pkg/mod/github.com/golang/[email protected]/glog_file.go:118:7: Symlink not declared by package os

 

 

This is being called from tinygo/loader/goroot.go

 

 

 

GCCGO

 

Run all the following commands inside of a linux container in docker to have all the dependencies installedL:

 

apt install -y software-properties-common wget git gcc-11

add-apt-repository -y ppa:ubuntu-toolchain-r/test

add-apt-repository -y pya:ubuntu-toolchain-r/test

update-alternatives --remove-all gcc

update-alternatives --remove-all g++

 

update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 10

update-alternatives --install /usr/bin/gccgo gccgo /usr/bin/gccgo-11 10

update-alternatives --install /usr/bin/gccgo gccgo /usr/bin/gccgo-9 20

 

update-alternatives --set cc /usr/bin/gcc

 

 

wget https://go.dev/dl/go1.17.6.linux-amd64.tar.gz

rm -rf /usr/local/go && tar -C /usr/local -xzf go1.17.6.linux-amd64.tar.gz

export PATH=$PATH:/usr/local/go/bin

 

 

The current error at the moment is:

/root/go/pkg/mod/github.com/hyperledger/aries-framework-go/component/storage/[email protected]/indexeddb.go:19:2: package syscall/js is not in GOROOT (/usr/local/go/src/syscall/js)

 

Tinygo The first step was to add placeholders for functions not implemented in the following files:

modified: loader/goroot.go
modified: src/net/ipsock.go
modified: src/net/net.go
modified: src/net/tcpsock.go
modified: src/os/errors.go
modified: src/os/file_unix.go
modified: src/os/file_windows.go
modified: src/reflect/type.go
modified: src/reflect/value.go
modified: src/runtime/extern.go
modified: src/runtime/symtab.go

And add a new package “user” for with the following content:

package user

func lookupUser(username string) (*User, error) {
panic("unimplemented: User.lookupUser()")
}

type User struct {
// Uid is the user ID.
// On POSIX systems, this is a decimal number representing the uid.
// On Windows, this is a security identifier (SID) in a string format.
// On Plan 9, this is the contents of /dev/user.
Uid string
// Gid is the primary group ID.
// On POSIX systems, this is a decimal number representing the gid.
// On Windows, this is a SID in a string format.
// On Plan 9, this is the contents of /dev/user.
Gid string
// Username is the login name.
Username string
// Name is the user's real or display name.
// It might be blank.
// On POSIX systems, this is the first (or only) entry in the GECOS field
// list.
// On Windows, this is the user's display name.
// On Plan 9, this is the contents of /dev/user.
Name string
// HomeDir is the path to the user's home directory (if they have one).
HomeDir string
}

func Current() (*User, error) {
panic("unimplemented: user.Current()")
}

I had to change a readonly file just to avoid having the issue of a int conversion error and the file is:

Changes in ../../../../../go/pkg/mod/github.com/google/tink/[email protected]/aead/subtle/aes_gcm.go

And had some issues with os/user that were bypassed by changing the following content in
tinygo/loader/goroot.go
// The boolean indicates whether to merge the subdirs. True means merge, false
// means use the TinyGo version.
func pathsToOverride(needsSyscallPackage bool) map[string]bool {
paths := map[string]bool{
"": true,
"crypto/": true,
"crypto/rand/": false,
"device/": false,
"examples/": false,
"internal/": true,
"internal/bytealg/": false,
"internal/reflectlite/": false,
"internal/task/": false,
"machine/": false,
"net/": true,
"os/": true,
"os/user/": true,
"reflect/": false,
"runtime/": false,
"sync/": true,
"testing/": true,
}
if needsSyscallPackage {
paths["syscall/"] = true // include syscall/js
}
return paths
}

After that make again and the final operation I am working on right now is passing the following issue:

gopkg.in/yaml.v3

/Users/victor.rada/development/securekey/fork/tinygo/src/reflect/type.go:376:15: (reflect.Type).Elem() called on map type
%0 = call i32 @"(reflect.rawType).elem"(i32 %t, i8* undef), !dbg !17187

traceback:
/Users/victor.rada/development/securekey/fork/tinygo/src/reflect/type.go:376:15:
%0 = call i32 @"(reflect.rawType).elem"(i32 %t, i8* undef), !dbg !17187
/Users/victor.rada/development/securekey/fork/tinygo/src/reflect/type.go:375:18:
%ret = call %runtime._interface @"(reflect.rawType).Elem"(i32 %unpack.int, i8* %1), !dbg !17183
/Users/victor.rada/go/pkg/mod/gopkg.in/[email protected]:
%15 = call %runtime._interface @"interface:{Align:func:{}{basic:int},AssignableTo:func:{named:reflect.Type}{basic:bool},Bits:func:{}{basic:int},ChanDir:func:{}{named:reflect.ChanDir},Comparable:func:{}{basic:bool},ConvertibleTo:func:{named:reflect.Type}{basic:bool},Elem:func:{}{named:reflect.Type},Field:func:{basic:int}{named:reflect.StructField},FieldAlign:func:{}{basic:int},FieldByIndex:func:{slice:basic:int}{named:reflect.StructField},FieldByName:func:{basic:string}{named:reflect.StructField,basic:bool},Implements:func:{named:reflect.Type}{basic:bool},In:func:{basic:int}{named:reflect.Type},IsVariadic:func:{}{basic:bool},Key:func:{}{named:reflect.Type},Kind:func:{}{named:reflect.Kind},Len:func:{}{basic:int},MethodByName:func:{basic:string}{named:reflect.Method,basic:bool},Name:func:{}{basic:string},NumField:func:{}{basic:int},NumIn:func:{}{basic:int},NumMethod:func:{}{basic:int},NumOut:func:{}{basic:int},Out:func:{basic:int}{named:reflect.Type},PkgPath:func:{}{basic:string},Size:func:{}{basic:uintptr},String:func:{}{basic:string}}.Elem$invoke"(i8* %14, i32 %invoke.func.typecode, i8* undef), !dbg !17184

This is being called from tinygo/loader/goroot.go

GCCGO

Run all the following commands inside of a linux container in docker to have all the dependencies installedL:

apt install -y software-properties-common wget git gcc-11
add-apt-repository -y ppa:ubuntu-toolchain-r/test
add-apt-repository -y pya:ubuntu-toolchain-r/test
update-alternatives --remove-all gcc
update-alternatives --remove-all g++

update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 10
update-alternatives --install /usr/bin/gccgo gccgo /usr/bin/gccgo-11 10
update-alternatives --install /usr/bin/gccgo gccgo /usr/bin/gccgo-9 20

update-alternatives --set cc /usr/bin/gcc

wget https://go.dev/dl/go1.17.6.linux-amd64.tar.gz
rm -rf /usr/local/go && tar -C /usr/local -xzf go1.17.6.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin

The current error at the moment is:
/root/go/pkg/mod/github.com/hyperledger/aries-framework-go/component/storage/[email protected]/indexeddb.go:19:2: package syscall/js is not in GOROOT (/usr/local/go/src/syscall/js)

[victro]

Adding -ldflags "-s -w" reduced dist directory from 8.3MB to 8MB

@victro this is the compressed file size. Can you add uncompressed wasm size as well ?

34MB

[victro]

The following bdd tests are failing with the tinygo build and working fine with the gc compiler:

1) "before all" hook for "Router is running on "http://0.0.0.0:10091,ws://0.0.0.0:10092" with controller "http://0.0.0.0:10093""
     DID-Exchange between an Edge Agent and a router
     Error: timout waiting for aries to initialize
    at test/didexchange/edgeagent_mediator.js:74:31

2) "after all" hook for "Edge Agent validates that the connection's state is 'completed'"
     DID-Exchange between an Edge Agent and a router
     TypeError: Cannot read properties of undefined (reading 'destroy')
    at Context.<anonymous> (test/didexchange/edgeagent_mediator.js:80:20)

3) "before all" hook for "Router is running on "http://0.0.0.0:10091,ws://0.0.0.0:10092" with controller "http://0.0.0.0:10093""
     DID-Exchange between two Edge Agents using the router
     Error: timout waiting for aries to initialize
    at node_modules/@hyperledger/aries-framework-go/dist/web/aries.js:1:18374

4) "after all" hook for "Alice Edge Agent validates that the connection's state is 'completed'"
     DID-Exchange between two Edge Agents using the router
     TypeError: Cannot read properties of undefined (reading 'destroy')
    at Context.<anonymous> (test/didexchange/edgeagent_mediator.js:113:26)

5) "before all" hook for "Router is running on "http://0.0.0.0:10091,ws://0.0.0.0:10092" with controller "http://0.0.0.0:10093""
     Registers multiple routers
     Error: timout waiting for aries to initialize
    at node_modules/@hyperledger/aries-framework-go/dist/web/aries.js:1:18374

6) "after all" hook for "Alice Edge Agent validates router`s connections"
     Registers multiple routers
     TypeError: Cannot read properties of undefined (reading 'destroy')
    at Context.<anonymous> (test/didexchange/edgeagent_mediator.js:202:26)

7) "before all" hook for "Alice constructs an out-of-band invitation"
     Outofband - New connection after Alice sends an ouf-of-band invitation to Bob wasm
     Error: timout waiting for aries to initialize
    at node_modules/@hyperledger/aries-framework-go/dist/web/aries.js:1:18374

8) "after all" hook for "Bob accepts the invitation and connects with Alice"
     Outofband - New connection after Alice sends an ouf-of-band invitation to Bob wasm
     TypeError: Cannot read properties of undefined (reading 'destroy')
    at Context.<anonymous> (test/outofband/outofband.js:41:27)

9) "before all" hook for "Bob and Carol have established connection with Alice"
     Introduce - Alice has Carol's public out-of-band invitation wasm
     Error: timout waiting for aries to initialize
    at node_modules/@hyperledger/aries-framework-go/dist/web/aries.js:1:18374

10) "after all" hook for "Bob wants to know Carol and sends introduce response with approve"
     Introduce - Alice has Carol's public out-of-band invitation wasm
     TypeError: destroy is not a function
    at Context.<anonymous> (test/introduce/introduce.js:268:15)

11) "before all" hook for "Bob and Carol have established connection with Alice"
     Introduce - Alice has Carol's public out-of-band invitation. The protocol starts with introduce request. wasm
     Error: timout waiting for aries to initialize
    at node_modules/@hyperledger/aries-framework-go/dist/web/aries.js:1:18374

12) "after all" hook for "Bob wants to know Carol and sends introduce response with approve"
     Introduce - Alice has Carol's public out-of-band invitation. The protocol starts with introduce request. wasm
     TypeError: destroy is not a function
    at Context.<anonymous> (test/introduce/introduce.js:210:15)

13) "before all" hook for "Bob and Carol have established connection with Alice"
     Introduce - Bob sends a response with approve and an out-of-band invitation. wasm
     Error: timout waiting for aries to initialize
    at node_modules/@hyperledger/aries-framework-go/dist/web/aries.js:1:18374

14) "after all" hook for "Carol wants to know Bob and sends introduce response with approve"
     Introduce - Bob sends a response with approve and an out-of-band invitation. wasm
     TypeError: destroy is not a function
    at Context.<anonymous> (test/introduce/introduce.js:142:15)

15) "before all" hook for "Bob and Carol have established connection with Alice"
     Introduce - Bob sends a response with approve and an out-of-band invitation. The protocol starts with introduce request wasm
     Error: timout waiting for aries to initialize
    at node_modules/@hyperledger/aries-framework-go/dist/web/aries.js:1:18374

16) "before all" hook for "Holder requests credential from the Issuer"
     Issue credential (v2) - The Holder begins with a request wasm
     Error: timout waiting for aries to initialize
    at node_modules/@hyperledger/aries-framework-go/dist/web/aries.js:1:18374

17) "after all" hook for "Checks credential"
     Issue credential (v2) - The Holder begins with a request wasm
     TypeError: Cannot read properties of undefined (reading 'destroy')
    at Context.<anonymous> (test/issuecredential/issuecredential.js:118:27)

18) "before all" hook for "Holder requests credential from the Issuer"
     Issue credential (v3) - The Holder begins with a request wasm
     Error: timout waiting for aries to initialize
    at node_modules/@hyperledger/aries-framework-go/dist/web/aries.js:1:18374

19) "after all" hook for "Checks credential"
     Issue credential (v3) - The Holder begins with a request wasm
     TypeError: Cannot read properties of undefined (reading 'destroy')
    at Context.<anonymous> (test/issuecredential/issuecredential.js:118:27)

20) "before all" hook for "Alice create key set"
     KMS Test wasm
     Error: timout waiting for aries to initialize
    at node_modules/@hyperledger/aries-framework-go/dist/web/aries.js:1:18374

21) "after all" hook for "Alice import p256 key"
     KMS Test wasm
     TypeError: Cannot read properties of undefined (reading 'destroy')
    at Context.<anonymous> (test/kms/kms.js:32:21)

22) "before all" hook for "[wasm] Alice imports extra JSON-LD contexts"
     JSON-LD API Test wasm
     Error: timout waiting for aries to initialize
    at node_modules/@hyperledger/aries-framework-go/dist/web/aries.js:1:18374

23) "after all" hook for "[wasm] Alice deletes remote context provider"
     JSON-LD API Test wasm
     TypeError: Cannot read properties of undefined (reading 'destroy')
    at Context.<anonymous> (test/ld/ld.js:51:21)

24) "before all" hook for "receiver registers basic message service"
     Basic Messaging
     Error: timout waiting for aries to initialize
    at node_modules/@hyperledger/aries-framework-go/dist/web/aries.js:1:18374

25) "after all" hook for "sender gets updated list of registered message services"
     Basic Messaging
     TypeError: Cannot read properties of undefined (reading 'destroy')
    at Context.<anonymous> (test/messaging/messaging.js:63:27)

26) "before all" hook for "Alice constructs an out-of-band invitation"
     Outofband - New connection after Alice sends an ouf-of-band invitation to Bob wasm
     Error: timout waiting for aries to initialize
    at node_modules/@hyperledger/aries-framework-go/dist/web/aries.js:1:18374

27) "after all" hook for "Bob accepts the invitation and connects with Alice"
     Outofband - New connection after Alice sends an ouf-of-band invitation to Bob wasm
     TypeError: Cannot read properties of undefined (reading 'destroy')
    at Context.<anonymous> (test/outofband/outofband.js:41:27)

28) "before all" hook for "Verifier sends a request presentation to the Prover"
     Present Proof (v2) - The Verifier begins with a request presentation wasm
     Error: timout waiting for aries to initialize
    at node_modules/@hyperledger/aries-framework-go/dist/web/aries.js:1:18374

29) "after all" hook for "Verifier checks presentation"
     Present Proof (v2) - The Verifier begins with a request presentation wasm
     TypeError: Cannot read properties of undefined (reading 'destroy')
    at Context.<anonymous> (test/presentproof/presentproof.js:321:25)

30) "before all" hook for "Verifier sends a request presentation to the Prover"
     Present Proof (v3) - The Verifier begins with a request presentation wasm
     Error: timout waiting for aries to initialize
    at node_modules/@hyperledger/aries-framework-go/dist/web/aries.js:1:18374

31) "after all" hook for "Verifier checks presentation"
     Present Proof (v3) - The Verifier begins with a request presentation wasm
     TypeError: Cannot read properties of undefined (reading 'destroy')
    at Context.<anonymous> (test/presentproof/presentproof.js:321:25)

32) "before all" hook for "Verifier sends a request presentation to the Prover"
     Present Proof (v2) - The Verifier begins with a request presentation (BBS+) wasm
     Error: timout waiting for aries to initialize
    at node_modules/@hyperledger/aries-framework-go/dist/web/aries.js:1:18374

33) "after all" hook for "Verifier checks presentation"
     Present Proof (v2) - The Verifier begins with a request presentation (BBS+) wasm
     TypeError: Cannot read properties of undefined (reading 'destroy')
    at Context.<anonymous> (test/presentproof/presentproof.js:156:25)

34) "before all" hook for "Verifier sends a request presentation to the Prover"
     Present Proof (v3) - The Verifier begins with a request presentation (BBS+) wasm
     Error: timout waiting for aries to initialize
    at node_modules/@hyperledger/aries-framework-go/dist/web/aries.js:1:18374

35) "after all" hook for "Verifier checks presentation"
     Present Proof (v3) - The Verifier begins with a request presentation (BBS+) wasm
     TypeError: Cannot read properties of undefined (reading 'destroy')
    at Context.<anonymous> (test/presentproof/presentproof.js:156:25)

36) "before all" hook for "Alice create peer did"
     VDR
     Error: timout waiting for aries to initialize
    at node_modules/@hyperledger/aries-framework-go/dist/web/aries.js:1:18374

37) "after all" hook for "Alice validates that she has the did"
     VDR
     TypeError: Cannot read properties of undefined (reading '0')
    at Context.<anonymous> (test/vdr/vdr.js:28:21)

38) "before all" hook for "[wasm] Alice stores the verifiable credential received from the college"
     Verifiable Store Test wasm
     Error: timout waiting for aries to initialize
    at node_modules/@hyperledger/aries-framework-go/dist/web/aries.js:1:18374

39) "after all" hook for "[wasm] Alice generates the signed  verifiable presentation to pass it to the employer"
     Verifiable Store Test wasm
     TypeError: Cannot read properties of undefined (reading 'destroy')
    at Context.<anonymous> (test/verifiable/verifiable.js:66:21)

[victro]

Changes in tinygo are in this commit:
victro/tinygo@20d5436

[victro]

At the moment after adding the runtime package manually to tinygo the current error to fix is:

# github.com/hyperledger/aries-framework-go/pkg/didcomm/protocol/didexchange.test
panic: runtime error: index out of range [2336751994] with length 9

[victro]

Putting this ticket on hold, the latest goal for the ticket was to update gjson that doesn't use reflect in latest version