Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prepare pipeline #19

Merged
merged 10 commits into from
Oct 3, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 0 additions & 49 deletions .github/workflows/pipeline.yml

This file was deleted.

6 changes: 6 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
[submodule "souper"]
path = souper
url = https://github.com/KTH/souper.git
[submodule "wabt"]
path = wabt
url = git://github.com/WebAssembly/wabt.git
[submodule "binaryen"]
path = binaryen
url = git://github.com/WebAssembly/binaryen.git
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
1 change: 1 addition & 0 deletions binaryen
Submodule binaryen added at fc6d2d
80 changes: 80 additions & 0 deletions utils/pipeline/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# pipeline

### Workflow

1. __`.wast`__ → __`.wasm`__ (by `wat2wasm` from [`wabt`](https://github.com/WebAssembly/wabt))

`wat2wasm xxx.wat -o xxx.wasm`

2. __`.wasm`__ → __`.opt`__ (by `wasm-opt` from [`binaryen`](https://github.com/WebAssembly/binaryen))

`wasm-opt --flatten --souperify xxx.wasm`

3. __`.opt`__ → __`.ll`__ (based on `utils/souper2llvm.in` from [`souper`](https://github.com/google/souper))

4. __`.ll`__ → __`.bc`__ (by `llvm-as` from [`llvm`](https://llvm.org/docs/index.html))

`llvm-as xxx.ll`

5. __`.bc`__ → __`.opt`__ (based on `souper` from [`souper`](https://github.com/google/souper)) → __`.ll`__ (same as step3)

----

if we need assembly filetype:
__`.ll`__ → __`.s`__ (by `llc` from [`llvm`](https://llvm.org/docs/index.html))
```
$ llc -march=wasm32 -filetype=asm xxx.ll
```
or we need binary filetype:
__`.ll`__ → __`.o`__ (by `llc` from [`llvm`](https://llvm.org/docs/index.html))
```
$ llc -march=wasm32 -filetype=obj xxx.ll
```

### Demo
```
sh pipeline.sh demo/2.wast
sh pipeline.sh demo/infer.opt
```

### Build

__souper__

manually install `re2c` `z3`

```
cd souper
./build_deps.sh
mkdir build
cd build
cmake ..
make
```

__wabt__

```
cd wabt
git submodule update --init
mkdir build
cd build
cmake ..
cmake --build .
```

__binaryen__

```
cmake . && make
```

### Requirements

Python >= 3.6

LLVM >= 8.0

### Note
1. current solution for __`.opt`__ → __`.ll`__ is not so reliable
2. for reference, my env is `llv=9.0.0`, `gcc=9.2.0`, `cmake=3.15.3`
57 changes: 57 additions & 0 deletions utils/pipeline/bc2opt2ll.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
import os
import sys
import subprocess


def split_candidates(file_path, prefix):
with open(file_path, 'r') as file:
lines = file.readlines()
lines.append('\n')

num = 1
buffer = list()
file_names = list()
for line in lines:
if line is '\n':
if len(buffer) > 0 and buffer[0].startswith('; Static profile'):
file_name = f'{prefix}.{num}'
file_names.append(file_name)
with open(f'{file_name}.opt', 'w') as file:
file.writelines(buffer)
num += 1
buffer = list()
else:
buffer.append(line)
return file_names


def batch_opt2ll(file_names):
for file_name in file_names:
try:
result = subprocess.run(
f'python3 souper2llvm.py {file_name}.opt > {file_name}.ll',
check=True,
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
# print(result.stdout)
# print(result.stderr)
except Exception as e:
print(e)


def main():
try:
file_path = sys.argv[1]
dir_name = os.path.dirname(file_path)
base_name = os.path.basename(file_path)
file_name, file_ext = os.path.splitext(base_name)
prefix = dir_name + os.path.sep + file_name
file_names = split_candidates(file_path, prefix)
batch_opt2ll(file_names)
except Exception as e:
print(e)


main()
19 changes: 19 additions & 0 deletions utils/pipeline/bc2opt2ll.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
#!/bin/sh

name=$(echo $1 | sed 's/\.[^.]*$//')
# ext=$(echo $1 | sed 's/^.*\.//')
souper=$2

# echo "invoke souper ..."
if [ "$(uname)"=="Darwin" ]; then
# for macOS
${souper} ${name}.bc -z3-path=/usr/local/Cellar/z3/4.8.6/bin/z3 > ${name}.candidates
else
# for ubuntu
${souper} ${name}.bc -z3-path=/usr/bin/z3 > ${name}.candidates
fi

# echo "handle results ..."
python3 bc2opt2ll.py ${name}.candidates

rm ${name}.candidates
Empty file added utils/pipeline/demo/12.opt
Empty file.
Binary file added utils/pipeline/demo/2.bc
Binary file not shown.
8 changes: 8 additions & 0 deletions utils/pipeline/demo/2.ll
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"


define i32 @foo(i32 %x1) {
entry:
%0 = mul i32 2, %x1
ret i32 %0
}
Binary file added utils/pipeline/demo/2.o
Binary file not shown.
8 changes: 8 additions & 0 deletions utils/pipeline/demo/2.opt
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@

; function: $0

; start LHS (in $0)
%0:i32 = var
%1 = mul 2:i32, %0
infer %1

17 changes: 17 additions & 0 deletions utils/pipeline/demo/2.s
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
.text
.file "2.ll"
.section .text.foo,"",@
.globl foo # -- Begin function foo
.type foo,@function
foo: # @foo
.functype foo (i32) -> (i32)
# %bb.0: # %entry
local.get 0
i32.const 1
i32.shl
# fallthrough-return-value
end_function
.Lfunc_end0:
.size foo, .Lfunc_end0-foo
# -- End function

Binary file added utils/pipeline/demo/2.wasm
Binary file not shown.
24 changes: 24 additions & 0 deletions utils/pipeline/demo/2.wast
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
(module

;;(2.0 * x / size) - 1.5

;;Subtree size 7

(func $Booiqoeeoi (param i32) (param i32) (result f32)
i32.const 2

;;x
get_local 0
i32.mul
f32.convert_s/i32

;;size
get_local 1
f32.convert_s/i32
f32.div
f32.const 1.5
f32.sub

)
(export "Booiqoeeoi" (func $Booiqoeeoi))
)
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"


define i1 @foo(i1 %x1, i64 %x2) {
define None @foo(i1 %x1, i64 %x2) {
entry:
%0 = icmp ult i64 135637824071393761, %x2
%1 = or i1 %x1, %0
%2 = select i1 %1, i64 135637824071393761, i64 %x2
%3 = icmp ult i64 135637824071393761, %2
ret i1 %3
}
ret None %3
}
Binary file not shown.
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ entry:
%2 = select i1 %1, i64 135637824071393761, i64 %x2
%3 = icmp ult i64 135637824071393761, %2
ret i1 %3
}
}
File renamed without changes.
File renamed without changes.
27 changes: 27 additions & 0 deletions utils/pipeline/demo/infer.s
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
.text
.file "infer.ll"
.section .text.foo,"",@
.globl foo # -- Begin function foo
.type foo,@function
foo: # @foo
.functype foo (i32, i64) -> (i32)
# %bb.0: # %entry
i64.const 135637824071393761
i64.const 135637824071393761
local.get 1
local.get 1
i64.const 135637824071393761
i64.gt_u
i64.select
local.get 0
i32.const 1
i32.and
i64.select
i64.const 135637824071393761
i64.gt_u
# fallthrough-return-value
end_function
.Lfunc_end0:
.size foo, .Lfunc_end0-foo
# -- End function

35 changes: 35 additions & 0 deletions utils/pipeline/pipeline.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
#!/bin/sh

name=$(echo $1 | sed 's/\.[^.]*$//')
ext=$(echo $1 | sed 's/^.*\.//')

if [ "${ext}" == "wast" ]; then
echo "### step1 wast2wasm \c"
../../wabt/bin/wat2wasm ${name}.wast -o ${name}.wasm
echo "okay"

echo "### step2 wasm2opt \c"
../../binaryen/bin/wasm-opt ${name}.wasm --flatten --souperify > ${name}.opt
echo "okay"
fi

echo "### step3 opt2ll \c"
python3 souper2llvm.py ${name}.opt > ${name}.ll
# ../../souper/build/souper2llvm ${name}.opt > ${name}.ll # it seems not so good
echo "okay"

echo "### step4 ll2bc \c"
llvm-as ${name}.ll
echo "okay"

echo "### step5 bc2opt2ll \c"
sh bc2opt2ll.sh ${name}.bc ../../souper/build/souper
echo "okay"

echo "### extra ll2s \c"
llc -march=wasm32 -filetype=asm ${name}.ll
echo "okay"

echo "### extra ll2o \c"
llc -march=wasm32 -filetype=obj ${name}.ll
echo "okay"
Original file line number Diff line number Diff line change
Expand Up @@ -117,10 +117,11 @@ def parseInst(i):
elif i[0][0] == "%":
assert len(i) >= 3, f"wrong instruction length {len(i)}, {i}"
tmp = i[0].split(":")
assert len(tmp) == 2, f"wrong reg length {len(tmp)}, {tmp}"
# assert len(tmp) == 2, f"wrong reg length {len(tmp)}, {tmp}"
assert i[1] == "=", f"expecting '=', got {i[1]}"
reg = tmp[0]
width = tmp[1]
# FIXME (this is one quick patch)
width = tmp[1] if len(tmp) == 2 else "i32"
inst = i[2]
if "overflow" in i[2]:
width = f"i{int(width[1:]) - 1}"
Expand Down Expand Up @@ -475,12 +476,11 @@ def main():
res.extend(genPCs())
res.extend(genLHSFuncFooter())

with open(f'{dir_name}/{file_name}.ll', 'w') as f:
f.write('\n'.join(res))
print("\n".join(res))


main()

# based on souper/utils/souper2llvm.in (Sep 13, 2019)
# based on souper/utils/souper2llvm.in (Sept. 13, 2019)
# todo sync latest version
# todo fix compatibility issue with llc (like negate.opt)
# todo fix possible issues
Loading