Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recover High Level IR from Patchable IR #342

Draft
wants to merge 164 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
164 commits
Select commit Hold shift + click to select a range
f702e7c
first lift of a func
2over12 Nov 16, 2022
d853624
fix entry
2over12 Nov 17, 2022
4d4c77e
keep memory pointer consistent
2over12 Nov 17, 2022
8b65e82
add inlining
2over12 Nov 17, 2022
998acf0
Use per-block context assignments
frabert Nov 17, 2022
8732f1c
fix PC semantics
2over12 Nov 17, 2022
e18567a
added conditional interprocedural control flow
2over12 Nov 20, 2022
1d7fb45
add own call utilities for 4 arg functions
2over12 Nov 21, 2022
729752e
Pass declaration to `AddFunctionToContext`
frabert Nov 21, 2022
e18dc31
Move basic block funcs into new context
frabert Nov 21, 2022
07866fd
Remove unused passes
frabert Nov 21, 2022
7f4b015
delete old test case
2over12 Nov 21, 2022
e6a8377
dont try to relift basic blocks for decls:
2over12 Nov 21, 2022
13cf91b
Parse local variable decls
frabert Nov 21, 2022
8898b9d
add test
2over12 Nov 21, 2022
c5a37b4
add contexts
2over12 Nov 21, 2022
d7bcb86
add start of param pass
2over12 Nov 22, 2022
88fa7f1
add pass plans
2over12 Nov 22, 2022
d12d438
First attempt at recovering params/locals
frabert Nov 22, 2022
888b3ee
Keep track of register offsets?
frabert Nov 28, 2022
4943211
Revert "First attempt at recovering params/locals"
2over12 Nov 28, 2022
b33c574
working on transforms
2over12 Nov 28, 2022
d016dc8
fix cloning and attach gv to module
2over12 Nov 29, 2022
908c135
do sig pass before opt
2over12 Nov 29, 2022
7cd6bcd
update spec
2over12 Nov 29, 2022
c347322
fix ordering in basic block functions, need to fix parents
2over12 Nov 29, 2022
ffb4b64
fix iterator invalidation and referencing wrong state
2over12 Nov 29, 2022
f0315cc
fix func names
2over12 Nov 29, 2022
b56c3b8
insert offsets
2over12 Dec 3, 2022
1528b3c
Add API to obtain basic block functions' addresses
frabert Dec 5, 2022
d90e942
fix thumb mode
2over12 Dec 5, 2022
7f0f284
fix switch branch types
2over12 Dec 5, 2022
6a748cf
fix interproc control flow that is terminal in a basic block function
2over12 Dec 6, 2022
5daa0f3
add frame size
2over12 Dec 6, 2022
a000f38
separate out state ptr
2over12 Dec 6, 2022
ba4d71c
remove deadcode
2over12 Dec 6, 2022
f8c6142
shared functionality into codelifter
2over12 Dec 7, 2022
f6609a8
pulled decl info into context
2over12 Dec 7, 2022
d0d50c3
move construction to code lifter
2over12 Dec 7, 2022
67fae08
back to building
2over12 Dec 8, 2022
9568946
add caller to bblifter
2over12 Dec 8, 2022
ebeff62
added pack and unpack
2over12 Dec 8, 2022
73b03eb
fix memory issue
2over12 Dec 8, 2022
d90fb04
fix constructor
2over12 Dec 8, 2022
6891e44
add stack offsets to context
2over12 Dec 8, 2022
3c851f8
use same struct type
2over12 Dec 8, 2022
6ca8711
fix state ref
2over12 Dec 8, 2022
ca78a0a
mostly lifting... weird debug info
2over12 Dec 9, 2022
86a15b6
lifts again
2over12 Dec 9, 2022
476c98d
disable optimization until figure out what's going on
2over12 Dec 9, 2022
b5e3da8
Fix spec loading bug
frabert Dec 9, 2022
a0ee5a8
Convert type specs to LLVM metadata
frabert Dec 9, 2022
5cd4ef3
Annotate values with type metadata
frabert Dec 9, 2022
f4c96dd
symbolic init stack
2over12 Dec 9, 2022
42ec613
Begin add pointer lifting stuff
frabert Dec 9, 2022
bdf898f
First conversion heuristics
frabert Dec 11, 2022
d8802a6
remove old transform
2over12 Dec 11, 2022
0da8b48
Convert global loads to pointers
frabert Dec 12, 2022
754d06f
Report if there were any changes
frabert Dec 12, 2022
5ae55c8
Improve recognition of indices
frabert Dec 12, 2022
b4851cd
Recognize scaled index access
frabert Dec 12, 2022
a130f06
basic stack vars
2over12 Dec 12, 2022
ecc0586
remove old decomp stack
2over12 Dec 12, 2022
4997be3
cast
2over12 Dec 12, 2022
5513b8c
fix off by ones in stack
2over12 Dec 12, 2022
a93fb2e
add function lifting
2over12 Dec 13, 2022
824923a
replace calls
2over12 Dec 13, 2022
dfd2622
start add return pass
2over12 Dec 13, 2022
e0bf8f2
remove optimistic remove of function returns
2over12 Dec 14, 2022
bc5a8a7
uncomment some opts
2over12 Dec 14, 2022
e26c58e
add liveness to spec
2over12 Dec 15, 2022
068914e
live loads working
2over12 Dec 15, 2022
e63d4d1
save live registers at exit
2over12 Dec 15, 2022
601a003
Merge branch 'frabert/ptr-prop' into ian/low-level-function-lifts
frabert Dec 16, 2022
31c5570
Begin porting pointer lifter to low level lifter
frabert Dec 16, 2022
38c7f74
remove mac only header def
Ninja3047 Dec 16, 2022
a67f34a
start porting to split params
2over12 Dec 19, 2022
7be0fb5
finish pass params and vars directly
2over12 Dec 19, 2022
878effe
small memory pointer fix
2over12 Dec 20, 2022
75f21ed
fix stack pointer, also update stackp pointer replacement to referenc…
2over12 Dec 21, 2022
4910c2f
initilaize the program counter later
2over12 Dec 21, 2022
3c02941
check if have overlapping variable
2over12 Dec 21, 2022
1629fba
reset insert point
2over12 Dec 21, 2022
16de1c8
fix call function to insert SP at ir
2over12 Dec 21, 2022
32d8263
handle pointer typed stack var sizes
2over12 Dec 21, 2022
af1f626
enable avx by default
2over12 Dec 21, 2022
1ebd897
use datalayout size
2over12 Dec 21, 2022
3fe263c
update angha script
2over12 Dec 22, 2022
6d9eb56
bump tool ci
2over12 Dec 22, 2022
1df0ee5
refactor to lift all live variables (#341)
2over12 Jan 6, 2023
9a59b07
formatting
2over12 Jan 6, 2023
d440843
add abstract fallback stack
2over12 Jan 7, 2023
6c279f1
added falling back to abstract stack... some really odd pointers need…
2over12 Jan 7, 2023
0921d06
stack fixes
2over12 Jan 9, 2023
d3aa66a
add better logging
2over12 Jan 10, 2023
adb25be
add more logging
2over12 Jan 10, 2023
460e7c0
Use live exits vector instead of live entries again
tetsuo-cpp Jan 11, 2023
637ccfd
temporary no alias
2over12 Jan 11, 2023
45d919e
allow stack references to be resolved wrt to a base
2over12 Jan 12, 2023
0648c6f
Don't run the first set of function passes again
tetsuo-cpp Jan 12, 2023
c1bda73
fix lte for case where there is no variable greater than the current …
2over12 Jan 12, 2023
ca29ee6
Revert "Don't run the first set of function passes again"
2over12 Jan 12, 2023
06eaf86
adapt pointer to integer when needed
2over12 Jan 12, 2023
58224b7
Fix CI Build (#343)
2over12 Jan 30, 2023
ab8c130
Reference fixes (#344)
2over12 Feb 2, 2023
02c8766
PowerPC Support (#340)
tetsuo-cpp Feb 6, 2023
69e79b9
Stop unnecessarily parsing the semantic module for each basic block (…
tetsuo-cpp Feb 8, 2023
491541b
Update CI (#345)
Ninja3047 Feb 13, 2023
2b466fd
dont require lifter options to provide pointer from args (#350)
2over12 Feb 16, 2023
4a95869
Add global register support (#351)
Ninja3047 Feb 16, 2023
7d84a1c
Respect taint pc flag when lifting constant symvals (#355)
2over12 Feb 21, 2023
6493170
Adds a control flow simplification pass to support idiomatic block go…
2over12 Feb 22, 2023
7197615
Rework load lifted value and store native to operate over composite l…
2over12 Feb 24, 2023
59d317e
hack to force pack lives to occur before recovered anvill returns (#357)
2over12 Feb 24, 2023
8c27f41
adapt register types to address types as needed
2over12 Mar 12, 2023
23a2ee3
simple inlining
2over12 Mar 10, 2023
dd89a23
add inline option
2over12 Mar 12, 2023
b7dfc5b
add namespace to pass name
Ninja3047 Mar 17, 2023
e844855
Always enable statistics, collect anvill_sp stores/loads
Ninja3047 Mar 17, 2023
29fdbbd
make increment consistent with llvm
Ninja3047 Mar 17, 2023
7d68247
count references to anvill stack/pc by counting uses
Ninja3047 Mar 21, 2023
4a229d9
convert to module pass
Ninja3047 Mar 22, 2023
bd64694
count number of functions that contain stack/pc
Ninja3047 Mar 22, 2023
3ebbffb
simplify convoluted logic
Ninja3047 Mar 22, 2023
233cbd7
accumlate total for all modules
Ninja3047 Mar 22, 2023
512d0c2
Adds back splitting the stack around the return address preventing ex…
2over12 Mar 27, 2023
df58f86
Tail call control flow (#364)
frabert Mar 29, 2023
d6605c7
Ian/fix hash for typespecs (#367)
2over12 Mar 29, 2023
9fad6d2
make registers no capture (#368)
2over12 Apr 3, 2023
a52d812
Keep memory consistent through basic block calls and returns (#370)
2over12 Apr 7, 2023
6a68180
use irene ghidra install (#374)
2over12 May 4, 2023
5c16d8e
Update remill for anvill (#371)
2over12 May 4, 2023
1364c44
Swap AArch64 to use Sleigh Semantics (#377)
2over12 May 4, 2023
9f5518c
Split symvals into equalities at entry and exit (#375)
frabert May 4, 2023
046702e
bump remill again
2over12 May 10, 2023
5741f82
Performance fixes (#348)
tetsuo-cpp May 10, 2023
9426467
Get Anvill building with LLVM 16 (#376)
tetsuo-cpp May 19, 2023
8ebf563
Pointer lifter (#373)
frabert May 19, 2023
09cef2b
Add image base spec (#379)
Ninja3047 Jun 1, 2023
ba87273
Add missing `<utility>` header for `std::exchange` (#384)
tetsuo-cpp Jul 5, 2023
2772efc
reconstruct floats during a load (#385)
2over12 Jul 7, 2023
0a9dd3f
Add `required_globals` to spec (#386)
frabert Jul 10, 2023
9f3e123
Add `-j` flag to `test-amp-challenge-bins.sh` and update ignore list …
tetsuo-cpp Jul 11, 2023
465ecdc
Debug utils (#369)
Ninja3047 Aug 2, 2023
f7f498d
Remove challenge testing (#389)
tetsuo-cpp Aug 8, 2023
66bb39d
Use encoded type for constant value mappings (#388)
tetsuo-cpp Aug 16, 2023
e04ff9d
Ian/allow for typehints in spec (#390)
2over12 Sep 18, 2023
4413bfb
Revert "Debug utils (#369)" (#391)
tetsuo-cpp Sep 21, 2023
875fcd5
Bump `lifting-tools-ci` (#394)
tetsuo-cpp Sep 25, 2023
c3f8be4
Support lifting x87 80-bit float constants (#393)
tetsuo-cpp Sep 26, 2023
9722e02
Callsite override (#398)
Ninja3047 Oct 4, 2023
3a506a5
check if index is in range (#397)
2over12 Oct 11, 2023
91c7162
Add Angha50 test to CI (#392)
tetsuo-cpp Oct 12, 2023
610205a
UID codeblocks refactor (#396)
ekilmer Oct 12, 2023
befde9c
llvm 17 fixes (#395)
2over12 Oct 13, 2023
cadafba
Fix accesses to globals that occur in the middle of the global (#399)
2over12 Oct 24, 2023
0b5b329
fix typo oredered_locs => ordered_locs (#401)
Ninja3047 Oct 27, 2023
9cfd0d6
Sleigh sparc (#387)
Ninja3047 Nov 10, 2023
d569305
Fix infinite loop with self-referential global struct (#403)
ekilmer Nov 17, 2023
e8ca92c
Bump lifting-tools-ci to use clang-14 for SPARC (#404)
Ninja3047 Dec 5, 2023
70209a8
Basic pointer use heuristic (#402)
2over12 Dec 5, 2023
30969fb
bump remill and fix initial state pc (#407)
2over12 Feb 28, 2024
b47f275
return early if we cannot find the entry block in the cfg (#408)
Ninja3047 Feb 28, 2024
52f9638
Invalid func fix pt2 (#409)
Ninja3047 Feb 29, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
193 changes: 103 additions & 90 deletions .github/workflows/build.yml

Large diffs are not rendered by default.

3 changes: 2 additions & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ find_package(glog CONFIG REQUIRED)
find_package(Z3 CONFIG REQUIRED)
find_package(doctest CONFIG REQUIRED)
find_package(LLVM CONFIG REQUIRED)
include_directories(SYSTEM ${LLVM_INCLUDE_DIRS})
llvm_map_components_to_libnames(llvm_libs support core irreader bitreader bitwriter)

find_package(sleigh CONFIG)
Expand Down Expand Up @@ -56,7 +57,7 @@ if(ANVILL_ENABLE_INSTALL)
endif(ANVILL_ENABLE_INSTALL)

set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_EXTENSIONS OFF)

if(ANVILL_ENABLE_TESTS)
Expand Down
12 changes: 6 additions & 6 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
ARG LLVM_VERSION=15
ARG LLVM_VERSION=17
ARG ARCH=amd64
ARG UBUNTU_VERSION=20.04
ARG CXX_COMMON_VERSION=0.2.16
ARG UBUNTU_VERSION=22.04
ARG CXX_COMMON_VERSION=0.6.0
ARG DISTRO_BASE=ubuntu${UBUNTU_VERSION}
ARG BUILD_BASE=ubuntu:${UBUNTU_VERSION}
ARG LIBRARIES=/opt/trailofbits
Expand All @@ -15,7 +15,7 @@ ARG LLVM_VERSION
ARG CXX_COMMON_VERSION
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
apt-get install -qqy --no-install-recommends git libdbus-1-3 curl unzip python3 python3-pip python3.8 python3.8-venv python3-setuptools xz-utils cmake && \
apt-get install -qqy --no-install-recommends git libdbus-1-3 curl unzip python3 python3-pip python3-setuptools xz-utils cmake && \
rm -rf /var/lib/apt/lists/*

#### NOTE ####
Expand All @@ -32,7 +32,7 @@ ARG CXX_COMMON_VERSION
ARG LIBRARIES

RUN apt-get update && \
apt-get install -qqy xz-utils python3.8-venv make rpm && \
apt-get install -qqy xz-utils python3 python3.10-venv make rpm && \
rm -rf /var/lib/apt/lists/*

# Build dependencies
Expand All @@ -59,7 +59,7 @@ ENV VIRTUAL_ENV=/opt/trailofbits/venv
ENV PATH="${VIRTUAL_ENV}/bin:${PATH}"

# create a virtualenv in /opt/trailofbits/venv
RUN python3.8 -m venv ${VIRTUAL_ENV}
RUN python3 -m venv ${VIRTUAL_ENV}

# Needed for sourcing venv
SHELL ["/bin/bash", "-c"]
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ plugin that is currently closed source. You can checkout the tag: [binja-final-v

## Getting Help

If you are experiencing undocumented problems with Anvill then ask for help in the `#binary-lifting` channel of the [Empire Hacking Slack](https://empireslacking.herokuapp.com/).
If you are experiencing undocumented problems with Anvill then ask for help in the `#binary-lifting` channel of the [Empire Hacking Slack](https://slack.empirehacking.nyc/).

## Supported Platforms

Expand Down Expand Up @@ -103,10 +103,10 @@ Or you can tell CMake where to find the remill installation prefix by passing `-

### Docker image

To build via Docker run, specify the architecture, base Ubuntu image and LLVM version. For example, to build Anvill linking against LLVM 14 on Ubuntu 20.04 on AMD64 do:
To build via Docker run, specify the architecture, base Ubuntu image and LLVM version. For example, to build Anvill linking against LLVM 16 on Ubuntu 20.04 on AMD64 do:

```shell
ARCH=amd64; UBUNTU_VERSION=20.04; LLVM=14; \
ARCH=amd64; UBUNTU_VERSION=20.04; LLVM=16; \
docker build . \
-t anvill-llvm${LLVM}-ubuntu${UBUNTU_VERSION}-${ARCH} \
-f Dockerfile \
Expand Down
19 changes: 15 additions & 4 deletions bin/Decompile/Main.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
#include <anvill/Optimize.h>
#include <anvill/Providers.h>
#include <anvill/Specification.h>
#include <anvill/Utils.h>
#include <anvill/Version.h>
#include <gflags/gflags.h>
#include <glog/logging.h>
Expand Down Expand Up @@ -42,6 +43,11 @@ DEFINE_bool(add_breakpoints, false,
"lifted bitcode.");

DEFINE_bool(add_names, false, "Try to apply symbol names to lifted entities.");
DEFINE_bool(disable_opt, false, "Dont apply optimization passes");
DEFINE_bool(llvm_debug, false, "Enable LLVM debug flag");
DEFINE_bool(inline_basic_blocks, false,
"Enables inlining of basic blocks for high level output");


DEFINE_string(
default_callable_spec, "",
Expand Down Expand Up @@ -105,9 +111,6 @@ int main(int argc, char *argv[]) {
remill::GetReference(maybe_buff);

llvm::LLVMContext context;
#if LLVM_VERSION_NUMBER < LLVM_VERSION(15, 0)
context.enableOpaquePointers();
#endif
llvm::Module module("lifted_code", context);

auto maybe_spec =
Expand Down Expand Up @@ -158,6 +161,7 @@ int main(int argc, char *argv[]) {
anvill::SpecificationControlFlowProvider cfp(spec);
anvill::SpecificationMemoryProvider mp(spec);
anvill::LifterOptions options(spec.Arch().get(), module, *tp.get(), cfp, mp);
options.should_inline_basic_blocks = FLAGS_inline_basic_blocks;

// options.state_struct_init_procedure =
// anvill::StateStructureInitializationProcedure::kNone;
Expand Down Expand Up @@ -241,7 +245,14 @@ int main(int argc, char *argv[]) {
llvm::EnableStatistics();
}

anvill::OptimizeModule(lifter, module);
if (FLAGS_llvm_debug) {
llvm::DebugFlag = true;
}

if (!FLAGS_disable_opt) {
anvill::OptimizeModule(lifter, module, spec.GetBlockContexts(), spec);
}


int ret = EXIT_SUCCESS;

Expand Down
176 changes: 0 additions & 176 deletions bin/Decompile/tests/scripts/roundtrip.py

This file was deleted.

File renamed without changes.
Loading
Loading