Middleware'19 Artifact Evaluation

This page contains a link to the artifacts (in the form of a docker container) as well as detailed instruction explaining how to reproduce the security and performance evaluation from the paper.

Introduction

This wiki is written for artifact evaluation and for those who want to reproduce our numbers. We provide a docker image that contains all the sources and scripts we used to get our results.

In order to get the docker up and running, use following commands:

docker pull bcliu430/slimguard
docker run -it bcliu430/slimguard /bin/bash

You will then get a bash shell.

Macro-benchmark

Single thread with all benchmarks

Estimated time: 2 hours

In the /home directory, there is a script called run_parsec.sh. simply type ./run_parsec.sh.

This will generate 4 files, including: vanilla, openbsd, guarder, and slimguard. Each file contains three columns, the first column is name of the benchmark, the second column is execution time in seconds, the third column is Maximum memory foorprints.

Figure 8 is obtained based on this benchmark.

Multi-thread with selected benchmarks

Estimated time: 1 hour

In the /home directory, there is a script called run_parsec_mt.sh. simply type ./run_parsec_mt.sh. It will run a subset of parsec, and save the results to files.Each file contains three columns, the first column is name of the benchmark, the second column is execution time in seconds, the third column is Maximum memory foorprints.

Figure 9 is obtained based on this benchmark.

MiBench

Estimated time: 30s

In /home/mibench, there is a run.sh, in order to run the mibench results, you will need to just use ./run.sh. It will generate 2 files for each allcoator, one file is memory consumption in MB, the other one is execution time in us. Refer run.sh for more details.

Micro-benchmark

Entropy Study

Estimated time: 10min

We believe, ideally, number of entropy bits has negligible impact on performance. We setup different number of entropy bits(2, 4, 6, 8, 10) for both Guarder and SlimGuard, we use a simple malloc/free microbenchmark to evaluate the impacts. To modify the entropy bits for each allocator, slimguard has a macro defined in /home/SlimGuard/include/slimguard.h:20; guarder defines its entropy bits at /home/Guarder/xdefines.hh:118.

In order to run the benchmarks, following:

cd /home/microbench/entropy_study
./run.sh

The results are in slimguard*.csv and guarder*.csv, each should contain two columns first column is malloc time, and the second column is free time, both in CPU ticks, please refer Figure 5 for more details. There is also another result file named slimguard_op.txt, it contains the over-provisioning ratio on different situations, see table 2 for more details.

Figure 5 and Table 2 are obtained based on this benchmark.

Guard Pages

Estimated time: 10min

We measure the impact of numbers of guard pages on both SlimGuard and Guarder. We setup different guard page ratios for both allocators. In SlimGuard, we set up the guard page ratio at /home/SlimGuard/include/slimguard.h:19; for Guarder, we can modify its guard page ratio at /home/Guarder/xdefines.hh:137.

To run this set of benchmark, run:

cd /home/microbench/guard_page
./run.sh

The results are in slimguard.csv and guarder.csv, each should contain two columns first column is malloc time, and the second column is free time, both in CPU ticks.

Figure 6 is obtained based on this benchmark.

Initialization Time

Estimated time: 10min

In order to measure the initialization time between SlimGuard and guarder, we use a very simple program with only a malloc() and free() in it. We also use different guard page ratios obtained from previous step to see the impact for initialization time. Use the following commands to obtain results.

cd /home/microbench/init_time
./run.sh

There are two files generated, guarder.csv and slimguard.csv, it contains 50 execution times of each allocator.

Figure 7 is obtained based on this benchmark.

Security Evaluation

Since some of the vulnerabilities are patched, or it needs to convert from stack to heap, we have provided the source code in order to easily reproduce the bugs.

See Section 5.3 of SlimGuard paper for more details.

gzip

cd /home/bugbench/gzip-1.2.4/script
./run_bug

ncompress

cd /home/bugbench/ncompress/script
./run_bug

gnu-ed

cd /home/gnu-ed-1.14.1
./run_bug.sh

ImageMagick

cd /home/ImageMagick-7.0.7-25/utilities
./run_bug.sh