(Artifact) Case Study: Neural Network Malware Detection Verification for Feature and Image Datasets

This artifact is used to reproduce the results shown in Case Study: Neural Network Malware Detection Verification for Feature and Image Datasets.

Requirements

The following resources are required to run this artifact:

1. Docker
2. MATLAB license
3. ~20 GB of disk space

1. Installation

1. Clone this repository and navigate to the verify_malware directory by running:

git clone https://github.com/pkrobinette/verify_malware && cd verify_malware

2. Download the following dataset files from here into the verify_malware/archive folder:

• bodmas.mat

• bodmas.npz

• malimg_dataset.tar.gz

  NOTE: Running ls archive should return the following:

• bodmas-feature-analysis-NEW.csv

• bodmas.mat

• bodmas.npz

• malimg_dataset.tar.gz

• malimg_verification_image_paths.csv

2. Smoke Test Instructions (Requires ~20 GB of disk space && ~ 15 min. runtime)

Instructions to quickly test the dockerfiles and tools needed to reproduce the artifacts of the paper. If the smoke test is successful, proceed to Artifact Instructions (see below).

1. Open a terminal, make sure you are in the verify_malware directory, and run the following:

chmod +x run_smoke_test.sh && ./run_smoke_test.sh

2. This smoke test will run 2 dockerfiles: (1) a MATLAB dockerfile designated to run an NNV smoke test and (2) a dockerfile designated to run an nnenum smoke test. A license is required for the MATLAB dockerfile. Once the dockerfile is built, you will be prompted to enter an email associated with your mathworks account. Follow the prompted directions before leaving the smoke test to run.

3. If successful, you should see the following message displayed at the bottom of the terminal:

   **********************************************
          smoke test passed.         
   **********************************************
   

3. Artifact Instructions (Requires ~20 GB of disk space && ~ 2 hr runtime)

Instructions to reproduce the results of the paper. NOTE: Results will not exactly match paper results as random seeds could be affected by docker image.

1. Open a terminal, make sure you are in the verify_malware directory, and run the following:

chmod +x run_artifact.sh && ./run_artifact.sh

2. This artifact will run 2 dockerfiles: (1) a MATLAB dockerfile for NNV experiments and (2) a dockerfile for nnenum experiments. A license is required for the MATLAB dockerfile. Once the dockerfile is built, you will be prompted to enter an email associated with your mathworks account. Follow the prompted directions before leaving the artifact to run.

4. Results

After the artifact has finished running, all results are saved to results/artifact. Artifacts and corresponding paper labels are listed below:

Artifact	Result Location	Description
Table 5:	results/artifacts/bodmas/table_5.txt	BODMAS certified robustness accuracy + avg. time results.
Table 6:	results/artifacts/malimg/table_6.txt	MALIMG certified robustness accuracy + avg. time results.
Table 7:	results/artifacts/malimg/table_7.txt	MALIMG per class certified robustness accuracy results.
Figure 4a:	results/artifacts/bodmas/figures/Fig_4a_all	BODMAS - all features CRA results.
Figure 4b:	results/artifacts/bodmas/figures/Fig_4b_all	BODMAS - all features time results.
Figure 5a:	results/artifacts/bodmas/figures/Fig_5a_continuous_discrete	BODMAS - discrete and continuous features CRA results.
Figure 5b:	results/artifacts/bodmas/figures/Fig_5b_continuous_discrete	BODMAS - discrete and continuous features time results.
Figure 6a:	results/artifacts/bodmas/figures/Fig_6a_discrete	BODMAS - discrete features CRA results.
Figure 6b:	results/artifacts/bodmas/figures/Fig_6b_discrete	BODMAS - discrete features time results.
Figure 7a:	results/artifacts/bodmas/figures/Fig_7a_continuous	BODMAS - continuous features CRA results.
Figure 7b:	results/artifacts/bodmas/figures/Fig_7b_continuous	BODMAS - continuous features time results.
Figure 8a:	results/artifacts/malimg/figures/Fig_8a	Malimg CRA results.
Figure 8b:	results/artifacts/malimg/figures/Fig_8b	Malimg time results

[Extra] Smoke Test Instructions for nnenum ONLY (Requires ~20 GB of disk space && ~ 10 min. runtime)

Instructions to quickly test the dockerfiles and tools needed to reproduce the artifacts of the paper for nnenum.

1. Open a terminal, make sure you are in the verify_malware directory, and run the following:

chmod +x run_smoke_test_nnenum_only.sh && ./run_smoke_test_nnenum_only.sh

2. If successful, you should see the following message displayed at the bottom of the terminal:

   **********************************************
          smoke test passed.         
   **********************************************
   

Notes

If there is not enough space to run, try:

docker system prune