The tutorial document for this repository is in (tutorial.pdf)
This repository includes a few software packages, all of which are dedicated for the analysis of deep neural netowrks (or tree ensembles) over its safety and/or security properties.
- DeepConcolic, a coverage-guided testing tool for convolutional neural networks. Now, it includes a major upgrade based on Bayesian Network based Abstraction.
- testRNN, a coverage-guided testing tool for Long short-term memory models (LSTMs). LSTMs are a major class of recurrent neural networks.
- EKiML, a tool for backdoor embedding and detection for tree ensembles.
- GUAP: a generalised universal adversarial perturbation. It generates universersal adversarial perburbation that may be applied to many inputs at the same time.
In the following, after the installation and download of example models, we will present them one by one.
First of all, please set up a conda environment
conda create --name deepconcolic python==3.7
conda activate deepconcolicThis should be followed by installing software dependencies:
conda install opencv nltk matplotlib
conda install -c pytorch torchvision
pip3 install numpy==1.19.5 scipy==1.4.1 tensorflow\>=2.4 pomegranate==0.14 scikit-learn scikit-image pulp keract np_utils adversarial-robustness-toolbox parse tabulate pysmt saxpy keras menpo patool z3-solver pyvisWe use Fashion-MNIST dataset as the running example. The following are two pre-trained mmodels, one larger and one smaller.
wget -P saved_models https://cgi.csc.liv.ac.uk/~acps/models/small_model_fashion_mnist.h5
wget -P saved_models https://cgi.csc.liv.ac.uk/~acps/models/large_model_fashion_mnist.h5Concolic testing alternates between CONCrete program execution and symbOLIC analysis to explore the execution paths of a software program and to increase code coverage. In this paper, we develop the first concolic testing approach for Deep Neural Networks (DNNs). More specifically, we utilise quantified linear arithmetic over rationals to express test requirements that have been studied in the literature, and then develop a coherent method to perform concolic testing with the aim of better coverage. Our experimental results show the effectiveness of the concolic testing approach in both achieving high coverage and finding adversarial examples.
The paper is available at https://arxiv.org/abs/1805.00089.
In the following, we first present the original ASE2018 version, and then introduce two new upgrades (fuzzing engine and Bayesian network based abstraction).
usage: python3 -m deepconcolic.main [-h] --dataset
{OpenML:har,cifar10,fashion_mnist,mnist}
--model MODEL --outputs DIR --criterion
{nc,ssc,ssclp,bfc,bfdc} --norm {l0,linf}
[--setup-only] [--init INT]
[--max-iterations INT] [--save-all-tests]
[--rng-seed SEED]
[--extra-tests DIR [DIR ...]]
[--filters {LOF}] [--norm-factor FLOAT]
[--lb-hard FLOAT] [--lb-noise FLOAT]
[--mcdc-cond-ratio FLOAT]
[--top-classes CLS]
[--layers LAYER [LAYER ...]]
[--feature-index INT] [--dbnc-spec SPEC]
[--dbnc-abstr PKL]
Concolic testing for Neural Networks
optional arguments:
-h, --help show this help message and exit
--dataset {OpenML:har,cifar10,fashion_mnist,mnist}
selected dataset
--model MODEL the input neural network model (.h5 file or "vgg16")
--outputs DIR the output test data directory
--criterion {nc,ssc,ssclp,bfc,bfdc}
the test criterion
--norm {l0,linf} the norm metric
--setup-only only setup the coverage critierion and analyzer, and
terminate before engine initialization and startup
--init INT number of test samples to initialize the engine
--max-iterations INT maximum number of engine iterations (use < 0 for
unlimited)
--save-all-tests save all generated tests in output directory; only
adversarial examples are kept by default
--rng-seed SEED Integer seed for initializing the internal random
number generator, and therefore get some(what)
reproducible results
--extra-tests DIR [DIR ...], +i DIR [DIR ...]
additonal directories of test images
--filters {LOF} additional filters used to put aside generated test
inputs that are too far from training data (there is
only one filter to choose from for now; the plural is
used for future-proofing)
--norm-factor FLOAT norm distance upper threshold above which generated
inputs are rejected by the oracle (default is 1/4)
--lb-hard FLOAT hard lower bound for the distance between original and
generated inputs (concolic engine only---default is
1/255 for image datasets, 1/100 otherwise)
--lb-noise FLOAT extra noise on the lower bound for the distance
between original and generated inputs (concolic engine
only---default is 1/10)
--mcdc-cond-ratio FLOAT
the condition feature size parameter (0, 1]
--top-classes CLS check the top-CLS classifications for models that
output estimations for each class (e.g. VGG*)
--layers LAYER [LAYER ...]
test layers given by name or index
--feature-index INT to test a particular feature map
--dbnc-spec SPEC Feature extraction and discretisation specification
--dbnc-abstr PKL, --bn-abstr PKL
input BN abstraction (.pkl)
The neural network model under tested is specified by --model and a set of raw test data should be given
by using --inputs. Some popular datasets like MNIST and CIFAR10 can be directly specified by using the
--dataset option directly. --criterion is used to choose the coverage
criterion and --norm helps select the norm metric to measure the distance between inputs. Some examples
to run DeepConcolic are in the following.
To run an MNIST model
python -m deepconcolic.main --model saved_models/mnist_complicated.h5 --dataset mnist --outputs outs/To run an CIFAR10 model
python -m deepconcolic.main --model saved_models/cifar10_complicated.h5 --dataset cifar10 --outputs outs/To test a particular layer
python -m deepconcolic.main --model saved_models/cifar10_complicated.h5 --dataset cifar10 --outputs outs/ --layers 2To run MC/DC for DNNs on the CIFAR-10 model
python -m deepconcolic.main --model saved_models/cifar10_complicated.h5 --criterion ssc --mcdc-cond-ratio 0.1 --dataset cifar10 --outputs outsTo run Concolic Sign-sign-coverage (MC/DC) for DNNs on the MNIST model
python -m deepconcolic.main --model saved_models/mnist_complicated.h5 --dataset mnist --outputs outs --criterion ssclpTo run Concolic BN-based Feature coverage (BFCov) for DNNs on the MNIST model
python -m deepconcolic.main --model saved_models/mnist_complicated.h5 --criterion bfc --norm linf --dataset mnist --outputs outs --dbnc-spec dbnc/example.yamlSee the example YAML specification for details on how to configure the BN-based abstraction.
To run Concolic BN-based Feature-dependence coverage (BFdCov) for DNNs on the MNIST model
python -m deepconcolic.main --model saved_models/mnist_complicated.h5 --criterion bfdc --norm linf --dataset mnist --outputs outs --dbnc-spec dbnc/example.yamlYou could adjust the following two parameters in the DBNC specification file defined by --dbnc-spec to dump the generated bayesian network to files bn4trained.yml and bn4tests.yml.
dump_bn_with_trained_dataset_distribution: True,
dump_bn_with_final_dataset_distribution: True,DeepConcolic additionally features an experimental fuzzing engine. The following command illustrates how to exercise this engine on a classifier for the CIFAR10 dataset: it will generate at most 1000 images obtained by mutating inputs randomly drawn from the CIFAR10 validation dataset, and save them into the outs/cifar10-fuzzing-basic directory. Aversarial examples can be identified in the latter directory by searching for files named <test-number>-adv-<wrong-label>.png, derived from file <test-number>-original-<true-label>.png. Passed tests are named in a similar way, as <test-number>-ok-<label>.png.
python3 -m deepconcolic.fuzzer --dataset cifar10 --model saved_models/cifar10_complicated.h5 --processes 2 --outputs outs/cifar10-fuzzing-basic -N 1000Further options are available to use this engine. It is for instance possible to specify a set of files used as seeds for fuzzing with the option --inputs, as in:
python3 -m deepconcolic.fuzzer --dataset mnist --model saved_models/mnist_complicated.h5 --inputs data/mnist-seeds --processes 5 --outputs outs/mnist-fuzzing-given-seeds -N 1000or sample N inputs from the validation dataset beforehand with --sample N:
python3 -m deepconcolic.fuzzer --dataset cifar10 --model saved_models/cifar10_complicated.h5 --sample 10 --processes 5 --outputs outs/cifar10-fuzzing-sub-sample10 -N 1000DeepConcolic provides means for working with additional datasets, that can be provided via a dedicate plugin system.
Such plugins are Python modules that are loaded when the tool starts, and are searched within any directory listed in the colon-separated environment variable DC_PLUGINS_PATH if this variable is defined, or else within the ./dc_plugins directory if it exists (note the latter is relative to the current working directory).
Then, a new dataset can be registered by calling the deepconcolic.datasets.register_dataset function with a name for the dataset as first argument, and a function that loads and returns a dataset description as second argument.
The latter function must accept any set of named arguments (for future extensions), and return a tuple with: (i) a pair of arrays containting trainting data and labets; (ii) a similar pair for validation; (iii) the shape of each individual input element; (iv) a descriptor string in {image, normalized, unknown} (used for determining the input feature encoding—note the format of this descriptor is likely to be refined in future versions); and (v) a list of strings showing the individual label names.
The dataset arrays can be given using numpy.ndarray or pandas.Dataframe dataframes.
The typical pattern is as follows (for loading, e.g., the MNIST dataset provided by tensorflow, as already done in deepconcolic.datasets):
def load_mnist_data (**_):
import tensorflow as tf
img_shape = 28, 28, 1
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data ()
x_train = x_train.reshape (x_train.shape[0], *img_shape).astype ('float32') / 255
x_test = x_test.reshape (x_test.shape[0], *img_shape).astype ('float32') / 255
return (x_train, y_train), (x_test, y_test), img_shape, 'image', \
[ str (i) for i in range (0, 10) ]
register_dataset ('mnist', load_mnist_data)For further illustrative purposes, we provide an example dataset plugin, which can be used to randomly generate classification tasks.
This plugin registers several datasets (named, e.g., rand10_2, rand10_5, and rand100_5) upon startup of DeepConcolic, which should then show as valid options for the --dataset option.
We also provide a utility script to construct and train small DNNs for the above toy datasets:
To train a classifier for the rand10_2 dataset, and then print a short classification report:
# The following saves the trained model under `/tmp' on Unix-style systems:
python3 -m utils.train4random rand10_2
python3 -m deepconcolic.eval_classifier --dataset rand10_2 --model /tmp/rand10_2_dense_50_50_10_10.h5To run the fuzzer on the newly trained model, using a sample of 10 initial test data and 5 processes:
python3 -m deepconcolic.fuzzer --dataset rand10_2 --model /tmp/rand10_2_dense_50_50_10_10.h5 --sample 10 --processes 5 --outputs outs/rand10_2-fuzz1000 -N 1000The above command outputs new inputs within a file outs/rand10_2-fuzz1000/new_inputs.csv.
For long short-term memory models (LSMTs), we design new coverage metrics to consider the internal behaviour of the LSTM layers in processing sequential inputs. We consider not only the tighter metric that quantifies the temporal behaviour (i.e., temporal coverage) but also some looser metrics that quantify either the gate values (i.e., Neuron Coverage and Boundary Coverage) or value change in one step (i.e., Stepwise Coverage).
The paper is available at https://arxiv.org/pdf/1911.01952.pdf.
- Neuron Coverage (NC),
- Boundary Coverage (BC),
- Stepwise Coverage (SC),
- Temporal Coverage (TC)
- Fashion-MNIST
- Sentiment Analysis,
- MNIST Handwritten Digits,
- UCF101 (need to download and put into the dataset file)
As running example, we download the pre-trained Fasion-MNIST model as follows.
wget -P saved_models https://cgi.csc.liv.ac.uk/~acps/models/fashion_mnist_lstm.h5We have two commands to run testing procedure and to run result analysis procedure, respectively.
python -m testRNN.main --model <modelName>
--TestCaseNum <Num. of Test Cases>
--threshold_SC <SC threshold>
--threshold_BC <BC threshold>
--symbols_TC <Num. of symbols>
--seq <seq in cells to test>
--mode <modeName>
--outputs <output directory>
where:
<modelName>is in {sentiment,mnist,fashion_mnist,ucf101}<Num. of Test Cases>is the expected number of test cases<Mutation Method>is in {random,genetic}<SC threshold>is in [0, 1]<BC threshold>is in [0, 1]<Num. of symbols>is in {1, 2, 3...}<seq in cells to test>is in {mnist: [4, 24], fashion_mnist: [4, 24], sentiment: [400, 499], ucf101: [0, 10]}<modeName>is in {train,test} with default valuetest<output directory>specifies the path of the directory to save the output record and generated examples
For example, we can run the following
python -m testRNN.main --model fashion_mnist --TestCaseNum 10000 --Mutation random --threshold_SC 0.6 --threshold_BC 0.7 --symbols_TC 3 --seq [4,24] --outputs testRNN_outputwhich says that, we are working with Fashion-MNIST model, and the genetic algorithm based test case generation will terminate when the number of test cases is over 10000. We need to specify other parameters including threshold_SC, threshold_BC, symbols_TC, and seq. Moreover, the log is generated to the file testRNN_output/record.txt. Also the output of adversarial examples can be found in testRNN_output/adv_output
In this tool, we consider embedding knowledge into machine learning models. The knowledge expression we considered can express e.g., robustness and resilience to backdoor attack, etc. That is, we can "embed" knowledge into a tree ensemble, representing a backdoor attack on the tree ensemble. Also, we can "detect" if a tree ensemble has been attacked.
The paper is available at https://arxiv.org/pdf/2010.08281.pdf.
As the running example, we download the pre-trained HAR tree model as follows.
wget -P saved_models https://cgi.csc.liv.ac.uk/~acps/models/har_tree_black-box.npy
wget -P saved_models https://cgi.csc.liv.ac.uk/~acps/models/har_forest_black-box.npypython -m EKiML.main --Dataset <DatasetName>
--Mode <modeName>
--Embedding_Method <embeddingMethod>
--Model <modeType>
--Pruning <pruningFlag>
--SaveModel <saveModelFlag>
--workdir <workDirectory>
where the flags have multiple options:
<DatasetName>is in {'iris', 'har', 'breast_cancer', 'mushroom', 'nursery, 'cod-rna', 'sensorless', 'mnist'}.<modeName>is in {'embedding', 'synthesis'}, where 'synthesis' denotes the "extraction".<embeddingMethod>is in {'black-box', 'white-box'}<modeType>is in {'forest', 'tree'}<pruningFlag>is in {True, False}, with default value False<saveModelFlag>is in {True, False}, with default value False<workDirectory>is the working directory, with default value 'EKiML_workdir'<Datadir>is the directory where dataset files are located (default is 'EKiML/dataset')
For example, we can run the following
python -m EKiML.main --Dataset har --Mode synthesis --Embedding_Method black-box --Model tree --workdir 'EKiML_har' --Datadir 'datasets'which suggests that we are considering the HAR dataset, tryng to synthesise knowledge from a pre-trained tree by applying our black-box synthesis algorithm.
Tool for generating spatial-transfermed or additive universarial perturbations, the paper 'Generalizing Universal Adversarial Attacks Beyond Additive Perturbations' was accepted by ICDM 2020.
Please cite Yanghao Zhang, Wenjie Ruan, Fu Wang, and Xiaowei Huang, Generalizing Universal Adversarial Attacks Beyond Additive Perturbations, The IEEE International Conference on Data Mining (ICDM 2020), November 17-20, 2020, Sorrento, Italy
The paper is avaiable at: https://arxiv.org/pdf/2010.07788.pdf
In this paper, for the first time we propose a unified and flexible framework, which can capture the distribution of the unknown additive and non-additive adversarial perturbations jointly for crafting Generalized Universal Adversarial Perturbations. Specifically, GUAP can generate either additive (i.e., l_inf-bounded) or non-additive (i.e., spatial transformation) perturbations, or a combination of both, which considerably generalizes the attacking capability of current universal attack methods.
There is also a notebook demo Colab_GUAP.ipynb, which can be run on the Colab.
pip install torch torchvision matplotlibwget -P saved_models https://cgi.csc.liv.ac.uk/~acps/models/cifar10_vgg19.pth
wget -P saved_models https://cgi.csc.liv.ac.uk/~acps/models/cifar10_resnet101.pth
wget -P saved_models https://cgi.csc.liv.ac.uk/~acps/models/cifar10_dense121.pth
wget -P saved_models https://cgi.csc.liv.ac.uk/~acps/models/fashion_mnist_modela.pth(from within the GUAP sub-directory)
usage: run_xxxxxx.py [-h] [--dataset DATASET] [--lr LR]
[--batch-size BATCH_SIZE] [--epochs EPOCHS]
[--l2reg L2REG] [--beta1 BETA1] [--tau TAU]
[--eps EPS] [--model MODEL]
[--manualSeed MANUALSEED] [--gpuid GPUID] [--cuda]
[--resume] [--outdir OUTDIR]
optional arguments:
-h, --help show this help message and exit
--dataset DATASET Fashion-MNIST
--lr LR Learning rate
--batch-size BATCH_SIZE
--epochs EPOCHS number of epochs to train for
--l2reg L2REG weight factor for l2 regularization
--beta1 BETA1 beta1 for adam
--tau TAU max flow magnitude
--eps EPS allow for linf noise
--model MODEL modelA
--manualSeed MANUALSEED
manual seed
--gpuid GPUID multi gpuid
--cuda enables cuda
--resume load pretrained model
--outdir OUTDIR output dir
python run_fashion_mnist.py --cuda --gpuid 0 --resumepython run_cifar.py --cuda --gpuid 0 --model VGG19 --tau 0.1 --eps 0.03python run_imagenet.py --cuda --gpuid 0,1 --model ResNet152 --tau 0.1 --eps 0.03
@inproceedings{swrhkk2018,
AUTHOR = { Sun, Youcheng
and Wu, Min
and Ruan, Wenjie
and Huang, Xiaowei
and Kwiatkowska, Marta
and Kroening, Daniel },
TITLE = { Concolic Testing for Deep Neural Networks },
BOOKTITLE = { Automated Software Engineering (ASE) },
PUBLISHER = { ACM },
PAGES = { 109--119 },
ISBN = { 978-1-4503-5937-5 },
YEAR = { 2018 }
}
@article{sun2018testing,
AUTHOR = { Sun, Youcheng
and Huang, Xiaowei
and Kroening, Daniel },
TITLE = { Testing Deep Neural Networks },
JOURNAL = { arXiv preprint arXiv:1803.04792 },
YEAR = { 2018 }
}
@article{10.1145/3358233,
author = {Sun, Youcheng and Huang, Xiaowei and Kroening, Daniel and Sharp, James and Hill, Matthew and Ashmore, Rob},
title = {Structural Test Coverage Criteria for Deep Neural Networks},
year = {2019},
issue_date = {October 2019},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
volume = {18},
number = {5s},
issn = {1539-9087},
url = {https://doi.org/10.1145/3358233},
doi = {10.1145/3358233},
journal = {ACM Trans. Embed. Comput. Syst.},
articleno = {Article 94},
numpages = {23},
keywords = {test criteria, Neural networks, test case generation} }
@article{testRNN,
author = {Wei Huang and
Youcheng Sun and
Xingyu Zhao and
James Sharp and
Wenjie Ruan and
Jie Meng and
Xiaowei Huang},
title = {Coverage Guided Testing for Recurrent Neural Networks},
journal = {IEEE transactions on Reliability},
year = {2021},
}
@article{DBLP:journals/corr/abs-2010-08281,
author = {Wei Huang and
Xingyu Zhao and
Xiaowei Huang},
title = {Embedding and Synthesis of Knowledge in Tree Ensemble Classifiers},
journal = {Machine Learning},
year = {2021}
}
@inproceedings{zhang2020generalizing,
title={Generalizing Universal Adversarial Attacks Beyond Additive Perturbations},
author={Yanghao Zhang and Wenjie Ruan and Fu Wang and Xiaowei Huang},
year={2020},
booktitle = {ICDM 2020}
}