03-03-adversarial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Synthesizing [adversarial examples](https://arxiv.org/abs/1312.6199) for neural networks is surprisingly easy: small, carefully-crafted perturbations to inputs can cause neural networks to misclassify inputs in arbitrarily chosen ways. Given that adversarial examples [transfer to the physical world](https://arxiv.org/abs/1607.02533) and [can be made extremely robust](https://blog.openai.com/robust-adversarial-inputs/), this is a real security concern.\n",
    "\n",
    "In this notebook, we'll give a brief introduction to algorithms for synthesizing adversarial examples, and we'll walk through the process of implementing attacks in [TensorFlow](https://www.tensorflow.org/), building up to synthesizing a robust adversarial example following [this technique](https://arxiv.org/abs/1707.07397)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Setup\n",
    "\n",
    "We'll attack an [Inception v3](https://arxiv.org/abs/1512.00567) network trained on [ImageNet](http://www.image-net.org/). In this section, we load a pre-trained network from the [TF-slim image classification library](https://github.com/tensorflow/models/tree/master/slim). This part isn't particularly interesting, so feel free to skip reading this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow.contrib.slim as slim\n",
    "import tensorflow.contrib.slim.nets as nets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.logging.set_verbosity(tf.logging.ERROR)\n",
    "sess = tf.InteractiveSession()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we set up the input image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = tf.placeholder(tf.float32, (299, 299, 3)) # InceptionV3 takes a 299x299x3 input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we load the Inception v3 model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def inception(x, reuse):\n",
    "    preprocessed = tf.multiply(tf.subtract(x, 0.5), 2.0)\n",
    "    arg_scope = nets.inception.inception_v3_arg_scope(weight_decay=0.0)\n",
    "    with slim.arg_scope(arg_scope):\n",
    "        logits, _ = nets.inception.inception_v3(\n",
    "            preprocessed, 1001, is_training=False, reuse=reuse)\n",
    "        logits = logits[:,1:] # ignore background class\n",
    "        probs = tf.nn.softmax(logits) # probabilities\n",
    "    return logits, probs\n",
    "\n",
    "logits, probs = inception(tf.expand_dims(x, 0), reuse=False)\n",
    "# gives us logits (pre-softmax) and probabilities (post-softmax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we load pre-trained weights. This Inception v3 has a top-5 accuracy of 93.9%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tempfile\n",
    "from urllib.request import urlretrieve\n",
    "import tarfile\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = tempfile.mkdtemp()\n",
    "inception_tarball, _ = urlretrieve(\n",
    "    'http://download.tensorflow.org/models/inception_v3_2016_08_28.tar.gz')\n",
    "tarfile.open(inception_tarball, 'r:gz').extractall(data_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "restore_vars = [\n",
    "    var for var in tf.global_variables()\n",
    "    if var.name.startswith('InceptionV3/')\n",
    "]\n",
    "saver = tf.train.Saver(restore_vars)\n",
    "saver.restore(sess, os.path.join(data_dir, 'inception_v3.ckpt'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we write some code to show an image, classify it, and show the classification result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('resources/imagenet.json') as f:\n",
    "    imagenet_labels = json.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def classify(img, correct_class=None, target_class=None):\n",
    "    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 8))\n",
    "    fig.sca(ax1)\n",
    "    p = sess.run(probs, feed_dict={x: img})[0]\n",
    "    ax1.imshow(img)\n",
    "    fig.sca(ax1)\n",
    "    \n",
    "    topk = list(p.argsort()[-10:][::-1])\n",
    "    topprobs = p[topk]\n",
    "    barlist = ax2.bar(range(10), topprobs)\n",
    "    if target_class in topk:\n",
    "        barlist[topk.index(target_class)].set_color('r')\n",
    "    if correct_class in topk:\n",
    "        barlist[topk.index(correct_class)].set_color('g')\n",
    "    plt.sca(ax2)\n",
    "    plt.ylim([0, 1.1])\n",
    "    plt.xticks(range(10),\n",
    "               [imagenet_labels[i][:15] for i in topk],\n",
    "               rotation='vertical')\n",
    "    fig.subplots_adjust(bottom=0.2)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example image\n",
    "\n",
    "We load our example image and make sure it's classified correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import PIL.Image\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img_class = 281\n",
    "img = PIL.Image.open('resources/cat.jpg')\n",
    "# take a square center crop of the image and convert it to a float32 image with pixels in [0, 1]\n",
    "big_dim = max(img.width, img.height)\n",
    "wide = img.width > img.height\n",
    "new_w = 299 if not wide else int(img.width * 299 / img.height)\n",
    "new_h = 299 if wide else int(img.height * 299 / img.width)\n",
    "img = img.resize((new_w, new_h)).crop((0, 0, 299, 299))\n",
    "img = (np.asarray(img) / 255.0).astype(np.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classify(img, correct_class=img_class)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we expect, it's classified as a cat."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adversarial examples\n",
    "\n",
    "Given an image $\\mathbf{x}$, our neural network outputs a probability distribution over labels, $P(y \\mid \\mathbf{x})$. When we craft an adversarial input, we want to find an $\\hat{\\mathbf{x}}$ where $\\log P(\\hat{y} \\mid \\hat{\\mathbf{x}})$ is maximized for a target label $\\hat{y}$: that way, our input will be misclassified as the target class. We can ensure that $\\hat{\\mathbf{x}}$ doesn't look too different from the original $\\mathbf{x}$ by constraining ourselves to some $\\ell_\\infty$ box with radius $\\epsilon$, requiring that $\\left\\lVert \\mathbf{x} - \\hat{\\mathbf{x}} \\right\\rVert_\\infty \\le \\epsilon$.\n",
    "\n",
    "In this framework, an adversarial example is the solution to a constrained optimization problem that we can solve using backpropagation and projected gradient descent, basically the same techniques that are used to train networks themselves. The algorithm is simple:\n",
    "\n",
    "We begin by initializing our adversarial example as $\\hat{\\mathbf{x}} \\leftarrow \\mathbf{x}$. Then, we repeat the following until convergence:\n",
    "\n",
    "1. $\\hat{\\mathbf{x}} \\leftarrow \\hat{\\mathbf{x}} + \\alpha \\cdot \\nabla \\log P(\\hat{y} \\mid \\hat{\\mathbf{x}})$\n",
    "2. $\\hat{\\mathbf{x}} \\leftarrow \\mathrm{clip}(\\hat{\\mathbf{x}}, \\mathbf{x} - \\epsilon, \\mathbf{x} + \\epsilon)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_hat = tf.placeholder(tf.int32, ()) # a placeholder for the target class\n",
    "labels = tf.one_hot(y_hat, 1000) # InceptionV3 uses one-hot encoded labels\n",
    "loss = tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=[labels])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parameters\n",
    "\n",
    "Before we perform our attack, let's choose some concrete parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "epsilon = 2.0/255.0 # a really small perturbation\n",
    "target = 924 # this is \"guacamole\"; you could try choosing anything else"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Projected gradient descent\n",
    "\n",
    "We use projected gradient descent to maximize the log probability of the target class (or equivalently, minimize the [cross entropy](https://en.wikipedia.org/wiki/Cross_entropy)) while keeping the adversarial example visually close to the original image (what happens if we don't do this?)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grad, = tf.gradients(loss, x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can evaluate the gradient `grad` at a particular $x = $ `x_hat` and $\\hat{y}$ = `target` by running `grad.eval({x: x_hat, y_hat: target})`.\n",
    "\n",
    "Finally, we're ready to run projected gradient descent to find our adversarial example `x_hat`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = 1e-1\n",
    "iterations = 30\n",
    "\n",
    "x_hat = np.copy(img) # initial guess\n",
    "\n",
    "upper = np.clip(img + epsilon, 0, 1) # an upper bound for pixels in the image\n",
    "lower = np.clip(img - epsilon, 0, 1) # a lower bound for pixels in the image\n",
    "\n",
    "for i in range(iterations):\n",
    "    # gradient descent step\n",
    "    # TODO\n",
    "    \n",
    "    # projection step; what happens if you don't do this (and e.g. clip to [0, 1] instead)?\n",
    "    # TODO\n",
    "    \n",
    "    # print progress\n",
    "    l = loss.eval({x: x_hat, y_hat: target})\n",
    "    print('step %d, loss %f' % (i, l))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This adversarial image is visually indistinguishable from the original, with no visual artifacts. However, it's classified as \"guacamole\" with high confidence!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classify(x_hat, correct_class=img_class, target_class=target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Robust adversarial examples\n",
    "\n",
    "Now, we go through a more advanced example. We follow an approach for synthesizing robust adversarial examples to find a single perturbation of our cat image that's simultaneously adversarial under some chosen distribution of transformations.  We could choose any distribution of differentiable transformations; in this post, we'll synthesize a single adversarial input that's robust to rotation by $\\theta \\in [-\\pi/4, \\pi/4]$.\n",
    "\n",
    "Before we proceed, let's check if our previous example is still adversarial if we rotate it, say by an angle of $\\theta = \\pi/8$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ex_angle = np.pi/8\n",
    "\n",
    "angle = tf.placeholder(tf.float32, ())\n",
    "rotated_image = tf.contrib.image.rotate(x, angle)\n",
    "rotated_example = rotated_image.eval(feed_dict={x: x_hat, angle: ex_angle})\n",
    "classify(rotated_example, correct_class=img_class, target_class=target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like our original adversarial example is not rotation-invariant!\n",
    "\n",
    "So, how do we make an adversarial example robust to a distribution of transformations? Given some distribution of transformations $T$, we can maximize $\\mathbb{E}_{t \\sim T} \\log P\\left(\\hat{y} \\mid t(\\hat{\\mathbf{x}})\\right)$, subject to $\\left\\lVert \\mathbf{x} - \\hat{\\mathbf{x}} \\right\\rVert_\\infty \\le \\epsilon$. We can solve this optimization problem via projected gradient descent, noting that $\\nabla \\mathbb{E}_{t \\sim T} \\log P\\left(\\hat{y} \\mid t(\\hat{\\mathbf{x}})\\right)$ is $\\mathbb{E}_{t \\sim T} \\nabla \\log P\\left(\\hat{y} \\mid t(\\hat{\\mathbf{x}})\\right)$ and approximating with samples at each gradient descent step.\n",
    "\n",
    "Rather than manually implementing the gradient sampling, we can get TensorFlow to do it for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_samples = 10 # samples per step\n",
    "\n",
    "rotated = []\n",
    "for i in range(num_samples):\n",
    "    rotated.append(tf.contrib.image.rotate(\n",
    "        x, tf.random_uniform((), minval=-np.pi/4, maxval=np.pi/4)))\n",
    "rotated = tf.stack(rotated)\n",
    "rotated_logits, _ = inception(rotated, reuse=True)\n",
    "\n",
    "duplicated_labels = tf.tile(tf.expand_dims(labels, 0), (num_samples, 1))\n",
    "\n",
    "average_loss = tf.reduce_mean(\n",
    "    tf.nn.softmax_cross_entropy_with_logits(logits=rotated_logits, labels=duplicated_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Projected gradient descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can run PGD just like we did last time to generate our adversarial input. As in the previous example, we'll choose \"guacamole\" as our target class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grad_robust, = tf.gradients(average_loss, x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "epsilon = 8.0/255.0 # still a pretty small perturbation\n",
    "learning_rate = 2e-1\n",
    "iterations = 50\n",
    "\n",
    "x_hat_robust = np.copy(img) # initial guess\n",
    "\n",
    "upper = np.clip(img + epsilon, 0, 1) # an upper bound for pixels in the image\n",
    "lower = np.clip(img - epsilon, 0, 1) # a lower bound for pixels in the image\n",
    "\n",
    "for i in range(iterations):\n",
    "    # gradient descent step\n",
    "    # TODO\n",
    "    \n",
    "    # projection step; what happens if you don't do this (and e.g. clip to [0, 1] instead)?\n",
    "    # TODO\n",
    "    \n",
    "    # print progress\n",
    "    l = average_loss.eval({x: x_hat_robust, y_hat: target})\n",
    "    print('step %d, average loss %f' % (i, l))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This adversarial image is classified as \"guacamole\" with high confidence, even when it's rotated!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rotated_example = rotated_image.eval(feed_dict={x: x_hat_robust, angle: ex_angle})\n",
    "classify(rotated_example, correct_class=img_class, target_class=target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "Let's examine the rotation-invariance of the robust adversarial example we produced over the entire range of angles, looking at $P(\\hat{y} \\mid \\hat{\\mathbf{x}})$ over $\\theta \\in [-\\pi/4, \\pi/4]$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "thetas = np.linspace(-np.pi/4, np.pi/4, 51)\n",
    "\n",
    "p_naive = []\n",
    "p_robust = []\n",
    "for theta in thetas:\n",
    "    rotated = rotated_image.eval(feed_dict={x: x_hat_robust, angle: theta})\n",
    "    p_robust.append(probs.eval(feed_dict={x: rotated})[0][target])\n",
    "    \n",
    "    rotated = rotated_image.eval(feed_dict={x: x_hat, angle: theta})\n",
    "    p_naive.append(probs.eval(feed_dict={x: rotated})[0][target])\n",
    "\n",
    "robust_line, = plt.plot(thetas, p_robust, color='b', linewidth=2, label='robust')\n",
    "naive_line, = plt.plot(thetas, p_naive, color='r', linewidth=2, label='naive')\n",
    "plt.ylim([0, 1.05])\n",
    "plt.xlabel('rotation angle')\n",
    "plt.ylabel('target class probability')\n",
    "plt.legend(handles=[robust_line, naive_line], loc='lower right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's super effective!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# More exploration\n",
    "\n",
    "Finished early? Try making this work for other types of transformation, such as random crops, rescaling, or shearing."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}