1a_conv_nets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/khipu-ai/practicals-2019/blob/master/1a_conv_nets.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NcAHF4g8Xa93"
   },
   "source": [
    "# Practical 1a: Convolutional Networks\n",
    "\n",
    "© Deep Learning Indaba. Apache License 2.0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zFOuMk56XjC8"
   },
   "source": [
    "## Introduction\n",
    "In this practical, we will cover the basics of convolutional neural networks, or \"ConvNets\". ConvNets were invented in the late 1980s/early 1990s, and have had tremendous success especially with vision although they have also been used very successfully in speech processing pipelines, and more recently, for machine translation.\n",
    "\n",
    "## Learning Objectives\n",
    "* Be able to explain what a convolutional layer does and how it's different from a fully-connected layer \n",
    "* Understand  the assumptions and trade-offs that are being made when using convolutional architectures\n",
    "* Be able to build a convolutional architecture using Tensorflow 2.0 and Keras Layers\n",
    "* Be able to use Keras to train a model on a dataset\n",
    "* Implement either batch normalisation or a very small residual network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TPiFyajJtDmj"
   },
   "source": [
    "**IMPORTANT: Please fill out the exit ticket form before you leave the practical: https://forms.gle/LzHwg1BGqdYUdK5q8**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zFOuMk56XjC8"
   },
   "source": [
    "## Running on GPU\n",
    "For this practical, you will need to use a GPU to speed up training. To do this, go to the \"Runtime\" menu in Colab, select \"Change runtime type\" and then in the popup menu, choose \"GPU\" in the \"Hardware accelerator\" box. This is all you need to do, Colab and Tensorflow will take care of the rest! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bJG0K_T7wifE"
   },
   "outputs": [],
   "source": [
    "#@title Imports (RUN ME!) { display-mode: \"form\" }\n",
    "\n",
    "!pip install tensorflow-gpu==2.0.0-beta0 > /dev/null 2>&1\n",
    "!pip -q install pydot_ng > /dev/null 2>&1\n",
    "!pip -q install graphviz > /dev/null 2>&1\n",
    "!apt install graphviz > /dev/null 2>&1\n",
    "\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from IPython import display\n",
    "%matplotlib inline\n",
    "\n",
    "print(\"TensorFlow executing eagerly: {}\".format(tf.executing_eagerly()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "u5_zoqk-YK0D"
   },
   "source": [
    "## Convolutional Architectures\n",
    "When modelling an image using a regular feed-forward network, we find that the number of model parameters grows exponentially.\n",
    "\n",
    "**QUESTION**: How many parameters would there be in a feed-forward network with 2 hidden layers consisting of 512 and 256 neurons respectively, an output size of 10 and an input image of shape [32, 32, 3]? (Note that we represent each pixel in a colour image using three real-numbers for the Red, Green and Blue values -- [called \"channels\"](https://www.quora.com/What-do-channels-refer-to-in-a-convolutional-neural-network) -- and hence the 32x32**x3** shape.)\n",
    "\n",
    "ConvNets address this model parameter issue by exploiting structure in the inputs to the network (in particular, by making the assumption that the input is a 3-D *volume*, which applies to images for example, where the 3 dimensions consist of the three RGB channels). The two key differences between a ConvNet and a Feed-forward network are:\n",
    "\n",
    "* ConvNets have neurons that are arranged in 3 dimensions: width, height, depth. Note that *depth* here means channels, i.e. the depth of the input volume, not the depth of a deep neural network!\n",
    "* The neurons in each layer are only connected to a small region of the layer before it.\n",
    "\n",
    "**QUESTION**: Unfortunately there is no such thing as a free lunch. What trade-off do you think a ConvNet makes for the reduction in memory required by fewer parameters?\n",
    "\n",
    "Generally, a ConvNet architecture is made up of different types of layers, the most common being convolutional layers, pooling layers and fully-connected layers that we encountered in the last practical."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SFvYLebFZzGG"
   },
   "source": [
    "### Optional extra reading: The rise of deep convolutional architectures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kEtb0aaeZ6os"
   },
   "source": [
    "ConvNet architectures were key to the tremendous success of deep learning in machine vision. In particular, the first deep learning model to win the ImageNet competition in 2012 was called AlexNet (after Alex Krizhevsky, one of its inventors). It had 5 convolutional layers followed by 3 fully-connected layers. Later winners included GoogLeNet and ResNet. If you're curious, have a look at [this link](https://medium.com/towards-data-science/neural-network-architectures-156e5bad51ba) for a great summary of different ConvNet architectures."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vOp67cfpaTlC"
   },
   "source": [
    "### Convolutional Layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6JbXSVKulDoD"
   },
   "source": [
    "A 2-dimensional convolutional layer maps an input *volume* (meaning, a 3-D input tensor, e.g. [width, height, channels]) to an output *volume* through a set of learnable filters, which make up the parameters of the layer. Every filter is small spatially (along width and height), but extends through the full depth of the input volume. (Eg: A filter in the first layer of a ConvNet might have size [5, 5, 3]). During the forward pass, we convolve (\"slide\") each filter across the width and height of the input volume and compute element-wise dot products between the entries of the filter and the input at any position. As we slide the filter over the width and height of the input volume we will produce a 2-dimensional activation map that gives the responses of that filter at every spatial position. Each convolutional layer will have such a set of filters, and each of them will produce a separate 2-dimensional activation map. We then stack these activation maps along the depth-dimension to produce the output volume.\n",
    "\n",
    "By using these filters which map to a small sub-volume of the input, we can to a large extent, control the parameter explosion that we would get with a (fully-connected) feed-forward network. This **parameter sharing** actually also tends to improve the performance of the model on inputs like natural images because it provides the model with some limited **translation invariance**. Translation invariance means that if the image (or a feature in the image) is translated (moved), the model will not be significantly affected. Think about why this is the case!\n",
    "\n",
    "The following animation illustrates these ideas, make sure you understand them!\n",
    "\n",
    "![Convolution Animation](https://i.stack.imgur.com/FjvuN.gif)\n",
    "\n",
    "If the parameter sharing aspect of CNNs is still not clear, consider the following diagram which compares a simplified 1-D convolutional layer with a fully-connected layer. The diagram shows how a 1-dimensional input  $\\mathbf{x}$ is mapped to a 1-dimensional output  $\\mathbf{y}$ using both a fully-connected layer and a convolution layer, both without bias parameters. The colours of the edges represent the value of the weight parameters in the layers. For the fully-connected layer, the number of weights is the product of the input and output sizes, in this case, $6 \\times 4 = 24$. On the other hand, the number of weights in the convolutional layer depends only on the filter size of the convolution, in this case, $3$, and is independent of the input and output sizes.\n",
    "\n",
    "![Weight Sharing](https://i.imgur.com/gcmmZz4.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eLiuT6TcmXw8"
   },
   "source": [
    "The hyper-parameters of a convolutional layer are as follows:\n",
    "* **Filters** defines the number of filters in the layer\n",
    "* **Kernel Size** defines the width and height of the filters (also called \"kernels\") in the layer. Note that kernels always have the same depth as the inputs to the layer.\n",
    "* **Stride** defines the number of pixels by which we move the filter when \"sliding\" it along the input volume. Typically this value would be 1, but values of 2 and 3 are also sometimes used.\n",
    "* **Padding** refers to the addition of 0-value pixels to the edges of the input volume along the width and height dimensions. In Tensorflow you can set this to \"VALID\", which essentially does no padding or \"SAME\" which pads the input such that the output width and height are the same as the input.\n",
    "\n",
    "Let's look at a very simple, dummy example to see how the values of the hyper-parameters affect the output size of a convolutional layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iluvPEPInWKn"
   },
   "outputs": [],
   "source": [
    "# Create a random colour \"image\" of shape 10x10 with a depth of 3 (for red, green and blue)\n",
    "dummy_input = np.random.uniform(size=[10, 10, 3])\n",
    "fig, ax = plt.subplots(1, 1)\n",
    "plt.imshow(dummy_input)\n",
    "ax.grid(False)\n",
    "print('Input shape: {}'.format(dummy_input.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZU1gD5hnsHbc"
   },
   "source": [
    "Now adjust the hyperparameters using the sliders on the right and see how the output shape changes for a [10, 10, 3] input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OF_KxVpSpE6y"
   },
   "outputs": [],
   "source": [
    "#@title Convolutional layer parameters {run: \"auto\"}\n",
    "filters = 1  #@param { type: \"slider\", min:0, max: 10, step: 1 }\n",
    "kernel_size = 2 #@param { type: \"slider\", min:1, max: 10, step: 1 }\n",
    "stride = 1 #@param { type: \"slider\", min:1, max: 3, step: 1 }\n",
    "\n",
    "conv_layer = tf.keras.layers.Conv2D(\n",
    "    filters=filters,\n",
    "    kernel_size=kernel_size,\n",
    "    strides=stride,\n",
    "    padding=\"valid\",\n",
    "    input_shape=[10, 10, 3])\n",
    "\n",
    "# Convert the image to a tensor and add an extra batch dimension which\n",
    "# the convolutional layer expects.\n",
    "input_tensor = tf.convert_to_tensor(dummy_input[None, :, :, :])\n",
    "convoluted = conv_layer(input_tensor)\n",
    "\n",
    "print('The output dimension is: {}'.format(list(convoluted.shape)[1:]))\n",
    "print('The number of parameters is: {}'.format(conv_layer.count_params()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dzh-9TL1sMCT"
   },
   "source": [
    "Note especially how output width and height are related to ```kernel_size``` and ```stride```, and how the output depth is related to ```filters```.\n",
    "\n",
    "**Question:** Can you come up with a formula for the output shape given the input shape, the hyperparameters of the layer and assuming no padding? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oW5FkOM2EOr3"
   },
   "source": [
    "### Optional extra reading: Building up complex filters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "F__JkeqMEbzv"
   },
   "source": [
    "One of the reasons that CNNs have been so successful is their ability to build up complex filters by composing more simple filters. For example, imagine a 5 layer CNN that has been trained to detect faces. The first 4 layers are convolutional and the last layer is fully-connected and outputs the final prediction (is there a face or not). We might find that the filters in each convolution layer pick out the following features:\n",
    "\n",
    "1. lines (horizontal, vertical, diagonal), and colour gradients,\n",
    "2. corners, circles and other simple shapes, and simple textures,\n",
    "3. noses, mouths, and eyes,\n",
    "4. whole faces.\n",
    "\n",
    "The neural net has learned to pick out complex objects like facial features and even whole faces! The reason for this is that each successive layer can combine the filters from the previous layer to detect more and more sophisticated features. The following diagram (adapted from [this paper](http://web.eecs.umich.edu/~honglak/icml09-ConvolutionalDeepBeliefNetworks.pdf)) shows some really cool examples of this kind of behaviour. The lower level features (shown above) detect noses, eyes, and mouths, in the case of faces, and wheels, doors, and windows, for cars. The higher level features are then able to detect whole faces and cars.\n",
    "\n",
    "![Imgur](https://i.imgur.com/653uIty.jpg)\n",
    "\n",
    "The diagrams on page 7 of [this classic paper](https://cs.nyu.edu/~fergus/papers/zeilerECCV2014.pdf) show more examples of this phenomena and are definitely worth checking out!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LhgDU-fVx2Jv"
   },
   "source": [
    "### (Max) Pooling\n",
    "A pooling layer reduces the spatial size of the representation. There are different reasons why we may want to do this. One is to reduce the number of parameters in the network. Imagine a convnet for the MNIST dataset. If the feature tensor produced by the final conv/pool/relu layer was say, of size 20x20 and had 100 feature channels, the final dense layer would have 20x20x100x10=400k parameters. However, if we down-sampled that layer to a 4x4 spatial size, we would have only 20k parameters. A big difference!\n",
    "\n",
    "Another reason is that we want later features (deeper in the network) to have larger *receptive fields* (input regions that they look at), in order to represent bigger objects and object parts for instance. In particular, pooling stride gives later features much larger receptive fields so that they can effectively combine smaller features together.\n",
    "\n",
    "A pooling layer has no trainable parameters. It applies some 2-D aggregation operation (usually a max(), but others like average() may also be used) to regions of the input volume. This is done independently for each depth dimension of the input. For example, a 2x2 max pooling operation with a stride of 2, downsamples every depth slice of the input by 2 along both the width and height.\n",
    "\n",
    "The hyper-parameters of a pooling layer are as follows:\n",
    "* **Pool Size** defines how many values are aggregated together.\n",
    "* **Stride** defines the number of pixels by which we move the pooling filter when sliding it along the input. Typically this value would be equal to the pool size.\n",
    "* **Padding** refers to the addition of 0-value pixels to the edges of the input volume along the width and height dimensions. In Tensorflow you can set this to \"VALID\", which essentially does no padding or \"SAME\" which pads the input such that the output width and height are the same as the input.\n",
    "\n",
    "#### Question\n",
    "Do 2x2 max-pooling by hand, with a stride of 2, and \"VALID\" padding, on the following 2D input. What is the size of the output?\n",
    "\n",
    "\\begin{bmatrix}\n",
    "  9 & 5 & 4 & 5 & 6 & 4 \\\\\n",
    "  6 & 6 & 3 & 5 & 8 & 2 \\\\\n",
    "  4 & 6 & 9 & 1 & 3 & 6 \\\\\n",
    "  9 & 7 & 1 & 5 & 8 & 1 \\\\\n",
    "  4 & 9 & 9 & 5 & 7 & 3 \\\\\n",
    "  7 & 3 & 6 & 4 & 9 & 1 \n",
    "\\end{bmatrix}\n",
    "\n",
    "\n",
    "Reveal the cell below by double-clicking and running it, to check your answer when you're done!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "e1oIm3Nlb9zY"
   },
   "outputs": [],
   "source": [
    "#@title Answer { display-mode: \"form\" }\n",
    "X = np.array([[9, 5, 4, 5, 6, 4],\n",
    "              [6, 6, 3, 5, 8, 2],\n",
    "              [4, 6, 9, 1, 3, 6],\n",
    "              [9, 7, 1, 5, 8, 1],\n",
    "              [4, 9, 9, 5, 7, 3],\n",
    "              [7, 3, 6, 4, 9, 1]])\n",
    "\n",
    "max_pool_layer = tf.keras.layers.MaxPooling2D((2, 2), strides=2)\n",
    "max_pool_layer(tf.convert_to_tensor(X[None, :, :, None])).numpy().squeeze()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SozPs8Ewx3tg"
   },
   "source": [
    "### Optional extra reading: Receptive fields"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XD8JDduMNh7V"
   },
   "source": [
    "Earlier we mentioned that one reason to do pooling is to increase the sizes of the receptive fields of our features. Let's take a closer look at what we meant by this. \n",
    "\n",
    "The diagram below shows the effective receptive field of one output \"neuron\" in each layer of a few simple networks. What the diagram tells us is how many of the input values have an effect on each output value.\n",
    "\n",
    "We can see that in the first two examples, with single convolutional layers, the receptive field is simply equal to the kernel size. \n",
    "\n",
    "However, the next two examples are a little more interesting. Here we have drastically increased the receptive field size, without a large increase in the number of parameters, by stacking convolution and pooling layers. The interesting thing here is that by using a pooling layer we increased our receptive field size by a much smaller cost (in the number of parameters) than if we'd simply increased the kernel sizes of our convolution layers.\n",
    "\n",
    "You can read more about receptive fields [here](https://medium.com/mlreview/a-guide-to-receptive-field-arithmetic-for-convolutional-neural-networks-e0f514068807).\n",
    "\n",
    "\n",
    "![Receptive Fields](https://i.imgur.com/TjxEsG4.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "e4Vsrgyudd2E"
   },
   "source": [
    "## The CIFAR10 Dataset\n",
    "Now that we understand convolutional, max-pooling and feed-forward layers, we can combine these as building blocks to build a ConvNet classifier for images. For this practical, we will use the colour image dataset CIFAR10 (pronounced \"seefar ten\") which consists of 50,000 training images and 10,000 test images. As we did in Practical 1, we take 10,000 images from the training set to form a validation set and visualise some example images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "flWYFg3ydvMU"
   },
   "outputs": [],
   "source": [
    "cifar = tf.keras.datasets.cifar10\n",
    "(train_images, train_labels), (test_images, test_labels) = cifar.load_data()\n",
    "cifar_labels = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QSzdYWpZd8RE"
   },
   "outputs": [],
   "source": [
    "# Take the last 10000 images from the training set to form a validation set\n",
    "train_labels = train_labels.squeeze()\n",
    "validation_images = train_images[-10000:, :, :]\n",
    "validation_labels = train_labels[-10000:]\n",
    "train_images = train_images[:-10000, :, :]\n",
    "train_labels = train_labels[:-10000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "x8jf1myGQP1O"
   },
   "source": [
    "What are the shapes and data-types of train_images and train_labels?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wzLutGn-P7mg"
   },
   "outputs": [],
   "source": [
    "print('train_images.shape = {}, data-type = {}'.format(train_images.shape, train_images.dtype))\n",
    "print('train_labels.shape = {}, data-type = {}'.format(train_labels.shape, train_labels.dtype))\n",
    "\n",
    "print('validation_images.shape = {}, data-type = {}'.format(validation_images.shape, validation_images.dtype))\n",
    "print('validation_labels.shape = {}, data-type = {}'.format(validation_labels.shape, validation_labels.dtype))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ynwzGIAneBbb"
   },
   "source": [
    "### Visualise examples from the dataset\n",
    "Run the cell below multiple times to see various images. (They might look a bit blurry because we've blown up the small images.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8nMTxCOjd9WW"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,10))\n",
    "for i in range(25):\n",
    "  plt.subplot(5,5,i+1)\n",
    "  plt.xticks([])\n",
    "  plt.yticks([])\n",
    "  plt.grid('off')\n",
    "\n",
    "  img_index = np.random.randint(0, 40000)\n",
    "  plt.imshow(train_images[img_index])\n",
    "  plt.xlabel(cifar_labels[train_labels[img_index]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wN-XUzp-fpyp"
   },
   "source": [
    "## A ConvNet Classifier\n",
    "Finally, we build a simple convolutional architecture to classify the CIFAR images. We will build a mini version of the AlexNet architecture, which consists of 5 convolutional layers with max-pooling, followed by 3 fully-connected layers at the end. In order to investigate the effect each of these two layers has on the number of parameters, we'll build the model in two stages. \n",
    "\n",
    "First, the convolutional layers + max-pooling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "q9zloewLws0b"
   },
   "outputs": [],
   "source": [
    "# Define the convolutinal part of the model architecture using Keras Layers.\n",
    "model = tf.keras.models.Sequential([\n",
    "    tf.keras.layers.Conv2D(filters=48, kernel_size=(3, 3), activation=tf.nn.relu, input_shape=(32, 32, 3), padding='same'),\n",
    "    tf.keras.layers.MaxPooling2D(pool_size=(3, 3)),\n",
    "    tf.keras.layers.Conv2D(filters=128, kernel_size=(3, 3), activation=tf.nn.relu, padding='same'),\n",
    "    tf.keras.layers.MaxPooling2D(pool_size=(3, 3)),\n",
    "    tf.keras.layers.Conv2D(filters=192, kernel_size=(3, 3), activation=tf.nn.relu, padding='same'),\n",
    "    tf.keras.layers.Conv2D(filters=192, kernel_size=(3, 3), activation=tf.nn.relu, padding='same'),\n",
    "    tf.keras.layers.Conv2D(filters=128, kernel_size=(3, 3), activation=tf.nn.relu, padding='same'),\n",
    "    tf.keras.layers.MaxPooling2D(pool_size=(3, 3)),\n",
    "])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VHoTJ0XTgWKN"
   },
   "source": [
    "How many parameters are there in the convolutional part of the architecture? We can easily inspect this using the model summary function in Keras:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "R4Hr3Wgtw72b"
   },
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sbJAaisIgZyX"
   },
   "source": [
    "Now we add a fully-connected part. Note that we also add \"Dropout\" after the first fully-connected layer. Dropout is a regularization technique which randomly zeros out (\"drops\") connections between neurons, and it was one of the key innovations of the AlexNet paper in 2012."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gaHgWaNb1C0W"
   },
   "outputs": [],
   "source": [
    "model.add(tf.keras.layers.Flatten())  # Flatten \"squeezes\" a 3-D volume down into a single vector.\n",
    "model.add(tf.keras.layers.Dense(1024, activation=tf.nn.relu))\n",
    "model.add(tf.keras.layers.Dropout(rate=0.5))\n",
    "model.add(tf.keras.layers.Dense(1024, activation=tf.nn.relu))\n",
    "model.add(tf.keras.layers.Dense(10, activation=tf.nn.softmax))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WOK9_UWx1JkL"
   },
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5uzeJaMB7DPa"
   },
   "source": [
    "### Optional extra reading: Random initialization schemes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AUBKcjfJ6gRs"
   },
   "source": [
    "You might have wondered what values we are using for the initial values of the weights and biases in our model. The short answer is that we typically use random initialization. In this case, we have just been using the default Keras initializers for each layer, which are usually sufficient.\n",
    "\n",
    "The longer answer is that just using completely random numbers does not always work best in practice and that there are a number of common initialization schemes (which are available in most deep learning frameworks such as TensorFlow and Keras).\n",
    "\n",
    "Let's consider a few examples:\n",
    "\n",
    " * When using the ReLU activation it is common to initialize the biases with small positive numbers because this encourages the ReLU activations to start off in the _on_ state, which helps to counteract the _dying ReLU problem_.\n",
    "\n",
    " * The deeper neural networks become the more likely it is that gradients will either shrink to the point that they vanish, or grow to the point that they overflow (the _vanishing_ and _exploding_ gradients problems). To help combat this we can initialize our weights to have a (model-specific) appropriate scale. One method for doing this is called [Xavier or Glorot](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf) initialization.\n",
    "\n",
    " * The _Xavier_ initialization scheme was designed with the _traditional_ activations Sigmoid and TanH in mind and does not work as well for ReLU activations. An alternative is [He or Kaiming](https://arxiv.org/pdf/1502.01852.pdf) initialization which is a modification of Xavier initialization for ReLU activations.\n",
    "\n",
    " [This blog](http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization) goes into more detail on _He_ and _Xavier_ initialization. [The Keras documentation](https://keras.io/initializers/) lists a number of common schemes. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nNMAg7s3W0gg"
   },
   "source": [
    "###Visualizing the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wCL9cH8a3F1W"
   },
   "source": [
    "Let's build a flow-diagram of the model we've constructed to see how information flows between the different layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dFXnOsd0W5pq"
   },
   "outputs": [],
   "source": [
    "tf.keras.utils.plot_model(model, to_file='small_lenet.png', show_shapes=True, show_layer_names=True)\n",
    "display.display(display.Image('small_lenet.png'))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XFrH0OVORco2"
   },
   "source": [
    "### Training and Validating the model\n",
    "In the last practical we wrote out the dataset pipeline, loss function and training-loop to give you a good appreciation for how it works. This time, we use the training loop built-in to Keras. For simple, standard datasets like CIFAR, doing it this way will work fine, but it's important to know what goes on under the hood because you may need to write some or all of the steps out manually when working with more complex datasets! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wJ6JAqUL1TDu"
   },
   "outputs": [],
   "source": [
    "batch_size = 128\n",
    "num_epochs = 10  # The number of epochs (full passes through the data) to train for\n",
    "\n",
    "# Compiling the model adds a loss function, optimiser and metrics to track during training\n",
    "model.compile(optimizer=tf.keras.optimizers.Adam(),\n",
    "              loss=tf.keras.losses.sparse_categorical_crossentropy,\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "# The fit function allows you to fit the compiled model to some training data\n",
    "model.fit(x=train_images,\n",
    "          y=train_labels,\n",
    "          batch_size=batch_size,\n",
    "          epochs=num_epochs,\n",
    "          validation_data=(validation_images, validation_labels.astype(np.float32)))\n",
    "\n",
    "print('Training complete')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "waS75-jNaEL1"
   },
   "source": [
    "### Test performance\n",
    "Finally, we evaluate how well the model does on the held-out test-set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "AVAdjH6o13tQ"
   },
   "outputs": [],
   "source": [
    "metric_values = model.evaluate(x=test_images, y=test_labels)\n",
    "\n",
    "print('Final TEST performance')\n",
    "for metric_value, metric_name in zip(metric_values, model.metrics_names):\n",
    "  print('{}: {}'.format(metric_name, metric_value))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2lsBthNB4FlL"
   },
   "source": [
    "Note that we achieved roughly 80% training set accuracy, but our test accuracy is only around 67%. What do you think may be the reason for this?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UlrSyMQpoV9f"
   },
   "source": [
    "### Classifying examples\n",
    "We now use our trained model to classify a sample of 25 images from the test set. We pass these 25 images to the  ```model.predict``` function, which returns a [25, 10] dimensional matrix. The entry at position $(i, j)$ of this matrix contains the probability that image $i$ belongs to class $j$. We obtain the most-likely prediction using the ```np.argmax``` function which returns the index of the maximum entry along the columns. Finally, we plot the result with the prediction and prediction probability labelled underneath the image and true label on the side. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BjzP384wm9OW"
   },
   "outputs": [],
   "source": [
    "img_indices = np.random.randint(0, len(test_images), size=[25])\n",
    "sample_test_images = test_images[img_indices]\n",
    "sample_test_labels = [cifar_labels[i] for i in test_labels[img_indices].squeeze()]\n",
    "\n",
    "predictions = model.predict(sample_test_images)\n",
    "max_prediction = np.argmax(predictions, axis=1)\n",
    "prediction_probs = np.max(predictions, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ol-f9SacnySQ"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,10))\n",
    "for i, (img, prediction, prob, true_label) in enumerate(\n",
    "    zip(sample_test_images, max_prediction, prediction_probs, sample_test_labels)):\n",
    "  plt.subplot(5,5,i+1)\n",
    "  plt.xticks([])\n",
    "  plt.yticks([])\n",
    "  plt.grid('off')\n",
    "\n",
    "  plt.imshow(img)\n",
    "  plt.xlabel('{} ({:0.3f})'.format(cifar_labels[prediction], prob))\n",
    "  plt.ylabel('{}'.format(true_label))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Z0618tOoMv0Y"
   },
   "source": [
    "### Question\n",
    "What do you think of the model's predictions? Looking at the model's confidence (the probability assigned to the predicted class), look for examples of the following cases:\n",
    "1. The model was correct with high confidence\n",
    "2. The model was correct with low confidence\n",
    "3. The model was incorrect with high confidence\n",
    "4. The model was incorrect with low confidence\n",
    "\n",
    "What do you think the (relative) loss values would be in those cases? \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "py0V6UwC6_kH"
   },
   "source": [
    "### Optional extra reading: Uncertainty in deep learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lpGl8VzY6B3c"
   },
   "source": [
    "Deep neural networks are not considered to be very good at estimating the uncertainty in their predictions. However, knowing your model's uncertainty can be very important for many applications. For example, consider a deep learning tool for diagnosing diseases, in this case, a false negative could have massive impacts on a person's life! We would really like to know how confident our model is in its prediction. This is a budding field of research, for example, see [this blog](https://www.cs.ox.ac.uk/people/yarin.gal/website/blog_3d801aa532c1ce.html) for a nice introduction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OJzCooQO66D3"
   },
   "source": [
    "### Optional extra reading: CNN architectures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "n9Sjwpsm5_TD"
   },
   "source": [
    "Deciding on the architecture for a CNN, i.e. the combination of convolution, pooling, dense, and other layers, can be tricky and often can seem arbitrary. On top of that, one also has to make decisions such as what kind of pooling, which activation functions, and what size of convolution to use, among other things. For new and old practitioners of deep learning, these choices can be overwhelming. \n",
    "\n",
    "However, by examining existing successful CNN architectures we can learn a lot about what works and what doesn't. (We can even apply these existing architectures to our problems since many deep learning libraries, such as TensorFlow and Keras, have them [built in](https://keras.io/applications/#available-models) and it is even possible to fine-tune pre-trained models to our specific problem using [transfer learning](https://cs231n.github.io/transfer-learning/).)\n",
    "\n",
    "[This article](https://medium.com/@sidereal/cnns-architectures-lenet-alexnet-vgg-googlenet-resnet-and-more-666091488df5) describes many of the most successful CNN architectures in recent years, including [ResNet](https://arxiv.org/abs/1512.03385), [Inception](https://arxiv.org/pdf/1512.00567v3.pdf) and [VGG](https://arxiv.org/pdf/1409.1556.pdf). For a more detailed and technical description of these models and more see [these slides](http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture9.pdf). Reading through these resources should give you insights into why these architectures are successful as well as best practices and current trends for CNNs that will help you design your own architectures.\n",
    "\n",
    "For example, one of the practices you might pick up on is the use of 3x3 convolutions. You'll notices that older architectures such as [AlexNet](https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf) used a range of convolutions from 7x7 down to 3x3. However, newer architectures such as VGG and ResNet use 3x3 convolutions almost exclusively. In short, the reason is that stacking 3x3 convolutions gives you the same receptive field as a larger convolution but with more non-linearity. \n",
    "\n",
    "Here are some other questions you may want to think about while investigating these architectures:\n",
    "\n",
    "* Why do modern architectures use less max-pooling?\n",
    "* What does a 1x1 convolution do?\n",
    "* What is a residual connection?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-3bIU8BErhiJ"
   },
   "source": [
    "## Your Tasks\n",
    "1. [**ALL**] Experiment with the network architecture, try changing the numbers, types and sizes of layers, the sizes of filters, using different padding etc. How do these decisions affect the performance of the model? In particular, try building a *fully convolutinoal* network, with no (max-)pooling layers. \n",
    "2. [**ALL**] Add BATCH NORMALISATION ([Tensorflow documentation](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/layers/BatchNormalization) and [research paper](http://proceedings.mlr.press/v37/ioffe15.pdf)) to improve the model's generalisation.\n",
    "3. [**ADVANCED**] Read about Residual networks ([original paper](https://arxiv.org/pdf/1512.03385.pdf), ) and add **shortcut connections** to the model architecture. Try to build a simple reusable \"residual block\" as a [Keras Model](https://www.tensorflow.org/api_docs/python/tf/keras/Model). \n",
    "4. [**OPTIONAL**]. Visualise the filters of the convolutional layers using Matplotlib. **HINT**: You can retrieve a reference to an individual layer from the sequential Keras model by calling```model.get_layer(name)```, replacing \"name\" with the name of the layer. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TPiFyajJtDmj"
   },
   "source": [
    "**IMPORTANT: Please fill out the exit ticket form before you leave the practical: https://forms.gle/LzHwg1BGqdYUdK5q8**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0e-IdtqUknDK"
   },
   "source": [
    "##Additional Resources\n",
    "\n",
    "Here's some more information on ConvNets:\n",
    "\n",
    "* Chris Colah's blog post on [Understanding Convolutions](https://colah.github.io/posts/2014-07-Understanding-Convolutions/)\n",
    "* [How do convolutional neural networks work?](http://brohrer.github.io/how_convolutional_neural_networks_work.html)\n",
    "* The [CS231n course](https://cs231n.github.io/) which is a great resource that covers just about all the basics of CNNs\n",
    "* [Building blocks of interpretability](https://distill.pub/2018/building-blocks/) (some really cool CNN feature visualisations)\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "SFvYLebFZzGG",
    "5uzeJaMB7DPa",
    "py0V6UwC6_kH"
   ],
   "include_colab_link": true,
   "name": "Practical 1a: Convolutional Neural Networks",
   "private_outputs": true,
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5rc1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}