06a-cnns.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Fashion MNIST - multiclass + CNN\n",
    "Now we are back to the **fashion MNIST dataset**. This time we will use a CNN to process the images. For every image, we want to classify it into **10 distinct clothing categories**.\n",
    "\n",
    "The big difference between processing images using a CNN compared to a dense network is that a **CNN** will process the images **\"as images\"**. This roughly means that we process the image by looking at pixels close together in the image and detect patterns in pixel values.\n",
    "\n",
    "In this notebook, we will process the image by using a **convolutional layers**. The **more layers we add, the more complex patterns** can be detected. This approach is more reasonable than a naive approach (dense) since we use way **fewer parameters** by applying the same filter on many locations in the image.\n",
    "\n",
    "Convolutional layers in Keras expect the input to have 3D dimensions `(width, height, colours)` but in the Fashion MNIST dataset each image is 2D, `(width, height)`. This is because the images are grayscale (range from black to white). If an image has colour, which consists of 3 colour values per pixel; Red, Green, Blue, we need to add an additional dimension to our tensor, making it 3D. This means that we need to add one dimension to our data. We will use the reshape function to achieve this. This means that our whole dataset will become a 4D tensor with dimensions `(examples, width, height, colours)` the training set this will have dimension `(60000, 28, 28, 1)`.\n",
    "\n",
    "We will preprocess the data by first scaling the data using **Min/Max scaling**, map the labels to their **one-hot encoding representation** and then reshape the training data. Today we will not split the dataset further as we are provided with the train and test sets. After preprocessing the data we will then create a simple CNN to solve the multiclass classification task by using a **softmax layer** at the end which outputs a 10 class probability distribution and train the CNN using **categorical_crossentropy**. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the data set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(tensorflow)\n",
    "library(keras)\n",
    "source(\"06-helpers.R\")\n",
    "\n",
    "use_multi_cpu(2L)\n",
    "\n",
    "data <- dataset_fashion_mnist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us quickly inspect the dimensions of our dataset so that we see how many examples we have and the size of the images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dim(data$train$x)\n",
    "length(data$train$y)\n",
    "dim(data$test$x)\n",
    "length(data$test$y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are 60000 training images, each one is 28 by 28 pixels. These are grayscale images, so the \"colour\" ranges from 0 (black) to 255 (white). Let's print out a single pixel in the first image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data$train$x[1,15,15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Preprocessing\n",
    "As a first step in our preprocessing we scale the images by making sure that the maximum value each feature (pixel) takes is at most 1 and the minimum is 0. To do this use Min/Max scaling.\n",
    "\n",
    "$$\n",
    "x' = \\frac{x - \\min(\\boldsymbol{x})}{\\max(\\boldsymbol{x}) - \\min(\\boldsymbol{x})}\n",
    "$$\n",
    "\n",
    "Where $x$ is a single example and $x'$ is our new scaled value. $\\min(\\boldsymbol{x})$ is the smallest value in the training set and $\\max(\\boldsymbol{x})$ is the largest value.\n",
    "\n",
    "In this case we know that the minimum pixel value is 0 and the maximum is 255. This means that just need to divide each pixel value by 255."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 Scale the train/test sets\n",
    "Scale the train and test sets by dividing all of the pixel values by 255. For `data$train$x` store the result in `x_train_scaled` and for `data$test$x` store the result in `x_test_scaled`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<FILL IN>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Reshaping\n",
    "Next we reshape our data. We need to add one extra dimension to our data, otherwise the `layer_conv_2d` in Keras will complain. For each image a `(28, 28)` we need to reshape it into `(28, 28, 1)`. This is a change \"on paper\" and will not change anything.\n",
    "\n",
    "[`Reshape`](https://rstudio.github.io/reticulate/reference/array_reshape.html) `x_train_scaled` and `x_test_scaled` to `c(60000, 28, 28, 1)` and `c(10000, 28, 28, 1)`, respectively. Check your work using the dim function. Store the result as `x_train` and `x_test`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<FILL IN>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 One hot-encoding\n",
    "We also need to preprocess the labels of the images. Each image in your dataset is associated with a certain label, what type of clothing it is. \n",
    "\n",
    "| Label | Description |\n",
    "| --- | --- |\n",
    "| 0 | T-shirt/top |\n",
    "| 1 | Trouser |\n",
    "| 2 | Pullover |\n",
    "| 3 | Dress |\n",
    "| 4 | Coat |\n",
    "| 5 | Sandal |\n",
    "| 6 | Shirt |\n",
    "| 7 | Sneaker |\n",
    "| 8 | Bag |\n",
    "| 9 | Ankle boot |\n",
    "\n",
    "These labels range from 0 to 9. We need to map these labels to their corresponding one-hot encodings. We do this using the [`to_categorical()`](https://www.rdocumentation.org/packages/keras/versions/2.2.4/topics/to_categorical) function.\n",
    "\n",
    "Apply this function to `data$train$y` and `data$test$y` and store the output in `y_train` and `y_test`, respectively. Verify your work with the `dim` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<FILL IN>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. The models\n",
    "Now we want to create a few simple CNN models. Let us start with a model which consists of a single convolutional layer.\n",
    "\n",
    "## 2.1 A single conv layer\n",
    "Create an initial model with the following properties.\n",
    "- a 2d convolutional layer, `layer_conv_2d`, with `filters = 32`, `kernel_size = c(3, 3)`, `strides = c(2, 2)` and `padding = 'valid'`. For more information about the 2d conv layer see the [`layer_conv_2d`](https://www.rdocumentation.org/packages/keras/versions/2.2.4/topics/layer_conv_2d) documentation.\n",
    "- After the `layer_conv_2d` apply the `layer_activation` with the `relu` activation function.\n",
    "- Then flatten the output using `layer_flatten`.\n",
    "- Then apply a `layer_dense` with 10 units and `softmax` activation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model <- keras_model_sequential() %>%\n",
    "    <FILL IN>\n",
    "\n",
    "cat(summary(model))\n",
    "\n",
    "model %>% compile(\n",
    "    optimizer = optimizer_adam(lr = 0.01),\n",
    "    loss = \"categorical_crossentropy\",\n",
    "    metric = \"acc\"\n",
    ")\n",
    "\n",
    "history <- model %>% fit(\n",
    "    x = x_train,\n",
    "    y = y_train,\n",
    "    validation_split = 0.2,\n",
    "    epochs = 15,\n",
    "    batch_size = 256,\n",
    "    callbacks=list(Progress$new())\n",
    ")\n",
    "\n",
    "plot(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us decipher this plot. \n",
    "- training loss continues to go down, this seems to imply that our model is sufficiently expressive.\n",
    "- validation loss goes up in the end, we are overfitting.\n",
    "- validation accuracy is fairly stable, our overfitting has not impacted our accuracy, yet.\n",
    "\n",
    "Let us try to reduce overfitting in the next step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Regularise\n",
    "Let us add some dropout to our model in hopes of reducing overfitting. Copy the model from above and add `layer_dropout` after the `relu` activation. We used a dropout of `0.4` and discuss those results below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at this plot. We can see:\n",
    "- Training loss seems to reduce slower now\n",
    "- Validation loss is stable, thus less overfitting.\n",
    "\n",
    "It seems that we could train this model longer and achieve better performance, but instead let us increase the complexity of the model by adding an additional convolutional layer. This will allow the model to detect more complex patterns which will undoubtably improve the performance of the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 Two convolutional layers\n",
    "Let us try increasing the model complexity by adding one more `layer_conv_2d` and `layer_activation` with `relu` to our model.\n",
    "Copy the model from above and:\n",
    "- add another `layer_conv_2d` and `layer_activation` with `relu` directly after the first `dropout` activation. Use the same parameters as first layer. Remove the `input_shape` parameter.\n",
    "- Add dropout with 0.2 after the new `relu` activation.\n",
    "- Reduce the dropout we specified earlier to 0.2.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this plot we can see.\n",
    "- Training loss and validation loss are close and decreasing slowly, this implies that we might want to increase model complexity to see if we can further improve our model.\n",
    "\n",
    "We have improved our model somewhat from the initial model and we have without a doubt a better model than the dense network we trained in notebook 03a. You can confirm this be evaluating the model below.\n",
    "\n",
    "We can continue improving this model, and we leave this as a bonus exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model %>% evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we performed multi-class classification on the Fashion MNIST dataset. Today we processed the images using convolutional layers instead of a dense network. This allowed us to extra local pixel patterns and combine these patterns to create more complex patters. We started with a simple model using a single convolutional layer which we then slowly improved."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bonus exercise\n",
    "Continue improving the model from 2.3. Try getting above 90% accuracy. Consider these options\n",
    "- Add more convolutional layers (with activations)\n",
    "- Add batch normalisation\n",
    "- Increase the number of filters in later convolutional layers. \n",
    "- Add a dense later after all the convolutions but before the softmax.\n",
    "- Use \"same\" padding in early convolutions.\n",
    "- Adding more epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}