03b-regression-regularisation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Fashion MNIST - multiclass + improved\n",
    "In this notebook we will work, yet again, with the Fashion MNIST dataset. We will start with simple model (the same as we used last time) which fits the training data quite well.\n",
    "\n",
    "We start by apply batch normalisation and we will see how batch normalisation allows us to train the models faster by reducing the training error faster and more consistently (not as sensitive to hyperparameters). After that we add a dropout layer. This will have a regularising effect and reduce overfitting. Lastly, we apply L2 regularisation.\n",
    "\n",
    "As usual we start by loading our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(keras)\n",
    "source(\"02-helpers.R\")\n",
    "\n",
    "use_multi_cpu()\n",
    "\n",
    "data <- dataset_fashion_mnist()\n",
    "data_train <- data$train\n",
    "data_test <- data$test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess\n",
    "Now we will preprocess the data for you. You can check the lines below and see if they make sense to you.\n",
    "- Scaling the pixel values\n",
    "- Map the labels to one-hot encodings\n",
    "- Split the training set to 80% train, 20% test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train <- data_train$x / 255\n",
    "x_test <- data_test$x / 255\n",
    "y_train = to_categorical(y = data_train$y, num_classes = 10)\n",
    "y_test = to_categorical(y = data_test$y, num_classes = 10)\n",
    "\n",
    "index <- 0.8 * dim(y_train)[1]\n",
    "x_val <- x_train[-(1:index),,]\n",
    "y_val <- y_train[-(1:index),]\n",
    "\n",
    "x_train <- x_train[1:index,,]\n",
    "y_train <- y_train[1:index,]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Baseline\n",
    "To begin with, let us start with a simple model so that we have a baseline to compare with.\n",
    "\n",
    "This is the same model as the basic one we created in the previous notebook. It will actually fit the data quite well but we still want to see if we can improve it and see the effects of batch normalisation, dropout and L2 regularisation. There is one slight difference between this network and the one we had before, the learning rate is slightly higher and you will see that the network will learn very slowly after a few epochs.\n",
    "\n",
    "## Exercise 1\n",
    "Create a simple model by performing the following steps\n",
    "- Flatten the input\n",
    "- Dense layer with 256 units\n",
    "- ReLu activation\n",
    "- Another dense layer with 10 units (1 for each class)\n",
    "- Softmax activation\n",
    "\n",
    "We have provided you with a skeleton which defines the compile step and fit step. You only need to fill in the layers. You should not have to change the batch size or any other hyperparameter.\n",
    "\n",
    "The layers you need\n",
    "\n",
    "    layer_dense()\n",
    "    layer_flatten()\n",
    "    layer_activation_relu()\n",
    "    layer_activation_softmax()\n",
    "\n",
    "**Hint:**\n",
    "For these exercises it is better to apply the activations as a separate layer, instead of adding it to the dense layer. That is, instead of doing \n",
    "\n",
    "    layer_dense(units = 256, activation = \"relu\")\n",
    "    \n",
    "do \n",
    "    \n",
    "    layer_dense(units = 256) %>% \n",
    "    layer_activation_relu()\n",
    "This will allow us to add the batch normalisation more easily between these two layers in a later exercises."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model <- keras_model_sequential() %>%\n",
    "    <FILL IN>\n",
    "\n",
    "cat(summary(model))\n",
    "\n",
    "model %>% compile(\n",
    "    optimizer = optimizer_adam(lr = 0.01),\n",
    "    loss = \"categorical_crossentropy\",\n",
    "    metrics = c(\"accuracy\")\n",
    ")\n",
    "\n",
    "history <- model %>% fit(\n",
    "    x = x_train,\n",
    "    y = y_train,\n",
    "    validation_data = list(x_val, y_val),\n",
    "    epochs = 80,\n",
    "    batch_size = 8096,\n",
    "    callbacks=list(Progress$new())\n",
    ")\n",
    "plot(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this plot you should see that our training is slow as the training quickly slows down. You should also see that the training accuracy is slightly lower than the validation accuracy. \n",
    "\n",
    "The training accuracy should be around 0.91-0.92 and validation accuracy around 0.87-0.88."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Batch normalisation\n",
    "The first step we take in improving our network will be to speed up learning. Batch normalisation has been shown to make networks more robust to hyperparameter selection and a larger learning rate can usually be used with batch normalisation.\n",
    "\n",
    "The batch normalisation layer will normalise the output of the dense layer to have mean 0 and variance 1 and then scale out the activation based on the learnt beta and gamma parameters.\n",
    "\n",
    "## Exercise 2.1\n",
    "Copy the baseline network you created in excercise 1 to the cell below and add a `layer_batch_normalization` after the `layer_dense` and before the `layer_activation_relu`. Then run the model and see next question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A small note on parameters for the interested\n",
    "When using batch normalisation 1024 parameters are added to the model, 512 of which are \"non-trainable\". The 512 trainable parameters are the beta and gamma parameters for 256 units. The non-trainable parameters are the standard deviation and mean of a mini-batch computed for each 256 unit. These are considered as parameters to the model but are not trained but rather computed for each iteration."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2.2\n",
    "If you compare this graph to the baseline. In particular, what do you notice with the training accuracy? Do you think it is reasonable to assume that the learning rate might have been too large for the baseline model (you can also experiment with it)? Do you think that the Batch normalisation layer is helpful in this case? What about the validation accuracy? Why do you think it goes down? Do you think we should reduce the number of epochs?\n",
    "\n",
    "Answer these questions with full sentences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dropout\n",
    "Let us improve our batch normalisation network by adding some dropout. Dropout has a regularising effect on the model, that is, it will reduce overfitting. We should notice reduced training accuracy and usually a higher validation accuracy.\n",
    "\n",
    "## Exercise 3.1\n",
    "Copy the baseline network you created in exercise 2.1 to the cell below and add a `layer_dropout(rate = ?)` after `layer_activation_relu`. If you reduced the number of epochs in the model in exercise 2.1, keep that number of epochs.\n",
    "\n",
    "Dropout takes a single parameter, the \"drop-rate\". This is the fraction of neurons in the layer it will set to 0. If we set the rate to 1, we will set all the activations to 0. If we set the rate to 0 we will let the activations remain the same. The best performing values are usually between 0.2 and 0.5.\n",
    "\n",
    "Try out a few values, but do not try to close the gap between the training accuracy and validation accuracy completely (you can't)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 3.2\n",
    "If you compare this graph to the batch normalisation network graph, what do you notice? Do you think we are reducing overfitt? Has our validation accuracy increased? Should we prefer this model over the baseline? Do you think that we could do away with fewer epochs? Try answering these quests with full sentences in the cell below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## L2 Regularisation\n",
    "Let us now close the gap between our training and validation set by using L2 regularisation. \n",
    "\n",
    "L2 regularisation associates costs to the weights our network learns. We need to select a hyperparameter, lambda, which is a discount factor of the L2 term. If we set lambda too high, the network will not be able to update the weights large enough. If we set lambda too low, it will have no regularising effects and we will continue to overfit.\n",
    "\n",
    "Notice that when we train using L2 regularisation, the L2 term will not affect the loss function much at first, since most weights have value around 0 (weights are initialised with mean 0 and variance 1, bias 0). When the network trains more and more, the weights for certain features will grow (positively or negatively) and the L2 term will grow as well and become a greater factor of the overall loss. Our goal is to make the L2 term start growing when we would have otherwise started overfitting.\n",
    "\n",
    "Let us try it out!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 4.1\n",
    "Copy the baseline network you created in exercise 2.1, not the one from 3.1 (Dropout) and add `kernel_regularizer = regularizer_l2(l = lambda)` as an additional parameter to `layer_dense`. You will need to set the value of `lambda`.\n",
    "\n",
    "    layer_dense(units = 128, kernel_regularizer = regularizer_l2(l = lambda))\n",
    "    \n",
    "Try a few different values for lambda, 0.0001 up to 0.0000001, using 80 epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Things are getting hairy! You should notice that the loss and accuracy have high variance now."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 4.2\n",
    "What do you notice when using lambda = 0.0001? And what about 0.0000001? Try answering these quiestion in full sentences below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 4.3\n",
    "Which model do you think is better, the one which uses dropout or L2 regularisation? Do you think that L2 regularisation is helping us in this case? Do you think that we would be better off using fewer epochs? Select the model which you think is best and evaluate it using the cell below. What is your score?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model %>% evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bonus exercise\n",
    "Like in the previous notebook, try improving this model by making the network thinner but deeper. Try using batch normalisation and dropout at multiple (correct/meaningful) locations in the network. Lastly, try using different optimisers, a more basic one, such as [mini-batch gradient descent](https://keras.rstudio.com/reference/optimizer_sgd.html) or [RMSprop](https://keras.rstudio.com/reference/optimizer_rmsprop.html) in your network. Do you notice any differences?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model <- keras_model_sequential() %>%\n",
    "    <FILL IN>\n",
    "\n",
    "cat(summary(model))\n",
    "\n",
    "model %>% compile(\n",
    "    optimizer = optimizer_adam(lr = 0.01),\n",
    "    # optimizer = optimizer_rmsprop (lr = 0.01),\n",
    "    # optimizer = optimizer_sgd(lr = 0.01),\n",
    "    loss = \"categorical_crossentropy\",\n",
    "    metrics = c(\"accuracy\")\n",
    ")\n",
    "\n",
    "history <- model %>% fit(\n",
    "    x = x_train,\n",
    "    y = y_train,\n",
    "    validation_data = list(x_val, y_val),\n",
    "    epochs = 80,\n",
    "    batch_size = 8096,\n",
    "    callbacks=list(Progress$new())\n",
    ")\n",
    "plot(history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}