17-Layer-wise-Differential-learning-rates.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TqOt6Sv7AsMi"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import os\n",
        "import tensorflow as tf"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ro4oYaEmxe4r"
      },
      "outputs": [],
      "source": [
        "_URL = 'https://storage.googleapis.com/mledu-datasets/cats_and_dogs_filtered.zip'\n",
        "path_to_zip = tf.keras.utils.get_file('cats_and_dogs.zip', origin=_URL, extract=True)\n",
        "PATH = os.path.join(os.path.dirname(path_to_zip), 'cats_and_dogs_filtered')\n",
        "\n",
        "train_dir = os.path.join(PATH, 'train')\n",
        "validation_dir = os.path.join(PATH, 'validation')\n",
        "\n",
        "BATCH_SIZE = 32\n",
        "IMG_SIZE = (160, 160)\n",
        "\n",
        "train_dataset = tf.keras.utils.image_dataset_from_directory(train_dir,\n",
        "                                                            shuffle=True,\n",
        "                                                            batch_size=BATCH_SIZE,\n",
        "                                                            image_size=IMG_SIZE)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cAvtLwi7_J__"
      },
      "outputs": [],
      "source": [
        "validation_dataset = tf.keras.utils.image_dataset_from_directory(validation_dir,\n",
        "                                                                 shuffle=True,\n",
        "                                                                 batch_size=BATCH_SIZE,\n",
        "                                                                 image_size=IMG_SIZE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EZqCX_mpV3Mx"
      },
      "source": [
        "As the original dataset doesn't contain a test set, you will create one. To do so, determine how many batches of data are available in the validation set using `tf.data.experimental.cardinality`, then move 20% of them to a test set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uFFIYrTFV9RO"
      },
      "outputs": [],
      "source": [
        "val_batches = tf.data.experimental.cardinality(validation_dataset)\n",
        "test_dataset = validation_dataset.take(val_batches // 5)\n",
        "validation_dataset = validation_dataset.skip(val_batches // 5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Q9pFlFWgBKgH"
      },
      "outputs": [],
      "source": [
        "print('Number of validation batches: %d' % tf.data.experimental.cardinality(validation_dataset))\n",
        "print('Number of test batches: %d' % tf.data.experimental.cardinality(test_dataset))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MakSrdd--RKg"
      },
      "source": [
        "### Configure the dataset for performance"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "p3UUPdm86LNC"
      },
      "outputs": [],
      "source": [
        "AUTOTUNE = tf.data.AUTOTUNE\n",
        "\n",
        "train_dataset = train_dataset.prefetch(buffer_size=AUTOTUNE)\n",
        "validation_dataset = validation_dataset.prefetch(buffer_size=AUTOTUNE)\n",
        "test_dataset = test_dataset.prefetch(buffer_size=AUTOTUNE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MYfcVwYLiR98"
      },
      "source": [
        "### Use data augmentation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bDWc5Oad1daX"
      },
      "source": [
        "When you don't have a large image dataset, it's a good practice to artificially introduce sample diversity by applying random, yet realistic, transformations to the training images, such as rotation and horizontal flipping. This helps expose the model to different aspects of the training data and reduce [overfitting](https://www.tensorflow.org/tutorials/keras/overfit_and_underfit). You can learn more about data augmentation in this [tutorial](https://www.tensorflow.org/tutorials/images/data_augmentation)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3P99QiMGit1A"
      },
      "outputs": [],
      "source": [
        "data_augmentation = tf.keras.Sequential([\n",
        "  tf.keras.layers.RandomFlip('horizontal'),\n",
        "  tf.keras.layers.RandomRotation(0.2),\n",
        "])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bAywKtuVn8uK"
      },
      "source": [
        "### Rescale pixel values\n",
        "\n",
        "In a moment, you will download `tf.keras.applications.MobileNetV2` for use as your base model. This model expects pixel values in `[-1, 1]`, but at this point, the pixel values in your images are in `[0, 255]`. To rescale them, use the preprocessing method included with the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cO0HM9JAQUFq"
      },
      "outputs": [],
      "source": [
        "preprocess_input = tf.keras.applications.mobilenet_v2.preprocess_input"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OkH-kazQecHB"
      },
      "source": [
        "## Create the base model from the pre-trained convnets\n",
        "You will create the base model from the **MobileNet V2** model developed at Google. This is pre-trained on the ImageNet dataset, a large dataset consisting of 1.4M images and 1000 classes. ImageNet is a research training dataset with a wide variety of categories like `jackfruit` and `syringe`. This base of knowledge will help us classify cats and dogs from our specific dataset.\n",
        "\n",
        "First, you need to pick which layer of MobileNet V2 you will use for feature extraction. The very last classification layer (on \"top\", as most diagrams of machine learning models go from bottom to top) is not very useful. Instead, you will follow the common practice to depend on the very last layer before the flatten operation. This layer is called the \"bottleneck layer\". The bottleneck layer features retain more generality as compared to the final/top layer.\n",
        "\n",
        "First, instantiate a MobileNet V2 model pre-loaded with weights trained on ImageNet. By specifying the **include_top=False** argument, you load a network that doesn't include the classification layers at the top, which is ideal for feature extraction."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "19IQ2gqneqmS"
      },
      "outputs": [],
      "source": [
        "# Create the base model from the pre-trained model MobileNet V2\n",
        "IMG_SHAPE = IMG_SIZE + (3,)\n",
        "base_model = tf.keras.applications.MobileNetV2(input_shape=IMG_SHAPE,\n",
        "                                               include_top=False,\n",
        "                                               weights='imagenet')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AqcsxoJIEVXZ"
      },
      "source": [
        "This feature extractor converts each `160x160x3` image into a `5x5x1280` block of features. Let's see what it does to an example batch of images:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rlx56nQtfe8Y"
      },
      "source": [
        "## Feature extraction\n",
        "In this step, you will freeze the convolutional base created from the previous step and to use as a feature extractor. Additionally, you add a classifier on top of it and train the top-level classifier."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CnMLieHBCwil"
      },
      "source": [
        "### Freeze the convolutional base"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7fL6upiN3ekS"
      },
      "source": [
        "It is important to freeze the convolutional base before you compile and train the model. Freezing (by setting layer.trainable = False) prevents the weights in a given layer from being updated during training. MobileNet V2 has many layers, so setting the entire model's `trainable` flag to False will freeze all of them."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wdMRM8YModbk"
      },
      "source": [
        "### Add a classification head"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DgzQX6Veb2WT"
      },
      "outputs": [],
      "source": [
        "global_average_layer = tf.keras.layers.GlobalAveragePooling2D()\n",
        "prediction_layer = tf.keras.layers.Dense(1)\n",
        "\n",
        "inputs = tf.keras.Input(shape=(160, 160, 3))\n",
        "x = data_augmentation(inputs)\n",
        "x = preprocess_input(x)\n",
        "x = base_model(x)\n",
        "x = global_average_layer(x)\n",
        "x = tf.keras.layers.Dropout(0.2)(x)\n",
        "outputs = prediction_layer(x)\n",
        "model = tf.keras.Model(inputs, outputs)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install tensorflow_addons"
      ],
      "metadata": {
        "id": "t2N8RjnRX_gv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RpR8HdyMhukJ"
      },
      "outputs": [],
      "source": [
        "import tensorflow_addons as tfa\n",
        "\n",
        "optimizers = [tf.keras.optimizers.Adam(learning_rate=0.000001),\n",
        "              tf.keras.optimizers.Adam(learning_rate=0.00001),\n",
        "              tf.keras.optimizers.Adam(learning_rate=0.0001)]\n",
        "\n",
        "optimizers_and_layers = [(optimizers[0], base_model.layers[:100]),\n",
        "                         (optimizers[1], base_model.layers[100:]),\n",
        "                         (optimizers[2], model.layers[-1])\n",
        "                         ]\n",
        "\n",
        "opt = tfa.optimizers.MultiOptimizer(optimizers_and_layers)\n",
        "model.compile(opt,\n",
        "              loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),\n",
        "              metrics=[tf.keras.metrics.BinaryAccuracy(threshold=0, name='accuracy')])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RxvgOYTDSWTx"
      },
      "source": [
        "### Train the model\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Om4O3EESkab1"
      },
      "outputs": [],
      "source": [
        "epochs = 20\n",
        "\n",
        "loss0, accuracy0 = model.evaluate(validation_dataset)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JsaRFlZ9B6WK"
      },
      "outputs": [],
      "source": [
        "history = model.fit(train_dataset,\n",
        "                    epochs=epochs,\n",
        "                    validation_data=validation_dataset)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hd94CKImf8vi"
      },
      "source": [
        "### Learning curves\n",
        "\n",
        "Let's take a look at the learning curves of the training and validation accuracy/loss when using the MobileNetV2 base model as a fixed feature extractor."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "53OTCh3jnbwV"
      },
      "outputs": [],
      "source": [
        "acc = history.history['accuracy']\n",
        "val_acc = history.history['val_accuracy']\n",
        "\n",
        "loss = history.history['loss']\n",
        "val_loss = history.history['val_loss']\n",
        "\n",
        "plt.figure(figsize=(8, 8))\n",
        "plt.subplot(2, 1, 1)\n",
        "plt.plot(acc, label='Training Accuracy')\n",
        "plt.plot(val_acc, label='Validation Accuracy')\n",
        "plt.legend(loc='lower right')\n",
        "plt.ylabel('Accuracy')\n",
        "plt.ylim([min(plt.ylim()),1])\n",
        "plt.title('Training and Validation Accuracy')\n",
        "\n",
        "plt.subplot(2, 1, 2)\n",
        "plt.plot(loss, label='Training Loss')\n",
        "plt.plot(val_loss, label='Validation Loss')\n",
        "plt.legend(loc='upper right')\n",
        "plt.ylabel('Cross Entropy')\n",
        "plt.ylim([0,1.0])\n",
        "plt.title('Training and Validation Loss')\n",
        "plt.xlabel('epoch')\n",
        "plt.show()"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "transfer_learning.ipynb",
      "private_outputs": true,
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}