12_Transfer_learning_part1.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "irxcLXiLy7zs"
      },
      "source": [
        "<center><div style=\"direction:rtl;font-family:B Lotus, B Nazanin, Tahoma\">به نام خدا</div></center>\n",
        "<h1><center><div style=\"direction:rtl;font-family:B Lotus, B Nazanin, Tahoma\"> Transfer Learning - Part 1</h1>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": true,
        "id": "avDL8zTby7zu"
      },
      "source": [
        "<div style=\"direction:rtl;text-align:right;font-family:Tahoma\">\n",
        "در این نوت‌بوک رویکرد نسخت Transfer Learning یعنی استفاده به عنوان یک Feature extractor بررسی خواهد شد.\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "id": "XoO_5Lwdy7zv"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import matplotlib.pyplot as plt\n",
        "from tensorflow import keras"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tPK6xuBay7zx"
      },
      "source": [
        "<div style=\"direction:rtl;text-align:right;font-family:Tahoma\">\n",
        "کدها  برگرفته از فصل پنج کتاب\n",
        "</div>\n",
        "\n",
        "[Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python?a_aid=keras&a_bid=76564dff)\n",
        "\n",
        "<div style=\"direction:rtl;text-align:right;font-family:Tahoma\">\n",
        "و گیت هاب نویسنده کتاب و توسعه دهنده کراس\n",
        "</div>\n",
        "\n",
        "[François Chollet](http://nbviewer.jupyter.org/github/fchollet/deep-learning-with-python-notebooks/blob/master/5.3-using-a-pretrained-convnet.ipynb)\n",
        "\n",
        "<div style=\"direction:rtl;text-align:right;font-family:Tahoma\">است.</div>\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "id": "aej7Ue9Wy7z4"
      },
      "outputs": [],
      "source": [
        "from keras.applications import VGG16\n",
        "\n",
        "conv_base = VGG16(weights='imagenet',\n",
        "                  include_top=False,\n",
        "                  input_shape=(150, 150, 3))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5fnqca66y7z6"
      },
      "source": [
        "We passed three arguments to the constructor:\n",
        "\n",
        "* `weights`, to specify which weight checkpoint to initialize the model from\n",
        "* `include_top`, which refers to including or not the densely-connected classifier on top of the network. By default, this\n",
        "densely-connected classifier would correspond to the 1000 classes from ImageNet. Since we intend to use our own densely-connected\n",
        "classifier (with only two classes, cat and dog), we don't need to include it.\n",
        "* `input_shape`, the shape of the image tensors that we will feed to the network. This argument is purely optional: if we don't pass it,\n",
        "then the network will be able to process inputs of any size.\n",
        "\n",
        "Here's the detail of the architecture of the VGG16 convolutional base: it's very similar to the simple convnets that you are already\n",
        "familiar with."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "id": "IJ_0E7Z7y7z9",
        "outputId": "817840fb-0550-4e8b-ab9c-fab87f6c7d15",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"vgg16\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_2 (InputLayer)        [(None, 150, 150, 3)]     0         \n",
            "                                                                 \n",
            " block1_conv1 (Conv2D)       (None, 150, 150, 64)      1792      \n",
            "                                                                 \n",
            " block1_conv2 (Conv2D)       (None, 150, 150, 64)      36928     \n",
            "                                                                 \n",
            " block1_pool (MaxPooling2D)  (None, 75, 75, 64)        0         \n",
            "                                                                 \n",
            " block2_conv1 (Conv2D)       (None, 75, 75, 128)       73856     \n",
            "                                                                 \n",
            " block2_conv2 (Conv2D)       (None, 75, 75, 128)       147584    \n",
            "                                                                 \n",
            " block2_pool (MaxPooling2D)  (None, 37, 37, 128)       0         \n",
            "                                                                 \n",
            " block3_conv1 (Conv2D)       (None, 37, 37, 256)       295168    \n",
            "                                                                 \n",
            " block3_conv2 (Conv2D)       (None, 37, 37, 256)       590080    \n",
            "                                                                 \n",
            " block3_conv3 (Conv2D)       (None, 37, 37, 256)       590080    \n",
            "                                                                 \n",
            " block3_pool (MaxPooling2D)  (None, 18, 18, 256)       0         \n",
            "                                                                 \n",
            " block4_conv1 (Conv2D)       (None, 18, 18, 512)       1180160   \n",
            "                                                                 \n",
            " block4_conv2 (Conv2D)       (None, 18, 18, 512)       2359808   \n",
            "                                                                 \n",
            " block4_conv3 (Conv2D)       (None, 18, 18, 512)       2359808   \n",
            "                                                                 \n",
            " block4_pool (MaxPooling2D)  (None, 9, 9, 512)         0         \n",
            "                                                                 \n",
            " block5_conv1 (Conv2D)       (None, 9, 9, 512)         2359808   \n",
            "                                                                 \n",
            " block5_conv2 (Conv2D)       (None, 9, 9, 512)         2359808   \n",
            "                                                                 \n",
            " block5_conv3 (Conv2D)       (None, 9, 9, 512)         2359808   \n",
            "                                                                 \n",
            " block5_pool (MaxPooling2D)  (None, 4, 4, 512)         0         \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 14,714,688\n",
            "Trainable params: 14,714,688\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "conv_base.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BkFjdhety7z-"
      },
      "source": [
        "The final feature map has shape `(4, 4, 512)`. That's the feature on top of which we will stick a densely-connected classifier.\n",
        "\n",
        "At this point, there are two ways we could proceed:\n",
        "\n",
        "* Running the convolutional base over our dataset, recording its output to a Numpy array on disk, then using this data as input to a\n",
        "standalone densely-connected classifier similar to those you have seen in the first chapters of this book. This solution is very fast and\n",
        "cheap to run, because it only requires running the convolutional base once for every input image, and the convolutional base is by far the\n",
        "most expensive part of the pipeline. However, for the exact same reason, this technique would not allow us to leverage data augmentation at\n",
        "all.\n",
        "* Extending the model we have (`conv_base`) by adding `Dense` layers on top, and running the whole thing end-to-end on the input data. This\n",
        "allows us to use data augmentation, because every input image is going through the convolutional base every time it is seen by the model.\n",
        "However, for this same reason, this technique is far more expensive than the first one.\n",
        "\n",
        "We will cover both techniques. Let's walk through the code required to set-up the first one: recording the output of `conv_base` on our\n",
        "data and using these outputs as inputs to a new model.\n",
        "\n",
        "We will start by simply running instances of the previously-introduced `ImageDataGenerator` to extract images as Numpy arrays as well as\n",
        "their labels. We will extract features from these images simply by calling the `predict` method of the `conv_base` model."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "#for run in PC\n",
        "base_dir = 'D:/dataset/catDog/catVsdog'\n",
        "train_dir = os.path.join(base_dir, 'train')\n",
        "validation_dir = os.path.join(base_dir, 'validation')\n",
        "test_dir = os.path.join(base_dir, 'test')"
      ],
      "metadata": {
        "id": "zH_JzKGbzSGg"
      },
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#or"
      ],
      "metadata": {
        "id": "Dena4DsCzVVj"
      },
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import os, shutil\n",
        "#for run in Colab\n",
        "\n",
        "!mkdir ~/.kaggle\n",
        "!mv kaggle.json ~/.kaggle\n",
        "! kaggle competitions download -c dogs-vs-cats\n",
        "!unzip dogs-vs-cats.zip\n",
        "!unzip -qq train.zip\n",
        "!rm -r dogs-vs-cats.zip sampleSubmission.csv test1.zip train.zip sample_data/\n",
        "\n",
        "# The path to the directory where the original\n",
        "# dataset was uncompressed\n",
        "original_dataset_dir = '/content/train'\n",
        "\n",
        "# The directory where we will\n",
        "# store our smaller dataset\n",
        "base_dir = '/content/subset'\n",
        "os.makedirs(base_dir, exist_ok=True)\n",
        "\n",
        "\n",
        "# Directories for our training,\n",
        "# validation and test splits\n",
        "train_dir = os.path.join(base_dir, 'train')\n",
        "os.mkdir(train_dir)\n",
        "validation_dir = os.path.join(base_dir, 'validation')\n",
        "os.mkdir(validation_dir)\n",
        "test_dir = os.path.join(base_dir, 'test')\n",
        "os.mkdir(test_dir)\n",
        "# Directory with our training cat pictures\n",
        "train_cats_dir = os.path.join(train_dir, 'cats')\n",
        "os.mkdir(train_cats_dir)\n",
        "\n",
        "# Directory with our training dog pictures\n",
        "train_dogs_dir = os.path.join(train_dir, 'dogs')\n",
        "os.mkdir(train_dogs_dir)\n",
        "\n",
        "# Directory with our validation cat pictures\n",
        "validation_cats_dir = os.path.join(validation_dir, 'cats')\n",
        "os.mkdir(validation_cats_dir)\n",
        "\n",
        "# Directory with our validation dog pictures\n",
        "validation_dogs_dir = os.path.join(validation_dir, 'dogs')\n",
        "os.mkdir(validation_dogs_dir)\n",
        "\n",
        "# Directory with our validation cat pictures\n",
        "test_cats_dir = os.path.join(test_dir, 'cats')\n",
        "os.mkdir(test_cats_dir)\n",
        "\n",
        "# Directory with our validation dog pictures\n",
        "test_dogs_dir = os.path.join(test_dir, 'dogs')\n",
        "os.mkdir(test_dogs_dir)\n",
        "\n",
        "# Copy first 1000 cat images to train_cats_dir\n",
        "fnames = ['cat.{}.jpg'.format(i) for i in range(1000)]\n",
        "for fname in fnames:\n",
        "    src = os.path.join(original_dataset_dir, fname)\n",
        "    dst = os.path.join(train_cats_dir, fname)\n",
        "    shutil.copyfile(src, dst)\n",
        "\n",
        "# Copy next 500 cat images to validation_cats_dir\n",
        "fnames = ['cat.{}.jpg'.format(i) for i in range(1000, 1500)]\n",
        "for fname in fnames:\n",
        "    src = os.path.join(original_dataset_dir, fname)\n",
        "    dst = os.path.join(validation_cats_dir, fname)\n",
        "    shutil.copyfile(src, dst)\n",
        "\n",
        "# Copy next 500 cat images to test_cats_dir\n",
        "fnames = ['cat.{}.jpg'.format(i) for i in range(1500, 2000)]\n",
        "for fname in fnames:\n",
        "    src = os.path.join(original_dataset_dir, fname)\n",
        "    dst = os.path.join(test_cats_dir, fname)\n",
        "    shutil.copyfile(src, dst)\n",
        "\n",
        "# Copy first 1000 dog images to train_dogs_dir\n",
        "fnames = ['dog.{}.jpg'.format(i) for i in range(1000)]\n",
        "for fname in fnames:\n",
        "    src = os.path.join(original_dataset_dir, fname)\n",
        "    dst = os.path.join(train_dogs_dir, fname)\n",
        "    shutil.copyfile(src, dst)\n",
        "\n",
        "# Copy next 500 dog images to validation_dogs_dir\n",
        "fnames = ['dog.{}.jpg'.format(i) for i in range(1000, 1500)]\n",
        "for fname in fnames:\n",
        "    src = os.path.join(original_dataset_dir, fname)\n",
        "    dst = os.path.join(validation_dogs_dir, fname)\n",
        "    shutil.copyfile(src, dst)\n",
        "\n",
        "# Copy next 500 dog images to test_dogs_dir\n",
        "fnames = ['dog.{}.jpg'.format(i) for i in range(1500, 2000)]\n",
        "for fname in fnames:\n",
        "    src = os.path.join(original_dataset_dir, fname)\n",
        "    dst = os.path.join(test_dogs_dir, fname)\n",
        "    shutil.copyfile(src, dst)"
      ],
      "metadata": {
        "id": "ThUiHW_zzYSC",
        "outputId": "a12fdcfb-358e-401b-a59d-005ffb10f92b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "mkdir: cannot create directory ‘/root/.kaggle’: File exists\n",
            "Warning: Your Kaggle API key is readable by other users on this system! To fix this, you can run 'chmod 600 /root/.kaggle/kaggle.json'\n",
            "Downloading dogs-vs-cats.zip to /content\n",
            " 99% 801M/812M [00:04<00:00, 177MB/s]\n",
            "100% 812M/812M [00:04<00:00, 180MB/s]\n",
            "Archive:  dogs-vs-cats.zip\n",
            "  inflating: sampleSubmission.csv    \n",
            "  inflating: test1.zip               \n",
            "  inflating: train.zip               \n",
            "rm: cannot remove 'sample_data/': No such file or directory\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(train_dir, validation_dir, test_dir)"
      ],
      "metadata": {
        "id": "ZTN3L1RJz4L8",
        "outputId": "01c64f6e-8104-4765-bb73-26ef23d0c71b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/subset/train /content/subset/validation /content/subset/test\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "id": "-2o4cOnjy70A"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "from keras.preprocessing.image import ImageDataGenerator\n",
        "\n",
        "\n",
        "datagen = ImageDataGenerator(rescale=1./255)\n",
        "batch_size = 20\n",
        "\n",
        "def extract_features(directory, sample_count):\n",
        "    features = np.zeros(shape=(sample_count, 4, 4, 512))\n",
        "    labels = np.zeros(shape=(sample_count))\n",
        "    generator = datagen.flow_from_directory(\n",
        "        directory,\n",
        "        target_size=(150, 150),\n",
        "        batch_size=batch_size,\n",
        "        class_mode='binary')\n",
        "    i = 0\n",
        "    for inputs_batch, labels_batch in generator:\n",
        "        features_batch = conv_base.predict(inputs_batch, verbose=0)\n",
        "        features[i * batch_size : (i + 1) * batch_size] = features_batch\n",
        "        labels[i * batch_size : (i + 1) * batch_size] = labels_batch\n",
        "        i += 1\n",
        "        if i * batch_size >= sample_count:\n",
        "            # Note that since generators yield data indefinitely in a loop,\n",
        "            # we must `break` after every image has been seen once.\n",
        "            break\n",
        "    return features, labels\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "train_features, train_labels = extract_features(train_dir, 2000)\n",
        "validation_features, validation_labels = extract_features(validation_dir, 1000)\n",
        "test_features, test_labels = extract_features(test_dir, 1000)"
      ],
      "metadata": {
        "id": "exF35Zz10NAl",
        "outputId": "45a81818-a36b-41ae-99f2-2af6c2a850a9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Found 2000 images belonging to 2 classes.\n",
            "Found 1000 images belonging to 2 classes.\n",
            "Found 1000 images belonging to 2 classes.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UyY1zPSVy70C"
      },
      "source": [
        "The extracted features are currently of shape `(samples, 4, 4, 512)`. We will feed them to a densely-connected classifier, so first we must\n",
        "flatten them to `(samples, 8192)`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "collapsed": true,
        "id": "nOILLTIvy70C"
      },
      "outputs": [],
      "source": [
        "train_features = np.reshape(train_features, (2000, 4 * 4 * 512))\n",
        "validation_features = np.reshape(validation_features, (1000, 4 * 4 * 512))\n",
        "test_features = np.reshape(test_features, (1000, 4 * 4 * 512))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gV8CZ8v0y70D"
      },
      "source": [
        "At this point, we can define our densely-connected classifier (note the use of dropout for regularization), and train it on the data and\n",
        "labels that we just recorded:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jN4QbXKoy70E",
        "outputId": "4f48883c-d003-462a-ea77-eaaff7a3f1c3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Train on 2000 samples, validate on 1000 samples\n",
            "Epoch 1/30\n",
            "2000/2000 [==============================] - 7s 3ms/step - loss: 0.5870 - acc: 0.6915 - val_loss: 0.4264 - val_acc: 0.8390\n",
            "Epoch 2/30\n",
            "2000/2000 [==============================] - 6s 3ms/step - loss: 0.4143 - acc: 0.8025 - val_loss: 0.3636 - val_acc: 0.8510\n",
            "Epoch 3/30\n",
            "2000/2000 [==============================] - 6s 3ms/step - loss: 0.3546 - acc: 0.8470 - val_loss: 0.3194 - val_acc: 0.8890\n",
            "Epoch 4/30\n",
            "2000/2000 [==============================] - 5s 3ms/step - loss: 0.3211 - acc: 0.8630 - val_loss: 0.2976 - val_acc: 0.8920\n",
            "Epoch 5/30\n",
            "2000/2000 [==============================] - 6s 3ms/step - loss: 0.2848 - acc: 0.8825 - val_loss: 0.2869 - val_acc: 0.8910\n",
            "Epoch 6/30\n",
            "2000/2000 [==============================] - 6s 3ms/step - loss: 0.2650 - acc: 0.8965 - val_loss: 0.2785 - val_acc: 0.8900\n",
            "Epoch 7/30\n",
            "2000/2000 [==============================] - 6s 3ms/step - loss: 0.2425 - acc: 0.9110 - val_loss: 0.2660 - val_acc: 0.8990\n",
            "Epoch 8/30\n",
            "2000/2000 [==============================] - 6s 3ms/step - loss: 0.2315 - acc: 0.9170 - val_loss: 0.2619 - val_acc: 0.8970\n",
            "Epoch 9/30\n",
            "2000/2000 [==============================] - 6s 3ms/step - loss: 0.2204 - acc: 0.9135 - val_loss: 0.2563 - val_acc: 0.9030\n",
            "Epoch 10/30\n",
            "2000/2000 [==============================] - 6s 3ms/step - loss: 0.2036 - acc: 0.9205 - val_loss: 0.2631 - val_acc: 0.8930\n",
            "Epoch 11/30\n",
            "2000/2000 [==============================] - 6s 3ms/step - loss: 0.1943 - acc: 0.9285 - val_loss: 0.2455 - val_acc: 0.9030\n",
            "Epoch 12/30\n",
            "2000/2000 [==============================] - 6s 3ms/step - loss: 0.1874 - acc: 0.9335 - val_loss: 0.2489 - val_acc: 0.9020\n",
            "Epoch 13/30\n",
            "2000/2000 [==============================] - 6s 3ms/step - loss: 0.1808 - acc: 0.9355 - val_loss: 0.2447 - val_acc: 0.8980\n",
            "Epoch 14/30\n",
            "2000/2000 [==============================] - 5s 3ms/step - loss: 0.1730 - acc: 0.9385 - val_loss: 0.2504 - val_acc: 0.9000\n",
            "Epoch 15/30\n",
            "2000/2000 [==============================] - 5s 3ms/step - loss: 0.1646 - acc: 0.9390 - val_loss: 0.2412 - val_acc: 0.9050\n",
            "Epoch 16/30\n",
            "2000/2000 [==============================] - 5s 3ms/step - loss: 0.1527 - acc: 0.9505 - val_loss: 0.2382 - val_acc: 0.9050\n",
            "Epoch 17/30\n",
            "2000/2000 [==============================] - 5s 3ms/step - loss: 0.1530 - acc: 0.9470 - val_loss: 0.2386 - val_acc: 0.9030\n",
            "Epoch 18/30\n",
            "2000/2000 [==============================] - 5s 3ms/step - loss: 0.1420 - acc: 0.9495 - val_loss: 0.2382 - val_acc: 0.9020\n",
            "Epoch 19/30\n",
            "2000/2000 [==============================] - 5s 3ms/step - loss: 0.1383 - acc: 0.9525 - val_loss: 0.2485 - val_acc: 0.9010\n",
            "Epoch 20/30\n",
            "2000/2000 [==============================] - 5s 3ms/step - loss: 0.1330 - acc: 0.9525 - val_loss: 0.2373 - val_acc: 0.9040\n",
            "Epoch 21/30\n",
            "2000/2000 [==============================] - 5s 3ms/step - loss: 0.1284 - acc: 0.9525 - val_loss: 0.2396 - val_acc: 0.9000\n",
            "Epoch 22/30\n",
            "2000/2000 [==============================] - 5s 3ms/step - loss: 0.1207 - acc: 0.9580 - val_loss: 0.2498 - val_acc: 0.9000\n",
            "Epoch 23/30\n",
            "2000/2000 [==============================] - 5s 3ms/step - loss: 0.1165 - acc: 0.9620 - val_loss: 0.2403 - val_acc: 0.8990\n",
            "Epoch 24/30\n",
            "2000/2000 [==============================] - 5s 3ms/step - loss: 0.1110 - acc: 0.9665 - val_loss: 0.2476 - val_acc: 0.8980\n",
            "Epoch 25/30\n",
            "2000/2000 [==============================] - 5s 3ms/step - loss: 0.1036 - acc: 0.9720 - val_loss: 0.2490 - val_acc: 0.8990\n",
            "Epoch 26/30\n",
            "2000/2000 [==============================] - 5s 3ms/step - loss: 0.1025 - acc: 0.9670 - val_loss: 0.2380 - val_acc: 0.9040\n",
            "Epoch 27/30\n",
            "2000/2000 [==============================] - 5s 3ms/step - loss: 0.0968 - acc: 0.9665 - val_loss: 0.2388 - val_acc: 0.9040\n",
            "Epoch 28/30\n",
            "2000/2000 [==============================] - 5s 3ms/step - loss: 0.0970 - acc: 0.9675 - val_loss: 0.2394 - val_acc: 0.9080\n",
            "Epoch 29/30\n",
            "2000/2000 [==============================] - 5s 3ms/step - loss: 0.0897 - acc: 0.9755 - val_loss: 0.2442 - val_acc: 0.8990\n",
            "Epoch 30/30\n",
            "2000/2000 [==============================] - 5s 3ms/step - loss: 0.0868 - acc: 0.9725 - val_loss: 0.2409 - val_acc: 0.8990\n"
          ]
        }
      ],
      "source": [
        "from keras import models\n",
        "from keras import layers\n",
        "from keras import optimizers\n",
        "\n",
        "model = models.Sequential()\n",
        "model.add(layers.Dense(256, activation='relu', input_dim=4 * 4 * 512))\n",
        "model.add(layers.Dropout(0.5))\n",
        "model.add(layers.Dense(1, activation='sigmoid'))\n",
        "\n",
        "model.compile(optimizer=optimizers.RMSprop(learning_rate=2e-5),\n",
        "              loss='binary_crossentropy',\n",
        "              metrics=['acc'])\n",
        "\n",
        "history = model.fit(train_features, train_labels,\n",
        "                    epochs=30,\n",
        "                    batch_size=20,\n",
        "                    validation_data=(validation_features, validation_labels))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2kQaggQgy70F"
      },
      "source": [
        "Training is very fast, since we only have to deal with two `Dense` layers -- an epoch takes less than one second even on CPU.\n",
        "\n",
        "Let's take a look at the loss and accuracy curves during training:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a6B3iDw_y70G",
        "outputId": "5f3d07e4-0d6c-4d67-875e-2968225734e6"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XucFNWd9/HPjxEQEBVhTJSBGTQoDncYMQZUNKJIVNRg\nBDERjSEm6rqabB5vjxI2arLGVXfjJsGsTxJFCNFoSKJrvCtJzDIoF8GACCiDCCMgt0G5/Z4/TjX0\n3KuHnunp7u/79epX1+VU1amumV+dOnXqlLk7IiKSH9pkOgMiItJyFPRFRPKIgr6ISB5R0BcRySMK\n+iIieURBX0Qkjyjo5yEzKzCzbWbWM51pM8nMPmdmaW9/bGZnmtmqpPGlZnZKnLRN2NYvzOyWpi4v\nEsdBmc6ANM7MtiWNdgQ+BfZE49909+mprM/d9wCHpDttPnD349OxHjO7CrjM3UcmrfuqdKxbpCEK\n+lnA3fcF3agkeZW7P19fejM7yN13t0TeRBqjv8fWRdU7OcDMfmBmvzGzGWa2FbjMzE42s9fN7GMz\nW2tm/2FmbaP0B5mZm1lJNP5oNP8ZM9tqZn8zs16ppo3mn2Nmy8xss5n9p5n9xcwm1ZPvOHn8ppkt\nN7NNZvYfScsWmNl9ZrbBzFYAoxv4fW41s5k1pj1oZv8eDV9lZm9H+/NuVAqvb10VZjYyGu5oZo9E\neVsMDK2R9jYzWxGtd7GZnR9N7w/8BDglqjr7KOm3nZK0/NXRvm8ws6fM7Kg4v00qv3MiP2b2vJlt\nNLMPzex7Sdv5v9FvssXMys3s6Lqq0sxsTuI4R7/nq9F2NgK3mVlvM3sp2sZH0e92WNLyxdE+Vkbz\nHzCzg6M8n5CU7igzqzKzrvXtrzTC3fXJog+wCjizxrQfADuB8wgn8g7AicBJhKu5Y4BlwLVR+oMA\nB0qi8UeBj4AyoC3wG+DRJqQ9EtgKjI3m3QjsAibVsy9x8vh74DCgBNiY2HfgWmAxUAR0BV4Nf851\nbucYYBvQKWnd64GyaPy8KI0BZwA7gAHRvDOBVUnrqgBGRsM/Bl4GugDFwJIaab8CHBUdk0ujPHwm\nmncV8HKNfD4KTImGz4ryOAg4GPgv4MU4v02Kv/NhwDrgeqA9cCgwLJp3M7AA6B3twyDgCOBzNX9r\nYE7iOEf7thv4FlBA+Hs8Dvgi0C76O/kL8OOk/Xkr+j07RemHR/OmAXcmbec7wJOZ/j/M5k/GM6BP\niges/qD/YiPLfRf4bTRcVyD/WVLa84G3mpD2SuC1pHkGrKWeoB8zj59Pmv874LvR8KuEaq7EvDE1\nA1GNdb8OXBoNnwMsbSDtH4FrouGGgv77yccC+HZy2jrW+xbwpWi4saD/K+CupHmHEu7jFDX226T4\nO38VmFtPuncT+a0xPU7QX9FIHsYltgucAnwIFNSRbjiwErBofD5wUbr/r/Lpo+qd3LE6ecTM+pjZ\nn6LL9S3AVKBbA8t/mDRcRcM3b+tLe3RyPjz8l1bUt5KYeYy1LeC9BvIL8BgwIRq+NBpP5ONcM/t7\nVPXwMaGU3dBvlXBUQ3kws0lmtiCqovgY6BNzvRD2b9/63H0LsAnonpQm1jFr5HfuQQjudWloXmNq\n/j1+1sxmmdmaKA+/rJGHVR4aDVTj7n8hXDWMMLN+QE/gT03Mk6A6/VxSs7nizwkly8+5+6HA7YSS\nd3NaSyiJAmBmRvUgVdOB5HEtIVgkNNakdBZwppl1J1Q/PRblsQPwOHA3oerlcODPMfPxYX15MLNj\ngJ8Sqji6Ruv9R9J6G2te+gGhyiixvs6EaqQ1MfJVU0O/82rg2HqWq2/e9ihPHZOmfbZGmpr79yNC\nq7P+UR4m1chDsZkV1JOPXwOXEa5KZrn7p/WkkxgU9HNXZ2AzsD26EfbNFtjmH4EhZnaemR1EqCcu\nbKY8zgL+2cy6Rzf1/k9Did39Q0IVxC8JVTvvRLPaE+qZK4E9ZnYuoe45bh5uMbPDLTzHcG3SvEMI\nga+ScP77BqGkn7AOKEq+oVrDDODrZjbAzNoTTkqvuXu9V04NaOh3ng30NLNrzay9mR1qZsOieb8A\nfmBmx1owyMyOIJzsPiQ0GCgws8kknaAayMN2YLOZ9SBUMSX8DdgA3GXh5ngHMxueNP8RQnXQpYQT\ngBwABf3c9R3gcsKN1Z8Tbrg2K3dfB1wC/Dvhn/hY4E1CCS/defwp8AKwCJhLKK035jFCHf2+qh13\n/xi4AXiScDN0HOHkFccdhCuOVcAzJAUkd18I/Cfwv1Ga44G/Jy37HPAOsM7MkqtpEsv/D6Ea5slo\n+Z7AxJj5qqne39ndNwOjgC8TTkTLgNOi2fcATxF+5y2Em6oHR9V23wBuIdzU/1yNfavLHcAwwsln\nNvBEUh52A+cCJxBK/e8TjkNi/irCcf7U3f+a4r5LDYmbIyJpF12ufwCMc/fXMp0fyV5m9mvCzeEp\nmc5LttPDWZJWZjaa0FJmB6HJ3y5CaVekSaL7I2OB/pnOSy5Q9Y6k2whgBaEu+2zgQt14k6Yys7sJ\nzwrc5e7vZzo/uUDVOyIieUQlfRGRPNLq6vS7devmJSUlmc6GiEhWmTdv3kfu3lATaaAVBv2SkhLK\ny8sznQ0RkaxiZo09lQ6oekdEJK8o6IuI5BEFfRGRPNLq6vTrsmvXLioqKvjkk08ynRVpwMEHH0xR\nURFt29bXnYyIZFpWBP2Kigo6d+5MSUkJoeNGaW3cnQ0bNlBRUUGvXr0aX0BEMiIrqnc++eQTunbt\nqoDfipkZXbt21dWYSJLp06GkBNq0Cd/Tp2c6R1kS9AEF/CygYySy3/TpMHkyvPceuIfvyZPrDvwt\neXLImqAvIpKq5gimcdd5661QVVV9WlVVmF5zfXFPDumgoB/Dhg0bGDRoEIMGDeKzn/0s3bt33ze+\nc+fOWOu44oorWLp0aYNpHnzwQaa3hus/kRzQHME0lXW+X0/3cDWnxz05pE2mX9Jb8zN06FCvacmS\nJbWmNeTRR92Li93Nwvejj6a0eIPuuOMOv+eee2pN37t3r+/Zsyd9G8pSqR4rkYRU/m/jpC0udg+h\nufqnuLjpeUxlnXHTmtWdziy1vAHlno8vRm/JS6Xly5dTWlrKxIkT6du3L2vXrmXy5MmUlZXRt29f\npk6dui/tiBEjmD9/Prt37+bwww/npptuYuDAgZx88smsX78egNtuu437779/X/qbbrqJYcOGcfzx\nx/PXv4YXBm3fvp0vf/nLlJaWMm7cOMrKypg/f36tvN1xxx2ceOKJ9OvXj6uvvhqPelNdtmwZZ5xx\nBgMHDmTIkCGsWrUKgLvuuov+/fszcOBAbm22IoZI3VKt/46TNm5JOxWprPPOO6Fjx+rTOnYM05P1\nrOftzvVNP2Bxzgwt+TnQkn5znN2TJZf033nnHTcznzt37r75GzZscHf3Xbt2+YgRI3zx4sXu7j58\n+HB/8803fdeuXQ74008/7e7uN9xwg999993u7n7rrbf6fffdty/99773PXd3//3vf+9nn322u7vf\nfffd/u1vf9vd3efPn+9t2rTxN998s1Y+E/nYu3evjx8/ft/2hgwZ4rNnz3Z39x07dvj27dt99uzZ\nPmLECK+qqqq2bFOopC9N0Rwl6FRjQXNcPcRZ56OPunfsWH19HTumXkNBvpb0m+Ps3pBjjz2WsrKy\nfeMzZsxgyJAhDBkyhLfffpslS5bUWqZDhw6cc845AAwdOnRfabumiy66qFaaOXPmMH78eAAGDhxI\n375961z2hRdeYNiwYQwcOJBXXnmFxYsXs2nTJj766CPOO+88IDxM1bFjR55//nmuvPJKOnToAMAR\nRxyR+g8hcgBS+b+NmzZuSRviXz2ksk6AiRNh1SrYuzd8T6zjLccTJ8K0aVBcDGbhe9q0utOmQ84F\n/Za+VOrUqdO+4XfeeYcHHniAF198kYULFzJ69Og62623a9du33BBQQG7d++uc93t27dvNE1dqqqq\nuPbaa3nyySdZuHAhV155pdrPS6uWyv9t3LSpBNO4N1ObK0DHOTmkS84F/VTPxOm0ZcsWOnfuzKGH\nHsratWt59tln076N4cOHM2vWLAAWLVpU55XEjh07aNOmDd26dWPr1q088cQTAHTp0oXCwkL+8Ic/\nAOGht6qqKkaNGsXDDz/Mjh07ANi4cWPa8y25Jd1NIVP5v00lbdxgmsqVRksG6OaQc0G/pS+Vkg0Z\nMoTS0lL69OnD1772NYYPH572bVx33XWsWbOG0tJSvv/971NaWsphhx1WLU3Xrl25/PLLKS0t5Zxz\nzuGkk07aN2/69Once++9DBgwgBEjRlBZWcm5557L6NGjKSsrY9CgQdx3331pz7dkhzjBPNXGEnHW\nmcr/bXP8j7f4zdRMilPx35KfdDTZzGW7du3yHTt2uLv7smXLvKSkxHft2pXhXO2nY9X6xG0KGfeG\nYio3M9N1k7K5ZUs+G0LMG7mxAjEwGlgKLAduqmN+MfACsBB4GShKmrcHmB99Zje2LQX9hm3atMmH\nDBniAwYM8P79+/uzzz6b6SxVo2PVuqQSzJqjXXlzt6ZLp+Z8vqclpC3oAwXAu8AxQDtgAVBaI81v\ngcuj4TOAR5LmbYuTkcRHQT+76Vi1nHQ3MYwbzJtjnXLg4gb9OHX6w4Dl7r7C3XcCM4GxNdKUAi9G\nwy/VMV9EYoh7g7Q5HlCKW6+dyo3UvKorzxJxgn53YHXSeEU0LdkC4KJo+EKgs5l1jcYPNrNyM3vd\nzC6oawNmNjlKU15ZWZlC9kVyRyo3SOM2MUwl6MYN5qncSM1kazqpR2OXAsA44BdJ418FflIjzdHA\n74A3gQcIJ4bDo3ndo+9jgFXAsQ1tT9U72U3Hqumao9ok1RuUzVGvne115dmCNFbvrAF6JI0XRdOS\nTxwfuPtF7j4YuDWa9nH0vSb6XkG4yTs4pbOSSA6IU23THFUxqTZvbI426Nnerj3XxAn6c4HeZtbL\nzNoB44HZyQnMrJuZJdZ1M/BwNL2LmbVPpAGGA7WfJmrlTj/99FoPWt1///1861vfanC5Qw45BIAP\nPviAcePG1Zlm5MiRlJeXN7ie+++/n6qka/kxY8bw8ccfx8m6NKN01783R1UMKOhKDXEuB4AxwDJC\nK55bo2lTgfN9fxXQO1GaXwDto+lfABYR6vwXAV9vbFutsXrn5z//uU+aNKnatJNOOslfeeWVBpfr\n1KlTo+s+7bTTqnXYVpfi4mKvrKxsPKOtQKaPVTqku5OsuNU2raEqRrIX6Wyn35Kf1hj0N2zY4IWF\nhf7pp5+6u/vKlSu9R48evnfvXt+6daufccYZPnjwYO/Xr58/9dRT+5ZLBP2VK1d637593d29qqrK\nL7nkEu/Tp49fcMEFPmzYsH1B/+qrr/ahQ4d6aWmp33777e7u/sADD3jbtm29X79+PnLkSHevfhK4\n9957vW/fvt63b999PXSuXLnS+/Tp41dddZWXlpb6qFGj9vWgmWz27Nk+bNgwHzRokH/xi1/0Dz/8\n0N3dt27d6pMmTfJ+/fp5//79/fHHH3d392eeecYHDx7sAwYM8DPOOKPO3yrTx+pANccDSqk0W1Qg\nl6bK2aB//fXup52W3s/11zf+g37pS1/aF9Dvvvtu/853vuPu4QnZzZs3u7t7ZWWlH3vssb537153\nrzvo33vvvX7FFVe4u/uCBQu8oKBgX9BPdGm8e/duP+2003zBggXuXruknxgvLy/3fv36+bZt23zr\n1q1eWlrqb7zxhq9cudILCgr2dbl88cUX+yOPPFJrnzZu3Lgvrw899JDfeOON7u7+ve99z69P+lE2\nbtzo69ev96KiIl+xYkW1vNaU7UE/3x9QkuwVN+jnXN87zWXChAnMnDkTgJkzZzJhwgQgnDRvueUW\nBgwYwJlnnsmaNWtYt25dvet59dVXueyyywAYMGAAAwYM2Ddv1qxZDBkyhMGDB7N48eI6O1NLNmfO\nHC688EI6derEIYccwkUXXcRrr70GQK9evRg0aBBQf/fNFRUVnH322fTv35977rmHxYsXA/D8889z\nzTXX7EvXpUsXXn/9dU499VR69eoF5G73y3FvpjZX/btIczso0xlIVfRiqRY3duxYbrjhBt544w2q\nqqoYOnQoEDowq6ysZN68ebRt25aSkpImdWO8cuVKfvzjHzN37ly6dOnCpEmTDqg75ES3zBC6Zk70\noJnsuuuu48Ybb+T888/n5ZdfZsqUKU3eXms3fXpow/7++yEw33ln3Tc0e/YMN1rrmp7szjvDzdjk\ntvIN3UiFeNsXaW4q6cd0yCGHcPrpp3PllVfuK+UDbN68mSOPPJK2bdvy0ksv8V5dESPJqaeeymOP\nPQbAW2+9xcKFC4HQLXOnTp047LDDWLduHc8888y+ZTp37szWrVtrreuUU07hqaeeoqqqiu3bt/Pk\nk09yyimnxN6nzZs30717eM7uV7/61b7po0aN4sEHH9w3vmnTJj7/+c/z6quvsnLlSiC7ul9O5aGn\n5nhAKZFeLWikNVDQT8GECRNYsGBBtaA/ceJEysvL6d+/P7/+9a/p06dPg+v41re+xbZt2zjhhBO4\n/fbb910xDBw4kMGDB9OnTx8uvfTSat0yT548mdGjR3P66adXW9eQIUOYNGkSw4YN46STTuKqq65i\n8OD4j0FMmTKFiy++mKFDh9KtW7d902+77TY2bdpEv379GDhwIC+99BKFhYVMmzaNiy66iIEDB3LJ\nJZfE3k6mxX16FVLv4leBXLJOnIr/lvy0xtY7El86jlUqLVjipFWnX5IPiHkjN+vq9CW3JapiEiXz\nRFUM1C5Jx00bt55eJB+oekdaTJwnWFOpiombVq1nRPbLmqAfrl6kNWvoGDVHV8Bx02byFZoirY21\ntmBaVlbmNfuiWblyJZ07d6Zr166YWYZyJnXZsAHWrIGdOx2zDezYsZURI3rVSldSUncVS3FxuAma\narpU04rkOjOb5+5ljaXLijr9oqIiKioqUF/7rcv27SHou4cWLMuXH8yPflTEv/1b7VJ03FJ5Ku3f\nU0krIkFWBP22bdvuexJUWo/6Stq33lo76Me9mZrKg0x66EkkdVlRvSOtU5s2oZRfk1ko+Ser2dIG\nQqlcdesi6RG3eidrbuRK65NK/zO6mSrSOijoS5Ol2hRST7CKZJ6CvjSZSu8i2ScrbuRK6zVxooK8\nSDZRSV/qFPf9ryKSXVTSl1pS6f9GRLKLSvo5IN2l8lT6vxGR7KKgn+VSeUFI3JNDKv3fiEh2iRX0\nzWy0mS01s+VmdlMd84vN7AUzW2hmL5tZUdK8y83snehzeTozL/FL5amcHFJpfy8i2aXRJ3LNrABY\nBowCKoC5wAR3X5KU5rfAH939V2Z2BnCFu3/VzI4AyoEywIF5wFB331Tf9vREbmriPhWbSudkeno2\nu+3ZA8uXw1tvwaJF+7+3b4cePcLJu2fP6sM9e0LXruHvJlPcYf58WLIENm/e/9mypfp44rN1a91/\n+3Xp0gX69YP+/fd/H3cctGt34PnetSt0Orh6dbgaTv6sXg0ffBCOSRxlZfDcc03LRzo7XBsGLHf3\nFdGKZwJjgSVJaUqBG6Phl4CnouGzgefcfWO07HPAaGBGnJ2QxsXt0yaVKhv1aZNeO3fC738Ps2aF\n8cMO2/859NDq44lPhw7xAvDOnbB0afUAv2QJfPJJmG8Gn/tcCHSHHRaC0Jtvhvx8+mn1dXXoEE4E\nRUXxg2FhIXzhCzBiBJSWhkJIKrZsCUHu6afhmWdg7drq89u2rf17HXNMGO7cGQoK4m1n3brw2zzz\nDOzevX/dxx9f/URw/PFhfl0nmZonn/Xrw//HBx/UPvl07Rr+b3r1Cr9N27bx8llcHC/dgYgT9LsD\nq5PGK4CTaqRZAFwEPABcCHQ2s671LNu9ybnNI9Onxwu6cXuaTPXtUWp/f+BWrYKHHoL//u8QdI4+\nOgStRPDYvj292zv66BC8rrlmfxA74YTaT01DCFKVlbVLp6tXh1Lrtm3xtvnmm/DII2H48MPDCWD4\n8BDoTjwxnEhqbvftt0OQf/ppeO21EGQPPxzOPhvGjIGTTgrjhx4KBx+c3quPTz+tfZL8619hRoxi\naJs21U/ShYVw1lm1r5iKiqBTp/TlOd3S1WTzu8BPzGwS8CqwBoh5QQNmNhmYDNBTFccpNZmMWypv\nDd0Qb90KCxaEf5yePcM/Tqr/0Il/2uRqi3fege7dq5fY+vYNJcGWtmdPCGY/+1koVZrBuefC1VeH\nAJFcMt29u/6qi0RJvTFt2uwvyXftGj+fZnDkkeEzdGhq+5jMHVasgDlz4C9/Cd9PPx3mtW0b1j18\nOAwYAH//e5iXqE7s3x+++90Q6E8+GQ5qgQbk7duHvAwYUH36li2weDEsWxZONHVdiR1ySGarv9Il\nTp3+ycAUdz87Gr8ZwN3vrif9IcA/3L3IzCYAI939m9G8nwMvu3u951XV6Tffy0GmT4dbbgknh+Li\n5q+yWbOmejBYsKD6fYbOnRuuY3bfH9gTQX7ZsuqX5336QO/eYVtvvVW99FxSUv1E0K8ffOYz9Qfa\n5Onbt4eSXM08ffazdVdhrF0bSvTTpoXS8lFHwVVXhU++lWM2bAil58Rxnzs3VEN17AhnnhmC/Jgx\n4beV9Elnnf5coLeZ9SKU4McDl9bYWDdgo7vvBW4GHo5mPQvcZWZdovGzovnSgHQ3mXSHhQtD4DQL\npZeRI6FbtxBA01HC2rs3lJSSg3zixNWxI3z+83DbbTBsWLjaqFmtMG9eqG6oT69eIXBfcEH47t8/\nBPvkuudER241b2Am1+M2pkOHUKrr1ClUydSs5mjbNlxVJJ8Ili4NdeS7d8OoUXD//XDeefHrcXNN\n165h/887L4x/8kk4WR93XChFS2bF6k/fzMYA9wMFwMPufqeZTQXK3X22mY0D7ia00HkVuMbdP42W\nvRK4JVrVne7+/xralkr66Svpv/tuqKt87LFQj1pQEILSkUfCU0+Fku2RR8LFF8OECeESO+6NuKqq\nUIJLBPm//jWUkCGUhkeMCJ/hw2HgwHgBcMcOqKjYfyLYuzeUzvv2DZfWTZW42bloUSiF1nXjNHEZ\nn5xP97BPddV7J4bXrAnLXXFFqD7r3bvp+RQ5EHFL+nqJSit0IE0mP/ggtBJ57LEQlAFOOSUE9XHj\nQpUFhNLXM8+Ek8If/hDGi4th/PiQdsCA6vWX69eH4J4oxc+bt7/0XFq6/+bdiBGhVJ4LdZ9xJJri\nxW1FItJcFPSzXF2tdy69NJwI6qqT/vBDePJJePnlUEIdPDgE70suabxOecuWUD0xYwb8+c8hkJWW\nwoUXhpPInDnhZimEG2Ennri/FP+FL8ARRzT7zyEijVDQz2J79sArr4QgXF5e/UZjQ3XTvXuHQD9h\nQrjB2RSVlfD442Hbr70W6mcTpfjhw0NrjPbtm7ZuEWk+CvpZxj1Ux8yYAb/5TWgN0qkTnHpqKEnX\nVQ+dXBfdpUu4wZjOapWtW3OnmZpIrktn6x1pRkuWhEA/Y0a48dquXWjONmFCaN9d14M1LSUT7dxF\npHkp6KfJ3r2h5UljF05PPQU/+lEoybdtG/rtaNMGzjgjtKG/6KLwNKKISHNQ0E+DzZth7NhQD5+K\nXbtC4L/vvvDovIhIc1PQP0Dr18Po0aEN+F13hTbq9fmXfwntxJPt2gX33KOgLyItQ0H/ALz/fnjY\nafVqmD0bzjmn4fRf/3r96xERaQl6c1YT/eMfoQnjunWha9jGAj7o5SQiknkK+k0wb154ynXnzlCP\nP3x4vOXuvLN2a5yW7ulSRPKbgn6KXnkFTj89tKGfMyf0KxP33bMTJ4auFIqLQ9v34mK9jUpEWpbq\n9FPwhz+EzsmOOSZU6XTvnlrf94lpCvIikikq6cf06KOhL5oBA+DVV0PAh/gvJhcRaQ0U9GP4yU/g\nq1+F006DF14I/dAnpLvvexGR5qSg34h//Ve47rrw8o4//al21wRqkSMi2URBvwE/+xncfjtcfjn8\n9rd1v/VHLXJEJJso6NfjlVdCCX/MmPDu0/peKagWOSKSTdS1ch1WrgwvCikshNdfD10Xi4i0ZnG7\nVlZJv4Zt20LnaXv2hLdJKeCLSC5RO/0ke/fC174GixeH98ced1ymcyQikl4K+kmmTg3vmb3vPjjr\nrEznRkQk/VS9E3niCfj+92HSJLj++kznRkSkecQK+mY22syWmtlyM7upjvk9zewlM3vTzBaa2Zho\neomZ7TCz+dHnZ+negXRYsCBU65x8cmimqXfCikiuajTom1kB8CBwDlAKTDCz0hrJbgNmuftgYDzw\nX0nz3nX3QdHn6jTlO23Wr4fzzw8vH//d76B9+/3z4nakJiKSLeLU6Q8Dlrv7CgAzmwmMBZYkpXHg\n0Gj4MOCDdGayuezcCePGhcA/Z071t16l2pGaiEg2iFO90x1YnTReEU1LNgW4zMwqgKeB65Lm9Yqq\nfV4xs1Pq2oCZTTazcjMrr6ysjJ/7A+AeHr567TV4+GEYOrT6fHWkJiK5KF03cicAv3T3ImAM8IiZ\ntQHWAj2jap8bgcfM7NCaC7v7NHcvc/eywsLCNGWpYT/9aXhy9uabYcKE2vPVkZqI5KI4QX8N0CNp\nvCialuzrwCwAd/8bcDDQzd0/dfcN0fR5wLtAxlu/P/cc/NM/wXnnwQ9+UHcadaQmIrkoTtCfC/Q2\ns15m1o5wo3Z2jTTvA18EMLMTCEG/0swKoxvBmNkxQG9gRboy3xSPPQZf+hKUloY+8tvU8wuoIzUR\nyUWNBn133w1cCzwLvE1opbPYzKaa2flRsu8A3zCzBcAMYJKHTn1OBRaa2XzgceBqd9/YHDvSGHe4\n665wE/YLXwgdqh1aq6JpP3WkJiK5KC86XNu9G779bXjoIbj00nDjNrlppohItlOHa5GtW0Pd/UMP\nhZY3jz6qgC8i+Sun+95ZswbOPRcWLQpB/6qrMp0jEZHMytmgv2hReAHKxx/DH/8Io0dnOkciIpmX\nk9U7zz8PI0aErpJfe00BX0QkIeeC/i9/CeecE1rbvP46DBqU6RyJiLQeORP03WHKFLjiChg5MpTw\ne/RobCkDOgwPAAAMNklEQVQRkfySM0F/2TL44Q9Df/hPP63XHIqI1CVnbuQefzzMmxeetFV/+CIi\ndcuZoA/Qt2+mcyAi0rrlTPWOiIg0TkFfRCSPKOiLiOQRBX0RkTyioC8ikkcU9EVE8oiCvohIHlHQ\nFxHJIwr6IiJ5REFfRCSPKOiLiOQRBX0RkTwSK+ib2WgzW2pmy83spjrm9zSzl8zsTTNbaGZjkubd\nHC231MzOTmfmRUQkNY32smlmBcCDwCigAphrZrPdfUlSstuAWe7+UzMrBZ4GSqLh8UBf4GjgeTM7\nzt33pHtHRESkcXFK+sOA5e6+wt13AjOBsTXSOHBoNHwY8EE0PBaY6e6fuvtKYHm0PhERyYA4Qb87\nsDppvCKalmwKcJmZVRBK+delsGyLmj4dSkqgTZvwPX16JnMjItKy0nUjdwLwS3cvAsYAj5hZ7HWb\n2WQzKzez8srKyjRlqbbp02HyZHjvvfBO3ffeC+MK/CKSL+IE5jVA8ivGi6Jpyb4OzAJw978BBwPd\nYi6Lu09z9zJ3LyssLIyf+xTdeitUVVWfVlUVpouI5IM4QX8u0NvMeplZO8KN2dk10rwPfBHAzE4g\nBP3KKN14M2tvZr2A3sD/pivzqXr//dSmi4jkmkaDvrvvBq4FngXeJrTSWWxmU83s/CjZd4BvmNkC\nYAYwyYPFhCuAJcD/ANdksuVOz56pTRcRyTXm7pnOQzVlZWVeXl7eLOtO1OknV/F07AjTpsHEic2y\nSRGRFmFm89y9rLF0efVE7sSJIcAXF4NZ+FbAF5F80ujDWblm4kQFeRHJX3lV0hcRyXcK+iIieURB\nX0Qkjyjoi4jkEQV9EZE8oqAvIpJHFPRFRPKIgr6ISB5R0BcRySMK+iIieURBX0Qkjyjoi4jkEQV9\nEZE8oqAvIpJHFPRFRPKIgr6ISB5R0BcRySMK+iIieURBX0Qkjyjoi4jkkVhB38xGm9lSM1tuZjfV\nMf8+M5sffZaZ2cdJ8/YkzZudzsyLiEhqDmosgZkVAA8Co4AKYK6ZzXb3JYk07n5DUvrrgMFJq9jh\n7oPSl2UREWmqOCX9YcByd1/h7juBmcDYBtJPAGakI3MiIpJecYJ+d2B10nhFNK0WMysGegEvJk0+\n2MzKzex1M7ugnuUmR2nKKysrY2ZdRERSle4bueOBx919T9K0YncvAy4F7jezY2su5O7T3L3M3csK\nCwvTnCUREUmIE/TXAD2SxouiaXUZT42qHXdfE32vAF6men2/iIi0oDhBfy7Q28x6mVk7QmCv1QrH\nzPoAXYC/JU3rYmbto+FuwHBgSc1lRUSkZTTaesfdd5vZtcCzQAHwsLsvNrOpQLm7J04A44GZ7u5J\ni58A/NzM9hJOMD9MbvUjIiIty6rH6MwrKyvz8vLyTGdDRCSrmNm86P5pg/RErohIHlHQFxHJIwr6\nIiJ5REFfRCSPKOiLiOQRBX0RkTyioC8ikkcU9EVE8oiCvohIHlHQFxHJIwr6IiJ5REFfRCSPKOiL\niOQRBX0RkTyioC8ikkcU9EVE8oiCvohIHlHQFxHJIwr6IiJ5REFfRCSPKOiLiOQRBX0RkTwSK+ib\n2WgzW2pmy83spjrm32dm86PPMjP7OGne5Wb2TvS5PJ2ZFxGR1BzUWAIzKwAeBEYBFcBcM5vt7ksS\nadz9hqT01wGDo+EjgDuAMsCBedGym9K6FyIiEkuckv4wYLm7r3D3ncBMYGwD6ScAM6Lhs4Hn3H1j\nFOifA0YfSIZFRKTp4gT97sDqpPGKaFotZlYM9AJeTGVZM5tsZuVmVl5ZWRkn3yIi0gTpvpE7Hnjc\n3fekspC7T3P3MncvKywsTHOWREQkIU7QXwP0SBoviqbVZTz7q3ZSXVZERJpZnKA/F+htZr3MrB0h\nsM+umcjM+gBdgL8lTX4WOMvMuphZF+CsaJqIiGRAo6133H23mV1LCNYFwMPuvtjMpgLl7p44AYwH\nZrq7Jy270cz+lXDiAJjq7hvTuwsiIhKXJcXoVqGsrMzLy8sznQ0RkaxiZvPcvayxdHoiV0Qkjyjo\ni4jkEQV9EZE8oqAvIpJHFPRFRPKIgr6ISB5R0BcRySMK+iIieURBX0Qkjyjoi4jkEQV9EZE8oqAv\nIpJHFPRFRPKIgr6ISB5R0BcRySMK+iIieURBX0Qkjyjoi4jkEQV9EZE8oqAvIpJHFPRFRPJIrKBv\nZqPNbKmZLTezm+pJ8xUzW2Jmi83ssaTpe8xsfvSZna6Mi4hI6g5qLIGZFQAPAqOACmCumc129yVJ\naXoDNwPD3X2TmR2ZtIod7j4ozfkWEZEmiFPSHwYsd/cV7r4TmAmMrZHmG8CD7r4JwN3XpzebIiKS\nDnGCfndgddJ4RTQt2XHAcWb2FzN73cxGJ8072MzKo+kX1LUBM5scpSmvrKxMaQcSpk+HkhJo0yZ8\nT5/epNWIiOS0Rqt3UlhPb2AkUAS8amb93f1joNjd15jZMcCLZrbI3d9NXtjdpwHTAMrKyjzVjU+f\nDpMnQ1VVGH/vvTAOMHFiU3dJRCT3xCnprwF6JI0XRdOSVQCz3X2Xu68ElhFOArj7muh7BfAyMPgA\n81zLrbfuD/gJVVVhuoiI7Bcn6M8FeptZLzNrB4wHarbCeYpQysfMuhGqe1aYWRcza580fTiwhDR7\n//3UpouI5KtGg7677wauBZ4F3gZmuftiM5tqZudHyZ4FNpjZEuAl4F/cfQNwAlBuZgui6T9MbvWT\nLj17pjZdRCRfmXvKVejNqqyszMvLy1NapmadPkDHjjBtmur0RSQ/mNk8dy9rLF1OPJE7cWII8MXF\nYBa+FfBFRGpLV+udjJs4UUFeRKQxOVHSFxGReBT0RUTyiIK+iEgeUdAXEckjCvoiInmk1bXTN7NK\n4L0DWEU34KM0Zac1yLX9gdzbp1zbH8i9fcq1/YHa+1Ts7oWNLdTqgv6BMrPyOA8oZItc2x/IvX3K\ntf2B3NunXNsfaPo+qXpHRCSPKOiLiOSRXAz60zKdgTTLtf2B3NunXNsfyL19yrX9gSbuU87V6YuI\nSP1ysaQvIiL1UNAXEckjORP0zWy0mS01s+VmdlOm85MOZrbKzBaZ2XwzS+0lA62AmT1sZuvN7K2k\naUeY2XNm9k703SWTeUxVPfs0xczWRMdpvpmNyWQeU2FmPczsJTNbYmaLzez6aHpWHqcG9iebj9HB\nZva/ZrYg2qfvR9N7mdnfo5j3m+jNho2vLxfq9M2sgPBe3lGE9/XOBSY0x1u6WpKZrQLK3D0rHyox\ns1OBbcCv3b1fNO3fgI3u/sPo5NzF3f9PJvOZinr2aQqwzd1/nMm8NYWZHQUc5e5vmFlnYB5wATCJ\nLDxODezPV8jeY2RAJ3ffZmZtgTnA9cCNwO/cfaaZ/QxY4O4/bWx9uVLSHwYsd/cV7r4TmAmMzXCe\n8p67vwpsrDF5LPCraPhXhH/IrFHPPmUtd1/r7m9Ew1sJr0TtTpYepwb2J2t5sC0abRt9HDgDeDya\nHvsY5UrQ7w6sThqvIMsPdMSBP5vZPDObnOnMpMln3H1tNPwh8JlMZiaNrjWzhVH1T1ZUhdRkZiXA\nYODv5MBxqrE/kMXHyMwKzGw+sB54DngX+Dh6hzmkEPNyJejnqhHuPgQ4B7gmqlrIGR7qFrO/fhF+\nChwLDALWAvdmNjupM7NDgCeAf3b3LcnzsvE41bE/WX2M3H2Puw8Cigg1G32auq5cCfprgB5J40XR\ntKzm7mui7/XAk4SDne3WRfWuifrX9RnOzwFz93XRP+Ve4CGy7DhF9cRPANPd/XfR5Kw9TnXtT7Yf\nowR3/xh4CTgZONzMEq+8jR3zciXozwV6R3ez2wHjgdkZztMBMbNO0Y0ozKwTcBbwVsNLZYXZwOXR\n8OXA7zOYl7RIBMfIhWTRcYpuEv438La7/3vSrKw8TvXtT5Yfo0IzOzwa7kBosPI2IfiPi5LFPkY5\n0XoHIGqCdT9QADzs7ndmOEsHxMyOIZTuIbzA/rFs2yczmwGMJHQBuw64A3gKmAX0JHSh/RV3z5ob\no/Xs00hCtYEDq4BvJtWHt2pmNgJ4DVgE7I0m30KoB8+649TA/kwge4/RAMKN2gJCQX2Wu0+NYsRM\n4AjgTeAyd/+00fXlStAXEZHG5Ur1joiIxKCgLyKSRxT0RUTyiIK+iEgeUdAXEckjCvoiInlEQV9E\nJI/8f8VPE/svVwa/AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7fd5529c2e80>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuUFNW59/Hvw3XkElAgKtdBReV+cUR9EQGvqBHEIAHx\nAieG6IkxiTFLlhpjiKyDxqDR8HokeTVGR5GjUfEWNEcimkTkEgQRCYiAAwgDCoKAOMzz/rG7h2bo\nmeme6Zme7v591qrVXdW7q3d1zTy1+6ldu8zdERGR7NIg3RUQEZHUU3AXEclCCu4iIllIwV1EJAsp\nuIuIZCEFdxGRLKTgLnGZWUMz221mnVNZNp3M7AQzS3nfXzM718zWxcyvMrPBiZStxmf9wcxure77\nK1nvXWb2x1SvV9KnUborIKlhZrtjZpsBXwEHIvPfd/fCZNbn7geAFqkumwvc/aRUrMfMrgWudPeh\nMeu+NhXrluyn4J4l3L0suEZahte6+18rKm9mjdy9pC7qJiJ1T2mZHBH52f20mT1lZruAK83sDDN7\nx8x2mNlmM3vAzBpHyjcyMzez/Mj8E5HXXzWzXWb2TzPrmmzZyOsXmtm/zWynmT1oZn83swkV1DuR\nOn7fzNaY2edm9kDMexua2X1mtt3M1gLDK/l+bjOzWeWWzTCz6ZHn15rZysj2fBRpVVe0riIzGxp5\n3szMHo/UbQVwSrmyt5vZ2sh6V5jZiMjy3sDvgMGRlNe2mO/2zpj3XxfZ9u1m9ryZHZvId1MVMxsV\nqc8OM3vDzE6Kee1WM9tkZl+Y2Ycx23q6mS2JLN9iZr9O9POkFri7piybgHXAueWW3QXsBy4hHNSP\nAE4FTiP8gjsO+DdwQ6R8I8CB/Mj8E8A2oABoDDwNPFGNst8EdgEjI6/dBHwNTKhgWxKp4wtAKyAf\n+Cy67cANwAqgI9AGmB/+5ON+znHAbqB5zLq3AgWR+UsiZQw4G9gL9Im8di6wLmZdRcDQyPN7gb8B\nRwJdgA/KlR0DHBvZJ1dE6nB05LVrgb+Vq+cTwJ2R5+dH6tgPyAP+L/BGIt9NnO2/C/hj5Hn3SD3O\njuyjW4FVkec9gfXAMZGyXYHjIs8XAuMiz1sCp6X7fyGXJ7Xcc8vb7v6iu5e6+153X+juC9y9xN3X\nAjOBIZW8/xl3X+TuXwOFhKCSbNlvAUvd/YXIa/cRDgRxJVjH/3L3ne6+jhBIo581BrjP3YvcfTsw\nrZLPWQu8TzjoAJwHfO7uiyKvv+juaz14A/hfIO5J03LGAHe5++fuvp7QGo/93NnuvjmyT54kHJgL\nElgvwHjgD+6+1N33AZOBIWbWMaZMRd9NZcYCc9z9jcg+mkY4QJwGlBAOJD0jqb2PI98dhIN0NzNr\n4+673H1BgtshtUDBPbd8EjtjZieb2ctm9qmZfQFMAdpW8v5PY57vofKTqBWVbR9bD3d3Qks3rgTr\nmNBnEVqclXkSGBd5fkVkPlqPb5nZAjP7zMx2EFrNlX1XUcdWVgczm2Bm70XSHzuAkxNcL4TtK1uf\nu38BfA50iCmTzD6raL2lhH3Uwd1XAT8l7IetkTTfMZGiE4EewCoze9fMLkpwO6QWKLjnlvLdAB8m\ntFZPcPdvAHcQ0g61aTMhTQKAmRmHBqPyalLHzUCnmPmqumrOBs41sw6EFvyTkToeATwD/BchZdIa\neC3BenxaUR3M7DjgIeB6oE1kvR/GrLeqbpubCKme6PpaEtI/GxOoVzLrbUDYZxsB3P0Jdx9ESMk0\nJHwvuPsqdx9LSL39BnjWzPJqWBepJgX33NYS2Al8aWbdge/XwWe+BAwws0vMrBHwI6BdLdVxNvBj\nM+tgZm2AWyor7O6fAm8DfwRWufvqyEtNgSZAMXDAzL4FnJNEHW41s9YWrgO4Iea1FoQAXkw4zn2P\n0HKP2gJ0jJ5AjuMp4Ltm1sfMmhKC7FvuXuEvoSTqPMLMhkY++2eE8yQLzKy7mQ2LfN7eyFRK2ICr\nzKxtpKW/M7JtpTWsi1STgntu+ylwDeEf92HCic9a5e5bgO8A04HtwPHAvwj98lNdx4cIufHlhJN9\nzyTwnicJJ0jLUjLuvgP4CfAc4aTkaMJBKhG/IPyCWAe8CvwpZr3LgAeBdyNlTgJi89SvA6uBLWYW\nm16Jvv8vhPTIc5H3dybk4WvE3VcQvvOHCAee4cCISP69KXAP4TzJp4RfCrdF3noRsNJCb6x7ge+4\n+/6a1keqx0LKUyQ9zKwhIQ0w2t3fSnd9RLKFWu5S58xseCRN0RT4OaGXxbtprpZIVlFwl3Q4E1hL\n+Ml/ATDK3StKy4hINSgtIyKShdRyFxHJQmkbOKxt27aen5+fro8XEclIixcv3ubulXUfBtIY3PPz\n81m0aFG6Pl5EJCOZWVVXWgNKy4iIZCUFdxGRLKTgLiKShXQnJpEc8fXXX1NUVMS+ffvSXRVJQF5e\nHh07dqRx44qGFqqcgrtIjigqKqJly5bk5+cTBuOU+srd2b59O0VFRXTt2rXqN8SRUWmZwkLIz4cG\nDcJjYVK3fBbJbfv27aNNmzYK7BnAzGjTpk2NfmVlTMu9sBAmTYI9e8L8+vVhHmB8jcfBE8kNCuyZ\no6b7KmNa7rfddjCwR+3ZE5aLiMihMia4b9iQ3HIRqV+2b99Ov3796NevH8cccwwdOnQom9+/P7Fh\n3ydOnMiqVasqLTNjxgwKU5SzPfPMM1m6dGlK1lXXMiYt07lzSMXEWy4iqVdYGH4Zb9gQ/s+mTq1Z\nCrRNmzZlgfLOO++kRYsW3HzzzYeUcXfcnQYN4rc7H3300So/5wc/+EH1K5lFMqblPnUqNGt26LJm\nzcJyEUmt6Dmu9evB/eA5rtroxLBmzRp69OjB+PHj6dmzJ5s3b2bSpEkUFBTQs2dPpkyZUlY22pIu\nKSmhdevWTJ48mb59+3LGGWewdetWAG6//Xbuv//+svKTJ09m4MCBnHTSSfzjH/8A4Msvv+Tb3/42\nPXr0YPTo0RQUFFTZQn/iiSfo3bs3vXr14tZbbwWgpKSEq666qmz5Aw88AMB9991Hjx496NOnD1de\neWXKv7NEZEzLPdpiSGVLQkTiq+wcV238z3344Yf86U9/oqCgAIBp06Zx1FFHUVJSwrBhwxg9ejQ9\nevQ45D07d+5kyJAhTJs2jZtuuolHHnmEyZMnH7Zud+fdd99lzpw5TJkyhb/85S88+OCDHHPMMTz7\n7LO89957DBgwoNL6FRUVcfvtt7No0SJatWrFueeey0svvUS7du3Ytm0by5cvB2DHjh0A3HPPPaxf\nv54mTZqULatrGdNyh/BHtW4dlJaGRwV2kdpR1+e4jj/++LLADvDUU08xYMAABgwYwMqVK/nggw8O\ne88RRxzBhRdeCMApp5zCunXr4q77sssuO6zM22+/zdixYwHo27cvPXv2rLR+CxYs4Oyzz6Zt27Y0\nbtyYK664gvnz53PCCSewatUqbrzxRubOnUurVq0A6NmzJ1deeSWFhYXVvgippjIquItI3ajoXFZt\nneNq3rx52fPVq1fz29/+ljfeeINly5YxfPjwuP29mzRpUva8YcOGlJSUxF1306ZNqyxTXW3atGHZ\nsmUMHjyYGTNm8P3vfx+AuXPnct1117Fw4UIGDhzIgQMHUvq5iVBwF5HDpPMc1xdffEHLli35xje+\nwebNm5k7d27KP2PQoEHMnj0bgOXLl8f9ZRDrtNNOY968eWzfvp2SkhJmzZrFkCFDKC4uxt25/PLL\nmTJlCkuWLOHAgQMUFRVx9tlnc88997Bt2zb2lM9x1YGEcu5mNhz4LdAQ+IO7T4tTZgxwJ+DAe+5+\nRQrrKSJ1KJ3nuAYMGECPHj04+eST6dKlC4MGDUr5Z/zwhz/k6quvpkePHmVTNKUST8eOHfnVr37F\n0KFDcXcuueQSLr74YpYsWcJ3v/td3B0z4+6776akpIQrrriCXbt2UVpays0330zLli1Tvg1VqfIe\nqmbWEPg3cB5QBCwExrn7BzFlugGzgbPd/XMz+6a7b61svQUFBa6bdYjUnZUrV9K9e/d0V6NeKCkp\noaSkhLy8PFavXs3555/P6tWradSofvUxibfPzGyxuxdU8JYyiWzJQGCNu6+NrHgWMBKI/R3zPWCG\nu38OUFVgFxFJp927d3POOedQUlKCu/Pwww/Xu8BeU4lsTQfgk5j5IuC0cmVOBDCzvxNSN3e6+1/K\nr8jMJgGTADrr6iMRSZPWrVuzePHidFejVqXqhGojoBswFBgH/N7MWpcv5O4z3b3A3Qvatavy/q4i\nIlJNiQT3jUCnmPmOkWWxioA57v61u39MyNF3S00VRUQkWYkE94VANzPramZNgLHAnHJlnie02jGz\ntoQ0zdoU1lNERJJQZXB39xLgBmAusBKY7e4rzGyKmY2IFJsLbDezD4B5wM/cfXttVVpERCqXUM7d\n3V9x9xPd/Xh3nxpZdoe7z4k8d3e/yd17uHtvd59Vm5UWkcwzbNiwwy5Iuv/++7n++usrfV+LFi0A\n2LRpE6NHj45bZujQoVTVtfr+++8/5GKiiy66KCXjvtx5553ce++9NV5PqukKVRGpE+PGjWPWrEPb\nfbNmzWLcuHEJvb99+/Y888wz1f788sH9lVdeoXXrw/p9ZA0FdxGpE6NHj+bll18uuzHHunXr2LRp\nE4MHDy7rdz5gwAB69+7NCy+8cNj7161bR69evQDYu3cvY8eOpXv37owaNYq9e/eWlbv++uvLhgv+\nxS9+AcADDzzApk2bGDZsGMOGDQMgPz+fbdu2ATB9+nR69epFr169yoYLXrduHd27d+d73/sePXv2\n5Pzzzz/kc+JZunQpp59+On369GHUqFF8/vnnZZ8fHQI4OmDZm2++WXazkv79+7Nr165qf7fxZFev\nfRFJyI9/DKm+wVC/fhCJi3EdddRRDBw4kFdffZWRI0cya9YsxowZg5mRl5fHc889xze+8Q22bdvG\n6aefzogRIyq8j+hDDz1Es2bNWLlyJcuWLTtkyN6pU6dy1FFHceDAAc455xyWLVvGjTfeyPTp05k3\nbx5t27Y9ZF2LFy/m0UcfZcGCBbg7p512GkOGDOHII49k9erVPPXUU/z+979nzJgxPPvss5WOz371\n1Vfz4IMPMmTIEO644w5++ctfcv/99zNt2jQ+/vhjmjZtWpYKuvfee5kxYwaDBg1i9+7d5OXlJfFt\nV00tdxGpM7GpmdiUjLtz66230qdPH84991w2btzIli1bKlzP/Pnzy4Jsnz596NOnT9lrs2fPZsCA\nAfTv358VK1ZUOSjY22+/zahRo2jevDktWrTgsssu46233gKga9eu9OvXD6h8WGEI48vv2LGDIUOG\nAHDNNdcwf/78sjqOHz+eJ554ouxK2EGDBnHTTTfxwAMPsGPHjpRfIauWu0gOqqyFXZtGjhzJT37y\nE5YsWcKePXs45ZRTACgsLKS4uJjFixfTuHFj8vPz4w7zW5WPP/6Ye++9l4ULF3LkkUcyYcKEaq0n\nKjpcMIQhg6tKy1Tk5ZdfZv78+bz44otMnTqV5cuXM3nyZC6++GJeeeUVBg0axNy5czn55JOrXdfy\n1HIXkTrTokULhg0bxn/8x38cciJ1586dfPOb36Rx48bMmzeP9fFumBzjrLPO4sknnwTg/fffZ9my\nZUAYLrh58+a0atWKLVu28Oqrr5a9p2XLlnHz2oMHD+b5559nz549fPnllzz33HMMHjw46W1r1aoV\nRx55ZFmr//HHH2fIkCGUlpbyySefMGzYMO6++2527tzJ7t27+eijj+jduze33HILp556Kh9++GHS\nn1kZtdxFpE6NGzeOUaNGHdJzZvz48VxyySX07t2bgoKCKluw119/PRMnTqR79+5079697BdA3759\n6d+/PyeffDKdOnU6ZLjgSZMmMXz4cNq3b8+8efPKlg8YMIAJEyYwcOBAAK699lr69+9faQqmIo89\n9hjXXXcde/bs4bjjjuPRRx/lwIEDXHnllezcuRN358Ybb6R169b8/Oc/Z968eTRo0ICePXuW3VUq\nVaoc8re2aMhfkbqlIX8zT02G/FVaRkQkCym4i4hkIQV3kRySrjSsJK+m+0rBXSRH5OXlsX37dgX4\nDODubN++vUYXNqm3jEiO6NixI0VFRRQXF6e7KpKAvLw8OnbsWO33K7iL5IjGjRvTtWvXdFdD6ojS\nMiIiWUjBXUQkCym4i4hkIQV3EZEspOAuIpKFFNxFRLKQgruISBZScBcRyUIK7iIiWUjBXUQkCym4\ni4hkIQV3EZEspOAuIpKFFNxFRLJQQsHdzIab2SozW2Nmk+O8PsHMis1saWS6NvVVFRGRRFU5nruZ\nNQRmAOcBRcBCM5vj7h+UK/q0u99QC3UUEZEkJdJyHwiscfe17r4fmAWMrN1qiYhITSQS3DsAn8TM\nF0WWlfdtM1tmZs+YWad4KzKzSWa2yMwW6VZfIiK1J1UnVF8E8t29D/A68Fi8Qu4+090L3L2gXbt2\nKfpoEREpL5HgvhGIbYl3jCwr4+7b3f2ryOwfgFNSUz0REamORIL7QqCbmXU1sybAWGBObAEzOzZm\ndgSwMnVVFBGRZFXZW8bdS8zsBmAu0BB4xN1XmNkUYJG7zwFuNLMRQAnwGTChFussIiJVMHdPywcX\nFBT4okWL0vLZIiKZyswWu3tBVeV0haqISBZScBcRyUIK7iIiWUjBXUQkCym4i4hkIQV3EZEspOAu\nIpKFFNxFRLKQgruISBZScBcRyUIZGdy/+qrqMoWFkJ8PDRqEx8LC2q6ViEj9kXHB/Xe/g86dYe/e\nissUFsKkSbB+PbiHx0mTFOBFJHdkXHDv3h22boXnn6+4zG23wZ49hy7bsycsFxHJBRkX3IcNgy5d\n4I9/rLjMhg3JLRcRyTYZF9wbNIBrroHXX4dPPolfpnPn5JaLiGSbjAvuEIK7O/zpT/FfnzoVmjU7\ndFmzZmG5iEguyMjgftxxMGRISM3Eu9fI+PEwc2ZI35iFx5kzw3IRkVyQkcEdYOJEWLMG3n47/uvj\nx8O6dVBaGh4V2EUkl2RscB89Glq0qPzEqohIrsrY4N68OYwZA7Nnw5dfprs2IiL1S8YGdwipmd27\n4Zln0l0TEZH6JaOD+6BBcMIJ8Oij6a6JiEj9ktHB3QwmTIA334S1a9NdGxGR+iOjgzvA1VeHIP/Y\nY+muiYhI/ZHxwb1TJzjvvBDcS0vTXRsRkfoh44M7hNTM+vUwb166ayIiUj9kRXC/9FJo1UonVkVE\norIiuB9xBIwbB3/+M+zcme7aiIikX0LB3cyGm9kqM1tjZpMrKfdtM3MzK0hdFRMzcWK4gcfs2XX9\nySIi9U+Vwd3MGgIzgAuBHsA4M+sRp1xL4EfAglRXMhGnngo9eig1IyICibXcBwJr3H2tu+8HZgEj\n45T7FXA3sC+F9UtYtM/7P/8JH36YjhqIiNQfiQT3DkDsbTGKIsvKmNkAoJO7v5zCuiXtqqugYUMN\nJiYiUuMTqmbWAJgO/DSBspPMbJGZLSouLq7pRx/mmGPgwgvh8cfhwIGUr15EJGMkEtw3Ap1i5jtG\nlkW1BHoBfzOzdcDpwJx4J1Xdfaa7F7h7Qbt27apf60pMnAibNsFrr9XK6kVEMkIiwX0h0M3MuppZ\nE2AsMCf6orvvdPe27p7v7vnAO8AId19UKzWuwre+BW3a6MSqiOS2KoO7u5cANwBzgZXAbHdfYWZT\nzGxEbVcwWU2ahLsuvfACfPZZumsjIpIeCeXc3f0Vdz/R3Y9396mRZXe4+5w4ZYemq9UeNXEi7N8P\nTz6ZzlqIiKRPVlyhWl6/fmFSrxkRyVVZGdwhtN4XL4bly9NdExGRupe1wf2KK6BxY5g+Pd01ERGp\ne1kb3Nu2hR//OKRm/v73dNdGRKRuZW1wB7jjjnAzj+uug6+/jl+msBDy86FBg/BYWFiXNRQRqR1Z\nHdxbtIAHH4T334ff/vbw1wsLYdKkcKMP9/A4aZICvIhkPnP3tHxwQUGBL1pUNz0mR4yAN96AlStD\nSz4qPz8E9PK6dIF16+qkaiIiSTGzxe5e5bDqWd1yj3rggXB/1R/96NDlGzbEL1/RchGRTJETwT0/\nP+Tfn3sOXnrp4PLOneOXr2i5iEimyIngDnDTTeFmHj/8IezZE5ZNnQrNmh1arlmzsFxEJJPlTHBv\n0gQeeijk0u+6KywbPx5mzgw5drPwOHNmWC4iksly4oRqrAkTwpgzS5eGlryISCbRCdUK/PrXoYvk\nf/5n6P4oIpKNci64t2sH06bBm2+GOzaJiGSjnAvuANdeC6efDjffrDHfRSQ75WRwb9AA/vu/Q2C/\n9dZ010ZEJPVyMrgD9O0LN94Yese88066ayMiklo5G9wBfvlLaN8+DCxWUpLu2oiIpE5OB/eWLcOA\nYu+9B7/7XbprIyKSOjkd3AEuuwwuvBBuuw2efjrdtRERSY2cD+5m8Mgj4Z6rY8fCDTfAV1+lu1Yi\nIjWT88Ed4Jhj4G9/C10jZ8yAQYPg44/TXSsRkepTcI9o3Dhcvfr88/DRR9C/P7zwwqFldNcmEckU\nCu7ljBwJS5ZAt25w6aXw05+GW/Tprk0ikklybuCwRH31VUjT/O53cMYZ4QYeGzceXk53bRKRuqSB\nw2qoadNw/9VZs2D58viBHXTXJhGpnxTcq/Cd78DixSEnH4/u2iQi9ZGCewJOPBEefhgaNjx0ue7a\nJCL1lYJ7giZOhMceg7Ztw7xZuABq3LjDy6pXjYikW0LB3cyGm9kqM1tjZpPjvH6dmS03s6Vm9raZ\nZeU9jsaPh+LikH+/6CJ44gkYOjR0nYxSrxoRqQ+qDO5m1hCYAVwI9ADGxQneT7p7b3fvB9wDTE95\nTeuR9u3hxRfh0UfDuDR9+oSLn0pLwzAG0RtwR+3ZE5aLiNSVRFruA4E17r7W3fcDs4CRsQXc/YuY\n2eZA1t/Azizcj3XFChg8OAxbcN55oaUej3rViEhdSiS4dwA+iZkviiw7hJn9wMw+IrTcb4y3IjOb\nZGaLzGxRcXFxdepb73TsCK++GsaFf/fdEPTjUa8aEalLKTuh6u4z3P144Bbg9grKzHT3AncvaNeu\nXao+Ou3M4Hvfg/ffh+7dD39dvWpEpK4lEtw3Ap1i5jtGllVkFnBpTSqVqbp0CRc8XXPNwRZ8y5bw\nq1+Fk7EiInUlkeC+EOhmZl3NrAkwFpgTW8DMusXMXgysTl0VM0uDBvDHP8KaNSHI79sHP/sZjB4N\n//xnumsnIrmiyuDu7iXADcBcYCUw291XmNkUMxsRKXaDma0ws6XATcA1tVbjDHHccSHIr1sHt9wC\nb7wB/+f/hOnZZ+HAgXTXUESymQYOqyO7d4dgf999sHZtCP4/+Um4OKp583TXTkQyhQYOq2datAjd\nJf/9b3jmGTj6aPjhD6FTJ7j11nAytrQ03bUUkWyh4F7HGjaEb38b/vGPMJ19NkybBr17Q7t2YQz5\n6dNh0SIoKUl3bUUkUzVKdwVy2RlnhFb8J5+EnPz8+WGK3gGqRYtwy7+zzgrTqaeGoYhFRKqi4J5m\nhYVhaIING8KFTlOnwrBh8NZbB4N9dOiCpk3DCdkRI8Ido7p2TW/dRaT+0gnVNIoOMhY7Fk2zZuFq\n19h+8du3w9tvh0D/2mshPw9hTJtLLw1Tv34VXx0rItkj0ROqCu5plJ8ffyyaqm7d99FHIXXz/PPw\n97+HE7GdO4fW/KWXhrFuKrq5iIhkNgX3DNCgQRgWuDyzxHvOFBfDSy+FQP/aa+GiqSOPhIsvhnPO\ngdNPDzcbaaBT5yJZQcE9A1S35V6RL78MAf6FF8KQxJ99Fpa3bg2nnRam008Pj0cdVZOai0i6KLhn\ngERz7tVRWgoffggLFsA774Qpti/9iSceDPS9e0NeXkjlNG4MTZocfB67LLpcRNJHwT1DxOstEy+w\nJ1quMrt2hf7z0WD/zjuwdWty6zjpJLjgAjj//HAXKl1dK1K3FNyzSG218KO3AVy1Cr7+Okz79x98\nXn7asycMfvbmmyG336QJnHlmCPQXXBB672Rbbn/btnA+Y+7ccM3B9dfrWgNJLwX3LJLq3HxN7dsX\n+uHPnRty/MuXh+VHHx3uRnXBBeGiqw4dwhW5mWbbNnjuOfif/wkXlx04EG6Mvm1b+M7vuguuuCL7\nDmSSGRTcs0gqetXUpk2bQpB/7TV4/fUQBCHU+5hjQpCvbGrRIjV99EtKwgBtX34Z0kWtWiW+3ngB\n/fjj4fLLw9S/P/z1r2GEz3/9C/r2DcNGXHCBri+IKimBvXvDPQzqsw0bwq/VvXurnvbvDwf0Xr3C\nuanjjkv/QV3BPYsk23JPRX6+ukpLQ/BbuBA2bjx82rHj8PeYhTRTvOmIIw4+b9gwBO5du0IQ3737\n0Of79h263saNQ4u7XbuDj+Wn7dvDEBDz5oWAfsIJBwN6vAvDSkvh6afD9/vxx+Fq4rvvDkND1IaS\nkvB9bt0KAwbAscfWzudUV2lpuMDuqafC9/j55+GAd9VV4UrqZs3SWz93WL364NXe8+dXfJ/jKLPw\nd3fEEeFv6NNPD77WrBn07Hkw2PfuHZ4ffXTdHeQV3LNIMjn32uyBkwpffhla+tFgv2kTfPFFaCXt\n2RN/ir5WUhJa+dGpZcv4z5s3D8G+uPjwadu2ww8w0YA+ZkxokSfyT7p/Pzz8MEyZEtZ5+eXhINqt\nW9Xvrcy+feHAGA1E//hH2JaoDh1g4MCD0ymnhF8odckdliwJAf3pp6GoKATCESPCPYWjy1q2DIPk\nXXUVDBlSNym60tLQKyw2mG/ZEl5r1+7gOE39+oW/k2gQj055eeFcUuzfwO7d8MEHIf0Ynd5//9DO\nCG3bhlZ9s2ZhHdF1xT6PXXb++eEcVXUouGeZRFvj9S0/Xx99/XUIyMXF0KhRuO9tdVtdX3wBv/lN\nmL76KtxL96c/DdcRNG0apsqC2u7d4SR1NBAtWBDWA6FFGA1G7duHgPruu2Fas+bgOk4+OQT6U08N\nU/v2IbC2bJnagPrhhyGgP/VUaA03bhxa6ePGhcDeokUoV1oaTro//nhoze/aFQ5K48eHQN+rV/Xr\ncOBA2G/gx0FMAAAKnElEQVSbNh06bd4c/jcWLAi/HiAcaIYMOfgdnnRSalvXW7ceDPTLl4fP37cv\nTHv3HvoYfR5Noz78cGiEVYeCe46q7/n5bPXpp+FeuTNnHj5Uc8OGBwN906ahZRgN+qtXh4DVoEFI\nu0QD0ZlnQps2FX/eZ5+Fbq3RYP/uuwdbqLGOOOJgoC8/RVNdjRqFx+gUO9+oUTgYvvIKLF0a/o6G\nDQsB/bLLqr4Ybu9emDMHnngC/vKX8N307RsC/bHHHgx8+/aFg1rsfHT64ouDQXzLlvh3MfvmN8NB\n7ZRTDn6HXbrUr/Mh7mH79+0LB8a8vOqtR8E9RyXTck9nbj5brVkTTrxGg1Vl09dfh1b3WWeF0T5r\nciLSPaRCFi8OLdtduw6doucnYqc9e0KgLCkJj/GeR8PDaaeFgD5mTPXz/sXFIWXz+OPhYBRPNN8d\nTV/k5YVfBO3bH5yOPfbQ+aOPzq2L6xTcc1SiOff6npuX+sE9/OJLdb68qCgcAPPywq+YaCBv1Kh+\ntbbrIwX3HJZIi1y5eZHMpOAulVJuXiQz6QbZUqnOnZNbLiKZRcE9R02devgFJs2aheXxFBaGVE6D\nBuGxsLC2aygiNaHgnqPGjw8nT6Pdxbp0qfhkavTk6/r1BwcbmzRJAV6kPlPOXaqkk68i9Ydy7pIy\nGzYkt1xE0k/BXaqkk68imUfBXaqU7MlXEUm/hIK7mQ03s1VmtsbMJsd5/SYz+8DMlpnZ/5pZl9RX\nVdIl2ZOv6lUjkn5VnlA1s4bAv4HzgCJgITDO3T+IKTMMWODue8zsemCou3+nsvXqhGr20ZAGIrUv\nlSdUBwJr3H2tu+8HZgEjYwu4+zx3j/5LvwN0TLbCkvluu+3QwA5h/rbb4pdXK1+k9iQS3DsAn8TM\nF0WWVeS7wKvxXjCzSWa2yMwWFRcXJ15LyQjJ9KpR33mR2pXSE6pmdiVQAPw63uvuPtPdC9y9oF27\ndqn8aKkHkulVk2wrX0SSk0hw3wh0ipnvGFl2CDM7F7gNGOHuX6WmepJJkulVk2wrX+kbkeQkEtwX\nAt3MrKuZNQHGAnNiC5hZf+BhQmDfGmcdkgOS6VWTaCtf6RuR6klo+AEzuwi4H2gIPOLuU81sCrDI\n3eeY2V+B3sDmyFs2uPuIytap3jK5LdGeNRr6QORQKR1+wN1fcfcT3f14d58aWXaHu8+JPD/X3Y92\n936RqdLALpJoKz/ZoQ+UwhEJdIWqpM348aH1XVoaHmuSvoHkUjg6CEi2U3CXei2Zk7SJ9sBRHl9y\ngYK71GvJnKRNNIWjbpiSCxqluwIiVRk/PrHhCzp3jn/ytXwKR0MYSy5Qy12yRqIpHA1hLLlAwV2y\nRqIpHA1hLLlAwV2ySiI9cJLJ40PiPWvUA0fqE91DVaQSiV5speGOpa4kehGTgrtIJRK9QlZX0kpd\n0Q2yRVIg0Z41upJW6hsFd5FKJNqzRlfSSn2j4C5SiUR71uhKWqlvFNxFKpFozxpdSSv1jU6oitSx\nRE++NmgQWuzlmYWunuUVFobAv2FDSAdNnaqeOtlIJ1RF6qnauJJWKRwpT8FdpI7VxpW0SuFIeUrL\niNRjiaZakk3hSOZSWkYkCyQynAIkn8JR98rsp+AukgUSTeEkm5vXgSBzKbiLZIFE8/jJ5OZ1sVVm\nU85dJIckk5tPtMtmsoOmqctmzSjnLiKHSSY3XxsXW6nLZt1RcBfJIcl0r0z0QJDMoGnqsll3FNxF\nckgywyTUxsVWyRwIlMevGQV3kRyTaPfK2rjYKtEDgdI3NafgLiIVSvVtCxM9ECSbvlEr/3AK7iJS\nY6n+NZBs+qY2umxm+gFDXSFFpN5J5raFtdFlsz7fE1ddIUUkYyWTx6+NLpvZ0KsnoeBuZsPNbJWZ\nrTGzyXFeP8vMlphZiZmNTn01RSSXJJPHr40um9nQq6fK4G5mDYEZwIVAD2CcmfUoV2wDMAF4MtUV\nFJHclGgevza6bNZWr566PBAk0nIfCKxx97Xuvh+YBYyMLeDu69x9GaDBRUWkTtVGl83a6NVT1907\nEwnuHYBPYuaLIsuSZmaTzGyRmS0qLi6uzipERA6T6i6btdGrp67z+I1qZ7XxuftMYCaE3jJ1+dki\nIuPHJ97bJZGynTvH76lT06tzUyGRlvtGoFPMfMfIMhGRnFYbV+emSiLBfSHQzcy6mlkTYCwwp3aq\nIyKSOWrj6txUqTK4u3sJcAMwF1gJzHb3FWY2xcxGAJjZqWZWBFwOPGxmK2qnuiIi9Uuqr85NFV2h\nKiKSQXSFqohIDlNwFxHJQgruIiJZSMFdRCQLKbiLiGShtPWWMbNiIM61XQlpC2xLYXXqg2zbpmzb\nHsi+bcq27YHs26Z429PF3dtV9ca0BfeaMLNFiXQFyiTZtk3Ztj2QfduUbdsD2bdNNdkepWVERLKQ\ngruISBbK1OA+M90VqAXZtk3Ztj2QfduUbdsD2bdN1d6ejMy5i4hI5TK15S4iIpVQcBcRyUIZF9zN\nbLiZrTKzNWY2Od31qSkzW2dmy81sqZll5DCZZvaImW01s/djlh1lZq+b2erI45HprGMyKtieO81s\nY2Q/LTWzi9JZx2SZWSczm2dmH5jZCjP7UWR5Ru6nSrYnY/eTmeWZ2btm9l5km34ZWd7VzBZEYt7T\nkftqVL2+TMq5m1lD4N/AeYR7uS4Exrn7B2mtWA2Y2TqgwN0z9sILMzsL2A38yd17RZbdA3zm7tMi\nB+Ej3f2WdNYzURVsz53Abne/N511qy4zOxY41t2XmFlLYDFwKTCBDNxPlWzPGDJ0P5mZAc3dfbeZ\nNQbeBn4E3AT82d1nmdl/A++5+0NVrS/TWu4DgTXuvtbd9wOzgJFprlPOc/f5wGflFo8EHos8f4zw\nj5cRKtiejObum919SeT5LsKNdzqQofupku3JWB7sjsw2jkwOnA08E1me8D7KtODeAfgkZr6IDN+h\nhJ33mpktNrNJ6a5MCh3t7psjzz8Fjk5nZVLkBjNbFknbZET6Ih4zywf6AwvIgv1Ubnsgg/eTmTU0\ns6XAVuB14CNgR+SOeJBEzMu04J6NznT3AcCFwA8iKYGs4iH3lzn5v/geAo4H+gGbgd+ktzrVY2Yt\ngGeBH7v7F7GvZeJ+irM9Gb2f3P2Au/cDOhIyFSdXd12ZFtw3Ap1i5jtGlmUsd98YedwKPEfYodlg\nSyQvGs2Pbk1zfWrE3bdE/vFKgd+Tgfspksd9Fih09z9HFmfsfoq3PdmwnwDcfQcwDzgDaG1mjSIv\nJRzzMi24LwS6Rc4eNwHGAnPSXKdqM7PmkZNBmFlz4Hzg/crflTHmANdEnl8DvJDGutRYNABGjCLD\n9lPkZN3/A1a6+/SYlzJyP1W0PZm8n8ysnZm1jjw/gtBxZCUhyI+OFEt4H2VUbxmASNem+4GGwCPu\nPjXNVao2MzuO0FoHaAQ8mYnbY2ZPAUMJw5NuAX4BPA/MBjoThnYe4+4ZcZKygu0ZSvip78A64Psx\nuep6z8zOBN4ClgOlkcW3EvLUGbefKtmecWTofjKzPoQTpg0JDe/Z7j4lEidmAUcB/wKudPevqlxf\npgV3ERGpWqalZUREJAEK7iIiWUjBXUQkCym4i4hkIQV3EZEspOAuIpKFFNxFRLLQ/wfrVYYmygVP\nVgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7fd550e7c828>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "acc = history.history['acc']\n",
        "val_acc = history.history['val_acc']\n",
        "loss = history.history['loss']\n",
        "val_loss = history.history['val_loss']\n",
        "\n",
        "epochs = range(len(acc))\n",
        "\n",
        "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
        "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
        "plt.title('Training and validation accuracy')\n",
        "plt.legend()\n",
        "\n",
        "plt.figure()\n",
        "\n",
        "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
        "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
        "plt.title('Training and validation loss')\n",
        "plt.legend()\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_jgTHEwqy70H"
      },
      "source": [
        "\n",
        "We reach a validation accuracy of about 90%, much better than what we could achieve in the previous section with our small model trained from\n",
        "scratch. However, our plots also indicate that we are overfitting almost from the start -- despite using dropout with a fairly large rate.\n",
        "This is because this technique does not leverage data augmentation, which is essential to preventing overfitting with small image datasets.\n",
        "\n",
        "Now, let's review the second technique we mentioned for doing feature extraction, which is much slower and more expensive, but which allows\n",
        "us to leverage data augmentation during training: extending the `conv_base` model and running it end-to-end on the inputs. Note that this\n",
        "technique is in fact so expensive that you should only attempt it if you have access to a GPU: it is absolutely intractable on CPU. If you\n",
        "cannot run your code on GPU, then the previous technique is the way to go.\n",
        "\n",
        "Because models behave just like layers, you can add a model (like our `conv_base`) to a `Sequential` model just like you would add a layer.\n",
        "So you can do the following:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 41,
      "metadata": {
        "collapsed": true,
        "id": "Ed5yQ-Gry70I"
      },
      "outputs": [],
      "source": [
        "from keras import models\n",
        "from keras import layers\n",
        "\n",
        "model = models.Sequential()\n",
        "model.add(conv_base)\n",
        "model.add(layers.Flatten())\n",
        "model.add(layers.Dense(256, activation='relu'))\n",
        "model.add(layers.Dense(1, activation='sigmoid'))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U-NIUwsay70J"
      },
      "source": [
        "This is what our model looks like now:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 42,
      "metadata": {
        "id": "-14Ugj-Ly70K",
        "outputId": "d4dfb086-f3be-4478-a828-87823f053319",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential_3\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " vgg16 (Functional)          (None, 4, 4, 512)         14714688  \n",
            "                                                                 \n",
            " flatten_1 (Flatten)         (None, 8192)              0         \n",
            "                                                                 \n",
            " dense_6 (Dense)             (None, 256)               2097408   \n",
            "                                                                 \n",
            " dense_7 (Dense)             (None, 1)                 257       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 16,812,353\n",
            "Trainable params: 16,812,353\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wFy8ao9Fy70M"
      },
      "source": [
        "As you can see, the convolutional base of VGG16 has 14,714,688 parameters, which is very large. The classifier we are adding on top has 2\n",
        "million parameters.\n",
        "\n",
        "Before we compile and train our model, a very important thing to do is to freeze the convolutional base. \"Freezing\" a layer or set of\n",
        "layers means preventing their weights from getting updated during training. If we don't do this, then the representations that were\n",
        "previously learned by the convolutional base would get modified during training. Since the `Dense` layers on top are randomly initialized,\n",
        "very large weight updates would be propagated through the network, effectively destroying the representations previously learned.\n",
        "\n",
        "In Keras, freezing a network is done by setting its `trainable` attribute to `False`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 43,
      "metadata": {
        "id": "1srdO_WDy70N",
        "outputId": "9a69b5c5-e45f-4966-f98c-4a0bce4c04c7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "This is the number of trainable weights before freezing the conv base: 30\n"
          ]
        }
      ],
      "source": [
        "print('This is the number of trainable weights '\n",
        "      'before freezing the conv base:', len(model.trainable_weights))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 44,
      "metadata": {
        "collapsed": true,
        "id": "4LgwzAl1y70N"
      },
      "outputs": [],
      "source": [
        "conv_base.trainable = False"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 45,
      "metadata": {
        "id": "WCzgO3qMy70O",
        "outputId": "1f3276ce-7809-469e-fdfa-855598b70bb6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "This is the number of trainable weights after freezing the conv base: 4\n"
          ]
        }
      ],
      "source": [
        "print('This is the number of trainable weights '\n",
        "      'after freezing the conv base:', len(model.trainable_weights))"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model.summary()"
      ],
      "metadata": {
        "id": "qr3eNG1A0hCN",
        "outputId": "459e118a-c18c-4864-c79f-7a3611fc4d20",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential_3\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " vgg16 (Functional)          (None, 4, 4, 512)         14714688  \n",
            "                                                                 \n",
            " flatten_1 (Flatten)         (None, 8192)              0         \n",
            "                                                                 \n",
            " dense_6 (Dense)             (None, 256)               2097408   \n",
            "                                                                 \n",
            " dense_7 (Dense)             (None, 1)                 257       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 16,812,353\n",
            "Trainable params: 2,097,665\n",
            "Non-trainable params: 14,714,688\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8VDX-Y2dy70O"
      },
      "source": [
        "With this setup, only the weights from the two `Dense` layers that we added will be trained. That's a total of four weight tensors: two per\n",
        "layer (the main weight matrix and the bias vector). Note that in order for these changes to take effect, we must first compile the model.\n",
        "If you ever modify weight trainability after compilation, you should then re-compile the model, or these changes would be ignored.\n",
        "\n",
        "Now we can start training our model, with the same data augmentation configuration that we used in our previous example:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aQu65cjly70P",
        "outputId": "2a16f47b-9386-4ae8-b761-e1a99d0aa15d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Found 2000 images belonging to 2 classes.\n",
            "Found 1000 images belonging to 2 classes.\n",
            "Epoch 1/30\n",
            "74s - loss: 0.4465 - acc: 0.7810 - val_loss: 0.2056 - val_acc: 0.9120\n",
            "Epoch 2/30\n",
            "72s - loss: 0.2738 - acc: 0.8905 - val_loss: 0.1239 - val_acc: 0.9550\n",
            "Epoch 3/30\n",
            "72s - loss: 0.2088 - acc: 0.9145 - val_loss: 0.1194 - val_acc: 0.9560\n",
            "Epoch 4/30\n",
            "72s - loss: 0.1835 - acc: 0.9280 - val_loss: 0.1025 - val_acc: 0.9550\n",
            "Epoch 5/30\n",
            "72s - loss: 0.1642 - acc: 0.9330 - val_loss: 0.0903 - val_acc: 0.9680\n",
            "Epoch 6/30\n",
            "72s - loss: 0.1360 - acc: 0.9410 - val_loss: 0.0794 - val_acc: 0.9740\n",
            "Epoch 7/30\n",
            "72s - loss: 0.1426 - acc: 0.9465 - val_loss: 0.0968 - val_acc: 0.9560\n",
            "Epoch 8/30\n",
            "72s - loss: 0.1013 - acc: 0.9580 - val_loss: 0.1411 - val_acc: 0.9430\n",
            "Epoch 9/30\n",
            "72s - loss: 0.1177 - acc: 0.9500 - val_loss: 0.2105 - val_acc: 0.9310\n",
            "Epoch 10/30\n",
            "72s - loss: 0.0949 - acc: 0.9620 - val_loss: 0.0900 - val_acc: 0.9710\n",
            "Epoch 11/30\n",
            "72s - loss: 0.0915 - acc: 0.9655 - val_loss: 0.1204 - val_acc: 0.9630\n",
            "Epoch 12/30\n",
            "72s - loss: 0.0782 - acc: 0.9645 - val_loss: 0.0995 - val_acc: 0.9650\n",
            "Epoch 13/30\n",
            "72s - loss: 0.0717 - acc: 0.9755 - val_loss: 0.1269 - val_acc: 0.9580\n",
            "Epoch 14/30\n",
            "72s - loss: 0.0670 - acc: 0.9715 - val_loss: 0.0994 - val_acc: 0.9680\n",
            "Epoch 15/30\n",
            "71s - loss: 0.0718 - acc: 0.9735 - val_loss: 0.0558 - val_acc: 0.9790\n",
            "Epoch 16/30\n",
            "72s - loss: 0.0612 - acc: 0.9780 - val_loss: 0.0870 - val_acc: 0.9690\n",
            "Epoch 17/30\n",
            "71s - loss: 0.0693 - acc: 0.9765 - val_loss: 0.0972 - val_acc: 0.9720\n",
            "Epoch 18/30\n",
            "71s - loss: 0.0596 - acc: 0.9785 - val_loss: 0.0832 - val_acc: 0.9730\n",
            "Epoch 19/30\n",
            "71s - loss: 0.0497 - acc: 0.9800 - val_loss: 0.1160 - val_acc: 0.9610\n",
            "Epoch 20/30\n",
            "71s - loss: 0.0546 - acc: 0.9780 - val_loss: 0.1057 - val_acc: 0.9660\n",
            "Epoch 21/30\n",
            "71s - loss: 0.0568 - acc: 0.9825 - val_loss: 0.2012 - val_acc: 0.9500\n",
            "Epoch 22/30\n",
            "71s - loss: 0.0493 - acc: 0.9830 - val_loss: 0.1384 - val_acc: 0.9610\n",
            "Epoch 23/30\n",
            "71s - loss: 0.0328 - acc: 0.9905 - val_loss: 0.1281 - val_acc: 0.9640\n",
            "Epoch 24/30\n",
            "71s - loss: 0.0524 - acc: 0.9860 - val_loss: 0.0846 - val_acc: 0.9760\n",
            "Epoch 25/30\n",
            "71s - loss: 0.0422 - acc: 0.9845 - val_loss: 0.1002 - val_acc: 0.9670\n",
            "Epoch 26/30\n",
            "71s - loss: 0.0617 - acc: 0.9825 - val_loss: 0.0858 - val_acc: 0.9760\n",
            "Epoch 27/30\n",
            "71s - loss: 0.0568 - acc: 0.9830 - val_loss: 0.0889 - val_acc: 0.9700\n",
            "Epoch 28/30\n",
            "71s - loss: 0.0296 - acc: 0.9915 - val_loss: 0.1406 - val_acc: 0.9620\n",
            "Epoch 29/30\n",
            "71s - loss: 0.0432 - acc: 0.9890 - val_loss: 0.1535 - val_acc: 0.9650\n",
            "Epoch 30/30\n",
            "71s - loss: 0.0354 - acc: 0.9885 - val_loss: 0.1832 - val_acc: 0.9510\n"
          ]
        }
      ],
      "source": [
        "from keras.preprocessing.image import ImageDataGenerator\n",
        "\n",
        "train_datagen = ImageDataGenerator(\n",
        "      rescale=1./255,\n",
        "      rotation_range=40,\n",
        "      width_shift_range=0.2,\n",
        "      height_shift_range=0.2,\n",
        "      shear_range=0.2,\n",
        "      zoom_range=0.2,\n",
        "      horizontal_flip=True,\n",
        "      fill_mode='nearest')\n",
        "\n",
        "# Note that the validation data should not be augmented!\n",
        "test_datagen = ImageDataGenerator(rescale=1./255)\n",
        "\n",
        "train_generator = train_datagen.flow_from_directory(\n",
        "        # This is the target directory\n",
        "        train_dir,\n",
        "        # All images will be resized to 150x150\n",
        "        target_size=(150, 150),\n",
        "        batch_size=20,\n",
        "        # Since we use binary_crossentropy loss, we need binary labels\n",
        "        class_mode='binary')\n",
        "\n",
        "validation_generator = test_datagen.flow_from_directory(\n",
        "        validation_dir,\n",
        "        target_size=(150, 150),\n",
        "        batch_size=20,\n",
        "        class_mode='binary')\n",
        "\n",
        "model.compile(loss='binary_crossentropy',\n",
        "              optimizer=optimizers.RMSprop(learning_rate=2e-5),\n",
        "              metrics=['acc'])\n",
        "\n",
        "history = model.fit(\n",
        "      train_generator,\n",
        "      epochs=30,\n",
        "      validation_data=validation_generator,\n",
        "      verbose=2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "ORN3BHzQy70P"
      },
      "outputs": [],
      "source": [
        "model.save('cats_and_dogs_small_3.h5')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pxZdq3tJy70P"
      },
      "source": [
        "Let's plot our results again:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3Be7j8xPy70Q",
        "outputId": "27143205-9364-4bef-c097-517215fa097d"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4FdX5wPHvCwIh7CQIArKoVERljaAVBFzRirjgAnGv\nxVpRsdr+VKhQFbu41KVWxYpLjSBqqdAKFAMWlaqEQkBABBUwbIZFZFMIeX9/nElyE+9N5ib35m7v\n53nuk3tnzsw9M3PzzplzzpwRVcUYY0xqqBPrDBhjjKk9FvSNMSaFWNA3xpgUYkHfGGNSiAV9Y4xJ\nIRb0jTEmhVjQT0EiUldE9ohIh0imjSUROUZEIt7/WETOFJF1AZ9Xi8gAP2mr8V1/FZF7qru8MX4c\nFusMmKqJyJ6Aj+nA98Ah7/ONqpoTzvpU9RDQONJpU4GqHhuJ9YjIDcCVqjooYN03RGLdxlTGgn4C\nUNXSoOuVJG9Q1XdCpReRw1S1qDbyZkxV7PcYX6x6JwmIyAMi8pqITBGR3cCVInKKiHwoIt+IyGYR\neUJE6nnpDxMRFZFO3udXvPmzRGS3iPxXRDqHm9abf66IfCYiu0TkSRH5QESuDZFvP3m8UUTWishO\nEXkiYNm6IvInEdkuIl8AQyrZP2NFZGqFaU+JyKPe+xtEZJW3PZ97pfBQ6yoQkUHe+3QR+ZuXtxVA\nnwppx4nIF956V4jIBd70E4E/AwO8qrNtAft2QsDyP/e2fbuI/ENEjvCzb8LZzyX5EZF3RGSHiGwR\nkV8HfM9vvH3yrYjkiUjbYFVpIvJ+yXH29ucC73t2AONEpIuIzPe+Y5u335oFLN/R28ZCb/7jIpLm\n5fm4gHRHiMg+EckItb2mCqpqrwR6AeuAMytMewA4AAzFncgbAicB/XBXc0cBnwGjvfSHAQp08j6/\nAmwDsoB6wGvAK9VIeziwGxjmzfslcBC4NsS2+MnjW0AzoBOwo2TbgdHACqA9kAEscD/noN9zFLAH\naBSw7q+BLO/zUC+NAKcD+4Hu3rwzgXUB6yoABnnvHwbeBVoAHYGVFdJeBhzhHZORXh5ae/NuAN6t\nkM9XgAne+7O9PPYE0oC/APP87Jsw93MzYCtwG9AAaAr09ebdDeQDXbxt6Am0BI6puK+B90uOs7dt\nRcBNQF3c7/FHwBlAfe938gHwcMD2fOLtz0Ze+lO9eZOAiQHfcwcwPdb/h4n8inkG7BXmAQsd9OdV\nsdydwOve+2CB/JmAtBcAn1Qj7fXAewHzBNhMiKDvM48nB8z/O3Cn934BrpqrZN55FQNRhXV/CIz0\n3p8LrK4k7T+Bm733lQX9DYHHAvhFYNog6/0E+In3vqqg/xLwYMC8prh2nPZV7Zsw9/NVwKIQ6T4v\nyW+F6X6C/hdV5GF4yfcCA4AtQN0g6U4FvgTE+7wUuDjS/1ep9LLqneTxVeAHEekqIv/yLte/Be4D\nMitZfkvA+31U3ngbKm3bwHyo+y8tCLUSn3n09V3A+kryC/AqMMJ7P9L7XJKP80XkI6/q4RtcKbuy\nfVXiiMryICLXiki+V0XxDdDV53rBbV/p+lT1W2An0C4gja9jVsV+PhIX3IOpbF5VKv4e24jINBHZ\n6OXhxQp5WKeu00A5qvoB7qqhv4icAHQA/lXNPBmsTj+ZVOyu+CyuZHmMqjYF7sWVvKNpM64kCoCI\nCOWDVEU1yeNmXLAoUVWX0mnAmSLSDlf99KqXx4bAG8DvcFUvzYF/+8zHllB5EJGjgKdxVRwZ3no/\nDVhvVd1LN+GqjErW1wRXjbTRR74qqmw/fwUcHWK5UPP2enlKD5jWpkKaitv3B1yvsxO9PFxbIQ8d\nRaRuiHy8DFyJuyqZpqrfh0hnfLCgn7yaALuAvV5D2I218J3/BHqLyFAROQxXT9wqSnmcBowRkXZe\no97/VZZYVbfgqiBexFXtrPFmNcDVMxcCh0TkfFzds9883CMizcXdxzA6YF5jXOArxJ3/foYr6ZfY\nCrQPbFCtYArwUxHpLiINcCel91Q15JVTJSrbzzOADiIyWkQaiEhTEenrzfsr8ICIHC1OTxFpiTvZ\nbcF1GKgrIqMIOEFVkoe9wC4RORJXxVTiv8B24EFxjeMNReTUgPl/w1UHjcSdAEwNWNBPXncA1+Aa\nVp/FNbhGlapuBS4HHsX9Ex8NLMGV8CKdx6eBXGA5sAhXWq/Kq7g6+tKqHVX9BrgdmI5rDB2OO3n5\nMR53xbEOmEVAQFLVZcCTwMdemmOBjwKWnQusAbaKSGA1Tcnys3HVMNO95TsA2T7zVVHI/ayqu4Cz\ngEtwJ6LPgIHe7IeAf+D287e4RtU0r9ruZ8A9uEb9YypsWzDjgb64k88M4M2APBQB5wPH4Ur9G3DH\noWT+Otxx/l5VF4a57aaCksYRYyLOu1zfBAxX1fdinR+TuETkZVzj8IRY5yXR2c1ZJqJEZAiup8x+\nXJe/g7jSrjHV4rWPDANOjHVekoFV75hI6w98gavLPge4yBreTHWJyO9w9wo8qKobYp2fZGDVO8YY\nk0KqLOmLyGQR+VpEPgkxX7zbrdeKyDIR6R0w7xoRWeO9rolkxo0xxoSvypK+iJyGu338ZVU9Icj8\n84BbcHdE9gMeV9V+XteuPNzt+gosBvqo6s7Kvi8zM1M7depUjU0xxpjUtXjx4m2qWlkXacBHQ66q\nLhBvsK0QhuFOCAp86PVZPgIYBMxV1R0AIjIXNyjWlMq+r1OnTuTl5VWVLWOMMQFEpKq70oHINOS2\no/wt1wXetFDTf0BERnkj+OUVFhZGIEvGGGOCiYveO6o6SVWzVDWrVasqr06MMcZUUySC/kbKjz/S\n3psWaroxxpgYiUTQnwFc7fXiORnYpaqbgTnA2SLSQkRa4EYunBOB7zPGGFNNVTbkisgUXKNspogU\n4MbQqAegqs8Ab+N67qzFDe96nTdvh4jcjxsXBeC+kkZdY4wxseGn986IKuYrcHOIeZOBydXLmjHG\nxJ+cHBg7FjZsgA4dYOJEyK7uUHgxEBcNucaY5JWTA506QZ067m9OTqxzVH05OTBqFKxfD6ru76hR\nibVNFvSNMVGTDEEy0NixsG9f+Wn79rnpicKCvjEmapIhSAbaEGLIt1DT/V7l1ObVkA2tbIyJmnCD\nZLzr0MFdrQSbXlHJVU7JSa/kKgfKtwH4TRcpVtI3xkRNsGAYanoi1P1PnAjp6eWnpae76RX5vcqp\n7ashC/rGmKjxGyQTpe4/OxsmTYKOHUHE/Z00KXiJ3O9VTm1fDVnQN8ZEjd8gGW5pN5ZXBdnZsG4d\nFBe7v6GqYPxe5YRzNRQJFvSNMWELJ+j6CZLhlHajdVUQ6ROJ36uccKqMIkJV4+rVp08fNcbExiuv\nqHbsqCri/r7ySvA06emqLuS6V3p68LR+dexYfn0lr44da5bWz/ZEa5vC/X4/6SoD5KmPGBvzIF/x\nZUHfmNjwG/jCCbqR/m5VFxiDfb9I9dcZjW2qbX6DvlXvGJOgIl0d4bdePRoNj+E0kPqtAw+nnSDZ\nupZWxoK+MXEknJt5Il2v7TfwRavh0W8Dqd868HACeW03psaSBX1jqinSJe1wAnk4pVi/+fQb+Gq9\n4bECv1cF4QTyWG9TrfJTB1SbL6vTN7G2fr3qpZeqzp8fOk24DX9+GurCqVeORr12uGlr2vAYbdE4\nRvEMa8g1qSDS/6hffqnaqZP7z6hfX/W114KnC7cHiZ/g4zeQh/P94TZQJnrgqyjZtqcyFvRN0ot0\nN7vPP1ft0EG1RQvVuXNVTz3VBYvHHvth2lgH6GicSExi8xv0rU7fJKxIjlmydi0MHAh798Kdd8IN\nN8AHH0BaGowZA7/+tWtgLBFOfbHfBsVw6pWjUa8da0VFcPXVcPrpsDFCT9PeuRMuvBD69oU5c9wp\nL+X5OTPU5stK+mWKi1WLimKdi/gVqVLs6tWqbduqZmaqPvjgD0vQhx3m/mZnq37/vVsmWn3AI10d\nEa2bjiKtuFj1uutc/ho0UG3TRvWDD2q2zk8+UT3mGNV69VSPPNKte/Bg1Y8/jkye4w1WvZO4iotV\nZ85UPfFE1R/9SHXPnljnKD5FIpiuXOkCzOGHqy5fHnqdzZu7v2eeqfrtt5WvM9h3+w28RUWqW7eq\nfvddDXeOj22PF8XFqr/8pdsv48e7YH300S5YT5pUvXVOn67auLFq69aq77/v9ufjj7sTO6gOH+5O\n9snEgn6CWrhQdcCAsuAFqr/6VaxzFZ/8BtNQ6X7/exfs27RxwV+18quHF15QrVtXtVcv1c2bw8vr\nyy+rHnGEW1dGhuoVV7hAl53tTiTdu7sAVaeOS9Ohg+q6dRHZTXHvgQfcNt96qzsBqKru2KF6zjlu\n+s9/XnaFVZVDh1Tvvdct17evakFB+fm7drn5jRq5Y3njjaqbNkV2e2LFgn6CWblS9cIL3RFp3Vr1\nL39RPXBA9ac/dT/O/PxY5zC0b79V3bgxNt9dk66Qdeq4QPzpp1WnLbl6ePttd8Lo3LnqkuKXX7qS\n6qWXukBfcZ0l6zn5ZNVhw1RHjVIdN0714YfdlUWXLq7Un8z+/Ge3L66+2gXsQEVFqv/3f25+//6q\nW7ZUvq5du1QvuMClv/Za1f37Q6fdskV19GhXdZeernrPParffFPz7YklC/oJ4quvXGCvU0e1SRPV\n++9X3b27bP62be6S9OSTf/hPES+uvtpdij/8cHzmMVTpHVQ/+6x8Wj9XDx9/rNqqlTsuH35YNn3H\nDtU333Ql06OPLlu+bVvVa65xpf0PPlBdu7b8MQ7mgw9UGzZU7dlTdefOiO2KuPLKK27/XHCB6sGD\nodNNmeL2Rbt2oevjV69W7drVFZCeeKLsiqEqa9eqjhjh8tGypeojj7jCViKyoB/ntm931TZpaa4/\n+JgxqoWFwdO+9JI7Us8+W7t59KOoyHVxbNrU5fGss+LvcjlU6b1t2+Dp/Vw9rFmjetRR7oRw662u\nKqGkaqZxY9Xzz3d1yCtX+g9AFc2e7U6m/fur7t1bvXXEq5kzXYAePLjyEnmJJUvcsWjQwP0/BPrn\nP93vLzOz8hvqKrN4serZZ7vjd9JJ7vgmGgv6caq4uOzyXUT1qqtcNUBVywwe7Jap6hK3tv33v+5X\nNGWK6jPPuBJZZqbqjBmxzlmZYKX3tLSaN2hu2eICRN26qqec4uqK33svsiXFadPc7+Tcc/3Xa8e7\nd991+z8rq6xR3I/CQvd/AKq33eb288SJbv/06hWZNpA33nCFmEaNVCdPrv4JOxYs6Mep8ePdXj/3\n3PDq6T/91F0RZGdHLWvVMmGC+6fbts19XrnSVUmA6i9+obpvX1naWPYi+dWvyqp52rWL3HcXF0e/\nFD5pksv35ZcnfhfevDxXjXnccaGvbCtz8KC7KoayhvERIyJ7DDZsUB00yK37sstctV0isKAfh0qq\naa67rnoliJJeCXPnRj5v1XXyyar9+pWf9t13ZV3wunVzJ7dYjuvy4ouuwa5nz/irevLrj3/U0p4s\n4f52vvvO/fZefz06efNr1Sp3Fdixo2vLqomXXnI9rx56KDql8aIi1d/9zv1ujjxS9T//ifx3RJoF\n/Tgzf76rnz399Opfpu/f72426dLFXz1otG3f7uqx7703+Pw5c1x3yPr13SVzVX3qDxxw3Sjr1/d3\ncqhKcbG72QpUzzjD9e5IZHfd5bbl7rv9pd+2zXUMaN1aS3sr1fSGp+pav161fXuXl0SqL//4Y/c/\nV6eO61kVTtVdQYErcIwZE34X3+qwoB9HVq1y9fHHHVfznhhz57qjFirQ1qbXXnN5adMmdKn8669V\nhw4NHvBLXldf7UrhFYN9VTdcVaaoSPXmm92yI0cmR314cbHrVw6u5B/K6tXuiqBhQ5f2nHNU33rL\nDSR31FHh1aPXNL8rV7reNEcf7f4Hli6tne+OpN27y+4W7tfP9fgJ5ttvXVvWrbe6//XA3++Pfxz9\n36AF/Tjx9dfuH+3ww1W/+CIy6xw50gXIwP7lsTBw4A+Dc7BSeXGx6w4XKqC3bas6ZIjqr39d+cnB\nb5/1/ftVL7nELXPHHfHZjbS6iopc3T6oPvdc2fTiYtdAOnSoOwHXr696/fXuLuMSCxa4eTfcEL38\nbdmimpPj+sm3b1927Lp0id1VRqS89ppqs2aud9bLL7v2hQ8+cO1a/fuXDdfRsKE70T70kDvJTZ2q\npVVz0WRBPw7s3+96daSlle/PXVNbtrhS06BBsetdUFzseq34LZW/8orbD4HpGjRQffrp8ulCda8E\nV0X0179WHsR37Ci7o/nRRyO5xfHj++9dRwAR1VdfdUG2Tx+3zRkZqr/5TeheXiVVRG+9FZm87N2r\nOmuWO7l2717+WA0f7roZR6qwEw/Wry/7fZVcSYm4nkh3362amxu86rXkJrPAE3WkWdCPsUOHykpk\nb7wR+fU/84xbd8U+y7Vl+fLQwTnUgGd+GmhDNfj+4Q+qp53mPg8YUDZsQqANG1SPP961nUyZEsmt\njT9797qhn0v20bHHut9EYG+pYL7/3nVvbNWqZt1/Dx1y3SZLquTq13ftVQ8+qLpoUeL3MqpMUZHq\nn/7kSu7TppX1XKtqmbPPdvspkgXAQBb0Y+yee7TKuteaOHTIXUVkZvr70alGtlfMQw+FDvrh1r/7\nzeehQ6rPP+9KkfXquRJtSalq+XLXFbNJE1faSgU7d7peUjNnhleFtWKFu+r6yU+qd6VYVOSqb8D9\nnT07+W4ei4bt292wG23bRqdh14J+DD3/vNuzo0ZFt/olP99Vsfz0p1WnjfQQu2ee6epsYzFs79at\nqldeqaV1xU884aq7jjgiMRsKY+Hxx93+e+aZ8JYrKnI3FIKryzbhyc93/yP9+0e+YdeCfozMnesa\ndM4+u3bG8PjVr9xRXLCg8nThPjavMnv2uMvUO+6I7Q1Xc+eWjXFz7LGpMyplJBw65E7c6ek/HH8o\nlIMHy8apuf/+6OYvmU2Z4vbhL34R2fVa0K+h4mJXbfLpp24EST8B/JNP3BggJ5xQeyP27dnjgm23\nbpWXHCL52Lx//cst++9/VzvbEbNvn+tJsX17rHOSeAoKXFVZv36VD3im6n7/l17qjvvvflc7+Utm\nJYW155+P3Dr9Bv3DavtJXfFg71748EPYsgW2bg3++vpr9/i2QBkZ0Lp12evww8veZ2S4x+qlp8O/\n/gXNmtXOtjRqBH/+Mwwd6h7p99hjwdN16ADr1wefHq7Zs6FhQxgwIPxlI61hQ7jqqljnIjG1awfP\nPAOXX+4eyTh+fPB0Bw7AiBHw97/DQw+5x0mamnnwQViyBG66CU44wT3OsbaIO0HEj6ysLM3Ly4vq\nd9x0k/uxl6hfv3wwDwzqmZmwZ0/ok8Pu3WXrSU+HBQugT5+oZj+oMWPg8cfhySdh9Ogfzs/JgVGj\nyj9TNj09+HNVq3LssXD00fD22zXLs4kPV10FU6a4ZwL361d+3oEDcNll8NZb8Kc/ud+ZiYzt2yEr\nCw4ehMWLXcypCRFZrKpZVSb0czlQm6/aqN7p3NnVZ376qesBUZPG1n373CiZH37ougzGSlGRuzGn\nTh031Gwwkah//+ILd1n6+OM1ya2JJ998457U1aVL+UdzfvedGyIaVJ98Mnb5S2ZLlrj+/gMG1LwN\nkEjW6QNDgNXAWuCuIPM7ArnAMuBdoH3AvEPAUu81o6rvinbQLwlaTzwR1a+Jid27VXv3dsPCLllS\ns3WFOkE8/bTbf8n2fNFU9+677liX3DW6f7+7AQzcU9xM9Lz6qtvPt9xSs/VELOgDdYHPgaOA+kA+\n0K1CmteBa7z3pwN/C5i3x09GSl7RDvrPPee2esWKqH5NzGzc6EYFbNu2+iMZVta9c9gwN4ZLIo0z\nbvwpaVx8/fWyB4pU98HkJjx33OH29wsvVH8dkQz6pwBzAj7fDdxdIc0K4EjvvQDfBsyLq6A/YoQb\nICyZg1Z+vrtJqWfP6g2uFap7Z4cObr3RHkPExMZ335UNpSAS2Z4lpnIHD7o7mvv2rf5YUX6Dfh0f\n7QPtgK8CPhd40wLlAxd77y8CmohIhvc5TUTyRORDEbkw2BeIyCgvTV5hYaGPLFWPKsybB6efDiJR\n+5qY694dXn8dli+HK674YS+kqmzYEHr67t1wzjk1z6OJPw0auAb/Y4+FF1+E66+PdY5Sx2GHuf/Z\nefOgjp+oXAORWv2dwEARWQIMBDbi6vIBOqprUR4JPCYiR1dcWFUnqWqWqma1atUqQln6oZUrXY+b\nM86I2lfERE4OdOrkfiydOrnP55wDTz3letiMGeNOeH6F6sbZtKn7cZ5+eiRybeLRCSfAp5/C1VfH\nOiepp2VL1wU72vz0098IHBnwub03rZSqbsIr6YtIY+ASVf3Gm7fR+/uFiLwL9MK1EdS63Fz3N5mC\nVsWumOvXu88AN94Ia9fCww/DMcf47243cWLw7p0tWkDPni74G2MSk5+S/iKgi4h0FpH6wBXAjMAE\nIpIpIiXruhuY7E1vISINStIApwIrI5X5cM2bB0cd5UrDyWLs2PLBGdznsWPd+z/8AS6+GH75S9fX\n2o/sbNd/v2NHVw3WsaO7KWf9ehgyJLL5N8bUriqDvqoWAaOBOcAqYJqqrhCR+0TkAi/ZIGC1iHwG\ntAYmetOPA/JEJB+YD/xeVWMS9IuK4N13k6uUD5XXv4Or8vnb3+Ckk2DkSPB731t2NqxbB8XF7m+T\nJm661ecbk9h8DcOgqm8Db1eYdm/A+zeAN4IstxA4sYZ5jIglS2DXruSrz/czvEJ6OsyY4e62HDoU\nPvoo/OEX5sxxdyj37Fmz/BpjYivK7cTxo6Q+f/Dg2ObDr2CNs8FMnOiCeqD0dDc9UOvWrlF3/344\n7zzYscN/XoqLXdA/++zo9ywwxkRXyvwLz5vneibUdHyL2lDSOLt+vet1U9I4GyzwB6t/DzWeTrdu\nMH06rFnjAv+ePf7ys2QJbNtm9fnGJIOUCPrffw/vv5849flVNc5WVLH+vbIB1AYPhtdeg0WL4KKL\n3L6pyuzZ7u9ZZ/nJvTEmnqVE0P/wQ1etkSj1+VU1ztbUhRfC5Mnwzjuucbeqm7fmzHEjhx5+eGS+\n3xgTOykR9HNzXV30aafFOif+hGpkrc7Y96Fcc40be//vf3dVR6Fu3tq1CxYutF47xiSLlAn6WVnQ\nvHmsc+KP38bZmrrtNrj3XnjhBfdgjGCBf948OHTI6vONSRZJH/R374aPP06cqh0Ir3G2piZMgFtu\ngUcfdU/zqWj2bNdH/+STI//dxpjal/RB/733XJ11vDTi+u2KGU7jbE2IuGqeq66CcePgL38pm6fq\n6vPPPBPq1YvO9xtjalfSPyN33jz3OMRTT411TiofJydaQd2POnXg+edd/f3o0a4abORIWL3a5fHu\nu2OXN2NMZCV9ST83F378Y/cA7VgLtytmbapXz3XlHDjQjbD4z3+6Uj5YI64xySSpg/727bB0afzU\n50e7K2ZNpaW54Rp69YJLL4Wnn4auXZNrgDpjUl1SB/35893feKnPr42umDXVpAnMmuUC/erVVso3\nJtkkddCfNw8aN3YjTEZTpMfJibXMTJg7F4YPd2PyG2OSR1I35ObmuhuyotnzJJzG2ZLPY8e6Kp0O\nHVzAj2Ujbijt27vHtxljkotoOM/RqwVZWVma53fQ90oUFMCRR8Ijj7gHiERLp07Bhzbu2NF1tTTG\nmNogIou9R9NWKmmrd+bNc3+jXZ8f742zxhgTKKmDfkYGdO8e3e9JhMZZY4wpkZRBX9XV5w8eHP2H\nfiRK46wxxkCSBv21a12dfm30z6/NcXKMMaamkrL3TsmjEWvrpqzsbAvyxpjEkJQl/dxc1+XwmGNi\nnRNjjIkvSRf0i4vdnbhnnOGqW4wxxpRJuqC/bJkbcydehl4wxph4knRBv7b65xtjTCJKuqCfmws/\n+pGr0zfGGFNeUgX9gwdhwYLI9drxO5CaMcYkiqTqsrloEezZE5mqnXh9ypUxxtREUpX0S+rzBw+u\n+bri+SlXxhhTXUkV9HNzoWdPN+ZOTdlAasaYZJQ0QX//fli4MHL1+TaQmjEmGSVN0N+1C664As4/\nPzLrs4HUjDHJKGkactu0gZdeitz6EukpV8YY41fSBP1osIHUjDHJJmmqd4wxxlTNgr4xxqQQC/rG\nGJNCLOgbY0wKsaBvjDEpxFfQF5EhIrJaRNaKyF1B5ncUkVwRWSYi74pI+4B514jIGu91TSQzb4wx\nJjxVBn0RqQs8BZwLdANGiEi3CskeBl5W1e7AfcDvvGVbAuOBfkBfYLyItIhc9o0xxoTDT0m/L7BW\nVb9Q1QPAVGBYhTTdAG+4M+YHzD8HmKuqO1R1JzAXGFLzbBtjjKkOP0G/HfBVwOcCb1qgfOBi7/1F\nQBMRyfC5LCIySkTyRCSvsLDQb96NMcaEKVINuXcCA0VkCTAQ2Agc8ruwqk5S1SxVzWrVqlWEsmSM\nMaYiP8MwbASODPjc3ptWSlU34ZX0RaQxcImqfiMiG4FBFZZ9twb5NcYYUwN+SvqLgC4i0llE6gNX\nADMCE4hIpoiUrOtuYLL3fg5wtoi08Bpwz/amGWOMiYEqg76qFgGjccF6FTBNVVeIyH0icoGXbBCw\nWkQ+A1oDE71ldwD3404ci4D7vGnGGGNiQFQ11nkoJysrS/Py8mKdDWOMSSgislhVs6pKl3J35Obk\nQKdOUKeO+5uTE+scGWNM7Ump8fRzcmDUqLIHnq9f7z6DjZtvjEkNKVXSHzu2LOCX2LfPTTfGmFSQ\nUkF/w4bwphtjTLJJqaDfoUN4040xJtmkVNCfOBHS08tPS093040xJhWkVNDPzoZJk6BjRxBxfydN\nskZcY0zqSKneO+ACvAV5Y0yqSqmSvjHGpDoL+sYYk0Is6BtjTAqxoG+MMSnEgr4xxqQQC/rGGJNC\nLOgbY0wKsaBvjDEpxIK+McakEAv6xhiTQizoG2NMCrGgb4wxKcSCvjHGpBAL+sYYk0Is6BtjTAqx\noG+MMSnEgr4xxqQQC/rGGJNCLOgbY0wKsaBvjDEpxIK+McakEAv6xhiTQizoG2NMCrGgb4wxKcSC\nvjHGpBBbJzS3AAARfUlEQVQL+sYYk0Is6BtjTAqxoG+MMSnEgr4xxqQQC/rGGJNCfAV9ERkiIqtF\nZK2I3BVkfgcRmS8iS0RkmYic503vJCL7RWSp93om0htgjDHGv8OqSiAidYGngLOAAmCRiMxQ1ZUB\nycYB01T1aRHpBrwNdPLmfa6qPSObbWOMMdXhp6TfF1irql+o6gFgKjCsQhoFmnrvmwGbIpdFY4wx\nkeIn6LcDvgr4XOBNCzQBuFJECnCl/FsC5nX2qn3+IyIDgn2BiIwSkTwRySssLPSfe2OMMWGJVEPu\nCOBFVW0PnAf8TUTqAJuBDqraC/gl8KqINK24sKpOUtUsVc1q1apVhLJkjDGmIj9BfyNwZMDn9t60\nQD8FpgGo6n+BNCBTVb9X1e3e9MXA58CPapppY4wx1eMn6C8CuohIZxGpD1wBzKiQZgNwBoCIHIcL\n+oUi0sprCEZEjgK6AF9EKvPGGGPCU2XvHVUtEpHRwBygLjBZVVeIyH1AnqrOAO4AnhOR23GNuteq\nqorIacB9InIQKAZ+rqo7orY1xhhjKiWqGus8lJOVlaV5eXmxzoYxxiQUEVmsqllVpbM7co0xJoVY\n0DfGmBRiQd8YY1KIBX1jjEkhFvSNMSaFWNA3xpgUYkHfGGNSiAV9Y4xJIRb0jTEmhVjQN8aYFGJB\n3xhjUogFfWOMSSEW9I0xJoVY0DfGmBRiQd8YY1KIBX1jjEkhFvSNMSaFWNA3xpgUYkHfGGNSiAV9\nY4xJIRb0jTEmhVjQN8aYFGJB3xhjUogFfWOMSSEW9I0xJoVY0DfGmBRiQd8YY1KIBX1jjEkhFvSN\nMSaFWNA3xpgUYkHfGGNSyGGxzoAxJn4cPHiQgoICvvvuu1hnxYSQlpZG+/btqVevXrWWt6BvjClV\nUFBAkyZN6NSpEyIS6+yYClSV7du3U1BQQOfOnau1DqveMcaU+u6778jIyLCAH6dEhIyMjBpdiVnQ\nN8aUYwE/vtX0+FjQN8aYFGJB3xhTbTk50KkT1Knj/ubk1Gx927dvp2fPnvTs2ZM2bdrQrl270s8H\nDhzwtY7rrruO1atXV5rmqaeeIqemmU1Q1pBrjKmWnBwYNQr27XOf1693nwGys6u3zoyMDJYuXQrA\nhAkTaNy4MXfeeWe5NKqKqlKnTvAy6wsvvFDl99x8883Vy2AS8FXSF5EhIrJaRNaKyF1B5ncQkfki\nskRElonIeQHz7vaWWy0i50Qy88aY2Bk7tizgl9i3z02PtLVr19KtWzeys7M5/vjj2bx5M6NGjSIr\nK4vjjz+e++67rzRt//79Wbp0KUVFRTRv3py77rqLHj16cMopp/D1118DMG7cOB577LHS9HfddRd9\n+/bl2GOPZeHChQDs3buXSy65hG7dujF8+HCysrJKT0iBxo8fz0knncQJJ5zAz3/+c1QVgM8++4zT\nTz+dHj160Lt3b9atWwfAgw8+yIknnkiPHj0YG42dVYUqg76I1AWeAs4FugEjRKRbhWTjgGmq2gu4\nAviLt2w37/PxwBDgL976jDEJbsOG8KbX1Keffsrtt9/OypUradeuHb///e/Jy8sjPz+fuXPnsnLl\nyh8ss2vXLgYOHEh+fj6nnHIKkydPDrpuVeXjjz/moYceKj2BPPnkk7Rp04aVK1fym9/8hiVLlgRd\n9rbbbmPRokUsX76cXbt2MXv2bABGjBjB7bffTn5+PgsXLuTwww9n5syZzJo1i48//pj8/HzuuOOO\nCO0d//yU9PsCa1X1C1U9AEwFhlVIo0BT730zYJP3fhgwVVW/V9UvgbXe+owxCa5Dh/Cm19TRRx9N\nVlZW6ecpU6bQu3dvevfuzapVq4IG/YYNG3LuuecC0KdPn9LSdkUXX3zxD9K8//77XHHFFQD06NGD\n448/Puiyubm59O3blx49evCf//yHFStWsHPnTrZt28bQoUMBd0NVeno677zzDtdffz0NGzYEoGXL\nluHviBryE/TbAV8FfC7wpgWaAFwpIgXA28AtYSyLiIwSkTwRySssLPSZdWNMLE2cCOnp5aelp7vp\n0dCoUaPS92vWrOHxxx9n3rx5LFu2jCFDhgTtu16/fv3S93Xr1qWoqCjouhs0aFBlmmD27dvH6NGj\nmT59OsuWLeP666+P+7uZI9V7ZwTwoqq2B84D/iYivtetqpNUNUtVs1q1ahWhLBljoik7GyZNgo4d\nQcT9nTSp+o244fj2229p0qQJTZs2ZfPmzcyZMyfi33Hqqacybdo0AJYvXx70SmL//v3UqVOHzMxM\ndu/ezZtvvglAixYtaNWqFTNnzgTcTW/79u3jrLPOYvLkyezfvx+AHTt2RDzfVfHTe2cjcGTA5/be\ntEA/xdXZo6r/FZE0INPnssaYBJWdXTtBvqLevXvTrVs3unbtSseOHTn11FMj/h233HILV199Nd26\ndSt9NWvWrFyajIwMrrnmGrp168YRRxxBv379Sufl5ORw4403MnbsWOrXr8+bb77J+eefT35+PllZ\nWdSrV4+hQ4dy//33RzzvlZGSluaQCUQOAz4DzsAF7EXASFVdEZBmFvCaqr4oIscBubhqnG7Aq7h6\n/Lbe9C6qeijU92VlZWleXl6NNsoYUz2rVq3iuOOOi3U24kJRURFFRUWkpaWxZs0azj77bNasWcNh\nh8W+p3uw4yQii1U1K8QiparMvaoWichoYA5QF5isqitE5D4gT1VnAHcAz4nI7bhG3WvVnU1WiMg0\nYCVQBNxcWcA3xph4sWfPHs444wyKiopQVZ599tm4CPg15WsLVPVtXANt4LR7A96vBIJeX6nqRCBK\nTTvGGBMdzZs3Z/HixbHORsTZMAzGGJNCLOgbY0wKsaBvjDEpxIK+McakEAv6xpi4MXjw4B/caPXY\nY49x0003Vbpc48aNAdi0aRPDhw8PmmbQoEFU1R38scceY1/AKHLnnXce33zzjZ+sJwwL+saYuDFi\nxAimTp1abtrUqVMZMWKEr+Xbtm3LG2+8Ue3vrxj03377bZo3b17t9cWjxO90aoyJijFjIMhIwjXS\nsyd4IxoHNXz4cMaNG8eBAweoX78+69atY9OmTQwYMIA9e/YwbNgwdu7cycGDB3nggQcYNqz82I/r\n1q3j/PPP55NPPmH//v1cd9115Ofn07Vr19KhDwBuuukmFi1axP79+xk+fDi//e1veeKJJ9i0aROD\nBw8mMzOT+fPn06lTJ/Ly8sjMzOTRRx8tHaXzhhtuYMyYMaxbt45zzz2X/v37s3DhQtq1a8dbb71V\nOqBaiZkzZ/LAAw9w4MABMjIyyMnJoXXr1uzZs4dbbrmFvLw8RITx48dzySWXMHv2bO655x4OHTpE\nZmYmubm5ETsGFvSNMXGjZcuW9O3bl1mzZjFs2DCmTp3KZZddhoiQlpbG9OnTadq0Kdu2bePkk0/m\nggsuCPnM2Keffpr09HRWrVrFsmXL6N27d+m8iRMn0rJlSw4dOsQZZ5zBsmXLuPXWW3n00UeZP38+\nmZmZ5da1ePFiXnjhBT766CNUlX79+jFw4EBatGjBmjVrmDJlCs899xyXXXYZb775JldeeWW55fv3\n78+HH36IiPDXv/6VP/7xjzzyyCPcf//9NGvWjOXLlwOwc+dOCgsL+dnPfsaCBQvo3LlzxMfnsaBv\njAmqshJ5NJVU8ZQE/eeffx5wY97fc889LFiwgDp16rBx40a2bt1KmzZtgq5nwYIF3HrrrQB0796d\n7t27l86bNm0akyZNoqioiM2bN7Ny5cpy8yt6//33ueiii0pH+rz44ot57733uOCCC+jcuTM9e/YE\nQg/fXFBQwOWXX87mzZs5cOAAnTt3BuCdd94pV53VokULZs6cyWmnnVaaJtLDLydNnX6kn9VpjImN\nYcOGkZuby//+9z/27dtHnz59ADeAWWFhIYsXL2bp0qW0bt26WsMYf/nllzz88MPk5uaybNkyfvKT\nn9RoOOSSYZkh9NDMt9xyC6NHj2b58uU8++yzMR1+OSmCfsmzOtevB9WyZ3Va4Dcm8TRu3JjBgwdz\n/fXXl2vA3bVrF4cffjj16tVj/vz5rF+/vtL1nHbaabz66qsAfPLJJyxbtgxwwzI3atSIZs2asXXr\nVmbNmlW6TJMmTdi9e/cP1jVgwAD+8Y9/sG/fPvbu3cv06dMZMGCA723atWsX7dq5R4m89NJLpdPP\nOussnnrqqdLPO3fu5OSTT2bBggV8+eWXQOSHX06KoF+bz+o0xkTfiBEjyM/PLxf0s7OzycvL48QT\nT+Tll1+ma9eula7jpptuYs+ePRx33HHce++9pVcMPXr0oFevXnTt2pWRI0eWG5Z51KhRDBkyhMGD\nB5dbV+/evbn22mvp27cv/fr144YbbqBXr16+t2fChAlceuml9OnTp1x7wbhx49i5cycnnHACPXr0\nYP78+bRq1YpJkyZx8cUX06NHDy6//HLf3+NHlUMr17bqDK1cp44r4VckAsXFEcqYMSnAhlZODDUZ\nWjkpSvq1/axOY4xJVEkR9Gv7WZ3GGJOokiLox/JZncYkm3ir8jXl1fT4JE0//Vg9q9OYZJKWlsb2\n7dvJyMgIedOTiR1VZfv27aSlpVV7HUkT9I0xNde+fXsKCgooLCyMdVZMCGlpabRv377ay1vQN8aU\nqlevXumdoCY5JUWdvjHGGH8s6BtjTAqxoG+MMSkk7u7IFZFCoPJBNSqXCWyLUHbiQbJtDyTfNiXb\n9kDybVOybQ/8cJs6qmqrqhaKu6BfUyKS5+dW5ESRbNsDybdNybY9kHzblGzbA9XfJqveMcaYFGJB\n3xhjUkgyBv1Jsc5AhCXb9kDybVOybQ8k3zYl2/ZANbcp6er0jTHGhJaMJX1jjDEhWNA3xpgUkjRB\nX0SGiMhqEVkrInfFOj+RICLrRGS5iCwVkfAeJxYHRGSyiHwtIp8ETGspInNFZI33t0Us8xiuENs0\nQUQ2esdpqYicF8s8hkNEjhSR+SKyUkRWiMht3vSEPE6VbE8iH6M0EflYRPK9bfqtN72ziHzkxbzX\nRKS+r/UlQ52+iNQFPgPOAgqARcAIVV0Z04zVkIisA7JUNSFvKhGR04A9wMuqeoI37Y/ADlX9vXdy\nbqGq/xfLfIYjxDZNAPao6sOxzFt1iMgRwBGq+j8RaQIsBi4EriUBj1Ml23MZiXuMBGikqntEpB7w\nPnAb8Evg76o6VUSeAfJV9emq1pcsJf2+wFpV/UJVDwBTgWExzlPKU9UFwI4Kk4cBL3nvX8L9QyaM\nENuUsFR1s6r+z3u/G1gFtCNBj1Ml25Ow1NnjfaznvRQ4HXjDm+77GCVL0G8HfBXwuYAEP9AeBf4t\nIotFZFSsMxMhrVV1s/d+C9A6lpmJoNEissyr/kmIqpCKRKQT0Av4iCQ4ThW2BxL4GIlIXRFZCnwN\nzAU+B75R1SIvie+YlyxBP1n1V9XewLnAzV7VQtJQV7eY+PWL8DRwNNAT2Aw8EtvshE9EGgNvAmNU\n9dvAeYl4nIJsT0IfI1U9pKo9gfa4mo2u1V1XsgT9jcCRAZ/be9MSmqpu9P5+DUzHHexEt9Wrdy2p\nf/06xvmpMVXd6v1TFgPPkWDHyasnfhPIUdW/e5MT9jgF255EP0YlVPUbYD5wCtBcREoehOU75iVL\n0F8EdPFas+sDVwAzYpynGhGRRl5DFCLSCDgb+KTypRLCDOAa7/01wFsxzEtElARHz0Uk0HHyGgmf\nB1ap6qMBsxLyOIXangQ/Rq1EpLn3viGuw8oqXPAf7iXzfYySovcOgNcF6zGgLjBZVSfGOEs1IiJH\n4Ur34B5r+WqibZOITAEG4YaA3QqMB/4BTAM64IbQvkxVE6ZhNMQ2DcJVGyiwDrgxoD48rolIf+A9\nYDlQ7E2+B1cPnnDHqZLtGUHiHqPuuIbauriC+jRVvc+LEVOBlsAS4EpV/b7K9SVL0DfGGFO1ZKne\nMcYY44MFfWOMSSEW9I0xJoVY0DfGmBRiQd8YY1KIBX1jjEkhFvSNMSaF/D8bKEGQJUzoLwAAAABJ\nRU5ErkJggg==\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7fd55306e400>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8FOX9wPHPFwg3clo8OD0qdzgiaBERQYpSpVi0HB5o\nbdRqsaW2UrAWsbRqqVIs9Se2Wi0BpGoUKkitogityCGCCAjFgAFEQEAwKCT5/v54dnOxSWaz9+T7\nfr3y2t3Z2Zlndjbfeeb7PPOMqCrGGGP8pUaiC2CMMSb6LLgbY4wPWXA3xhgfsuBujDE+ZMHdGGN8\nyIK7Mcb4kAV3E5KI1BSRoyLSJprzJpKInCMiUe/7KyKDRCSnxOstItLPy7xVWNdfRGRiVT9fwXJ/\nIyJ/i/ZyTeLUSnQBTHSIyNESL+sDXwMFgde3qmpWOMtT1QKgYbTnrQ5U9bxoLEdEbgGuU9VLSiz7\nlmgs2/ifBXefUNWi4BqoGd6iqv8ub34RqaWq+fEomzEm/iwtU00ETrufE5G5InIEuE5ELhSRd0Tk\nkIjsEZEZIpIWmL+WiKiItAu8nh14f7GIHBGR/4pI+3DnDbx/uYh8JCKHReQxEVkhImPLKbeXMt4q\nIttE5KCIzCjx2Zoi8qiIHBCR7cCQCr6fSSIyr8y0mSLySOD5LSKyKbA9/wvUqstbVq6IXBJ4Xl9E\n/h4o20agV5l57xWR7YHlbhSRqwLTuwJ/AvoFUl77S3y3k0t8/rbAth8QkZdE5HQv301lRGR4oDyH\nROQNETmvxHsTRWS3iHwhIptLbOsFIrI2MH2viPze6/pMDKiq/fnsD8gBBpWZ9hvgOHAl7qBeDzgf\n6IM7gzsL+Ai4MzB/LUCBdoHXs4H9QAaQBjwHzK7CvN8AjgDDAu+NB04AY8vZFi9lfBloDLQDPg9u\nO3AnsBFoBTQHlrmffMj1nAUcBRqUWPZnQEbg9ZWBeQS4FDgGdAu8NwjIKbGsXOCSwPNpwJtAU6At\n8GGZea8FTg/sk9GBMrQMvHcL8GaZcs4GJgeeDw6UsTtQF/gz8IaX7ybE9v8G+FvgecdAOS4N7KOJ\nwJbA887ADuC0wLztgbMCz1cBowLPGwF9Ev2/UJ3/rOZevSxX1YWqWqiqx1R1laquVNV8Vd0OzAL6\nV/D551V1taqeALJwQSXceb8DrFPVlwPvPYo7EITksYy/U9XDqpqDC6TBdV0LPKqquap6AHiwgvVs\nBz7AHXQALgMOqurqwPsLVXW7Om8ArwMhG03LuBb4jaoeVNUduNp4yfXOV9U9gX0yB3dgzvCwXIAx\nwF9UdZ2qfgVMAPqLSKsS85T33VRkJLBAVd8I7KMHcQeIPkA+7kDSOZDa+zjw3YE7SJ8rIs1V9Yiq\nrvS4HSYGLLhXL5+UfCEiHUTkFRH5VES+AKYALSr4/KclnudRcSNqefOeUbIcqqq4mm5IHsvoaV24\nGmdF5gCjAs9HB14Hy/EdEVkpIp+LyCFcrbmi7yro9IrKICJjReT9QPrjENDB43LBbV/R8lT1C+Ag\ncGaJecLZZ+UttxC3j85U1S3Az3D74bNAmu+0wKw3AZ2ALSLyrohc4XE7TAxYcK9eynYDfAJXWz1H\nVU8B7sOlHWJpDy5NAoCICKWDUVmRlHEP0LrE68q6as4HBonImbga/JxAGesBzwO/w6VMmgD/8liO\nT8srg4icBTwO3A40Dyx3c4nlVtZtczcu1RNcXiNc+meXh3KFs9wauH22C0BVZ6tqX1xKpibue0FV\nt6jqSFzq7Q/ACyJSN8KymCqy4F69NQIOA1+KSEfg1jis859ATxG5UkRqAXcBp8aojPOBn4jImSLS\nHLinoplV9VNgOfA3YIuqbg28VQeoDewDCkTkO8DAMMowUUSaiLsO4M4S7zXEBfB9uOPcD3E196C9\nQKtgA3IIc4EfiEg3EamDC7Jvq2q5Z0JhlPkqEbkksO6f49pJVopIRxEZEFjfscBfIW4DrheRFoGa\n/uHAthVGWBZTRRbcq7efATfi/nGfwDV8xpSq7gW+DzwCHADOBt7D9cuPdhkfx+XGN+Aa+5738Jk5\nuAbSopSMqh4Cfgpk4xolR+AOUl78GncGkQMsBp4tsdz1wGPAu4F5zgNK5qlfA7YCe0WkZHol+PlX\ncemR7MDn2+Dy8BFR1Y247/xx3IFnCHBVIP9eB3gY107yKe5MYVLgo1cAm8T1xpoGfF9Vj0daHlM1\n4lKexiSGiNTEpQFGqOrbiS6PMX5hNXcTdyIyJJCmqAP8CtfL4t0EF8sYX7HgbhLhImA77pT/28Bw\nVS0vLWOMqQJLyxhjjA9Zzd0YY3woYQOHtWjRQtu1a5eo1RtjTEpas2bNflWtqPswkMDg3q5dO1av\nXp2o1RtjTEoSkcqutAYsLWOMMb5kwd0YY3zIgrsxxviQ3YnJmGrixIkT5Obm8tVXXyW6KMaDunXr\n0qpVK9LSyhtaqGIW3I2pJnJzc2nUqBHt2rXDDcZpkpWqcuDAAXJzc2nfvn3lHwghpdIyWVnQrh3U\nqOEes8K65bMx1dtXX31F8+bNLbCnABGhefPmEZ1lpUzNPSsLMjMhL8+93rHDvQYYE/E4eMZUDxbY\nU0ek+yplau6TJhUH9qC8PDfdGGNMaSkT3HfuDG+6MSa5HDhwgO7du9O9e3dOO+00zjzzzKLXx497\nG/b9pptuYsuWLRXOM3PmTLKilLO96KKLWLduXVSWFW8pk5Zp08alYkJNN8ZEX1aWOzPeudP9n02d\nGlkKtHnz5kWBcvLkyTRs2JC777671DyqiqpSo0boeufTTz9d6XruuOOOqhfSR1Km5j51KtSvX3pa\n/fpuujEmuoJtXDt2gGpxG1csOjFs27aNTp06MWbMGDp37syePXvIzMwkIyODzp07M2XKlKJ5gzXp\n/Px8mjRpwoQJE0hPT+fCCy/ks88+A+Dee+9l+vTpRfNPmDCB3r17c9555/Gf//wHgC+//JLvfe97\ndOrUiREjRpCRkVFpDX327Nl07dqVLl26MHHiRADy8/O5/vrri6bPmDEDgEcffZROnTrRrVs3rrvu\nuqh/Z16kTM09WGOIZk3CGBNaRW1csfif27x5M88++ywZGRkAPPjggzRr1oz8/HwGDBjAiBEj6NSp\nU6nPHD58mP79+/Pggw8yfvx4nnrqKSZMmHDSslWVd999lwULFjBlyhReffVVHnvsMU477TReeOEF\n3n//fXr27Flh+XJzc7n33ntZvXo1jRs3ZtCgQfzzn//k1FNPZf/+/WzYsAGAQ4cOAfDwww+zY8cO\nateuXTQt3lKm5g7uR5WTA4WF7tECuzGxEe82rrPPPrsosAPMnTuXnj170rNnTzZt2sSHH3540mfq\n1avH5ZdfDkCvXr3IyckJueyrr776pHmWL1/OyJEjAUhPT6dz584Vlm/lypVceumltGjRgrS0NEaP\nHs2yZcs455xz2LJlC+PGjWPJkiU0btwYgM6dO3PdddeRlZVV5YuQIpVSwd0YEx/ltWXFqo2rQYMG\nRc+3bt3KH//4R9544w3Wr1/PkCFDQvb3rl27dtHzmjVrkp+fH3LZderUqXSeqmrevDnr16+nX79+\nzJw5k1tvvRWAJUuWcNttt7Fq1Sp69+5NQUFBVNfrhQV3Y8xJEtnG9cUXX9CoUSNOOeUU9uzZw5Il\nS6K+jr59+zJ//nwANmzYEPLMoKQ+ffqwdOlSDhw4QH5+PvPmzaN///7s27cPVeWaa65hypQprF27\nloKCAnJzc7n00kt5+OGH2b9/P3llc1xxkDI5d2NM/CSyjatnz5506tSJDh060LZtW/r27Rv1dfz4\nxz/mhhtuoFOnTkV/wZRKKK1ateKBBx7gkksuQVW58sorGTp0KGvXruUHP/gBqoqI8NBDD5Gfn8/o\n0aM5cuQIhYWF3H333TRq1Cjq21CZhN1DNSMjQ+1mHcbEz6ZNm+jYsWOii5EU8vPzyc/Pp27dumzd\nupXBgwezdetWatVKrvpuqH0mImtUNaOcjxRJri0xxpg4OHr0KAMHDiQ/Px9V5Yknnki6wB4pf22N\nMcZ40KRJE9asWZPoYsSUNagaY4wPWXA3xhgfsuBujDE+ZMHdGGN8yIK7MSYuBgwYcNIFSdOnT+f2\n22+v8HMNGzYEYPfu3YwYMSLkPJdccgmVda2ePn16qYuJrrjiiqiM+zJ58mSmTZsW8XKizYK7MSYu\nRo0axbx580pNmzdvHqNGjfL0+TPOOIPnn3++yusvG9wXLVpEkyZNqry8ZGfB3RgTFyNGjOCVV14p\nujFHTk4Ou3fvpl+/fkX9znv27EnXrl15+eWXT/p8Tk4OXbp0AeDYsWOMHDmSjh07Mnz4cI4dO1Y0\n3+233140XPCvf/1rAGbMmMHu3bsZMGAAAwYMAKBdu3bs378fgEceeYQuXbrQpUuXouGCc3Jy6Nix\nIz/84Q/p3LkzgwcPLrWeUNatW8cFF1xAt27dGD58OAcPHixaf3AI4OCAZW+99VbRzUp69OjBkSNH\nqvzdhmL93I2phn7yE4j2DYa6d4dAXAypWbNm9O7dm8WLFzNs2DDmzZvHtddei4hQt25dsrOzOeWU\nU9i/fz8XXHABV111Vbn3EX388cepX78+mzZtYv369aWG7J06dSrNmjWjoKCAgQMHsn79esaNG8cj\njzzC0qVLadGiRallrVmzhqeffpqVK1eiqvTp04f+/fvTtGlTtm7dyty5c3nyySe59tpreeGFFyoc\nn/2GG27gscceo3///tx3333cf//9TJ8+nQcffJCPP/6YOnXqFKWCpk2bxsyZM+nbty9Hjx6lbt26\nYXzblfNUcxeRISKyRUS2icjJAyYXz/c9EVERqfTSWGNM9VMyNVMyJaOqTJw4kW7dujFo0CB27drF\n3r17y13OsmXLioJst27d6NatW9F78+fPp2fPnvTo0YONGzdWOijY8uXLGT58OA0aNKBhw4ZcffXV\nvP322wC0b9+e7t27AxUPKwxufPlDhw7Rv39/AG688UaWLVtWVMYxY8Ywe/bsoith+/bty/jx45kx\nYwaHDh2K+hWylS5NRGoCM4HLgFxglYgsUNUPy8zXCLgLWBnVEhpjoq6iGnYsDRs2jJ/+9KesXbuW\nvLw8evXqBUBWVhb79u1jzZo1pKWl0a5du5DD/Fbm448/Ztq0aaxatYqmTZsyduzYKi0nKDhcMLgh\ngytLy5TnlVdeYdmyZSxcuJCpU6eyYcMGJkyYwNChQ1m0aBF9+/ZlyZIldOjQocplLctLzb03sE1V\nt6vqcWAeMCzEfA8ADwFV/yaNMb7WsGFDBgwYwM0331yqIfXw4cN84xvfIC0tjaVLl7Ij1A2TS7j4\n4ouZM2cOAB988AHr168H3HDBDRo0oHHjxuzdu5fFixcXfaZRo0Yh89r9+vXjpZdeIi8vjy+//JLs\n7Gz69esX9rY1btyYpk2bFtX6//73v9O/f38KCwv55JNPGDBgAA899BCHDx/m6NGj/O9//6Nr167c\nc889nH/++WzevDnsdVbEy3nAmcAnJV7nAn1KziAiPYHWqvqKiPy8vAWJSCaQCdDG7mxtTLU0atQo\nhg8fXqrnzJgxY7jyyivp2rUrGRkZldZgb7/9dm666SY6duxIx44di84A0tPT6dGjBx06dKB169al\nhgvOzMxkyJAhnHHGGSxdurRoes+ePRk7diy9e/cG4JZbbqFHjx4VpmDK88wzz3DbbbeRl5fHWWed\nxdNPP01BQQHXXXcdhw8fRlUZN24cTZo04Ve/+hVLly6lRo0adO7cueiuUtFS6ZC/IjICGKKqtwRe\nXw/0UdU7A69rAG8AY1U1R0TeBO5W1Qo7ndqQv8bElw35m3oiGfLXS1pmF9C6xOtWgWlBjYAuwJsi\nkgNcACywRlVjjEkcL8F9FXCuiLQXkdrASGBB8E1VPayqLVS1naq2A94Brqqs5m6MMSZ2Kg3uqpoP\n3AksATYB81V1o4hMEZGrYl1AY0z0JOrOayZ8ke4rTx0rVXURsKjMtPvKmfeSiEpkjImJunXrcuDA\nAZo3b17uxUEmOagqBw4ciOjCJrtC1ZhqolWrVuTm5rJv375EF8V4ULduXVq1alXlz1twN6aaSEtL\no3379okuhokTGzjMGGN8yIK7Mcb4kAV3Y4zxIQvuxhjjQxbcjTHGhyy4G2OMD1lwN8YYH7Lgbowx\nPmTB3RhjfMiCuzHG+JAFd2OM8SEL7sYY40MW3I0xxocsuBtjjA9ZcDfGGB+y4G6MMT5kwd0YY3zI\ngrsxxviQBXdjjPEhC+7GGONDFtyNMcaHLLgbY4wPWXA3xhgfsuBujDE+ZMHdGGN8yIK7Mcb4kAV3\nY4zxIQvuxhjjQxbcjTHGhyy4G2OMD1lwN8YYH7LgbowxPmTB3RhjfMiCuzHG+JAFd2OM8SEL7sYY\n40MW3I0xxoc8BXcRGSIiW0Rkm4hMCPH+bSKyQUTWichyEekU/aIaY4zxqtLgLiI1gZnA5UAnYFSI\n4D1HVbuqanfgYeCRqJfUGGOMZ15q7r2Bbaq6XVWPA/OAYSVnUNUvSrxsAGj0imiMMSZctTzMcybw\nSYnXuUCfsjOJyB3AeKA2cGmoBYlIJpAJ0KZNm3DLaowxxqOoNaiq6kxVPRu4B7i3nHlmqWqGqmac\neuqp0Vq1McaYMrwE911A6xKvWwWmlWce8N1ICmWMMSYyXoL7KuBcEWkvIrWBkcCCkjOIyLklXg4F\ntkaviMYYY8JVac5dVfNF5E5gCVATeEpVN4rIFGC1qi4A7hSRQcAJ4CBwYywLbYwxpmJeGlRR1UXA\nojLT7ivx/K4ol8sYY0wE7ApVY4zxIQvuxhjjQxbcjTHGh3wb3LOyoF07qFHDPWZlJbpExhgTP54a\nVFNNVhZkZkJennu9Y4d7DTBmTOLKZYwx8eLLmvukScWBPSgvz003xpjqwJfBfefO8KYbY4zf+DK4\nlzcmmY1VZoypLnwZ3KdOhfr1S0+rX99NN8aY6sCXwX3MGJg1C9q2BRH3OGuWNaYaY6oPX/aWARfI\nLZgbY6orX9bcjTGmurPgbowxPmTB3RhjfMiCuzHG+JAFd2OM8SEL7sYY40MW3I0xxocsuBtjjA9Z\ncDfGGB+y4G6MMT5kwd0YY3zIgrsxxviQBXdjjPEhC+7GGONDFtyNMcaHLLgbY4wPWXA3xhgfsuBu\njDE+ZMHdGGN8yIK7Mcb4kAV3Y4zxIQvuxhjjQxbcjTHGh6p9cM/KgnbtoEYN95iVlegSGWNM5Kp1\ncM/KgsxM2LEDVN1jZmb1CvD/+x88+WSiS2GMibZqHdwnTYK8vNLT8vLc9Opi8mR3QNu/P9ElMcZE\nU8oF9wMH4NVXo7OsnTvDm+43x4/DP//pnr//fmLLYoyJLk/BXUSGiMgWEdkmIhNCvD9eRD4UkfUi\n8rqItI1+UZ3HH4crrohOTbNNm/Cm+82bb8KhQ+65BXdj/KXS4C4iNYGZwOVAJ2CUiHQqM9t7QIaq\ndgOeBx6OdkGDLrvM5cdffz3yZU2dCvXrl55Wv76bXh28+CI0aACnnmrBvbrJz4cvvkh0KUwseam5\n9wa2qep2VT0OzAOGlZxBVZeqajB7/Q7QKrrFLJaRAU2awL/+FfmyxoyBWbOgbVsQcY+zZrnpfldQ\nAC+9BEOHQs+eFtyrmylT4Lzz4OuvE10SEytegvuZwCclXucGppXnB8DiUG+ISKaIrBaR1fv27fNe\nyhJq1oSBA+G111wNPlJjxkBODhQWusfqENgB3nkH9u6F4cMhPR0+/NDl4E31sGQJfPopLFqU6JKY\nWIlqg6qIXAdkAL8P9b6qzlLVDFXNOPXUU6u8nsGD4ZNP4KOPqryIau/FF6F2bdd+0b07nDgBmzcn\nulQmHvLyYO1a93zOnMSWxcSOl+C+C2hd4nWrwLRSRGQQMAm4SlVjerJ32WXuMRqpmepIFbKz3fd4\nyimu5g6Wmqku3n3X5dzPOw8WLrTcu195Ce6rgHNFpL2I1AZGAgtKziAiPYAncIH9s+gXs7T27eHs\ns11qxoTv/ffh449dSgbgm9+EOnUsuFcXK1a4x0cfdTn37OzElsfERqXBXVXzgTuBJcAmYL6qbhSR\nKSJyVWC23wMNgX+IyDoRWVDO4qJm8GBYutSlE0x4XnzRDbdwVWDv1aoFXbpYcK8uli+Hzp1hyBA4\n66zqdUV2deIp566qi1T1m6p6tqpODUy7T1UXBJ4PUtWWqto98HdVxUuM3GWXwdGjrmEwXvwyDk12\nNvTr57pABqWnu+AejUZqk7wKC+G//4WLLnI9xEaPdt2KP/000SUz0ZZyV6gGDRjggmy8UjN+GYfm\no4/ggw/g6qtLT09Ph3377J/c7zZuhMOHoW9f93r0aBfw589PbLlM9KVscG/SBPr0iV9w98s4NMH8\najDfHmSNqtXD8uXuMRjcO3Z0vaWs10x8fP01fO97sHJl7NeVssEdXGrm3Xfh4MHYr8sv49BkZ7sL\nwVq3Lj29Wzf3aMHd31asgNNPd50SgkaPdsFm27bElau6+MUvXJtXPM6QUzq4Dx7sTimXLo39uvww\nDk1urvsnLpuSAWja1G2LBXd/W77c1dpFiqeNGuVez52buHJVB9nZMGMG/OQnMGxY5fNHKqWDe+/e\n0KhRfPq7+2Ecmpdeco9lUzJBwUZV40+7drm2omBKJqhVK7j4Ytd+ZA3qsZGTAzff7M6aH3ooPutM\n6eCeluYaVuORd/fDODTZ2S7H2qFD6PfT02HLFvjqq/iWy8RHsH/7RRed/N7o0W7fr1sX3zJVB8eP\nw/e/77IMzz3nrgyPh5QO7uBSM9u3uzsKxVoqj0Ozfz+89VbolExQerobUGzjxviVy8TP8uXubDPY\neF7SiBGuspRqvb9SwS9/6doGn3rKXVcQLykf3INDEdjVqhVbuNAF7vJSMmA9ZvxuxQq44AIXxMtq\n1gwuv9zl3QsK4l82v1q4EB55BO64w/WSiaeUD+7nnusaApMpuCfjxU7Z2e576tmz/HnOPtuN727B\n3X+OHHEpl7L59pJGj4bdu+Htt+NXLj/buRNuvBF69IBp0+K//pQP7iIuNfPGG24wpERLxoudjhxx\njc5XX126l0RZNWpA164W3P1o5UqXTqwouF95pTu4W5/3yJ044Xoh5ee7C8Tq1o1/GVI+uINLzRw6\nBKtXJ7okyXmx0+LF7uKJilIyQTYMgT+tWOEO3hdeWP489eu738g//mE38YjUr34F//mP63RxzjmJ\nKYMvgvvAga5GmgypmWS82Ck7240jU1GtLSg93R0oP/mk8nlN6li+3J2VnXJKxfONGeP2f7RuQl8d\nLV7sujtmZsLIkYkrhy+Ce/Pm0KtXcozvnmwXO331Ffzzn+6iiZo1K58/2KhqXeL8Iz/fDbAXqgtk\nWQMHuoqApWaqZtcuuOEGd8X39OmJLYsvgju41Mw777j8ciIl28VOr7/uRs+sqAtkSV27ukfLu/vH\nhg3uN+DlzC0tDa69FhYsSPz/UqrJz3d59mPHXJ69Xr3Elsc3wX3wYPflvvlmYsuRbBc7ZWe7U/FL\nL/U2f6NGrteMBXf/KDtYWGXGjHFnfHYTj/Dcf7/rafR//+fucpVovgnuF17oasjJkJpJloud8vPh\n5Zdh6FB3pyWvbBgCf1mxwg0U5zU1eMEFrguvpWa8mz/fnZ3ffDNcd12iS+P4JrjXqQP9+ydHo2qy\nWL7cXZnqNSUTlJ7urvg9ejQ25TLxo1o8WJhXwZt4/PvfsHdv7MrmB5s3u5vMf//7rj/7Y48lukTF\nfBPcwaVmtmxJvWF4YyU72/WvHTIkvM+lp7ugsGFDbMpl4mfnTtfI56UxtaTRo92VqnYTj9AOHYKf\n/cy1Ua1YAX/4g7vDVdn2tkTyVXD381AE4V71qurGjR48GBo2DG9dNgyBfwQHCwun5g7uHqvp6Zaa\nKaugAJ580t1U/tFHYexYd3ez8ePjNyCYV74K7p06wRln+C+4V+Wq19Wr3fjt4aZkwDUCN25swd0P\nli93jeTBXlDhGD3a9UDbvj365UpFb78N55/v/ve++U1YtcoF+pYtE12y0HwV3EVc7f3f/3aNmX5R\nlates7Ndv/Yrrwx/fSKun64F99S3YoXrbODlGoeyghfgVPebeOzc6b6Liy929xmeO9cF+l69El2y\nivkquIML7gcOwHvvJbok3nhJt4R71asqvPACXHKJG+2vKtLTYf16fx0kq5vDh127SbgpmaA2baBf\nv+p7Ew9V+N3v3P0PXn4Z7rvPNaCOHFnxGE3JwnfBfdAg95gKqRmv6ZZwr3pdvNjlASPpgpmeDl9+\naafkqey//3W/q6oGd3C/oU2bXHqmulm2DCZOdO1Wmze7fuwNGiS6VN75Lri3bOkCUzL0d6+M13RL\nOFe9qsLkye4sIJL+tt27u0dLzaSuFStcOqZPn6ovY8QI1/7Srx9cf3316kE1Y4Y7850717VDpRrf\nBXdwqZkVK04OnMnGa7olnKteFy92DT333hv6pgxede7sUkUW3FPX8uXuIB1ub6mSmjd3AX3cONeO\n062buyhu2TJ/p2p27nT3HP7hDxM/jEBV+TK4Dx7s7lu4bFmiS1KxcNItXq56LVlrv+GGyMpWr567\nhNqCe2o6ccKN4R5u//ZQWrd2dxPauRMeeMBVHvr3h299ywVAP7bL/PnP7vFHP0psOSLhy+B+0UXu\nitVkT81Ee5CxYK190qTIau1BNgxB6lq3zg1gFUm+vaxmzdwZ4Y4dMHOmu3p1+HDXBfmvf/XPGPDH\njrkujt/9buJGc40GXwb3evVcjjDZG1WjOchYNGvtQenp7h/50KHoLM/ET7iDhYWjXj1Xo/3oI5g3\nz1VIbrnF3fz5ySejv754mzMHPv/cpaJSmS+DO7jUzAcfwJ49iS5JxaI1yNirrxbX2qN1pVzwStX1\n66OzvEj8/veuX/FnnyW6JKlhxQpo395d1BcrtWq5MVXWrHEVqfbtXW+vZPi9VJWqGx+ma1fXrz2V\n+Ta4B4cieOgh1+f9xInElieWYlFrh+QZhuCll+AXv4C1a+Gaa/y9L6NB1QX3WNTaQxFxXZAXLnTD\nS0+eHJ+UTh41AAASr0lEQVT1xsLbb7vf+7hxqdGXvSK+De7durmj7x//CD17ukuwL7gA7rwTnnkG\nNm5040T4wauvwrvvhl9rr+wCqtNPhxYtEhvcP/zQdcHr3Rueeso1ko8fn7jypILt2+HTT6PTmBqO\npk3dvsnOdgfiVDRjhtuO0aMTXZIoUNWE/PXq1UtjrbBQdds21XnzVO++W7V/f9WGDVVd3Ua1QQPV\niy5S/elPVRcscPOnmsJC1d69Vdu1U/36a++fmz1btX794u8C3OvZs0vPN3CgakZGeGWaN0916FDV\nPXvC+1xZBw+qnnOOasuWqrm5btrPfubK+tRTkS3bz555xn1HGzbEf92HDqk2bar6ne/Ef92R2rFD\ntWZN1V/8ItElqRiwWj3EWF8H91AKClQ3bVJ99lnVceNUv/Ut1Xr13DcxapTqkSMJKVaVLVrkyj5r\nVnifa9u2dGAP/rVtW3q+8eNV69ZVPXHC23I3biz+Ps85RzUnJ7xyBeXnq15+uWpamury5cXTT5xQ\nHTRItXZt1Xfeqdqy/e6HP1Rt0sT91hNh6lS3/1euTMz6q2rCBNUaNar+m40XC+5hOHHC/SBr1FDt\n0CExNZ6qCNba27YNr9auqioSOriLlJ4vWAv88MPKl5mXp9q1q+qpp6q++KILMK1bq27ZEl7ZVN0/\nGqg+8cTJ7+3fr9q+veoZZ0R+dpAK5s9XHTZM9dVXvZ1dduqkesUVsS9Xeb74QrV5c9UhQxJXhnDl\n5bkyDx+e6JJUzoJ7FbzxhksB1Kvnglqyq2qtXdV7zX3dOjd97tzKl3nHHW7eRYvc6/fec4H+G99w\ny/Hquefccm67rfx51q1zaaRvfSv8A1sq+fhjl0qsUcN9J927q86ZU/6Z1IEDbr6pU+NazJM89JAr\nx4oViS2HV3/9qyvv0qWJLknlLLhX0e7dLjcPqrfc4o7oySiSWruq95z711+71MiECRUvLzvbLWP8\n+NLTN29WbdXK1eL/+9/KyxUM2n37Vr5dwYPArbdWvtxUVFCgOmCAC+5btrgA1KGD2+b27VX/9CfV\nL78s/ZmFC937b72VmDIHHT3qDuoDBya2HF4UFrqDZpcuqdHuZsE9AidOqE6c6L6d9HTVjz5KdIlO\nFkmtPWj2bHdwEHGPZQN7ULduLv9dnp07XSNar16hA3JOjsu/N2ig+vrr5S9n/37XMHzmmd7TLRWl\nb1Ldn/508j4uKFB96SXVCy9077VooXr//e67U3XfR1paclRKHnnElfHNNxNdkootWxb5/1I8WXCP\ngldeUW3WTLVRI9Xnn090aYpFWmsP1/XXu/x2KCdOqPbr52qXFR0E9+xxNaM6dVzPpFDLGTjQvf/u\nu97Llp/vcrtpaamTAvBi61Z3BjNkSOjaZGGhC0pDhxafdd11l2qPHqp9+sS/vKHk5amefrrqxRcn\nd434mmtc5aTsWVCyimpwB4YAW4BtwIQQ718MrAXygRFelpkKwV3V1Tp793bf1F13JUd+d/Hiimsa\nXmvkXk2b5tb32Wcnvzd5snvv2WcrX86BA6rnn++6m82ZU/q9n/7ULacqbR2ff6569tmqp51W3GUy\nleXnuy66jRurfvJJ5fNv2OAOwLVqacjUWCI99pgr07//neiShLZzp/s9/vzniS6Jd1EL7kBN4H/A\nWUBt4H2gU5l52gHdgGf9FtxVXUAfN859W336uL7ziVJY6MrQpk3oA43XXHo4XnvNLee110pPf+st\n19B3/fXel/XFF65NQ6Q4lfLss8UHz6r64AOX9undW/XYsaovJxkE0xl/+1t4n9uxQ/W3v3WPyeLY\nMdfm8q1vJWftfeJE9xv++ONEl8S7aAb3C4ElJV7/EvhlOfP+zY/BPegf/3ApmrQ0V9M8cCD+ZQjW\n2svLMXvtBROOzz5zy5g2rXja/v3un/acc1zADkdenuuqB6o//rHrRz9ggOrx41Uvo6rqCy+4Zd58\nc3IGEi82b3bfx5VXpu42lPX4426/vPpq+J89eNBdlxILwe6P3/1ubJYfK9EM7iOAv5R4fT3wp3Lm\n9XVwV3Wn/Tff7GqeTZq4gPfVV/FZd2W1dlXv/dfD1aSJqxmLuPX36uUOcqtXV215X3/tcp3BA0+o\nlE9V3Htv1Wq9yeDECbd/mzXzV//9r792+/j888M7YL36qsvZi7hKQLQvMHzqKfdbeeON6C431pIy\nuAOZwGpgdZs2beLxPcTM+vWusQtcD485c2J/ReDcuRXX2lVjU3OfPbu4n3XJv9Gjq75MVZdbfuwx\nV1uNloICl5pp0yZ+B92gDz5Qvekm1csuq9rVsw8+6L7Xsu0RfvCXv7htW7iw8nnz8orToJ07q2Zm\nFrcfLVkSnfIUFrrG51Tp/liSpWXi5F//ct0lwdVMYtHtq6BA9YEH3A+8R4+KG3VjkXMv74AR6vgc\n7cbcqliyxJXvz3+O/bqCvVa+853i77ply+LapteU1YYNbkiF730v9YKNF8ePq551lvv9VrR9a9e6\nK2xB9Sc/KW4/Wb68uI//2LGRp0TffrvyilKyimZwrwVsB9qXaFDtXM681S64q7oa6N/+5nLQoHrV\nVdHLEx48WBw4Ro92F4dUJtoB1muqJxYHlqooLHQXQZ15ZuwaV/PzXY7/ggu0qL/5lCmuLeLwYdU7\n73TfT6tWqi+/XPGyjh9X7dnTXc0brfRUMgoOZfHiiye/l5/vrmpNS3Pdbv/1r5PnOXZMddIk17ul\nZcvIuidfe61LNXr5f0o20e4KeQXwUaDXzKTAtCnAVYHn5wO5wJfAAWBjZcv0U3APystzvRUaNXI/\nwFtvjaznwvvvuy5+tWq59EWianReUz2xSAlV1euvu3XPmBHd5R475mp7557rln/WWe4MIVQf6Xfe\ncWPtgOrVV5ffTfP++908yXQtRSycOKH6zW+676RkCjMnp/iq8BEjii/IKs9777kzgOD3unt3eOX4\n5BP3/3n33WFvQlKwi5gSaO9eN85KWpr7y8wMv6vV7NlujJvTT0/8xTnBslRWI49VY25VFBa6gHHa\nadG5WvPzz914LS1bum3KyHADeuXnV/y548dVf/c71wPmlFNUZ84sHdjee88dvEeNiryMqSAry31/\n8+e717Nnu++lUSNXs/dagTlxwrVR1KnjauBPP33yZ7/6yp1BL1yo+uij7n/y2992Z3Qiqtu3R3XT\n4saCexLYsUP1Rz9yudRatVwvm61bK/7M11+7XC24K/uSpdfE7Nkux15RqieZau6qrv0DXL/xSOzY\n4dIu4IZhWLo0/LOobdvcUMXghg7YsMHt627d3AEoEd1qEyE/3+XUO3ZUHTnSfR99+1Y90G7e7K6Q\nBneF8623usdgWrLk77BxY3dQHjkytRutLbgnkdxc1/pft27xRT+heojs2uUu9gheZRhpv+94S5ac\ne0kDB7oBrKqaW83PdwfZhg1VV62KrCyFhe6CrebN3cH+oovUcw8SP5k/3213rVqqv/mN93sFlKeg\nwKXGGjd2322fPqpjxqj++tfut/fOOy7V45eGagvuSWjPHncnofr1Xa1i5EjXfU7VXe3ZsqXrS/7c\nc4ktZyTCacyNR8+a5cvdr/zhh6v2+d/+VqPeb37fPtUbbtCinh/VTUGB6h/+EPnBsiy/BO/KWHBP\nYnv3qt5zjwvkwdPJmjVVzzvP3cmoOohnLf/b33Y1unCvpF21ytUur7kmNoFjy5bUOzsziec1uIub\nN/4yMjJ09erVCVl3sjhwAKZPdzflvewydwPoU05JdKnio1072LHj5Olt20JOTnTXtXKluzn6b38L\nv/ylt898+aW7sXpenrtBeLNm0S2TMVUlImtUNaOy+WrEozAmtObN4YEH4PPP4fnnq09gB9i50/v0\nrCx3MKhRwz1mZYW3rj59YOhQ+P3v4YsvvC1z/HjYuhWefdYCu0lNFtyTQM2aiS5B/LVp4216VhZk\nZrpavqp7zMwMP8Dffz8cPAh//GPly3z5ZZg1C+6+GwYM8L6OSA9CxkSVl9xNLP6qc87deM+5h9u9\nsqJG2mHDim/aXd4yd+923R579AhvbJpk7Clk/AlrUDXJzktvmXAujKoswL73Xuhllfz79rddl9UP\nPwxvW5Ktj7/xL6/B3RpUTVILp+HVy7wjRsCLL7rQW1bTpi51M3Mm/OhH4ZWzRo3QyxSBwsLwlmVM\nRaxB1fjC1KlQv37pafXru+lleWmknTzZPdaqVXqeunXh6FHX8Hr77aXf85JL99qGYGLD2jtC8FK9\nj8WfpWWMV14vdvKaGvn+913qpVWr4puPtG7trmTdu/fkdXvJpYebc0+GoZH9orq1d2A5d1PdeP0n\n37jRBdV77nGvgzfnfuWVk5cZTi7da8BOdDDy24ElFvsomVlwN9WS13/e0aOLAyq4EQNDicVIl7Fq\nfPWy7Yk+sMRCqt1vIFIW3I2pwObNxbcO7Nix/GGBYxGIY3HAiFXX0liIdu05Fe83EAkL7sZUYuxY\nNxzze++VP088b1sYSRrB6zITPeZ+LL5Pr8tM9LZHiwV3Yypx/Li3O2VFu6YZi0Zar4ErmheFVUUi\nU1JWc7fgbkzMRTsYeZ03nANGLGrZiaw9W87dgrsxSSGaV+eWnTeaqZ5wJLr2HIveMvHugWPB3ZgU\nl+gUSiIbfoPzJnu3xUScDVhwNybFJTqNYF02K5eIMxGvwd3GljEmiWVlwaRJbgiFNm3csAtjxsRv\n3ZmZ7oYlQfXru+GQY12GeN7MJRKJGFPIxpYxxgfGjHHBrLDQPcYrsAfXPWuWC6gi7jEegR3Cu5lL\nIoU7plA8x8Cx4G6MKZfXg0u0g1aqDMQWzsB20brxjFcW3I0xEYlF0AonaCZSOGc3kyaVTnGBez1p\nUmzKZsHdGBORWAStRKaEgryejXg9u4l3qskaVI0xEfHjjUpi0ZgcrUZia1A1xsRFquTHwxGLs5F4\np5osuBtjIpIq+fFwxCKFEu9UkwV3Y0xEkiE/Hm2xOhuJZ9dWC+7GmIglsj9+LPjhbMSCuzHGlOGH\ns5Falc9ijDHVz5gxqRXMy7KauzHG+JAFd2OM8SEL7sYY40MW3I0xxocsuBtjjA8lbGwZEdkHhBhp\nwZMWwP4oFicZ+G2b/LY94L9t8tv2gP+2KdT2tFXVUyv7YMKCeyREZLWXgXNSid+2yW/bA/7bJr9t\nD/hvmyLZHkvLGGOMD1lwN8YYH0rV4D4r0QWIAb9tk9+2B/y3TX7bHvDfNlV5e1Iy526MMaZiqVpz\nN8YYUwEL7sYY40MpF9xFZIiIbBGRbSIyIdHliZSI5IjIBhFZJyIpeVNZEXlKRD4TkQ9KTGsmIq+J\nyNbAY9NEljEc5WzPZBHZFdhP60TkikSWMVwi0lpElorIhyKyUUTuCkxPyf1Uwfak7H4Skboi8q6I\nvB/YpvsD09uLyMpAzHtORGp7Wl4q5dxFpCbwEXAZkAusAkap6ocJLVgERCQHyFDVlL3wQkQuBo4C\nz6pql8C0h4HPVfXBwEG4qarek8hyelXO9kwGjqrqtESWrapE5HTgdFVdKyKNgDXAd4GxpOB+qmB7\nriVF95OICNBAVY+KSBqwHLgLGA+8qKrzROT/gPdV9fHKlpdqNffewDZV3a6qx4F5wLAEl6naU9Vl\nwOdlJg8Dngk8fwb3j5cSytmelKaqe1R1beD5EWATcCYpup8q2J6Upc7RwMu0wJ8ClwLPB6Z73kep\nFtzPBD4p8TqXFN+huJ33LxFZIyKZiS5MFLVU1T2B558CLRNZmCi5U0TWB9I2KZG+CEVE2gE9gJX4\nYD+V2R5I4f0kIjVFZB3wGfAa8D/gkKrmB2bxHPNSLbj70UWq2hO4HLgjkBLwFXW5v9TJ/4X2OHA2\n0B3YA/whscWpGhFpCLwA/ERVvyj5XirupxDbk9L7SVULVLU70AqXqehQ1WWlWnDfBbQu8bpVYFrK\nUtVdgcfPgGzcDvWDvYG8aDA/+lmCyxMRVd0b+McrBJ4kBfdTII/7ApClqi8GJqfsfgq1PX7YTwCq\neghYClwINBGR4C1RPce8VAvuq4BzA63HtYGRwIIEl6nKRKRBoDEIEWkADAY+qPhTKWMBcGPg+Y3A\nywksS8SCATBgOCm2nwKNdX8FNqnqIyXeSsn9VN72pPJ+EpFTRaRJ4Hk9XMeRTbggPyIwm+d9lFK9\nZQACXZumAzWBp1R1aoKLVGUichautg7uZuVzUnF7RGQucAlueNK9wK+Bl4D5QBvc0M7XqmpKNFKW\nsz2X4E71FcgBbi2Rq056InIR8DawASgMTJ6Iy1On3H6qYHtGkaL7SUS64RpMa+Iq3vNVdUogTswD\nmgHvAdep6teVLi/VgrsxxpjKpVpaxhhjjAcW3I0xxocsuBtjjA9ZcDfGGB+y4G6MMT5kwd0YY3zI\ngrsxxvjQ/wPcz2v1dLH/YgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7fd544300160>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "acc = history.history['acc']\n",
        "val_acc = history.history['val_acc']\n",
        "loss = history.history['loss']\n",
        "val_loss = history.history['val_loss']\n",
        "\n",
        "epochs = range(len(acc))\n",
        "\n",
        "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
        "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
        "plt.title('Training and validation accuracy')\n",
        "plt.legend()\n",
        "\n",
        "plt.figure()\n",
        "\n",
        "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
        "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
        "plt.title('Training and validation loss')\n",
        "plt.legend()\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "twdDu5Qpy70Q"
      },
      "source": [
        "As you can see, we reach a validation accuracy of about 96%. This is much better than our small convnet trained from scratch."
      ]
    }
  ],
  "metadata": {
    "anaconda-cloud": {},
    "kernelspec": {
      "display_name": "tensorflow",
      "language": "python",
      "name": "tensorflow"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.8"
    },
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}