01_part3_Keras.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "01_part3_Keras.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/MatchLab-Imperial/deep-learning-course/blob/master/01_part3_Keras.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bb1Zl0tVSKCN"
      },
      "source": [
        "# **Starting with Keras**\n",
        "\n",
        "Keras is a high-level Deep Learning API  written in Python and capable of running on top of TensorFlow, CNTK, or Theano. It was developed with a focus on enabling fast experimentation. In other words, Keras is especially good for beginners, it is modular, minimalist, and it makes everything easy to get a neural network up and running in no time. \n",
        "\n",
        "The following code is based on [this tutorial](https://elitedatascience.com/keras-tutorial-deep-learning-in-python), and it is just a brief introduction to Keras. You can refer to [Keras documentation](https://keras.io/) to learn more details in the future about it."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rF1rW5C1dS4R"
      },
      "source": [
        "## Importing Libraries and Modules\n",
        "\n",
        "Google Colab already has everything you need installed. We highly recommend to use Colab for these tutorials, however, if you want to use your own environment, you will need to set up a Python ecosystem and install Keras on it. \n",
        "\n",
        "You can find multiple guides that will help you with the task of installing Keras on your machine. Here are some examples for [Windows](http://inmachineswetrust.com/posts/deep-learning-setup/) or [Ubuntu](https://medium.com/@naomi.fridman/install-conda-tensorflow-gpu-and-keras-on-ubuntu-18-04-1b403e740e25)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wzb418Yih5SN"
      },
      "source": [
        "Let's start with an easy example. First of all, we will import NumPy and define the seed for the pseudorandom number generator. Specifying the seed is a common practice because different initializations might lead to varying results. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VTilKf6uRhML"
      },
      "source": [
        "import numpy as np\n",
        "np.random.seed(1234)  # for reproducibility"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ob8EYZ89h96k"
      },
      "source": [
        "Now we are ready to import some of the Keras' modules.\n",
        "\n",
        "First, we will import the `Sequential` module from Keras. A `Sequential` module is a stack of neural network layers. We will need it for creating our first feed-forward neural network."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tE570PePip7f"
      },
      "source": [
        "from keras.models import Sequential"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IC1aRifsj4Eu"
      },
      "source": [
        "As mentioned before, Keras is a wrapper that allows us to use either Theano or TensorFlow as a backend. It is easy to switch between them if required. However, newer versions of Keras will focus on the TensorFlow backend, dropping support for the rest of the backends. Thus, in this tutorial, we will use the TensorFlow backend. Similar to the NumPy pseudorandom number generator, we also fix Keras' pseudorandom number generator for reproducibility."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "81_cpwH9UhRn"
      },
      "source": [
        "import tensorflow as tf\n",
        "tf.random.set_seed(123)"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8zCje2VIWFIV"
      },
      "source": [
        "Next, we import two of the core layers used to build a network on Keras: [`Dense`](https://keras.io/layers/core/#dense) and [`Activation`](https://keras.io/layers/core/#activation). In short, the `Dense` defines a fully-connected network where all the neurons are connected to all of the elements of the input data, and the `Activation` defines a non-linear function applied to the network. We will explain more about those layers in the following tutorials."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ua1JaDKTj5XO"
      },
      "source": [
        "from keras.layers import Dense, Activation"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vbm3UwLeKdDv"
      },
      "source": [
        "In addition, we need to import some utils, that will help us later with the visualisation and transformation of the dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yUVkzIz3KtXt"
      },
      "source": [
        "from keras.utils import np_utils\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZBQn3bnIkpOE"
      },
      "source": [
        "## Loading Datasets from Keras\n",
        "\n",
        "Besides the modules to build our neural networks, Keras library also contains a set of [standard datasets](https://keras.io/datasets/). The fact that those datasets are inside Keras is handy and can save us a lot of time when learning how to create our first neural network. \n",
        "\n",
        "Therefore, let's import the [MNIST](https://en.wikipedia.org/wiki/MNIST_database) dataset. MNIST is a popular dataset on the computer vision community that contains a large number of binary images of handwritten digits. Once loaded, Keras allows us to easily split the dataset into the commonly used train and test data splits."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "92F5A9XEIiVQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fc9dfb17-bbd0-4b17-e22d-d3fd7cd6a13e"
      },
      "source": [
        "from keras.datasets import mnist\n",
        "\n",
        "# the data, shuffled and split between train and test sets\n",
        "(X_train, y_train), (X_test, y_test) = mnist.load_data()"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
            "11493376/11490434 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nuHAmcCaImS8"
      },
      "source": [
        "Automatically, Keras has provided us with the `X_train` and `X_test` images for training and testing, respectively. Moreover, Keras also stores into `y_train` and `y_test` the corresponding numeric labels associated with `X_train` and `X_test`.\n",
        "\n",
        "We can now look into the shape of the imported data and visualise some examples:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l5o9qsfjJdNv",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 319
        },
        "outputId": "cfd66970-3bee-4afd-ffe6-130288c5efae"
      },
      "source": [
        "print('Image shape: {0}'.format(X_train.shape[1:]))\n",
        "print('Total number of training samples: {0}'.format(X_train.shape[0]))\n",
        "print('Total number of test samples: {0}'.format(X_test.shape[0]))\n",
        "\n",
        "N=5\n",
        "start_val = 0 # pick an element for the code to plot the following N**2 values\n",
        "fig, axes = plt.subplots(N,N)\n",
        "items = list(range(0, 10))\n",
        "for row in range(N):\n",
        "  for col in range(N):\n",
        "    idx = start_val+row+N*col\n",
        "    axes[row,col].imshow(X_train[idx], cmap='gray')\n",
        "    fig.subplots_adjust(hspace=0.5)\n",
        "    y_target = int(y_train[idx])\n",
        "    target = str(items[y_target])\n",
        "    axes[row,col].set_title(target)\n",
        "    axes[row,col].set_xticks([])\n",
        "    axes[row,col].set_yticks([])"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Image shape: (28, 28)\n",
            "Total number of training samples: 60000\n",
            "Total number of test samples: 10000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUYAAAD7CAYAAAAW9D3vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydeXhV1bn/P+ucnCHznJABCEmAyBAiAsogyCAKKIiCCiIWbavWobX2Wmnrbavt9fZXq9XeqhUsKC1aFaxMigwyT4JMMiYhA2Se5+RM7++PmFNIQMm4M+zP86znSc6w17u/Z+/vXsO711Yigo6Ojo7OfzBoHYCOjo5OZ0M3Rh0dHZ1G6Maoo6Oj0wjdGHV0dHQaoRujjo6OTiN0Y9TR0dFphG6MOjo6Oo1od2NUSm1TStUqpSq/KWfau87OjlLKopR6WymVoZSqUEodUUpN0zourVFK/UMplaOUKldKnVVKfV/rmLTkonOmoTiVUn/ROq7OglKq/zfe8o+23nZHtRgfFxGfb8rADqqzM+MBnAcmAP7Ar4APlFIxGsbUGXgRiBERP2Am8Dul1HUax6QZF50zPkAvoAb4UOOwOhN/Bb5sjw3rXWkNEJEqEfmNiKSLiEtE1gFpQI81AQAROSEidQ3/flPiNAypM3EXkA/s1DqQzoBS6l6gFNjSHtvvKGN8USlVqJTarZS6qYPq7DIopcKBAcAJrWPRGqXU60qpauA0kANs0DikzsIDwLui38OLUsoPeB74aXvV0RHG+HMgFogC3gLWKqX0VsA3KKVMwD+Bd0TktNbxaI2I/AjwBW4EVgN13/6N7o9Sqi/1wy7vaB1LJ+EF4G0RudBeFbS7MYrIfhGpEJE6EXkH2A1Mb+96uwJKKQOwArABj2scTqdBRJwisguIBh7VOp5OwP3ALhFJ0zoQrVFKJQFTgFfasx6P9tz4FRBAaVBvp0IppYC3gXBguojYNQ6pM+KBPsYIsBD4X62D6CTcBMQAmfWnED6AUSk1SESGt1Ul7dpiVEoFKKVuUUpZlVIeSqn7gPHAZ+1ZbxfhDeAa4HYRqdE6GK1RSoUppe5VSvkopYxKqVuAebTT4HpXQSk1hvphKH02up63qL9YJn1T3gTWA7e0ZSXt3WI0Ab8DEgAn9QPqd4jI2Xaut1PzzZjRw9SPn+V+c+UDeFhE/qlZYNoi1Heb36T+gp0B/ERE1mgalfY8AKwWkQqtA+kMiEg1UN3wv1KqEqgVkYK2rEfpk1w6Ojo6l6LnMero6Og0QjdGHR0dnUboxqijo6PTCN0YdXR0dBqhG6OOjo5OI5qVrqOU0nQKW0Q6XWK4rklTdE0uj65LUzqrJnqLUUdHR6cRujHq6OjoNEKLe6V1dC7BYrEQGBhIdHQ0/fv3p6CggOLiYlwuFxcuXKCoqAj9RgSd78LT05O+ffsSFhbGl19+SU1Ny++07TBj9PD4T1XBwcFYLBY8PDxwuVzk5ubSp08fQkJCcLlc5Ofnk5eXR1VVVUeFpykGgwGj0Yi3tzdWqxWn00lhYWGPMAM/Pz9iY2MZNWoUo0aN4pZbbuH06dOcO3cOp9PJJ598wq5du3rMsdCTUUphNBoBcLlcuFyuZn0/PDyc2267jZEjR/L8889z4kTLlzdtc2M0GAzuHfTw8MDDwwOTyURwcLD7MzNnziQqKorAwEAqKipYtmwZzz33HLNmzaKqqorly5fz9ttvc+TIkbYOr1NhNBrx8vLC398fHx8fEhMTiYuLo7y8nJUrV1JSUqJ1iO3OkCFDePDBB5k3bx5eXl4AREdHX/KZgoICjh07hsPh0CJEnQ7C19eXoKAgLBYL+fn5zT7++/Xrx3XXXcfo0aO58847SU5OxmaztSwYEbnqwn+Wm79i6d27t8THx8u0adPk2WeflWXLlsmaNWvE6XQ2KWVlZXL8+HFZs2aNOBwOKSsrk927d8t9990nERERTbbdnFg7qlyNJpcrHh4eEh8fL2+88YYcPHhQKioqxOFwiMPhkKysLPmf//mfq9qO1vvfWk28vLzk0UcflaKiIrkc1dXV8uGHH8r1119/1dvUev/b+li5uCil3MVgMFxSuqIuF8c3a9YsWb9+vaSmpsojjzzSbG3i4+Plj3/8ozgcDsnLy5O4uLgWa9KmLcakpCRWrVpFVFSUu+t80coxlyAiHDhwgOXLl1NbW8s777xDTk4OxcXFZGRktGp8oDMTEBDAtddey8KFC5kzZw6enp4opcjPzyc7OxsvLy9CQ0OZPXs277zzDmfOdO+HKlZXV7Njxw6WLl3KE088gaenJ+fPnycqKgqDwYCnpycDBgxg1KhR7N+/X+twNeeRRx7Bz8+PyMhIRo4cyciRI3E6nZw5c4akpKQGs+nS9O7dm8WLF5OVlcXatWuv+nt+fn74+fm1SQxtaowXLlygtLSUiIiIyxriV199RXl5OYmJiXh5eXHq1Ck++ugj94/pcrkQEZxOZ1uG1Snw8vJi1qxZzJ07l5EjR7q7DAAZGRk888wzpKSkMHfuXH7+858THBxMYGCgxlF3DKdPn2bVqlXMnDmThIQEKioqLjnBT5061awTpDvh6+vLsGHDGDhwIKNHj2bmzJlYrVaUUphMJvfQ1YABA9i1axfTp0+nrKxM67BbhcFgoFevXvzlL3/h/PnzHD9+/Ds9wWq1MmXKFMaPHw/UT+jNnDmTV15p2ULfbWqMJSUlvPnmm4wfP56amhpiYmKYPHkyTqeTlJQUHn30USorK+nfvz/jx48nOTmZurqe8UiP4cOHM3v2bG666SZ8fHwwGOozpbKysnj66afZuXMnJpOJ2tpa4D8TMj0Bp9PJ+fPneeedd3jwwQfp16/fJfsuIs0eiO+qBAUF8fDDD7sbD2azmdDQUAIDA/Hz86OwsJDo6Gg8PT0B3BcQk8lEZGRklz5mysrK3KZuNBrp1asXP/rRj/jv//5v8vPzv/UY6NWrF4MHD6Zv375AfU+14QLSklZ0mxqj0+nks88+48yZM9TW1jJ8+HBCQ0OJj4/n3//+N0ePHsVut5OVlcW5c+d6jCkCXH/99VxzzTX4+fnhcDgoLCwkMzOTd955hx07dlBSUkJsbGybdQW6GsXFxXz88cfk5eXx/PPPExkZ6b549OvXjwkTJrBixQqNo2xfEhIS+N73vsf06dOJiorCZDK5W4ZGo5Hnn3+ejIwMwsLCCAwMJDExkXHjxuHr6wtAXl5el76AnD59ms8//5yoqCjGjh2Lh4cHU6dOZcOGDWzdupXy8vIrftdut1NXV4fNZsNsNmO32zl48GCLhxbafFb6/Pnz5OXl4XQ6MZvNpKamEh8fj91ud88qVlRUtGoqvSuSl5fHiRMnKCgoICcnh/Pnz3PmzBk++eQTSkpKEBECAgLw9/fXOlRNUEqhlCInJ6fJe97e3oSEhGgQVcehlGLGjBnMnz+f8PBwjEYjIkJ5eTkHDx4kPz+fJUuWUFlZidVq5dprryUsLMzdIqqoqGDVqlVdurGRl5fHtm3bCAkJITExEV9fX6Kjo5k2bRpff/01VVVV39mlbhjCczqdnDt3rsWxtEseY8MUeWlpKWlpadTV1TF9+nRee+01iouLu8UAcXNZu3YtOTk5eHl5cebMGfLy8pqMBQUEBBAQEKBRhNpgMpkICQlh0KBB3HTTTcTGxhIVFXXJGHVBQQGnT3ffJ8sqpQgICODOO+8kLCwMo9GIw+GgoKCAQ4cOsXz5ck6fPk1ubi4AlZWVREREkJCQgI+PDw6Hg7Nnz7J06VL3UExXRES4cOECO3fuJD09naFDhwIwefJkPvjgA7Kzs6mudj/VAKUU4eHhAAwaNIi+ffvi7e3dJrG0a4L36dOn+ec//8mQIUOYMmUKkyZNYuvWrdTU1FBXV9ctJ1muRFlZGVu2XPpcp4YZ6Qb69etHdHQ0IoLNZsNu7/4PDgwNDWXevHn8+Mc/vuIYWUFBQbeenTebzdx0000MHToUDw8PHA4Hubm5fPLJJ7zxxhuXXBQautbBwcEEBAQgIlRVVbFlyxZKS0u7fKOj4YJw7NgxtzHGxMTQu3dvjh07Rm1trTs/2mKxsGjRIgBuvPFGEhMT3dtxOp2t0qJdjdHhcHDq1Cmee+45Ro8ezZ/+9Cd27drF8ePHWbt2LSdOnOjyP2RL8fb25gc/+AGenp7usbSkpCQiIiKorKzkwIEDHDx4UOMo25+wsDBiYmLc3cfLMWjQIGbNmtXiGcbOjqenJ4888oh7QuXEiRO89957rF27tskFwd/fn3HjxnHzzTcTGxtLRUUF+/btY/Pmzd3mXKqsrOTw4cPcd9997tcGDRrEwYMHMZlMDBs2jKFDh+Lr68uzzz7bJAOmpqaG1NTU1s3Od0SCqsVikTlz5khJSYk7ifnDDz+UO+64Q4KDg7t04m5zdDCbzRIVFSWzZ8+Wzz77zK2Fy+VyJ71XVFTIxo0bJSYmpkdoAkhiYqK8/PLLUltbK5cjNTVVXnjhhS59nHybLhaLRebOnSuHDh2Sbdu2ybhx48TLy+uyn33yySclLS1NHA6HFBQUyPvvvy/h4eFdVpcrxerv7y8rV6685IaQhvPF4XCI0+m85LxpXEpLS2XRokUt1qRdD/jGP/64ceNk/fr1UlpaKjU1NbJ//3556qmnJCwsrNv9sBcXDw8PSUhIkNdee01SU1OlqKhIiouLJTs7Wz755BMpLi4Wu93uNsaDBw/KD37wAzGZTN1Wk4uL0WiUyMhImTdvnjz88MPy8MMPy6OPPionTpwQm80mqamp8vvf/77bGiPUXzQDAgLE399fPDw8LvuZqVOnSnp6utTV1YnNZpMdO3bIvHnz5Js1DbukLleK1WQyybhx49znxeXKtxljRUWF/PWvf/1Oba4UV4ctIlFXV8ehQ4f45S9/yYQJE1iwYAGDBg3i+9//PpGRkfzXf/1XR4XSoXh4eDBixAgWL17M9ddfT25uLh9//DHHjh3j2LFjGI1G3njjDQYPHozFYsFqtTJgwACeeOIJ0tLS2LFjR8vv9+wiOJ1O8vLy+PTTT93daYPBQHBwMPfddx9hYWHExsYSFhZGfn6+xtG2Dzab7Vt/Z4PBwCuvvEKvXr0wGo1s3LiRFStWsHHjxgaD6VaIiPvut8vtn8vlIicnB5H6O+jKy8sZPXo0AwcOdH/m4oVrmkuHLjtWU1PDiRMnKC4u5vrrr+faa68lNjaWm2++mQEDBnD27NmODKfdMRqNzJo1iwULFjBu3DiqqqrYuXMn7733HufPn6empoaf/exn7juFjh07RklJCVFRUfTv359nnnmGoUOHsmXLFsrKynC5XFRWVnbLxSWcTielpaXu/z08PBgwYAChoaEYjUb3akw9EYvFwm233UZCQoL7tZMnT3Lw4MFueSxcCZvNRmVlJdnZ2ezcuZOdO3ciImRlZeHv7090dPQlxtgaOvRICwoKIiYmhpCQEHe+nsFgcN8a150wm82MGjWKhx56iBtvvJGzZ8+ya9cuNmzYwJkzZwgICGDatGncf//9eHl5sXv3bjZs2EBubi5xcXGMGTOG2NhYHnjgAfcge01NDQcOHGDjxo1a7167M2LECK677jqCg4NJT08nNTW1xy49ZrVaufXWW905i8eOHePIkSPdtvUM9a3EsrIyTp8+TUBAAJWVlSQnJ5OSkkJycjJ79+69ZPWtAQMGUFlZ6f7faDTSu3dvLBZLi1KY2t0YGxYCCA8PZ+TIkUycOJEBAwYwaNAglFLU1taSkZFBcnJye4fSYZhMJmJiYnj44YeZPHkyZ86cYcWKFezatYvCwkIGDhzIqFGj+NGPfoTBYGD37t0sWbKEPXv2UFZWRmBgIDt27GDixInMmjWLe+65B4PBQFpaGsXFxV3aGD08PIiPj6e4uJji4uLLLiVmMpm4//77CQsLAyA9PZ1t27Z1+XuAW4LVaiUqKoqhQ4ciIhQVFfHee++xb9++b70TpKvjdDrJzc1l5cqVREREkJ+fz969ezl69CiFhYXf+f2G2WsfHx/q6uqaPdzQbsaolMLDwwN/f3+GDBnCwoULmTZtGqGhoSilcLlcVFdXU1BQQHp6ercaJwkNDWX+/PnMmzePmpoali5dyo4dOwgODmbatGlMmTKFESNGYLfb+d3vfsdHH31EVlaWO2+xoKCAgoICduzYwdGjR5k5cyYmk4k1a9bwxRdfaLx3LcdgMBAdHc0rr7zC+vXrWb16Nfn5+e581oY1PGNiYrj99tsJCgrCZrORlpbGoUOHNI6+4/Hy8iI2Npa77rqLESNGUFlZyZo1a1i/fj2ZmZlah9fuVFZW8uKLL17VZxsWthUR911UoaGhBAQEUFJS0vyc6faabQwICJDx48fLSy+9JBkZGU2m3VNSUuSvf/2rjB07tkvPNl4uzhEjRkheXp44HA6prq6Wo0ePysmTJ92zz3a7XUpKSmTx4sViNBpbPNPflTQBxM/PT/75z3+Ky+WSlJQUWbRokSQlJUl4eLj4+vrKDTfcIAsWLJDDhw+Lw+EQEZGNGzfK3LlzxWw2d2lNmnv+WK1Wuf3222Xz5s3icDjEbrfLk08+ecU0nq6qS2uO/YtLRESEvP7661JUVORO6bHb7bJs2TIJCAhotiZtvhOhoaFyww03yFtvvSXnz59vMo3+5ZdfyuOPPy5xcXFXtbhmV/xhhw0bJsePH78k78rhcMiRI0fkrbfekoceekji4uKumJbRHTWB+hSMsWPHyu7du8Vut4vL5ZL8/Hw5ePCgbN++XcrLy8XlckkDaWlpsmDBArFYLF1ek6s9fxpKr1695Ne//rU4HA6pq6uTU6dONft86Qq6tJUxAhIbGyu//vWvpaCg4JLzrk+fPlfU7opxtcVOKKUkKChIli5dKvv27ZOcnByprq52J2LW1dXJ3r175YEHHpA+ffqIl5dXi35krX/Eq9XEbDZLXFycPProo/Lmm2/K73//exk3bpxER0eLv7+/WCyWVh/kXU2Ti7X585//LM8++6xkZmaKy+Vyt6IvprS0VCZOnNhsU+ysmnyXLheX6Oho+eEPfyhbtmyRqqoqOXDggAwZMqTHHSvNLQaDQfr06SOPPvqoFBYWuo1x2rRp4uPj0yxNWjXGGBgYyLBhw5gxYwaDBw9m5MiReHl5YbFY3OOIOTk5rF27lqVLl5KWlkZ5eXm3v0faZrORkZHBhx9+yIYNG7Db7ZSUlFBXV9ell4VqC2w2Gy+//DIulwulFJGRkQQEBLhX6Yb6saWf//znHDt2rEuvFtNS7rjjDu6++24SExMpLCxk1apV3S6VrT1wuVxkZ2ezadMmevXqxT333MOePXs4depUs2emW2WMERERTJ48mTlz5hASEoKXlxf5+fmcOXOGqqoqMjIy2LdvH8eOHePrr7/uUQ8zalhz8Wpm0HoamZmZGAwGPvzwQ7y8vLBarQQHB7sfgmWz2di0aVOPnIUODw+nX79+REVFUVFRwfbt29m9e3e3T/JvKxwOBxcuXGDlypUcOXKE8+fPk5ub22zvaZUxVldXk5qayqeffup+LS8vj7Nnz1JdXc358+c5ceJEj7zq63w7LpeLlJQUrcPodIwYMYKEhATMZjMnT55k3bp13XplofagtraWM2fOtEq3Vhljeno6y5cvZ/ny5a3ZjI6OzjfEx8czePBgKioq2LlzJ9u3b6egoEDrsHocPfMeKx2dTsr+/fvJzMxk9+7drF69mry8PK1D6pGob2aGru7D9StVaIaIXP5ZrBqia9IUXZPLo+vSlM6qSXNbjIVARuvDaRF9Nar3u9A1aYquyeXRdWlKp9SkWS3GlqCUCgLeBqZSL8JiEVnZrpV2cpRSjwPfA4YC74nI9zQNqJOglNoG3AA0TCFmiUjbLJfSBVFKWYDXgSlAEJBK/fnz6bd+sZvTEZ7SEWOMfwVsQDiQBKxXSh0VkZ71mMBLyQZ+B9wCeGocS2fjcRFZqnUQnQQP4DwwAcgEpgMfKKWGiki6loFpTLt7iqGtNnQ5lFLewF3AcyJSKSK7gDXA/e1Zb2dHRFaLyL+BIq1j0em8iEiViPxGRNJFxCUi64A04DqtY9OKjvKUdjVGYADgEJGL0/aPAoPbuV6drsuLSqlCpdRupdRNWgfTmVBKhVN/TvXk3laHeEp7G6MP0HjRuDLAt53r1ema/ByIBaKAt4C1Sqk4bUPqHCilTMA/gXdEpPs+ZPu76RBPaW9jrAT8Gr3mB1S0c706XRAR2S8iFSJSJyLvALupH1fr0SilDMAK6sfVHtc4HK3pEE9pb2M8C3gopfpf9NowenZXQOfqEaDT5d51JKr+oclvUz/RcJeI2DUOSWs6xFPa1RhFpApYDTyvlPJWSo0FZlF/9euxKKU8lFJWwAgYlVJWpVSPvgtJKRWglLqlQQul1H3AeOAzrWPTmDeAa4DbRaRG62C0pqM8pb1bjAA/oj4lJR94D3i0h6fqAPwKqAGeBRZ88/evNI1Ie0zUpzAVUJ+b9gRwR6NB9h6FUqov8DD1KSm5SqnKb8p9GoemNe3uKe2e4K2jo6PT1eiIFqOOjo5Ol0I3Rh0dHZ1G6Maoo6Oj0wjdGHV0dHQa0awUkc66dpqW6Jo0Rdfk8ui6NKWzaqK3GHV0dHQaoRujjo6OTiN0Y9TR0dFpRIffhhYcHEx8fDz9+vUDIC0tjeTkZIqLizs6FM3w9fVl+PDhXHPNNRQXF7N582ZKS0txuVxah6aj02Xx9/fnmmuuYdiwYdhsNj777DNycnJatK0ONUYvLy+mTJnC9OnTmTBhAiLC9u3bWbduHR999FFHhqIZBoOBPn368OMf/5jZs2dz7tw50tPTOXbsGLW1tVqHp9OJsFqtBAQE4Ofnh5eXF1arFYCamhouXLhASUmJfjH9hsDAQJKSknjggQe4//77qays5IknnmDTpk3k5eU1XycRuepC/WonLSomk0kmTJggGRkZYrPZxOFwiN1ul9raWjl79qx4eHh85zaaE2tHlebq4OPjI7fddpvk5+eL0+kUh8Mhc+fOFX9//xbpqvX+t/Vx0hZF6/1vrS5KKfHy8pIRI0bI4sWLZfXq1XL06FGpqKiQuro6OXbsmCxatEiCg4O7vC5t8XtbLBa566675P3335fz58+Lw+EQm80mR44ckV/96lfi7+8v38x+X7UmHbITJpNJbrjhBsnMzBS73S5Op1MqKiqksLDQXcaMGSMWi6Xb/7AeHh6SlJQkGzZs0I2xFUUpJQaDwV0uPvC13v/W6hIYGChPP/20HDlyRGprayU7O1vWrFkja9askcLCQnE4HJKdnS2//vWvr3jC95RjxWAwyLx58+SDDz6Q8+fPi91uF4fDIU6nU+x2u9jtdklKShKz2dx5jNHDw0NCQ0Nl2rRpkpaW5g7Y6XTKuXPnZMmSJfLLX/7S7fCLFy8WT0/Pbv/DxsTEyFtvvaUbYwtOgj59+sjAgQNl8eLFcuzYMcnNzZUDBw7I4sWLO7UmzdFl2bJlkpaWJjt37pQnn3xSgoKCRCklSikZMGCA7N27V2pra2Xfvn3yve99r8cdK0ajUUJCQmTixImSkpIiZWVlUldXJ3a7XQoKCuT48eNy5syZVhlju44x3nDDDTz55JNMmTIFHx+fS96Ljo6mb9++JCcnc/DgQUaOHMmgQYPw8OjeyxKazWbCwsIYOLDHPhW02URERDBq1CgmTZrEzJkz8fPzw2w2Y7FYMBgMGI1GBg0apHWYbcKYMWMYPnw4hYWFrFy5kmXLllFXV9dgIqSkpLBy5Uq8vLwYMGAAU6dOZfny5doG3YH06dOHiRMnMmvWLMaPH4+/v/8l73/99df861//ws/Pj9///vctrqfdXKhXr16MHDmSsWPHuoM/fPgw+/fvZ/bs2Xh7e5Obm8tXX32Fy+Vi1KhRmM3m9gqn02A0GvH19SUiIsL92sCBAzl58iQ2m42amh6/FqmbBx54gKSkJAYMGED//v0JCAggMDCQ2tpavvjiC/dF9dy5cxQVdY8HLnp6elJSUsLx48c5evRokwk5l8vF7t27mTp1KrGxsfj6+uLl5UV1dbVGEXccgwcP5t5772X+/PkEBQXh5+eHwWC4ZGLF6XRSU1OD2WzGYGh5NmK7GGNYWBh33303c+fOJTg4GIfDwcmTJ3nxxRcxm82YzWZyc3PZvn07R48epaSkBIChQ4fSt29fzp49i81ma4/QNMdut5OXl8fRo0eJi6t/ztOdd95JYWEhFRUVZGZmahyh9nh7ezNjxgwWLlxI3759MRqNlJaWsmnTJnJycqiqquLw4cMUFBSQnZ1NaWmpu0XV1UlJSeGVV14hNzeX5OTky36muLiYmpoarFYrkZGRjBgxgh07dnRwpB3L4MGDueeee5g5cyZ9+vTBbrdTVFSE1WrF09MTpRROpxObzeb2jtYcE21ujH5+ftxwww3cdttt7nyi0tJSNm/ezPr164mJiSE3N5fz589z/vx56urqUEohIvTr14+JEyeSl5dHQUFBW4fWKXA4HFy4cIHVq1dz5513ApCYmEhcXFyT4YaeypgxY3jooYcICgriq6++IjU1lYyMDPbs2UNeXp77mOouZngxWVlZZGdn43Q6r5hiUlFRQUlJCbW1tQQHBzNs2LBubYx9+vTh3nvvZebMmfTv3x+bzUZ6ejqrV68mMTGRKVOmYDabSUlJ4dChQ2RmZhIaGtp5jFEpxfDhw7nzzjsZOHAgxcXFnD17lvT0dDZt2kRtbS2nT5/m9OnLP/3RZDIxdepUPv30025rjAClpaVs2bKF+ucc6UB9zl54eDghISGMHTuWsLAwNm7cyAcffMDx48ex23vGM6AcDsd3fqa0tJTc3FzKy8sxmUyEhIR0QGTaEBoayqxZs7j//vuJiorCZrORlZXFZ599xn//93+zcOFCRowYgd1uZ/369Xz88cekpaURExNDZWVlixsbbWqMXl5ePPbYY0yePBmXy8XatWt54403SElJwWKxXNU2QkNDe8RYI+BuKTf83VOxWq0kJCTw0EMPcfPNN/PUU0+xb98+Dh06RFFRkZ7E3Ain03lJi7I7tpyh/pyYM2cOv/jFLwgJCaG6upq0tDS2bDsxWOwAACAASURBVNnC66+/johgs9n46quv2LVrF5988glnz9Y/Imjbtm2cPHmSUaNGtajuNjXGYcOGMWnSJPz8/PjTn/7EkiVLSElJuarv9kRjcLlc7oO6ux7cV8OwYcN47LHHmDNnDqtWrWLbtm36JNS3YDKZsFgsmEwmbDYblZWVTT5z8UW3q6KU4umnnyYoKAiATZs2sWTJEjZu3Oj+zIEDB9i6dStFRUWXtLarqqpaNU/RZsZoMBj4wx/+gI+PD0eOHGHnzp1XbYrwH2PsiQbZ07n22mtZsGABZWVlfPjhh7opfgcRERHEx8cTFhbG8ePH+eKLL9zvBQUFMWDAAIYMGUJlZSW7d+/m/PnzGkbbMjw8PBgyZAi+vr4YjUbGjRvH/v37m5h9amrqFbehlGrxzHSbGKPZbGbSpEkkJiZiNBrZu3dvs3+MhsTKU6dO9YjUA53/UFhYSHJyMv369ePFF1+kvLyc3bt395hxxavFbDbTq1cvhg8fTlhYGAAxMTG89NJLHDt2jKFDhxIYGEhkZCQGg4GUlBSio6N56aWXNI68eXh6epKYmMjf/vY3AgMDgUt7V9+FwWDA29sbk8nU4mGYNjFGpRS+vr54enpSWlrKjh07rirtxGAwEBkZyX333YeIcPDgQV5//XXy8vLaIqxOz8U5WAMHDuzWg+iNGThwINnZ2VRWVrJz505+/vOfc9dddzFnzhxeeuklHnzwQU6fPt1t07auhNlsxs/Pj+joaPr168eAAQNISEgA6g0jPj4ePz8/wsPDgfrUpqSkJIKCgvD29mbr1q0cPXqU/Px8cnJyOHfunJa70yKioqL48Y9/TP/+/Vs0JODv78/EiRMZPHhwi2No0660UoqamhrOnz9PeXn5d34+NjaW6dOnM3/+fIqLi1m1ahWnTp3qMavMXPyDX3vttSQkJHD06FHKyso0jKr9UErh7+/PY489xoABA/j3v//Nnj17KCgoYOfOnYgI8+fPZ+jQoYSGhpKamtpjjNFsNjNjxgySkpLo06cPwcHBBAYGEhwcTHBwMHV1dTgcDqKjo913h+Xm5nLo0CG+/PJL96RDSkqKO9ezpqaGuro6LXerRVgsFnr37u2ehF23bt1VL0vo6enJoEGDWLhwId7e3hQVFWGz2Zptrm2ex5ienk5paSlOp/OKn/H39yc+Pp6JEycyffp0DAYDb775JuvWraO6urrLDxpfDQ6HgxMnTnDNNdcA9eNGSUlJHDhwgCNHjmgcXftgtVpZuHAhDz74oHtsyOVy4XK5cDgc7vGgvLw8Kisrv/UY6k5YLBamT5/Oww8/TEJCgruBUVRURHJyMqdOnaKwsJDi4mIWLFhAr169sNvtHD58mL///e8cOHCArKwsrXej3dizZ893NrTgP13wu+66ixtuuIGamhpWr15NcXFxs4+lNjfG3Nzcbx0bCg8PZ9SoUUydOpXRo0djNptZuXIlL774YluH0qmx2Wx8/vnnbmME6N+/P/379++2xujp6cmiRYvo1asXGzduxG63ExcXR79+/QgODubuu++mrq6OjRs3kpaW1mNai15eXixevJihQ4dy6tQpDhw4wPnz50lNTXXfBZafn4+npye33norAQEBpKam8vnnn7Nhw4Yu2Sq8GhomYsPDw781hc9gMNC7d28GDBjAjBkzuPvuuxER9u/fz8svv0x+fr426zFaLBa599573cshjR07ViwWixiNRjEajWK1WsXf31+Cg4PlmWeekRMnTkh5ebkcOXJEnnnmmR63OgggVqtVfvCDH0htba171aHt27fLggULxGAwdEtNPD095Sc/+Ynk5eVJdXW1ewm6kpISycjIkIqKCjl06JAMGzZMjEZji1fh0Xr/m6tLWFiYiIikpqbK2LFjm6wwZTAYJCQkRF577TUpKiqS9PR0eeqppyQqKqrL63K5OBMSEmTTpk2XrMY1ZcoUCQgIEF9fX7FarWKxWCQgIEACAgIkKipK/vKXv0h6errU1tZKXl6ebNmyReLi4lqsSZu1GBs2GBISwpNPPsnHH3/MsWPHUEoxYsQI5syZQ2JiIpGRkeTn53PgwAE2b97MunXr2iqELkVtbS1vv/02Tz/9NHFxcSilGD9+PBcuXOCLL77oll2jmpoaXn31VZRSJCYmMnr0aPcCI3a7nbVr1/Kb3/yG1NTUHtONbkBEKC8vJzk52X2bLNS3hsLCwli6dCm33nordXV1/OIXv2DNmjUtXra/s5OTk8Prr79OUlISAQEBAPzhD3+gvLyc9PR09u/fT2FhIY899hhKKUwmE4MGDcJsNrNv3z7++c9/snXr1m9N5fku1DeufXUfvsIzYC0WC3fccQf/+Mc/3D9oWVmZexKhd+/e7tcPHTrEvn37WLFiBYcOHaI59Us3ey6uUopVq1Zx6623YjabUUqxcuVKnn322as2xq6qyZXyVZtzPFyJzqgJXFmX4OBg9uzZQ//+/XnvvfewWCzk5OSQnp6O2Wxm4cKFxMfHc+jQIZ555hkOHTpEVVVVs+vvjLpcSROz2czYsWP54IMP8Pf3x2g0Nm5puvMUXS4Xhw8fZuPGjaxfv559+/Zddf1X1KQtugIGg0FiY2PlwIEDUldXJw6Hw/3oApvNJna7XfLz8+Wtt96SqKgo8fb2vqpHGTQuWjf7m6PJ1ZY77rhDSkpK3N2GFStWNKuLpPX+t4cmrS1a739zdVFKSVBQkLz00kuSm5srFRUVUlZWJiUlJZKWlibvvfee3H333RIQENCi86Yz6/Jt8ZpMJhk8eLC88MILkpWV5faUi0tubq6sXbtWxo0b1yJ9rhRXm7QYod7hb7jhBubOncucOXPcCaiFhYWsX7+eTz75hKNHj5KZmdnipEvpQle8q2XgwIG89957DBw4EKvV2mNajO1JZ9QEvluX6OhowsPDMRqN7tccDgdlZWUUFha2Oo2rM+ryXZqYTCZCQ0OZPXs2M2fOJCkpCS8vL9LT01m1ahXp6ens27ePzMxMampqaI6fwZU1aTNjhPpZxwEDBjBo0CC8vb2B+nsWk5OTOXv27FVNuX8bXfGH/S58fHyYNGkSQUFBeHh4kJqayv79+6/67p/uqElr6YyagK7L5bhaTXr37k18fDy9evXCYrFQUlLCiRMnqKioID8/v9mG2ECHGGN705V/2PZC16QpnVET0HW5HJ1Vk5av/a2jo6PTTdGNUUdHR6cRujHq6OjoNKK5Cd6FQEYL67IAg4ESIK0F3+/bwnrbm5Zq0g/wBYyAHcj9ZlvNQdekKZ1VE2iZLmagD+BDfYpJCdCSJ6Z1Vl1aoomVek28AAdwAShtQd1X1KRZky+tQSn1OeAJZIjIgg6ptBOjlBoMpIhInVIqAdgGzBCRQ9pGph26Jk1RSm0A8oFHgABgE7BERF7TNDCNUEp5ACeBN4FXgQnAWuBaETnbVvV0SFdaKXUv9Y6+pSPq6wqIyAkRabj7vyHhNE7DkDRH1+Sy9AM+EJFaEckFPqO+59VTSQAigVdExCkiW4HdwP1tWUm7G6NSyg94Hvhpe9fV1VBKva6UqgZOAznABo1D0hxdkyb8GbhXKeWllIoCplFvjjr/QQFD2nKDHdFifAF4W0QudEBdXQoR+RH1Y2o3AquB7rl+VDPQNWnCDupbiOXUj6UdBP6taUTacob6oYX/UkqZlFJTqe9Oe7VlJe1qjEqpJGAK8Ep71tOV+aY7sAuIBh7VOp7OgK5JPUopA/Wtw9WANxACBAJ/0DIuLRERO3AHMIP6ybmngQ+ov2i0GW2+UG0jbgJigMxvVlPxAYxKqUEiMryd6+5qeKCPpzWmp2sSRP3s6/99M/Zap5RaBvwOeEbTyDRERI5R30oEQCm1B3inLeto7670W9Qf2EnflDeB9cAt7Vxvp0YpFaaUulcp5aOUMiqlbgHm0YMnp3RNmiIihdSntj2qlPJQSgUADwDHtI1MW5RSiUop6zfjrj8DIoDlbVlHuxqjiFSLSG5DASqBWhEpaM96uwBCfRfxAvV5aS8BPxGRNZpGpS26JpfnTuBWoABIoT6/8ylNI9Ke+6mfmMsHJgM3X5TN0CZ0WB6jjo6OTldBvyVQR0dHpxG6Mero6Og0QjdGHR0dnUboxqijo6PTCN0YdXR0dBrRrATvzroMuZbomjRF1+Ty6Lo0pbNqorcYdXR0dBqhG6OOjo5OI3Rj1NHR0WlEey8icVUopfDz86N///6cPn2aqqqqFj8ntrNjNBoxmUwYjUZ8fX2JiYm54mdLSko4e/Zsl9ZCKYWnpyc+Pj6ICN7e3hiNRoKDg6murqa2trbJ/tlsNkpLS6mqqsLlcmkUedfBbDaTkJCA2Wzm5MmTLXrwvM6ltMoYDQYDRqMRpRQ2m63F2zGZTIwZM4a33nqLu+66i2PHjlFbW9ua0DodBoMBs9lMaGgokZGReHp6cv311/Piiy8CXNYcNm/ezKJFiyguLu6yBuHj40NiYiIjRozA5XIxePBgvL29uf322zlz5gxZWVlUVVVd8p2CggK2bNnCsWPHKC0txWazYbfbu6wGbYnBYEAphVIKEcFoNNK7d2/effddgoODmTFjBqdPn27V+djVMBgMWK1WrFYrHh4euFwu6urqqKioaPlGReSqC/9Zbl4AiYiIkMmTJ8u0adOk8XvNKV5eXrJ48WJZu3atDB8+XKxW62U/15xYO6pc7T7GxsbKD3/4Q/n4448lIyNDsrKyxOFwiNPpFKfTKQ6Ho0mx2+2yfPlyiYyMFKPR2CU1mTNnjnz++eficrm+szidzkv+TklJkb///e9yzz33SERExFUfT1rvf2uPlcZFKSUGg0EMBoNERkbKkCFDJCkpSRISEiQxMVGefPJJcblccurUKendu7cYDIYuo0trfKNBmz59+sizzz4rR44cEbvdLvn5+fLmm2+26lhpVYsxMjKSyZMn4+vry6efftri7ZhMJmJiYujbty8GQ/cc9nzkkUeYM2cOffr0uervKKVYsGAB//rXv9i5cyeVlZXtGGH7MGzYMJKSki55rbi4mBMnTuB0Ot2v1dTUkJWVRXx8PAEBAQwdOpTY2FhiY2MJDQ0lIyODnJycjg5fU4KCgrjuuuu45557iI6OBiAuLo7IyEgsFguHDx9m586dJCQkoJQiOzsbo9GIwWDoEa1rpRQPPPAAL7zwAqWlpWzcuJG///3vJCYmcuedd/LKK69w5syZFm27VcYYHx/P5MmTOXjwYIu3YTKZ6N27NwsWLGDNmjWkpKR0u240wOHDhxk3bhx9+vTB4XCQm5vL6tWr3V0igJiYGIYMGUJsbKzG0bYdS5YsYevWrfj7+3PhQv0iy9XV1eTk5Fxy8ooILpfLPfb66aef0r9/f8xmM3379mX06NHs27dPq93QhF69ejFlyhTmz5/vbjA4nU527dpFaGgo/fr1Y9CgQRiNRlwuF++//z55eXk4HA6NI29flFL4+vry0EMPsXjxYj788EPeffddzp49i5eXFzU1NcycObNVDYlWjzFardbWbILevXvz05/+FKvVSlpaWrcdG9m8eTNpaWmEhobicrmorq4mOTn5ks/4+flx11138ZOf/AR/f38AduzYwalTp6ipqdEi7FaTnZ1NUVERRqORurr6JfNcLhd2u/2K3xk9ejTh4eGYTCacTifp6ens3bu3o0LuFIwbN4577rmHmTNnopRi06ZNlJSU8OGHH5Kamkp4eDhz5sxhwYIFWCwWSkpK+PTTT90ad2cCAwO57bbb+PGPf8y7777Lm2++SVZWFrW1tURERBAbG0txcTH5+fktrqPFxhgQEECvXr3w9vZuceUAnp6eJCQkAPWtqu56tSssLKS8vBwPj3rJnU5nk5ZxbGyse8Ya6ltR586do6Sk5JJuZ1fC4XBc9W/q7e3NnXfeyX333Ye/vz9KKT799FPeffddTp8+3c6Rdi5CQ0OJi4sjLCyM7Oxs3nrrLfLy8jh58iRms5mBAwcSHh6O2WymsrKSjz/+mNzc3G7fhfbw8KBPnz7Mnz+fvLw8VqxYQVpaGk6nE29vbwYPHszo0aOx2+3fevH9znpa+sU+ffoQGxuLl1fLH85lNBrx8fEhKCgIEeHMmTNd1gC+CxGhrq7uilf06OhoRo8eTWJi4iWt8G3btnXLoYWL8fb2JiIigsGDB/PDH/6QUaNGYTQaOXLkCKtXr+aLL76gtLRU6zA7lOLiYvLy8qiurkYphdFo5PDhw+4MjlmzZjF8+HAqKirYv38/K1as6LaNiouJj49n1qxZ9O3bl9dee42vv/7a7RlJSUncfPPN9OvXjyNHjrSqnhYbY0xMDDExMSilWtzNCw4O5pprriE0NJSamhqKi4vd4209hZCQECIjIxk/fjw33XQTSUlJ7lZlZWUla9as6bbDCwAWi4WEhASmTZvGhAkTGDt2LFDf0ly3bh0HDhygrKxM4yg7nuTkZA4ePMiIESOIj4/ngQce4ODBgwwcOJAFCxYwfvx4TCYT+/btY+nSpezYsUPrkNsdX19fJk6cyD333MPBgwf5xz/+4TbFgIAAZsyYwa233orNZmPDhtY9jrzFxhgUFERwcDC1tbWkpqZe9feUUphMJsxmM6NHj2bhwoX4+vpy9OhRKisru60xGgwG9343dJWVUsyZM4eHH37Y3fpWSuF0OikvL+fIkSPY7fZuqwnUjzHfe++9PPXUU5dkJIgINpsNs9mMr68vtbW12Gy2btujaExDt/ncuXMkJCQwatQopkyZwrPPPuu+KWDHjh0sXbqUTz75RNtgO4ihQ4cyatQoamtrefXVV915ikopJk6cyNixYwkMDGTPnj189NFHraqr1Xe+VFZWcvz48Su+/81jU91/+/r6Mnr0aKZOncrEiRMZPHgw1dXVvPzyy926u9SrVy8mTJjAxIkT3WOqSinGjBkD/CfB22azcebMGV555RW2bNnSZSddrpakpCSGDRvWJE3LZDLx/PPP89BDD3Ho0CG2bdvGF198wcmTJ7v1haIBp9NJTU0NVVVVGAwGevXqxdtvv+3OYti5cycfffRRj5qUGjlyJPHx8Zw+ffqSrnJYWBiPPfYYI0aM4OTJk6xatYqioqJW1dVqYzSZTAQGBl7yWnx8PN7e3vTu3ZuhQ4cSFhaGxWJh3rx5GAwGbDYb586dc2eo22w2Dh8+3NpQOi0JCQm8/PLLjB49Gl9f32/9bG5uLp9//jnvvvtuB0WnLYcPH2bt2rVYrVYGDRpEUFDQJe/37duXvn37Mnv2bCorK/nb3/7GSy+9REFBQbc3yMrKSvLy8pq8vmfPHpYtW8Znn33WqpnXrsagQYMwm818/vnn7vFUX19f/v73vzNy5EgOHDjA8uXLW91ahFYYY8OsT9++fXn11Vf56U9/6n4vPj4eLy8vDAYDIkJNTQ2pqam8//77fP311xw6dIisrCxmzpzJCy+8gN1ub5K60t1oSLy9uAUNuDVqoHfv3kydOpX169f3iHGjtLQ0lixZwvvvv09UVBSBgYGEhYUxdepUbr/9dkJCQoD/9DYee+wxEhISWLhwYbfuYSil3BNSFx8z27dv52c/+xlff/11tx57bkxSUhIRERFkZmayd+9eTCYT48eP549//CMJCQkUFRXx/vvv8+9//7vJLaYtocXG+Nlnn1FVVcXMmTOJiYm55EdKTU0lJSWFs2fPkpmZSXJyMrW1tVRWVlJVVUVVVRWDBg1i8ODBWCwWMjIyuvXYUUZGBs8//zw33ngjnp6epKamNkkluPHGG7n55puJjY3Fw8MDk8mkUbQdi8vlora2lrq6OsrKytz7vn//fg4ePMi0adO47rrriIiIAOrTuyZMmMC8efP48MMPKSws1HgP2oc77riDBx54gFGjRl3yus1mo6KiokfkK15MTU0NNTU1DB8+nD/+8Y/ue8Sjo6OxWq1s3bqVI0eOUF5e3ib1tdgYi4qK2LlzJxcuXGjS/Wl4v6ioiLKyMkpKSpq8HxcXx4ABA6iurm7VnTNdgZqaGr766itycnIwGAyUlJQ0yTc7d+4c0dHR9OvXT6MotaVhsqXhAltWVsbatWs5efIk06dP595776V3795AffdpyJAhrF+/XsuQ2wU/Pz/3LW1jx47FYrGQlpZGdnY2Y8aMwcfHp9veNvtt5OTksGnTJkwmExEREVRWVrJt2zbuvvtukpOTWbduHSkpKW02vNKqMcYG82sJAQEBBAQEUFZWxq5du1oTRpegpqaGc+fOXfH9goKCbp+v2FwuXLhAVlYWVquVkSNHuo0R6LaJzLGxscyfP59x48bhdDrZtm2bu+s4evToyy7T1hMoLy9n8+bNFBQUEBMTQ1VVFRaLBaPRyPr16/nyyy8pLi5us/o0X4+xurqaEydOaB1Gu6CUwt/fn4qKim8dKggJCWHSpEn07du3A6PTFpPJ5O5pNAyxXI6GMerGuYwZGRnd7kLi5+fHzTffzG233YaHhwdbtmzh/fff59ChQyxatAiAzMzMHteNbiA9PZ309HSMRiMRERG8/PLLVFRU8K9//Yvc3Nw2vWB0ijZ5Q15fd0EphdlsJiQkhGnTpuHn53fZ7o9SCqvVyp133slPf/rTJqvQdFcMBgP9+vXj/vvvZ9asWcTHx1/2cw2L3EZGRl4yxFBXV8fx48eprq7uqJA7hOuvv54nn3ySkJAQVq5cyYsvvsj27duJjo7mwQcfRClFbm5uq2516w74+fkxadIk7rjjDpYsWdIu67dqboyenp7drqXk6+vL5MmT+cc//sGKFSvo27cvJpPJvcBowy1ewcHBzJ07l+eee474+HiMRiMOh4Pq6upunb8YHh7O22+/zW9/+1usVitnz55t8hmlFMHBwcyZM4cHH3yQxMRE93tnzpxh//79XXIZtiuhlGL8+PEEBQVx6tQptm3bxsmTJ/H392fSpEn4+fkB9bcK9nRjjIuL43//93/Jzs7m//7v/9rlXNG8Ky0iTVJYujr9+vXj//2//8c111wDwGOPPUZRUdEl3Wmz2cyECRMYPny4+7WzZ8+yefNmPv74Y/bs2dPhcXcESin+9Kc/MWLECCwWCyEhIYSFhZGTk+OeeLFarTz++OM89NBD9OvXD7PZ7E5srqio4KmnnmqTlIzOiMvlQkQwGAzExsYyd+5cnn/+ecrKyli2bBlvvvlm61am7uKEhoYydOhQrFYrzz//fLtpobkx+vj4kJSUxL/+9S+tQ2k35s+ff9nJgoYTvqioiL1797J06VJ27tzZbU/6Bnbu3Mntt9+OxWLhqaee4t577yU9Pd29335+flx33XXu58NA/Z0geXl5PPbYY+zatatbtpqCg4MxGAyUlZWxaNEioqOjSUhIoLKykieeeIJ169Z1q1Zyc/Hw8GD8+PE88sgjZGRk8MEHH7RfXe225auk4R7i7sSFCxf4wx/+wD333MO0adOwWCyXvG+z2cjPz6euro6jR4+yZs0ajhw5QmZmJhUVFd161lFE2LNnD5s2bWLixIn4+/sTExNDRESE++JhMBjcy9mJiLslvXv3brZu3dotTRHqJ1ZcLpe7F2EwGMjPz2ft2rVs3bq1R7cUARITE5k0aRJms5nnnnuuXRP8NTPGrKwssrOz3Uu2dyfKysrYtGkTOTk5FBcXM2PGDPz8/MjKyiIlJcV9r2fD0lLJyckUFxd36yT3izl37hx/+ctfOHv2rPs2L5fL5Z6Eyc3NBepnnnNzczl58iRfffUVFy5caLME3s6GiLB161YGDRrEzTffjMvlIjU1lb1797Jy5cpum8jeHEaOHMmQIUO4cOECe/bsadeULc2M8dSpU6xatYpTp051uyXrHQ4HeXl5FBUVUVdXx4ULF9zGmJqaSmpqKsnJyT22BVBRUcGOHTvIyckhLi7uEmN0uVzu+38zMjLIz8+noKDgsjcJdDe+/vprli1bxrlz53A6nWRkZHDixAmOHj2qdWidgr59+2I2m/nqq6/a/XZQ1Zxum1JK0z6eiHS6WRpdk6bomlweXZemNEeTxx9/nJiYGNauXcv27dvbpP4raaIbYyvRNWmKrsnl0XVpSmfVRPM8Rh0dHZ3ORnPHGAuBjPYI5CrorFnguiZN0TW5PLouTemcmohIuxbgceAgUAcsb+/6ukLRNflWbe4FTgFVQCpwo9YxaV10TZro0e7nT0fMSmcDvwNuATw7oL6ugK7JZVBK3Qz8AbgHOABEaBuR9uiaXJZ2P3/a3RhFZDWAUmoE0P2SFluArskV+S3wvIg05G9laRlMJ0HXpBEdcf7oky86nQKllBEYAYQqpVKUUheUUv+nlOqxLWpdE+3QjVGnsxAOmIA5wI1AEnAt8Cstg9IYXRON0I1Rp7PQsHbUX0QkR0QKgZeB6RrGpDW6JhqhG6NOp0BESoALwMUJv913NY2rQNdEO9rdGJVSHkopK2AEjEopq1JK81V9tETX5IosA55QSoUppQKBp4B1GsekNbomjeiQ86cDco5+Q/1V7uLyG61zoTTOw9I1ubwuJuB1oBTIBV4DrFrHpWvSuUpHnD/NuldaR0dHpyegjzHq6OjoNEI3Rh0dHZ1G6Maoo6Oj0wjdGHV0dHQaoRujjo6OTiOalfvTWVfb1RJdk6bomlweXZemdFZN9KRinU5LeHg4CxcuZMyYMZw5c4Znn31W65B0egi6Mep0SmbPns2NN97ofvZ0wyNVdSAgIICRI0cyYsQIAN5++233kxV12gbdGHU6FV5eXkydOpUf/OAHXHvttZjNZr788ku++OILrUPrFISGhnLbbbcxceJE4uLiyMzMxGw2ax1Wt0NTY/T29iY8PJywsDAATp48SUVFBT3hbhyDwYCPjw99+vTh1KlTOJ1OrUPSFKUUPj4+DBs2jN/+9rdcc801FBQUcOLECTZs2MD69eu1DlFzAgICmDZtGosWLcLlcvHVV1+xe/fudn/GptnKxQAAHWhJREFUcleiwVNCQ0PJyckhMzOzZRtq5j2Kje9PbHExmUwybtw4effdd8XpdIrT6ZSbbrpJTCbTFb+j9T2abaWJUkp8fHxk9uzZsnfvXgkKCmqxjlrvf1tp4unpKSNGjJCNGzeKw+EQp9Mpr776qsyePVvi4uK6vCatPX88PDxk3rx5kp2dLUuWLJEbbrjhW8+VrqJLW3oKIKNGjZJly5ZJVVWV/PnPfxZPT88WaaLZTowZM0befvttqa6uFofDIXa7XSZMmNAjjNHDw0MSEhLE5XJJTk6OJCQkNPsg726a+Pr6ym233SYOh8NdfvnLXzbbFDurJq05f4xGoyQkJEhFRYVs2rRJEhISevSxcqVitVrlhz/8oRQWForT6ZSamhr5/ve/LxaLpdmaaNaVDgoKIjQ0FJPJpFUInYLw8HB69epFZmYmdrtd63A0wd/fn5tuuolXX30VpeqzJxYtWsS6desoLi7WODptUUoRGhrKG2+8wcmTJ3nooYfIyurxj31pgtFo5KabbuKWW24hMDAQl8tFXl4e+/btw+FwNH+DWrh7UlKSvPvuu1JU9P/bu/eoqMv8gePv79wYYEBnUGZEYFAoJRFJEwzNYF3TspuZbZ7N1VyzrW23Pdtav/Zs26m1LG3Xc7Z+lbtd7LIWWqQmm6WpKYlcxCAhDRBjEEEYAYcB5sbz+0OZ33rb0hhmZL6vc+ZU4uH5zNP3+Xyf73P7WoXNZhN79+4VaWlpQqfTidPrmgb0Ha+3x9grKytLhIWFBW0vYPz48eLVV18VNptNbNmyRaSmporQ0ND/ei1cbnVyqddKXFycWLFihWhvbxdjxowRKpXqktudv79/X9XJ+T4pKSkiJydHdHV1CY/HI1pbW8XcuXO/t74uFFe/9xjHjBnD73//e7Kzs4mIiMBisfCvf/2LyspKXC5Xb2UFBSEEkiQREhKCQhGcm5Dmz5/PvHnzSE5Opq6ujpUrV3Lw4EGcTqe/Q/M7hUKB0Whk5syZlJeXU11dfWm9nwHOaDSyaNEiJk6ciEajwW63U1RUxBdffHHJ9dXvrTE7O5v09HSio6Ox2WyUlZWRl5eH0+kMqqTYSwhBfHw8ISEh/g6l3yUlJfGTn/yEa6+9ltDQUPbu3Ut1dbWcFE+Lj4/n+uuvR6/X88477+BwOPwdUkB68MEHmTlzJiaTCY/HQ0NDAxs2bMBqtV7y7+zXHqNWq2XGjBmYTCYAampq+Pzzzzl8+HB/hhEQenp6cDqdaDQaEhISgi4xajQaxo0bR2JiIpGRkVgsFg4dOkR7e/sZf2/48OEYjUY0Gg1ut5uSkhI/Rdz/4uPjSU9Pp7GxkU8++cTf4QSs2bNnM3LkSNRqNU1NTRQWFrJ169YftQSu3xJjSEgI11xzDZMnT0an03Hs2DEKCgrYtm1bf4UQMHp6enA4HBw/fpzY2FiioqKCbhJKq9Vy5ZVXEhUVBYDL5aKpqcm7Js9oNKJQKMjOziYjI4OoqCg6Ozt57bXXKC0tHfC9Sq1Wi9lsJiYmhh07dmCxWLw/Cw8PB8DpdAbthF2vhIQEhg8f7n2ELi8vZ8OGDVRXV/+o39sviVGpVDJixAheeukldDodLpeLL7/8ks2bN3Po0KH+CCGg9PT00NbWRllZGbGxsf4Oxy+USiVXXHGFNzH2DqsAqFQqfvaznxEZGcl1113HhAkTGDx4MEII7r33XsaNG8ehQ4cGdFJISEjg6quvRq1W8+qrrwKnkmV4eDhpaWlIkkR9fT21tbVB+YgtSRIREREsXbqUsLAwPB4PBw8eZMOGDWze/OPfFdYvY4whISFMmzaNq666CqVSyf79+8nJyWH37t39UbwsAE2cOJFJkyYxdOhQOjo6OHr0qLe3mJKSQkZGBr/73e+YNm0a4eHhnDhxgpqaGgBee+01Ro0aNaC3wl1zzTWMHDmSqqoq6uvrkSSJu+++m9LSUvLy8ti8eTMvv/wyt9xyi79D9QudTsfzzz/P7bffjlqt5vDhw7z44ou8++67fXLD9HmP0WAwMGvWLJYtW4YkSRQWFvLoo4+yb98+uru7fV38ZUGv16NSBc+2dZ1Oh9lsJiQkhPb2dvbv38+mTZuwWCyMGjWKRx55hNmzZ9Pd3U1+fj7//ve/2b59O3FxcXzwwQcMGjTI31/B5zIzM9FoNLzxxhs4nU60Wi1/+tOfePLJJ711MX/+fP74xz9itVqDai+5Tqdj0qRJ3HrrrZhMJiRJYuPGjZSUlNDR0dEnZfi0Nep0OiZPnsxLL71EeHg4kiRhsVhobGykq6vLl0VfVqZNmxYUjb1XRkYGf/7znzEajWzdupU333yTLVu2YDabeeqpp7jllltwu93s3LmTf/7znxw4cIAJEybwxBNP0NHRwe7du6mrqxuw44xms5moqCgkScLj8aBSqUhJSWHnzp18/PHHtLa20tDQgFarJSwsjCVLlgRVYhwyZAhz587FaDQiSRKHDx9m+/btl74v+jx8+ig9atQobrnlFu9gsRCC9957Tz4iCXC73ezbtw8hBBqNJmjWMUqSRHJyMpGRkSgUCnbs2MG2bdswmUwsW7aM6dOnExISwpNPPslTTz1FbW0tc+bM4bnnnsNsNvPRRx/x7LPP9lnPIBDp9XpCQ0Pp6OigsbERj8dDXV0dL774Im1tbfT09OB2u/nmm28oKCjwHj8WDBITE7n77ru59dZbvW1m6dKlFBUV0dnZ2Wfl+KzHqNfrGTt2LBMnTvSuTywrK6O0tBS73e6rYi8bHo+HI0eOAKeSxaBBgwgJCQmKgfQhQ4agUCiorq6mpqYGl8tFYmIi6enpDBo0iA8++ICPP/4Ym83G7NmzmTNnDgaDgY0bN7J69WpvvQ1UKpUKSZKw2+20tLQghKC1tZX29vYzlqC0tbVRX1+PTqfzY7T9JywsjNTUVGbMmMHQoUPxeDzk5+dTUFDgvWH0FZ8lxszMTG688UbMZjNCCJqamlizZg3Hjx8P+iO24FTv2WazAacaQkREBBqNJigSY++kybFjx+jo6ECr1WIwGLxPFlVVVUyePBm9Xk9WVhYej4e3336bvLw8CgsL/Rl6v3K73d5x+PNNKGg0GsLCwgb07Hyv8PBwMjMzuemmm0hJScHtdlNeXs4//vEPWltb+zQpgg8T4/Tp07npppvQarV0d3dTUFDAG2+8IU+4nNbT00NLSwudnZ2o1WqMRiM6nc6bLAeyEydOIIQgMjKSyMhIIiIiCAsL8x4gkZaWxsKFCwkJCaGqqoqcnBzefffdPn1UCnQ/ZGjFZDKRnJxMQ0NDP0TkXyaTiTlz5jB79mz0ej12u53169fz/vvv+2THnE8GtsLCwoiMjPRe7E6nk5ycnKC6sL+Py+Vi9+7dVFVVoVAomDJlCnFxcf4Oy+eEEOTn5+N0Ohk3bhxTp05lypQppKamEhYWBsCNN97IsGHD6Onp4fPPP+ezzz7jxIkTQXNTdbvduN1uFArFBZckhYeHk5qaytixY/tk3V6gGzZsGGazGb1ejxACu93Ozp07fbaN2Cc9xkceeYQpU6agUCi8A8XFxcW+KOqyJoTgiy++IDY2lldeeYV9+/b5O6R+4XQ6sdvtRERE8NBDDwF4L/Def9rtdrZv386WLVs4ePCg32L1B4vFQl1dHWazmezsbDZs2HBGAlCr1dx1113ccccd2O12Xn75ZT9G2z8iIiK822ZtNht79uzh22+/9Vl5fZ4Yx4wZQ1ZWFsOGDcPtdlNfX8/bb78tz0T/F0IInE5n0Iy9lpaWct9997FgwQKmTp1KTEwMAF1dXaxbt44DBw7w9ddfU1xcfM7e6WBgtVrJzc1l0aJFLFu2jEGDBpGbm8vIkSNJTk5m8eLFREdHs3nzZlatWhUUZ1YuWbLEO/ve2NjImjVrfPpKhz5PjBEREQwZMoTQ0FDsdjv79+/n+eefD5rHoEuh0+m47rrrOH78eJ+uxQpkO3bsoKCgALVa7R1PE0LgcDjweDzex8lglZ+fj81mY/78+axcuZIVK1agVCrp7u5my5YtLF++nKKioqAYk05KSvJugigrK+P111/ns88+82mZfZ4YlUolCoUCSZIQQuB2u+WxxQuQJIlp06YhSRLNzc1BtYzJ5XIFxWzqpXK5XFRUVPDXv/6Vd955B6VSCZxa5tXS0kJLS8uAXsv5n1JSUoiKikKlUuFwODh58qTPV2/0eWJsamqitraW2NhY7yyj7MKsVis6nY62trYBu5NDdmm6urqoq6sLmqeICzlw4ABWqxW73U5jY2O/HFPok8S4bt06Kisr0Wg0VFZW9nURA4YQgrfffhuTySQf0CqTXUB1dTXvv/8+hYWFVFRUUFVV5fMypYuZ7j79Dg6/EUIEXBdUrpNzyXVyfnK9nCtQ6yQ4NujKZDLZRZATo0wmk53lYscYW4DvfBHID2D2U7nfR66Tc8l1cn5yvZwrIOvkosYYL4UkScnA/wITgGZgqRDiI58WepmQJOkK4GvgAyHEPf6Ox58kSXoIWAiMBd4TQiz0a0ABQm4/5+qPa8Wnj9KSJKmAjcBmwAAsAd6VJOlKX5Z7GflfQN4reUoDsAx4w9+BBAq5/VyQz68VX48xjgZigFVCCI8QYjvwJTDfx+UGPEmS7gbagM/9HUsgEELkCiE2AJf+MuCBR24/59Ef14o/Jl8kIMUP5QYMSZIigaeB3/s7FtllJ+jbT3/wdWI8BBwHlkqSpJYk6QbgeiDMx+UGur8Arwsh6v0diCygye3HT3yaGIUQLuB2YBbQCDwCrAOCNiFIkpQG/BRY5e9YZIFNbj/+4/N3dgohyjl1lwNAkqQ9wFu+LjeAZQEJQN3pveQ6QClJ0lVCiPF+jEsWgOT24x8+T4ySJKUC33Kqd/ogMAxY4+tyA9g/gPf/47//wKlE+YBfogkQp2dgVYCSUzcKLeAWQgTv2WPI7ed8+uNa6Y/Jl/nAMU6NlUwDpgshBv4bny5ACNEphGjs/QAdQLcQotnfsfnZn4Au4H+Ae07/+5/8GlFgkNvPuXx+rfh8gbdMJpNdbuS90jKZTHYWOTHKZDLZWeTEKJPJZGeRE6NMJpOd5aKW6wTqabv+JNfJueQ6OT+5Xs4VqHUi9xhlMpnsLD5f4C2TyWS+dM8995CYmEhRURFFRUVYrT/+0B05MV4GZsyYgUajoaysLOhfpSmTne3GG29k8uTJjB49GoAtW7bwY9dny4/SAS4hIYFHH32U3/zmN1x11VX+DsdntFoto0aNIj09HY1G4+9wZJcJjUZDbGws0dHRzJw5k6ysLJRK5Y/+vXJiDFCSJBEREcFjjz3G+PHjaW1t5eTJk/4Oy2cSEhJ44YUX2Lp1K/Hx8SgU8qUp+34mk4nRo0cTGhqKTqcjIiKiT66dfrv6JElCoVB4P5IknfGRnSk8PJwnnniChQsXolQqKS0tpaGhwd9h+YRKpSIqKooJEyag0+nIzs6We43ncXYb6m1HwWzw4MHea+Xrr7+mrKwMl8v1o3+vzxOjTqdj6tSp/P3vf6e0tJTOzk4cDgdr165l8eLF/PKXv2TFihWkp6f7OpTLikqlIj09HbVazf79+9m4cSNHjhzxd1g+4Xa7aWtro6amBoDo6Oigb/D/SZIkDAYDS5YsobKyko6ODhwOB/n5+dxzT/C+Q02tVvOHP/yB0NBQAIqLiykpKfnR44sACCF+8AcQF/NJSEgQDz/8sKiurhY2m004nU5RUFAgysvLRUNDg2hraxNtbW3CarWK119/XZxe03TBz8XE2l+fi60TQERERIhp06aJnJwcodfrz/neBoNBLFiwQDQ1NYnq6moxY8YModPpBnSdDBkyRDzwwAPC4/GINWvWXPD7/pCPv79/X9ZLeHi4yMrKEp9++qlob28XTqdTuN1u4fF4RHd3t6iurhbPPPPMZVsvl/r/ODw8XDz33HOiq6tL9Fq9erUYP358n1wrPpmVVigUxMTEcNttt/HrX/8ao9HIwYMHWbNmDQUFBSgUCubNm8dtt93GiBEjsNvtfPedv14t2/+GDx/O448/ztVXX01iYiJfffUVbvf/HyVnMplYuHAher2e5cuXU1xcTGdnpx8j9j273U55eTmSJHHDDTdgMpk4cuTIGfUSbDQaDcnJyTz22GNMmjQJt9vNp59+SlFREWazmWuvvZbY2FgmTpyISqUKmroyGAzcdNNNzJs3D61W65MyfJIYU1JSmDNnDjfffDPR0dFUVFSwcuVKCgoKaGpqIiUlBYPBwODBg3G73TQ3N7Nt27beO8iAFxISQkxMDJGRkURERJzx2BgeHo7ZbOaqq65CoVBw7NgxOjo66Onp8WPEvufxeLDb7UiSRHR0NOPHj6e5uZn29nZ/h+Y3qampLF68mAkTJnDkyBFeeOEF6urqqK+vJz4+HpfLxbx584iLiyMmJiYolnKZTCays7O5//77GT58OHV1dcTGxvb5ZF2fJ8a0tDRvbzAqKori4mJyc3PJy8uju7sbgKysLMaPH8/gwYOxWq1s3bqV8vLyvg4lIA0dOpSxY8ei0+lobW2lvb3dm/QUCgVms5nMzEx0Oh1NTU189913Az4pnk2pVGI2mwkJCfF3KH41fPhwMjIyCA0NZdeuXbz77rvezoPNZqO2tha3241arfZZz8mf1Go1UVFRDBo0CL1ez/Dhw0lOTiYzM5NrrrmGgoICvvzyS3772996xxn7Sp8mxtjYWO677z7mzJlDWFgYe/fu5a233iI3N5fu7m4kSWLQoEFMmzaNhIQE7+PTm2++SUdHR1+GEpCUSiWpqanceeed6PV6tm/fTm1trTfxmUwmpkyZwvTp03E4HGzatIl9+/YFzSMS4G34KpUq6Cdgenp6cLvduN1uSktLkSQJrVaLXq8nPj4eg8FAW1sbZWVlnDhxwt/h9ilJkoiMjOSGG27giiuuIDExkbS0NKKjo+ns7KSkpMQ7obtkyZLATYwKhYIHH3yQX/ziFwCUlpby3nvvsX79epxOJ0qlkqFDh5KVlUViYiJqtZqSkhLeeustCgsL+yqMgKVUKomJiSE9PZ3JkyfT3NzM8uXLaW1tBU49Xs+aNYt7772XMWPGUF5ezgMPBPVrYIKeQqHw3iB6enowGo2MGDGCmTNncuWVV2IwGNi1axfPPvssLS0t/g63TykUCoYMGcL999/PFVdcgVKppLOzk7KyMj755BPWr19PfX09cXFxZwzB9dnNtC9mkCRJEgaDQRw7dky4XC6xc+dOMX36dO/P1Gq1GD16tNi7d6/o7u4WbrdbFBUViV/96ldCq9Ve1rONPzT2pKQk8corr4impiZhsVjEX/7ylzNmo3/+85+LL7/8UjidTnHo0CExf/78AV8n//nRaDQiLS1NCCFET0+PePzxx4XRaAzqWekFCxYIq9Uquru7RWFhoSgqKvK2n5dfflmkpaVd9vXyQ66L0aNHizFjxojExESh1+vP+LnZbBZWq1UIIcSaNWvEpEmTAmtWWqPREBUVhSRJrFq1igMHDjBlyhRuvvlmrr/+epKTk4mIiIBT0VBYWMiePXu8444DlVKpZO7cuTz99NMkJSUBp8aHpk6dymOPPcaqVasYPHgwM2bMYOTIkVRVVbF27VreeecdP0fuP8H+CN2ru7ubtrY2QkNDmTBhgrdeuru7sVgsNDU1+TlC33M6nRw8ePCCP+/o6KCpqYmIiAhSU1MZN24ce/fu/dHl9llidDqdWK1WhgwZwpo1a3C5XCiVSjQaDUqlEqvVit1uJzo6mpMnT7Jv3z4qKir6qviANXv2bFauXInRaEQIgc1mw2azkZGRQVpaGtnZ2ZhMJsxmM0IIPvvsM/72t7/5O2y/Ot2TCHobNmxgz549pKSkEBMTw+rVq5Ekifz8fIqLi4MiMX6fEydOYLFYiIuLo6amxrtJ4Mfqk8QohODkyZMsXbqU5cuXYzQaOXnyJFVVVRQVFZGfn09raytPP/00UVFRFBYWcuTIETweT18UH5DUajVZWVksW7aMqKgo7HY7ubm55OfnExkZycKFCxk7diyTJk1Co9GgUqno6enh5ptvRpIknn/+eY4dO+bvr9HvepPi4cOHB/zTxPdxOBwcPXqUkydPek+OgVMJs6KiIuhWK5yPEMKbR9rb22lra+uT39tnPcbexaculwudTofD4cBqtXL06FGOHj1KXFwc8fHxKJVKvvvuuz77AoFKq9Uye/Zs9Ho9H374IXv27KG4uBiLxeIdUnjooYcYMWKE9xFJkiRCQkJQKpXYbDZ/hu939fX1OJ1Of4fhdz09PXR1dWEwGLzXSXNz84Bf8H8pOjo6+qxe+nS5TnNzMzk5Oef8ee96JJPJBMChQ4f65DDJQKfValm7di2bNm2iuLjYuySpo6PjjM3uX3zxBS0tLbjdbhoaGti1a1dQLF/6b2w224B+orgYvUtXABobG3E4HEE53BAaGkpoaCgqlcrbU+zu7qayspLRo0fjcrlQqVSEhoai1WpJS0ujtraWurq6i+5d98tBtUII7HY7QggkSeKbb76hubm5P4r2G6fTySeffMKuXbs4fvy490KWJImwsDASEhKIi4vD4XDw4osvUllZicPhoKurK6h3e/SSJ2BOUSgUhIWFodfrcTqdFBQUcPz48T45QeZy0PsUFRUVRVJSErGxsYSGhiKEoLu7mxMnTnh3SMXExJCRkcGwYcPQ6/UsWLCAtWvXsm7dOhwOx0WV2y+J0e1289VXX3kTYzBwOBysX7/+nD/XarWMGDGCG264Aa1Wy/79+9mxY8eAH1q4WOHh4X1y4OjlTqfTMWHCBO644w6OHTvGBx98wKFDh+jq6vJ3aD6nUCgwGAwkJSUxa9Ysbr31VpKTk1Gr1fT09OByuWhvbycsLAyFQsGVV17JzJkz6ezsRKPREBYWRklJySVdR/2SGDUaDdnZ2UGTFP+b5ORkFi1axNy5c2lsbGTOnDkD+gDaS3X99ddTU1MTFAngQjQaDePHj+fNN9/EaDTy+OOPs3HjxqCoE4VCgdFoZNWqVdx+++3e7aFCCBwOB83NzQghqKyspKKi4ry7w4QQ5OTkXHRvEfopMSqVSkaOHNkfRQU0g8FAZmYms2bNorOzk9dff536+vqgHC86m8vlorGxkW+++Ybk5GR/hxMQJk2axMMPP4zBYCAnJ4dVq1YFzUx0SEgId999N3fddZe3Q/X111+zefNmcnNzKS8v/0HDCZfatvolMbpcLgoLC4P+uPr777+f++67D5VKRU5ODitWrJCT4mlCCJxOp3eJTmZmJuvXrw/atXpJSUned5i0t7ezadOmoEmK8P9j9IsWLcJqtbJjxw7y8vKorKzE6XT6/PyAfhtjPHjwIBaLhZiYGKKjowkPDw+6dWo1NTUcPnyY+Ph4duzYIS+5OIvb7ebbb7/l6quvJjIyEpUqOF9iGRUVxYIFC5g7dy5NTU289NJLbNu2zd9h9SuPx0NtbS3z5s3D7XbT3t5Oa2trv+WMfrvyehc4L168mOnTp9PY2EhRUVFQzcAWFBTQ2trKoEGDguLgjIvlcDhYt24dEydO5KuvvgrKG4darWbq1KlkZmai1WrZvn07Gzdu9B42EkwcDgcHDhzwS9n9lhiFEHz00Uf89Kc/JSMjg5aWFmw2G8XFxUGzXs1isWCxWPwdRsByOp3s3r2b1atXU1JSEpTJYNSoUdx2220MGzaM/fv3k5ubK18zftCvzyolJSV8/PHH3oMlmpqaqKioCPpdHrJThBC0tLSwcuVKf4fiFzqdjttvv52xY8disVj48MMPycvL83dYQalfE2NnZyfPPPMMDQ0N3HnnnYwYMYLBgwfLiVEm49SEy4wZM9i2bRt5eXmUlZUF1SHFgUS6mFnR0+cH+o0QIuAWQsp1ci65Ts5PrpdzBWqdXGyPsQXw1+v8zH4q9/vIdXIuuU7OT66XcwVknVxUj1Emk8mCQXCvuJbJZLLzkBOjTCaTnUVOjDKZTHYWOTHKZDLZWeTEKJPJZGeRE6NMJpOdRU6MMplMdhY5McpkMtlZ5MQok8lkZ/k/T0vaxGVFoxgAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 25 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WoNhnQtRNhHu"
      },
      "source": [
        "We have 60000 training samples and 10000 test samples, where each image has a shape of 28×28 pixels. Visualising the data before deploying any algorithm is always a good idea: it is a quick sanity check that can prevent avoidable mistakes."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DohVlsvsN9np"
      },
      "source": [
        "## Preprocessing Keras Datasets\n",
        "\n",
        "In this first example, we train a simple model to classify the digits on MNIST dataset. \n",
        "\n",
        "As will be studied in the lectures, convolutional layers are usually employed to classify images. However, for the time being, we use only `Dense` layers. Thus, we have to reshape the images into a 1D array to have a single dimension, and define an architecture that could be used in any 1D data, *e.g.*, time series. We will see on following weeks that when using Convolutional Neural Networks (CNNs) we must define a 3D shape for images (depth, width, height). CNNs usually have higher performance compared to models with only `Dense` layers when applied to 2D data.\n",
        "\n",
        "Let's transform the 2D images into 1D arrays."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aa6-T9T9PABQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9762b8ef-2581-4e01-cee6-24bf7244282d"
      },
      "source": [
        "X_train_flatten = X_train.reshape(X_train.shape[0], X_train.shape[1] * X_train.shape[2])\n",
        "X_test_flatten = X_test.reshape(X_test.shape[0], X_test.shape[1] * X_test.shape[2])\n",
        "\n",
        "print('New X_train shape: {0}'.format(X_train_flatten.shape))"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "New X_train shape: (60000, 784)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aVLnq7URVwxH"
      },
      "source": [
        "A standard practise is to normalise the dataset to the range $[0, 1]$."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jbOITHGSV9WS"
      },
      "source": [
        "X_train_flatten = X_train_flatten.astype('float32')\n",
        "X_test_flatten = X_test_flatten.astype('float32')\n",
        "X_train_flatten /= 255\n",
        "X_test_flatten /= 255"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eZTwrNtOWNBe"
      },
      "source": [
        "We train a classifier in a supervised manner, and therefore, we need to process the labels so that our pipeline can use them. The `y_train` and `y_test` labels indicate the numerical values belonging to the `X_train` and `X_test` images. \n",
        "\n",
        "We use a Categorical CrossEntropy as the loss function. To feed the loss function with the labels, we need to transform them into a [one-hot encoded vector](https://en.wikipedia.org/wiki/One-hot)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z-KifrbWXreU",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2e24fcaf-d167-4fdf-dfa6-1cd582226eab"
      },
      "source": [
        "Y_train_class = np_utils.to_categorical(y_train, 10)\n",
        "Y_test_class = np_utils.to_categorical(y_test, 10)\n",
        "\n",
        "print('New Y_train shape: {0}'.format(Y_train_class.shape))"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "New Y_train shape: (60000, 10)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jokxf-U8thD8"
      },
      "source": [
        "To see the difference between one-hot encoded and integer representation, let's print the first element of `y_train` and `Y_train_class`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0oYrx0kStdfy",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a622b5d7-539f-422e-a69e-534b4a0e139e"
      },
      "source": [
        "print('Label: {0}.    One-hot Vector: {1}'.format(y_train[0], Y_train_class[0]))"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Label: 5.    One-hot Vector: [0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OcgAG0OZLZ3W"
      },
      "source": [
        "## Model Architecture\n",
        "\n",
        "\n",
        "Defining a model architecture is a critical step when trying to solve a machine learning problem. In this tutorial, we will understand how models can be built into Keras. \n",
        "\n",
        "First, we need to declare a `Sequential` module. As mentioned, the `Sequential` module allows us to combine neural network layers linearly. We create a model by passing a list of layer instances to the `Sequential` module:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9_sJo3MbS1Lz"
      },
      "source": [
        "model = Sequential([\n",
        "    Dense(10, input_shape=(784,)),\n",
        "    Activation('softmax'),\n",
        "])"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iEVpTbQ9TX_u"
      },
      "source": [
        "The previous model is composed of a dense layer and activation function (Softmax). Another way of defining architectures is to simply add layers via the `.add()` method:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w-i9u2fbTZCk"
      },
      "source": [
        "model = Sequential()\n",
        "model.add(Dense(10, input_dim=784))\n",
        "model.add(Activation('softmax'))"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xGsQ9NTBBHFC"
      },
      "source": [
        "We now have a simple model ready to go! \n",
        "\n",
        "Note that the model needs to know the shape of the input data. For this reason, the first layer in the Sequential model needs information about the input shape (the following layers can automatically infer the shape, and you do not need to specify it). In our case, we tell the network that the `input_shape` is 784. The 784 size vector comes from the flatten operation of our images (28 x 28). If we were using directly images, the input shape would have been (28, 28, 1), which corresponds to the (width, height, depth) of each digit image. The formatting (width, height, depth) is called `channels_last`, which is used by default in Keras. However, Keras also accepts the format (depth, width, height), that is called `channels_first` and it is commonly used in PyTorch. Keep in mind that you can change the formatting by passing the argument `data_format` when defining a layer.\n",
        "\n",
        "Let's print the model shape output:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h5CZJUssUW1b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f0dace5d-565a-4c6a-fd34-4bf82f9ec205"
      },
      "source": [
        "print(model.output_shape)"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(None, 10)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kpz5BNJ8UnuP"
      },
      "source": [
        "We define the output of our layer to be 10, that corresponds to the 10 different classes in MNIST (one, two, three, ..., nine.). Keras automatically handles the connections between layers, so there is no need for us to manually set up anything within the architecture. \n",
        "\n",
        "We see how our model looks like by using the `.summary()` method. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a37n1PQzU3BX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7277b91f-4816-49a7-a622-4b887bc6f631"
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_1\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "dense_1 (Dense)              (None, 10)                7850      \n",
            "_________________________________________________________________\n",
            "activation_1 (Activation)    (None, 10)                0         \n",
            "=================================================================\n",
            "Total params: 7,850\n",
            "Trainable params: 7,850\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TPQEWWDuwCzk"
      },
      "source": [
        "In our example, the first row of the table has $784\\cdot 10+ 10=7850$ parameters, where the first term refers to the connections between input data and neurons,  and the second term to the output bias."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q_zb2cNDVaJi"
      },
      "source": [
        "## Compilation\n",
        "\n",
        "Before training our model, we need to configure the learning process, which is done via the compile method. We must define three important parameters here:\n",
        "\n",
        "*   The loss function that the model will try to minimise.\n",
        "*   The optimiser used to minimise the loss function and update the weights of the network.\n",
        "*   The list of metrics you want the model to compute in every step.\n",
        "\n",
        "In our example, as we are doing classification, we will define the `categorical_crossentropy` as the loss function, and we will compute the accuracy metric. You can refer to Keras documentation to learn more about [losses](https://keras.io/losses/) or [optimisers](https://keras.io/optimizers/). Many other parameters could be tuned, e.g., learning rate, decay factor, or weight normalisation. We will explore in more detail all of these parameters in the following tutorials. For now, we will use the default values for them. \n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kHasLrmvWtSf"
      },
      "source": [
        "model.compile(loss='categorical_crossentropy',\n",
        "              optimizer='adam',\n",
        "              metrics=['accuracy'])"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iqImvb8kWznJ"
      },
      "source": [
        "## Training\n",
        "\n",
        "To train the model, we call `.fit()` after compiling the model. We have to define the batch size and the number of total epochs the model is going to train. We are finally ready to start the learning of our classifier! "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "trVlIdWNXUwd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "96e4539a-71e8-4426-807f-178c9a7877b6"
      },
      "source": [
        "history = model.fit(X_train_flatten, Y_train_class, batch_size=32, epochs=10)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "1875/1875 [==============================] - 5s 2ms/step - loss: 0.7159 - accuracy: 0.8152\n",
            "Epoch 2/10\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.3191 - accuracy: 0.9114\n",
            "Epoch 3/10\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.2786 - accuracy: 0.9213\n",
            "Epoch 4/10\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.2728 - accuracy: 0.9231\n",
            "Epoch 5/10\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.2646 - accuracy: 0.9262\n",
            "Epoch 6/10\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.2578 - accuracy: 0.9272\n",
            "Epoch 7/10\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.2552 - accuracy: 0.9289\n",
            "Epoch 8/10\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.2538 - accuracy: 0.9303\n",
            "Epoch 9/10\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.2550 - accuracy: 0.9304\n",
            "Epoch 10/10\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.2509 - accuracy: 0.9314\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yj8MJvUnXzHj"
      },
      "source": [
        "Great, we have trained our first model! \n",
        "\n",
        "Imagine that we need to train the model until it converges, and we do not know how many epochs it will take. Then, we can define some mechanism to control the learning curves and stop them whenever we see fit. Those mechanisms are called callbacks in Keras. Thus, callbacks allow us to define early-stopping rules. Additionally, we use [callbacks methods](https://keras.io/callbacks/) to save the model's weights or log history along the training process, among many other things. An example of using the early stopping callback is given later in this tutorial.\n",
        "\n",
        "Most of the times, the training curves give important information about the training process. To visualise them, we can use the information saved on the training history:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OGy405W2boBB",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 312
        },
        "outputId": "03c91f04-3d7e-4bbe-9a35-2b97a0f020d1"
      },
      "source": [
        "fig, axes = plt.subplots(2,1)\n",
        "axes[0].plot(history.history['accuracy'])\n",
        "axes[0].set_title('Model Accuracy')\n",
        "axes[0].set_ylabel('Accuracy')\n",
        "axes[0].set_xlabel('Epoch')\n",
        "\n",
        "fig.subplots_adjust(hspace=0.5)\n",
        "\n",
        "axes[1].plot(history.history['loss'], 'g')\n",
        "axes[1].set_title('Model Loss')\n",
        "axes[1].set_ylabel('Cross Entropy Loss')\n",
        "axes[1].set_xlabel('Epoch')"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Text(0.5, 0, 'Epoch')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxcZb3H8c83e5qk6ZLSvaS0ZSlrIUBBrkBZRFYRF1A2RREVBBERtIIgi7L0soh6AUEWlYtc0aooZVGQpdAWKLRla0uXdKF70yV7fvePc5JO0kk7yczkJJnf+/Wa15zznHNmfjnQ85vnec55HpkZzjnnXFtZUQfgnHOue/IE4ZxzLi5PEM455+LyBOGccy4uTxDOOefi8gThnHMuLk8QLuNJKpdkknIS2Pd8SS91RVzORc0ThOtRJC2SVCeprE35m+FFvjyayFrFUixps6R/RB2Lc8nwBOF6oo+As5pXJO0L9IkunO2cAdQCx0ka0pVfnEgtyLlEeYJwPdEjwLkx6+cBD8fuIKlU0sOSVktaLGmypKxwW7ak2yStkbQQOCnOsb+RtELSMkk3SMruQHznAb8G3gbObvPZR0h6RdIGSUslnR+WF0q6PYx1o6SXwrKjJFW2+YxFko4Nl38i6QlJj0qqAs6XdIikV8PvWCHpF5LyYo7fW9IzktZJ+ljSDyUNkbRV0sCY/Q4Mz19uB/5214t4gnA90XSgr6S9wgv3mcCjbfa5GygFdgOOJEgoXwm3fR04GZgAVACfa3Psb4EGYGy4z/HA1xIJTNKuwFHA78LXuW22/SOMbRBwAPBWuPk24CDgcGAAcCXQlMh3AqcBTwD9wu9sBL4LlAGHAccA3wpjKAGeBf4JDAv/xufMbCXwb+ALMZ97DvCYmdUnGIfrZTxBuJ6quRZxHPAusKx5Q0zSuNrMNpnZIuB2ggseBBfBO8xsqZmtA26OOXYwcCJwmZltMbNVwH+Hn5eIc4C3zWwe8Biwt6QJ4bYvAc+a2R/MrN7M1prZW2HN5qvApWa2zMwazewVM6tN8DtfNbM/m1mTmVWb2Swzm25mDeHf/j8ESRKCxLjSzG43s5rw/LwWbnuIsMYTnsOzCM6zy1DeXul6qkeAF4HRtGleIvjlnAssjilbDAwPl4cBS9tsa7ZreOwKSc1lWW3235FzgfsAzGyZpBcImpzeBEYCC+IcUwYUtLMtEa1ik7Q7MIWgdtSH4N/5rHBzezEA/AX4taTRwB7ARjN7vZMxuV7AaxCuRzKzxQSd1ScCf2qzeQ1QT3CxbzaKbbWMFQQXythtzZYSdDCXmVm/8NXXzPbeWUySDgfGAVdLWilpJXAo8KWw83gpMCbOoWuAmna2bSGmAz78ZT+ozT5th2T+FfAeMM7M+gI/BJqz3VKCZrftmFkN8DhBLeIcvPaQ8TxBuJ7sAmCSmW2JLTSzRoIL3Y2SSsK2/8vZ1k/xOPAdSSMk9Qeuijl2BTANuF1SX0lZksZIOpKdOw94BhhP0L9wALAPUAh8mqB/4FhJX5CUI2mgpAPMrAl4AJgiaVjYiX6YpHzgA6BA0klhZ/FkIH8ncZQAVcBmSXsC34zZ9jdgqKTLJOWH5+fQmO0PA+cDp+IJIuN5gnA9lpktMLOZ7Wy+hODX90LgJeD3BBdhCJqAngZmA2+wfQ3kXCAPmAesJ+gAHrqjWCQVEPRt3G1mK2NeHxFcaM8zsyUENZ7vAesIOqj3Dz/iCuAdYEa47edAlpltJOhgvp+gBrQFaHVXUxxXEPR3bAr/1v9t3mBmmwj6bU4BVgIfAkfHbH+ZoHP8jbCW5jKYfMIg51wsSc8Dvzez+6OOxUXLE4RzroWkgwmayUaGtQ2XwbyJyTkHgKSHCJ6RuMyTgwOvQTjnnGuH1yCcc87F1WselCsrK7Py8vKow3DOuR5l1qxZa8ys7bM1QC9KEOXl5cyc2d4dj8455+KR1O7tzN7E5JxzLq5eU4NwzrmuVNfQRHVdI41mNDYZZtay3NTEtmULXvHKW7aH5U1NQVnLshlNRvzypmBbY5NRVpLPqfsPS/nf6AnCOZdRGhqb2FzbwKaa4LW5toHNtfWt1jfV1LO5poFN4X6bY8vDstqGREdjT78DRvbzBOGcy1z1jU1sqW2+oDfEuYDHucg37xfuu7mmger6xp1+V3aWKM7PoaQgp+V9UEk+o8uKKC7IoSQsK8jNJidLZGcJKXjPlsjKEtlZkBWWZUkty7HlzfsG6633z85qXm6vfNv23Oz09BZ4gnDOpU1jk7Vc0LeEF/MtMRf4VtvC97blHfnFniXCC3ouxfk5FBfk0L8oj5ED+lBSEFMeXuCDBJAbvLdc+HMpyM0iZrj3jOUJwjlHU5NR29BEdX0jNfWNVNc3Ul3XSG1DI9V1TdvK6hvZEveC3sjm2vrwot7YkggS+bUOkJ+TRUlBDkXhxbs4P4chfQsoDstK8mO2xfyq33ahDy78ffKy/cKeQp4gnOvGGpuMLXUNbK3ddtGuaWikJnyPvXjXxFzca+qbWpXV1De1HFtdF1MWbu9Me3peTlbLxbz5Ij6oOJ/RZc2/0rMpzs+lKD+71cW/bSIoys9JWxOJS44nCOdSpLHJ2FoX/IJuvqhvqWtga13wC3trbQNb6ra9b6ltaLPf9mU19R2/cGdnicLcbApysyjIzaYgN7tlvTg/h7Li/Jb1wnB7QW42hXnZFORkBe+x5W32LcrPoSg/m/yc7DScRdedeIJwLkZtQyOrqmpZsbGGFRurWbGxhg1b68OLfOou5nk5WRTlhRfbvBz65GdTlJdDWXFeq/U+ecHFuE9e0HxS0OZiHVzUsynIy2q5mPuvcZcqniBcxqhtaGTlxhpWbKxpeW9OAs3razbXbndcXnZWy0W6+b04P4eBRXkUhe3eQft3Tqv9Yi/0RfnZ4b7B/n4Rdz2BJwjXK9TUN7Z70V+xsZqVG2tYu6Vuu+P6FuQwtLSQof0K2Gd4X4b0DZaHlgavIaWFFOf7PxOXmfz/fNftVdc1tlzkV2ysYWVVDcs3VLdKAOu31m93XL8+uQzpG1zo9x/Zj6F9CxhSWtCSEIb0LaDIL/7Otcv/dbjIba5tYOm6rcFrfTVL122lcv1WKtdXs7Iq6ANoq3+f3OBCX1rAhFH9GNavsCUZNCeBwjzvRHUuGZ4gXNrVNjSybH11y8V/6fqtVK6rZun6ICm0/fVfnJ/DiP6FjOhfyMHlA8ILfkFLQhhSWkBBrl/8nUs3TxAuaY1Nxsqqmla1gMowESxdV83Hm2qInbgwLzuL4WEC2GffoYzs34dRA/owckAhI/v3oV+fXH/YybluIK0JQtIJwJ1ANnC/mf2szfZdgQeAQcA64Gwzq5R0APAroC/QCNxoZv+bzlhd+8yMtVvqtmsCWhrWApZvqKa+cVsGkGBo3wJGDOjDJ8aWtVz4R4ZJYHBJAVlZngCc6+7SliAkZQP3AMcBlcAMSVPNbF7MbrcBD5vZQ5ImATcD5wBbgXPN7ENJw4BZkp42sw3pitcFg6G9XbmBt5ZujKkNBH0BW+taD5kwsCiPEQP6sO/wUk4MawHNiWBYv0Lycvw2Tud6unTWIA4B5pvZQgBJjwGnAbEJYjxwebj8L+DPAGb2QfMOZrZc0iqCWoYniBRqaGxi7vIqXl24llcWrGXmonUtiaC5H2DXgUUcMXZQq1rAiP6FfvePcxkgnf/KhwNLY9YrgUPb7DMb+CxBM9TpQImkgWa2tnkHSYcAecCCtl8g6ULgQoBRo0alNPjeqKnJeHdlFa8uWMv0hWt5beE6NtU2ADBul2I+d9AIDtttIBXlAygrzvN+AOcyXNQ/A68AfiHpfOBFYBlBnwMAkoYCjwDnmdl24xiY2b3AvQAVFRXWdnumMzM+XLWZVxesDZLCR2tbbhkdXVbEyfsP47AxA5m42wB2KSmIOFrnXHeTzgSxDBgZsz4iLGthZssJahBIKgbOaO5nkNQX+DvwIzObnsY4ew0zY9HarbyyYE1YS1jXMnTE8H6FHLfXYA4bM5DDxgxkaGlhxNE657q7dCaIGcA4SaMJEsOZwJdid5BUBqwLawdXE9zRhKQ84EmCDuwn0hhjj7d03VZeXbi2pZawsqoGgMF98/mvcWUctluQEEYO6BNxpM65nmanCULSKcDf4zXx7IiZNUi6GHia4DbXB8xsrqTrgZlmNhU4CrhZkhE0MX07PPwLwCeBgWHzE8D5ZvZWR2LojVZurOHVhUEN4ZUFa6lcXw0EdxVNHDOQw8cM5LDdBjK6rMj7EJxzSZHZjpvuJT0KHAb8H8FF/r2uCKyjKioqbObMmVGHkXKrN9UyfeHallrCR2u2AFBamMvE3QZw+JgyDhszkHG7FHtCcM51mKRZZlYRb9tOaxBmdnbYH3AW8Nvw1/6DwB/MbFNqQ3Xrt9Tx2kdhk9HCtXzw8WYASvJzOGT0AL586CgOGzOQvYb09YfNnHNplVAfhJlVSXoCKAQuI7gl9fuS7jKzu9MZYKZYum4rF//hTd6u3IAZFOZmc/DoAZw+YQSHjxnI3sP6kuNzCDjnulAifRCnAl8BxgIPA4eY2SpJfQgeevMEkSQz40d/nsOCVZu5/NjdOWzMQPYb0c+fRnbORSqRGsQZwH+b2YuxhWa2VdIF6Qkrs/xzzkpe/GA1154ynq98YnTU4TjnHJBYgvgJsKJ5RVIhMNjMFpnZc+kKLFNsrm3gur/OY/zQvpwzcdeow3HOuRaJtGH8EYi9xbUxLHMpcOezH7CyqoYbTt/H+xicc91KIlekHDNrmcw3XM5LX0iZ472VVTzw8iLOOmQkB47qH3U4zjnXSiIJYnXYUQ2ApNOANekLKTM0NRmTn5xDaWEuV35qz6jDcc657STSB3ER8DtJvwBEMELruWmNKgM88UYlMxev55bP7Uf/Iq+QOee6n0QelFsATAwH08PMNqc9ql5u/ZY6bn7qXSp27c/nDhwRdTjOORdXQg/KSToJ2BsoaB7OwcyuT2NcvdotT79PVU0DP/3MPv40tHOu29ppH4SkXwNfBC4haGL6POD3Y3bSG0vW89iMJXz1E+XsNbRv1OE451y7EumkPtzMzgXWm9l1BAP37Z7esHqnhsYmJj85h8ElBVx6rJ9C51z3lkiCqAnft0oaBtQDQ9MXUu/1yPTFzFtRxTWnjKfY53R2znVziVyl/iqpH3Ar8AZgwH1pjaoXWlVVw+3TPuCTuw/i0/sMiToc55zbqR0mCElZwHPhNKD/J+lvQIGZbeyS6HqRG/7+LnWNTVx/6t4+b4NzrkfYYRNTOIvcPTHrtZ4cOu6lD9cwdfZyvnXUGMrLiqIOxznnEpJIH8Rzks6Q/+ztlNqGRq75yxx2HdiHi44cE3U4zjmXsEQSxDcIBuerlVQlaZOkqjTH1Wvc9+JCFq7ZwvWn7UNBbnbU4TjnXMISeZK6pCsC6Y2WrtvK3c/P56R9h3Lk7oOiDsc55zokkRnlPhmvvO0EQq41M+PaqXPJyRI/Pnl81OE451yHJXKb6/djlguAQ4BZwKSdHSjpBOBOIBu438x+1mb7rsADwCBgHXC2mVWG284DJoe73mBmDyUQa7cxbd7HPP/eKiaftBdDSguiDsc55zoskSamU2LXJY0E7tjZcZKyCe6AOg6oBGZImmpm82J2uw142MwekjQJuBk4R9IA4FqgguC5i1nhsesT/LsitbWugeumzmXPISWcd3h51OE451yndGYKs0pgrwT2OwSYb2YLw0mGHgNOa7PPeOD5cPlfMds/BTxjZuvCpPAMcEInYo3EXc/NZ/nGGm74zD7k+ixxzrkeKpE+iLsJfsVDkFAOIHiiemeGE8wd0awSOLTNPrOBzxI0Q50OlEga2M6xw+PEdiFwIcCoUaMSCCn9Pvh4E/f/ZyFfqBhBRfmAqMNxzrlOS6QPYmbMcgPwBzN7OUXffwXwC0nnAy8CywjmvE6Imd0L3AtQUVFhO9k97cyMyX+eQ3FBDld9OpFKlnPOdV+JJIgngBoza4Sgb0FSHzPbupPjlgEjY9ZHhGUtzGw5QQ2CcEKiM8xsg6RlwFFtjv13ArFG6sk3l/H6R+u4+bP7MsBniXPO9XAJPUkNFMasFwLPJnDcDGCcpNGS8oAzgamxO0gqC8d7Aria4I4mgKeB4yX1l9QfOD4s67Y2bq3npqfeZcKofnyxYuTOD3DOuW4ukQRREDvNaLjcZ2cHmVkDcDHBhf1d4HEzmyvpekmnhrsdBbwv6QNgMHBjeOw64KcESWYGcH1Y1m3dOu091m2p4wafJc4510sk0sS0RdKBZvYGgKSDgOpEPtzMngKealN2TczyEwRNWPGOfYBtNYpubfbSDfzutSWcf3g5ew8rjToc55xLiUQSxGXAHyUtJ5hydAjBFKQOaGwKOqYHFedz+XE+S5xzrvdI5EG5GZL2BPYIi943s/r0htVz/P61xbyzbCN3nTWBkoLcqMNxzrmU2WkfhKRvA0VmNsfM5gDFkr6V/tC6v1Wbarjl6fc5YmwZp+zns7A653qXRDqpvx7OKAdA+GTz19MXUs9x81PvUVvfxPWn+SxxzrneJ5EEkR07WVA4xlLG3+T/6oK1PPnmMr5x5G7sNqg46nCccy7lEumk/ifwv5L+J1z/BvCP9IXU/dU1NPHjv8xh5IBCvn302KjDcc65tEgkQfyAYLyji8L1twnuZMpY97+0kPmrNvPg+Qf7LHHOuV5rp01MZtYEvAYsIhihdRLBg28ZqXL9Vu567kM+tfdgjt5zl6jDcc65tGm3BiFpd+Cs8LUG+F8AMzu6a0Lrnq776zyEuOaUvaMOxTnn0mpHTUzvAf8BTjaz+QCSvtslUXVTz877mGfmfcxVn96T4f0Kd36Ac871YDtqYvossAL4l6T7JB1D8CR1Rqqua+Qnf53LuF2KueCI0VGH45xzaddugjCzP5vZmcCeBLO9XQbsIulXko7vqgC7i1/860Mq11f7LHHOuYyRSCf1FjP7fTg39QjgTYI7mzLG/FWbuffFhXz2wOEcutvAqMNxzrku0aGfwma23szuNbNj0hVQd2NmXPOXORTmZnO1zxLnnMsg3layE1NnL+eVBWv5/gl7MqgkP+pwnHOuy3iC2IGqmnp++rd32X9EKV86ZFTU4TjnXJdK5EnqjDVl2ges3VLLg+cfTLbPEuecyzBeg2jHnGUbefjVRZwzcVf2HeGzxDnnMo8niDgam4wf/XkOA4ry+d7xe+z8AOec64U8QcTx2IwlzF66gckn7UVpoc8S55zLTJ4g2lizuZZb/vk+E3cbwGkHDIs6HOeci0xaE4SkEyS9L2m+pKvibB8l6V+S3pT0tqQTw/JcSQ9JekfSu5KuTmecsX72j/fYUtvADZ/Zx2eJc85ltLQliHDmuXuATwPjgbMkjW+z22TgcTObAJwJ/DIs/zyQb2b7AgcB35BUnq5Ym73+0TqemFXJ1z+5G2N3KUn31znnXLeWzhrEIcB8M1toZnXAY8BpbfYxoG+4XAosjykvkpQDFAJ1QFUaY6W+sYnJf36H4f0K+c6kcen8Kuec6xHSmSCGA0tj1ivDslg/Ac6WVAk8BVwSlj8BbCEYTXYJcJuZrUtjrDz48kd88PFmfnLq3hTm+SxxzjkXdSf1WcBvzWwEcCLwiKQsgtpHIzAMGA18T9JubQ+WdKGkmZJmrl69utNBLN9QzR3Pfsixe+3CceMHd/pznHOuN0lnglgGjIxZHxGWxboAeBzAzF4FCoAy4EvAP82s3sxWAS8DFW2/IBw4sMLMKgYNGtTpQK//6zyazLjWZ4lzzrkW6UwQM4BxkkZLyiPohJ7aZp8lwDEAkvYiSBCrw/JJYXkRMJFghruUW7B6M8+8+zGXTBrHyAF90vEVzjnXI6VtLCYza5B0MfA0kA08YGZzJV0PzDSzqcD3gPvCqUwNON/MTNI9wIOS5hLMYvegmb2djjjHDCrmqe/8F6PLitLx8c4512PJzKKOISUqKips5syZUYfhnHM9iqRZZrZdEz5E30ntnHOum+o1NQhJq4HFSXxEGbAmReH0dH4uWvPz0Zqfj216w7nY1czi3uXTaxJEsiTNbK+alWn8XLTm56M1Px/b9PZz4U1Mzjnn4vIE4ZxzLi5PENvcG3UA3Yifi9b8fLTm52ObXn0uvA/COedcXF6DcM45F5cnCOecc3FlfILY2ax3mUTSyHCGv3mS5kq6NOqYoiYpO5zx8G9RxxI1Sf0kPSHpvXCmx8OijilKkr4b/juZI+kPkgqijinVMjpBJDjrXSZpAL5nZuMJBkj8doafD4BLgXejDqKbuJNglOU9gf3J4PMiaTjwHaDCzPYhGG/uzGijSr2MThAkNutdxjCzFWb2Rri8ieAC0HaSp4whaQRwEnB/1LFETVIp8EngNwBmVmdmG6KNKnI5QGE482Ufts2I2WtkeoJIZNa7jBTOAT4BeC3aSCJ1B3Al0BR1IN3AaIKh+B8Mm9zuD4fiz0hmtgy4jWBqghXARjObFm1UqZfpCcLFIakY+D/gMjNL61zg3ZWkk4FVZjYr6li6iRzgQOBXZjaBYErgjO2zk9SfoLVhNMHMl0WSzo42qtTL9ASRyKx3GUVSLkFy+J2Z/SnqeCL0CeBUSYsImh4nSXo02pAiVQlUmllzjfIJgoSRqY4FPjKz1WZWD/wJODzimFIu0xNEIrPeZQxJImhjftfMpkQdT5TM7GozG2Fm5QT/XzxvZr3uF2KizGwlsFTSHmHRMcC8CEOK2hJgoqQ+4b+bY+iFnfZpm1GuJ2hv1ruIw4rSJ4BzgHckvRWW/dDMnoowJtd9XAL8LvwxtRD4SsTxRMbMXpP0BPAGwd1/b9ILh93woTacc87FlelNTM4559rhCcI551xcniCcc87F5QnCOedcXJ4gnHPOxeUJwrkOkNQo6a2YV8qeJpZULmlOqj7PuWRl9HMQznVCtZkdEHUQznUFr0E4lwKSFkm6RdI7kl6XNDYsL5f0vKS3JT0naVRYPljSk5Jmh6/mYRqyJd0XzjMwTVJhZH+Uy3ieIJzrmMI2TUxfjNm20cz2BX5BMBIswN3AQ2a2H/A74K6w/C7gBTPbn2BMo+Yn+McB95jZ3sAG4Iw0/z3OtcufpHauAyRtNrPiOOWLgElmtjAc8HClmQ2UtAYYamb1YfkKMyuTtBoYYWa1MZ9RDjxjZuPC9R8AuWZ2Q/r/Mue25zUI51LH2lnuiNqY5Ua8n9BFyBOEc6nzxZj3V8PlV9g2FeWXgf+Ey88B34SWea9LuypI5xLlv06c65jCmJFu84C9gNxwvb+ktwlqAWeFZZcQzMJ2E1AC7BeWXwrcK+kCgprCNwlmJnOu2/A+CJcRwj6CYcAwM1sTU/4mcAAw2swWdfAzy4GPCBLEfIIJ7Ne0s+/5wNfM7IgdfY6ZNXQkBufSyZuYXCb5iG2/7JG0L8Fk8865ODxBuEzyCHBuzPp5wMOxO0gqlfSwpNWSFkuaLCkr3JYt6TZJayQtBE5qPi6cea5e0m8krZC0TNINkrKTCVjSMElTJa2TNF/S12O2HSJppqQqSR9LmhKWF0h6VNJaSRskzZA0OJk4XGbyBOEyyXSgr6S9wgv3mUDbeabvBkqB3YAjCRJK88xpXwdOBiYAFcDn2hz7W4LZxcaG+xwPfC3JmB8jmA96WPh9N0maFG67E7jTzPoCY4DHw/Lzwr9hJDAQuAioTjIOl4E8QbhM01yLOI5gDuFlzRtiksbVZrYp7JO4nWAaVoAvAHeY2VIzWwfcHHPsYOBE4DIz22Jmq4D/ZtsdTB0maSTBNLA/MLMaM3sLuJ9ttaB6YKykMjPbbGbTY8oHAmPNrNHMZplZVWfjcJnLE4TLNI8AXwLOp03zElBG0OG8OKZsMTA8XB4GLG2zrdmu4bErwmadDcD/ALskEeswYJ2ZbWonnguA3YH3wmakk8PyRwjmWX9M0vJwCJBcnOsgTxAuo5jZYoLO6hOBP7XZvIbg1/euMWWj2FbLWEHQbBO7rdlSgttby8ysX/jqGw6Z0VnLgQGSSuLFY2YfmtlZBEno58ATkorMrN7MrjOz8cDhBM1i5+JcB3mCcJnoAoJhMbbEFppZI0E7/o2SSiTtClzOtn6Kx4HvSBohqT9wVcyxK4BpwO2S+krKkjRG0pEdiCs/7GAukFRAkAheAW4Oy/YLY38UQNLZkgaZWRPBuE0ATZKOlrRv2GRWRZD0mjoQh3OAJwiXgcxsgZnNbGfzJcAWYCHwEvB74IFw230ETTezgTfYvgZyLsHDc/OA9cATwNAOhLaZoDO5+TWJ4LbccoLaxJPAtWb2bLj/CcBcSZsJOqzPNLNqYEj43VUE/SwvEDQ7Odch/qCcc865uLwG4ZxzLi5PEM455+JKKkFIKop5ynR3Saf67XTOOdc7JNUHIWkW8F9Af+BlYAZQZ2ZfTk14zjnnopLscN8ys63hkMW/NLNbYoZC7lJlZWVWXl4exVc751yPNWvWrDVmNijetqQThKTDCCZCuSAsS2pwss4qLy9n5sz27lx0zjkXj6TF7W1LtpP6MuBq4EkzmytpN+BfSX5ml9tQs4Em8+eInHMuVlIJwsxeMLNTzeznYWf1GjP7Topi6xIfrv2QMXeN4ZHZ/hyRc87FSvYupt+HwwoUAXOAeZK+n5rQusbYAWMZN2AcP3z+h2yp27LzA5xzLkMk28Q0PhxG+DPAP4DRbBsauUeQxJRPTWH5puXc9sptUYfjnHPdRrIJIjd87uEzwFQzqwd63Ngdh488nC/s/QVueeUWllUt2/kBzjmXAZJNEP8DLAKKgBfD0S975MQkPzvmZzQ0NTD5X5OjDsU557qFZDup7zKz4WZ2ogUWA0enKLYuNbr/aL478bs89NZDvLHijajDcc65yCXbSV0qaUo4cfpMSbcT1CZ6pKuPuJqyPmVc/vTl+Ci3zrlMl2wT0wPAJoK5er9A0Lz0YLJBRaW0oJTrj76eFxa/wF/e/0vU4TjnXKSSHYvpLTM7YGdlXaGiosJS8SR1Q1MD+/96f+oa65j7rbnkZeelIDrnnOueJM0ys4p425KtQVRLOiLmiz5BMBNWj5WTlcPtx9/O/HXzuef1e6IOxyMMMBUAABPcSURBVDnnIpNsgrgIuEfSIkmLgF8A30g6qoidMPYEThh7Ate/eD1rt66NOhznnItEsncxzTaz/YH9gP3MbALBPLo93m3H3UZVbRXXv3B91KE451wkUjKjnJlVhU9UA1yeis+M2t677M2FB17IL2f+kvfXvB91OM451+XSMeWo0vCZkbju6OsozCnkymevjDoU55zrculIEL3mAYJdinbhR//1I6a+P5XnP3o+6nCcc65LdSpBSNokqSrOaxMwLMUxRurSiZdS3q+cy5++nMamxqjDcc65LtOpBGFmJWbWN86rxMySnaWuWynIKeDnx/6c2R/P5qHZD0UdjnPOdZl0NDH1Op8f/3kOG3EYP3r+R2yu2xx1OM451yU8QSSgec6IlZtXcsvLt0QdjnPOdQlPEAmaOGIiZ+1zFre9chtLNy6NOhznnEu7ZEdzvURS/1QF093dfMzNGMYPn/9h1KE451zaJVuDGAzMkPS4pBMk9ZpnIOLZtd+uXD7xch59+1FmLJsRdTjOOZdWyQ61MRkYB/wGOB/4UNJNksakILZu6aojrmKXol24fJrPGeGc692S7oOw4Cq5Mnw1AP2BJyS125sb1jbelzRf0lU72O8MSSYp7lC0USjJL+GGo2/gpSUv8ad3/xR1OM45lzbJ9kFcKmkWcAvwMrCvmX0TOAg4o51jsoF7gE8D44GzJI2Ps18JcCnwWjIxpsNXJ3yVfXfZlyufvZLahtqow3HOubRItgYxAPismX3KzP5oZvUAZtYEnNzOMYcA881soZnVAY8Bp8XZ76fAz4GaJGNMueysbG4//nYWrl/I3a/fHXU4zjmXFsn2QVwLDJT0nfCOpgNjtr3bzmHDgdj7RCvDshbh54w0s7/v6PslXdg8H/bq1as790d00nFjjuOkcSfx0xd/yuotXfvdzjnXFZJtYvox8BAwECgDHpQ0OcnPzAKmAN/b2b5mdq+ZVZhZxaBBg5L52k659bhb2VK3heteuK7Lv9s559It2Sams4GDzezasDYxEThnJ8csA0bGrI8Iy5qVAPsA/w5nqZsITO1OHdXN9hq0FxdVXMSvZ/6aeavnRR2Oc86lVLIJYjlQELOeT+uLfTwzgHGSRkvKA84EpjZvNLONZlZmZuVmVg5MB041s5lJxpoW1x55LcV5xXz/me9HHYpzzqVUsgliIzBX0m8lPQjMATZIukvSXfEOMLMG4GLgaeBd4HEzmyvpekmnJhlPlxtUNIjJn5zMUx8+xbQF06IOxznnUkbJPOwl6bwdbTezLhsfu6KiwmbOjKaSUdtQy/hfjqdPbh/e+sZbZGdlRxKHc851lKRZZha3CT+puRvM7KGwmWj3sOj95ltdM0l+Tj63HHsLn/vj53jgzQf4+kFfjzok55xLWrJ3MR0FfEjw4NsvgQ8kfTIFcfU4n93rsxwx6ggm/2syVbVVUYfjnHNJS7YP4nbgeDM70sw+CXwK+O/kw+p5JDHl+Cms2rKKn730s6jDcc65pCWbIHLN7P3mFTP7AMhN8jN7rIOHH8zZ+53NlFensHjD4qjDcc65pCSbIGZJul/SUeHrPqBb3o7aVW6adBNZyuLq566OOhTnnEtKsgniImAe8J3wNQ/4ZrJB9WQjS0dyxeFX8Ic5f2B65fSow3HOuU7r9G2u4aisc81sz9SG1DlR3uba1ua6zYy7exyj+43m5a++TC+fR8k514Pt6DbXTtcgzKwReF/SqE5H1ksV5xVz46QbebXyVR6f+3jU4TjnXKck28TUn+BJ6uckTW1+pSKwnu68/c/jgCEH8INnf0BNQ7cbsdw553YqqQflgB+nJIpeqHnOiGMePoY7p9/JD474QdQhOedchyRbgzjRzF6IfQEnpiKw3mDS6Emcusep3PifG/l488dRh+Occx2SbII4Lk7Zp5P8zF7llmNvobqhmmv/fW3UoTjnXId0KkFI+qakd4A9JL0d8/oIeCe1IfZse5TtwbcqvsV9b9zHnFVzog7HOecS1tkaxO+BUwjmcTgl5nWQmX05RbH1GtcceQ2l+aVcMe2KqENxzrmEdSpBhJP6LDKzswjmlK4HDCj22163N7DPQK458hqeXvA0/5z/z6jDcc65hCQ7muvFwMfAM8Dfw9ffUhBXr/Otg7/F2AFj+d6079HQ1BB1OM45t1PJdlJfBuxhZnub2b7ha79UBNbb5GXncetxtzJv9Tzum3Vf1OE459xOJZsglhJMO+oScNoep3Hkrkdyzb+vYWONnzbnXPeWbIJYCPxb0tWSLm9+pSKw3kgSUz41hbVb13LTf26KOhznnNuhZBPEEoL+hzygJObl2nHg0AM574DzuOO1O/ho/UdRh+Occ+3q9Giu7X6glGNmXd4L251Gc92ZZVXL2P0Xu3PSuJN4/PM+mJ9zLjopH81V0ksxy4+02fx6Zz4zkwzvO5wrD7+SP877Iy8veTnqcJxzLq7ONjEVxSzv02abT36QgCsOv4JhJcO4fNrlNFlT1OE459x2OpsgrJ3leOsujqK8Im6adBOvL3udx+Y8FnU4zjm3nc4miH6STpd0Rrj82fB1BlCawvh6tXP2P4cDhx7IVc9eRXV9ddThOOdcK51NEC8ApwInh8vNYzGdDLyYmtB6vyxlMeX4KSytWsqUV6dEHY5zzrXSqQmDzOwrqQ4kUx1ZfiSn73k6N790MxcceAFDiodEHZJzzgHJPwfRKZJOkPS+pPmSroqz/SJJ70h6S9JLksZHEWdX+fmxP6eusY4fP+8T9Dnnuo9kpxztMEnZwD0Ekw1VAjMkTTWzeTG7/d7Mfh3ufyowBTihq2PtKuMGjuPiQy7mjul3sHDDQg4dfigTR0zk0OGHMrh4cNThOecyVJcnCOAQYL6ZLQSQ9BhwGtCSIMysKmb/IjLgzqjrjroOgBcXv8itr9zaMuJreb/ylmQxccREJgyZQH5OfpShOucyRFIJQtLngX+a2SZJk4EDgRvM7I0dHDacYJC/ZpXAoXE++9vA5QTDeExq5/svBC4EGDWqZ09DUZJfwpRPBR3V1fXVvLHiDaZXTue1Za/x8pKXW26FzcvO44AhBzBx+MQgcYw4lNH9RiP54yfOudRKaqgNSW+b2X6SjgBuAG4FrjGz7S74Mcd8DjjBzL4Wrp8DHGpmF7ez/5eAT5nZeTuKpScNtdEZyzct57XK15heOZ3py6Yzc/lMttZvBWBQn0GtahkHDz+Yvvl9I47YOdcT7GiojWSbmBrD95OAe83s75Ju2Mkxy4CRMesjwrL2PAb8qvMh9g7DSoZx+l6nc/pepwPQ0NTAnFVzgqSxbDrTK6fz1w/+CoAQ4weNb5U0xg8aT3ZWdpR/gnOuh0m2BvE3gov7cQTNS9XA62a2/w6OyQE+AI4Jj50BfMnM5sbsM87MPgyXTwGubS/DNevtNYhEbKjZwOvLXm9pmppeOZ111esAKM4r5uBhBzNxxETvAHfOtdhRDSLZBNGH4O6id8zsQ0lDgX3NbNpOjjsRuAPIBh4wsxslXQ/MNLOpku4EjiWY63o9cHFsAonHE8T2zIwF6xcEzVJh0nhr5VutOsCbaxjeAe5cZkpnghgDVJpZraSjgP2Ah81sQ6c/tJM8QSSmuQO8uYbx2rLXWLJxCQC5WblMGDqBg4YeRHm/ckaVjmJU6ShG9h3J0JKh5GRFcdObcy6d0pkg3gIqgHLgKeAvwN5mdmKnP7STPEF0XnMHeHPSmP3xbDbUtM7x2cpmeN/hjOw7slXiaFkuHUn/gv5+N5VzPUw6E8QbZnagpCuBajO7W9KbZjah0x/aSZ4gUmtT7SaWVi1lycYlLa/Y9cqqSuoa61odU5Rb1JIsRvXdljiak8iIviMoyCmI6C9yzsWTzruY6iWdBZxLMFgfQG6Sn+m6gZL8EsYPGs/4QfFHOWmyJlZtWbUteWwMk0dVsDx75Ww+3vLxdsftUrTLdrWP2PXBxYPJUiQjwDjn2kg2QXwFuAi40cw+kjQaaDvDnOuFspTFkOIhDCkewiHDD4m7T01DDcuqlsWthby35j2mLZjGlvotrY7JzcplRN8RjCodxdCSofQv6E//gv4MKBxA/8Jgue17cV6xN205lwZJz0ktKQ/YPVx938zqk46qE7yJqecxMzbUbNiu+ap5ecWmFayvWc+Gmg07nHUvJyuHfgX9tkseAwraTyqeXJwLpK2JKbxz6SFgEcFUoyMlnWdmPieE2ylJwYW6sD/7D2n30RmarIlNtZtYX7Oe9dXrd/xes5511etYsH5BS1lHk8uAwgGt1ksLSumb37flVZJXsm05v8Tv7nK9VrL/Z98OHG9m7wNI2h34A3BQsoE51yxLWZQWlFJaUEp5v/IOHWtmbKrbxLrqdSlPLs0KcwpbJYx2k0lem21t9i3KLfLajOtWkk0Quc3JAcDMPpDkndSu25DUcgHubHLZWLORqtoqNtVtoqq2quW1qbbNesz2xRsWt6xvrNlIfdPOW16FWpJGvGTSJ6cPhbmFFOQUtLwKc7atx27bUXledp4nIpeQZBPELEn3A4+G618GvCPA9QqxySVZtQ21HUowsevLNi2jqraK6vpqahpqqG6oTqhmsyMdSjBt9inMKaQwt5A+uX0ozAnf26y3LfOk1DMlmyAuAr4NfCdc/w/wyyQ/07leJz8nn/ycfMr6lCX9WWZGQ1MDNQ01LQmjZbl+23LbbW23b7ctZn1jzca426rrq7FOTM+Spax2k8l26zvZLzbp5Gbnkped1+qVm9W6LCcrx5NTJ3U6QYQzw802sz0JZnxzznUBSeRm55KbnUtJfkmXfreZUddYR3VDNVvrt1JdH763sx53nzb7bq3fypqta+J+VmeSUTxtk0aiyaVlv6z4x+ZmBf8dcrJyyM0K3nOyclrKYsvjlcWW7+z45ldXJrtOJwgzawznlR5lZktSGZRzrnuS1FIb6lfQL63fZWbUNta2m4Tqm+qpa6xredU3tl5vKW9KvLy+sZ7NdZsT2rd50Muulq3s7RLIwcMP5u9f+nvKvyvZJqb+wFxJrwMtTzyZ2alJfq5zLsNJaun76F/YP+pwttNcm2q0Ruob62loaqChqYH6pmA5Xllsebyyzu47qjQ9M2ommyB+nJIonHOuh2muTQG9doChTiUISWOBwWb2QpvyI4AVqQjMOedctDo7KtodQFWc8o3hNueccz1cp8ZikjTDzA5uZ9s7ZrZv0pF1PKbVwOIkPqIMWJOicHo6Pxet+flozc/HNr3hXOxqZoPibehsH8SObl8o7ORnJqW9PzBRkmbubN7rTOHnojU/H635+dimt5+LzjYxzZT09baFkr4GzEouJOecc91BZ2sQlwFPSvoy2xJCBZAHnJ6KwJxzzkWrUwnCzD4GDpd0NLBPWPx3M3s+ZZF1vXujDqAb8XPRmp+P1vx8bNOrz0XSEwY555zrnXzyX+ecc3F5gnDOORdXxicISSeEgw7Ol3RV1PFESdJISf+SNE/SXEmXRh1T1CRlS3pT0t+ijiVqkvpJekLSe5LelXRY1DFFSdJ3w38ncyT9QVJB1DGlWkYniHDI8nuATwPjgbMkjY82qkg1AN8zs/HARODbGX4+AC4F3o06iG7iTuCf4RD/+5PB50XScIJ5cCrMbB8gGzgz2qhSL6MTBHAIMN/MFppZHfAYcFrEMUXGzFaY2Rvh8iaCC8DwaKOKjqQRwEnA/VHHEjVJpcAngd8AmFmdmW2INqrI5QCFknKAPsDyiONJuUxPEMOBpTHrlWTwBTGWpHJgAvBatJFE6g7gSiC5+T17h9HAauDBsMntfklFUQcVFTNbBtwGLCEYoHSjmU2LNqrUy/QE4eKQVAz8H3CZmcUblLHXk3QysMrMfGSAQA5wIPArM5tAMP9LxvbZSepP0NowGhgGFEk6O9qoUi/TE8QyYGTM+oiwLGNJyiVIDr8zsz9FHU+EPgGcKmkRQdPjJEmPRhtSpCqBSjNrrlE+QZAwMtWxwEdmttrM6oE/AYdHHFPKZXqCmAGMkzRaUh5BJ9PUiGOKjILJbn8DvGtmGT3PuJldbWYjzKyc4P+L582s1/1CTJSZrQSWStojLDoGmBdhSFFbAkyU1Cf8d3MMvbDTPtkZ5Xo0M2uQdDHwNMFdCA+Y2dyIw4rSJ4BzgHckvRWW/dDMnoowJtd9XAL8LvwxtRD4SsTxRMbMXpP0BPAGwd1/b9ILh93woTacc87FlelNTM4559rhCcI551xcniCcc87F5QnCOedcXJ4gnHPOxeUJwrkOkNQo6a2YV8qeJpZULmlOqj7PuWRl9HMQznVCtZkdEHUQznUFr0E4lwKSFkm6RdI7kl6XNDYsL5f0vKS3JT0naVRYPljSk5Jmh6/mYRqyJd0XzjMwTVJhZH+Uy3ieIJzrmMI2TUxfjNm20cz2BX5BMBIswN3AQ2a2H/A74K6w/C7gBTPbn2BMo+Yn+McB95jZ3sAG4Iw0/z3OtcufpHauAyRtNrPiOOWLgElmtjAc8HClmQ2UtAYYamb1YfkKMyuTtBoYYWa1MZ9RDjxjZuPC9R8AuWZ2Q/r/Mue25zUI51LH2lnuiNqY5Ua8n9BFyBOEc6nzxZj3V8PlV9g2FeWXgf+Ey88B34SWea9LuypI5xLlv06c65jCmJFuIZijuflW1/6S3iaoBZwVll1CMAvb9wlmZGseAfVS4F5JFxDUFL5JMDOZc92G90E4lwJhH0SFma2JOhbnUsWbmJxzzsXlNQjnnHNxeQ3COedcXJ4gnHPOxeUJwjnnXFyeIJxzzsXlCcI551xc/w+3d/QNeYd9gwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1xMgl2FYaSA9"
      },
      "source": [
        "## Evaluating Model\n",
        "\n",
        "Finally, we check the metrics of our model on the test data by using the method `.evaluate()`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dac7-bRJab05",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e5c15cd9-aeb5-471d-d7df-ace11afc7369"
      },
      "source": [
        "score = model.evaluate(X_test_flatten, Y_test_class, verbose=0)\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Test loss: 0.26406407356262207\n",
            "Test accuracy: 0.9269999861717224\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jzdmn3rCgo8l"
      },
      "source": [
        "# Training a Simple Multi-layer Perceptron\n",
        "\n",
        "Earlier, we showed how to create a simple network that maps directly from $784$ (input size) to $10$ (output size). In the following sections, we explain the basics of creating more complex models, models that combine different sequential layers to perform more accurate classifications. \n",
        "\n",
        "Hence, we start by defining a network with an extra layer and study how that affects the final test accuracy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WRRqajaThGgf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "72079fbd-17f7-4ec0-97f9-985bcbfff914"
      },
      "source": [
        "model2 = Sequential()\n",
        "model2.add(Dense(100, activation='relu', input_shape=(784,)))\n",
        "model2.add(Dense(10, activation='softmax'))\n",
        "\n",
        "model2.summary()"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_2\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "dense_2 (Dense)              (None, 100)               78500     \n",
            "_________________________________________________________________\n",
            "dense_3 (Dense)              (None, 10)                1010      \n",
            "=================================================================\n",
            "Total params: 79,510\n",
            "Trainable params: 79,510\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RK4s-S-ihT2v"
      },
      "source": [
        "We have increased the number of parameters from 7850 to 79510 by adding this layer. Remember that the number of units given to the last layer in Keras defines the dimensionality of the output space, thus, we need the last layer to have the same size that the total number of classes to classify. \n",
        "\n",
        "Note that we use ReLU (Rectified Linear Unit) activation function after the first dense layer. ReLU is a common activation function, we will give more details about it in future tutorials. However, for now, you can learn more about ReLU [here](https://keras.io/activations/#relu).\n",
        "\n",
        "We train our new model and visualise the model curves:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ni6ov50rkltw",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 652
        },
        "outputId": "1111e277-c133-44d0-d232-363bbf1bfbf6"
      },
      "source": [
        "model2.compile(loss='categorical_crossentropy',\n",
        "              optimizer='adam',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "history2 = model2.fit(X_train_flatten, Y_train_class, batch_size=32, epochs=10)\n",
        "\n",
        "fig, axes = plt.subplots(2,1)\n",
        "axes[0].plot(history2.history['accuracy'])\n",
        "axes[0].set_title('Model Accuracy')\n",
        "axes[0].set_ylabel('Accuracy')\n",
        "axes[0].set_xlabel('Epoch')\n",
        "\n",
        "fig.subplots_adjust(hspace=0.5)\n",
        "\n",
        "axes[1].plot(history2.history['loss'], 'g')\n",
        "axes[1].set_title('Model Loss')\n",
        "axes[1].set_ylabel('Cross Entropy Loss')\n",
        "axes[1].set_xlabel('Epoch')"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "1875/1875 [==============================] - 4s 2ms/step - loss: 0.4421 - accuracy: 0.8766\n",
            "Epoch 2/10\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.1275 - accuracy: 0.9623\n",
            "Epoch 3/10\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.0803 - accuracy: 0.9764\n",
            "Epoch 4/10\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.0601 - accuracy: 0.9813\n",
            "Epoch 5/10\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.0460 - accuracy: 0.9863\n",
            "Epoch 6/10\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.0367 - accuracy: 0.9893\n",
            "Epoch 7/10\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.0312 - accuracy: 0.9909\n",
            "Epoch 8/10\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.0252 - accuracy: 0.9920\n",
            "Epoch 9/10\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.0216 - accuracy: 0.9934\n",
            "Epoch 10/10\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.0153 - accuracy: 0.9953\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Text(0.5, 0, 'Epoch')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxdVbn/8c83UzOnaRs6tymUlrZ2ACKzgiDKDJYrMosXUVQUVK6CP0cuilcBmZwQUIYCaimKAsoMKgqdaIEOtHSgLZ2HNM08PL8/9k56kp60Sc5Jd5LzvF+v8zp7rz2cJ+fV7uestfZeS2aGc84511Za1AE455zrmTxBOOeci8sThHPOubg8QTjnnIvLE4Rzzrm4PEE455yLyxOES3mSSiWZpIwO7HuZpH/uj7ici5onCNerSFolqU7SoDbl88OLfGk0kbWKJV/SLklPRx2Lc4nwBOF6o5XABc0rkiYDudGFs4dzgVrgZElD9ucHd6QW5FxHeYJwvdGDwKUx658GHojdQVKRpAckbZa0WtK3JaWF29Il3Sxpi6QVwOlxjr1X0npJ6yTdKCm9E/F9GvgVsBC4uM25j5P0qqQdktZIuiwsz5F0SxhruaR/hmUnSFrb5hyrJH00XP6+pJmSHpK0E7hM0hGS/h1+xnpJd0nKijl+kqRnJW2TtFHStyQNkVQlaWDMfoeF319mJ/5214d4gnC90X+AQkkTwgv3+cBDbfa5EygCDgSOJ0gonwm3XQGcARwKlAH/1ebY3wENwNhwn48Bn+1IYJJGAycAM8LXpW22PR3GVgJMA94IN98MHA4cAwwAvgE0deQzgbOBmUD/8DMbga8Cg4CjgZOAL4YxFADPAX8DhoV/4/NmtgF4CTgv5ryXAI+aWX0H43B9jCcI11s11yJOBhYD65o3xCSN682swsxWAbcQXPAguAjeZmZrzGwbcFPMsYOB04BrzKzSzDYBPwvP1xGXAAvNbBHwKDBJ0qHhtguB58zsETOrN7OtZvZGWLP5b+BqM1tnZo1m9qqZ1XbwM/9tZn8ysyYzqzazuWb2HzNrCP/2XxMkSQgS4wYzu8XMasLv57Vw2/2ENZ7wO7yA4Ht2KcrbK11v9SDwCjCGNs1LBL+cM4HVMWWrgeHh8jBgTZttzUaHx66X1FyW1mb/vbkU+A2Ama2T9DJBk9N8YCTwbpxjBgHZ7WzriFaxSRoH3EpQO8ol+H8+N9zcXgwAfwZ+JWkMMB4oN7PXuxiT6wO8BuF6JTNbTdBZfRowq83mLUA9wcW+2Sh21zLWE1woY7c1W0PQwTzIzPqHr0Izm7SvmCQdAxwMXC9pg6QNwJHAhWHn8RrgoDiHbgFq2tlWSUwHfPjLvqTNPm2HZP4lsAQ42MwKgW8BzdluDUGz2x7MrAb4A0Et4hK89pDyPEG43uxy4EQzq4wtNLNGggvdDyUVhG3/X2N3P8UfgK9IGiGpGLgu5tj1wDPALZIKJaVJOkjS8ezbp4FngYkE/QvTgA8AOcCpBP0DH5V0nqQMSQMlTTOzJuA+4FZJw8JO9KMl9QPeAbIlnR52Fn8b6LePOAqAncAuSYcAX4jZ9ldgqKRrJPULv58jY7Y/AFwGnIUniJTnCcL1Wmb2rpnNaWfzlwl+fa8A/gk8THARhqAJ6O/AAmAee9ZALgWygEXAdoIO4KF7i0VSNkHfxp1mtiHmtZLgQvtpM3uPoMbzdWAbQQf11PAU1wJvArPDbf8HpJlZOUEH8z0ENaBKoNVdTXFcS9DfURH+rb9v3mBmFQT9NmcCG4BlwEditv+LoHN8XlhLcylMPmGQcy6WpBeAh83snqhjcdHyBOGcayHpgwTNZCPD2oZLYd7E5JwDQNL9BM9IXOPJwYHXIJxzzrXDaxDOOefi6jMPyg0aNMhKS0ujDsM553qVuXPnbjGzts/WAH0oQZSWljJnTnt3PDrnnItHUru3M3sTk3POubj6TA3COef6iqYmo7ahier6RmrqG6mub6S6LliuqQ/Kq+sbqalrpKahkaKcTM6eNnzfJ+4kTxDOOddJdQ1N7Kiqo7y6PryIN7W5iDe2uYg3UV23u6w25qJfXd+0e71+dxLojKkjijxBOOdcsjU2GeXV9WyrrGVbZT3bKuvYXlUXvFfWsa2q+b2e7WFZRW1Dpz4jJzOdnKx0sjPSyM5KD9Yz08nNymBAXrAtJzON7LA8O3zlZKYFx7WsN58nnZystFZl3cEThHOuzzAzdtY0tL6wt1zw69tc8IP3HdX1tPc4WG5WOsW5WQzIy6I4L4sxA3MpzstiQG6wXpSTufui3XJxTwsv+EFZv4w0YoaO71U8QTjnIlffuLuJprqukaq63c0tzcvVdQ3BtvpGKmsb2B7+oo9NADuq6mhoin+1z0pPozgvk+LcLAbmZzFxaGFw4Y9JAMGFP7OlPDuze36Z9xaeIJxz+2RmVNU1Ul5dT1VdA9V1TcF7Szt6cCFvfUGPt62B6vqm4GIfU17f2LkRHdJEy0W8OC+LAwflc/joLAaECSD2gt+8nJeV3mt/yUfFE4RzKaS5Caa8qp4d1XXsqKpnR3U95VW7l3dU1VNeved6Zy7iWelpZGemkZuV0dLckpOVTl6/DAbmN7e/B00wuTHbc7Ji1zPa3a83N9v0Jp4gnOuFGpuMndXNF/C68CK/ezm4qLfZVh2UNbbTBAOQl5VO/9ygbb1/bibjBudTlJNF/9xM+udkUpSTSV6/jFYX9Jy2F/nMdDLS/RGrvsAThHM9RFOTsa2qjg3lNWzcWcOGnTVsLA/fd9ayvflXflUdO2v2fhdNYXYG/XODC3tRTiYjinPpH170g4t/Vst6UBYkhawMv7C73TxBOLcf1NQ3Bhf9lgt+DRvKa1sSwYbyGjZV1OzRjJMmGJTfjyFF2QzMy+KgkvyWX/f9wwt9Ucxy/5xMCnMySU/z5heXOE8QziXAzNhWWdfqot/6l3/wvqOqfo9jc7PSGVKYzeDCbI4YM4DBhdkMKQySweDCbIYUZVOS38+ba1xkPEE4147GJmPDzhrWba9uddGPXd60s5a6xtZPvUowMK8fQ4r6MaI4h8NHFzM05qI/pDCbwUXZFPTL8I5W16N5gnAprbahkbXbq1m9tZLVW6vCVyWrt1Wxdlv1Hhf/7My0ll/9h48ublmO/dV/QEE/Mv1Xv+sDPEG4Pm9XbQOrt1by3tYqVm2t4r1tu5PB++XVrZ6izc1KZ/TAPMYdUMDJEwYzamAuI4pzGVIY/PIvzPFf/S51eIJwvZ6ZsbWyjtXhxX/Vlire21bVUivYWlnXav8BeVmMGpDLB0uLGTVwBKMH5FI6KJdRA/IYlJ/lCcC5ULcmCEmnALcD6cA9ZvbjNttHA/cBJcA24GIzWxtu+wlwOsGcFc8CV5tPoJ2ymvsDVm8Jmn9amoK2BslgV8zgaRIMLcxm1MBcTp4Y1AJGD8hj9MBcRg3MpTA7M8K/xLneo9sShKR04OfAycBaYLakJ8xsUcxuNwMPmNn9kk4EbgIukXQMcCwwJdzvn8DxwEvdFa/rGcqr63lnYwVLNlTw7qZd7fYHZKaLkcXBBf+IMQMYNSCX0QOD14ji3JQfQ8e5ZOjOGsQRwHIzWwEg6VHgbCA2QUwEvhYuvwj8KVw2IBvIAgRkAhu7MVa3n9U2NPLupkqWbtzJkg0VLA1f68trWvZp2x8wemBYCxiQy7D+OX6vv3PdrDsTxHBgTcz6WuDINvssAKYTNEN9AiiQNNDM/i3pRWA9QYK4y8wWt/0ASZ8DPgcwatSo5P8FLmFNTcba7dUs3VjB0g27k8HKLZUto25mpouDSvI5YswAxg8p4JAhBYwfUsiwomzvD3AuQlF3Ul8L3CXpMuAVYB3QKGksMAEYEe73rKQPmdk/Yg82s7uBuwHKysq8fyJi2yrrWLJhZ0ttYMmGCpZtrKCyrrFlnxHFORwypICPTRrM+CGFHDKkgDGD8vy2UOd6oO5MEOuAkTHrI8KyFmb2PkENAkn5wLlmtkPSFcB/zGxXuO1p4GigVYJw0aiua2TZpoqW2kBzn8HmitqWfYpzMxk/pIBPlo1k3OACxg8JXvn9ov5N4pzrqH3+b5V0JvCkmXVuklSYDRwsaQxBYjgfuLDNuQcB28JzX09wRxPAe8AVkm4iaGI6Hritk5/vEtTYZKzeWtlSG1i6oYKlGytYtbWy5dmBfhlpHDw4nw8fXBI2DQVNRCUF/bx5yLleriM/5z4F3CbpMeA+M1vSkRObWYOkq4C/E9zmep+ZvS3pBmCOmT0BnADcJMkImpi+FB4+EzgReJOgw/pvZvaXTvxdrgt21Tbw/OKN/GPZFpZuqGDZpoqWydMlKB2Yx/jBBZw1dRiHDClg3JACSgfmeWexc32UOvJogaRC4ALgMwQX7N8Cj5hZRfeG13FlZWU2Z86cqMPodZqTwpML1/PSO5upa2hiQF4wHeP4mBrBwQcUdNvE6M656Eiaa2Zl8bZ1qEHYzHZKmgnkANcQ3HH0P5LuMLM7kxeq2x8qaup5fvEmnnxzPS+HSWFwYT8uPGIUp08ZyuGjiknzWoFzKa8jfRBnEdQcxgIPAEeY2SZJuQTPNHiC6AX2lhTOmDKUwzwpOOfa6EgN4lzgZ2b2SmyhmVVJurx7wnLJUFFTz3OLN/Lkwg28sixICkMKs7noyFGcPtmTgnNu7zqSIL5P8MAaAJJygMFmtsrMnu+uwFzX7KypD/sUNvDKO5upa9ydFM6YMpRDR3pScM51TEcSxB+BY2LWG8OyD3ZLRK7TdtbU89yijTz15npeeWcLdY1NDC3K5uKjRnP6lCGeFJxzXdKRBJFhZi3jJZtZnaSsbozJdUB7SeGSo0dz2uShHDqyvycF51xCOpIgNks6K3xuAUlnA1u6NywXT3n17qTwj2VBUhjmScE51006kiCuBGZIuovgqeY1wKXdGpVrEZsUXlm2mfpGY1hRNpcePZrTpgxl2ghPCs657rHPBGFm7wJHhWMl0Tw+kus+5dX1PNtSUwiSwvD+OXz66FJOnzKUaSP7+zAWzrlu16EH5SSdDkwCspsvTGZ2QzfGlXKamow/L1jHXxa0TgqXHVPKaZM9KTjn9r+OPCj3KyAX+AhwD/BfwOvdHFfKue35Zdzx/LKWpHD6lGFMHVHkScE5F5mO1CCOMbMpkhaa2Q8k3QI83d2BpZJX393CnS8sY/phw7nlk1M9KTjneoSOzNLSPAdklaRhQD0wtPtCSi1bd9Xy1d+/wZhBefzv2R/w5OCc6zE6UoP4i6T+wE+BeQSjuf6mW6NKEU1NxrV/XMD2qnruu+yD5PlkOs65HmSvNQhJacDzZrbDzB4DRgOHmNl3O3JySadIWippuaTr4mwfLel5SQslvSRpRMy2UZKekbRY0iJJpZ36y3qB+/61kheXbubbp09g0rCiqMNxzrlW9pogwpnefh6zXmtm5R05saT08NhTgYnABZImttntZuABM5sC3ADcFLPtAeCnZjYBOALY1JHP7S0WrNnB//1tCR+fNJhLjhoddTjOObeHjvRBPC/pXHW+cfwIYLmZrQiH6ngUOLvNPhOBF8LlF5u3h4kkw8yeheDZCzOr6uTn91g7a+r58iPzOaAgm5+c653SzrmeqSMJ4vMEg/PVStopqULSzg4cN5zgqetma8OyWAuA6eHyJ4ACSQOBccAOSbMkzZf007BG0oqkz0maI2nO5s2bOxBS9MyMb816k3U7qrn9/GkU5WZGHZJzzsW1zwRhZgVmlmZmWWZWGK4XJunzrwWOlzQfOB5YRzBabAbwoXD7B4EDgcvixHa3mZWZWVlJSUmSQupev5+9hr8uXM/XTh5HWemAqMNxzrl2deRBuQ/HK287gVAc64CRMesjwrLYc7xPWIMIh/I418x2SFoLvGFmK8JtfwKOAu7dV7w92TsbK/j+X97muLGD+MLxB0UdjnPO7VVH7qv8n5jlbIK+hbnAifs4bjZwsKQxBInhfODC2B0kDQK2hZ3h1wP3xRzbX1KJmW0OP2tOB2LtsWrqG7nq4Xnk98vg1k9N9QH2nHM9XkcG6zszdl3SSOC2DhzXIOkq4O9AOnCfmb0t6QZgTjh8+AnATZIMeAX4Unhso6RrCTrIRZCQevWzFz/4yyLe2biLB/77CA4oyI46HOec26euPJm1FpjQkR3N7CngqTZl341ZngnMbOfYZ4EpXYivx/nrwvd55PX3uPL4g/jwuN7RV+Kccx3pg7iT4OlpCDq1pxE8Ue06YM22Kq5/7E0OHdWfr39sXNThOOdch3WkBhHb9t8APGJm/+qmePqUuoYmrnpkPgjuOP9QMtM7clexc871DB1JEDOBGjNrhOAJaUm5fenBte5yyzNLWbBmB7+86DBGDsiNOhznnOuUDj1JDeTErOcAz3VPOH3HS0s38etXVnDxUaM4dbIPfuuc6306kiCyY6cZDZf95/BebNxZw9f/sIBDhhTw7dPbDj/lnHO9Q0cSRKWkw5pXJB0OVHdfSL1bY5Px1d+/QVVdI3ddeCjZmXuMEOKcc71CR/ogrgH+KOl9QMAQ4FPdGlUv9osXl/Pqu1v5yX9NYewBBVGH45xzXdaRB+VmSzoEGB8WLTWz+u4Nq3d6feU2fvbcO5w9bRifPHzEvg9wzrkebJ9NTJK+BOSZ2Vtm9haQL+mL3R9a77Kjqo6rH53PyAG53HiOTx3qnOv9OtIHcYWZ7WheMbPtwBXdF1LvY2Zc+8eFbNlVy10XHEZBtg/h7Zzr/TqSINJjJwsK52XI6r6Qep/7X13Fc4s3ct2pE5g8wqcOdc71DR3ppP4b8HtJvw7XPw883X0h9S5vrSvnR08t4aRDDuC/jy2NOhznnEuajiSIbwKfA64M1xcS3MmU8nbVNvDlR+YzIC+Ln37Spw51zvUtHZlRrgl4DVhFMBfEicDi7g2rd/jun95i9dZKbj9/GgPyvNXNOde3tJsgJI2T9D1JS4A7gfcAzOwjZnZXR04u6RRJSyUtl3RdnO2jJT0vaaGklySNaLO9UNJaSR36vP1p5ty1zJq/jqtPGseRBw6MOhznnEu6vdUglhDUFs4ws+PM7E6C+aI7JOzM/jlwKjARuEBS23EnbgYeMLMpwA3ATW22/y/BREI9yrubd/GdP73FUQcO4KoTx0YdjnPOdYu9JYjpwHrgRUm/kXQSwZPUHXUEsNzMVphZHfAocHabfSYCL4TLL8ZuD4f0GAw804nP7HbB1KHzyclK5/bzDyXdpw51zvVR7SYIM/uTmZ0PHEJw8b4GOEDSLyV9rAPnHg6siVlfG5bFWkCQiAA+ARRIGigpDbgFuHZvHyDpc5LmSJqzefPmDoSUuB89tZjF63dy8yenMLjQpw51zvVdHemkrjSzh8O5qUcA8wnubEqGa4HjJc0HjgfWETRjfRF4yszW7iO2u82szMzKSkq6fyrPv721gQf+vZrPHjeGEw8Z3O2f55xzUerUnNThU9R3h699WQeMjFkfEZbFnu99whqEpHzgXDPbIelo4EPhkB75QJakXWa2R0f3/rJ2exXfmLmAKSOK+MYph0QVhnPO7TedShCdNBs4WNIYgsRwPnBh7A6SBgHbwltprwfuAzCzi2L2uQwoizI51Dc2cfWjb9BkcOcFh5KV4VOHOuf6vm670plZA3AV8HeC5yb+YGZvS7pB0lnhbicASyW9Q9Ah/cPuiicRtz33DnNXb+em6ZMZPTAv6nCcc26/kJlFHUNSlJWV2Zw5c5J+3n8u28Il973Gp8pG8uNzpyT9/M45FyVJc82sLN42byvZi80VtVzz+zcYW5LP986cFHU4zjm3X3VnH0Sv1tRkfO0Pb1BRU8+Mzx5JTpZPHeqcSy1eg2jHr19ZwT+WbeF7Z05i/BCfOtQ5l3o8QcQxd/V2bn5mKadPHsoFR4zc9wHOOdcHeYJoo7y6nq88Mp+hRdncdO5kH8LbOZeyvA8ihplx3WML2bizhplfOIZCnzrUOZfCvAYRY8Zr7/H0Wxv4xinjmTayf9ThOOdcpDxBhBav38kNf13E8eNK+OxxB0YdjnPORc4TBFBV18BVD8+jf04mt5w3lTQfwts557wPAuD7T7zNii2VzLj8SAbl94s6HOec6xFSvgbx7uZdzJq3jqs+MpZjxg6KOhznnOsxUr4GcVBJPn/58nEcfEB+1KE451yPkvIJAmDC0MKoQ3DOuR4n5ZuYnHPOxddnhvuWtBlYncApBgFbkhROb+ffRWv+fbTm38dufeG7GG1mceds7jMJIlGS5rQ3Jnqq8e+iNf8+WvPvY7e+/l14E5Nzzrm4PEE455yLyxPEbndHHUAP4t9Fa/59tObfx259+rvwPgjnnHNxeQ3COedcXJ4gnHPOxZXyCULSKZKWSlou6bqo44mSpJGSXpS0SNLbkq6OOqaoSUqXNF/SX6OOJWqS+kuaKWmJpMWSjo46pihJ+mr4/+QtSY9Iyo46pmRL6QQhKR34OXAqMBG4QNLEaKOKVAPwdTObCBwFfCnFvw+Aq4HFUQfRQ9wO/M3MDgGmksLfi6ThwFeAMjP7AJAOnB9tVMmX0gkCOAJYbmYrzKwOeBQ4O+KYImNm681sXrhcQXABGB5tVNGRNAI4Hbgn6liiJqkI+DBwL4CZ1ZnZjmijilwGkCMpA8gF3o84nqRL9QQxHFgTs76WFL4gxpJUChwKvBZtJJG6DfgG0BR1ID3AGGAz8Nuwye0eSXlRBxUVM1sH3Ay8B6wHys3smWijSr5UTxAuDkn5wGPANWa2M+p4oiDpDGCTmc2NOpYeIgM4DPilmR0KVAIp22cnqZigtWEMMAzIk3RxtFElX6oniHXAyJj1EWFZypKUSZAcZpjZrKjjidCxwFmSVhE0PZ4o6aFoQ4rUWmCtmTXXKGcSJIxU9VFgpZltNrN6YBZwTMQxJV2qJ4jZwMGSxkjKIuhkeiLimCIjSQRtzIvN7Nao44mSmV1vZiPMrJTg38ULZtbnfiF2lJltANZIGh8WnQQsijCkqL0HHCUpN/x/cxJ9sNM+pScMMrMGSVcBfye4C+E+M3s74rCidCxwCfCmpDfCsm+Z2VMRxuR6ji8DM8IfUyuAz0QcT2TM7DVJM4F5BHf/zacPDrvhQ20455yLK9WbmJxzzrXDE4Rzzrm4PEE455yLyxOEc865uDxBOOeci8sThHOdIKlR0hsxr6Q9TSypVNJbyTqfc4lK6ecgnOuCajObFnUQzu0PXoNwLgkkrZL0E0lvSnpd0tiwvFTSC5IWSnpe0qiwfLCkxyUtCF/NwzSkS/pNOM/AM5JyIvujXMrzBOFc5+S0aWL6VMy2cjObDNxFMBIswJ3A/WY2BZgB3BGW3wG8bGZTCcY0an6C/2Dg52Y2CdgBnNvNf49z7fInqZ3rBEm7zCw/Tvkq4EQzWxEOeLjBzAZK2gIMNbP6sHy9mQ2StBkYYWa1MecoBZ41s4PD9W8CmWZ2Y/f/Zc7tyWsQziWPtbPcGbUxy414P6GLkCcI55LnUzHv/w6XX2X3VJQXAf8Il58HvgAt814X7a8gneso/3XiXOfkxIx0mwVMADLD9WJJCwlqAReEZV8mmIXtR0ABMCUsvxq4W9LlBDWFLxDMTOZcj+F9EC4lhH0Ew4BhZrYlpnw+MA0YY2arOnnOUmAlQYJYTjCB/ZZ29r0M+KyZHbe385hZQ2dicK47eROTSyUr2f3LHkmTCSabd87F4QnCpZIHgUtj1j8NPBC7g6QiSQ9I2ixptaRvS0oLt6VLulnSFkkrgNObjwtnnquXdK+k9ZLWSbpRUnoiAUsaJukJSdskLZd0Rcy2IyTNkbRT0kZJt4bl2ZIekrRV0g5JsyUNTiQOl5o8QbhU8h+gUNKE8MJ9PtB2nuk7gSLgQOB4goTSPHPaFcAZwKFAGfBfbY79HcHsYmPDfT4GfDbBmB8lmA96WPh5P5J0YrjtduB2MysEDgL+EJZ/OvwbRgIDgSuB6gTjcCnIE4RLNc21iJMJ5hBe17whJmlcb2YVYZ/ELQTTsAKcB9xmZmvMbBtwU8yxg4HTgGvMrNLMNgE/Y/cdTJ0maSTBNLDfNLMaM3sDuIfdtaB6YKykQWa2y8z+E1M+EBhrZo1mNtfMdnY1Dpe6PEG4VPMgcCFwGW2al4BBBB3Oq2PKVgPDw+VhwJo225qNDo9dHzbr7AB+DRyQQKzDgG1mVtFOPJcD44AlYTPSGWH5gwTzrD8q6f1wCJBMnOskTxAupZjZaoLO6tOAWW02byH49T06pmwUu2sZ6wmabWK3NVtDcHvrIDPrH74KwyEzuup9YICkgnjxmNkyM7uAIAn9HzBTUp6Z1ZvZD8xsInAMQbPYpTjXSZ4gXCq6nGBYjMrYQjNrJGjH/6GkAkmjga+xu5/iD8BXJI2QVAxcF3PseuAZ4BZJhZLSJB0k6fhOxNUv7GDOlpRNkAheBW4Ky6aEsT8EIOliSSVm1kQwbhNAk6SPSJocNpntJEh6TZ2IwznAE4RLQWb2rpnNaWfzl4FKYAXwT+Bh4L5w228Imm4WAPPYswZyKcHDc4uA7cBMYGgnQttF0Jnc/DqR4LbcUoLaxOPA98zsuXD/U4C3Je0i6LA+38yqgSHhZ+8k6Gd5maDZyblO8QflnHPOxeU1COecc3F5gnDOORdXQglCUl7MU6bjJJ3lt9M551zfkFAfhKS5wIeAYuBfwGygzswuSk54zjnnopLocN8ys6pwyOJfmNlPYoZC3q8GDRpkpaWlUXy0c871WnPnzt1iZiXxtiWcICQdTTARyuVhWUKDk3VVaWkpc+a0d+eic865eCStbm9bop3U1wDXA4+b2duSDgReTPCc+92Wqi347b7OOddaQgnCzF42s7PM7P/CzuotZvaVJMW2X7yz9R3G3zWee+ffG3UozjnXoyR6F9PD4bACecBbwCJJ/5Oc0PaPsQPGMm3INK7+29W8s/WdqMNxzrkeI9EmponhMMLnAE8DY9g9NHKvkKY0HjjnAfql9+PCxy6krrEu6pCcc65HSDRBZIbPPZwDPGFm9UCva8wfXjice866h7nr5/L9l74fdTjOOdcjJJogfg2sAvKAV8LRL3vlxCTTJ0zns4d+lh//88e8tEzgFjwAABYtSURBVOqlqMNxzrnIJX2wPkkZZtaQ1JN2QFlZmSV6m+uuul0c9uvDqG6oZuGVCynOKU5SdM451zNJmmtmZfG2JdpJXSTp1nDi9DmSbiGoTfRK+Vn5zJg+gw27NvD5v37eb311zqW0RJuY7gMqCObqPY+geem3iQYVpQ8O/yA3nHADf1z0Rx5Y0HZGSuecSx2JJoiDzOx7ZrYifP0AODAZgUXpG8d+g+NHH89VT1/Fu9vejToc55yLRKIJolrScc0rko4lmAmrV0tPS+fBTzxIRloGF826iPrG+qhDcs65/S7RBHEl8HNJqyStAu4CPp9wVD3AyKKR/PqMX/Pautf431f+N+pwnHNuv0t0qI0FZjYVmAJMMbNDCebR7RPOm3Qen576aX74jx/yz/f+GXU4zjm3XyVlRjkz2xk+UQ3wtWScs6e449Q7KO1fysWzLqa8pjzqcJxzbr/pjilH1Q3njExhv0JmTJ/B2p1r+dJTX4o6HOec22+6I0H0uYcHjhpxFN87/nvMeHMGMxbOiDoc55zbL7qUICRVSNoZ51UBDEtyjD3C9R+6nmNHHssXn/oiK7evjDoc55zrdl1KEGZWYGaFcV4FZpboLHU9UkZaBg9NfwiASx6/hIam/T6aiHPO7Vfd0cTUZ5X2L+UXp/2Cf635Fzf946aow3HOuW4VSYKQdIqkpZKWS7ouzvavSVokaaGk58NRYnuEi6ZcxIWTL+QHL/+A/6z9T9ThOOdct9nvCUJSOvBz4FRgInCBpIltdpsPlJnZFGAm8JP9G+Xe/eK0XzCicAQXzbqInbW9cnRz55zbp0RHc/2ypM6OiX0EsDwcu6kOeBQ4O3YHM3vRzKrC1f8AIxKJM9mKsot4aPpDrNqxiq883aum4HbOuQ5LtAYxGJgt6Q9hs1FHnoEYDqyJWV8blrXncoLpTPcg6XPNQ41v3ry5w0Enw3GjjuP/fej/cf+C+/n9W7/fr5/tnHP7Q6JDbXwbOBi4F7gMWCbpR5IOSkJsSLoYKAN+2s7n321mZWZWVlJSkoyP7JTvfPg7HDn8SK588kreK39vv3++c851p4T7ICyYVWdD+GoAioGZktrrN1gHjIxZHxGWtSLpo8D/A84ys9pE4+wOmemZzJg+g4amBi59/FIamxqjDsk555Im0T6IqyXNJehE/hcw2cy+ABwOnNvOYbOBgyWNkZQFnA880ea8hxLMd32WmW1KJMbudtCAg7jr1Lt4efXL/ORfPaov3TnnEpJoDWIAMN3MPm5mfzSzegAzawLOiHdAOF/1VcDfgcXAH8zsbUk3SDor3O2nQD7wR0lvSHoi3rl6ikunXsp5k87juy99l9nrZkcdjnPOJYUSnXdZ0mHAcQRjMP3LzOYlI7DOKisrszlz5kTx0QBsr97OlF9NIScjh3mfn0d+Vn5ksTjnXEdJmmtmZfG2JdrE9B3gfmAgMAj4raRvJ3LO3qo4p5gHP/Egy7ct56t/+2rU4TjnXMISbWK6GPhgOC/194CjgEsSD6t3OqH0BL557De5Z/49zFo8K+pwnHMuIYkmiPeB7Jj1fsS5IymV/OAjP6BsWBlX/OUK1u1M6a/COdfLJZogyoG3Jf1O0m+Bt4Adku6QdEfi4fU+WelZzJg+g5qGGi7906U0WVPUITnnXJckOjT34+Gr2UsJnq9PGDdwHLefcjtX/OUKbv33rVx7zLVRh+Scc52WUIIws/vDZxnGhUVLm291TXWXH3o5Ty17im89/y1OGnMShw49NOqQnHOuUxK9i+kEYBnB6Ky/AN6R9OEkxNXrSeI3Z/6GkrwSLpx1IVX1Vfs+yDnnepBE+yBuAT5mZseb2YeBjwM/SzysvmFg7kAeOOcBlmxZwtf//vWow3HOuU5JNEFkmtnS5hUzewfITPCcfcpJB57EtUdfy6/m/oonlvboB8Kdc66VRBPEXEn3SDohfP0GiO5x5h7qxhNvZNqQaVz+xOWsr1gfdTjOOdchiSaIK4FFwFfC1yLgC4kG1df0y+jHw9MfZlfdLi7782V+66tzrlfocoIIpw5dYGa3mtn08PWznjo0d9QmlEzg1o/dyjPvPsOdr90ZdTjOObdPXU4QZtYILJU0Konx9GlXll3JmePO5BvPfYOFGxdGHY5zzu1Vok1MxQRPUj8v6YnmVzIC64skce9Z91KcXcyFj11IdX111CE551y7En2S+jtJiSKFlOSV8LtzfsepM07lm899kztOTckRSZxzvUCiNYjTzOzl2BdwWjIC68tOGXsKVx95NXe+fidPLXsq6nCccy6uRBPEyXHKTk3wnCnhxx/9MZMPmMxn/vwZNlX26FlVnXMpqksJQtIXJL0JjJe0MOa1EngzuSH2TdkZ2Tx87sOU15TzmT9/hkRn9nPOuWTrag3iYeBM4Inwvfl1uJldlKTY+rwPHPABfnryT3lq2VP8YvYvog7HOeda6VKCMLNyM1tlZhcAa4F6gjmp8/2218656oirOGXsKVz77LW8ventqMNxzrkWiY7mehWwEXgWeDJ8/TUJcaUMSfz27N9SkFXAhbMu5K1Nb3lzk3OuR0j0NtdrgPFmtjUZwaSqIflD+N05v+PsR89m8i8nM27gOKYfMp3pE6ZTNqwMSVGH6JxLQUrk16qkF4GTzawheSF1TVlZmc2Z07vHCVxfsZ4/L/0zsxbP4oWVL9BojYwsHMn0CUGyOHbksaSnpUcdpnOuD5E018zK4m5LMEHcC4wnaFpqGYPJzG7dx3GnALcD6cA9ZvbjNts/DNwGTAHON7OZ+4qlLySIWNuqt/GXpX9h1pJZ/H3536ltrOWAvAM4Z/w5TJ8wnY+M+QhZ6VlRh+mc6+W6M0F8L165mf1gL8ekA+8QPEOxFpgNXGBmi2L2KQUKgWuBJ1IxQcSqqK3g6eVPM2vxLJ5c9iS76nbRP7s/Z447k+kTpvPxgz5OTmZO1GE653qhvSWIROek3iMRSNrXOY8AlpvZinD/R4GzCYYKbz7vqnCbj4sNFPQr4LxJ53HepPOoaajh2XefZdaSWfx5yZ95cOGD5GbmctrBpzH9kOmcPu50CvsVRh2yc64P6FKCkPRPMzsuXH7QzC6J2fw6cNheDh8OrIlZXwsc2ZU4UlF2RjZnjj+TM8efSf0Z9by8+mVmLZ7F40seZ+aimWSlZ3HygSczfcJ0zhp/FoNyB0UdsnOul+pqDSIvZvkDbbbtt1tuJH0O+BzAqFGp9/hFZnomHz3wo3z0wI9y12l38e81/2bW4lnMWhI0RaUrneNLj2f6IdM555BzGF44POqQnXO9SFefg7B2luOtt7UOGBmzPiIs63wQZnebWZmZlZWUlHTlFH1GmtI4dtSx3PLxW1jxlRXM+9w8rjvuOtZXrOeqp69ixM9GcMy9x3DzqzezYvuKqMN1zvUCXeqklrQC+DpBgvkpQWcyBLWHn5jZQXs5NoOgk/okgsQwG7jQzPZ4jFjS74C/pnondaIWb17cUrOYt34eAFMHT+XcCecyfcJ0JpZM9GctnEtRSb+LSdJv97bdzD6zj+NPI7iNNR24z8x+KOkGYI6ZPSHpg8DjBBMS1QAbzGzS3s7pCaJjVu1YFSSLxbN4dc2rGMa4geNaksXhQw/3ZOFcCum221x7Ek8Qndf8YN5jix/jxZUv0miNjCoaxZnjzqRsWBnThkxjYslEf97CuT7ME4Tbp9gH855b8RxV9VUAZKRlMLFkItOGTGPa4GlMGzKNqUOmMiBnQMQRO+eSwROE65TGpkbe3f4ub2x4o+W1YOMC3q94v2WfkYUjg6QRvqYOnsqY4jGkKdE5qJxz+5MnCJcUmyo3sWDDgiBpbHyDBRsWsGTLEhqtEYCCrAKmDpnKtMFBLWPakGl84IAPkJ2RHXHkzrn2dOdQG58E/mZmFZK+TfCA3I1mNq/LJ+0iTxDRqK6v5u3Nb7eqaSzYsICKugoA0pXOIYMOaallNNc4SvJS+7Zk53qK7kwQC81siqTjgBsJbnn9rpnt9yejPUH0HE3WxMrtK1sSRnPyWLNz9wP0wwqG7ZE0xg4Y601Uzu1n3TYWE9AYvp8O3G1mT0q6McFzul4uTWkcNOAgDhpwEOdOPLelfGvV1pYaxhsbg6TxzLvP0NAUjBafl5nHlMFTWpLGpAMmMab/GIYWDPXE4VwEEq1B/JXgYbeTCZqXqoHXzWxqcsLrOK9B9E61DbUs2ryoVU3jjQ1vUF5b3rJPVnoWo4pGUdq/lDH9x1Dav7TVa0j+EE8gznVRdzYx5QKnAG+a2TJJQ4HJZvZMl0/aRZ4g+g4zY3X5apZsWcKqHav2eG2s3Nhq/6z0LEYXjWZM8RhKi0r3SCCD8wd7AnGuHd3ZxDQUeNLMaiWdQDDBzwMJntOlOEktF/d4quqrWL1jdevEUb6KldtXMn/9fDZXbW61f3ZGNqOLRu+ROFoSSN5gf3rcuTgSTRCPAWWSxgJ3A38GHgZOSzQw59qTm5nLhJIJTCiZEHd7ZV0lq8t3J5CV21eyqjxYnrt+LluqtrTaPzsje3fCaFMDGVM8hpLcEk8gLiUlmiCazKxB0nTgTjO7U9L8ZATmXFflZeUxsWQiE0smxt2+q24Xq3esZuWOlXs0X81eN5ut1Vtb7Z+Zlsng/MEMzhvMkPwhu9/z91wv6lfkycT1GYkmiHpJFwCXAmeGZZkJntO5bpWflc+kAyYx6YD44z/urN3Z0oS1csdK3q94n42VG9mwawPvV7zPvPXz2FS5qeUBwVj90vvtmTjaSSj5WfmeTFyPlmiC+AxwJfBDM1spaQzwYOJhORedwn6FTB48mcmDJ7e7T5M1sbVqa0vi2LgrfK/c/b66fDWvr3udzVWbabI9Z8/NyciJXxOJk1DysvLiROFc90p4qA1JWcC4cHWpmdUnHFUX+F1MrqdqbGpkS9WW1gkkTkLZsGvDHv0jzfKz8hmYM5DinGKKs4t3v4fLA3IGtC4P3/tn9yc9LX0//8WuN+m2u5jCO5fuB1YRTBY0UtKnzeyVRM7rXF+SnpYe9GHkD97nvvWN9Wyu2rxnAtm1ka3VW9les53t1dtZsmUJ26u3s71mOzUNNXs9Z1G/onYTS3F2mFzabs8ppqhfkSeXFJdoE9MtwMfMbCmApHHAI8DhiQbmXCrKTM9kWMEwhhUM6/AxNQ01bK/ezrbqbS0JJO57TbDPos2LWsprG2vbPa8QRdlFeySOgqwC8jLzyM/KJy8reM/Pym8piy2PLctKz/I+l14m0QSR2ZwcAMzsHUneSe3cfpSdkc3QgqEMLRja6WOr66v3SCZ7JJqY5fc3vc+uul3sqttFZX0ldY11Hf6sdKXvNYHkZeWRn9nxpFOUXURBVoHXcrpRoglirqR7gIfC9YsA7whwrpfIycwhJzOnUzWWWPWN9VTWV+5OGnWVLcmjbVlseez2LVVbWLVjVavyziSe/Kx8CvsVUtivkKJ+RcF7dhGFWeF7vG0x64X9Cv2OsnYkmiCuBL4EfCVc/wfwiwTP6ZzrJTLTM+mf3p/+2f2Tet7YxBMvwVTUVlBRV0F5TTk7a3dSXtv6fV3FupZtzUPP702a0ijIKoibPPaWWAr7FZKbmdvqlZOZQ0ZaopfWnqHLf4WkdGCBmR0C3Jq8kJxzqS6ZiaexqZFddbt2J5G2SSVcb5toNldtZvm25S3r+7oZIFZWehY5GTl7JI99vTpzTE5mTrePMdblBGFmjZKWShplZu8lMyjnnEuW9LR0irKLKMouSug8dY11LYkkNrFUN1RTVV9FdX3wvserofX6+l3rW5ZjjzE6/8hBdkY2ORk5HD3yaJ688MmE/r54Eq0HFQNvS3odqGwuNLOzEjyvc871KFnpWQzKHcSg3EFJP7eZUdtYGz/BtHnFS0TDC4cnPSZIPEF8JylROOdcCpNEdkY22RnZDMgZEHU4LbqUIMLRWweb2cttyo8D1icjMOecc9Hqag/HbcDOOOXl4TbnnHO9XJfGYpI028w+2M62N82s/VHOuomkzcDqBE4xCIg/EE7q8e+iNf8+WvPvY7e+8F2MNrOSeBu62gext3vPcrp4zoS09wd2lKQ57Q1YlWr8u2jNv4/W/PvYra9/F11tYpoj6Yq2hZI+C8xNLCTnnHM9QVdrENcAj0u6iN0JoQzIAj6RjMCcc85Fq0sJwsw2AsdI+gjwgbD4STN7IWmR7X93Rx1AD+LfRWv+fbTm38duffq7SHjCIOecc31T9w7k4ZxzrtfyBOGccy6ulE8Qkk4JBx1cLum6qOOJkqSRkl6UtEjS25KujjqmqElKlzRf0l+jjiVqkvpLmilpiaTFko6OOqYoSfpq+P/kLUmPSMqOOqZkS+kEEQ5Z/nPgVGAicIGkidFGFakG4OtmNhE4CvhSin8fAFcDi6MOooe4HfhbOMT/VFL4e5E0nGAenDIz+wCQDpwfbVTJl9IJAjgCWG5mK8ysDngUODvimCJjZuvNbF64XEFwAeieYSJ7AUkjgNOBe6KOJWqSioAPA/cCmFmdme2INqrIZQA5kjKAXOD9iONJulRPEMOBNTHra0nhC2IsSaXAocBr0UYSqduAbwBNUQfSA4wBNgO/DZvc7pGUF3VQUTGzdcDNwHsEA5SWm9kz0UaVfKmeIFwckvKBx4BrzCzeoIx9nqQzgE1m5iMDBDKAw4BfmtmhBPO/pGyfnaRigtaGMcAwIE/SxdFGlXypniDWASNj1keEZSlLUiZBcphhZrOijidCxwJnSVpF0PR4oqSHog0pUmuBtWbWXKOcSZAwUtVHgZVmttnM6oFZwDERx5R0qZ4gZgMHSxojKYugk+mJiGOKjCQRtDEvNrOUnmfczK43sxFmVkrw7+IFM+tzvxA7ysw2AGskjQ+LTgIWRRhS1N4DjpKUG/6/OYk+2Gmf6IxyvZqZNUi6Cvg7wV0I95nZ2xGHFaVjgUuANyW9EZZ9y8yeijAm13N8GZgR/phaAXwm4ngiY2avSZoJzCO4+28+fXDYDR9qwznnXFyp3sTknHOuHZ4gnHPOxeUJwjnnXFyeIJxzzsXlCcI551xcniCc6wRJjZLeiHkl7WliSaWS3krW+ZxLVEo/B+FcF1Sb2bSog3Buf/AahHNJIGmVpJ9IelPS65LGhuWlkl6QtFDS85JGheWDJT0uaUH4ah6mIV3Sb8J5Bp6RlBPZH+VSnicI5zonp00T06ditpWb2WTgLoKRYAHuBO43synADOCOsPwO4GUzm0owplHzE/wHAz83s0nADuDcbv57nGuXP0ntXCdI2mVm+XHKVwEnmtmKcMDDDWY2UNIWYKiZ1Yfl681skKTNwAgzq405RynwrJkdHK5/E8g0sxu7/y9zbk9eg3Aueayd5c6ojVluxPsJXYQ8QTiXPJ+Kef93uPwqu6eivAj4R7j8PPAFaJn3umh/BelcR/mvE+c6JydmpFsI5mhuvtW1WNJCglrABWHZlwlmYfsfghnZmkdAvRq4W9LlBDWFLxDMTOZcj+F9EM4lQdgHUWZmW6KOxblk8SYm55xzcXkNwjnnXFxeg3DOOReXJwjnnHNxeYJwzjkXlycI55xzcXmCcM45F9f/B+AbYG+wPv5MAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LDcMSVr_l6C3"
      },
      "source": [
        "We have now trained two different models. Let's plot their metrics to see how that extra layer affected the results in the network performance."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cnk6_wuemCXO",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "509589b5-3153-454c-ad0c-3b3e67ed7e64"
      },
      "source": [
        "print('Old model:')\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])\n",
        "\n",
        "print('New model:')\n",
        "score2 = model2.evaluate(X_test_flatten, Y_test_class, verbose=0)\n",
        "print('Test loss:', score2[0])\n",
        "print('Test accuracy:', score2[1])"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Old model:\n",
            "Test loss: 0.26406407356262207\n",
            "Test accuracy: 0.9269999861717224\n",
            "New model:\n",
            "Test loss: 0.0948835015296936\n",
            "Test accuracy: 0.9750999808311462\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a-DHj7P7n1Eh"
      },
      "source": [
        "We have improved accuracy results on the test set. Even though there is still margin for parameter tuning, the reached accuracy is already pretty high. We could add more layers or change the number of neurons in each layer to see if we could boost even further the results.\n",
        "\n",
        "Let's now explore some techniques that will prove useful in the following tutorials."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9XyUx6ThaT47"
      },
      "source": [
        "## Saving/Loading Model\n",
        "\n",
        "Some networks require long training times (hours, days or even weeks), hence, it is essential to know how to save the models for using them in future times without the need of retraining them every time. \n",
        "\n",
        "We can save and load the trained model in different ways.\n",
        "\n",
        "The first way is to save everything into a single HDF5 file, which will contain:\n",
        "\n",
        "*   the architecture of the model, allowing to re-create the model\n",
        "*   the weights of the model\n",
        "*   the training configuration (loss, optimiser)\n",
        "*   the state of the optimiser, which permits us to resume training exactly where you left it off.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0KJp2Pj2aTOc"
      },
      "source": [
        "from keras.models import load_model\n",
        "\n",
        "model.save('my_model.h5')  # creates a HDF5 file 'my_model.h5'\n",
        "del model  # deletes the existing model\n",
        "\n",
        "# Load a compiled model identical to the previous one\n",
        "model = load_model('my_model.h5') "
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vD5h5qsUblji"
      },
      "source": [
        "An alternative method is to save the architecture and the weights independently.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dJXTYfTDcaWs"
      },
      "source": [
        "model.save_weights('my_model_weights.h5') # creates a HDF5 file 'my_model_weights.h5'\n",
        "\n",
        "with open('my_model_architecture.json', 'w') as f:\n",
        "    f.write(model.to_json()) # creates a json file 'my_model_architecture.json'"
      ],
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wPgCrD4Leony"
      },
      "source": [
        "If we want to load the model, without having to define it, we can use the `model_from_json()`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gXr0L8nqeipI"
      },
      "source": [
        "from keras.models import model_from_json\n",
        "\n",
        "# Model reconstruction from JSON file\n",
        "with open('my_model_architecture.json', 'r') as f:\n",
        "    model = model_from_json(f.read())"
      ],
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NzmL07T_dh7Z"
      },
      "source": [
        "Once we have a model defined, we can load the weights with the method `load_weights(file_path)`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yBahatkweEZx"
      },
      "source": [
        "model.load_weights('my_model_weights.h5')"
      ],
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nwXWMk1vdm48"
      },
      "source": [
        "Imagine that we create a slightly different version of the model. For instance, we design a standard classification network, but we modify the last layer, so it can now classify among more classes. Let's suppose that we have trained previously a model that shares the same architecture but with different output size. We can use the pre-trained network and load only the shared parts into our new model (all layers but the last one) by using the layer names. Keep in mind that loading weights by name can be done since you will face this situation when dealing with fine-tuning or transfer-learning problems."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KAGF3VbPdroU"
      },
      "source": [
        "model.load_weights('my_model_weights.h5', by_name=True)"
      ],
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UWyW0AMhGMDc"
      },
      "source": [
        "Additionally, you can refer to [Keras' documentation](https://keras.io/getting_started/faq/#how-can-i-regularly-save-keras-models-during-training) and check how to create a callback which saves our model every 100 steps."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mX2BJRnjIxmN"
      },
      "source": [
        "### Download Models / Save to Google Drive\n",
        "One of the problems you may face is that the models, or any other file, saved in Colab will not be there permanently. In some cases, you will want to store your model weights in a more lasting way. There are two ways to save your model. First, you can download any file to your computer. To do so, you can use the left-side menu in Colab, and follow the instructions on the image below.\n",
        "\n",
        "![Screenshot](https://i.ibb.co/yS5JgPd/Screenshot-2021-01-18-at-15-46-04.png)\n",
        "\n",
        "If you want to use any file you previously downloaded, you will need to manually upload your file using the Colab interface.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ez688r6XKFrn"
      },
      "source": [
        "Another way to save your weights is by using your Google Drive storage. This can be more convenient, as you can quickly load the models again from your Drive without having to upload manually the file. To do so, you can click on the folder with the Drive symbol highlighted in the following image.\n",
        "\n",
        "![Screenshot](https://i.ibb.co/PFGw6QR/Screenshot-2021-01-18-at-16-16-44.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xZ_A8BuHL1pi"
      },
      "source": [
        "After clicking on the Drive folder, the code below will appear in your Colab notebook. You need to run it and follow the instructions to have access to your Google Drive folder."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I-tn782jL7lM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e3dd6b3c-5cb0-412d-868a-795f9c1f940e"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zLZ7pcurNgkK"
      },
      "source": [
        "After the `drive` folder is mounted, you will see a new folder on your files section in Colab as in the following image.\n",
        "\n",
        "![Screenshot](https://i.ibb.co/NSM1RFK/Screenshot-2021-01-18-at-16-17-48.png)\n",
        "\n",
        "Now, you can save the model to your personal Google Drive and also load any model from it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C-0qOFZpNMmP"
      },
      "source": [
        "model.save_weights('/content/drive/MyDrive/my_model_weights.h5')\n",
        "model.load_weights('/content/drive/MyDrive/my_model_weights.h5')"
      ],
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nJVamZKmPDJU"
      },
      "source": [
        "Using your Google Drive storage is quite convenient. However, keep in mind that any file saved in your drive will also count towards your Google Drive storage limit."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-6v_S7oxfUQC"
      },
      "source": [
        "## Obtaining an Output of an Intermediate Layer\n",
        "\n",
        "Sometimes we need to check how features of intermediate layers look like. This can be used for extracting features, but also for debugging purposes. That is why we need to see how we can obtain information about any layer within the architecture.\n",
        "\n",
        "The easiest way is to design a new model that will have as final output our desired layer. We first define the model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dZa6sDFLgXoC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bce53eab-6014-404b-e7fd-28a6da295503"
      },
      "source": [
        "model = Sequential()\n",
        "model.add(Dense(64, activation='relu', input_shape=(784,), name='first_dense'))\n",
        "model.add(Dense(128, activation='relu', name='second_dense'))\n",
        "model.add(Dense(10, activation='softmax', name='final_dense'))\n",
        "\n",
        "model.summary()"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_3\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "first_dense (Dense)          (None, 64)                50240     \n",
            "_________________________________________________________________\n",
            "second_dense (Dense)         (None, 128)               8320      \n",
            "_________________________________________________________________\n",
            "final_dense (Dense)          (None, 10)                1290      \n",
            "=================================================================\n",
            "Total params: 59,850\n",
            "Trainable params: 59,850\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hoXvhyLzhKU2"
      },
      "source": [
        "Now, we specify the name of the layer in which we are interested in obtaining the output. In our example, it is the second dense layer (\"second_dense\")."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8Zu29bschSps",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "01e2bfcf-69cc-42c8-c60e-7d242c0751cc"
      },
      "source": [
        "from keras.models import Model\n",
        "\n",
        "layer_name = 'second_dense'\n",
        "intermediate_layer_model = Model(inputs=model.input,\n",
        "                                 outputs=model.get_layer(layer_name).output)\n",
        "intermediate_output = intermediate_layer_model.predict(X_test_flatten)\n",
        "final_output = model.predict(X_test_flatten)\n",
        "\n",
        "print('Second dense layer output shape: {0}'.format(intermediate_output.shape))\n",
        "print('Final output shape: {0}'.format(final_output.shape))"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Second dense layer output shape: (10000, 128)\n",
            "Final output shape: (10000, 10)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HFo0jHRqiJv2"
      },
      "source": [
        "Alternatively, we could also build a Keras function that returns the output of a specific layer:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "msvYp1c9iNEX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d8b7cb25-0c5f-4194-8e8c-ccaba33644f0"
      },
      "source": [
        "from keras import backend as K\n",
        "\n",
        "get_second_dense_layer_output = K.function([model.layers[0].input],\n",
        "                                  [model.layers[2].output])\n",
        "\n",
        "intermediate_output = get_second_dense_layer_output([X_test_flatten])[0]\n",
        "final_output = model.predict(X_test_flatten)\n",
        "\n",
        "print('Second dense layer output shape: {0}'.format(intermediate_output.shape))\n",
        "print('Final output shape: {0}'.format(final_output.shape))"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Second dense layer output shape: (10000, 10)\n",
            "Final output shape: (10000, 10)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iV22i-GFj5np"
      },
      "source": [
        "## A bit more about Callback Functions\n",
        "\n",
        "We have already talked a bit about callback functions and how to use them. You can check the complete list of available callbacks [in the documentation](https://keras.io/callbacks/).\n",
        "\n",
        "Controlling and managing callbacks is essential when training big models. Sometimes we want to train the network until convergence. However, we are not able to determine the number of epochs it will take. Keras provides a handy method for controlling the training loops, the `EarlyStopping` callback. This callback checks when the network is not \"learning\" anymore and allows us to stop the training at any time. \n",
        "\n",
        "Let's see an example of how to stop the training when the validation loss is not decreasing anymore:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GNShEzC-kdKN",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c8a0a92e-c754-4447-e0ef-d6b15d299b25"
      },
      "source": [
        "from keras.callbacks import EarlyStopping\n",
        "\n",
        "# Define the metric that will determine the earlyStopping.\n",
        "# 'patience' defines the number of epochs we wait before stopping training \n",
        "# when the monitored metric does not improve\n",
        "early_stopping = EarlyStopping(monitor='val_loss', patience=2)\n",
        "\n",
        "model.compile(loss='categorical_crossentropy',\n",
        "              optimizer='adam',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "model.fit(X_train_flatten, Y_train_class, validation_split=0.2, \n",
        "            epochs=100, batch_size=32, callbacks=[early_stopping])"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/100\n",
            "1500/1500 [==============================] - 4s 2ms/step - loss: 0.5244 - accuracy: 0.8434 - val_loss: 0.1490 - val_accuracy: 0.9558\n",
            "Epoch 2/100\n",
            "1500/1500 [==============================] - 3s 2ms/step - loss: 0.1316 - accuracy: 0.9598 - val_loss: 0.1198 - val_accuracy: 0.9655\n",
            "Epoch 3/100\n",
            "1500/1500 [==============================] - 3s 2ms/step - loss: 0.0876 - accuracy: 0.9719 - val_loss: 0.1213 - val_accuracy: 0.9659\n",
            "Epoch 4/100\n",
            "1500/1500 [==============================] - 3s 2ms/step - loss: 0.0704 - accuracy: 0.9784 - val_loss: 0.1000 - val_accuracy: 0.9713\n",
            "Epoch 5/100\n",
            "1500/1500 [==============================] - 3s 2ms/step - loss: 0.0558 - accuracy: 0.9822 - val_loss: 0.1036 - val_accuracy: 0.9711\n",
            "Epoch 6/100\n",
            "1500/1500 [==============================] - 3s 2ms/step - loss: 0.0442 - accuracy: 0.9858 - val_loss: 0.1023 - val_accuracy: 0.9729\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f4758bccc18>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "meMroifllkIN"
      },
      "source": [
        "The model stops training without reaching the 100 epochs. How convenient, right?\n",
        "\n",
        "We could also combine multiple callbacks when calling the fit module, learn more about it [here](https://keras.io/callbacks/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3w4AVBmLCDJe"
      },
      "source": [
        "## Freezing Keras Layers\n",
        "\n",
        "Keras also allows us to freeze some of the weights of specific layers. Freezing layers means that we can exclude them from training. A typical example where this proved useful is when fine-tuning a model. Remember that when fine-tuning a model, some layers are fixed, and normally only the last ones are trained to finetune the task-specific layers (e.g. classifiers) to the new task.\n",
        "\n",
        "To freeze layers we can set the trainable argument (Boolean) of the layer to be non-trainable or trainable:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q6x8F5hZDLWH"
      },
      "source": [
        "frozen_layer = Dense(32, trainable=False)"
      ],
      "execution_count": 34,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3s-5Bm8ODNe0"
      },
      "source": [
        "Additionally, we can access the layer arguments after initialisation. Therefore, we can set the trainable argument to `True` or `False` at any time. However, for this to take effect, we must call the `compile()` method after modifying any properties of the model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MTnjPeE7DjEs",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "78631a81-a3bc-4e7a-bc7b-f8fda7abed45"
      },
      "source": [
        "from keras.models import clone_model\n",
        " \n",
        "model = Sequential()\n",
        "model.add(Dense(10, activation='softmax', input_shape=(784,)))\n",
        "\n",
        "# We clone model in frozen_model\n",
        "frozen_model = clone_model(model)\n",
        "frozen_model.layers[0].trainable = False\n",
        "\n",
        "# with this model the weights of the layer will be updated during training\n",
        "model.compile(loss='categorical_crossentropy',\n",
        "              optimizer='adam',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "# in the model below, the weights of `layer` will not be updated during training\n",
        "frozen_model.compile(loss='categorical_crossentropy',\n",
        "              optimizer='adam',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "model.summary()\n",
        "frozen_model.summary()\n",
        "\n",
        "score_untrained = model.evaluate(X_test_flatten, Y_test_class, verbose=0)\n",
        "score_frozen_untrained = frozen_model.evaluate(X_test_flatten, Y_test_class, verbose=0)\n",
        "\n",
        "print('\\nTrain Model:')\n",
        "model.fit(X_train_flatten, Y_train_class, batch_size=32, epochs=5)\n",
        "\n",
        "print('\\nTrain Frozen Model:')\n",
        "frozen_model.fit(X_train_flatten, Y_train_class, batch_size=32, epochs=5)\n",
        "\n",
        "score = model.evaluate(X_test_flatten, Y_test_class, verbose=0)\n",
        "score_frozen = frozen_model.evaluate(X_test_flatten, Y_test_class, verbose=0)\n",
        "\n",
        "print('\\nBefore Training: Trainable Model Test Accuracy:', score_untrained[1])\n",
        "print('After Training:  Trainable Model Test Accuracy:', score[1]) \n",
        "\n",
        "print('\\nBefore Training: Frozen Model Test Accuracy:', score_frozen_untrained[1])\n",
        "print('After Training:  Frozen Model Test Accuracy:', score_frozen[1])"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_4\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "dense_5 (Dense)              (None, 10)                7850      \n",
            "=================================================================\n",
            "Total params: 7,850\n",
            "Trainable params: 7,850\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "Model: \"sequential_4\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "dense_5 (Dense)              (None, 10)                7850      \n",
            "=================================================================\n",
            "Total params: 7,850\n",
            "Trainable params: 0\n",
            "Non-trainable params: 7,850\n",
            "_________________________________________________________________\n",
            "\n",
            "Train Model:\n",
            "Epoch 1/5\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.4636 - accuracy: 0.8806\n",
            "Epoch 2/5\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.3036 - accuracy: 0.9146\n",
            "Epoch 3/5\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.2834 - accuracy: 0.9204\n",
            "Epoch 4/5\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.2727 - accuracy: 0.9233\n",
            "Epoch 5/5\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 0.2666 - accuracy: 0.9259\n",
            "\n",
            "Train Frozen Model:\n",
            "Epoch 1/5\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 2.3682 - accuracy: 0.1040\n",
            "Epoch 2/5\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 2.3682 - accuracy: 0.1040\n",
            "Epoch 3/5\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 2.3682 - accuracy: 0.1040\n",
            "Epoch 4/5\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 2.3682 - accuracy: 0.1040\n",
            "Epoch 5/5\n",
            "1875/1875 [==============================] - 3s 2ms/step - loss: 2.3682 - accuracy: 0.1040\n",
            "\n",
            "Before Training: Trainable Model Test Accuracy: 0.15070000290870667\n",
            "After Training:  Trainable Model Test Accuracy: 0.9236999750137329\n",
            "\n",
            "Before Training: Frozen Model Test Accuracy: 0.10790000110864639\n",
            "After Training:  Frozen Model Test Accuracy: 0.10790000110864639\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mLmeoTdqhmhX"
      },
      "source": [
        "We see in the model description that the frozen model has 0 trainable parameters, while it has 7,850 non-trainable parameters. This fact indicates that we successfully froze that model and the training does not affect it. We can verify this using the results after training both networks, where the trainable model improves accuracy while the frozen one does not change at all."
      ]
    }
  ]
}