02_neural_networks_python/01_introduction_mlp_with_solutions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MNIST handwritten digits classification with MLPs, by hand"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Author: Bethany Lusch adapting notebooks by Tanwi Mallick, Prasanna Balaprakash and Taylor Childers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Goal of notebook:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we'll train a multi-layer perceptron model (a basic kind of neural network) to classify handwritten digits. We'll build up the code by hand. Next week, we show how this can be done using existing Python libraries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Machine learning task:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial works through a supervised learning problem, specifically classification.\n",
    "\n",
    "Imagine you are making a machine for the post office that will automatically sort mail by zip code. The MNIST dataset contains thousands of examples of handwritten numbers, with each digit labeled 0-9. We will use deep learning to create a function that classifies each image of one number as a digit 0-9.\n",
    "<img src=\"images/mnist_task.png\"  align=\"left\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, the needed imports."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "import numpy\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MNIST data set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we'll load the MNIST handwritten digits data set. The first time we may have to download the data, which can take a while.\n",
    "\n",
    "<img src=\"images/MnistExamples.png\"  align=\"left\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MNIST is a popular dataset, so we can download it via the TensorFlow library. Note:\n",
    "- x is for the inputs (images of handwritten digits) and y is for the labels or outputs (digits 0-9)\n",
    "- We are given \"training\" and \"test\" datasets. Training datasets are used to fit the model. Test datasets are saved until the end, when we are satisfied with our model, to estimate how well our model generalizes to new data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We do some pre-processing on the images: convert from integer to float32 and normalize the pixels to be within 0 to 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 28, 28)\n"
     ]
    }
   ],
   "source": [
    "x_train = x_train.astype(numpy.float32)\n",
    "x_test  = x_test.astype(numpy.float32)\n",
    "\n",
    "x_train /= 255.\n",
    "x_test  /= 255.\n",
    "\n",
    "print(x_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training data (x_train) is a tensor of 60,000 images, each of size 28x28 pixels.\n",
    "\n",
    "For this notebook, we flatten each image to a vector, so x_train is a matrix of size (60000, 28*28)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 784)\n"
     ]
    }
   ],
   "source": [
    "x_train = x_train.reshape(x_train.shape[0], numpy.prod(x_train[0,:,:].shape))\n",
    "x_test = x_test.reshape(x_test.shape[0], numpy.prod(x_test[0,:,:].shape))\n",
    "\n",
    "print(x_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "y_train is a 60000-dimensional vector containing the correct classes (\"0\", \"1\", ..., \"9\") for each training sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "MNIST data loaded: train: 60000 test: 10000\n",
      "X_train: (60000, 784)\n",
      "y_train: (60000,)\n"
     ]
    }
   ],
   "source": [
    "y_train = y_train.astype(numpy.int32)\n",
    "y_test  = y_test.astype(numpy.int32)\n",
    "\n",
    "print()\n",
    "print('MNIST data loaded: train:',len(x_train),'test:',len(x_test))\n",
    "print('X_train:', x_train.shape)\n",
    "print('y_train:', y_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a closer look. Here are the first 10 training digits:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAABNCAYAAACi7r7XAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA7PUlEQVR4nO29d3Rc933o+flNL5gCDHoZ9F5IsJMiRbFIsmhLtpPYlsu6bOxd24nPxs9Zn3iTeP1233nZ+CU5LznZ5B1t7MS2FCeOLMV21FlEUSwSG0AQjQRAADOoMwMMBmUGwGDu/gHea4BiAQuIO/D9nMOjoyl3fl/87v39vr9vFZIkoaGhoaGhoaGxntGt9QA0NDQ0NDQ0NFYbTeHR0NDQ0NDQWPdoCo+GhoaGhobGukdTeDQ0NDQ0NDTWPZrCo6GhoaGhobHu0RQeDQ0NDQ0NjXXPfSs8QojvCSGefxCDUSuajMnPepcPNBnXC+tdxvUuH2gyqpUVKTxCiM8IIc4JIaaEEENCiNeEELtXe3ArQQjRK4SIXh/blBDizXu8jpplLBJCHBNCzAghOoQQB+/xOqqVUUYIsVcIIQkh/ss9fFe18gkh/m8hRIsQIi6E+N59XEfNMu4SQrwvhJgUQly613GpVUYhRKYQ4qdCiEEhxIQQ4qQQYvs9XkuVMsKDuVdVLt8xIURACBERQjQLIT56j9dRs4zrfr2RuZs9444KjxDiPwH/HfivQBbgBf4OuKebZJV4WpKklOv/nrjbLyeBjD8FLgIe4I+BF4UQGXdzgSSQESGEEfhr4L17+K7a5esCvg28cq8XULOMQog04FfAfwPcwPeBXwkhUu/yOqqVEUgBzgKbgTTgR8ArQoiUu7mIymWE+7xXk0C+/w3IkSTJCfwvwPNCiJy7uUASyLiu1xuZu94zJEm65T/ABUwBn7jNZ74HPL/k//8NGAYmgHeA2iXvHQLagElgAPjD66+nA/8BhIEx4ASgu93YllyzFzi4ks8mo4xABTALOJa8dgL46nqRccl1/4jFjfKfgP+y3uS7fo3nge+tw/v0I0DrDa9dAX53vch4i/FEgM3rUcZ7uVeTSb7r19kGxIBt61HGe5nDZJKRu9wz7mTh2QlYgJfv8LmlvAaUA5nABeCFJe/9APhfJUlyAHXA0euvfwvwAxksapL/ByABCCH+Tgjxd3f4zReumyjfFEJsuIuxgvplrAV6JEmaXPJa8/XXV4raZUQIUQj8z8D/dRdjlFG9fA+AZJBR3OT/6+5ivMkgo4IQYiNgYvE0vVKSSsZ7ICnkE0L8hxAixqJl4G3g3F2MNylkvE9UL+O97BmGO7zvAYKSJMVXekFJkn64ZEDfA8aFEC5JkiaAeaBGCNEsSdI4MH79o/NADlAoSVIXi1qefL2v3+EnP8viH1ewaKp8QwhRJUlSeIVDVruMKSxqzEuZAPJWOl7ULyPA3wB/KknSlBA37pt3JBnku1/ULuNpIFcI8WngReAzQClgW+l4Ub+MCkIIJ/AT4D9f/62VkjQy3iNJIZ8kSR+57g45CFRLkpRY6XhJEhnvk2SQ8a73jDtZeEJAuhDiTooRAEIIvRDi/xFCdAshIiy6m2DRbAXw2yyatvqEEMeFEDuvv/7fWDwlvSmE6BFC/NGKRg9IknRSkqSoJEkzkiT9GYumsT0r/T7ql3EKcN7wmpNF0+BKUbWMQoinWXTZ/esK5bkRVcv3gFC1jJIkhVj07f8nYAT4EHCYxdPbSlG1jEt+18pivNKZ62vO3ZAUMt4HSSOfJEnzkiS9BjwhhHjmLr6aNDLeB6qW8Z73jDv4x1zANPA7t/nM97juxwP+J6AdKGbR4uJm0TxVdsN3jMA3Ad9NrlcHjAIHbje224ynHXjmLj6vahlZjOGJsTyG5x3uPoZHzTL+dxZjIYav/4uyqOj9Yj3Id8P37sennhQyXv+uAegHnlxPMgJm4A0WzfX3EjOiehnv515NJvmWfP8w8M31KOO9zGEyyMg97hm3tfBIi6ao7wL/rxDiY0IImxDCKIR4Sgjx/Zt8xcFigG2IRVP2f5XfEEKYhBCfvW7imr8+2MT19z4ihCgTi3apCWBBfu92CCG8QohHrl/bIoT431nUKE/e6bvJIqMkSVeAJuD/vC7jx4EG4OfrRUbgT1lU7DZe//dL4P8DvrRO5OP6eCwsWlUN1+dSv5LvJpGMjdfH5AT+gsVF7Y31IqNYdIG8yOLi+gXp7twgSSGjLOe93qtql08IUXV9LNbr4/oc8ChwfCXyJYOM17+73tebe9szVqjtfZbFoK5pFrWpV4BdN9HyUoBfsOhu6QM+z3Utj8XgvtdZ9N1FWEzv3H39e99k0QQ2zaIJ/E+X/Pb/AP7HLcZVC1y6/r0QcATYcrfarJplvP5+EYuBdVGgk3vMSlOzjDeM85+4iyytZJDvukzSDf++uM5k/CmLi9YE8K9A5nq6T4G9168/w+JpUv63Z73I+KDuVbXKB1SzGKg8yWL4w1ng4+vpPv1NWW9uIu8d9wxx/cMaGhoaGhoaGusWrZeWhoaGhoaGxrpHU3g0NDQ0NDQ01j2awqOhoaGhoaGx7tEUHg0NDQ0NDY11j6bwaGhoaGhoaKx7bltFUQiR1ClckiTdsd60JqP6uZOM610+0GRMBjQZ1798oMmYDNxKRs3Co6GhoaGhobHu0RQeDQ0NDQ0NjXWPpvBoaGhoaGhorHs0hUdDQ0NDQ0Nj3bOi1u8a94der8dsNpOSkoLNZsPlcgEwMTHBzMwMU1NTzM7OsrCwsMYj1dDQSCZSUlJITU3F5XJhMpmYnZ1lamqKoaEh4vE4icRd9zfV0FAlKSkp5ObmApBIJPD5fMzOzt7VNTSF5yGQkpJCcXExe/bsYePGjTzxxBNIksRbb73FhQsXOHnyJD09PUQikbUeqoaGRhKxceNGPvaxj3Ho0CHy8/Pp6enh5MmT/Pmf/zljY2NMTU2t9RA1NB4IdXV1fPe730UIwezsLN/61rfo7u6+q2s8dIVHCIHBYECnW+5Ny8/PJz09HYPBgMFgwOFw4Pf7uXr1KrW1tWRlZVFYWIher2dhYYH+/n4GBwdpa2tjZmbmYYuxInQ6HampqZSXl/P0009TUVFBYWEhbrcbSZKor6/H4XCQnZ3NCy+8sO4VnvT0dNxuN4888gjBYJBTp04xMzNz11q6WrDb7VRXV5OXl0dRURFCCKLRKEeOHGF8fJxQKLTWQ9S4AXn90ev1GAwGsrOzsdlsmEwmAoEAfr8/KSytOp0Ou91OQUEBDQ0NpKenY7PZyM7OJisri7S0NMV6nIzo9XocDgdCCGUddbvdbN26FbPZzJ2aXg8PDxMKhTh58iSxWOyOn19rdDodaWlpOJ1OGhoaCAQCnDx5cq2HpQpMJhObNm1i27ZtFBUVMTIyQjgcvqfn9KEqPDqdDp1Oh9VqxWg0LnuvoaGBhoYGLBYLNpuNgoIC3n77bQKBAE888QRbtmzhySefxGQyMTc3x1tvvcXp06fx+/2qVXj0ej05OTls2bKFr33tax+Qe9OmTdTV1bF//35OnjxJR0fHGo529cnLy6OiooLvfOc7tLS00NXVxcjISNIqPE6nk4MHD7Jnzx4+9KEPodPpCAaDjI6OcuXKFU3hUSEGgwGr1YrZbMZisbBp0yY8Hg9ut5uLFy8yPDyMJEmqdwUZDAZSU1MpKipi8+bNpKSkIIQgIyODrKwssrKyGBsbW+th3jNGo3HZAbi2tpaSkhL+6I/+CKfTecf5ef/992lra6OlpYX5+Xni8fhDGvndo9Pp0Ov15ObmUlRUxFe/+lUuXryoKTzXsVqtHDp0iIaGBoqLiwkEAgSDQXUoPCaTCafTiV6vR6/X4/V6lQfTYrFgNpupra3F4/Es+57H48HpdKLT6ZAkibm5OQDS0tJ4/PHHKSoqIh6PMzU1xfDwMB0dHbS3txOLxR60CA+E3NxccnNz+f3f/33Ky8ux2+2KVWt2dpZEIkE8HsdgMGC32yksLKSsrAyfz8f8/PxDWXAtFosyLxaLhc7OTsbHx1ft93JycsjPz8fn8zE8PLxqv7Pa6HQ6tmzZQnV1Nb/9279Ndna2sknKJ0m1nyh/k3C73bhcLvbv309mZiZer5f09HRcLhdpaWmYTCaMRiN1dXUUFhby+uuv09fXt9bDvi0LCwtEIhEGBwdpbW2lurqatLS0tR7WfSHHIm3ZsoW8vDy2bdumeANyc3Nxu91YrdYVPVvl5eWkpqby5S9/mdbWVt58803m5uZUqfgYDAZsNhtf+MIX2LVrFx6PB7/fT2pqalJbwB8E+/bto6amhkOHDpGRkUE8Hsfv99PS0kI0Gr3r6z1QhUd2RRUVFSmaeXV1Nenp6dTU1Cgb65YtW8jMzLzldWKxGL29vbjdbkpLS3G73RiNRkKhEBMTE/T39+Pz+RgaGmJ+fv5BinBfyOZXWVsvLy9n27ZtZGVlYTAs/qllZW52dpZIJILL5cLj8ZCfn09paSlTU1PKv9XGaDSSk5ODw+HAbrfj8/lWVeFxOp3KQ3wvN6takBfgkpISysvLsVgsSJKEECJpFB35PjWbzRgMBoxGo/KazNTUFAsLCyQSCeUAIwfXq+m5uxmyTGazmby8PDIzM9m2bRt5eXmUlpaSkZGBy+VS5k12lU9PT3P27FnVu7YSiYSyhoyOjlJaWrrWQ7pvXC4X+fn5NDY2UlxczCOPPKJ4BdxuNxaLBVhcQ+/0nLndbsxmM1u3biWRSHDixAnlkKk2dDodRqORmpoatm7dyvj4uJLgIu8Vv6mUlJTQ0NBASUkJRqORubk5gsEgPp9PMYrcDQ9M4TEYDJSWlrJ7926+/e1vY7ValZOTTqdTNnydTofJZLrldSRJYmRkhOeee46pqSlisRhHjhwBIBQKEY1GiUQi+P1+gsGgahZeIQRms5n09HTy8vL4+te/zqZNmygqKlrmxpIkieHhYfr6+njppZd47LHH+OQnP8k3vvENPvOZz/DDH/6Q5uZm/uM//mPVx2y329m0aRPp6el4PB7a2toYGBhYld/S6XR4PB5yc3PJyclhdHQUIQRC3LHKueoQQpCenk5GRgYmk2mZkpAseDweCgoK2LVrF4WFhdTX15OWlkZOTg46nY65uTn+8R//EZ/Px9jYmBJDd+rUKfr6+ujo6FDl5gEoSk5DQwO7du3ikUceoaCgAJfLpRzEJicnGR0dJRqNYjQayc/Px+v14nA4OHHiBGNjY/h8PtUqPZIkEYvFCIfDDA0NqdbSfTc8+uijHDp0iEceeQS3243JZFLWhxtjPleC1WrlwIEDWCwWjh49Sm9vr+r/TnK8ksfjwePxEIvFmJ6eXuthrQlCCAoLC6mursZisTA5OUlHRwenTp3i2LFj96QIPlALTzweRwiBy+XC4XAoGvnNkCSJSCTCwsIC8XiclJQULBYLiUSC6elp2tvbmZmZURbVRCJBJBJhbm6OWCzGxMTEPWl4q4XNZqO+vp6CggLKysqoqKggOztbUfhu/KzFYmFmZoaJiQlCoRBOp5P09HSys7Pp7+9/KGM2m82UlpZitVrR6/WrpnwYDAZMJhO5ubl4vV5isRiTk5NMTU2pag5XQkpKCi6Xi5KSEgoKCtDpdMzPzzM7O0tnZyd9fX2MjIyoNljUaDSSkZFBRUUFGzZsoLGxkZycHEpLSzGbzZhMJgwGAxaLhQ0bNpCXl8fk5CSpqalkZ2czMTGByWSiq6tLVQqPfKgqLi4mPT2d6upqKioqaGhowOv1kpGRgcFgYGFhQZmrnp4e5ufnSUtLIzc3F5PJhMvlwm63Y7FYVK2My1YpudyFfKBMZubn54nFYpjNZqxWK4CyPwQCAaLR6E2tqE6nE4vFopT7WIrVasVisdw0UUaNyPOq1+vR6XSqvgfvBovFgt1uJy8vD5PJRFtb221LscjlFjIzM3G73cRiMUZHR7lw4QIDAwP3rLg+sKdkYWGBUChEKBRicnJSCQq8FfF4nJ6eHqanp5mZmaG8vJy8vDzi8ThjY2O88847qtfGl5KRkcFXvvIV6uvr2bRp0y0/J4RQ5LRarUQiEVpbW6mrq8NqtSqppQ8Dh8PBnj17CIfDXLt2bdV+x2w2k5qayrZt29i5c6cSI+H3+1ftN1eL3NxcSktLeeKJJygtLUWv1xOJRBgeHuav/uqvOH36NKFQSDWWxxtJSUlh586dHDhwgN/6rd/C6XQqJ+lgMEhrayspKSlYrVaeeOIJZeOBxXvX4XBw+fJljh07pqrn02QykZKSwic/+Unq6uo4cOCAEpy8lFgsxvDwMP/6r//KL3/5SwwGA/X19Tz++OOYzWaMRiNOp1MJAlYrcpZWRkYGRUVF2Gy2tR7SfTMwMEBLSwv79u1TXpubm2NycpJ33nkHn8+nvC7HOBoMBurq6sjKymLjxo1JodT8JuLxeCgtLeUTn/gE6enp/PEf/zGjo6O3PBjm5eWxadMmKisryc7OJhQK0dTUxI9+9KP72jcemMIjSRIzMzNcu3aNf//3f6ekpITs7GxGR0ex2+089thjiuk/FAoxMjLCP/zDPzA2NkYsFsPr9ZKTk0NDQwODg4Oqz5KQEUKQmZlJcXExtbW15OTkKAtlPB5X4mJ8Ph/19fUUFxczPT3N+Pg4fr+fWCzGzMwMubm5FBQUKHFAD4uH4Y6pqqpi//795OfnI0mSEouVTMgpvwcOHGDXrl3k5OQom2l/fz8nTpygp6eH8fFx5ubmVHX/yvdUVlYWJSUlPPvss5SWlmK32wkEAoTDYU6fPq2UeTCbzdhsNj796U/j9XopLCxU7slAIMDg4KDqXD1yaYAdO3ZQVlaGzWZbdm8nEgnC4TDt7e28+OKLvPfee4yNjZGWlpY0cVdLEUJgMpmw2WykpaUtc5vbbDYKCwuVw6dsSVc7PT09zMzMMD8/j9PpBFhmPb1VfOH7779PRUUFXq8Xu92uKOnxeJyrV6/S3t5Of38/k5OTD02W+0We3/WiwFVUVPDJT36SzZs3KzGuc3Nzt1R4CgsLefLJJ/F6veh0Ot566y2amprw+/33ZT1/oHbQubk5hoeHOX78OMFgkPz8fLq7u/F4POzcuROLxYJOp2N8fJze3l5eeeUVRkdHmZ2dJSsri4yMDObm5ohGo0mxCMnmx+zsbLxeL8XFxaSkpACLC+zc3JwSYN3U1ITb7SY/P59wOEwoFGJ0dJSJiQnC4TCHDh1S6gzJ5szV3DTlTfBhKFgFBQUcOHCA9PR0EokE4+PjqnX53Aq5VMK2bds4cOAAqampGAwGEokEQ0NDvP/++wwNDalSLr1ej8lkIj8/n+rqavbt26coBKOjo1y7do1f/OIX+Hw+Ojs7MRqNpKSksHnzZqxWK16vV3kex8bGGB4eVpVCByhuupqaGgoKCpTX5QDX+fl5gsEg7e3tvPzyy4yPjzMzM0NqaqqqLTm3Ql57TCaT4pKWkS3FPp+PYDDI9PR0Uig8g4ODDA8P4/P5FBddIpFgYWGBiYmJW8ZsOBwORkdH+cpXvqL8PWBRWert7aW3t5fR0VHVWl2XIj9ncqxrMsYH3oyCggL27dtHbm4u0WgUj8dDMBi86WeFEOTk5LBt2zYyMzOZm5vj7NmztLW13fI7K+WBO36DwSAnTpzg0qVL2Gw2JiYmyMvLo7y8nKqqKurr63nnnXd47733lNMwLFp9IpEIzz33nGqj6W8kIyODzMxM/uRP/oSqqipcLhd6vR5Jkujo6ODatWt8//vfJx6Pk5OTwxtvvEFzczNHjx5lYGCAjo4OhBAYjUaCwSAGg4EtW7YQjUapqKhgZGRkVbKmhBAUFBRQVFSkBMatJkajEbvdjsFgYH5+nqamJq5evbqqv/mgycrK4qmnnqKmpgaPx4NerycajSpBdG+88YZqT5BVVVVUVVXxta99jeLiYmWD6O3t5bnnnuPixYuKtTEej1NZWUllZSU7duygpKQEnU5HNBplcnKSU6dOqdLdXFFRwd69e3E4HMpr09PTTExM4PP5GBwc5Mc//jH9/f2MjIywsLCgLKzZ2dlJp/TE43GCwSCXL1/m1Vdf5SMf+QjFxcXA4un4S1/6EhkZGbjdbl5//fWkiJVLJBIkEglCodCy+ZAk6ZYKm06no7a2loaGBiWWZ+n3QqEQ4XCYubm5pDhEy6SkpFBZWcnY2BhDQ0NrPZz7ZmFhgVgspsyjHCt4I2azmYKCArxeL1lZWUq7FNndfL88cIUnHo8zMTFBNBrFYDAQi8UwmUyMjo6Sl5cH/PoGXppeGI/HicfjSZGuLKcRZmZmUlJSQkVFBUVFRcDiqWJhYYHBwUF6enro6upSTmOweIppaWkhFAopBROFEMTjcXQ6HS6Xi4yMDIqLi4lGo6um8KSlpeHxeDCbzat6ijAYDIqLRKfTEY/HGR8fV61ycDPkqq8VFRV4PB4l+HVmZoauri4lk0ltWCwW0tLSqKqqorGxkYqKCjIyMhQLa1NTE+3t7fT09CyzqmZlZVFRUUFaWpoSGxIOh+nt7WVwcJBgMKg6C4+cyCBb2eSEgGAwqIy7tbWVsbEx5aSv1+uxWq3Y7fY1Hv3dI0kS8Xic0dFRLl++zN69e5X3zGYzOTk5eDwe5RCWTKz0sGuz2bDb7RQXF1NQUPABF5AcUjA6Oqp6ZUeu45VIJBTrTmpq6m0zmpMBuYBkWloaFotFUXzkw9VShBBYrVaKi4vJzMxUigxPTU0xMTHxQKznqxbaPzc3p5wq5PoW8kKzdetWHA4Hv/rVr5iZmVH9zXgjVquVnJwcHnvsMXbu3KlkeEQiEaampohEIoqVa2JiglgsxsjIiJKGLSt7N8NkMlFcXMyzzz7LT37yk1UpgKbT6diwYQMbN25UMhhWA7kabEZGBjk5OVgsFmKxGGNjY0nTRkOn0+F0OikrK+PDH/6w4mqcnp6mv7+fF154QbXWqqKiIj71qU9x8OBBtm7disFgIBgM8stf/pITJ07w2muvKZmPS9m2bRuf+9znSE1NVV6TAwZbWlpUGX91+PBhzp8/zzPPPIPJZOLcuXNKEkU4HFYW2KXPnZxR6nQ6k87CI9PW1saVK1d45plnaGhoWOvhPFQqKiqorq7m6aefpri4WDlUyUxPT/PCCy+oPjlCNgBEo1FmZmaULMHc3FwlRCJZSU9P54tf/CK7d++mpKSEoaEh+vr6uHr16gcsV2azmezsbJ599lnq6+uxWq1cvXqVrq4ujh8/TiAQuO/xPJRcxqmpKd577z1sNhtlZWU4HA4KCwvxer0IIR6IIA8LnU5HRkYGe/fuZfPmzVRVVWEwGAiFQhw7doxQKEQgEKClpYXBwUFlkV2pUie7uGSNeDUQQuB2u3G73QghiMViq5JZ5HA42L17N1VVVaSkpCh1Q26m3asRq9WK0+nk8ccfZ+fOnRgMBmVx6uzspLW1lZ6eHtW1kDAajdTX1yt1aPLz89HpdAwMDHDt2jWOHTvGlStXmJycVOZBr9djs9nIyMggOzub1NRUjEYjMzMztLW10dTUREdHh2otczMzMyQSCc6ePYvBYKC/v5/p6Wmmp6eJRqM3dYnIrRgyMjKSVuGRLT1LC18mqyx3wmAwUFRURHp6OuXl5ZSXl+P1eqmqqiItLW2Z3JFIhFAolBRrTTweZ3Z2lkAgwNDQkOKaTPZ5tNlseDweGhsbycvLQ5IkmpubaW5u/kBMlk6nY/PmzVRXVyudGKampjh27Bjnz59ftlbdDw9F4YlEIrz55ptYrVbq6+upq6vD6XRSWVlJIpEgGAwmjZXHYDCQn5/PRz/6UWprayksLGRmZoahoSF++tOf4vf78fl8mEym2/qeb4fJZFIqha4GQghSU1OVgM3p6WmGh4cfeEVPl8vF008/TX19PSkpKQwMDCj1NJIhpsDhcJCXl8cXv/hFZRGCRYvlhQsXuHDhAleuXFHdgmqxWNi7dy9bt27l4MGDSu2Zrq4umpubeeWVV4hGo4qCu1TJrquro6CgALfbjcFgIBAIcOTIEU6ePMnly5fXWLJbI5vJ33333RV/R66YnZubm/Sby9Iinsmylt6JG+fEZDKxYcMGNmzYwMc//nEyMzOVYpJLkSSJYDDI0NAQ8Xhcde7XG5Hd4wMDA/T19eH1etd6SPeNEAKn00leXh7bt2/H6XSysLDAu+++yzvvvEMoFFq2BxgMBvbt28emTZtobGwkFosxPj7OSy+9xOHDhx/YuB6KwiOXQT979ixzc3P8wR/8AbW1tXzxi19U+mIEAgFCoRCnT59WpckcFifF6/VSXV3N5s2bcTqdzM/P84Mf/IBLly5x/vx55UQpm1bV/rDBYnzGlStX7quip5wxUlBQQHZ2tlJles+ePUqPn3fffZfz588zNjaWFOXSS0pKqK6upqSkhIyMDGCxVsjAwACvv/46HR0dqst+MZvNSv85ud2Az+fD5/Pxt3/7t1y5coWZmRll3HJV1y984QsUFBRQWVmppII2NzfT0dHByy+/zODg4FqKdV9s2bKFnJwcpd6QbAkxGAzU1NQopSTGx8eVPn09PT2qm9vbIVuRk1nZSUlJweFwsHHjRrKzsykrK1vmojKbzWzfvh2Px0N2dvayLCa58nRbWxvvv/8+ly5dwu/3L4vZ0ng4pKWlkZaWxpe//GVqa2tJS0vD7/fT3t7O2bNn6ezsXDYnDQ0NVFVV8fjjjytJEleuXOH48eMPfN15aOU5FxYWGBkZ4fz58/h8PrxeLzU1NWRnZ+NwOBgYGGB4eFg5McdisWXNGNWAXq+noKCA/Px8srKylNikCxcucPHiRUZGRu5LwVmrXkxyca+bLQxL+yvJtSGWprILIbBYLBiNRqxWK6WlpUoH54KCAiW+SZIk+vv76ejouKWLQS0sVd5KSkoUa5tcduHq1at0d3fj9/tVdX8CSpXkwsJCsrKykCSJQCDAlStXlGwsOW5L7qWWl5fHnj17KCgooLCwEFh8Xv1+P11dXVy9elUJsE8GhBBKdW+LxUJ1dTXFxcVK9WT5OdPr9eTk5Cjdt8fHx+nu7mZkZIRwOJwUh5WlJGvjWjmpIz09XSkg6PV6aWhoWBZwbTQaqa2txWg0fkBG2UrS19fHmTNnaG5uZnR0lFgslnR/D/h12ZBkqsMjV/7Ozs6msLCQnTt3UlZWhtVqJRaLMTAwwPT0tGLZkZ/FvLw86urqKC4uJiMjQ/nsxYsXH3is50OtRy5nT/zwhz/k/fff55vf/CZFRUWUlJQoViCn08nly5d5++23lfYDaiElJYWvfOUrVFVVIYTA7/fT29vL+fPnuXLlyn0rO2tlVpdL6t8s7S8zM5O0tDR0Oh0pKSlK+qfT6VS6/O7fvx+r1ap0uk8kEkqw9fz8vJLV5vP5VNeS4Gbk5+dTU1PD5z//eTZv3ozdbmdiYoKuri6ef/553njjDfx+vyqtVHq9XqkWLGcfHTlyhH/+538mFotRVFTEwYMHKSkpoaqqCq/Xi9vtJjMzc5lrIB6Pc/LkSS5evMj09LTq50xGbkvg9XrZvHkz+/btUxr4ys/Y0oOFyWRSlJ2jR4/y3HPP0d3dnXTKTjJjt9vJz8/n85//PE8++ST5+fnKIerGNfFWqckzMzM0Nzdz/PhxXn75ZaX4Z7LOo9FoVPqJJQNyMcGdO3fyO7/zOzz22GPKHiH3HtywYQPT09PU19fz85//nPn5eex2Oxs2bGDfvn04HA4ikQhvv/02r732Gi+//PIDX2MfqsIjx7TIvaJOnTpFfn4+5eXl2O123G43DQ0NWK1WAoEAPp+Pvr4+VVSudbvd5OTkUFBQQHp6OrBYhE0OjrzfDWHp6Swej980e+ZBsnQxyMzMVMqy31jYKS8vj/T0dMWSU1ZWhsViwWq1Mjc3p6SaT05OMj09TSQSYXJyku7ubpxO57IT2eTkpKpPzjqdDrPZTHFxMY8++ihFRUW4XC6lX1Y4HCYYDBIIBJifn1f1yXFhYYGFhQUl0HPnzp3Mz8/jcDjYvHkzOTk5FBYW4nK5lHRR+UQpW1h9Ph8DAwOqtsbBr9NZrVYrDQ0NZGRkkJeXR2VlJRUVFUo8UiQSUeLjlh4w5LmUA7flRVrN87uekPugORwOPB6P0u7kZtzqUChnhLrdbhwOB+FwOGmU9JshZ2klQ8kE2VJaUVHBI488Qnl5OR6PB51Op1jeDAYDmZmZ1NbWkpqayujoKPF4HJfLRU1NDVlZWQCMj49z+vTpVbMqr0nHuStXrnDt2jX8fj+NjY186UtforKykvz8fA4dOkQgEMBut3P69GkmJyeV9hNrSVlZGTU1NZSUlODxeJAkib6+Ps6fP//AJkZWBKanp+nt7V3V1O35+Xlloa+vr6esrEwJKF5Kbm6uovDICms8Hmdubg6fz6fEXY2OjtLd3c3Vq1fx+/2Mj49TXV2tpAnLmWzDw8OrJtP9YjAYyMjIYN++fXz7299e9t7c3BwjIyOqb4uxsLDA3NwckUhEscQ9++yzPPvsszfdwMfGxpTYLYvFQmpqqlLTpqWlhba2toctwl0hu0MyMzPJz8/ne9/7HpWVlXg8HhYWFpifn2dmZoZAIMClS5dITU2lsbHxA4XP5ADmHTt2EAgEiEQiWuzHGnA7S/ftArPtdjubNm3C5/Nx4cIFzp8/z+jo6KqPd7VITU1l69atiiKgZkwmE1u3bmXPnj18/etfV+ZI7jYwPDyslPYoKytjbm6OLVu2oNfrFQXXZrMptb7+/u//ftX2+zVrsSvH9Fy8eBFJkqipqVGaMtrtdnbt2qV0S33ppZfo6upaq6ECKDEPcmry1NQUPT09XLx48Z4VHr1eT3FxsaLhxuNx+vr6uHjxIr/4xS9WrYloPB7nzTffpKenR0lRd7lcRKPRD5yK5ubmGBoaoru7W0n1HB8fJxAIMDk5qdTViUajSpuMmZkZqqurFfeXJEnLqmqrFafTyYEDB6ioqPjAgjoyMsJbb7310DrZ3yuzs7OEQiGef/55NmzYwNNPP43ZbMZgMBAOhxXr29DQkKJU6/V6nnnmGbKyskhNTcXn8ynBzWpGr9fjdDopKCjgiSeeYPv27ZSUlLCwsMBbb71FIBBgeHgYv9+vuMdra2spKCggLS1NqXGi1+ux2+1UV1fjcDgIBAJYLBY6OztVYV1eKTempaenp1NSUrJq2Z4PilgsxtDQEK+++ip9fX1s2rQJg8FAX1+fYqmUWarwFBcXk5eXx8aNG5O+Xg2glI3YuXNn0hSLLCkpwev18ulPf1oJTZFr8L366qvKIbG6uppt27aRm5uLxWIhLy8PnU6HxWJR4kJtNhulpaV897vf5ezZs5w4ceKBezrWTOGR/eaTk5PK4ltVVcWOHTtIS0ujpqYGl8uFx+Ph5MmT9PT0rOnCo9frMRgM6PV6EokEMzMzDA4O0tXVdU9+Rnmyi4qKePTRR0lLS1Mqg7a3t3Pq1KlV03ITiQTnz5/H7/eTlZW1LFj1RoLBIAsLC5w6dYqRkRH6+voYGhq67cZvMBgoLi6moqICm83G9PQ04XBY1SdmIQQpKSls2rRpWe8oQCl3f/bsWdWfGmV36BtvvMH09DS7d+/GbrdjMpkIBAKMjo5y9uxZWltbOXfuHDMzMzgcDnbt2oXL5QJgdHSUq1evqjJGSUYIgdlsJjU1lcrKSvbt28eTTz5JIpFgYGCA06dPc+3aNbq6uujs7CQSiZCenq64VuUNUlbw5SB1r9er/F3kqs3yc7j0nlCbu+tm1o+0tDQKCgowm82qdtHNzc0RCoU4c+YM7e3tTExMoNfrOXfuHHNzc7d0TW3fvp2GhgYlJCLZCYVCyxrz3izmTG0UFBRQV1fHvn37lFprcmiDfKiWQxkyMzOV9h+y10BGkiTMZjO5ubl87nOfw2w2K8aEdaHwyMTjcaampmhpacHn8/GNb3xDKa+dnp6OzWajsbGRyclJ2tvbVbFpJhIJ5ufnmZiYIBQK3bWvWPZnPv300+zatYvHH38cIQQ+n4+XXnqJ1tZWpZDaahIKhfiXf/kXjEbjbU+BcmVh2ZV1pznQ6XTU1NRQXV2NTqfD7/dz6tQp1RXokxFCKOUGtm/fTk5OjvLe3Nwc58+f5/z58/T396vi/rsT8XicK1euMDg4yPHjx9Hr9QghFDemXDphZmaGoqIivF4vGzZsUArwyS0a1CqrTqfD4XDwiU98grq6Og4dOqQ0pm1paaG1tZWf//znjI+PE4lEMBgMFBQU8J3vfIfKykol3Xl2dpZXX32V6elp8vLy8Hq9lJaW8uyzz/Lkk09y5swZrl69yuHDh4nFYsrfIxaL0d/fr6pN6GZp6aWlpXg8HiorK5mdnWVwcFBVY76RSCTC9PQ0L7/8MoBShf9WYw6Hw1y8eJGDBw+SmZn5MIe6KoTDYUZHR0kkEkqmocvlIisri2AwqMqYpB07drBz504mJyfp6Ojg8OHDNDU10dXVxdDQELOzs4qBoKenhy984QvU19dTXl5+0wD0aDTKxYsX6ejoUAr3PkjWVOGRzVkpKSnKf2/0q8sBhGpifn6e8fHxm7qAbofRaFR6cHm9XiX90mQy0dnZSX9/v7JRPQxrltyLZzWw2+1KH6aJiQl6e3tV6yLR6XRUVFRQVVVFenq6clqMxWJMTk5y+fJluru71zyObKVIkkQ0GiUajd6xirnBYMBut2O327FarSQSCSKRCMPDw6pVeNLT08nMzKShoYGamhqKioqYmJigv7+f1tZW2traGBoaUk6G+fn5eL1e6uvrlSKDgUCAYDBIc3Mzk5OTBINBJicnkSQJh8NBamoqtbW1SmxBLBZjdnaWeDzO2NgYfr9fVcHc8uFLDsgGlMzJ3NxcRkZGGBoaUrXCI7uvVtoRe2Zm5qEcDB8WSw+TsmVHPoyqNT1dzszt6Oigt7eXixcvcunSJXp7e5d9LhwOMzw8vKxn3/T0NKOjo8useOFwmLa2Nvx+/6qEQKyZwiNnj3i9XrZv347X6yU7O1spegaLGv/g4CCXL1+mtbVVNQvM2NgYJ06cuOsutnJPqU996lNUVlZy8OBBfD4fJ06c4C/+4i84d+5cUlQGvVsGBwc5ceKEKhtswqI741vf+hYbN25cVqLe5/PR3d3NX/7lXzIyMrLGo1wd5Ng0+HXQfGdnp1IWQk3ItUmeeOIJtmzZwmc/+1mlxsfhw4c5efKkErsTDofJyMjA6/Xyu7/7uzQ2NlJfX8/8/Dx+v58XX3yRY8eOKe6rlJQUcnJyKC0tZc+ePZSUlLBr1y6qq6v56Ec/qpRbCAaDXLhwgXfffVc165EkSZw+fRohBB/72MewWq3Ke3q9Xuki39LSsq7WlqKiIqqrq9eFOwuWN9WWkZ9PtR36Zdra2giHw7zzzjuMjY0pFqobkUuf5OXlUVBQgF6vp6Ojg7/5m79haGhIsf7Pz88zMjKyaofjh6rwyF2z5QJDjY2NZGdnU1VVRWpqKi6XS3GtSJLE7OysUhBPTQ+qzWajpKQEp9O5os+7XC5KS0vZsGED5eXlbNmyBbfbrdTwOXr0KP39/aqOmbgfbvYgqw273Y7D4Vi2sHR3dyvNMpPFunO3BAIBnE6ncsKSXQhq2cyXkp6eTl5eHlu3bqWxsRGj0UggEOD8+fO8//77tLa2YjablUNUQUEBpaWlNDQ0kJ6eTktLCyMjI7S2tnLmzBl6enqUGIGpqSklVXZhYYGOjg5isRhZWVkUFRUBixaIEydOcPnyZVWtR/Dr3lE3G5der1ddEKzsssnJyVEC7Ve6RjgcDrxeL4888ghbtmxZ8TqsdoaHh7FarYRCIQwGg1JPasOGDQSDQVXuD11dXYyMjDAyMnLTgrKylSojI4OKigqlA/zk5CTDw8NcvnxZcWXC4jM2NTW1au67h6rw2Gw2srKy2L9/P9XV1ezbtw+Xy4Xb7V72Ofk0NTs7qwS7rvVmuTSAzOVyUV9fj8fjWdH35GajTz31FFu3blUK2b355pu89tpr/PSnP30IEmjcCp1Oh81mw2KxLIsZaG9v57333lvVB3CtGRgYAFClgnMjOTk57Ny5k71791JbW0ssFqOvr49/+7d/o729nYGBAWpqavB6vRw6dEgpJWE0GpmamlLi444ePcr4+Piyoqazs7OMjo4yOjpKR0cHVquVYDBIeXk5jz76KLDocvjJT35Cd3e36u4HuUZUMswjLCphFouFyspKIpEIkUhkxYp2Wloa27Zt49ChQzz66KOr1mT5YdPX10c0GmVoaAir1YrD4aC8vJzdu3fz3nvvEQ6H13qIH6C1tfW27+t0OqxWK7m5uTQ2NpKRkYHRaGR4eBifz0dTU9PDGeh1Vl3hcTgcuN1uNm7cSFVVFY2NjUqdDI/HsyxwSZIk/H4/w8PDHDlyhK6uLtra2ujs7FztYd6RpUGBBoMBt9vN/v370ev1nD17VknHdrvdZGVlUV9fT3Z2tlLHpry8HJvNxtTUFJ2dnfj9fl577TXa29vXWrRVQ1YQ7XY7OTk5SvVlNbF37162bdtGZmamYt2RsyLk1gpq29weJCUlJZSXl3+gAaMaqaio4Nlnn1XicHQ6HXl5eXz84x/nqaeeIh6PU1RUhNPpJDMzE7vdjtFo5MSJE3R2dvKzn/2M0dFRpXDk7ZidneXMmTO0tLRw/PhxYNHld+3aNaanp9f8AHYj586dY2RkhE984hPk5eWRmpoKLLpEHn30UVJTU3nxxRcZGxtbU1dlWloaOTk57N69m+LiYnbs2MHFixeVEgk31gGTsVgs2Gw2pdzFZz7zGYqKipZloM3Pzysp0WqzwK2U+fl5xYKYlpbGu+++y89+9jPVJnzcCZfLxcGDB9mzZw8f+tCHsNvt+P1+/uzP/mxNanytyion91/S6/VkZWWRnZ2t+NC3bdtGenq6opXLN+rs7CyxWIxr165x7do13n33XXp6elSh7NyIfDopLS1VirQFAgEmJiaW9RHxer0UFxdjs9lwOBwEg0GlUee1a9fo6OhYcYBeMmMymXA4HKoyq8sm9ZKSEsXqtnThlLv1jo2NJe3iuRLS0tLIzMxU1dzcCrfbTVlZmRKzodPpcDqdVFVVYTKZMBqNeDwe9Ho98/PzxONxAoEAbW1tNDc309nZydTU1IqUlUQioRTJ7O7uXlW5HgQjIyNIkkQ4HCY1NVVReIQQ5OfnEw6HSUlJWfNWPU6nU3nmqquraWxsZG5uTqnMe6vWEW63W6ncXl9fT319PRaLRamCLmfNhsNh1ffqux1yCQy5UvTAwADt7e2qr2F2M3Q6nVLfqqKigsLCQsWKeubMGcW6/DB54AqPyWTC6XSSlZVFZmYmX/3qVyktLaWkpESJOF+6uEYiEXp6ejhx4gRNTU2cPHmSsbExZmZmVH2y1ul0NDY2UldXx969e4nFYkxNTSnFEuVKrnq9nkgkQktLC11dXfT29vLjH/+YwcFBotHout5Ml24sa9kr7GakpKQomXJbt27FZrMp5Qba2tqUmiDhcFh1p/kHicfjIScnJykUnlgsRigUwm63YzablX5DDocDWNwsmpqaGB4eprW1lZ6eHtrb2/H7/UQikRUrO8nKwsICfX19igthacVbtTRi3rp1K9/4xjcoKytTekV5vV4+//nPE41Gb7mx19fXk5OTQ3V1NTabDavVihCCRCJBe3s7fX19vPbaa1y+fJlz586pNsNwpSyNpVNDSMfdIvfPKisr45lnniE7O1sJrm9qasLv969JxfoHovDIXVIrKipwuVxkZGSQnZ1NRkYG5eXl5ObmKkXN4NdNRH0+HyMjI3R0dNDU1ERnZydDQ0NK/QU1Ibe4GB8fx2g0KsXcTCYTWVlZilXAbrcrQaDz8/MMDQ3h8/k4ffo0AwMDSoDXaraNUBOSJGGxWEhLS7vl6W0t0Ov1ip/c4XCg0+lIJBJEo1H6+/t57733CAaDSbnY3A1ykbClFXrVit/v5+2331Zc4g6HQ7HIjY2NMTExQWtrK8FgkJ6eHgYHB+nr62NiYoLZ2dl1PY+wWGbi2rVrylqrxjmVs3WsVqvSL0u23MTj8ZsecoUQFBUVkZaWpmRRzs/PK7E/cpHJ1tZWBgcHk9IaciOyl8TtdpOdnb0iN6ya0Ol0bNq0Seltl0gk8Pl8XL58mUuXLq1ZF/sHovBYrVYyMzP5vd/7PQoLCykqKiI9PX2ZkrMUv99Pf38/L7zwAj09PTQ1Nd22oqYakM1vXV1dLCwsUFZWBiw+jPIJc+kEyq0Wjhw5wvvvv88PfvCDdb/g3gq5Gq6ayr/L9WccDofSIHRubo7x8XHOnTvHj370o7Ue4kPB5/NhNptZWFhQdUVXWGw2fOHCBaWMRVVVFYlEgnA4zJkzZ2hra1OyOpNpc3hQxGIx3n77bYQQfPjDH17r4ayYzMxMnnrqqRV9VgjB7Oys0uuts7OTv/7rv8bn86kyi+lekdudlJaWsmXLFo4fP67qHn43YjQa+epXv0pjYyM5OTl0dHRw/vx5Xn755YceqLyUe1J4TCaTUjOnsrJS6ai9fft2XC6XUj5aRu4ALgeBHj16lO7ubi5fvqycvtTu2pmenmZoaIgf//jHlJWVsWPHDqqrq5e1ZJCrJXd1ddHc3Izf7+fSpUuqL/i1WqjNjXUjS4OUl/KbNFdjY2MMDQ0xOjqK0WjE4XBgt9vxeDxMTk6q6hASj8eJRqN0dHQwMDBAR0eHUr5ieHhY6bvzmzR/S0kkEgQCAYaGhhgaGsLlcqmuRk1HRwfPP/88O3bsoLCwkNra2ttafqPRKNPT00pMlpzU0tzcTHd3N4ODg0ln/bgdRqORuro6ioqKiMfjTExMMDIyoqrncCXIVc+NRiPZ2dmEQiFaWlrWvLbXPSk8RqOR3NxcNm3axP79+5US5hkZGUrlxUQioUyS3CK+t7eX06dP8/rrr3P16tU1M2vdC7FYjLm5OQ4fPkxPTw/xeBybzbasJ4gkSfh8Ps6ePav0EVFbRdaHxdKmf2pUeuR7NJFIsLCwkBRZSquBXGU4EAjgcDhISUkhJSWF9PR0xT2glmdUni+fz7fWQ1EliUSCsbExRemRD5FCCGKxmCriePr6+njzzTfR6XRMTU1RXFyM3W5f9vzJ40wkEkxOThIIBJidnSUajdLc3ExXVxdHjhxhaGhItcVM7xWDwUBhYaHSTHpycpLx8fGkVHiuXr2K3W5n586dSn++tQ6av6dV3uFwsHv3bnbs2MGjjz6qdGPW6XTMzMwwMjJCV1eX0iIhGAzy2muvEQqFlAj0ZFJ2ZOSGp5cvX6avr4+f//znSvsEmWg0ytTUFJFIhNnZ2d9IZWepdr9//35VznM0GqWvr48rV67Q2tqaNKnZq8HMzAwvvvgiW7du5bOf/azSSPQHP/gBvb29qu/BpLHIwsICgUCAw4cPc/nyZQwGg3JPx2Ixent719wSEg6HmZ6eJhgM4vF46OjooK6ujqeeegqr1Yper+fChQsMDw/T1tbGtWvXaG9vJxKJEIvFiEajzM7OEolE1lyW1WB+fp7W1lYmJyfJyckhHA4nZfxZPB7nlVde4ejRo/zoRz9icnKSUCikFBhcK+5phZebSY6OjtLV1bXsPVnhkc2NCwsLhEIhmpublZs1mZED66anp5W0VY3lSJLEwMAADoeDc+fO0dfXR19fn6p6aclNa7u7u7lw4QKTk5MYDAaCwSCDg4NrPbyHSjwe5+rVq2RkZBCJRJSYq5KSEqXUu9orZWssMj8/TzgcVmWROvj1+im7IOX4zczMTCwWCwaDgYsXLyoKT39/P93d3etWwbmR+fl52tvbFVfz4OCgYp1LNuTaQX6/f41H8mvE7RYxIcQt35QtOjemsy4t0CdP0lqVqpck6Y6+lNvJmAyoVUadTqc0f13q4ryXTfNOMt6PfHLpgKVuyYWFhYdqQl7rORRC4HQ62b59O3/4h39IeXk52dnZHDt2jKamJr7//e8rcRT3ylrL+DDQZLx7+fR6/Qf2kaUuraX7yMNADXMo984SQii9FR/kYUMNMq42t5Lxnm34yeZT1Hi4yPEWar9PbpUK+5uEJElKm4Zf/epXSkB+dnY2paWluFwupR6IhsaDRI710+6tX6P9LVaP38ygBQ0NjWXMzs7S3d3NP/3TPxEMBolEInzsYx9Dr9eTmppKLBZb8wwLDQ0Njfvhnl1aycBvsuluKetdxvUuHzwcGYUQSjsYj8dDdnY2s7OzNDU1EYvF7iv+Ti0yriaajOtfPtBkTAZuJaOm8Ggyqh5tkdVkTAY0Gde/fKDJmAzck8KjoaGhoaGhobEe0K31ADQ0NDQ0NDQ0VhtN4dHQ0NDQ0NBY92gKj4aGhoaGhsa6R1N4NDQ0NDQ0NNY9msKjoaGhoaGhse7RFB4NDQ0NDQ2Ndc//D/uVLAc1Sqw5AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x72 with 10 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pltsize=1\n",
    "plt.figure(figsize=(10*pltsize, pltsize))\n",
    "\n",
    "for i in range(10):\n",
    "    plt.subplot(1,10,i+1)\n",
    "    plt.axis('off')\n",
    "    plt.imshow(numpy.reshape(x_train[i,:], (28, 28)), cmap=\"gray\")\n",
    "    plt.title('Class: '+str(y_train[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's begin with a simple linear model: linear regression, like last week. \n",
    "We add one complication: each example is a vector (flattened image), so the \"slope\" multiplication becomes a dot product.\n",
    "\n",
    "Note, like before, we consider multiple examples at once. input_images is a matrix where each row is an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def linear_model(W, input_images):\n",
    "    # f(x) = xW returns m-length vector, where m is the number of examples\n",
    "    return numpy.dot(input_images, W)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like last week, the loss is mean squared error (MSE):\n",
    "\n",
    "$\\large{MSE = \\frac{1}{n} \\sum_{i=1}^{n} (y - \\hat{y})^{2}}$ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(W, input_images, true_labels):\n",
    "    predicted_label = linear_model(W, input_images)\n",
    "    MSE = numpy.mean((true_labels - predicted_label)**2) \n",
    "    return MSE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Update weights using gradient decent \n",
    "$\\large{W = W - \\eta \\frac{\\partial J(W)}{\\partial W} }$,  where, $W$ is the network weight, $\\eta$ is the learning rate and $J(W)$ is the objective function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def learn(input_images, true_labels, current_W, learning_rate=0.000001):\n",
    "    # first we need dJW/dW where JW = MSE \n",
    "    n = input_images.shape[0] # get number of examples to average over\n",
    "    label_predictions = linear_model(current_W, input_images)\n",
    "    # calculate gradient: one entry per partial derivative for an entry in vector W\n",
    "    dJW_dW = (2./n) * numpy.dot(input_images.transpose(), label_predictions - true_labels)\n",
    "    # now we update W\n",
    "    new_W = current_W - (learning_rate * dJW_dW)  # gradient update step\n",
    "    return new_W"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In practice, we usually don't use all of the training data to calculate each step. We use a random subset. This makes the steps faster and noisier. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# At the moment we take the simple route and use a fixed subset. \n",
    "batch_size = 100\n",
    "\n",
    "x_train_batch = x_train[:batch_size, :]\n",
    "y_train_batch = y_train[:batch_size,numpy.newaxis]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f4439ff7220>]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAYjElEQVR4nO3da5Bc5X3n8d+/rzM996s0ICEhIBjZJgLLoCwuFuzYwSRlhw2VwpvyarOuUipOapPdrUrhZGs3W5Xdyia12dgpX0KCY1444FzsMkU2wVjAGq8dicEWICOEhBhJoMuMZjSae1+ffXFO9/Tc0GhuR0/391PVdU4/58zM/5ka/frR00+fY845AQD8E4u6AADAyhDgAOApAhwAPEWAA4CnCHAA8FRiI39Yd3e32759+0b+SADw3ksvvXTBOdczv31DA3z79u3q7+/fyB8JAN4zs5OLtTOFAgCeIsABwFMEOAB4igAHAE8R4ADgKQIcADxFgAOAp7wI8P1HzutLzx+PugwAuKp4EeDPHx3SX77wVtRlAMBVxYsAj5lULHHjCQCo5keAx0wl7hwEAHP4EeBmKjECB4A5vAjweMxEfgPAXF4EuJlUZAoFAObwIsDjZnIEOADM4UWAx8xYhQIA83gS4GIOHADm8SPAYyZJTKMAQBU/AtyCAGcaBQBmeRHg8XAETn4DwCwvAjwcgPNpTACo4kWAx608AifAAaDMiwBnDhwAFvIjwJkDB4AF/Ajw8hw4CQ4AFV4E+OwqFAIcAMq8CHArz4ET4ABQ4UWAl1ehkN8AMMuLAC/PgbMKBQBm+RHgzIEDwAJ+BHj5gzyliAsBgKvIZQPczLaa2XNm9pqZ/cTMfits7zSzZ8zsWLjtWK8i42GVjMABYNZyRuAFSf/JObdT0h5Jv2FmOyU9LGm/c+4mSfvD5+tTJB+lB4AFLhvgzrmzzrkfhfvjko5IulbSJyU9Fp72mKRfXKcaK8sICXAAmHVFc+Bmtl3SbZIOSNrknDsbHjonadMSX7PPzPrNrH9oaGhFRc5ezGpFXw4ANWnZAW5mzZL+XtJvO+fGqo+54FY5i8arc+4R59xu59zunp6elRXJMkIAWGBZAW5mSQXh/XXn3DfD5vNm1hce75M0uD4lsowQABaznFUoJulRSUecc39SdehJSXvD/b2Svr325QVYRggACyWWcc5dkj4t6VUzOxS2/a6kP5T0N2b2GUknJf3yulQolhECwGIuG+DOue9LsiUOf2Rty1kcF7MCgIW8+CTm7MWsCHAAKPMiwGdvqRZxIQBwFfEjwJkDB4AF/AjwyioUAhwAyrwI8Dg3NQaABbwI8MonMZlCAYAKTwKcT2ICwHx+BThzKABQ4VeAk98AUOFHgLOMEAAW8CPAmUIBgAW8CHCWEQLAQl4EOMsIAWAhTwKci1kBwHxeBTi3VAOAWV4EOHPgALCQFwEeDsBZhQIAVbwI8Dg3NQaABbwI8Bi3VAOABbwI8MoInCkUAKjwIsATYYAXCHAAqPAiwMsj8EKRAAeAMi8CPBkPymQEDgCzvAjw8gi8WOK29ABQ5keAh6tQ8kyhAECFFwEei5lixkfpAaCaFwEuSYl4jDlwAKjiT4DHjDlwAKjiTYDHY8YcOABU8SbAk/EYc+AAUMWbAI/HjDlwAKjiTYAnYqZCkTlwACjzJ8DjxhQKAFTxJ8BjLCMEgGreBHgwB84UCgCUeRPgwRw4I3AAKLtsgJvZV81s0MwOV7X9vpm9Y2aHwsf961smc+AAMN9yRuBfk3TfIu3/2zm3K3z8n7Uta6E4c+AAMMdlA9w59z1JIxtQy7tKMAcOAHOsZg78N83slXCKpWOpk8xsn5n1m1n/0NDQin8Yc+AAMNdKA/zLkm6QtEvSWUn/a6kTnXOPOOd2O+d29/T0rPDHMQcOAPOtKMCdc+edc0XnXEnSX0i6Y23LWigeiylPgANAxYoC3Mz6qp4+IOnwUueuFS4nCwBzJS53gpk9LukeSd1m9rak/yrpHjPbJclJGpD0a+tXYoA5cACY67IB7pz71CLNj65DLe+KOXAAmMubT2KyDhwA5vImwJOsAweAObwJ8Dhz4AAwhzcBnkzElOeGDgBQ4U2Ap+IxZQsEOACUeRPg6SQBDgDV/AnweEy5QknOMQ8OAJJHAZ5KBKXmeSMTACR5GOA53sgEAEkeBXg6EZckZfPFiCsBgKuDNwHOCBwA5vInwONhgLMSBQAkeRTg6WRQKksJASDgTYAzAgeAufwJ8AQjcACo5k2AV1ahFFiFAgCSRwFeWYXCCBwAJHkU4GkCHADm8CbAmQMHgLm8CXBG4AAwlzcB3pgM3sSc5qP0ACDJowDPpBOSpOkcAQ4AkkcBXh6BT+YKEVcCAFcHbwI8HjM1JuOaYgQOAJI8CnBJakrHNZllBA4AkmcBnkklGIEDQMizAGcEDgBl3gU4I3AACHgV4E3pBKtQACDkVYBnUnFNZRmBA4DkWYA3pRKayjMCBwDJswDPpOOaZAQOAJI8C/C2xqTGpvNyzkVdCgBEzqsAb29MqVByGmcpIQB4FuCZpCRpdDIfcSUAED2vArwjk5IkXZzKRVwJAETPqwCvjMCnGYEDwGUD3My+amaDZna4qq3TzJ4xs2PhtmN9ywy0hyPwUUbgALCsEfjXJN03r+1hSfudczdJ2h8+X3cd4Qj84iQBDgCXDXDn3Pckjcxr/qSkx8L9xyT94tqWtbi2xjDAp5hCAYCVzoFvcs6dDffPSdq01Ilmts/M+s2sf2hoaIU/LpCIx9SRSerCRHZV3wcAasGq38R0wadqlvxkjXPuEefcbufc7p6entX+OG1ua9S5SzOr/j4A4LuVBvh5M+uTpHA7uHYlvbvNrWmdGyPAAWClAf6kpL3h/l5J316bci6PETgABJazjPBxST+UdLOZvW1mn5H0h5I+ambHJP1s+HxD9LU1aHgyp2yBi1oBqG+Jy53gnPvUEoc+ssa1LMvm1gZJ0uBYVls7M1GUAABXBa8+iSlJm9uCAD8zOh1xJQAQLe8CvDzqPjUyFXElABAt7wJ8S0ejEjHTWxcmoy4FACLlXYAn4zFd15nRwDABDqC+eRfgkrS9u0knhghwAPXNywC/vrtJJ4enVCpxazUA9cvbAJ/OF3WWT2QCqGNeBvh7NrdIkl4/OxZxJQAQHT8DvK9VknSEAAdQx7wM8OZ0Qtu6MnqNAAdQx7wMcEna2deq184Q4ADql9cBPjA8pYlsIepSACAS/gb4NcE8OKNwAPXK2wDftbVdkvSjUxejLQQAIuJtgHc1p7Wju0n9AwQ4gPrkbYBL0u3bOvSjUxcV3JYTAOqL1wH+gW0dGpnMaWCYS8sCqD/eB7gk9Q+MRFwJAGw8rwP8xp5mdWSS+uGJ4ahLAYAN53WAx2Kmu27s1gvHLjAPDqDueB3gknT3TT0aGs/q6PnxqEsBgA3lfYB/6KZuSdL3j12IuBIA2FjeB/g17Y26sbdZ//eNoahLAYAN5X2AS9K9N/fon08Ma2wmH3UpALBhaiLA73tfn/JFp/1HzkddCgBsmJoI8Nu2tmtza4P+8dVzUZcCABumJgI8FjPd977Nev6NIS4vC6Bu1ESAS9L97+9TrlDS04cZhQOoDzUT4B/c3qHtXRl9o/901KUAwIaomQA3M/3yB7fq4FsjOjE0EXU5ALDuaibAJenB27coHjNG4QDqQk0FeG9rgz56yyY9cfC0JnkzE0CNq6kAl6R9/3KHLk3n9cSLjMIB1LaaC/Dbr+vQHds79egLJ5QvlqIuBwDWTc0FuCT9+j036MylGT1x8FTUpQDAuqnJAL/n5h7dcX2n/vS7x/hgD4CaVZMBbmb63ftv0fBkTl9+/njU5QDAulhVgJvZgJm9amaHzKx/rYpaC7u2tuuB267VI987oTe42QOAGrQWI/B7nXO7nHO71+B7ran//PO3qDmd0O/83SsqlrjlGoDaUpNTKGVdzWn9/ifeq0OnR/Vnzx6LuhwAWFOrDXAn6Ttm9pKZ7VvsBDPbZ2b9ZtY/NLTxd835xE9fo39127X6/P5jeuEYd+0BUDtWG+Afcs7dLunjkn7DzO6ef4Jz7hHn3G7n3O6enp5V/rgrZ2b6gwfep5t6m/XvH/8x10kBUDNWFeDOuXfC7aCkb0m6Yy2KWmuZVEKPfHq3Ymb69KMHdX5sJuqSAGDVVhzgZtZkZi3lfUkfk3R4rQpba9u7m/S1X71Do1M5/cpfHtC5S4Q4AL+tZgS+SdL3zexlSQcl/YNz7p/Wpqz18f4tbXr0335QZ0en9eBXfqCBC5NRlwQAK7biAHfOnXDO/XT4eK9z7r+vZWHrZc+OLj2+b48mswU98KX/xxubALxV08sIl3LrlnZ987N3qaclrb1fPagvPnecdeIAvFOXAS5J13c36VufvUs/f+s1+uOnj+qXvvwDHR9khQoAf9RtgEtSUzqhLzy0S59/aJcGhid1/xde0B/90+tcAAuAF+o6wKVgnfgnd12r7/yHu/Xx923Wl55/U/f88fP6+oGTyhaKUZcHAEsy5zZu7nf37t2uv/+quubVAodOj+oPnnpN/ScvanNrg/bdvUMP3bFVmVQi6tIA1Ckze2mx600R4ItwzumFYxf0xeeO68BbI2rPJPVLt2/Rv77zOt3Q0xx1eQDqDAG+Qv0DI/qrHwzo6cPnVCg57dnRqQc/sFUfe+8mtTYkoy4PQB0gwFdpcHxGf9v/tp548ZROj0wrFY/pnpt79Ild1+jem3vVlGaKBcD6IMDXiHNOh06P6smXz+gfXjmrwfGsUvGY7tzRqQ+/p1cffk+vtnU1RV0mgBpCgK+DYsnpxYER7T9yXs++Pqg3h4KP5u/oadJdN3Rrz44u7dnRqa7mdMSVAvAZAb4BTg5P6tnXB/X80SG9ODCiqVywDPGnNjVrz44u3Xl9l3Zd165r2hpkZhFXC8AXBPgGyxdLevWdS/rnE8P64ZvD6h+4qOl8EOg9LWndtrVdu65r166t7bp1S7uamUMHsAQCPGL5YklHzo7px6dGdeh08HgrvBpizILL3e7sa9Utfa3aeU2rdva1qrclzUgdwJIBzrBvgyTjMd26JRht7w3bLk7mdOjtUR06NarXzo7p0OlRPfXK2crXdDWldEtfq27pa9GNvc3Bo6dFbRmWLwIgwCPV0ZTSvTf36t6beyttl6bzev3smF47O6Yj4faxH55UrlCqnNPdnNaNvU26sbdZN/Q0V7abWxsUizFiB+oFAX6VaWtM6s4dXbpzR1elrVhyevvilI4PTsw+hib07UNnND4ze+GtVCKmrR2N2tbVpOs6M9rWFTyu62zS1s5GpRPxKLoEYJ0Q4B6Ix0zbupq0ratJH7llU6XdOaehiayOD07orQuTOjU8pZPDUzo5MqUDJ4Y1mZu9GJeZ1NfaoK2dGV3b0ahr2xt1Tfi4tr1BfW2NfBgJ8Az/Yj1mZuptaVBvS4P+xQ3dc4455zQ8mdPJ4SmdGpkMtsNTOjUypQMnRnRubGbBTSzaM0n1tQWBXg73vrbg+/e2prWptYHVMsBVhH+NNcrM1N2cVndzWh/Y1rHgeKFY0uB4VmdGp/XO6LTOjM7ozOi0zoxO6+2L0zr41ojGZhZeFz2TimtTa4N6W9LqbW3QppZ0Jdx7WtKVY83pBCtogHVGgNepRDxWGWUvWJsUmsgWdO7SjAbHZzQ4ltX5sRkNjs9uX317VN8dy1bWt1dLJWLqbkqpszmlrqa0uppT6m5Oq7Mppa6mqv2wvSHJ/DxwpQhwLKk5nagsX1yKc04T2YLOj2UrQT84PqPhyZyGJ3IanshqeDKn44MTujCRVbZqNU21TCqurjDsOzJJdWRSagu37Zmk2jMptTfOPm/LJNXCKB91jgDHqpiZWhqSamlIvmvQS0HYT+WKQbBPZme388J+aCKrN85P6NJ0/l1vbxePmdobk5WA78gk1dYYbNszSbU2JtXakFRLQ2LBflMqTvjDewQ4NoyZqSmdUFM6oeu6Msv6mlyhpEvTeV2azuniVF6jU3ldnMrpUrgdnc5rdCqn0am83hmd0WtnxnRxKr/otE61mEktDUm1NiZmg70hCP3F94NtUH9czemEGpO8CCBaBDiuaqlETD0tafW0XNkVHWfyRY3PFDQ2kw+20/nK8+r98rGxmbxOjUxVjo0v48bWMQtujN0cvigF+/HK8+ptc9Xx8rktc74uoTgfwsIVIsBRkxqScTUk41cc/GXFktNEOfCrgn4yV9BEtqjJbEGT2YLGZ4JtuX1iJq8L4zlNlNtmCiqUlne9oYZkTJlUMLLPpMqPhDKpuBqrnjem4soky22JOefOnld1PBnnE7o1igAHFhGPmdrCN0tXwzmnbKEUBn5RE9lCEO7ztuX9qVxR07mipnJFTeWLms4VdG4sX2mbzBU0nSsu+0WhLJ2IqSk998WhMRVXY/hCN/uIqSFZbo/NPZaIqTEVn3M8nYhX2hoSMSXisVX9vnBlCHBgHZlZJQC71vB+2LlCKQj1/LzQzxXCbRD+lf387LHqcy9M5DSTD47P5EvKhvtX+gJRloybGhJxpZNxNaZiakjMBn56kReHxmRc6URM6XCbSsSC54ly++x+6l3aEzGry/cjCHDAQ6kwuNq0PlemLBRLmglfJGbyRWULRU3nSpopBM+nc0XNFEqayRerHqXwhSDYL7eX28ZnChoazy44d6mlpVciZgoCPTn7AjD7YhCbcyxVfhFY0F71AlH1YpJKxJSKB98vGbfZ9rCt/EjGTal4bENfSAhwAAsk4jE1x2MbcukE55zyRadsIQjzbCH4n0C2UFKu/LxQVDY/u3/59rnHcoWSLk7mlvz+ueLqX0TKUvHZUK/e/x8PvF93XN+5Zj9HIsABRMzMlEoEI9uWiGoolZxyxVIY+LMvJLkw3HOV/aJyBTe3rVCcfV50886d/R5N6bX/tDEBDqDuxWKmhlg8vKSDPzdM4S1jAPAUAQ4AniLAAcBTBDgAeIoABwBPEeAA4CkCHAA8RYADgKfMuZVdtGZFP8xsSNLJFX55t6QLa1iOD+hzfaDP9WE1fd7mnOuZ37ihAb4aZtbvnFvq/rs1iT7XB/pcH9ajz0yhAICnCHAA8JRPAf5I1AVEgD7XB/pcH9a8z97MgQMA5vJpBA4AqEKAA4CnvAhwM7vPzI6a2XEzezjqelbDzL5qZoNmdriqrdPMnjGzY+G2I2w3M/tC2O9XzOz2qq/ZG55/zMz2RtGX5TCzrWb2nJm9ZmY/MbPfCttruc8NZnbQzF4O+/zfwvbrzexA2LdvmFkqbE+Hz4+Hx7dXfa/Phe1HzeznIurSsplZ3Mx+bGZPhc9rus9mNmBmr5rZITPrD9s27m/bOXdVPyTFJb0paYeklKSXJe2Muq5V9OduSbdLOlzV9keSHg73H5b0P8P9+yX9oySTtEfSgbC9U9KJcNsR7ndE3bcl+tsn6fZwv0XSG5J21nifTVJzuJ+UdCDsy99Ieihs/4qkXw/3PyvpK+H+Q5K+Ee7vDP/e05KuD/8dxKPu32X6/h8l/bWkp8LnNd1nSQOSuue1bdjfduS/gGX8gn5G0tNVzz8n6XNR17XKPm2fF+BHJfWF+32Sjob7fy7pU/PPk/QpSX9e1T7nvKv5Ienbkj5aL32WlJH0I0l3KvgUXiJsr/xdS3pa0s+E+4nwPJv/t1593tX4kLRF0n5JH5b0VNiHWu/zYgG+YX/bPkyhXCvpdNXzt8O2WrLJOXc23D8naVO4v1TfvfydhP9Nvk3BiLSm+xxOJRySNCjpGQUjyVHnXCE8pbr+St/C45ckdcmzPkv6U0m/I6l8i/cu1X6fnaTvmNlLZrYvbNuwv21uanyVcc45M6u5tZ1m1izp7yX9tnNuzMwqx2qxz865oqRdZtYu6VuS3hNtRevLzH5B0qBz7iUzuyficjbSh5xz75hZr6RnzOz16oPr/bftwwj8HUlbq55vCdtqyXkz65OkcDsYti/Vd69+J2aWVBDeX3fOfTNsruk+lznnRiU9p2D6oN3MyoOm6vorfQuPt0kall99vkvSJ8xsQNITCqZRPq/a7rOcc++E20EFL9R3aAP/tn0I8Bcl3RS+m51S8IbHkxHXtNaelFR+53mvgnnicvu/Cd+93iPpUvhfs6clfczMOsJ3uD8Wtl11LBhqPyrpiHPuT6oO1XKfe8KRt8ysUcGc/xEFQf5geNr8Ppd/Fw9KetYFk6FPSnooXLFxvaSbJB3ckE5cIefc55xzW5xz2xX8G33WOfcrquE+m1mTmbWU9xX8TR7WRv5tR/0mwDLfKLhfweqFNyX9XtT1rLIvj0s6KymvYK7rMwrm/vZLOibpu5I6w3NN0hfDfr8qaXfV9/l3ko6Hj1+Nul/v0t8PKZgnfEXSofBxf433+VZJPw77fFjSfwnbdygIo+OS/lZSOmxvCJ8fD4/vqPpevxf+Lo5K+njUfVtm/+/R7CqUmu1z2LeXw8dPytm0kX/bfJQeADzlwxQKAGARBDgAeIoABwBPEeAA4CkCHAA8RYADgKcIcADw1P8HuCdC10BesqkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "num_features = x_train.shape[1] # this is the number of pixels\n",
    "\n",
    "# Randomly initialize W\n",
    "W = .01 * numpy.random.rand(num_features,1)\n",
    "\n",
    "# now iterate num_iters times, with the step size defined by learning_rate\n",
    "learning_rate = 0.0005  \n",
    "num_iters = 5000\n",
    "losses = numpy.zeros(num_iters,)\n",
    "\n",
    "for i in range(0, num_iters):\n",
    "    # all the magic here\n",
    "    W = learn(x_train_batch, y_train_batch, W, learning_rate)\n",
    "    losses[i] = evaluate(W, x_train_batch, y_train_batch)\n",
    "plt.plot(losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check results so far"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, this simple linear model f(x) = xW is not very accurate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAABNCAYAAACi7r7XAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA+rUlEQVR4nO29eXRb132o+23MIIiBAOcBnGeREjVLlmxLnmIldpKmSZQ4N05uk7WSNLm3venKa97ty8u7w+trXtv7mpeX9nrVaezYTZM6dobGsyzLsiRbIymKk0RSJAGOAEgQHACSIM77AzgnpExJlMQBVM63FpYongNwb+x99v7t3ygkSUJFRUVFRUVF5W5Gs94NUFFRUVFRUVFZbVSBR0VFRUVFReWuRxV4VFRUVFRUVO56VIFHRUVFRUVF5a5HFXhUVFRUVFRU7npUgUdFRUVFRUXlrkcVeFRUVFRUVFTuelZU4BFCPCeEGBRChIQQl4UQX7rOff8ghJhc8JoRQkzc6uesF0KITUKI14QQfiHEDRMZCSEqhBC/EkL4hBCjifdVXnPPnwohhhL9/ZEQwri6Pbhhe41CiKeFEL1CiAkhRKMQ4tHr3HuzcXxbCBFZcL1j7XpyfYQQXxdCnE2098fLfM8RIYQkhNAtce2+xLX/tuKNvU2EEEVCiJeFEGOJufWD67T9gBCiWQgRFEIEhBAvCSHyFlx3CiF+lrjmF0I8L4SwrW1vPsgtzlOjEOJ/CCEGEt/HD4UQ+gXXb3k+rAUr/Cw6E2M7lfi8z65dT27MLewby1p3hRDliXXnudVr9fK5lfl1o71ACLFXCHE6MRcuCiH2rXrjl8ly1/pE/7oT/RtIPJe6BdePivheGRJCNAkhPrqiDZUkacVeQC1gTPxcBQwB25bxvh8DP7rTz1mrF1AJ/BHw0fhXeMN7dybudQJ64L8C7QuuPwIMJ/qcBrwN/F/r2DcL8F2giLhA/BFgAii6jXF8G/jSeo/XEu38A+BjwN8DP17G/U8A7wASoLvmmh5oBN4D/tt6921Bu15OjIcJyAaagf+wxH1ZQG7iZyPwPeDXC67/EHgdsAF24E3gb5Ogf8uep8D/DhxPPIMZibH6P253PiRjH5d477XP4k+BnwGpwD5gHKhd7z4m2ras9X65625ivh4Hnlvvvt3K/OIGe0Fi7gaATwJa4HPAGJC23v1LtG9Zaz1QCjgW9Okt4D8tuF4vr7HArsR8z1mpdq6ohkeSpBZJkmbk/yZepTd6jxDCAnwCeOZOPmctkSSpQ5Kkp4GWZdx7WpKkpyVJGpUkaQ74H0ClEMKVuOVJ4OlEn8eIC0RfWK223wxJkqYkSfquJEk9kiTFJEn6N+AqsO1G71tqHJMVSZJelCTpl8QXkBsihLAT3zC/dZ1bvkl8gW1fsQauDMXAzyVJikiSNAS8SnwhXYQkScOSJA0s+NU8UHbN5/xSkqSQJEnjwEtLfc5ac4vz9DHg+4ln0Ad8H/j3Cz5r2fNhLVmpZ3HB//83SZImJUl6F/g18O9WtwfLY7nr/XLWXSHEYSAIHFmFpt4WtzC/brQX7AWGJEn6V0mS5iVJeg7wERemNgySJHVJkhRM/FcAMRasN5IkXZQkKSr/l/iBsmCl/v6K+/Ak1MXTxDeAQeInzRvxCeID984dfs5G4V7iE1ee/LVA04LrTUDWAoFoXRFCZAEV3Fy4W3Icgb9MqKBPCCHuX/kWrjr/J/GT2dC1F4QQhcQ3zv+y1o1aBv8PcFgIkZIwUT1KXOj5AEIItxAiCISBPyOu5ZH5/4CPCCHShBBpxMf5ldVs+O2wjHkqrvk5PyHMbhju4FmsAKKSJF1ecE8TSSC4yqzEep8wtf4X4D+tcPPWipvtBeKa+wWwaS0atkyWtdYLIT4rhAgBfmAz8D+vuf5vQogI8D5xzdHZlWrgigs8kiR9DbAC+4EXgZkbv4MngWelhA7rDj4n6RFC5BPfQBY+kKnE1csy8s/WtWrX9Uj4OTwPPCNJ0s00GEuN4/8ClAB5wFPAb4QQSaOpuxlCiO3APcD/e51bvk/i1Lx2rVo27xBfQEOAl/ii8culbpQkqU+SJAeQDvwFi7VV5wED8dNpgLgG6Ier1ejbYRnz9FXgPwohMoQQ2cB/SPw+Za3aeKfc4bOYSnweLGScJFhjZFZovf+vxDUk3pVs2xpyo73gFJArhPiMEEIvhHiSuBYsWebwstd6SZL+WZIkG3FB/B+Im/EWXv8I8T4fAl6XJCm2Uo1clSithMrtXSAf+Or17hNCuIH7gWfv5HNWGyHEEwucsW7rdCuEyCBu+vihJEk/XXBpkrh/hIz88wTriBBCA/wEmAW+fpN7lxxHSZLelyRpQpKkGUmSngFOEJ/ESU+i/z8E/uMCFevC648BVkmSfrbmjbsJiba/SnzjsBAXZNKAv7rR+yRJGiVuBvnVAkfCnwOXiS9ANqALSApnUFj2PP3vwAXivlYniQt+c1yz0CYrK/AsXrvGkPj/uq4x13In670QYgvwIHGXgY3KdfeChEXgo8QPy8PAh4j70yWFcHc7a70kSVeIays/cICSJGlOkqRXgIeFEI+vVDtXOyxdx419b/4dcEKSpO47/JxVRZKk5yVJSk28loySuBEJU8DrxJ1B//s1l1uIq/VkNgPDC0xea44QQgBPE3do/UTC9+hGLHccJT6olk1WbMB24GdCiCHgTOL3XiHEfuABYHsiomII+DTwJ0KIX61PcxfhBNzADxILUAD4J5YnbOqATH632G4B/mfCn2SS+IksKYTW5c5TSZLCkiR9XZKkPEmSSohrqs6t5MlxtVihZ/EyoBNClC/43WaW4YO4TtzOen8/cefuvsTz+GfAJ4QQ51e2aavKDfcCSZKOSZK0Q5IkJ/FxrgJOr30zl8Vy1/qbjfXK7v3SCnk/E18kDxNXy2mJe5xPAY/f4D0dwL+/089Z6xfxgTQBNcQH1kQiymCJe23EJ+UPrnP9Q8T9Q2oAB3Gv9XWL0kq06R+IR7KkLvP+pcbRkRg7U2LSPpEYx4okGD9dol1/SfzkbOKD0VeCeHST/NqRGOs84iYe6zXXf0b8dOlc7/4l2t8N/Hmirw7izsb/vMR9f0A8+kVDPILp58D5BdePEjfpmROvHwIn17t/tzJPE2OWmxjT3YAHePhW5kOy93HB/R94FhO//xfikVoW4mbapIjSupX1/kbrLnHTzsLn8a+BF4CMJOjjsuYXN9kLgAbiTrw24j56J9a7b4l2OVjmWg98CchM/FxDXMj728T/q4j7GpoT/fwcca3m1hVr6wp2OgM4RtxDPkQ8DPbLiWtu4uo694L79yS+FOtyPydZXsRPEtI1r54F118B/tfEz08mrk8lvoPJJb4LWU0ZIn4SX1J4WqO+FSbaG7mmvU/cxjieIa42DxJftB9a77FLtO27S4zfd5fq3xJjvuRGSDwMOJnC0rcQd/gbI+4c+HMgK3FtEtif+PkbxCN/pogvtv8CFC74nGLgN8S1IqPETWXlSdC/Zc9T4oECPcA0cYHgieXMh43Ux8T9Sz6LiWtO4qa8KaAP+Ox69y/RrmXvG9xk3V1iTJMlLH3Z6w032AuIC6zjidfPSAgO6/3iBms9cZ+syQX3/lOif1OJZ/L/BkyJa9XEHZXlzzkDfHwl2yoSf0hFRUVFRUVF5a5FLS2hoqKioqKictejCjwqKioqKioqdz2qwKOioqKioqJy16MKPCoqKioqKip3ParAo6KioqKionLXo7vRRSHEhg7hkiTppomP1D4mPzfr493eP1D7uBFQ+3j39w/UPm4ErtdHVcOjoqKioqKictejCjwqKioqKioqdz2qwKOioqKioqJy16MKPCoqKioqKip3PTd0WlZZGbRaLUajkdTUVFJSUrDb7QCMj48zPT3N5OQkMzMzzM/Pr3NLVVRUNhKpqamkpaVht9sxGAzMzMwwOTnJ4OAg0WiUWCzpC8KrqCyL1NRUcnNzAYjFYng8HmZmZm7pM1SBZw1ITU2luLiY/fv3s2XLFh5++GEkSeKNN97g/PnznDhxgu7ubkKh0Ho3VUVFZQOxZcsWPvaxj3Ho0CHy8/Pp7u7mxIkT/NVf/RWjo6NMTk6udxNVVFaETZs28Z3vfAchBDMzM3zzm9+kq6vrlj5jzQUeIQQ6nQ6NZrE1LT8/n/T0dHQ6HTqdDqvVitfr5cqVK9TW1pKVlUVhYSFarZb5+Xn6+voYGBigtbWV6enpte7GstBoNKSlpVFeXs5jjz1GRUUFhYWFOBwOJEmirq4Oq9VKdnY2zz///F0v8KSnp+NwOLjnnnvw+/2cPHmS6enpW5bSkwWLxUJ1dTV5eXkUFRUhhCAcDnPkyBHGxsYIBALr3USVa5DXH61Wi06nIzs7m5SUFAwGAz6fD6/XuyE0rRqNBovFQkFBAfX19aSnp5OSkkJ2djZZWVk4nU5Fe7wR0Wq1WK1WhBDKOupwONixYwdGo5GbFb0eGhoiEAhw4sQJIpHITe9fbzQaDU6nE5vNRn19PT6fjxMnTqx3s5ICg8HA1q1b2blzJ0VFRQwPDxMMBm/rOV1TgUej0aDRaDCbzej1+kXX6uvrqa+vx2QykZKSQkFBAW+//TY+n4+HH36Y7du388gjj2AwGJidneWNN97g1KlTeL3epBV4tFotOTk5bN++na9+9asf6PfWrVvZtGkTBw8e5MSJE7S3t69ja1efvLw8Kioq+Pa3v01zczOdnZ0MDw9vWIHHZrPx4IMPsn//fj70oQ+h0Wjw+/2MjIxw+fJlVeBJQnQ6HWazGaPRiMlkYuvWrbhcLhwOBxcuXGBoaAhJkpLeFKTT6UhLS6OoqIht27aRmpqKEIKMjAyysrLIyspidHR0vZt52+j1+kUH4NraWkpKSvjzP/9zbDbbTcfn9OnTtLa20tzczNzcHNFodI1afutoNBq0Wi25ubkUFRXxla98hQsXLqgCTwKz2cyhQ4eor6+nuLgYn8+H3+9PDoHHYDBgs9nQarVotVrcbrfyYJpMJoxGI7W1tbhcrkXvc7lc2Gw2NBoNkiQxOzsLgNPp5KGHHqKoqIhoNMrk5CRDQ0O0t7fT1tZGJBJZ6S6sCLm5ueTm5vL1r3+d8vJyLBaLotWamZkhFosRjUbR6XRYLBYKCwspKyvD4/EwNze3JguuyWRSxsVkMtHR0cHY2Niq/b2cnBzy8/PxeDwMDQ2t2t9ZbTQaDdu3b6e6uppPfOITZGdnK5ukfJJM9hPl7xMOhwO73c7BgwfJzMzE7XaTnp6O3W7H6XRiMBjQ6/Vs2rSJwsJCXn31VXp7e9e72Tdkfn6eUCjEwMAALS0tVFdX43Q617tZd4Tsi7R9+3by8vLYuXOnYg3Izc3F4XBgNpuX9WyVl5eTlpbGl770JVpaWnj99deZnZ1NSsFHp9ORkpLCk08+yd69e3G5XHi9XtLS0ja0BnwlOHDgADU1NRw6dIiMjAyi0Sher5fm5mbC4fAtf96KCjyyKaqoqEiRzKurq0lPT6empkbZWLdv305mZuZ1PycSidDT04PD4aC0tBSHw4FerycQCDA+Pk5fXx8ej4fBwUHm5uZWsgt3hKx+laX18vJydu7cSVZWFjpd/KuWhbmZmRlCoRB2ux2Xy0V+fj6lpaVMTk4qr9VGr9eTk5OD1WrFYrHg8XhWVeCx2WzKQ3w7kzVZkBfgkpISysvLMZlMSJKEEGLDCDryPDUajeh0OvR6vfI7mcnJSebn54nFYsoBRnauT6bnbinkPhmNRvLy8sjMzGTnzp3k5eVRWlpKRkYGdrtdGTfZVD41NcWZM2eS3rQVi8WUNWRkZITS0tL1btIdY7fbyc/Pp6GhgeLiYu655x7FKuBwODCZTEB8Db3Zc+ZwODAajezYsYNYLMbx48eVQ2ayodFo0Ov11NTUsGPHDsbGxpQAF3mv+H2lpKSE+vp6SkpK0Ov1zM7O4vf78Xg8ilLkVlgxgUen01FaWsq+ffv41re+hdlsVk5OGo1G2fA1Gg0Gg+G6nyNJEsPDwzz11FNMTk4SiUQ4cuQIAIFAgHA4TCgUwuv14vf7k2bhFUJgNBpJT08nLy+Pr33ta2zdupWioqJFZixJkhgaGqK3t5cXX3yR+++/n0996lN84xvf4LOf/Sw/+tGPaGpq4t/+7d9Wvc0Wi4WtW7eSnp6Oy+WitbWV/v7+VflbGo0Gl8tFbm4uOTk5jIyMIIRAiJtmOU86hBCkp6eTkZGBwWBYJCRsFFwuFwUFBezdu5fCwkLq6upwOp3k5OSg0WiYnZ3ln/7pn/B4PIyOjio+dCdPnqS3t5f29vak3DwARcipr69n79693HPPPRQUFGC325WD2MTEBCMjI4TDYfR6Pfn5+bjdbqxWK8ePH2d0dBSPx5O0Qo8kSUQiEYLBIIODg0mr6b4V7r33Xg4dOsQ999yDw+HAYDAo68O1Pp/LwWw288ADD2AymXjrrbfo6elJ+u9J9ldyuVy4XC4ikQhTU1Pr3ax1QQhBYWEh1dXVmEwmJiYmaG9v5+TJkxw9evS2BMEV1fBEo1GEENjtdqxWqyKRL4UkSYRCIebn54lGo6SmpmIymYjFYkxNTdHW1sb09LSyqMZiMUKhELOzs0QiEcbHx29LwlstUlJSqKuro6CggLKyMioqKsjOzlYEvmvvNZlMTE9PMz4+TiAQwGazkZ6eTnZ2Nn19fWvSZqPRSGlpKWazGa1Wu2rCh06nw2AwkJubi9vtJhKJMDExweTkZFKN4XJITU3FbrdTUlJCQUEBGo2Gubk5ZmZm6OjooLe3l+Hh4aR1FtXr9WRkZFBRUcHmzZtpaGggJyeH0tJSjEYjBoMBnU6HyWRi8+bN5OXlMTExQVpaGtnZ2YyPj2MwGOjs7EwqgUc+VBUXF5Oenk51dTUVFRXU19fjdrvJyMhAp9MxPz+vjFV3dzdzc3M4nU5yc3MxGAzY7XYsFgsmkymphXFZKyWnu5APlBuZubk5IpEIRqMRs9kMoOwPPp+PcDi8pBbVZrNhMpmUdB8LMZvNmEymJQNlkhF5XLVaLRqNJqnn4K1gMpmwWCzk5eVhMBhobW29YSoWOd1CZmYmDoeDSCTCyMgI58+fp7+//7YF1xV7Subn5wkEAgQCASYmJhSnwOsRjUbp7u5mamqK6elpysvLycvLIxqNMjo6yjvvvJP00vhCMjIy+PKXv0xdXR1bt2697n1CCKWfZrOZUChES0sLmzZtwmw2K6Gla4HVamX//v0Eg0GuXr26an/HaDSSlpbGzp072bNnj+Ij4fV6V+1vrha5ubmUlpby8MMPU1pailarJRQKMTQ0xN/+7d9y6tQpAoFA0mgeryU1NZU9e/bwwAMP8Ad/8AfYbDblJO33+2lpaSE1NRWz2czDDz+sbDwQn7tWq5VLly5x9OjRpHo+DQYDqampfOpTn2LTpk088MADinPyQiKRCENDQ/zsZz/j17/+NTqdjrq6Oh566CGMRiN6vR6bzaY4AScrcpRWRkYGRUVFpKSkrHeT7pj+/n6am5s5cOCA8rvZ2VkmJiZ455138Hg8yu9lH0edTsemTZvIyspiy5YtG0Ko+X3E5XJRWlrKJz/5SdLT0/nP//k/MzIyct2DYV5eHlu3bqWyspLs7GwCgQCNjY0888wzd7RvrJjAI0kS09PTXL16lV/+8peUlJSQnZ3NyMgIFouF+++/X1H9BwIBhoeH+cd//EdGR0eJRCK43W5ycnKor69nYGAg6aMkZIQQZGZmUlxcTG1tLTk5OcpCGY1GFb8Yj8dDXV0dxcXFTE1NMTY2htfrJRKJMD09TW5uLgUFBYof0FqxFuaYqqoqDh48SH5+PpIkKb5YGwk55PeBBx5g79695OTkKJtpX18fx48fp7u7m7GxMWZnZ5Nq/spzKisri5KSEg4fPkxpaSkWiwWfz0cwGOTUqVNKmgej0UhKSgqf+cxncLvdFBYWKnPS5/MxMDCQdKYeOTXA7t27KSsrIyUlZdHcjsViBINB2traeOGFF3j//fcZHR3F6XRuGL+rhQghMBgMpKSk4HQ6F5nNU1JSKCwsVA6fsiY92enu7mZ6epq5uTlsNhvAIu3p9fwLT58+TUVFBW63G4vFogjp0WiUK1eu0NbWRl9fHxMTE2vWlztFHt+7RYCrqKjgU5/6FNu2bVN8XGdnZ68r8BQWFvLII4/gdrvRaDS88cYbNDY24vV670h7vqJ60NnZWYaGhjh27Bh+v5/8/Hy6urpwuVzs2bMHk8mERqNhbGyMnp4efvvb3zIyMsLMzAxZWVlkZGQwOztLOBzeEIuQrH7Mzs7G7XZTXFxMamoqEF9gZ2dnFQfrxsZGHA4H+fn5BINBAoEAIyMjjI+PEwwGOXTokJJnSFZnruamKW+CayFgFRQU8MADD5Cenk4sFmNsbCxpTT7XQ06VsHPnTh544AHS0tLQ6XTEYjEGBwc5ffo0g4ODSdkvrVaLwWAgPz+f6upqDhw4oAgEIyMjXL16lV/96ld4PB46OjrQ6/Wkpqaybds2zGYzbrdbeR5HR0cZGhpKKoEOUMx0NTU1FBQUKL+XHVzn5ubw+/20tbXx0ksvMTY2xvT0NGlpaUmtybke8tpjMBgUk7SMrCn2eDz4/X6mpqY2hMAzMDDA0NAQHo9HMdHFYjHm5+cZHx+/rs+G1WplZGSEL3/5y8r3AXFhqaenh56eHkZGRpJW67oQ+TmTfV03on/gUhQUFHDgwAFyc3MJh8O4XC78fv+S9wohyMnJYefOnWRmZjI7O8uZM2dobW297nuWy4obfv1+P8ePH+fixYukpKQwPj5OXl4e5eXlVFVVUVdXxzvvvMP777+vnIYhrvUJhUI89dRTSetNfy0ZGRlkZmbyF3/xF1RVVWG329FqtUiSRHt7O1evXuV73/se0WiUnJwcXnvtNZqamnjrrbfo7++nvb0dIQR6vR6/349Op2P79u2Ew2EqKioYHh5elagpIQQFBQUUFRUpjnGriV6vx2KxoNPpmJubo7GxkStXrqzq31xpsrKyePTRR6mpqcHlcqHVagmHw4oT3WuvvZa0J8iqqiqqqqr46le/SnFxsbJB9PT08NRTT3HhwgVF2xiNRqmsrKSyspLdu3dTUlKCRqMhHA4zMTHByZMnk9LcXFFRwX333YfValV+NzU1xfj4OB6Ph4GBAZ599ln6+voYHh5mfn5eWVizs7M3nNATjUbx+/1cunSJl19+mY985CMUFxcD8dPxF7/4RTIyMnA4HLz66qsbwlcuFosRi8UIBAKLxkOSpOsKbBqNhtraWurr6xVfnoXvCwQCBINBZmdnN8QhWiY1NZXKykpGR0cZHBxc7+bcMfPz80QiEWUcZV/BazEajRQUFOB2u8nKylLKpcjm5jtlxQWeaDTK+Pg44XAYnU5HJBLBYDAwMjJCXl4e8LsJvDC8MBqNEo1GN0S4shxGmJmZSUlJCRUVFRQVFQHxU8X8/DwDAwN0d3fT2dmpnMYgfoppbm4mEAgoCROFEESjUTQaDXa7nYyMDIqLiwmHw6sm8DidTlwuF0ajcVVPETqdTjGRaDQaotEoY2NjSSscLIWc9bWiogKXy6U4v05PT9PZ2alEMiUbJpMJp9NJVVUVDQ0NVFRUkJGRoWhYGxsbaWtro7u7e5FWNSsri4qKCpxOp+IbEgwG6enpYWBgAL/fn3QaHjmQQdayyQEBfr9faXdLSwujo6PKSV+r1WI2m7FYLOvc+ltHkiSi0SgjIyNcunSJ++67T7lmNBrJycnB5XIph7CNxHIPuykpKVgsFoqLiykoKPiACUh2KRgZGUl6YUfO4xWLxRTtTlpa2g0jmjcCcgJJp9OJyWRSBB/5cLUQIQRms5ni4mIyMzOVJMOTk5OMj4+viPZ81Vz7Z2dnlVOFnN9CXmh27NiB1WrlN7/5DdPT00k/Ga/FbDaTk5PD/fffz549e5QIj1AoxOTkJKFQSNFyjY+PE4lEGB4eVsKwZWFvKQwGA8XFxRw+fJif/OQnq5IATaPRsHnzZrZs2aJEMKwGcjbYjIwMcnJyMJlMRCIRRkdHN0wZDY1Gg81mo6ysjA9/+MOKqXFqaoq+vj6ef/75pNVWFRUV8elPf5oHH3yQHTt2oNPp8Pv9/PrXv+b48eO88sorSuTjQnbu3MnnPvc50tLSlN/JDoPNzc1J6X/15ptvcu7cOR5//HEMBgNnz55VgiiCwaCywC587uSIUpvNtuE0PDKtra1cvnyZxx9/nPr6+vVuzppSUVFBdXU1jz32GMXFxcqhSmZqaornn38+6YMjZAVAOBxmenpaiRLMzc1VXCQ2Kunp6XzhC19g3759lJSUMDg4SG9vL1euXPmA5spoNJKdnc3hw4epq6vDbDZz5coVOjs7OXbsGD6f747bsyaxjJOTk7z//vukpKRQVlaG1WqlsLAQt9uNEGJFOrJWaDQaMjIyuO+++9i2bRtVVVXodDoCgQBHjx4lEAjg8/lobm5mYGBAWWSXK9TJJi5ZIl4NhBA4HA4cDgdCCCKRyKpEFlmtVvbt20dVVRWpqalK3pClpPtkxGw2Y7PZeOihh9izZw86nU5ZnDo6OmhpaaG7uzvpSkjo9Xrq6uqUPDT5+floNBr6+/u5evUqR48e5fLly0xMTCjjoNVqSUlJISMjg+zsbNLS0tDr9UxPT9Pa2kpjYyPt7e1Jq5mbnp4mFotx5swZdDodfX19TE1NMTU1RTgcXtIkIpdiyMjI2LACj6zpWZj4cqP25WbodDqKiopIT0+nvLyc8vJy3G43VVVVOJ3ORf0OhUIEAoENsdZEo1FmZmbw+XwMDg4qpsmNPo4pKSm4XC4aGhrIy8tDkiSamppoamr6gE+WRqNh27ZtVFdXK5UYJicnOXr0KOfOnVu0Vt0JayLwhEIhXn/9dcxmM3V1dWzatAmbzUZlZSWxWAy/379htDw6nY78/Hw++tGPUltbS2FhIdPT0wwODvLTn/4Ur9eLx+PBYDDc0PZ8IwwGg5IpdDUQQpCWlqY4bE5NTTE0NLTiGT3tdjuPPfYYdXV1pKam0t/fr+TT2Ag+BVarlby8PL7whS8oixDENZbnz5/n/PnzXL58OekWVJPJxH333ceOHTt48MEHldwznZ2dNDU18dvf/pZwOKwIuAuF7E2bNlFQUIDD4UCn0+Hz+Thy5AgnTpzg0qVL69yz6yOryd99991lv0fOmJ2bm7vhN5eFSTw3ylp6M64dE4PBwObNm9m8eTMf//jHyczMVJJJLkSSJPx+P4ODg0Sj0aQzv16LbB7v7++nt7cXt9u93k26Y4QQ2Gw28vLy2LVrFzabjfn5ed59913eeecdAoHAoj1Ap9Nx4MABtm7dSkNDA5FIhLGxMV588UXefPPNFWvXmgg8chr0M2fOMDs7y5/8yZ9QW1vLF77wBaUuhs/nIxAIcOrUqaRUmUN8UNxuN9XV1Wzbtg2bzcbc3BxPP/00Fy9e5Ny5c8qJUlatJvvDBnH/jMuXL99RRk85YqSgoIDs7Gwly/T+/fuVGj/vvvsu586dY3R0dEOkSy8pKaG6upqSkhIyMjKAeK6Q/v5+Xn31Vdrb25Mu+sVoNCr15+RyAx6PB4/Hww9+8AMuX77M9PS00m45q+uTTz5JQUEBlZWVSihoU1MT7e3tvPTSSwwMDKxnt+6I7du3k5OTo+QbkjUhOp2OmpoaJZXE2NiYUqevu7s76cb2Rsha5I0s7KSmpmK1WtmyZQvZ2dmUlZUtMlEZjUZ27dqFy+UiOzt7URSTnHm6tbWV06dPc/HiRbxe7yKfLZW1wel04nQ6+dKXvkRtbS1OpxOv10tbWxtnzpyho6Nj0ZjU19dTVVXFQw89pARJXL58mWPHjq34urNm6Tnn5+cZHh7m3LlzeDwe3G43NTU1ZGdnY7Va6e/vZ2hoSDkxRyKRRcUYkwGtVktBQQH5+flkZWUpvknnz5/nwoULDA8P35GAs161mOTkXkstDAvrK8m5IRaGsgshMJlM6PV6zGYzpaWlSgXngoICxb9JkiT6+vpob2+/rokhWVgovJWUlCjaNjntwpUrV+jq6sLr9SbV/ASULMmFhYVkZWUhSRI+n4/Lly8r0Viy35ZcSy0vL4/9+/dTUFBAYWEhEH9evV4vnZ2dXLlyRXGw3wgIIZTs3iaTierqaoqLi5XsyfJzptVqycnJUapvj42N0dXVxfDwMMFgcEMcVhayUQvXykEd6enpSgJBt9tNfX39IodrvV5PbW0ter3+A32UtSS9vb289957NDU1MTIyQiQS2XDfB/wubchGysMjZ/7Ozs6msLCQPXv2UFZWhtlsJhKJ0N/fz9TUlKLZkZ/FvLw8Nm3aRHFxMRkZGcq9Fy5cWHFfzzXNRy5HT/zoRz/i9OnT/Omf/ilFRUWUlJQoWiCbzcalS5d4++23lfIDyUJqaipf/vKXqaqqQgiB1+ulp6eHc+fOcfny5TsWdtZLrS6n1F8q7C8zMxOn04lGoyE1NVUJ/7TZbEqV34MHD2I2m5VK97FYTHG2npubU6LaPB5P0pUkWIr8/Hxqamr4/Oc/z7Zt27BYLIyPj9PZ2clzzz3Ha6+9htfrTUotlVarVbIFy9FHR44c4Z//+Z+JRCIUFRXx4IMPUlJSQlVVFW63G4fDQWZm5iLTQDQa5cSJE1y4cIGpqamkHzMZuSyB2+1m27ZtHDhwQCngKz9jCw8WBoNBEXbeeustnnrqKbq6ujacsLORsVgs5Ofn8/nPf55HHnmE/Px85RB17Zp4vdDk6elpmpqaOHbsGC+99JKS/HOjjqNer1fqiW0E5GSCe/bs4Q//8A+5//77lT1Crj24efNmpqamqKur4xe/+AVzc3NYLBY2b97MgQMHsFqthEIh3n77bV555RVeeumlFV9j11TgkX1a5FpRJ0+eJD8/n/LyciwWCw6Hg/r6esxmMz6fD4/HQ29vb1JkrnU4HOTk5FBQUEB6ejoQT8ImO0fe6Yaw8HQWjUaXjJ5ZSRYuBpmZmUpa9msTO+Xl5ZGenq5ocsrKyjCZTJjNZmZnZ5VQ84mJCaampgiFQkxMTNDV1YXNZlt0IpuYmEjqk7NGo8FoNFJcXMy9995LUVERdrtdqZcVDAbx+/34fD7m5uaS+uQ4Pz/P/Py84ui5Z88e5ubmsFqtbNu2jZycHAoLC7Hb7Uq4qHyilDWsHo+H/v7+pNbGwe/CWc1mM/X19WRkZJCXl0dlZSUVFRWKP1IoFFL84xYeMOSxlB235UU6mcf3bkKug2a1WnG5XEq5k6W43qFQjgh1OBxYrVaCweCGEdKXQo7S2ggpE2RNaUVFBffccw/l5eW4XC40Go2iedPpdGRmZlJbW0taWhojIyNEo1Hsdjs1NTVkZWUBMDY2xqlTp1ZNq7wuFecuX77M1atX8Xq9NDQ08MUvfpHKykry8/M5dOgQPp8Pi8XCqVOnmJiYUMpPrCdlZWXU1NRQUlKCy+VCkiR6e3s5d+7cig2MLAhMTU3R09OzqqHbc3NzykJfV1dHWVmZ4lC8kNzcXEXgkQXWaDTK7OwsHo9H8bsaGRmhq6uLK1eu4PV6GRsbo7q6WgkTliPZhoaGVq1Pd4pOpyMjI4MDBw7wrW99a9G12dlZhoeHk74sxvz8PLOzs4RCIUUTd/jwYQ4fPrzkBj46Oqr4bplMJtLS0pScNs3NzbS2tq51F24J2RySmZlJfn4+3/3ud6msrMTlcjE/P8/c3BzT09P4fD4uXrxIWloaDQ0NH0h8Jjsw7969G5/PRygUUn0/1oEbabpv5JhtsVjYunUrHo+H8+fPc+7cOUZGRla9vatFWloaO3bsUASBZMZgMLBjxw7279/P1772NWWM5GoDQ0NDSmqPsrIyZmdn2b59O1qtVhFwU1JSlFxff//3f79q+/26ldiVfXouXLiAJEnU1NQoRRktFgt79+5VqqW++OKLdHZ2rldTARSfBzk0eXJyku7ubi5cuHDbAo9Wq6W4uFiRcKPRKL29vVy4cIFf/epXq1ZENBqN8vrrr9Pd3a2EqNvtdsLh8AdORbOzswwODtLV1aWEeo6NjeHz+ZiYmFDy6oTDYaVMxvT0NNXV1Yr5S5KkRVm1kxWbzcYDDzxARUXFBxbU4eFh3njjjTWrZH+7zMzMEAgEeO6559i8eTOPPfYYRqMRnU5HMBhUtG+Dg4OKUK3Vann88cfJysoiLS0Nj8ejODcnM1qtFpvNRkFBAQ8//DC7du2ipKSE+fl53njjDXw+H0NDQ3i9XsU8XltbS0FBAU6nU8lxotVqsVgsVFdXY7Va8fl8mEwmOjo6kkK7vFyuDUtPT0+npKRk1aI9V4pIJMLg4CAvv/wyvb29bN26FZ1OR29vr6KplFko8BQXF5OXl8eWLVs2fL4aQEkbsWfPng2TLLKkpAS3281nPvMZxTVFzsH38ssvK4fE6upqdu7cSW5uLiaTiby8PDQaDSaTSfELTUlJobS0lO985zucOXOG48ePr7ilY90EHtluPjExoSy+VVVV7N69G6fTSU1NDXa7HZfLxYkTJ+ju7l7XhUer1aLT6dBqtcRiMaanpxkYGKCzs/O27IzyYBcVFXHvvffidDqVzKBtbW2cPHly1aTcWCzGuXPn8Hq9ZGVlLXJWvRa/38/8/DwnT55keHiY3t5eBgcHb7jx63Q6iouLqaioICUlhampKYLBYFKfmIUQpKamsnXr1kW1owAl3f2ZM2eS/tQom0Nfe+01pqam2LdvHxaLBYPBgM/nY2RkhDNnztDS0sLZs2eZnp7GarWyd+9e7HY7ACMjI1y5ciUpfZRkhBAYjUbS0tKorKzkwIEDPPLII8RiMfr7+zl16hRXr16ls7OTjo4OQqEQ6enpimlV3iBlAV92Une73cr3Imdtlp/DhXMi2cxdS2k/nE4nBQUFGI3GpDbRzc7OEggEeO+992hra2N8fBytVsvZs2eZnZ29rmlq165d1NfXKy4RG51AILCoMO9SPmfJRkFBAZs2beLAgQNKrjXZtUE+VMuuDJmZmUr5D9lqICNJEkajkdzcXD73uc9hNBoVZcJdIfDIRKNRJicnaW5uxuPx8I1vfENJr52enk5KSgoNDQ1MTEzQ1taWFJtmLBZjbm6O8fFxAoHALduKZXvmY489xt69e3nooYcQQuDxeHjxxRdpaWlREqmtJoFAgH/5l39Br9ff8BQoZxaWTVk3GwONRkNNTQ3V1dVoNBq8Xi8nT55MugR9MkIIJd3Arl27yMnJUa7Nzs5y7tw5zp07R19fX1LMv5sRjUa5fPkyAwMDHDt2DK1WixBCMWPKqROmp6cpKirC7XazefNmJQGfXKIhWfuq0WiwWq188pOfZNOmTRw6dEgpTNvc3ExLSwu/+MUvGBsbIxQKodPpKCgo4Nvf/jaVlZVKuPPMzAwvv/wyU1NT5OXl4Xa7KS0t5fDhwzzyyCO89957XLlyhTfffJNIJKJ8H5FIhL6+vqTahJYKSy8tLcXlclFZWcnMzAwDAwNJ1eZrCYVCTE1N8dJLLwEoWfiv1+ZgMMiFCxd48MEHyczMXMumrgrBYJCRkRFisZgSaWi328nKysLv9yelT9Lu3bvZs2cPExMTtLe38+abb9LY2EhnZyeDg4PMzMwoCoLu7m6efPJJ6urqKC8vX9IBPRwOc+HCBdrb25XEvSvJugo8sjorNTVV+fdau7rsQJhMzM3NMTY2tqQJ6Ebo9XqlBpfb7VbCLw0GAx0dHfT19Skb1Vpos+RaPKuBxWJR6jCNj4/T09OTtCYSjUZDRUUFVVVVpKenK6fFSCTCxMQEly5doqura939yJaLJEmEw2HC4fBNs5jrdDosFgsWiwWz2UwsFiMUCjE0NJS0Ak96ejqZmZnU19dTU1NDUVER4+Pj9PX10dLSQmtrK4ODg8rJMD8/H7fbTV1dnZJk0Ofz4ff7aWpqYmJiAr/fz8TEBJIkYbVaSUtLo7a2VvEtiEQizMzMEI1GGR0dxev1JpUzt3z4kh2yASVyMjc3l+HhYQYHB5Na4JHNV8utiD09Pb0mB8O1YuFhUtbsyIfRZA1PlyNz29vb6enp4cKFC1y8eJGenp5F9wWDQYaGhhbV7JuammJkZGSRFi8YDNLa2orX610VF4h1E3jk6BG3282uXbtwu91kZ2crSc8gLvEPDAxw6dIlWlpakmaBGR0d5fjx47dcxVauKfXpT3+ayspKHnzwQTweD8ePH+ev//qvOXv27IbIDHqrDAwMcPz48aQssAlxc8Y3v/lNtmzZsihFvcfjoauri7/5m79heHh4nVu5Osi+afA7p/mOjg4lLUQyIecmefjhh9m+fTtPPPGEkuPjzTff5MSJE4rvTjAYJCMjA7fbzR/90R/R0NBAXV0dc3NzeL1eXnjhBY4ePaqYr1JTU8nJyaG0tJT9+/dTUlLC3r17qa6u5qMf/aiSbsHv93P+/HnefffdpFmPJEni1KlTCCH42Mc+htlsVq5ptVqlinxzc/NdtbYUFRVRXV19V5izYHFRbRn5+Uy2Q79Ma2srwWCQd955h9HRUUVDdS1y6pO8vDwKCgrQarW0t7fz/e9/n8HBQUX7Pzc3x/Dw8KodjtdU4JGrZssJhhoaGsjOzqaqqoq0tDTsdrtiWpEkiZmZGSUhXjI9qCkpKZSUlGCz2ZZ1v91up7S0lM2bN1NeXs727dtxOBxKDp+33nqLvr6+pPaZuBOWepCTDYvFgtVqXbSwdHV1KcUyN4p251bx+XzYbDblhCWbEJJlM19Ieno6eXl57Nixg4aGBvR6PT6fj3PnznH69GlaWlowGo3KIaqgoIDS0lLq6+tJT0+nubmZ4eFhWlpaeO+99+ju7lZ8BCYnJ5VQ2fn5edrb24lEImRlZVFUVATENRDHjx/n0qVLSbUewe9qRy3VLq1Wm3ROsLLJJicnR3G0X+4aYbVacbvd3HPPPWzfvn3Z63CyMzQ0hNlsJhAIoNPplHxSmzdvxu/3J+X+0NnZyfDwMMPDw0smlJW1VBkZGVRUVCgV4CcmJhgaGuLSpUuKKRPiz9jk5OSqme/WVOBJSUkhKyuLgwcPUl1dzYEDB7Db7TgcjkX3yaepmZkZxdl1vTfLhQ5kdruduro6XC7Xst4nFxt99NFH2bFjh5LI7vXXX+eVV17hpz/96Rr0QOV6aDQaUlJSMJlMi3wG2traeP/991f1AVxv+vv7AZJSwLmWnJwc9uzZw3333UdtbS2RSITe3l7+9V//lba2Nvr7+6mpqcHtdnPo0CEllYRer2dyclLxj3vrrbcYGxtblNR0ZmaGkZERRkZGaG9vx2w24/f7KS8v59577wXiJoef/OQndHV1Jd18kHNEbYRxhLgQZjKZqKysJBQKEQqFli1oO51Odu7cyaFDh7j33ntXrcjyWtPb20s4HGZwcBCz2YzVaqW8vJx9+/bx/vvvEwwG17uJH6ClpeWG1zUaDWazmdzcXBoaGsjIyECv1zM0NITH46GxsXFtGppg1QUeq9WKw+Fgy5YtVFVV0dDQoOTJcLlcixyXJEnC6/UyNDTEkSNH6OzspLW1lY6OjtVu5k1Z6BSo0+lwOBwcPHgQrVbLmTNnlHBsh8NBVlYWdXV1ZGdnK3lsysvLSUlJYXJyko6ODrxeL6+88gptbW3r3bVVQxYQLRYLOTk5SvblZOK+++5j586dZGZmKtodOSpCLq2QbJvbSlJSUkJ5efkHCjAmIxUVFRw+fFjxw9FoNOTl5fHxj3+cRx99lGg0SlFRETabjczMTCwWC3q9nuPHj9PR0cHPf/5zRkZGlMSRN2JmZob33nuP5uZmjh07BsRNflevXmVqamrdD2DXcvbsWYaHh/nkJz9JXl4eaWlpQNwkcu+995KWlsYLL7zA6OjoupoqnU4nOTk57Nu3j+LiYnbv3s2FCxeUFAnX5gGTMZlMpKSkKOkuPvvZz1JUVLQoAm1ubk4JiU42DdxymZubUzSITqeTd999l5///OdJG/BxM+x2Ow8++CD79+/nQx/6EBaLBa/Xy1/+5V+uS46vVVnl5PpLWq2WrKwssrOzFRv6zp07SU9PV6RyeaLOzMwQiUS4evUqV69e5d1336W7uzsphJ1rkU8npaWlSpI2n8/H+Pj4ojoibreb4uJiUlJSsFqt+P1+pVDn1atXaW9vX7aD3kbGYDBgtVqTSq0uq9RLSkoUrdvChVOu1js6OrphF8/l4HQ6yczMTKqxuR4Oh4OysjLFZ0Oj0WCz2aiqqsJgMKDX63G5XGi1Wubm5ohGo/h8PlpbW2lqaqKjo4PJycllCSuxWExJktnV1bWq/VoJhoeHkSSJYDBIWlqaIvAIIcjPzycYDJKamrrupXpsNpvyzFVXV9PQ0MDs7KySmfd6pSMcDoeSub2uro66ujpMJpOSBV2Omg0Gg0lfq+9GyCkw5EzR/f39tLW1JX0Os6XQaDRKfquKigoKCwsVLep7772naJfXkhUXeAwGAzabjaysLDIzM/nKV75CaWkpJSUlisf5wsU1FArR3d3N8ePHaWxs5MSJE4yOjjI9PZ3UJ2uNRkNDQwObNm3ivvvuIxKJMDk5qSRLlDO5arVaQqEQzc3NdHZ20tPTw7PPPsvAwADhcPiu3kwXbizrWStsKVJTU5VIuR07dpCSkqKkG2htbVVyggSDwaQ7za8kLpeLnJycDSHwRCIRAoEAFosFo9Go1BuyWq1AfLNobGxkaGiIlpYWuru7aWtrw+v1EgqFli3sbFTm5+fp7e1VTAgLM94mSyHmHTt28I1vfIOysjKlVpTb7ebzn/884XD4uht7XV0dOTk5VFdXk5KSgtlsRghBLBajra2N3t5eXnnlFS5dusTZs2eTNsJwuSz0pUsGl45bRa6fVVZWxuOPP052drbiXN/Y2IjX612XjPUrIvDIVVIrKiqw2+1kZGSQnZ1NRkYG5eXl5ObmKknN4HdFRD0eD8PDw7S3t9PY2EhHRweDg4NK/oVkQi5xMTY2hl6vV5K5GQwGsrKyFK2AxWJRnEDn5uYYHBzE4/Fw6tQp+vv7FQev1SwbkUxIkoTJZMLpdF739LYeaLVaxU5utVrRaDTEYjHC4TB9fX28//77+P3+DbnY3ApykrCFGXqTFa/Xy9tvv62YxK1Wq6KRGx0dZXx8nJaWFvx+P93d3QwMDNDb28v4+DgzMzN39ThCPM3E1atXlbU2GcdUjtYxm81KvSxZcxONRpc85AohKCoqwul0KlGUc3Nziu+PnGSypaWFgYGBDakNuRbZSuJwOMjOzl6WGTaZ0Gg0bN26ValtF4vF8Hg8XLp0iYsXL65bFfsVEXjMZjOZmZn88R//MYWFhRQVFZGenr5IyFmI1+ulr6+P559/nu7ubhobG2+YUTMZkNVvnZ2dzM/PU1ZWBsQfRvmEuXAA5VILR44c4fTp0zz99NN3/YJ7PeRsuMmU/l3OP2O1WpUCobOzs4yNjXH27FmeeeaZ9W7imuDxeDAajczPzyd1RleIFxs+f/68ksaiqqqKWCxGMBjkvffeo7W1VYnq3Eibw0oRiUR4++23EULw4Q9/eL2bs2wyMzN59NFHl3WvEIKZmRml1ltHRwd/93d/h8fjScoopttFLndSWlrK9u3bOXbsWFLX8LsWvV7PV77yFRoaGsjJyaG9vZ1z587x0ksvrbmj8kJuS+AxGAxKzpzKykqlovauXbuw2+1K+mgZuQK47AT61ltv0dXVxaVLl5TTV7KbdqamphgcHOTZZ5+lrKyM3bt3U11dvagkg5wtubOzk6amJrxeLxcvXkz6hF+rRbKZsa5loZPyQn6fxmp0dJTBwUFGRkbQ6/VYrVYsFgsul4uJiYmkOoREo1HC4TDt7e309/fT3t6upK8YGhpS6u78Po3fQmKxGD6fj8HBQQYHB7Hb7UmXo6a9vZ3nnnuO3bt3U1hYSG1t7Q01v+FwmKmpKcUnSw5qaWpqoquri4GBgQ2n/bgRer2eTZs2UVRURDQaZXx8nOHh4aR6DpeDnPVcr9eTnZ1NIBCgubl53XN73ZbAo9fryc3NZevWrRw8eFBJYZ6RkaFkXozFYsogySXie3p6OHXqFK+++ipXrlxZN7XW7RCJRJidneXNN9+ku7ubaDRKSkrKopogkiTh8Xg4c+aMUkck2TKyrhULi/4lo9Ajz9FYLMb8/PyGiFJaDeQswz6fD6vVSmpqKqmpqaSnpyvmgWR5RuXx8ng8692UpCQWizE6OqoIPfIhUghBJBJJCj+e3t5eXn/9dTQaDZOTkxQXF2OxWBY9f3I7Y7EYExMT+Hw+ZmZmCIfDNDU10dnZyZEjRxgcHEzaZKa3i06no7CwUCkmPTExwdjY2IYUeK5cuYLFYmHPnj1Kfb71dpq/rVXearWyb98+du/ezb333qtUY9ZoNExPTzM8PExnZ6dSIsHv9/PKK68QCAQUD/SNJOzIyAVPL126RG9vL7/4xS+U8gky4XCYyclJQqEQMzMzv5fCzkLp/uDBg0k5zuFwmN7eXi5fvkxLS8uGCc1eDaanp3nhhRfYsWMHTzzxhFJI9Omnn6anpyfpazCpxJmfn8fn8/Hmm29y6dIldDqdMqcjkQg9PT3rrgkJBoNMTU3h9/txuVy0t7ezadMmHn30UcxmM1qtlvPnzzM0NERraytXr16lra2NUChEJBIhHA4zMzNDKBRa976sBnNzc7S0tDAxMUFOTg7BYHBD+p9Fo1F++9vf8tZbb/HMM88wMTFBIBBQEgyuF7e1wsvFJEdGRujs7Fx0TRZ4ZHXj/Pw8gUCApqYmZbJuZGTHuqmpKSVsVWUxkiTR39+P1Wrl7Nmz9Pb20tvbm1S1tOSitV1dXZw/f56JiQl0Oh1+v5+BgYH1bt6aEo1GuXLlChkZGYRCIcXnqqSkREn1nuyZslXizM3NEQwGkzJJHfxu/ZRNkLL/ZmZmJiaTCZ1Ox4ULFxSBp6+vj66urrtWwLmWubk52traFFPzwMCAop3baMi5g7xe7zq35HeIGy1iQojrXpQ1OteGsy5M0CcP0nqlqpck6aa2lBv1cSOQrH3UaDRK8deFJs7b2TRv1sc76Z+cOmChWXJ+fn5NVcjrPYZCCGw2G7t27eLP/uzPKC8vJzs7m6NHj9LY2Mj3vvc9xY/idlnvPq4Fah9vvX9arfYD+8hCk9bCfWQtSIYxlGtnCSGU2ooredhIhj6uNtfr423r8DeaTVFlbZH9LZJ9nlwvFPb3CUmSlDINv/nNbxSH/OzsbEpLS7Hb7Uo+EBWVlUT29VPn1u9Qv4vV4/fTaUFFRWURMzMzdHV18eMf/xi/308oFOJjH/sYWq2WtLQ0IpHIukdYqKioqNwJt23S2gj8PqvuFnK39/Fu7x+sTR+FEEo5GJfLRXZ2NjMzMzQ2NhKJRO7I/y5Z+riaqH28+/sHah83AtfroyrwqH1MetRFVu3jRkDt493fP1D7uBG4LYFHRUVFRUVFReVuQLPeDVBRUVFRUVFRWW1UgUdFRUVFRUXlrkcVeFRUVFRUVFTuelSBR0VFRUVFReWuRxV4VFRUVFRUVO56VIFHRUVFRUVF5a7n/we4EvSaZhCPPgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x72 with 10 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pltsize=1\n",
    "plt.figure(figsize=(10*pltsize, pltsize))\n",
    "predicted_labels = linear_model(W, x_train[:10,:])\n",
    "\n",
    "for i in range(10):\n",
    "    plt.subplot(1,10,i+1)\n",
    "    plt.axis('off')\n",
    "    plt.imshow(numpy.reshape(x_train[i,:], (28, 28)), cmap=\"gray\")\n",
    "    plt.title('%1.2f' % predicted_labels[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## In-class exercise: implement \"accuracy\" - number of images correctly labeled"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A solution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# one answer, following the format of the above evaluate function:\n",
    "\n",
    "def accuracy(true_labels, predicted_labels):\n",
    "    N = predicted_labels.shape[0]\n",
    "    acc = (true_labels == predicted_labels).sum() / N\n",
    "    return acc\n",
    "\n",
    "def evaluate_accuracy(W, input_images, true_labels):\n",
    "    predicted_labels = numpy.round(linear_model(W, input_images)) # round real numbers to integer labels\n",
    "    acc = accuracy(true_labels, predicted_labels) \n",
    "    return acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# can plot this definition of accuracy here to check that it looks reasonable\n",
    "\n",
    "# Randomly initialize W\n",
    "W = .01 * numpy.random.rand(num_features,1)\n",
    "\n",
    "# now iterate num_iters times, with the step size defined by learning_rate\n",
    "losses = numpy.zeros(num_iters,)\n",
    "accuracies = numpy.zeros(num_iters,)\n",
    "\n",
    "for i in range(0, num_iters):\n",
    "    # all the magic here\n",
    "    W = learn(x_train_batch, y_train_batch, W, learning_rate)\n",
    "    losses[i] = evaluate(W, x_train_batch, y_train_batch)\n",
    "    accuracies[i] = evaluate_accuracy(W, x_train_batch, y_train_batch)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f4439c6b910>]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmx0lEQVR4nO3de5xdZX3v8c9v3+eSuWQmlyG3STCCERRDBCJqOQoItMVrW6ACVlqOikettj1w8Nhq+7Jaz7HaU1qhSksrilpAIqIRAhZFgUzIhYTcbyQxyeQ2k5nMZd+e88dae2bPLZnJzJ49a+/v+/Xar73Ws9Ze+3l2dr57zbOetZY55xARkeALFbsCIiIyMRToIiIlQoEuIlIiFOgiIiVCgS4iUiIixXrjxsZG19zcXKy3FxEJpDVr1hx1zs0YblnRAr25uZmWlpZivb2ISCCZ2d6RlqnLRUSkRCjQRURKROAC/T9+vYdlf/MUPalMsasiIjKlBC7Qe9NZjnb2ksxki10VEZEpJXCBHgkZAOmMrkEjIpIvcIEejXhVTmsPXURkgOAFesirsrpcREQGCl6gR9TlIiIynMAFesTfQ09pD11EZIDABXo0nAt07aGLiOQbU6Cb2Twze8bMXjGzTWb2Sb98upk9aWbb/ef6wlQXomGvy0V76CIiA411Dz0NfMY5twS4DLjDzJYAdwKrnHOLgVX+fEHk9tDTWQW6iEi+MQW6c+6gc+4lf7oD2AzMAd4NPOCv9gDwngms4wARfw89mVaXi4hIvrPuQzezZuBNwAvALOfcQX/RIWDWCK+53cxazKzlyJEjZ/W+Me2hi4gM66wC3cyqgYeBTznnTuYvc845YNjdZ+fcfc65Zc65ZTNmDHs53zOKhDXKRURkOGMOdDOL4oX5g865R/ziw2bW5C9vAlonrooD9R8UVZeLiEi+sY5yMeBbwGbn3FfzFq0AbvWnbwUem5jqDRXVHrqIyLDGeseiy4GbgZfNbJ1f9r+ALwHfN7PbgL3A709YDQfRxblERIY3pkB3zv0SsBEWv3P81Tmz3B66ruUiIjJQYM8U1R66iMhAAQx0nSkqIjKcwAW6hi2KiAwvcIEe08W5RESGFbhAz536rzsWiYgMFLxAD6kPXURkOIELdDMjFg6RVJeLiMgAgQt0gHgkRG86U+xqiIhMKYEM9EQsTE9KgS4iki+YgR4N0ZNSH7qISL5ABnpFNEx3UnvoIiL5AhvoPepDFxEZIJCBHtceuojIEIEMdG8PXX3oIiL5AhnoiWiIHu2hi4gMEMhAVx+6iMhQwQz0mPrQRUQGC2SgxyNhunVikYjIAIEM9IpYmF6dWCQiMkAwAz0aJpnJ6hK6IiJ5Ahno1XHv3tadveki10REZOoIZKDXVkQBONmtQBcRyQlkoNfkAr0nVeSaiIhMHcEM9ITX5dLerUAXEckJZqD3dbko0EVEcoId6OpyERHpE8xA97tcdFBURKRfIAO9Oh4hEjKOdyWLXRURkSkjkIFuZsyYFqf1ZG+xqyIiMmUEMtABZtUkaO3oKXY1RESmjAAHepzDJxXoIiI5AQ70BIfaFegiIjmBDvSTPWm6khrpIiICZxHoZna/mbWa2ca8sulm9qSZbfef6ye2mkM1N1QBsOvIqUK/lYhIIJzNHvq/AdcMKrsTWOWcWwys8ucL6jUzqwHYeaSz0G8lIhIIYw5059yzwPFBxe8GHvCnHwDeM75qnVlzYyUhgx2tCnQREZi4PvRZzrmD/vQhYNZwK5nZ7WbWYmYtR44cGdcbxiNhmhur2HywY1zbEREpFRN+UNQ55wA3wrL7nHPLnHPLZsyYMe73Wjq/npdePYH3liIi5W2iAv2wmTUB+M+tE7Td03pzcz3HTyXZqQOjIiITFugrgFv96VuBxyZou6f15ubpAPx659HJeDsRkSntbIYtfhf4NXCeme03s9uALwFXmdl24Ep/vuAWNlaxsLGKlZsOT8bbiYhMaZGxvsA5d+MIi945zrqMmZlx7QWzuffZXZw4laS+KjbZVRARmTICe6ZoznUXNpHJOh5/+eCZVxYRKWGBD/TXn1PDBXNqePD5vRrtIiJlLfCBbmZ88NIFbDnUQcveE8WujohI0QQ+0AGuv+gc6iuj/OPTO4pdFRGRoimJQK+MRbj97efyX9uOsEZ76SJSpkoi0AFuWb6A6VUxvrJyi/rSRaQslUygV8Uj/OmVi3l+13F+tEEjXkSk/JRMoAPcdOkCLpxTy18//grtXaliV0dEZFKVVKCHQ8YX33shJ04lufORDep6EZGyUlKBDnDh3Fr+/F3n8ZONh/j283uLXR0RkUlTcoEO8CdvW8R/O28Gf/WjV3hm66Rc+FFEpOhKMtBDIeP/3bSU82dP444HX+LF3YNvsCQiUnpKMtABquMR/vVDb6apNsHN33qBZ7ZoT11ESlvJBjrAzJoE3//vy1k8q5rbHljNP/98J9msDpSKSGkq6UAHaKiO89Dty7n2wia+/NMtfPiB1ew/0VXsaomITLiSD3Twul/+8cY38fnrX88Lu45z9d8/yz//fCfdyUyxqyYiMmHKItDBuyrjrW9p5slPv53lixr48k+38PavPMP9v9xNR49OQhKR4LNinXyzbNky19LSUpT3Bnhx93H+78+28sLu41TFwrx36Rx+f9k8LpxTi5kVrV4iIqdjZmucc8uGXVaugQ7gnGP9/nb+49d7+dGG35BMZ5k3vYLrLmziytfN4qJ5dUTDZfNHjIgEgAJ9FNq7Uqx85RA/3nCQ53YcJZ11VMXCXLaogeXnNrB0QT1LmmpIRMPFrqqIlDEF+hi1d6X49a6j/HLHUZ7bcYzdR08BEA0br2uq4Y1z61hyTg2vnVXN4lnTqElEi1xjESkXCvRxOtTew7p9bazf38b6fW1s2N9OZ2+6b/nsmgSLZ1XzmpnVLJheyYKGKuZNr2RufYX26EVkQp0u0COTXZkgml2b4Jra2VxzwWwAslnHgbZuth3uYNvhTrYf7mBbawcPvbiP7lT/UEgzL+znTa9kXn0lTbUJZtUmaKpJMLvWe0yvjBEK6SCsiIyfAv0shELmhfT0St75ull95c45jnT2su94F3uPdfHqcf9xrIvndhyltaOHwSeqRsPGrJoEs2sSzKyJ01gdp6EqTkN1jMbqGA3Vfll1jGnxiEbgiMiIFOgTyMyYOS3BzGkJLl4wfcjydCbL0c4kh072cKi9m0PtPRw82cPh9h4Otvew5VAHxzqP0d49/Lj4WDhEQ3WMhuoY9ZUx6ipj1FVEqa2IUleZe471T1dEqa2MEo+o20ekHCjQJ1EkHOrramFe3YjrJdNZTnQlOdrZy7HOJMdOec9HO5Mc6+zlaGcvbd0pDpzopq07RVtXcsief76KaJhaP/inJSJUJyJMS0Spjke8+Xikb9qbj1Ltl9f461dEw/rrQGSKU6BPQbFIiFk1CWbVJEa1fjbr6Eymae9K0d6doq0rRVt3kjZ/vt0P/bauFJ29aY6fSvLqsS46etN09qQH9PuPJGT0BX9FLExlLEJlLOw/vLKqWJiKQeWVsbC/LPe6gdMV0bCOIYhMEAV6CQiFjJpElJpElHln8fpUJsup3jQdPWk6e71HR0+qf74nV5bmVG+armSGrqT3fLQzSVeyi+5khlPJDN3JDMlMdkzvH4uEiEdCJKJhEtEQiUi4fzoaJh7pn84tjw9aLx71p/u2k//6ELFIiFjYf/an9ReHlBoFuhANh/y+99iEbC+VydLlh3su+HM/Av3B31/ek87Qm8rSk8r4j2xfWWdvmqOdSXpzy9L96433SsiDA3646fhplsUiIeJDloUHzMcjISJhIxoOEfWfI6EQsYgRCYWIRkJEQ35533ohwvqrRc6CAl0mXDQcorYiRG1F4U64cs6Ryjh60l6496ay9Kb9H4NU3rP/w5DMZEmm/UcmS286fz4zYFky3b+8szc9ZFnu0evPF4KZ9zlGQ0Y04v8IhI1I3g/DwB+BYX4s/PL8H4po2Px1QkRCRjhk3rP/XuGQt344dPr5SCiUNz1wPhwyoqEQYX9Zbl5da4WnQJdAMjNiESMWCRX1TN3cD8uQsE9n6E1nSWWypLOOVCZLKuNIZ7J906lMlrT/2nSuLJsllXaks1m/3A1aP0sq60il87ebpSeVpbMnTXK49/DXT2W9skyRbvJiRl/4ez8ip/kxGGHe+4Hof33YvLJQqP85bN7rwiOU9T3yynKvH64sZP0/TAPKwt5z+HRlecvy3zseCRXkOlEKdJFxyP9hIV7s2oxONuv9cGSyjnTWkcnkzWf8sqz3QzBg3v8xSJ1mPt23zWz/tL+dTDbrrztovu912b765Ob71/X+GsufT2WzZLOOjPNek3Fe+YBHXtlUulnZX7/nAm6+bMGEb1eBLlJmQiEjHiq/cxOcGybks3g/HMOUZZ0jk5seRVkmm+1fNqgs4/84ZZw3vXR+XUHaqEAXkbJgfpdIKYeeLvYtIlIiFOgiIiWiaJfPNbMjwN6zfHkjcHQCqxMEanN5UJvLw3javMA5N2O4BUUL9PEws5aRrgdcqtTm8qA2l4dCtVldLiIiJUKBLiJSIoIa6PcVuwJFoDaXB7W5PBSkzYHsQxcRkaGCuocuIiKDKNBFREpE4ALdzK4xs61mtsPM7ix2fcbDzO43s1Yz25hXNt3MnjSz7f5zvV9uZvYPfrs3mNnSvNfc6q+/3cxuLUZbRsPM5pnZM2b2ipltMrNP+uWl3OaEmb1oZuv9Nn/eL19oZi/4bfuemcX88rg/v8Nf3py3rbv88q1m9q4iNWnUzCxsZmvN7HF/vqTbbGZ7zOxlM1tnZi1+2eR+t51zgXkAYWAnsAiIAeuBJcWu1zja83ZgKbAxr+zvgDv96TuBL/vT1wE/AQy4DHjBL58O7PKf6/3p+mK3bYT2NgFL/elpwDZgSYm32YBqfzoKvOC35fvADX75N4CP+tMfA77hT98AfM+fXuJ/3+PAQv//QbjY7TtD2z8NfAd43J8v6TYDe4DGQWWT+t0u+ocwxg9sObAyb/4u4K5i12ucbWoeFOhbgSZ/ugnY6k/fC9w4eD3gRuDevPIB603lB/AYcFW5tBmoBF4CLsU7SzDil/d9r4GVwHJ/OuKvZ4O/6/nrTcUHMBdYBbwDeNxvQ6m3ebhAn9TvdtC6XOYA+/Lm9/tlpWSWc+6gP30ImOVPj9T2QH4m/p/Vb8LbYy3pNvtdD+uAVuBJvD3NNudc2l8lv/59bfOXtwMNBKzNwNeAvwByt3RqoPTb7ICfmdkaM7vdL5vU73YpX0ky8JxzzsxKblypmVUDDwOfcs6dzL9Zcym22TmXAS4yszrgUeD84taosMzsd4BW59waM7uiyNWZTG91zh0ws5nAk2a2JX/hZHy3g7aHfgAG3Nh+rl9WSg6bWROA/9zql4/U9kB9JmYWxQvzB51zj/jFJd3mHOdcG/AMXndDnZnldqjy69/XNn95LXCMYLX5cuB6M9sDPITX7fJ1SrvNOOcO+M+teD/clzDJ3+2gBfpqYLF/tDyGdwBlRZHrNNFWALkj27fi9TPnym/xj45fBrT7f8qtBK42s3r/CPrVftmUY96u+LeAzc65r+YtKuU2z/D3zDGzCrxjBpvxgv0D/mqD25z7LD4APO28ztQVwA3+iJCFwGLgxUlpxBg55+5yzs11zjXj/R992jn3h5Rwm82sysym5abxvpMbmezvdrEPJJzFgYfr8EZH7ATuLnZ9xtmW7wIHgRReX9lteH2Hq4DtwFPAdH9dA+7x2/0ysCxvOx8GdviPPyp2u07T3rfi9TNuANb5j+tKvM1vANb6bd4IfM4vX4QXTjuAHwBxvzzhz+/wly/K29bd/mexFbi22G0bZfuvoH+US8m22W/bev+xKZdNk/3d1qn/IiIlImhdLiIiMgIFuohIiVCgi4iUiKKNQ29sbHTNzc3FensRkUBas2bNUTfCPUWLFujNzc20tLQU6+1FRALJzPaOtExdLiIiJUKn/ouIDHKgrZuth06O6TVL59dTVxkbUr732Cl2HukcUPbaWdOYW185rjoOR4EuIjLIJ767ljV7T4zpNTdeMp+/fd+FQ8pve6CFHa0DA/1v3nMBH7xswbjqOBwFuojIICdOJXnb4kb+7OrzRrX+//juWtq6kiNu69oLZvOR3zq3r2xOfcWE1HMwBbqIlJ3edIbndx0nnckOu7ytO8Wy5nreOK9uVNurr4yy/0Q3qzYfHrKsszfN3PqKUW9rPBToIlJ2Hlv7G/7i4Q2nXWfGtPiotzdjWoKnNh/mtgeGH7k3lm2NhwJdRMrOCb975AcfWU48MnSwn2GcN3vaqLf3tRsuYtegA585ITPOH8O2xkOBLiIlpaMnxYu7j5M9zXUHN/3GG8Fy8fx6QiEbecVRqo5HeMPcunFvZ7wU6CJSUu79r1384zM7zrheY3VsQsJ8KlGgi0hJOd6VpK4yyrdvu/S0682qSUxSjSaPAl1Eiubl/e0cbO+e0G3uPnKKaYkIF8ypndDtBsGoAt3MrsG7J2AY+KZz7kuDln8EuAPIAJ3A7c65Vya4riJSQnpSGd77T8+RPl1n91laOr9uwrcZBGcMdDML490q6Sq826StNrMVgwL7O865b/jrXw98FbimAPUVkRLR2ZsmnXV89Ipz+e0LmyZ02/OmT/xp9UEwmj30S4AdzrldAGb2EPBuoC/QnXP5Fz2owrtvpIjIEIfae1i/v43jp7yhg4saq8qye6QQRhPoc4B9efP7gSFHG8zsDuDTQAx4x3AbMrPbgdsB5s+fP9a6ikgJ+OwPN/JU3hmVk3XSTTmYsIOizrl7gHvM7Cbgs8Ctw6xzH3AfwLJly7QXL1KG2rqSvHFuLV9834XEI2HOnVFV7CqVjNEE+gFgXt78XL9sJA8B/zyeSolIcHUnM/xyx1Ey2eGvk9La0cvimdW8/hx1s0y00QT6amCxmS3EC/IbgJvyVzCzxc657f7sbwPbEZGy9P2Wffzlik2nXefy1zROUm3KyxkD3TmXNrOPAyvxhi3e75zbZGZfAFqccyuAj5vZlUAKOMEw3S0iUh5y10n58SfeSsiGPxNzkbpZCmJUfejOuSeAJwaVfS5v+pMTXC8RCah9x7uJhExdKkWge4qKyIQ60NZVkJOF5MwU6CIyoUJmLGmqKXY1ypKu5SIiZ62jJ8VzO46Rdf175Ifaewp2izU5PQW6iJy1B361h//zs21DypcuqC9CbUSBLiJnra0rRUU0zA/vuHxAeXNjeV5LpdgU6CKT7GRPiue2Hz3tHXWCYuvhDqri4THdrk0KR4EuMsm+9YvdfH1V6Zx7pwOgU4cCXWSStXUlmRaP8PDH3lLsqkyI2bWld+efoFKgi0ySl/e3s+fYKbYd7qQqHuG1s9RNIRNLgS4ySW765vN09KQBeOO8uuJWRkqSAl1kEmSyjo6eNLcuX8DNyxcwu1bjtGXiKdBFCqy9K8WqLd4NHebUV/CamepqkcJQoIsU2L3P7uSffr4TgFk1OoAohaNAFymwtu4UdZVRHrvjcuaX6c2LZXIo0EVOY/2+NvYcOzWubew43El1PMKCBl0DXApLgS5yGh/81gt9I1PG483NuraJFJ4CXWQE2byRKbe8pXlc22rSyTcyCRToU0Aqk+XZbUe4bFEDVfGB/yTOOX6+7Qg9yQzveN1M4pFwkWpZXjYeaGfzwZMAnFNXwbkzqotcI5Ez0w0upoCnt7Ry2wMt3PvsriHLdh7p5I/+dTUfffAlnnzlcBFqV57+5N9b+PP/3ADo1HYJDu2hTwG5Ptr9x7uGLGvrSvVNn+wef1+ujE57d4rfu3gun3jnYubqZg0SEAr0Ajt+Ksmz247gGPlaqS17jgOwrbWDR9fuH7Bs2+HOvunVe45TERv4R1XIjCvOm0lnb5oXdx8bsu1zZ1Tzhrl142hBYRzp6OW5HUdP+7lMpngkzJWvm0UsEuLZbUfoTmWYXZtgnoYZSoCMKtDN7Brg60AY+KZz7kuDln8a+GMgDRwBPuyc2zvBdQ2ke/9r57BdKcPZeOAkf/q99SMuf3TtAR5de2BI+Weuei1bDnXw45cPDlnWWB2j5bNXjb7Ck+QfVm3nP56fWl+Re2++mDfOreOW+18EoEmn50vAnDHQzSwM3ANcBewHVpvZCufcK3mrrQWWOee6zOyjwN8Bf1CICgdNe3eKxuoY//mR018qNRwyMiPc8aAqHiEWDnGiKzlk2bVf/wUne1K0d6d4/Tk13HPT0r5l//KLXXy/Zd/4GlAgbd0p5tZX8O3bLi12VTh0socb7nue9m7vcwT46/dcwI2XzCtyzUTGZjR76JcAO5xzuwDM7CHg3UBfoDvnnslb/3nggxNZycG2H+6goTrO9KpYId9mWKlMlidfOczmgydZ2FjF0vn1zK5N8NTmwyTTWQAuXdTAnLoKupJpfrzhINOrYzQ3jv+kktrK6JCyRDTEun1t7D56ivNn1wx4n9k1CVIZx8YD7Vwwp3bc7z9aPanMgM9jOLuPdjItEZ2Qz2W8piW8/wbP7zrGb9q6AZhbV4GZFbNaImM2mkCfA+Tv5u0HTrdbdRvwk+EWmNntwO0A8+fPH2UVh7rq759lxrQ4q+++8qy3cbZ+uf0oH3vwpb75ty1u5H1L5wzoKvntNzRxz01LWbHuN3T0plk0s3BD3k4lM6zb10Yq4+joHXjQtKnO6zL45ENrWfWZKwpWh8FWbjrEJx9ad8b13nn+zMJXZhSqExGq4xEeeam/O0vXXJEgmtCDomb2QWAZ8FvDLXfO3QfcB7Bs2bJxHQ070tE7npeftbbugd0e7d0p2v2RKCs+fjl3PfIyJ/0/29v853/90JsLVp+bLpnPv/1qDwBXDgrI9y+dww/XHmB7a0fB3n84uW6LH95xOfXD/FWRM1WGA8YjYZ77n+/o+7etiIWZOW1q1E1kLEYT6AeA/M7EuX7ZAGZ2JXA38FvOueKk7SR4ZsuRAfOtJ3tZvfcEAK+dNY3aiigHTnTz8Jr9rH3VK6+rGDnUxiu/2ynXdZBjZpw7o4qXXj3Bw2u80TOzaxNc/prGMb3Hmr3H2XN06JDKkazek/s8qqmMBWMgVW1ldNguLZEgGc3/ttXAYjNbiBfkNwA35a9gZm8C7gWucc61Tngt8zhXvGFu3ckMK9b/pm++JhHh0Mker5+8KkY8EuKcugp+tfMYn/mB1wXTVJsgFCpcX+w5df0jMZrqho7KmFNfQVcy01cfM1j3uaupHcOPzAe/+SLdqcyY6lVfGSWhs1pFJtUZA905lzazjwMr8YYt3u+c22RmXwBanHMrgK8A1cAP/ANJrzrnri9EhVOZ4gV6p99H/WdXv5ablzdTHY9w4IR3EK2uKoqZ8aX3Xcgn3rG47zXTqwt74Pb9S+ew/NwGQjb8MLs/edsirr2gCefgxy8f5Ms/3UJnb3rUgZ5MZ+lOZfjjty7kluXNo65XXVW0oD9kIjLUqP4eds49ATwxqOxzedOTdnQymRl55MRESmeyPLW5lcbqGLuPnqI7lek7o3NWTaIvEOc3DDzxJBIODSkrJDNjzjB75vnLcyfHzPHPePzh2gPMnBbvK3vLuUO7YJ7ecphjnUl6/JEqs2sTk9ouERm7YHRw5jndULiJ9Kudx/jIt9cMu+x0ATqVzanzDvR9ZeXWvrJwyHjlC+8acNGv/Se6+PC/tQx6bTDbLFJOFOgj6MwbArh8UQO/3uWdVv/i3e8M7AiIixdMZ/XdV9Lj94c//NJ+vvbUdrp6MwMCPTdK5YvvvZC3LW4kFglpGJ9IACjQB3l22xEaqmOs29fWV5Z/4DGoYZ4zw+9qgf5rdK/Ze4K27hRZ/4Dzq8e8ES1z6yt0LRORAAleoBewD72tK8kt979IyPr7mwEWNlb2jWApJXPqvLD+439vGbLMTDdlEAmawAV6IYct5g56Zh1ks95ZoF96/xs4pzbBhy5fSKTERm28dXEjTbUJDrb3MC0e4SefelvfsspYpCiXVhCRsxe4QB/h+lUTYtfR/psBt3enmD+9su9gYHU8cB/VqMz2A70qHmFuvbpXRIIscHcsyhZwD/0HeVcm7OxND+h2KVW5EC+HtoqUusDtdhYy0NMZx+yaBI/e4V3qdnYZjOz4+99/I3deez4N6l4RCbzA7aFPRJ4fPtnDo2v305XsH5p4tLOXn246xKyaOE21FTTVlsflUyPhEHPqKkhEdZq+SNAFLtAnYg/9yz/dwp9+bz2Pb+i/w8+3frkbQP3IIhJYAQz08W8jd+Pl3GVuATp6vOmv3XDR+N9ARKQIyq4Pff2+tr4Laq3f386qzYdp7ehl88EO5tRVEA0H7jdORAQIYKCPdxz6Jx5ay17/TMhntrTyo7zL4V66cPq4ti0iUkyBC/T8Lpds1o35Eq0nu1O8b+kcTnaneWrzYQD+6neXcM0FTTqRRkQCLXiBnpfoqWyWeGj0ozN+vfMYJ7pSNFbHB4zqmF1bMWVuhyYicraCF+h5e+ipjGMsJ3De+C/PA9BYHaO+0tsbN9OlYUWkNAQu0PP70NNneaGumy9rJhEN8f6lc4hFQtRVqqtFRIIvcIGev4c+lisv7s67TksiGsLMmFkGZ4KKSPkI3Bi9/GGLY7m/6Bef2Nw3XQ5ngIpI+Ql2oI/hZhcdPSkWz6xm999eV4hqiYgUXeAC3Q04KDq6QN/R2sHzu45zTl15XJ9FRMrTqALdzK4xs61mtsPM7hxm+dvN7CUzS5vZBya+mv3y99BH24f++R+9AsB83U5NRErYGQPdzMLAPcC1wBLgRjNbMmi1V4EPAd+Z6AoOln9QND3KPvSOnjRLmmr4/PWvL1CtRESKbzR76JcAO5xzu5xzSeAh4N35Kzjn9jjnNgCFvYMzA/fQNx88ecb1nXOs29fGvOkVYz6rVEQkSEYT6HOAfXnz+/2yMTOz282sxcxajhw5cjabGDAO/c5HXj7j+rnhiplC3rtORGQKmNSDos65+5xzy5xzy2bMmHFW2xhrLudu/HzjJfPP6v1ERIJiNIF+AJiXNz/XLyuKsV4+d4V/NcWKmO7IIyKlbTSBvhpYbGYLzSwG3ACsKGy1RjaWPfT2rlTfnYh0vRYRKXVnDHTnXBr4OLAS2Ax83zm3ycy+YGbXA5jZm81sP/B7wL1mtqlQFc71odckIly26PTXL+/o9e5C9Je/u4QFDVWFqpKIyJQwqmu5OOeeAJ4YVPa5vOnVeF0xBZfrcqmMRTjWmeTbz+/lskUNvGZm9ZB1V+85DkBjdXwyqiYiUlSBO1M06w+MjISN7a2dfPaHG/nbvOu05PunZ3YCME8nFIlIGQheoPt76LG8e3+eSqaHXTeVyfKO82dy0by6yaiaiEhRBS7Qc4NcIuH+k4Se33Wch9fs7xtr7pzjsXUHOH4qyQx1t4hImQheoOOF9vFTqQHln/nBel4+0A7AvuPdfPKhdZzsSdPcqIOhIlIeAhfouWGLF86pGbKs0z+J6GSPF/Zf+4OL+OgV505a3UREiimAgd4/ymWwbz+/l2zW624BaKjWreVEpHwEMNC950R06JmfP910iJa9J/iXX+hkIhEpP4EL9NyJRYmoV/XXn1PDhr+6mvtuvhiAQyd7APjfv7OERTOGjk0XESlVgQv0rL+LXuHvoUfDIWoSURr80Syf+O5aABqq1N0iIuUleIHud7nk7la080gnMLR7RScTiUi5CWCge4l+3uxpACydXw/A7NrEgPUuXlA/uRUTESmywAV67sSiKn+Ui+75LCLiCVyg5/bQ5zd4XSpXL5ndt+wDF3vXB1ukk4lEpAyZG+MNIybKsmXLXEtLy5hfl0xn6U1nqI5HSGUcsUj/b5Jzjt50lmg4RFj3DxWREmRma5xzy4ZbNqrL504lsUioL8RjkYGhbWbDjk8XESkHgetyERGR4SnQRURKRNH60M3sCLD3LF/eCBydwOoEgdpcHtTm8jCeNi9wzs0YbkHRAn08zKxlpIMCpUptLg9qc3koVJvV5SIiUiIU6CIiJSKogX5fsStQBGpzeVCby0NB2hzIPnQRERkqqHvoIiIyiAJdRKREBC7QzewaM9tqZjvM7M5i12c8zOx+M2s1s415ZdPN7Ekz2+4/1/vlZmb/4Ld7g5ktzXvNrf76283s1mK0ZTTMbJ6ZPWNmr5jZJjP7pF9eym1OmNmLZrbeb/Pn/fKFZvaC37bvmVnML4/78zv85c1527rLL99qZu8qUpNGzczCZrbWzB7350u6zWa2x8xeNrN1Ztbil03ud9s5F5gHEAZ2AouAGLAeWFLseo2jPW8HlgIb88r+DrjTn74T+LI/fR3wE8CAy4AX/PLpwC7/ud6fri9220ZobxOw1J+eBmwDlpR4mw2o9qejwAt+W74P3OCXfwP4qD/9MeAb/vQNwPf86SX+9z0OLPT/H4SL3b4ztP3TwHeAx/35km4zsAdoHFQ2qd/ton8IY/zAlgMr8+bvAu4qdr3G2abmQYG+FWjyp5uArf70vcCNg9cDbgTuzSsfsN5UfgCPAVeVS5uBSuAl4FK8swQjfnnf9xpYCSz3pyP+ejb4u56/3lR8AHOBVcA7gMf9NpR6m4cL9En9bgety2UOsC9vfr9fVkpmOecO+tOHgFn+9EhtD+Rn4v9Z/Sa8PdaSbrPf9bAOaAWexNvTbHPOpf1V8uvf1zZ/eTvQQMDaDHwN+Asg6883UPptdsDPzGyNmd3ul03qdztwl88tJ845Z2YlN67UzKqBh4FPOedOWt5tp0qxzc65DHCRmdUBjwLnF7dGhWVmvwO0OufWmNkVRa7OZHqrc+6Amc0EnjSzLfkLJ+O7HbQ99APAvLz5uX5ZKTlsZk0A/nOrXz5S2wP1mZhZFC/MH3TOPeIXl3Sbc5xzbcAzeN0NdWaW26HKr39f2/zltcAxgtXmy4HrzWwP8BBet8vXKe0245w74D+34v1wX8Ikf7eDFuirgcX+0fIY3gGUFUWu00RbAeSObN+K18+cK7/FPzp+GdDu/ym3ErjazOr9I+hX+2VTjnm74t8CNjvnvpq3qJTbPMPfM8fMKvCOGWzGC/YP+KsNbnPus/gA8LTzOlNXADf4I0IWAouBFyelEWPknLvLOTfXOdeM93/0aefcH1LCbTazKjOblpvG+05uZLK/28U+kHAWBx6uwxsdsRO4u9j1GWdbvgscBFJ4fWW34fUdrgK2A08B0/11DbjHb/fLwLK87XwY2OE//qjY7TpNe9+K18+4AVjnP64r8Ta/AVjrt3kj8Dm/fBFeOO0AfgDE/fKEP7/DX74ob1t3+5/FVuDaYrdtlO2/gv5RLiXbZr9t6/3Hplw2TfZ3W6f+i4iUiKB1uYiIyAgU6CIiJUKBLiJSIhToIiIlQoEuIlIiFOgiIiVCgS4iUiL+P1l4rQi9DgfaAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure()\n",
    "plt.subplot(2,1,1)\n",
    "plt.plot(losses)\n",
    "plt.subplot(2,1,2)\n",
    "plt.plot(accuracies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Improving the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Examples of basic ways to improve:\n",
    "- Add bias term: f(x) = xW + b \n",
    "- Reformulate as classification (output integers, not real numbers), like logistic regression\n",
    "- Minimize something other than mean squared error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sigmoid function encourages outputs of 0 and 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(x):\n",
    "    z = 1/(1 + numpy.exp(-x))\n",
    "    return(z)\n",
    "\n",
    "def classification_model(A, b, input_images):\n",
    "    # f(x) = sigmoid(xA + b) returns m-length vector, where m is the number of examples\n",
    "    return sigmoid(numpy.dot(input_images, W) + b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f4439bded00>]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAgAklEQVR4nO3deXRc9X338fdXuxd5lbzJ8gbGeAEbWxAgbGGzIcFu00DM05AFGpqFnvSkaUue9BAOyenzpGnyNDmlIbTZWAohpBA3MRGGkJIFg20wYHnB8i5hLV4lW5Y0y/f5Y67NICRrbI/mzow+r3PGM/fe38x8fefqo6vfvXN/5u6IiEjuKwi7ABERSQ8FuohInlCgi4jkCQW6iEieUKCLiOSJorDeuKKiwqdNmxbW24uI5KR169btc/fK3paFFujTpk1j7dq1Yb29iEhOMrNdfS1Tl4uISJ5QoIuI5AkFuohInlCgi4jkCQW6iEie6DfQzeyHZtZiZhv6WG5m9l0zqzezN8xsYfrLFBGR/qSyh/5jYMlJlt8AzAxudwLfO/OyRETkVPV7Hrq7v2hm007SZBnwkCeuw7vazEaZ2UR335uuIkUkf7k7XdE4XZE4ndEY3dE40bgTi8eJxJxY3InGnWjs+HwnEosH98eXx4m74w5xT7ymOzhJ83DiDri/04b3tk9MQzy4tPjxZQD+rrqTHicteff83p9wzezxzK8elaY1+I50fLGoCtiTNN0QzHtPoJvZnST24pkyZUoa3lpEwuTutHVGaW3vpKW9iwNHu2k7FqWtM0LbsUhwH6W9M0JbZ5SjXVG6onE6I7HglgjxwTIsg1niftyIsqwN9JS5+4PAgwA1NTWD5CMUyV3uTuOhY+ze38HuAx3sOpC4f/vQMVrbu2ht76IrGu/1uUUFRnlZESOGFDOirJjysiJGDx3KkJJCyooKKCsupKz4+H3hieniwgKKC42iggKKCoyiwsR9YYFRFMwvLDCKC4N5wXRhgVFgYBhmBLd35hUYYFBghvHuZVYARrAsaAPBaxx/bsDsnYmk2VgfbTIpHYHeCFQnTU8O5olIDonE4tS93caGxsNsbmpj8952Nje1c6QreqJNUYExefQQqkYP4cJpY6gsL2VceSmVwa1ieCkjyooZMaSIIcWFoQXbYJWOQF8B3GVmjwPvAw6r/1wk+3VH47y6+yCv7DjAKzsO8Orug3R0xwAoLyti9oQRfHhhFbMmlDN97DCmjB3KxJFDKCxQSGerfgPdzB4DrgIqzKwB+CpQDODuDwArgRuBeqAD+NRAFSsiZ6a9M8Lzm1pYtamZF7e00t4VxQxmjS/n5kWTuXD6GBZUj6Jq1BDtXeegVM5yubWf5Q58Pm0ViUhaxeLO7+v38fN1DdTWNdEVjVNZXsoHz5/I1eeO433TxzJyaHHYZUoahHb5XBEZWEe6ojyxZg8/+uMO9hw4xsghxdxSU82fXFDFBdWjKFDXSd5RoIvkmbbOCP/+4nZ+/IedtHdFqZk6mruXzObaOeMoLSoMuzwZQAp0kTzRGYnxyOpd3P9CPQc7Itx43gQ+ffkMLpgyOuzSJEMU6CJ54KVt+/nfT73Jjn1HuXxmBX+3+FzOmzwy7LIkwxToIjns8LEI/2flJh5fs4cpY4by0O0XccU5vQ43KYOAAl0kR63fc4jPP/oqTW2d/OWVM/jra85hSIn6yAczBbpIjnF3HnppF1//1UbGlZfx889eyoIBuC6I5B4FukgO6Y7G+fufv8FTrzVyzbnj+NYt8xk1tCTssiRLKNBFcsSRriiffWQdv9u6jy9edw53feBsnUsu76JAF8kB+4508akfrWHj3ja++ZHzubmmuv8nyaCjQBfJcvuPdPHR779E46FjPHjbIq6ZPT7skiRLKdBFslh7Z4RP/OgVGg4e4ye3X8TFM8aGXZJksVTGFBWREHRGYtzxk7Vs3tvOAx9bpDCXfmkPXSQLxePOXz32Gmt2HuBfPrqAD5w7LuySJAdoD10kC/3L81tZtbGZez40h2ULqsIuR3KEAl0kyzxb18R3n9/KzYsm88lLp4VdjuQQBbpIFqlvOcIXn3id8yeP5Gt/Mk+jBskpUaCLZInOSIzPPLKO0qICHvjYIsqKdV0WOTU6KCqSJf7p11uobznCw3dcxKRRQ8IuR3KQ9tBFssBL2/bzwz/s4OOXTOXymbr8rZweBbpIyNo7I3zpZ68zvWIYd99wbtjlSA5Tl4tIyL7+y03sPXyMJz97KUNL9CMpp0976CIhenn7fn66dg93XnEWCzX2p5whBbpISKKxOF9dUUfVqCF84ZqZYZcjeUCBLhKSR1/ezeamdv7hg7M1dJykhQJdJAT7j3TxrWe3cNnZFSyZNyHsciRPKNBFQvDN2i10dMe4d+kcfRtU0kaBLpJhm5va+OnaPXzy0mmcPa487HIkjyjQRTLsW8++xfCSIu66+uywS5E8o0AXyaD1ew6xamMzn75iBqOGloRdjuQZBbpIBn3r2S2MHlrM7ZdND7sUyUMpBbqZLTGzLWZWb2Z397J8ipm9YGavmdkbZnZj+ksVyW0vb9/P77bu47NXncXwUn0jVNKv30A3s0LgfuAGYA5wq5nN6dHsH4An3P0CYDnwb+kuVCSXuTv//OwWxpWX8vFLpoVdjuSpVPbQLwLq3X27u3cDjwPLerRxYETweCTwdvpKFMl9q7cfYM3Og9x19dm6zrkMmFQCvQrYkzTdEMxLdi/wMTNrAFYCf9XbC5nZnWa21szWtra2nka5Irnp+y9uY+ywEm6pqQ67FMlj6TooeivwY3efDNwIPGxm73ltd3/Q3WvcvaayUtd8lsFhc1Mbv93Syicvnaa9cxlQqQR6I5C8WzE5mJfsDuAJAHd/CSgDKtJRoEiue/DF7QwpLuS2S6aGXYrkuVQCfQ0w08ymm1kJiYOeK3q02Q1cA2Bms0kEuvpUZNB7+9AxVqx/m+UXVeu8cxlw/Qa6u0eBu4BaYBOJs1nqzOw+M1saNPsb4NNm9jrwGPBJd/eBKlokV/zoDztw4A6ddy4ZkNLJsO6+ksTBzuR59yQ93gi8P72lieS2ts4I//nybj50/kQmjx4adjkyCOiboiID5OfrGjjaHeMvLpsRdikySCjQRQaAu/PI6l0sqB7FeZNHhl2ODBIKdJEB8NL2/WxrPcptF+vMFskcBbrIAHhk9S5GDS3mg+dPDLsUGUQU6CJp1tzWSW1dM7fUVOuLRJJRCnSRNHvsld3E4s6fv29K2KXIIKNAF0mjSCzOY6/s5spzKpk6dljY5cggo0AXSaMXNrfQ3NbFx3QwVEKgQBdJoyfXNVAxvJQPzNLF5yTzFOgiabLvSBe/2dzChxdWUVSoHy3JPG11Imnyi/VvE407H1k0OexSZJBSoIukgbvzs7V7mD95JOeMLw+7HBmkFOgiaVD3dhubm9q1dy6hUqCLpMGT6xooKSxg6fyeozOKZI4CXeQMdUfj/GJ9I9fNHc/IocVhlyODmAJd5Ay9sKWFgx0RdbdI6BToImdoxfq3GTushMvP1jC6Ei4FusgZaO+M8NymZj54/kSdey6h0xYocgZWbWymKxpn6fxJYZciokAXORMrXn+bqlFDWDhldNiliCjQRU7X/iNd/G7rPm6aP4mCAgu7HBEFusjpWrmhiVjc1d0iWUOBLnKaVqxvZOa44cyeqK/6S3ZQoIuchsZDx1iz8yBL50/CTN0tkh0U6CKn4Zk39wJwk7pbJIso0EVOQ21dE+dOKGdahYaZk+yhQBc5Ra3tXazddZDr504IuxSRd1Ggi5yi5zY14w6L544PuxSRd1Ggi5yi2romJo8ewpyJI8IuReRdFOgip6C9M8If6/ezeO4End0iWSelQDezJWa2xczqzezuPtrcYmYbzazOzP4zvWWKZIcXtrTSHYuzWP3nkoWK+mtgZoXA/cB1QAOwxsxWuPvGpDYzgS8D73f3g2Y2bqAKFglTbV0TY4eVsGiqrt0i2SeVPfSLgHp33+7u3cDjwLIebT4N3O/uBwHcvSW9ZYqErysa47ebW7huzngKde0WyUKpBHoVsCdpuiGYl+wc4Bwz+4OZrTazJb29kJndaWZrzWxta2vr6VUsEpI/1u/naHdM3S2StdJ1ULQImAlcBdwK/LuZjerZyN0fdPcad6+prKxM01uLZEZtXRPDS4u49OyxYZci0qtUAr0RqE6anhzMS9YArHD3iLvvAN4iEfAieSEWd1ZtbOaqWZWUFhWGXY5Ir1IJ9DXATDObbmYlwHJgRY82T5PYO8fMKkh0wWxPX5ki4Vq36yD7j3aru0WyWr+B7u5R4C6gFtgEPOHudWZ2n5ktDZrVAvvNbCPwAvC37r5/oIoWybTauiZKCgu4apa6CiV79XvaIoC7rwRW9ph3T9JjB74Y3ETyirtTW9fE+88eS3lZcdjliPRJ3xQV6cfGvW00HDym7hbJegp0kX7U1jVTYHDtHF2MS7KbAl2kH8/WNVEzdQwVw0vDLkXkpBToIiexa/9RNje1c70ulSs5QIEuchK1dU0A6j+XnKBAFzmJ2rpm5kwcQfWYoWGXItIvBbpIH1raO3l190HtnUvOUKCL9GHVxmCouXnqP5fcoEAX6UNtXTNTxw5l1vjysEsRSYkCXaQXbZ0RXtq2T0PNSU5RoIv04oXNLURizmKdrig5RIEu0ovauiYqy0u5oFpDzUnuUKCL9NAZifHbLa1cN2c8BRpqTnKIAl2kh99v3UeHhpqTHKRAF+mhtq6J8rIiLpmhoeYktyjQRZJEY3Ge29TM1eeOo6RIPx6SW7TFiiRZs/MgBzsi6m6RnKRAF0lSW9dESVEBV56joeYk9yjQRQLuzqqNzVwxs4JhpSmNziiSVRToIoENjW00HjrG9epukRylQBcJ1NY1JYaam61vh0puUqCLBGrrmrhw2hjGDCsJuxSR06JAFwG2tx5ha8sRnd0iOU2BLkLiUrmAxg6VnKZAFyHR3TKvagSTR2uoOcldCnQZ9JoOd7J+zyEWz1F3i+Q2BboMeqs2NgGweJ4CXXKbAl0Gvdq6ZqZXDGPmuOFhlyJyRhToMqgd7oiwevt+rp87XkPNSc5ToMugtmpTM9G4s0SnK0oeUKDLoPbrDXuZNLKMBdWjwi5F5IylFOhmtsTMtphZvZndfZJ2f2ZmbmY16StRZGC0d0Z48a19LJk3Ud0tkhf6DXQzKwTuB24A5gC3mtmcXtqVA18AXk53kSID4TebW+iOxbnhPHW3SH5IZQ/9IqDe3be7ezfwOLCsl3ZfA74BdKaxPpEB88ybTYwrL2XRlNFhlyKSFqkEehWwJ2m6IZh3gpktBKrd/VcneyEzu9PM1prZ2tbW1lMuViRdOrqj/PatFhbPnUBBgbpbJD+c8UFRMysAvg38TX9t3f1Bd69x95rKSo0II+H5ny2tdEbU3SL5JZVAbwSqk6YnB/OOKwfmAb81s53AxcAKHRiVbLZyQxNjhpVw0bQxYZcikjapBPoaYKaZTTezEmA5sOL4Qnc/7O4V7j7N3acBq4Gl7r52QCoWOUOdkRi/2dTM4rnjKSrUmbuSP/rdmt09CtwF1AKbgCfcvc7M7jOzpQNdoEi6/W7rPo52x1gyb2LYpYikVUoj4br7SmBlj3n39NH2qjMvS2TgPLNhLyOHFHPpWWPDLkUkrfT3pgwq3dE4qzY2c+3s8RSru0XyjLZoGVT+uG0f7Z1RbtTZLZKHFOgyqDzzZhPDS4u4bGZF2KWIpJ0CXQaNrmiMX9c1ce3scZQWFYZdjkjaKdBl0HjxrX0cPhZh2YKq/huL5CAFugwaK15/m9FDi9XdInlLgS6DQkd3lOc2NnPjeRN1dovkLW3ZMiis2tjMsUiMpfMnhV2KyIBRoMugsGL920wcWcaFunaL5DEFuuS9Qx3dvLi1lZvmT9KlciWvKdAl7z2zoYlIzNXdInlPgS557+nXGplRMYy5k0aEXYrIgFKgS17bvb+Dl3cc4MMLqzQQtOQ9BbrktZ+/2oAZfHjh5LBLERlwCnTJW/G48+S6Bi47u4JJo4aEXY7IgFOgS95avWM/jYeO8ZFF2juXwUGBLnnrybUNlJcWsXiuLpUrg4MCXfJSe2eElRv28qH5kygr1pUVZXBQoEteWvnmXjojcXW3yKCiQJe89MTaBmZUDmPhlFFhlyKSMQp0yTub9raxbtdBll9YrXPPZVBRoEveeWT1LkqKCrh5UXXYpYhklAJd8kp7Z4SnX2vkpvMnMXpYSdjliGSUAl3yytOvNXK0O8Ztl0wNuxSRjFOgS95wdx5evYvzqkYyf/LIsMsRyTgFuuSNV3Yc4K3mI9x28VQdDJVBSYEueeORl3czoqyIm3TdcxmkFOiSFxoPHWPlm3u5uaaaISX6ZqgMTgp0yQs//P0OAG6/bHrIlYiER4EuOe9wR4THXtnN0vmTqNJlcmUQSynQzWyJmW0xs3ozu7uX5V80s41m9oaZPW9mOmdMMuaRl3fR0R3jzitmhF2KSKj6DXQzKwTuB24A5gC3mtmcHs1eA2rc/XzgSeCf0l2oSG86IzF+9IedXHlOJbMnasxQGdxS2UO/CKh39+3u3g08DixLbuDuL7h7RzC5GtAl7iQjnnqtkX1HuvhL7Z2LpBToVcCepOmGYF5f7gCe6W2Bmd1pZmvNbG1ra2vqVYr0IhqL8+CL2zmvaiSXnDU27HJEQpfWg6Jm9jGgBvhmb8vd/UF3r3H3msrKynS+tQxCT73WyI59R/n8B87SF4lEgKIU2jQCyZetmxzMexczuxb4CnClu3elpzyR3nVH43zn+a2cVzVSQ8yJBFLZQ18DzDSz6WZWAiwHViQ3MLMLgO8DS929Jf1lirzbT9fuoeHgMf7m+nO0dy4S6DfQ3T0K3AXUApuAJ9y9zszuM7OlQbNvAsOBn5nZejNb0cfLiZyxzkiMf/3NVi6cNporz1HXnchxqXS54O4rgZU95t2T9PjaNNcl0qeHX9pFc1sX31l+gfbORZLom6KSUw53RPje/2zj8pkVXDxDZ7aIJFOgS075f8+9xaGObv5+yblhlyKSdRTokjM27W3joZd28r/eN4V5VRrAQqQnBbrkBHfnqyvqGDmkmC9dPyvsckSykgJdcsJ/v7GXV3Yc4G8Xn8uooRr8WaQ3CnTJem2dEf7xV5uYVzWCj15Y3f8TRAaplE5bFAnTff+9kdYjXTxw2yIKC3SaokhftIcuWW3VxmaeXNfA5646iwXVo8IuRySrKdAla+0/0sWX/+sN5k4awV9dPTPsckSynrpcJCu5O195agNtx6I8+hcLKCnSvodIf/RTIlnpoZd28eu6Jr54/TnMmlAedjkiOUGBLlnnlR0H+NovN3Lt7HHceblGIhJJlQJdssrew8f43KPrmDJmKN/+6AIKdFaLSMrUhy5ZozMS47OPvMqx7hiPffpiRpQVh12SSE5RoEtWiMTifP7RV3m94RDf+/NFzByvfnORU6UuFwldPO586Wev8/zmFu5bNo8l8zSknMjpUKBLqNyde/+7jl+sf5u/XTyL2y6eGnZJIjlLXS4Smljc+YenN/DYK7v5yytm8Lmrzgq7JJGcpkCXUHRGYnzh8deorWvm8x84iy9dP0vDyYmcIQW6ZNyhjm7ufHgdr+w4wFdvmsOn3j897JJE8oICXTJq/Z5DfP7RV2lp7+Q7yxewbEFV2CWJ5A0FumSEu/PQS7v4+q82Mq68jCc/cynzdfVEkbRSoMuA23Ogg688vYEX32rl6nPH8e1b5mvUIZEBoECXAROLOz/+407+uXYLZnDvTXP4+CXT9HV+kQGiQJe0c3ee3djMN2u3UN9yhA/MquTrf3oeVaOGhF2aSF5ToEvaxOPO/7zVynd/s5XXdh9iRuUwHvjYQhbPnaBTEkUyQIEuZ6yjO8rTr73ND36/nW2tR5k4soxv/Nl5/NnCyRQV6svIIpmiQJfTEo87q3fs579ebeSZN/dytDvGvKoRfGf5Am48byLFCnKRjFOgS8qOdkX547b9PL+pmec2tbDvSBfDS4v40PmT+EjNZGqmjlbXikiIFOjSp0Md3azZeZA1Ow/w8o4DbGg8TCzulJcWceWsSq6fO4HrZo9nSElh2KWKCAp0IdEHvvtAB/UtR9i8t53NTW1s2ttO46FjAJQUFrCgehSfuXIGl8yo4KLpYzRos0gWSinQzWwJ8B2gEPgPd/+/PZaXAg8Bi4D9wEfdfWd6S5VT5e4c6YrS2t5FS3sXrcGtpb2L5rZOdh/oYNf+DvYd6TrxnMIC46zKYSyaOpo/v3gKi6aMZn71KMqKtRcuku36DXQzKwTuB64DGoA1ZrbC3TcmNbsDOOjuZ5vZcuAbwEcHouBc5O5E404suEVP3McT97FgmfuJ6e5YnM5IjM5IjK5o4nFXJE5nNLiPxOiMxuiMxGnvjNDeGaWtM0LbsSjtnRHaOqO0HYsQjft76ikuNMaVl1E9ZghXn1vJ1LHDqB4zlBkVw5g5fjilRQpvkVyUyh76RUC9u28HMLPHgWVAcqAvA+4NHj8J/KuZmbu/N03O0BNr9vD9F7cB4ME/x9/E3XHg+Ls6jvs70ydtc2J5MPfE8neec3x58vTx939PG5x4HKLxOL1kaloUFhhlRQWUlxUzYkgR5WXFVAwvYUblMMrLihhRVszIIcWMG1FK5fCy4L6UkUOK9W1NkTyUSqBXAXuSphuA9/XVxt2jZnYYGAvsS25kZncCdwJMmTLltAoePayEcyeMgCCPLPG6xycxe2fe8eUYHG/xzvIe8+xE63e1Scy1E/NIfu1elp+YZ0ZhgVFUkLgvNKOw8Ph0wYn5RQVGQVK7ooICCgugpKiAsqJCSosLKSsuoLQocV9WXEhZcSGlRQU6NVBE3iWjB0Xd/UHgQYCamprT2m+9bs54rpszPq11iYjkg1R28RqB6qTpycG8XtuYWREwksTBURERyZBUAn0NMNPMpptZCbAcWNGjzQrgE8HjjwC/GYj+cxER6Vu/XS5Bn/hdQC2J0xZ/6O51ZnYfsNbdVwA/AB42s3rgAInQFxGRDEqpD93dVwIre8y7J+lxJ3BzeksTEZFTodMkRETyhAJdRCRPKNBFRPKEAl1EJE9YWGcXmlkrsOs0n15Bj2+hZpFsrU11nRrVdeqytbZ8q2uqu1f2tiC0QD8TZrbW3WvCrqM32Vqb6jo1quvUZWttg6kudbmIiOQJBbqISJ7I1UB/MOwCTiJba1Ndp0Z1nbpsrW3Q1JWTfegiIvJeubqHLiIiPSjQRUTyRNYGupndbGZ1ZhY3s5oey75sZvVmtsXMFvfx/Olm9nLQ7qfBpX/TXeNPzWx9cNtpZuv7aLfTzN4M2q1Ndx19vOe9ZtaYVN+NfbRbEqzHejO7OwN1fdPMNpvZG2b2lJmN6qNdRtZZf/9/MysNPuf6YHuaNlC1JL1ntZm9YGYbg5+BL/TS5iozO5z0+d7T22sNUH0n/Wws4bvBOnvDzBZmoKZZSetivZm1mdlf92iTkXVmZj80sxYz25A0b4yZrTKzrcH96D6e+4mgzVYz+0RvbU7K3bPyBswGZgG/BWqS5s8BXgdKgenANqCwl+c/ASwPHj8AfHaA6/0WcE8fy3YCFRlef/cCX+qnTWGw/mYAJcF6nTPAdV0PFAWPvwF8I6x1lsr/H/gc8EDweDnw0wx8dhOBhcHjcuCtXuq6CvhlJrepVD8b4EbgGRKjMl4MvJzh+gqBJhJfwMn4OgOuABYCG5Lm/RNwd/D47t62e2AMsD24Hx08Hn0q7521e+juvsndt/SyaBnwuLt3ufsOoJ7EQNYnWGLQz6tJDFgN8BPgTwaq1uD9bgEeG6j3GCAnBgB3927g+ADgA8bdn3X3aDC5msQIWGFJ5f+/jMT2A4nt6Ro7PqjsAHH3ve7+avC4HdhEYtzeXLEMeMgTVgOjzGxiBt//GmCbu5/uN9HPiLu/SGJciGTJ21FfebQYWOXuB9z9ILAKWHIq7521gX4SvQ1a3XNjHwscSgqO3tqk0+VAs7tv7WO5A8+a2bpgoOxMuSv4k/eHffyJl8q6HEi3k9iT600m1lkq//93DYAOHB8APSOCLp4LgJd7WXyJmb1uZs+Y2dxM1UT/n03Y29Vy+t65CmudjXf3vcHjJqC3gZHPeL1ldJDonszsOWBCL4u+4u6/yHQ9vUmxxls5+d75Ze7eaGbjgFVmtjn4LT5gtQHfA75G4ofvayS6hG4/0/c807qOrzMz+woQBR7t42UGZJ3lEjMbDvwc+Gt3b+ux+FUSXQpHguMjTwMzM1Ra1n42wbGypcCXe1kc5jo7wd3dzAbkfPFQA93drz2Np6UyaPV+En/mFQV7Vb21SUuNlhgU+8PAopO8RmNw32JmT5H4U/+MfwBSXX9m9u/AL3tZlMq6THtdZvZJ4EPANR50HvbyGgOyzno4lQHQGyyDA6CbWTGJMH/U3f+r5/LkgHf3lWb2b2ZW4e4DfhGqFD6bAdmuUnQD8Kq7N/dcEOY6A5rNbKK77w26n1p6adNIop//uMkkjiGmLBe7XFYAy4OzD6aT+A37SnKDICReIDFgNSQGsB6oPf5rgc3u3tDbQjMbZmblxx+TOCi4obe26dSjz/JP+3jPVAYAT3ddS4C/A5a6e0cfbTK1zrJyAPSgj/4HwCZ3/3YfbSYc78s3s4tI/Cxn4hdNKp/NCuDjwdkuFwOHk7obBlqffy2Htc4CydtRX3lUC1xvZqODLtLrg3mpG+gjvmdwpPhPSfQhdQHNQG3Ssq+QODthC3BD0vyVwKTg8QwSQV8P/AwoHaA6fwx8pse8ScDKpDpeD251JLodMrH+HgbeBN4INqaJPWsLpm8kcRbFtkzUFnwee4D1we2BnnVlcp319v8H7iPxCwegLNh+6oPtaUYG1tFlJLrK3khaTzcCnzm+rQF3BevmdRIHly/N0HbV62fTozYD7g/W6ZsknaU2wLUNIxHQI5PmZXydkfiFsheIBBl2B4njLs8DW4HngDFB2xrgP5Kee3uwrdUDnzrV99ZX/0VE8kQudrmIiEgvFOgiInlCgS4ikicU6CIieUKBLiKSJxToIiJ5QoEuIpIn/j+RA09qa2/eTwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = numpy.arange(-10, 10, step=.1)\n",
    "plt.plot(x, sigmoid(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To handle multiple classes, it's common to use a one-hot encoding:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# one-hot encoding:\n",
    "nb_classes = 10\n",
    "y_train_onehot = tf.keras.utils.to_categorical(y_train, nb_classes)\n",
    "y_test_onehot = tf.keras.utils.to_categorical(y_test, nb_classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training sample 0 : class: 5 , one-hot encoded: [0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]\n",
      "Training sample 1 : class: 0 , one-hot encoded: [1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "Training sample 2 : class: 4 , one-hot encoded: [0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]\n",
      "Training sample 3 : class: 1 , one-hot encoded: [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "Training sample 4 : class: 9 , one-hot encoded: [0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]\n",
      "Training sample 5 : class: 2 , one-hot encoded: [0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]\n",
      "Training sample 6 : class: 1 , one-hot encoded: [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "Training sample 7 : class: 3 , one-hot encoded: [0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]\n",
      "Training sample 8 : class: 1 , one-hot encoded: [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "Training sample 9 : class: 4 , one-hot encoded: [0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAABNCAYAAACi7r7XAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA7PUlEQVR4nO29d3Rc933o+flNL5gCDHoZ9F5IsJMiRbFIsmhLtpPYlsu6bOxd24nPxs9Zn3iTeP1233nZ+CU5LznZ5B1t7MS2FCeOLMV21FlEUSwSG0AQjQRAADOoMwMMBmUGwGDu/gHea4BiAQuIO/D9nMOjoyl3fl/87v39vr9vFZIkoaGhoaGhoaGxntGt9QA0NDQ0NDQ0NFYbTeHR0NDQ0NDQWPdoCo+GhoaGhobGukdTeDQ0NDQ0NDTWPZrCo6GhoaGhobHu0RQeDQ0NDQ0NjXXPfSs8QojvCSGefxCDUSuajMnPepcPNBnXC+tdxvUuH2gyqpUVKTxCiM8IIc4JIaaEEENCiNeEELtXe3ArQQjRK4SIXh/blBDizXu8jpplLBJCHBNCzAghOoQQB+/xOqqVUUYIsVcIIQkh/ss9fFe18gkh/m8hRIsQIi6E+N59XEfNMu4SQrwvhJgUQly613GpVUYhRKYQ4qdCiEEhxIQQ4qQQYvs9XkuVMsKDuVdVLt8xIURACBERQjQLIT56j9dRs4zrfr2RuZs9444KjxDiPwH/HfivQBbgBf4OuKebZJV4WpKklOv/nrjbLyeBjD8FLgIe4I+BF4UQGXdzgSSQESGEEfhr4L17+K7a5esCvg28cq8XULOMQog04FfAfwPcwPeBXwkhUu/yOqqVEUgBzgKbgTTgR8ArQoiUu7mIymWE+7xXk0C+/w3IkSTJCfwvwPNCiJy7uUASyLiu1xuZu94zJEm65T/ABUwBn7jNZ74HPL/k//8NGAYmgHeA2iXvHQLagElgAPjD66+nA/8BhIEx4ASgu93YllyzFzi4ks8mo4xABTALOJa8dgL46nqRccl1/4jFjfKfgP+y3uS7fo3nge+tw/v0I0DrDa9dAX53vch4i/FEgM3rUcZ7uVeTSb7r19kGxIBt61HGe5nDZJKRu9wz7mTh2QlYgJfv8LmlvAaUA5nABeCFJe/9APhfJUlyAHXA0euvfwvwAxksapL/ByABCCH+Tgjxd3f4zReumyjfFEJsuIuxgvplrAV6JEmaXPJa8/XXV4raZUQIUQj8z8D/dRdjlFG9fA+AZJBR3OT/6+5ivMkgo4IQYiNgYvE0vVKSSsZ7ICnkE0L8hxAixqJl4G3g3F2MNylkvE9UL+O97BmGO7zvAYKSJMVXekFJkn64ZEDfA8aFEC5JkiaAeaBGCNEsSdI4MH79o/NADlAoSVIXi1qefL2v3+EnP8viH1ewaKp8QwhRJUlSeIVDVruMKSxqzEuZAPJWOl7ULyPA3wB/KknSlBA37pt3JBnku1/ULuNpIFcI8WngReAzQClgW+l4Ub+MCkIIJ/AT4D9f/62VkjQy3iNJIZ8kSR+57g45CFRLkpRY6XhJEhnvk2SQ8a73jDtZeEJAuhDiTooRAEIIvRDi/xFCdAshIiy6m2DRbAXw2yyatvqEEMeFEDuvv/7fWDwlvSmE6BFC/NGKRg9IknRSkqSoJEkzkiT9GYumsT0r/T7ql3EKcN7wmpNF0+BKUbWMQoinWXTZ/esK5bkRVcv3gFC1jJIkhVj07f8nYAT4EHCYxdPbSlG1jEt+18pivNKZ62vO3ZAUMt4HSSOfJEnzkiS9BjwhhHjmLr6aNDLeB6qW8Z73jDv4x1zANPA7t/nM97juxwP+J6AdKGbR4uJm0TxVdsN3jMA3Ad9NrlcHjAIHbje224ynHXjmLj6vahlZjOGJsTyG5x3uPoZHzTL+dxZjIYav/4uyqOj9Yj3Id8P37sennhQyXv+uAegHnlxPMgJm4A0WzfX3EjOiehnv515NJvmWfP8w8M31KOO9zGEyyMg97hm3tfBIi6ao7wL/rxDiY0IImxDCKIR4Sgjx/Zt8xcFigG2IRVP2f5XfEEKYhBCfvW7imr8+2MT19z4ihCgTi3apCWBBfu92CCG8QohHrl/bIoT431nUKE/e6bvJIqMkSVeAJuD/vC7jx4EG4OfrRUbgT1lU7DZe//dL4P8DvrRO5OP6eCwsWlUN1+dSv5LvJpGMjdfH5AT+gsVF7Y31IqNYdIG8yOLi+gXp7twgSSGjLOe93qtql08IUXV9LNbr4/oc8ChwfCXyJYOM17+73tebe9szVqjtfZbFoK5pFrWpV4BdN9HyUoBfsOhu6QM+z3Utj8XgvtdZ9N1FWEzv3H39e99k0QQ2zaIJ/E+X/Pb/AP7HLcZVC1y6/r0QcATYcrfarJplvP5+EYuBdVGgk3vMSlOzjDeM85+4iyytZJDvukzSDf++uM5k/CmLi9YE8K9A5nq6T4G9168/w+JpUv63Z73I+KDuVbXKB1SzGKg8yWL4w1ng4+vpPv1NWW9uIu8d9wxx/cMaGhoaGhoaGusWrZeWhoaGhoaGxrpHU3g0NDQ0NDQ01j2awqOhoaGhoaGx7tEUHg0NDQ0NDY11j6bwaGhoaGhoaKx7bltFUQiR1ClckiTdsd60JqP6uZOM610+0GRMBjQZ1798oMmYDNxKRs3Co6GhoaGhobHu0RQeDQ0NDQ0NjXWPpvBoaGhoaGhorHs0hUdDQ0NDQ0Nj3bOi1u8a94der8dsNpOSkoLNZsPlcgEwMTHBzMwMU1NTzM7OsrCwsMYj1dDQSCZSUlJITU3F5XJhMpmYnZ1lamqKoaEh4vE4icRd9zfV0FAlKSkp5ObmApBIJPD5fMzOzt7VNTSF5yGQkpJCcXExe/bsYePGjTzxxBNIksRbb73FhQsXOHnyJD09PUQikbUeqoaGRhKxceNGPvaxj3Ho0CHy8/Pp6enh5MmT/Pmf/zljY2NMTU2t9RA1NB4IdXV1fPe730UIwezsLN/61rfo7u6+q2s8dIVHCIHBYECnW+5Ny8/PJz09HYPBgMFgwOFw4Pf7uXr1KrW1tWRlZVFYWIher2dhYYH+/n4GBwdpa2tjZmbmYYuxInQ6HampqZSXl/P0009TUVFBYWEhbrcbSZKor6/H4XCQnZ3NCy+8sO4VnvT0dNxuN4888gjBYJBTp04xMzNz11q6WrDb7VRXV5OXl0dRURFCCKLRKEeOHGF8fJxQKLTWQ9S4AXn90ev1GAwGsrOzsdlsmEwmAoEAfr8/KSytOp0Ou91OQUEBDQ0NpKenY7PZyM7OJisri7S0NMV6nIzo9XocDgdCCGUddbvdbN26FbPZzJ2aXg8PDxMKhTh58iSxWOyOn19rdDodaWlpOJ1OGhoaCAQCnDx5cq2HpQpMJhObNm1i27ZtFBUVMTIyQjgcvqfn9KEqPDqdDp1Oh9VqxWg0LnuvoaGBhoYGLBYLNpuNgoIC3n77bQKBAE888QRbtmzhySefxGQyMTc3x1tvvcXp06fx+/2qVXj0ej05OTls2bKFr33tax+Qe9OmTdTV1bF//35OnjxJR0fHGo529cnLy6OiooLvfOc7tLS00NXVxcjISNIqPE6nk4MHD7Jnzx4+9KEPodPpCAaDjI6OcuXKFU3hUSEGgwGr1YrZbMZisbBp0yY8Hg9ut5uLFy8yPDyMJEmqdwUZDAZSU1MpKipi8+bNpKSkIIQgIyODrKwssrKyGBsbW+th3jNGo3HZAbi2tpaSkhL+6I/+CKfTecf5ef/992lra6OlpYX5+Xni8fhDGvndo9Pp0Ov15ObmUlRUxFe/+lUuXryoKTzXsVqtHDp0iIaGBoqLiwkEAgSDQXUoPCaTCafTiV6vR6/X4/V6lQfTYrFgNpupra3F4/Es+57H48HpdKLT6ZAkibm5OQDS0tJ4/PHHKSoqIh6PMzU1xfDwMB0dHbS3txOLxR60CA+E3NxccnNz+f3f/33Ky8ux2+2KVWt2dpZEIkE8HsdgMGC32yksLKSsrAyfz8f8/PxDWXAtFosyLxaLhc7OTsbHx1ft93JycsjPz8fn8zE8PLxqv7Pa6HQ6tmzZQnV1Nb/9279Ndna2sknKJ0m1nyh/k3C73bhcLvbv309mZiZer5f09HRcLhdpaWmYTCaMRiN1dXUUFhby+uuv09fXt9bDvi0LCwtEIhEGBwdpbW2lurqatLS0tR7WfSHHIm3ZsoW8vDy2bdumeANyc3Nxu91YrdYVPVvl5eWkpqby5S9/mdbWVt58803m5uZUqfgYDAZsNhtf+MIX2LVrFx6PB7/fT2pqalJbwB8E+/bto6amhkOHDpGRkUE8Hsfv99PS0kI0Gr3r6z1QhUd2RRUVFSmaeXV1Nenp6dTU1Cgb65YtW8jMzLzldWKxGL29vbjdbkpLS3G73RiNRkKhEBMTE/T39+Pz+RgaGmJ+fv5BinBfyOZXWVsvLy9n27ZtZGVlYTAs/qllZW52dpZIJILL5cLj8ZCfn09paSlTU1PKv9XGaDSSk5ODw+HAbrfj8/lWVeFxOp3KQ3wvN6takBfgkpISysvLsVgsSJKEECJpFB35PjWbzRgMBoxGo/KazNTUFAsLCyQSCeUAIwfXq+m5uxmyTGazmby8PDIzM9m2bRt5eXmUlpaSkZGBy+VS5k12lU9PT3P27FnVu7YSiYSyhoyOjlJaWrrWQ7pvXC4X+fn5NDY2UlxczCOPPKJ4BdxuNxaLBVhcQ+/0nLndbsxmM1u3biWRSHDixAnlkKk2dDodRqORmpoatm7dyvj4uJLgIu8Vv6mUlJTQ0NBASUkJRqORubk5gsEgPp9PMYrcDQ9M4TEYDJSWlrJ7926+/e1vY7ValZOTTqdTNnydTofJZLrldSRJYmRkhOeee46pqSlisRhHjhwBIBQKEY1GiUQi+P1+gsGgahZeIQRms5n09HTy8vL4+te/zqZNmygqKlrmxpIkieHhYfr6+njppZd47LHH+OQnP8k3vvENPvOZz/DDH/6Q5uZm/uM//mPVx2y329m0aRPp6el4PB7a2toYGBhYld/S6XR4PB5yc3PJyclhdHQUIQRC3LHKueoQQpCenk5GRgYmk2mZkpAseDweCgoK2LVrF4WFhdTX15OWlkZOTg46nY65uTn+8R//EZ/Px9jYmBJDd+rUKfr6+ujo6FDl5gEoSk5DQwO7du3ikUceoaCgAJfLpRzEJicnGR0dJRqNYjQayc/Px+v14nA4OHHiBGNjY/h8PtUqPZIkEYvFCIfDDA0NqdbSfTc8+uijHDp0iEceeQS3243JZFLWhxtjPleC1WrlwIEDWCwWjh49Sm9vr+r/TnK8ksfjwePxEIvFmJ6eXuthrQlCCAoLC6mursZisTA5OUlHRwenTp3i2LFj96QIPlALTzweRwiBy+XC4XAoGvnNkCSJSCTCwsIC8XiclJQULBYLiUSC6elp2tvbmZmZURbVRCJBJBJhbm6OWCzGxMTEPWl4q4XNZqO+vp6CggLKysqoqKggOztbUfhu/KzFYmFmZoaJiQlCoRBOp5P09HSys7Pp7+9/KGM2m82UlpZitVrR6/WrpnwYDAZMJhO5ubl4vV5isRiTk5NMTU2pag5XQkpKCi6Xi5KSEgoKCtDpdMzPzzM7O0tnZyd9fX2MjIyoNljUaDSSkZFBRUUFGzZsoLGxkZycHEpLSzGbzZhMJgwGAxaLhQ0bNpCXl8fk5CSpqalkZ2czMTGByWSiq6tLVQqPfKgqLi4mPT2d6upqKioqaGhowOv1kpGRgcFgYGFhQZmrnp4e5ufnSUtLIzc3F5PJhMvlwm63Y7FYVK2My1YpudyFfKBMZubn54nFYpjNZqxWK4CyPwQCAaLR6E2tqE6nE4vFopT7WIrVasVisdw0UUaNyPOq1+vR6XSqvgfvBovFgt1uJy8vD5PJRFtb221LscjlFjIzM3G73cRiMUZHR7lw4QIDAwP3rLg+sKdkYWGBUChEKBRicnJSCQq8FfF4nJ6eHqanp5mZmaG8vJy8vDzi8ThjY2O88847qtfGl5KRkcFXvvIV6uvr2bRp0y0/J4RQ5LRarUQiEVpbW6mrq8NqtSqppQ8Dh8PBnj17CIfDXLt2bdV+x2w2k5qayrZt29i5c6cSI+H3+1ftN1eL3NxcSktLeeKJJygtLUWv1xOJRBgeHuav/uqvOH36NKFQSDWWxxtJSUlh586dHDhwgN/6rd/C6XQqJ+lgMEhrayspKSlYrVaeeOIJZeOBxXvX4XBw+fJljh07pqrn02QykZKSwic/+Unq6uo4cOCAEpy8lFgsxvDwMP/6r//KL3/5SwwGA/X19Tz++OOYzWaMRiNOp1MJAlYrcpZWRkYGRUVF2Gy2tR7SfTMwMEBLSwv79u1TXpubm2NycpJ33nkHn8+nvC7HOBoMBurq6sjKymLjxo1JodT8JuLxeCgtLeUTn/gE6enp/PEf/zGjo6O3PBjm5eWxadMmKisryc7OJhQK0dTUxI9+9KP72jcemMIjSRIzMzNcu3aNf//3f6ekpITs7GxGR0ex2+089thjiuk/FAoxMjLCP/zDPzA2NkYsFsPr9ZKTk0NDQwODg4Oqz5KQEUKQmZlJcXExtbW15OTkKAtlPB5X4mJ8Ph/19fUUFxczPT3N+Pg4fr+fWCzGzMwMubm5FBQUKHFAD4uH4Y6pqqpi//795OfnI0mSEouVTMgpvwcOHGDXrl3k5OQom2l/fz8nTpygp6eH8fFx5ubmVHX/yvdUVlYWJSUlPPvss5SWlmK32wkEAoTDYU6fPq2UeTCbzdhsNj796U/j9XopLCxU7slAIMDg4KDqXD1yaYAdO3ZQVlaGzWZbdm8nEgnC4TDt7e28+OKLvPfee4yNjZGWlpY0cVdLEUJgMpmw2WykpaUtc5vbbDYKCwuVw6dsSVc7PT09zMzMMD8/j9PpBFhmPb1VfOH7779PRUUFXq8Xu92uKOnxeJyrV6/S3t5Of38/k5OTD02W+0We3/WiwFVUVPDJT36SzZs3KzGuc3Nzt1R4CgsLefLJJ/F6veh0Ot566y2amprw+/33ZT1/oHbQubk5hoeHOX78OMFgkPz8fLq7u/F4POzcuROLxYJOp2N8fJze3l5eeeUVRkdHmZ2dJSsri4yMDObm5ohGo0mxCMnmx+zsbLxeL8XFxaSkpACLC+zc3JwSYN3U1ITb7SY/P59wOEwoFGJ0dJSJiQnC4TCHDh1S6gzJ5szV3DTlTfBhKFgFBQUcOHCA9PR0EokE4+PjqnX53Aq5VMK2bds4cOAAqampGAwGEokEQ0NDvP/++wwNDalSLr1ej8lkIj8/n+rqavbt26coBKOjo1y7do1f/OIX+Hw+Ojs7MRqNpKSksHnzZqxWK16vV3kex8bGGB4eVpVCByhuupqaGgoKCpTX5QDX+fl5gsEg7e3tvPzyy4yPjzMzM0NqaqqqLTm3Ql57TCaT4pKWkS3FPp+PYDDI9PR0Uig8g4ODDA8P4/P5FBddIpFgYWGBiYmJW8ZsOBwORkdH+cpXvqL8PWBRWert7aW3t5fR0VHVWl2XIj9ncqxrMsYH3oyCggL27dtHbm4u0WgUj8dDMBi86WeFEOTk5LBt2zYyMzOZm5vj7NmztLW13fI7K+WBO36DwSAnTpzg0qVL2Gw2JiYmyMvLo7y8nKqqKurr63nnnXd47733lNMwLFp9IpEIzz33nGqj6W8kIyODzMxM/uRP/oSqqipcLhd6vR5Jkujo6ODatWt8//vfJx6Pk5OTwxtvvEFzczNHjx5lYGCAjo4OhBAYjUaCwSAGg4EtW7YQjUapqKhgZGRkVbKmhBAUFBRQVFSkBMatJkajEbvdjsFgYH5+nqamJq5evbqqv/mgycrK4qmnnqKmpgaPx4NerycajSpBdG+88YZqT5BVVVVUVVXxta99jeLiYmWD6O3t5bnnnuPixYuKtTEej1NZWUllZSU7duygpKQEnU5HNBplcnKSU6dOqdLdXFFRwd69e3E4HMpr09PTTExM4PP5GBwc5Mc//jH9/f2MjIywsLCgLKzZ2dlJp/TE43GCwSCXL1/m1Vdf5SMf+QjFxcXA4un4S1/6EhkZGbjdbl5//fWkiJVLJBIkEglCodCy+ZAk6ZYKm06no7a2loaGBiWWZ+n3QqEQ4XCYubm5pDhEy6SkpFBZWcnY2BhDQ0NrPZz7ZmFhgVgspsyjHCt4I2azmYKCArxeL1lZWUq7FNndfL88cIUnHo8zMTFBNBrFYDAQi8UwmUyMjo6Sl5cH/PoGXppeGI/HicfjSZGuLKcRZmZmUlJSQkVFBUVFRcDiqWJhYYHBwUF6enro6upSTmOweIppaWkhFAopBROFEMTjcXQ6HS6Xi4yMDIqLi4lGo6um8KSlpeHxeDCbzat6ijAYDIqLRKfTEY/HGR8fV61ycDPkqq8VFRV4PB4l+HVmZoauri4lk0ltWCwW0tLSqKqqorGxkYqKCjIyMhQLa1NTE+3t7fT09CyzqmZlZVFRUUFaWpoSGxIOh+nt7WVwcJBgMKg6C4+cyCBb2eSEgGAwqIy7tbWVsbEx5aSv1+uxWq3Y7fY1Hv3dI0kS8Xic0dFRLl++zN69e5X3zGYzOTk5eDwe5RCWTKz0sGuz2bDb7RQXF1NQUPABF5AcUjA6Oqp6ZUeu45VIJBTrTmpq6m0zmpMBuYBkWloaFotFUXzkw9VShBBYrVaKi4vJzMxUigxPTU0xMTHxQKznqxbaPzc3p5wq5PoW8kKzdetWHA4Hv/rVr5iZmVH9zXgjVquVnJwcHnvsMXbu3KlkeEQiEaampohEIoqVa2JiglgsxsjIiJKGLSt7N8NkMlFcXMyzzz7LT37yk1UpgKbT6diwYQMbN25UMhhWA7kabEZGBjk5OVgsFmKxGGNjY0nTRkOn0+F0OikrK+PDH/6w4mqcnp6mv7+fF154QbXWqqKiIj71qU9x8OBBtm7disFgIBgM8stf/pITJ07w2muvKZmPS9m2bRuf+9znSE1NVV6TAwZbWlpUGX91+PBhzp8/zzPPPIPJZOLcuXNKEkU4HFYW2KXPnZxR6nQ6k87CI9PW1saVK1d45plnaGhoWOvhPFQqKiqorq7m6aefpri4WDlUyUxPT/PCCy+oPjlCNgBEo1FmZmaULMHc3FwlRCJZSU9P54tf/CK7d++mpKSEoaEh+vr6uHr16gcsV2azmezsbJ599lnq6+uxWq1cvXqVrq4ujh8/TiAQuO/xPJRcxqmpKd577z1sNhtlZWU4HA4KCwvxer0IIR6IIA8LnU5HRkYGe/fuZfPmzVRVVWEwGAiFQhw7doxQKEQgEKClpYXBwUFlkV2pUie7uGSNeDUQQuB2u3G73QghiMViq5JZ5HA42L17N1VVVaSkpCh1Q26m3asRq9WK0+nk8ccfZ+fOnRgMBmVx6uzspLW1lZ6eHtW1kDAajdTX1yt1aPLz89HpdAwMDHDt2jWOHTvGlStXmJycVOZBr9djs9nIyMggOzub1NRUjEYjMzMztLW10dTUREdHh2otczMzMyQSCc6ePYvBYKC/v5/p6Wmmp6eJRqM3dYnIrRgyMjKSVuGRLT1LC18mqyx3wmAwUFRURHp6OuXl5ZSXl+P1eqmqqiItLW2Z3JFIhFAolBRrTTweZ3Z2lkAgwNDQkOKaTPZ5tNlseDweGhsbycvLQ5IkmpubaW5u/kBMlk6nY/PmzVRXVyudGKampjh27Bjnz59ftlbdDw9F4YlEIrz55ptYrVbq6+upq6vD6XRSWVlJIpEgGAwmjZXHYDCQn5/PRz/6UWprayksLGRmZoahoSF++tOf4vf78fl8mEym2/qeb4fJZFIqha4GQghSU1OVgM3p6WmGh4cfeEVPl8vF008/TX19PSkpKQwMDCj1NJIhpsDhcJCXl8cXv/hFZRGCRYvlhQsXuHDhAleuXFHdgmqxWNi7dy9bt27l4MGDSu2Zrq4umpubeeWVV4hGo4qCu1TJrquro6CgALfbjcFgIBAIcOTIEU6ePMnly5fXWLJbI5vJ33333RV/R66YnZubm/Sby9Iinsmylt6JG+fEZDKxYcMGNmzYwMc//nEyMzOVYpJLkSSJYDDI0NAQ8Xhcde7XG5Hd4wMDA/T19eH1etd6SPeNEAKn00leXh7bt2/H6XSysLDAu+++yzvvvEMoFFq2BxgMBvbt28emTZtobGwkFosxPj7OSy+9xOHDhx/YuB6KwiOXQT979ixzc3P8wR/8AbW1tXzxi19U+mIEAgFCoRCnT59WpckcFifF6/VSXV3N5s2bcTqdzM/P84Mf/IBLly5x/vx55UQpm1bV/rDBYnzGlStX7quip5wxUlBQQHZ2tlJles+ePUqPn3fffZfz588zNjaWFOXSS0pKqK6upqSkhIyMDGCxVsjAwACvv/46HR0dqst+MZvNSv85ud2Az+fD5/Pxt3/7t1y5coWZmRll3HJV1y984QsUFBRQWVmppII2NzfT0dHByy+/zODg4FqKdV9s2bKFnJwcpd6QbAkxGAzU1NQopSTGx8eVPn09PT2qm9vbIVuRk1nZSUlJweFwsHHjRrKzsykrK1vmojKbzWzfvh2Px0N2dvayLCa58nRbWxvvv/8+ly5dwu/3L4vZ0ng4pKWlkZaWxpe//GVqa2tJS0vD7/fT3t7O2bNn6ezsXDYnDQ0NVFVV8fjjjytJEleuXOH48eMPfN15aOU5FxYWGBkZ4fz58/h8PrxeLzU1NWRnZ+NwOBgYGGB4eFg5McdisWXNGNWAXq+noKCA/Px8srKylNikCxcucPHiRUZGRu5LwVmrXkxyca+bLQxL+yvJtSGWprILIbBYLBiNRqxWK6WlpUoH54KCAiW+SZIk+vv76ejouKWLQS0sVd5KSkoUa5tcduHq1at0d3fj9/tVdX8CSpXkwsJCsrKykCSJQCDAlStXlGwsOW5L7qWWl5fHnj17KCgooLCwEFh8Xv1+P11dXVy9elUJsE8GhBBKdW+LxUJ1dTXFxcVK9WT5OdPr9eTk5Cjdt8fHx+nu7mZkZIRwOJwUh5WlJGvjWjmpIz09XSkg6PV6aWhoWBZwbTQaqa2txWg0fkBG2UrS19fHmTNnaG5uZnR0lFgslnR/D/h12ZBkqsMjV/7Ozs6msLCQnTt3UlZWhtVqJRaLMTAwwPT0tGLZkZ/FvLw86urqKC4uJiMjQ/nsxYsXH3is50OtRy5nT/zwhz/k/fff55vf/CZFRUWUlJQoViCn08nly5d5++23lfYDaiElJYWvfOUrVFVVIYTA7/fT29vL+fPnuXLlyn0rO2tlVpdL6t8s7S8zM5O0tDR0Oh0pKSlK+qfT6VS6/O7fvx+r1ap0uk8kEkqw9fz8vJLV5vP5VNeS4Gbk5+dTU1PD5z//eTZv3ozdbmdiYoKuri6ef/553njjDfx+vyqtVHq9XqkWLGcfHTlyhH/+538mFotRVFTEwYMHKSkpoaqqCq/Xi9vtJjMzc5lrIB6Pc/LkSS5evMj09LTq50xGbkvg9XrZvHkz+/btUxr4ys/Y0oOFyWRSlJ2jR4/y3HPP0d3dnXTKTjJjt9vJz8/n85//PE8++ST5+fnKIerGNfFWqckzMzM0Nzdz/PhxXn75ZaX4Z7LOo9FoVPqJJQNyMcGdO3fyO7/zOzz22GPKHiH3HtywYQPT09PU19fz85//nPn5eex2Oxs2bGDfvn04HA4ikQhvv/02r732Gi+//PIDX2MfqsIjx7TIvaJOnTpFfn4+5eXl2O123G43DQ0NWK1WAoEAPp+Pvr4+VVSudbvd5OTkUFBQQHp6OrBYhE0OjrzfDWHp6Swej980e+ZBsnQxyMzMVMqy31jYKS8vj/T0dMWSU1ZWhsViwWq1Mjc3p6SaT05OMj09TSQSYXJyku7ubpxO57IT2eTkpKpPzjqdDrPZTHFxMY8++ihFRUW4XC6lX1Y4HCYYDBIIBJifn1f1yXFhYYGFhQUl0HPnzp3Mz8/jcDjYvHkzOTk5FBYW4nK5lHRR+UQpW1h9Ph8DAwOqtsbBr9NZrVYrDQ0NZGRkkJeXR2VlJRUVFUo8UiQSUeLjlh4w5LmUA7flRVrN87uekPugORwOPB6P0u7kZtzqUChnhLrdbhwOB+FwOGmU9JshZ2klQ8kE2VJaUVHBI488Qnl5OR6PB51Op1jeDAYDmZmZ1NbWkpqayujoKPF4HJfLRU1NDVlZWQCMj49z+vTpVbMqr0nHuStXrnDt2jX8fj+NjY186UtforKykvz8fA4dOkQgEMBut3P69GkmJyeV9hNrSVlZGTU1NZSUlODxeJAkib6+Ps6fP//AJkZWBKanp+nt7V3V1O35+Xlloa+vr6esrEwJKF5Kbm6uovDICms8Hmdubg6fz6fEXY2OjtLd3c3Vq1fx+/2Mj49TXV2tpAnLmWzDw8OrJtP9YjAYyMjIYN++fXz7299e9t7c3BwjIyOqb4uxsLDA3NwckUhEscQ9++yzPPvsszfdwMfGxpTYLYvFQmpqqlLTpqWlhba2toctwl0hu0MyMzPJz8/ne9/7HpWVlXg8HhYWFpifn2dmZoZAIMClS5dITU2lsbHxA4XP5ADmHTt2EAgEiEQiWuzHGnA7S/ftArPtdjubNm3C5/Nx4cIFzp8/z+jo6KqPd7VITU1l69atiiKgZkwmE1u3bmXPnj18/etfV+ZI7jYwPDyslPYoKytjbm6OLVu2oNfrFQXXZrMptb7+/u//ftX2+zVrsSvH9Fy8eBFJkqipqVGaMtrtdnbt2qV0S33ppZfo6upaq6ECKDEPcmry1NQUPT09XLx48Z4VHr1eT3FxsaLhxuNx+vr6uHjxIr/4xS9WrYloPB7nzTffpKenR0lRd7lcRKPRD5yK5ubmGBoaoru7W0n1HB8fJxAIMDk5qdTViUajSpuMmZkZqqurFfeXJEnLqmqrFafTyYEDB6ioqPjAgjoyMsJbb7310DrZ3yuzs7OEQiGef/55NmzYwNNPP43ZbMZgMBAOhxXr29DQkKJU6/V6nnnmGbKyskhNTcXn8ynBzWpGr9fjdDopKCjgiSeeYPv27ZSUlLCwsMBbb71FIBBgeHgYv9+vuMdra2spKCggLS1NqXGi1+ux2+1UV1fjcDgIBAJYLBY6OztVYV1eKTempaenp1NSUrJq2Z4PilgsxtDQEK+++ip9fX1s2rQJg8FAX1+fYqmUWarwFBcXk5eXx8aNG5O+Xg2glI3YuXNn0hSLLCkpwev18ulPf1oJTZFr8L366qvKIbG6uppt27aRm5uLxWIhLy8PnU6HxWJR4kJtNhulpaV897vf5ezZs5w4ceKBezrWTOGR/eaTk5PK4ltVVcWOHTtIS0ujpqYGl8uFx+Ph5MmT9PT0rOnCo9frMRgM6PV6EokEMzMzDA4O0tXVdU9+Rnmyi4qKePTRR0lLS1Mqg7a3t3Pq1KlV03ITiQTnz5/H7/eTlZW1LFj1RoLBIAsLC5w6dYqRkRH6+voYGhq67cZvMBgoLi6moqICm83G9PQ04XBY1SdmIQQpKSls2rRpWe8oQCl3f/bsWdWfGmV36BtvvMH09DS7d+/GbrdjMpkIBAKMjo5y9uxZWltbOXfuHDMzMzgcDnbt2oXL5QJgdHSUq1evqjJGSUYIgdlsJjU1lcrKSvbt28eTTz5JIpFgYGCA06dPc+3aNbq6uujs7CQSiZCenq64VuUNUlbw5SB1r9er/F3kqs3yc7j0nlCbu+tm1o+0tDQKCgowm82qdtHNzc0RCoU4c+YM7e3tTExMoNfrOXfuHHNzc7d0TW3fvp2GhgYlJCLZCYVCyxrz3izmTG0UFBRQV1fHvn37lFprcmiDfKiWQxkyMzOV9h+y10BGkiTMZjO5ubl87nOfw2w2K8aEdaHwyMTjcaampmhpacHn8/GNb3xDKa+dnp6OzWajsbGRyclJ2tvbVbFpJhIJ5ufnmZiYIBQK3bWvWPZnPv300+zatYvHH38cIQQ+n4+XXnqJ1tZWpZDaahIKhfiXf/kXjEbjbU+BcmVh2ZV1pznQ6XTU1NRQXV2NTqfD7/dz6tQp1RXokxFCKOUGtm/fTk5OjvLe3Nwc58+f5/z58/T396vi/rsT8XicK1euMDg4yPHjx9Hr9QghFDemXDphZmaGoqIivF4vGzZsUArwyS0a1CqrTqfD4XDwiU98grq6Og4dOqQ0pm1paaG1tZWf//znjI+PE4lEMBgMFBQU8J3vfIfKykol3Xl2dpZXX32V6elp8vLy8Hq9lJaW8uyzz/Lkk09y5swZrl69yuHDh4nFYsrfIxaL0d/fr6pN6GZp6aWlpXg8HiorK5mdnWVwcFBVY76RSCTC9PQ0L7/8MoBShf9WYw6Hw1y8eJGDBw+SmZn5MIe6KoTDYUZHR0kkEkqmocvlIisri2AwqMqYpB07drBz504mJyfp6Ojg8OHDNDU10dXVxdDQELOzs4qBoKenhy984QvU19dTXl5+0wD0aDTKxYsX6ejoUAr3PkjWVOGRzVkpKSnKf2/0q8sBhGpifn6e8fHxm7qAbofRaFR6cHm9XiX90mQy0dnZSX9/v7JRPQxrltyLZzWw2+1KH6aJiQl6e3tV6yLR6XRUVFRQVVVFenq6clqMxWJMTk5y+fJluru71zyObKVIkkQ0GiUajd6xirnBYMBut2O327FarSQSCSKRCMPDw6pVeNLT08nMzKShoYGamhqKioqYmJigv7+f1tZW2traGBoaUk6G+fn5eL1e6uvrlSKDgUCAYDBIc3Mzk5OTBINBJicnkSQJh8NBamoqtbW1SmxBLBZjdnaWeDzO2NgYfr9fVcHc8uFLDsgGlMzJ3NxcRkZGGBoaUrXCI7uvVtoRe2Zm5qEcDB8WSw+TsmVHPoyqNT1dzszt6Oigt7eXixcvcunSJXp7e5d9LhwOMzw8vKxn3/T0NKOjo8useOFwmLa2Nvx+/6qEQKyZwiNnj3i9XrZv347X6yU7O1spegaLGv/g4CCXL1+mtbVVNQvM2NgYJ06cuOsutnJPqU996lNUVlZy8OBBfD4fJ06c4C/+4i84d+5cUlQGvVsGBwc5ceKEKhtswqI741vf+hYbN25cVqLe5/PR3d3NX/7lXzIyMrLGo1wd5Ng0+HXQfGdnp1IWQk3ItUmeeOIJtmzZwmc/+1mlxsfhw4c5efKkErsTDofJyMjA6/Xyu7/7uzQ2NlJfX8/8/Dx+v58XX3yRY8eOKe6rlJQUcnJyKC0tZc+ePZSUlLBr1y6qq6v56Ec/qpRbCAaDXLhwgXfffVc165EkSZw+fRohBB/72MewWq3Ke3q9Xuki39LSsq7WlqKiIqqrq9eFOwuWN9WWkZ9PtR36Zdra2giHw7zzzjuMjY0pFqobkUuf5OXlUVBQgF6vp6Ojg7/5m79haGhIsf7Pz88zMjKyaofjh6rwyF2z5QJDjY2NZGdnU1VVRWpqKi6XS3GtSJLE7OysUhBPTQ+qzWajpKQEp9O5os+7XC5KS0vZsGED5eXlbNmyBbfbrdTwOXr0KP39/aqOmbgfbvYgqw273Y7D4Vi2sHR3dyvNMpPFunO3BAIBnE6ncsKSXQhq2cyXkp6eTl5eHlu3bqWxsRGj0UggEOD8+fO8//77tLa2YjablUNUQUEBpaWlNDQ0kJ6eTktLCyMjI7S2tnLmzBl6enqUGIGpqSklVXZhYYGOjg5isRhZWVkUFRUBixaIEydOcPnyZVWtR/Dr3lE3G5der1ddEKzsssnJyVEC7Ve6RjgcDrxeL4888ghbtmxZ8TqsdoaHh7FarYRCIQwGg1JPasOGDQSDQVXuD11dXYyMjDAyMnLTgrKylSojI4OKigqlA/zk5CTDw8NcvnxZcWXC4jM2NTW1au67h6rw2Gw2srKy2L9/P9XV1ezbtw+Xy4Xb7V72Ofk0NTs7qwS7rvVmuTSAzOVyUV9fj8fjWdH35GajTz31FFu3blUK2b355pu89tpr/PSnP30IEmjcCp1Oh81mw2KxLIsZaG9v57333lvVB3CtGRgYAFClgnMjOTk57Ny5k71791JbW0ssFqOvr49/+7d/o729nYGBAWpqavB6vRw6dEgpJWE0GpmamlLi444ePcr4+Piyoqazs7OMjo4yOjpKR0cHVquVYDBIeXk5jz76KLDocvjJT35Cd3e36u4HuUZUMswjLCphFouFyspKIpEIkUhkxYp2Wloa27Zt49ChQzz66KOr1mT5YdPX10c0GmVoaAir1YrD4aC8vJzdu3fz3nvvEQ6H13qIH6C1tfW27+t0OqxWK7m5uTQ2NpKRkYHRaGR4eBifz0dTU9PDGeh1Vl3hcTgcuN1uNm7cSFVVFY2NjUqdDI/HsyxwSZIk/H4/w8PDHDlyhK6uLtra2ujs7FztYd6RpUGBBoMBt9vN/v370ev1nD17VknHdrvdZGVlUV9fT3Z2tlLHpry8HJvNxtTUFJ2dnfj9fl577TXa29vXWrRVQ1YQ7XY7OTk5SvVlNbF37162bdtGZmamYt2RsyLk1gpq29weJCUlJZSXl3+gAaMaqaio4Nlnn1XicHQ6HXl5eXz84x/nqaeeIh6PU1RUhNPpJDMzE7vdjtFo5MSJE3R2dvKzn/2M0dFRpXDk7ZidneXMmTO0tLRw/PhxYNHld+3aNaanp9f8AHYj586dY2RkhE984hPk5eWRmpoKLLpEHn30UVJTU3nxxRcZGxtbU1dlWloaOTk57N69m+LiYnbs2MHFixeVEgk31gGTsVgs2Gw2pdzFZz7zGYqKipZloM3Pzysp0WqzwK2U+fl5xYKYlpbGu+++y89+9jPVJnzcCZfLxcGDB9mzZw8f+tCHsNvt+P1+/uzP/mxNanytyion91/S6/VkZWWRnZ2t+NC3bdtGenq6opXLN+rs7CyxWIxr165x7do13n33XXp6elSh7NyIfDopLS1VirQFAgEmJiaW9RHxer0UFxdjs9lwOBwEg0GlUee1a9fo6OhYcYBeMmMymXA4HKoyq8sm9ZKSEsXqtnThlLv1jo2NJe3iuRLS0tLIzMxU1dzcCrfbTVlZmRKzodPpcDqdVFVVYTKZMBqNeDwe9Ho98/PzxONxAoEAbW1tNDc309nZydTU1IqUlUQioRTJ7O7uXlW5HgQjIyNIkkQ4HCY1NVVReIQQ5OfnEw6HSUlJWfNWPU6nU3nmqquraWxsZG5uTqnMe6vWEW63W6ncXl9fT319PRaLRamCLmfNhsNh1ffqux1yCQy5UvTAwADt7e2qr2F2M3Q6nVLfqqKigsLCQsWKeubMGcW6/DB54AqPyWTC6XSSlZVFZmYmX/3qVyktLaWkpESJOF+6uEYiEXp6ejhx4gRNTU2cPHmSsbExZmZmVH2y1ul0NDY2UldXx969e4nFYkxNTSnFEuVKrnq9nkgkQktLC11dXfT29vLjH/+YwcFBotHout5Ml24sa9kr7GakpKQomXJbt27FZrMp5Qba2tqUmiDhcFh1p/kHicfjIScnJykUnlgsRigUwm63YzablX5DDocDWNwsmpqaGB4eprW1lZ6eHtrb2/H7/UQikRUrO8nKwsICfX19igthacVbtTRi3rp1K9/4xjcoKytTekV5vV4+//nPE41Gb7mx19fXk5OTQ3V1NTabDavVihCCRCJBe3s7fX19vPbaa1y+fJlz586pNsNwpSyNpVNDSMfdIvfPKisr45lnniE7O1sJrm9qasLv969JxfoHovDIXVIrKipwuVxkZGSQnZ1NRkYG5eXl5ObmKkXN4NdNRH0+HyMjI3R0dNDU1ERnZydDQ0NK/QU1Ibe4GB8fx2g0KsXcTCYTWVlZilXAbrcrQaDz8/MMDQ3h8/k4ffo0AwMDSoDXaraNUBOSJGGxWEhLS7vl6W0t0Ov1ip/c4XCg0+lIJBJEo1H6+/t57733CAaDSbnY3A1ykbClFXrVit/v5+2331Zc4g6HQ7HIjY2NMTExQWtrK8FgkJ6eHgYHB+nr62NiYoLZ2dl1PY+wWGbi2rVrylqrxjmVs3WsVqvSL0u23MTj8ZsecoUQFBUVkZaWpmRRzs/PK7E/cpHJ1tZWBgcHk9IaciOyl8TtdpOdnb0iN6ya0Ol0bNq0Seltl0gk8Pl8XL58mUuXLq1ZF/sHovBYrVYyMzP5vd/7PQoLCykqKiI9PX2ZkrMUv99Pf38/L7zwAj09PTQ1Nd22oqYakM1vXV1dLCwsUFZWBiw+jPIJc+kEyq0Wjhw5wvvvv88PfvCDdb/g3gq5Gq6ayr/L9WccDofSIHRubo7x8XHOnTvHj370o7Ue4kPB5/NhNptZWFhQdUVXWGw2fOHCBaWMRVVVFYlEgnA4zJkzZ2hra1OyOpNpc3hQxGIx3n77bYQQfPjDH17r4ayYzMxMnnrqqRV9VgjB7Oys0uuts7OTv/7rv8bn86kyi+lekdudlJaWsmXLFo4fP67qHn43YjQa+epXv0pjYyM5OTl0dHRw/vx5Xn755YceqLyUe1J4TCaTUjOnsrJS6ai9fft2XC6XUj5aRu4ALgeBHj16lO7ubi5fvqycvtTu2pmenmZoaIgf//jHlJWVsWPHDqqrq5e1ZJCrJXd1ddHc3Izf7+fSpUuqL/i1WqjNjXUjS4OUl/KbNFdjY2MMDQ0xOjqK0WjE4XBgt9vxeDxMTk6q6hASj8eJRqN0dHQwMDBAR0eHUr5ieHhY6bvzmzR/S0kkEgQCAYaGhhgaGsLlcqmuRk1HRwfPP/88O3bsoLCwkNra2ttafqPRKNPT00pMlpzU0tzcTHd3N4ODg0ln/bgdRqORuro6ioqKiMfjTExMMDIyoqrncCXIVc+NRiPZ2dmEQiFaWlrWvLbXPSk8RqOR3NxcNm3axP79+5US5hkZGUrlxUQioUyS3CK+t7eX06dP8/rrr3P16tU1M2vdC7FYjLm5OQ4fPkxPTw/xeBybzbasJ4gkSfh8Ps6ePav0EVFbRdaHxdKmf2pUeuR7NJFIsLCwkBRZSquBXGU4EAjgcDhISUkhJSWF9PR0xT2glmdUni+fz7fWQ1EliUSCsbExRemRD5FCCGKxmCriePr6+njzzTfR6XRMTU1RXFyM3W5f9vzJ40wkEkxOThIIBJidnSUajdLc3ExXVxdHjhxhaGhItcVM7xWDwUBhYaHSTHpycpLx8fGkVHiuXr2K3W5n586dSn++tQ6av6dV3uFwsHv3bnbs2MGjjz6qdGPW6XTMzMwwMjJCV1eX0iIhGAzy2muvEQqFlAj0ZFJ2ZOSGp5cvX6avr4+f//znSvsEmWg0ytTUFJFIhNnZ2d9IZWepdr9//35VznM0GqWvr48rV67Q2tqaNKnZq8HMzAwvvvgiW7du5bOf/azSSPQHP/gBvb29qu/BpLHIwsICgUCAw4cPc/nyZQwGg3JPx2Ixent719wSEg6HmZ6eJhgM4vF46OjooK6ujqeeegqr1Yper+fChQsMDw/T1tbGtWvXaG9vJxKJEIvFiEajzM7OEolE1lyW1WB+fp7W1lYmJyfJyckhHA4nZfxZPB7nlVde4ejRo/zoRz9icnKSUCikFBhcK+5phZebSY6OjtLV1bXsPVnhkc2NCwsLhEIhmpublZs1mZED66anp5W0VY3lSJLEwMAADoeDc+fO0dfXR19fn6p6aclNa7u7u7lw4QKTk5MYDAaCwSCDg4NrPbyHSjwe5+rVq2RkZBCJRJSYq5KSEqXUu9orZWssMj8/TzgcVmWROvj1+im7IOX4zczMTCwWCwaDgYsXLyoKT39/P93d3etWwbmR+fl52tvbFVfz4OCgYp1LNuTaQX6/f41H8mvE7RYxIcQt35QtOjemsy4t0CdP0lqVqpck6Y6+lNvJmAyoVUadTqc0f13q4ryXTfNOMt6PfHLpgKVuyYWFhYdqQl7rORRC4HQ62b59O3/4h39IeXk52dnZHDt2jKamJr7//e8rcRT3ylrL+DDQZLx7+fR6/Qf2kaUuraX7yMNADXMo984SQii9FR/kYUMNMq42t5Lxnm34yeZT1Hi4yPEWar9PbpUK+5uEJElKm4Zf/epXSkB+dnY2paWluFwupR6IhsaDRI710+6tX6P9LVaP38ygBQ0NjWXMzs7S3d3NP/3TPxEMBolEInzsYx9Dr9eTmppKLBZb8wwLDQ0Njfvhnl1aycBvsuluKetdxvUuHzwcGYUQSjsYj8dDdnY2s7OzNDU1EYvF7iv+Ti0yriaajOtfPtBkTAZuJaOm8Ggyqh5tkdVkTAY0Gde/fKDJmAzck8KjoaGhoaGhobEe0K31ADQ0NDQ0NDQ0VhtN4dHQ0NDQ0NBY92gKj4aGhoaGhsa6R1N4NDQ0NDQ0NNY9msKjoaGhoaGhse7RFB4NDQ0NDQ2Ndc//D/uVLAc1Sqw5AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x72 with 10 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pltsize=1\n",
    "plt.figure(figsize=(10*pltsize, pltsize))\n",
    "\n",
    "for i in range(10):\n",
    "    plt.subplot(1,10,i+1)\n",
    "    plt.axis('off')\n",
    "    plt.imshow(numpy.reshape(x_train[i,:], (28, 28)), cmap=\"gray\")\n",
    "    plt.title('Class: '+str(y_train[i]))\n",
    "    print('Training sample',i,': class:',y_train[i], ', one-hot encoded:', y_train_onehot[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To handle multi-class classification, it's common to use softmax instead of sigmoid. It's related but forces the outputs to sum to 1, like a probability distribution. The class with the highest value is the prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax(x):\n",
    "    normalization = numpy.sum(numpy.exp(x),axis=1)\n",
    "    z = numpy.exp(x) / normalization[:,None]\n",
    "    return(z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Randomly initialize W and b\n",
    "W = .01 * numpy.random.rand(num_features,10)\n",
    "b = .01 * numpy.random.rand(10,)\n",
    "\n",
    "def classification_model(W, b, input_images):\n",
    "    # f(x) = softmax(xW + b) returns m x 10 matrix, where m is the number of examples and 10 is the number of classes\n",
    "    return softmax(numpy.dot(input_images, W) + b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To demonstrate softmax, we can apply this model with random W & b\n",
    "predicted_labels = classification_model(W, b, x_train[:5,:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.09681239 0.10663882 0.10122395 0.10200088 0.09875957 0.1024364\n",
      " 0.09924979 0.0938935  0.09719685 0.10178786]\n",
      "1.0\n"
     ]
    }
   ],
   "source": [
    "# Here the class \"probabilities\" might all be very similar since the model hasn't been trained\n",
    "print(predicted_labels[0,:])\n",
    "print(sum(predicted_labels[0,:]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, for multi-class classification problem, it is common to minimize a different \"loss\" function instead of mean squared error, like categorical cross-entropy. You can read more [here](https://gombru.github.io/2018/05/23/cross_entropy_loss/). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nonlinear Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above improvements are not enough to classify these images. We move to a nonlinear model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A neural network has multiple layers. A basic layer is $\\sigma(xA + b)$, where $\\sigma$ is a nonlinear \"activation function.\" An example neural network with two layers adds another affine transformation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$f(x) = \\sigma(\\sigma(xW_1 + b_1)W_2 + b_2)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is often drawn as a network like this:\n",
    "\n",
    "\n",
    "<img src=\"images/tiny_network.png\" width=\"300\" hight=\"300\" align=\"left\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Writing it in numpy, with sigmoid as the activation function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def nonlinear_model(W1, W2, b1, b2, input_images):\n",
    "    return sigmoid(numpy.dot(sigmoid(numpy.dot(input_images, W1) + b1), W2) + b2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding another layer:\n",
    "\n",
    "$f(x) = \\sigma(\\sigma(\\sigma(xW_1 + b_1)W_2 + b_2)W_3 + b_3)$\n",
    "\n",
    "\n",
    "<img src=\"images/three_layer_network.png\" width=\"300\" hight=\"300\" align=\"left\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def nonlinear_model(A1, A2, A3, b1, b2, b3, input_images):\n",
    "    return sigmoid(numpy.dot(sigmoid(numpy.dot(sigmoid(numpy.dot(input_images, A1) + b1), A2) + b2), A3) + b3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will learn about other activation functions (nonlinearities) next week."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Optimizing a nonlinear model: \n",
    "- We still want to use stochastic gradient descent (or a variant), but now the gradients are more complicated\n",
    "- The gradients can be calculated with calculus (chain rule!). To save on computation, we move backward through the layers, saving intermediate results for re-use. This is called **back-propagation**.\n",
    "- Applying the current network to the data is often called the \"forward pass,\" and calculating the gradients is called the \"backward pass.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nonlinear neural networks can fit more complicated data than linear models. On the other hand, deep learning training can be tricky. \n",
    "\n",
    "1. Unlike linear regression, the objective function that you're minimizing (some measure of error) is non-convex, so there can be many local optima. As we learned about last week, the learning rate can help you jump into a new area, although too much jumping can be bad. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![multiple local minima](images/nonconvex.png)\n",
    "\n",
    "Image source: [firsttimeprogrammer.blogspot.com](http://firsttimeprogrammer.blogspot.com/2014/09/multivariable-gradient-descent.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are some fancier versions of gradient descent optimization algorithms that are more effective, such as:\n",
    "- Adam \n",
    "- RMSprop\n",
    "- Adadelta\n",
    "- Adagrad\n",
    "\n",
    "For far more information, and some cool animations, see https://ruder.io/optimizing-gradient-descent/ or https://distill.pub/2017/momentum/. It will be easier to try them out next week when we are using TensorFlow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Neural networks can be overly flexible/complicated and \"overfit\" your data. This is like what happens if you fit a high-degree polynomial:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<img src=\"images/bias_vs_variance.png\" width=\"800\" hight=\"500\" align=\"left\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. To improve the generalization of our model on previously unseen data, we employ a technique known as regularization, which constrains our optimization problem in order to discourage complex models. Next week, we'll learn about Dropout for regularization. A simpler form is to add a penalty for large weights (\"L2 regularization\")."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<img src=\"images/test_data_rule.png\" width=\"800\" hight=\"500\" align=\"left\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## In-class exercise: split the training data into training & validation, and track validation loss during the training loop. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tip: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html\n",
    "\n",
    "How do we know when to stop training? For example, you might stop when the validation loss stops improving."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A solution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train2, x_val, y_train2, y_val = train_test_split(x_train, y_train, test_size=0.33, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for simplicity and speed, we could again consider just a batch of training data & a batch of validation data\n",
    "x_train_batch2 = x_train2[:batch_size, :]\n",
    "y_train_batch2 = y_train2[:batch_size,numpy.newaxis]\n",
    "x_val_batch = x_val[:batch_size, :]\n",
    "y_val_batch = y_val[:batch_size,numpy.newaxis]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Randomly initialize W\n",
    "W = .01 * numpy.random.rand(num_features,1)\n",
    "\n",
    "# now iterate num_iters times, with the step size defined by learning_rate\n",
    "losses = numpy.zeros((num_iters,2)) # 1st column for training data, 2nd for validation data\n",
    "accuracies = numpy.zeros((num_iters,2))\n",
    "\n",
    "for i in range(0, num_iters):\n",
    "    W = learn(x_train_batch2, y_train_batch2, W, learning_rate)\n",
    "    losses[i,0] = evaluate(W, x_train_batch, y_train_batch)\n",
    "    losses[i,1] = evaluate(W, x_val_batch, y_val_batch)\n",
    "    accuracies[i,0] = evaluate_accuracy(W, x_train_batch2, y_train_batch2)\n",
    "    accuracies[i,1] = evaluate_accuracy(W, x_val_batch, y_val_batch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## By plotting both training and validation loss/accuracy, we see definite overfitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f4439d9b1f0>]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAw30lEQVR4nO3deXwb5Z348c9XI/l2EsfOaSexA+EIVwImCeVYjgABCqEFytmmbSBdFhZot7uF0mVpX3QXyq+00IUCBVra5ShHKSlXGiBcJUAcCCH3RUIcEjtxEseOT0nP748ZyZIsO7ItSx75+3699NLMM8/MPI+ifDV+5pnnEWMMSiml3M+T7gIopZRKDg3oSimVITSgK6VUhtCArpRSGUIDulJKZQhvuk5cUlJiysvL03V6pZRypaVLl+4yxoyIty1tAb28vJyqqqp0nV4ppVxJRLZ0tU2bXJRSKkNoQFdKqQzhvoD+0e/g/x0C7c3pLolSSg0oaWtD7zV/CzTWQKAdfLnpLo1SKsXa29uprq6mpaUl3UXpVzk5OZSVleHz+RLex30B3eMUOehPbzmUUmlRXV1NYWEh5eXliEi6i9MvjDHU1dVRXV1NRUVFwvu5r8klHNAD6S2HUiotWlpaKC4uzthgDiAiFBcX9/ivEBcHdL1CV2qwyuRgHtKbOmpAV0qpDOG6gN4atH+1TKA9zSVRSg1Ge/fu5YEHHujxfueeey579+5NfoEiuC6gv7upHoDWtrY0l0QpNRh1FdD9/u5bDV555RWGDRvWT6Wyua6XizhNLn6/XqErpVLv5ptvZuPGjUyZMgWfz0dOTg5FRUWsWbOGdevWceGFF7J161ZaWlq48cYbmTdvHtAx3EljYyPnnHMOJ510Eu+//z6lpaW8+OKL5Ob2vRu2+wK6ZRc5qAFdqUHvp39byaov9yX1mJPHDuG/zj+iy+133nknK1asYNmyZbz11lucd955rFixIty98LHHHmP48OE0Nzdz/PHHc9FFF1FcXBx1jPXr1/PUU0/xu9/9jm984xs8//zzXHXVVX0uu2sDut+vTS5KqfSbNm1aVF/x++67jxdeeAGArVu3sn79+k4BvaKigilTpgBw3HHHsXnz5qSUxXUB3eME9MAB2quUUpmvuyvpVMnPzw8vv/XWW7z++ussXryYvLw8Tj311Lh9ybOzs8PLlmXR3JycoUxcd1PUY2UB2uSilEqPwsJCGhoa4m6rr6+nqKiIvLw81qxZwwcffJDSsrnuCj3c5KLdFpVSaVBcXMyJJ57IkUceSW5uLqNGjQpvmzVrFg8++CCHH344hx56KDNmzEhp2VwX0D2WPVCNCWiTi1IqPZ588sm46dnZ2bz66qtxt4XayUtKSlixYkU4/Yc//GHSyuW+Jhev04berjdFlVIqkusCuuUE9KBeoSulVBTXBXQJ3RTVNnSllIrSo4AuIuNEZJGIrBKRlSJyo5M+XEQWish6572of4oLVvjBIr1CV0qpSD29QvcD/2aMmQzMAK4TkcnAzcAbxphJwBvOer+wvPZNUb1CV0qpaD0K6MaY7caYj53lBmA1UArMBh53sj0OXJjEMkbxaEBXSqm4et2GLiLlwFTgQ2CUMWa7s2kHMKqLfeaJSJWIVO3cubNX5w1doRttclFKuUBBQUHKztWrgC4iBcDzwE3GmKiRcYwxBjDx9jPGPGyMqTTGVI4YMaI3p8Zy+qEHg3qFrpRSkXr8YJGI+LCD+RPGmL84yTUiMsYYs11ExgC1ySxkJK9XHyxSSqXPzTffzLhx47juuusAuP322/F6vSxatIg9e/bQ3t7OHXfcwezZs1Neth4FdLEnuXsUWG2MuSdi03xgDnCn8/5i0koYw/JpQFdKOV69GXZ8ltxjjj4Kzrmzy82XXnopN910UzigP/PMMyxYsIAbbriBIUOGsGvXLmbMmMEFF1yQ8rlPe3qFfiLwTeAzEVnmpP0YO5A/IyJzgS3AN5JWwhhW+NF/bXJRSqXe1KlTqa2t5csvv2Tnzp0UFRUxevRovv/97/POO+/g8XjYtm0bNTU1jB49OqVl61FAN8a8B3T1k3NG34tzYFaWE9B1kmilVDdX0v3pkksu4bnnnmPHjh1ceumlPPHEE+zcuZOlS5fi8/koLy+PO2xuf3Pd4Fxe50lRDehKqXS59NJLueaaa9i1axdvv/02zzzzDCNHjsTn87Fo0SK2bNmSlnK5LqD7fE6RtQ1dKZUmRxxxBA0NDZSWljJmzBiuvPJKzj//fI466igqKys57LDD0lIu1wV0y7IIGNGArpRKq88+67gZW1JSwuLFi+Pma2xsTFWR3Dc4l8/y4MeL0X7oSikVxXUB3esRWvEiAR0PXSmlIrkuoFseoQ0fnqAGdKUGK/uB9MzWmzq6LqCL2AFdAq3pLopSKg1ycnKoq6vL6KBujKGuro6cnJwe7ee6m6IA7fjwaEBXalAqKyujurqa3g7w5xY5OTmUlZX1aB9XBvQ2ydI2dKUGKZ/PR0VFRbqLMSC5rskFwC96ha6UUrFcGtCz9KaoUkrFcGdA92gvF6WUiuXKgB70ZGFpQFdKqSiuDOgB0YCulFKxXBnQg1YWXg3oSikVxaUBPRuv0bFclFIqkisDurGy8KEBXSmlIrkyoAetbHxGm1yUUiqSKwM6VrZeoSulVAx3BnRvDjm0QwYPzqOUUj3l2oAOYPypn4RVKaUGKlcGdJOVD0Dr/n1pLolSSg0crgzoVm4hAPsb9qa3IEopNYC4MqD7cocA0NRYn+aSKKXUwOHKgJ7lXKG37NeArpRSIa4M6Nn5QwFo0St0pZQKc2VAzymwA3pbU0OaS6KUUgOHKwN6fqEd0P3N2stFKaVCXBnQ8wqLAPC36BW6UkqFuDKg5xfYvVwCGtCVUirMlQFdvNk0k41p2p3uoiil1IDhyoAOsM8zFKtZA7pSSoW4NqA3+4rIbtOArpRSIa4N6G1ZReT596a7GEopNWC4NqCb/BKGmn3UN+u46EopBS4O6DlDR1LMPjbUaF90pZQCFwf0ISPGkyttbKnelu6iKKXUgODagD60dBIAtV+sTXNJlFJqYOhxQBeRx0SkVkRWRKQNF5GFIrLeeS9KbjE78wyvAGDXVg3oSikFvbtC/wMwKybtZuANY8wk4A1nvX8NmwBA1r4v2FGvU9EppVSPA7ox5h0gtgP4bOBxZ/lx4MK+FSsB2QX480dziGcr8z/VdnSllEpWG/ooY8x2Z3kHMCpeJhGZJyJVIlK1c+fOPp/UW3Ycx2dt4dmqaoJB0+fjKaWUmyX9pqgxxgBxo6sx5mFjTKUxpnLEiBF9P1npVEoD26ipreH11TV9P55SSrlYsgJ6jYiMAXDea5N03O6NPwGA2UPWc+8b6wnoVbpSahBLVkCfD8xxlucALybpuN0bNwNyi5g3ag0rv9zHUx99kZLTKqXUQNSbbotPAYuBQ0WkWkTmAncCZ4rIemCms97/LC8ceh5lNW9yenkOdy9YS+0+7fGilBqcetPL5XJjzBhjjM8YU2aMedQYU2eMOcMYM8kYM9MYk7phEKddjbQ1ctdBy2n1B/j+M8v0BqlSalBy7ZOiYWOnQvnJjPjkf/n5rPH8Y0Md9y/akO5SKaVUyrk/oAOcdQc01fH1PY9x4ZSx/HLhOl5cpn3TlVKDS2YE9LFTYMa1SNUj3H3kF0yvGM4Pn/2UN7Qro1JqEMmMgA4w83YYMwXfi9fy6BnC5DFD+N6fluqVulJq0MicgO7Nhiv+DAUjKXjuUp4623DchCJufHoZ//PKavyBYLpLqJRS/SpzAjpA4Wj41ouQX0Le01/n/45bx1XTx/HQO5v4xkOLWbujId0lVEqpfpNZAR2gaALMXQjjpuN76QbuaL2Lh2aP5vNd+znvvnf5+curqGtsTXcplVIq6cQeeiX1KisrTVVVVf+dIBiAxffDm3eACM3HXsPde0/j95+1kOuz+OaMCVw5fQLji/P6rwxKKZVkIrLUGFMZd1vGBvSQPVtg0c9h+TPgsWiYeB5PNE/n15tKaTE+Tp5UwvnHjGXm4aMYnp/V/+VRSqk+GNwBPaRuIyx5BD55AlrrCWYVsKnweF6qr2Dh/oNYx3iOLR/BCQcVM72imKnjh5Hjs1JXPqUGq2AQTACCfvsvaxOw38PLkelBez2cxw8mGLEcu29MugnGHM/fg/PH7tvNeTqlx5TxxJtg8gW9+rg0oEfyt8Hnb8Pq+bDpLdhrD+jlFx9feEpZ3jaWDcFSdsgIPMPGMXR0BWPGTeSw0mLKS/IZPSQHj0dSX2418BnT8R+320DQXXowToBINIj5e3D+ngSxRIJlN2U8UFkGMvGAWOCxnHcveDwRy1bH9p7kmf49OOTs3hVJA3o36rfBF4thx3KoXU2wZjWefVujsgSNsI889poC6imgxTeMQM4wJLcIX04B2Xn55OQVklcwhALnZeXkg5UFHh9YPvsfNvQeXg5tswABEefdE7GcyDt2MMEJKLHLJuisxy53ta2L/cL/QSOCTug/c6e0QMw+EduMiZMWjA4gUccInSM2LSZ/1L6x5UkkiPX0ii8miJlASr6yvSYe+7vXKQBZEemJBCur41hxg1jM8aLO6XX2tRIoy4HOc6AyJuP8A+/irbuA7k11YQacoaVw1MX2C6fbT1sT7PsS6rdCfTUtOzfTXFcDDbsobNrNsJY9ZDdXk7e/kWzTSrYM8KuMTBK6Ygr9p4wMQlFp1oH/44b+83uzehnEugmKCQURb0yZEwmWfTj/AAxOKrk0oMeTlQclB9svIM95xdPqD1Bdv59de/ayZ+9e6uvr2dewj7amBppbmmlpaaWltYXWtlbaWttob2uDYDteCeAjgBc/XoII9lW1fd1t8GAQDFmWkGUJPkvwesDrsd99HsFrCV4Brwcsy7JfHg+W5XHeLSzLg9djv3s8drp4LCyPhPN7nJfXssJ5LK9zLOc4XsuDJxxYYv8MdYJQp4AaCk6J5vcQFaCjjmFpUFLqADSg91G216KseAhlxUOA8QfMb4yhpT1IY6uf5rYA+9v8NLX52d8aoKnNT1NbgP1tAZpa/eH3Xe0B2vxBWtoDtPqD9it2uSlIqz9Aa7ud1tYPT8aKgM/ykGV58FoSXvZZgtfy2D82lmB5wOsJYnkMXo9gecR5t/NYlsRPD61bXaSHt8dL90Sti2D/aIng8QgescvlkdBy5Dt4QnlF8HgIbw/lsUQQD1gScfzwsv7IqIFBA3qKiQi5WRa5WVa/nicYNLQFgrS2B2lxAn17MIg/YGgP2AG/3R/E7+Rr9wdpDxj8wSBtznJ7IOi87GV/IEhbnPTQyx8wBIIGfzD0bqc3tzvrgY706Hyh7Z3T3aLjx8P+0ej4Ien84+Bxfhg8Th5LOn6A7Hz29yTuOx3n6EiLXvc4x+vuPfJYUfsRWu/4oetI6zhG5HqofJHH9jg/clHHcY4flSdmn9hjh87tZI34POzyh/J3fAZ02qejDh15Y/chcv+oc0fv44k6b+S2jn2Q2M/S3sdOt9OyvR58VvKf69SAnqE8HiHHY5HjsxiKL93F6RVjDEFD9A9AIPoHI/oHoSMtaCBojLNsCAYhYELLEemGiGVDIGj/GAaNsfOH80YeL/bYdt5A0C5zwFk3zrHtZSc9SMS5YsoWNBgIl8s4+YyJTAva92Cj8oDBPk5k/tCxOvY34Xvhwahjd5wrtBx5ztCxTOS6e35rB6Q7LjySq2ZMSPpxNaCrAUtEsAQsT//+NaN6JzLAx/uxCDqdrYKxPxZE/FgEnf2I/LFwfgxjjmmfsyOvifihsdMJ/yiaqO3RP0qhY0Qeuyf7dHXuyH3oVF97Gafux44v6pd/Ew3oSqleCf/govcQBorMG5xLKaUGKQ3oSimVIdL2pKiI7AS29HL3EmBXEovjBlrnwUHrPDj0pc4TjDEj4m1IW0DvCxGp6urR10yldR4ctM6DQ3/VWZtclFIqQ2hAV0qpDOHWgP5wuguQBlrnwUHrPDj0S51d2YaulFKqM7deoSullIqhAV0ppTKE6wK6iMwSkbUiskFEbk53efpCRB4TkVoRWRGRNlxEForIeue9yEkXEbnPqfdyETk2Yp85Tv71IjInHXVJhIiME5FFIrJKRFaKyI1OeibXOUdEPhKRT506/9RJrxCRD526/VlEspz0bGd9g7O9POJYtzjpa0Wkd/OXpZCIWCLyiYi85KxndJ1FZLOIfCYiy0SkyklL7XfbhAfDGfgvwAI2AhOBLOBTYHK6y9WH+pwCHAusiEj7BXCzs3wzcJezfC7wKvaIoDOAD5304cAm573IWS5Kd926qO8Y4FhnuRBYB0zO8DoLUOAs+4APnbo8A1zmpD8IXOss/wvwoLN8GfBnZ3my833PBiqc/wdWuut3gLr/AHgSeMlZz+g6A5uBkpi0lH630/4h9PADOwFYELF+C3BLusvVxzqVxwT0tcAYZ3kMsNZZfgi4PDYfcDnwUER6VL6B/AJeBM4cLHXGnvjqY2A69lOCXic9/L0GFgAnOMteJ5/Eftcj8w3EF1AGvAGcDrzk1CHT6xwvoKf0u+22JpdSIHIG52onLZOMMsZsd5Z3AKOc5a7q7srPxPmzeir2FWtG19lpelgG1AILsa809xpjQpPRRpY/XDdnez1QjMvqDPwa+A8gNHVWMZlfZwP8XUSWisg8Jy2l320dPncAM8YYEcm4fqUiUgA8D9xkjNknEVO4ZWKdjTEBYIqIDANeAA5Lb4n6l4h8Fag1xiwVkVPTXJxUOskYs01ERgILRWRN5MZUfLfddoW+DRgXsV7mpGWSGhEZA+C81zrpXdXdVZ+JiPiwg/kTxpi/OMkZXecQY8xeYBF2c8MwEQldUEWWP1w3Z/tQoA531flE4AIR2Qw8jd3sci+ZXWeMMduc91rsH+5ppPi77baAvgSY5Nwtz8K+gTI/zWVKtvlA6M72HOx25lD6t5y74zOAeudPuQXAWSJS5NxBP8tJG3DEvhR/FFhtjLknYlMm13mEc2WOiORi3zNYjR3YL3ayxdY59FlcDLxp7MbU+cBlTo+QCmAS8FFKKtFDxphbjDFlxphy7P+jbxpjriSD6ywi+SJSGFrG/k6uINXf7XTfSOjFjYdzsXtHbARuTXd5+liXp4DtQDt2W9lc7LbDN4D1wOvAcCevAPc79f4MqIw4zneBDc7rO+muVzf1PQm7nXE5sMx5nZvhdT4a+MSp8wrgNid9InZw2gA8C2Q76TnO+gZn+8SIY93qfBZrgXPSXbcE638qHb1cMrbOTt0+dV4rQ7Ep1d9tffRfKaUyhNuaXJRSSnVBA7pSSmUIDehKKZUh0tYPvaSkxJSXl6fr9Eop5UpLly7dZbqYUzRtAb28vJyqqqp0nV4ppVxJRLZ0tU2bXJRSKkPoo/9KqUHr8137+XxXY8rPe8ioQsqK8pJ+XA3oSqlB69u//4gtdU0pP+8dFx7JVTMmJP24GtCVUoPW7sY2zj9mLFefVJHS85YW5fbLcTWgK6Uyykef7+bDTXUJ5d3f5mf88FyOGTesfwuVIhrQlVIZ5Y6XV7G8uj6hvCJ2e3am0ICulMoojS1+zjtqDPdeNiWh/F4rczr7aUBXSqXEpp2N/HLhOvyB4IEz98GX9c1UlhdlVKBOVEIBXURmYQ9QbwGPGGPujNn+z8B1QABoBOYZY1YluaxKKRd7c00tLy/fziGjCvBEzFKVbOXF+Zx66Mh+O/5AdsCALiIW9ri9Z2KP2b1ERObHBOwnjTEPOvkvAO4BZvVDeZVSLtXUFgDglRtOHpRXz6mQyBX6NGCDMWYTgIg8DcwGwgHdGLMvIn8+9iQGSimXe25pNfe+sY5kTJtQ39xOluXRYN6PEgno8Wahnh6bSUSuA34AZGHPIdiJMxP2PIDx48f3tKxKqRR7f8Mudje2cfaRo5NyvMljhiTlOCq+pN0UNcbcD9wvIlcAP6FjHr3IPA8DDwNUVlbqVbzKeA0t7QR7+U3Pz7ISupr1B4Lsd5ozkm1fSzulRbnc840p/XJ8lVyJBPSezkL9NPDbvhRKqUzwTNVW/uO55b3ef3rFcP78vRMOmG/2/f9g5Zf7Dpivt46bUNRvx1bJlUhAXwJMcmbd3oY9i/cVkRlEZJIxZr2zeh72hKhKDWqbdu7H6xFuOffwHu/78vIv+XzX/oTP85WDijnj8FE9Pk8ippUP75fjquQ7YEA3xvhF5HpgAXa3xceMMStF5GdAlTFmPnC9iMzEnr1+D3GaW5QaLLbXN4dH8cvP9jK3F+OEVO9pYs2OBt7fuKv7jAaa2wMcXz68V+dRmSWhNnRjzCvAKzFpt0Us35jkcinlWnMe+4h1NfaQrBNH5PfqGCUF2TS1Bbjidx8mmD+rV+dRmUWfFFUqyeoa2zhz8ijmnlTBhOLejXl99ckVHF8+nGAC/QW9HsmYwaVU32hAV6qPXl6+ne31zeH1hhY/5cV5zJhY3OtjZnstplVo27XqGQ3oSvVBfVM71z35caf0iSMK0lAaNdhpQFeqD/a1tAPws9lH8LWppQB4RMjP1v9aKvX0W6cGvddX1fDS8i95e91OjpswnIJsK+F9G1v9AAzPz6Iwx9dfRVQqIRrQ1aD3+/c/5x8b7BluXl9d0+MbmYeOKuSIsUP7o2hK9YgGdDXo7W8NUJTnY09TO16P8Pa/n5buIinVKxrQ1aD136+sZtGaWrbsbmL88Dz2NLUzemhOuoulVK9pQFeD1svLtwMw8/CRfH1qGW+v28k5RyVnVEGl0kEDegL8gSAiguWJnmUlGDQEjdHxndOkPRDs0zjdTW1+zjt6DHdceBQAMyf3z1goSqWKBvQDWF/TwJm/eofCHC8f/XgmuVkdPSDOve9d1uxo4JnvnaAPgaTYorW1zP3Dkl4PTRtSkK09U1Tm0IB+AJvrmgD76b+6/a2UZXX0gFizowGwJ7/VgJ5aG2sbCRq44YxJZHt79xeSCMyeUprkkimVPhrQu+EPBFlf2xBeX7Z1LyUF2eT4LDbubAyn1ze3p6N4g0pdYytf7G4Kr29yhpb919MPxqdNXkoBGtC79cfFW/jFa2vD69c/+QlzT6rg6pMrOOOXb4fTI4O76h9zfv8RK7ZFT+JQmOPVYK5UBA3o3ahtaMXrEZ6/9is0twe46ell7GxoZVdDGwD/duYh/HLhOrK9iT9ZqHqndl8rpxwygu+cWB5OG1eUm74CKTUAaUDvRtXm3eT6rPDQpEX5WayraWD+p/YMfFPHFzF2aA4rv6xnxbZ6jizNzKcF39+wK9zEkS77WtqZWJLPaYeOTGs5lBrINKB3Y9X2fbS0d0y+W16cx6srdrBmRwOWRygtymVCcT6LN9Xxo+eX8/INJ6extP3DGMN3/rCEVn8w3UXp9djiSg0WGtC70R4IMucr5eH131w+ld1NdnNLjs9iSI6PP86dxo1Pf8Jn2+rTVMr+1RYI0uoPcu2pB0U1d6SaR4SSguy0nV8pN3B9QG/1B3jk3c+55uSJZHXTfW3plt3c+sIKfnDmIZx1RMfTgMGg4Vevr2NXY1s4LRAMkuX10B4wDM3t6KfstTyMLIx+NNxneRiWl8XOhlZu+ctnAFxSWcax49M3U3ogaPjl39eyaed+ivI7T02W7fVw/ekHhwOkPxDknoXr2NPUubdOe8C+Mi8pyO5Ud6XUwJJQQBeRWcC92JNEP2KMuTNm+w+AqwE/sBP4rjFmS5LLGtfv/7GZuxesJdvr4eqTJ3aZb85jS2hs9TPvT0vZfOd54fSte5r4zZsbKMzxkuOzaG4LhIdEHTM0h6kJBObpFcN5fVUNr6+uoa6xldb2QFoD+sadjTzw1sbw+ojCjivbYNBQt7+No0qHctFxZU7+/Tzw1kaG5HjJ9nW+wVs6LJdjyjLz/oBSmeSAAV1ELOB+4EygGlgiIvONMasisn0CVBpjmkTkWuAXwKX9UeBYzW12G3dDi7/bfKEgHWt/q73/3Rcfzawjx/CnxZv5zxdXArD4ljMSKsPsKaXhB1TO+tXbNLUFDrBH/9ofU9clt84ML9c2tDDt52/QFHFvYH+bnf/ey6fqTUelXCyRK/RpwAZjzCYAEXkamA2EA7oxZlFE/g+Aq5JZyO54xB5fJXYy3ZVf1nPxbxfT3B5gzgkTorZd8L/vAXB02VCOLhsGQG6W/VHkZfWtFSovy8trK3dwz8J1vLW2ljknlIevhGPdPn8lH3+xh1lHjmb73hY+rd4bN9/0iuHcet5kALbXN3PT08tobu/6R6OrH69Q+QAeWLSBZ6u2RuXPi3N1rpRyj0SiVymwNWK9GpjeTf65wKvxNojIPGAewPjx4xMsYve8lh3Q/TGDenywaXc46D2+uKP1p7w4j+L8LDbt2s+zVdWMH273nDhoRD4AMw4qZtYRo3s9i/qoIXbzxoNvb6TNH2TByh1dBvRnq7ayvy1Amz/I57v2M3poDhNL8qPyrKtp5C8fbwsH9OXV9Xz4+W6OLy+ioItpzorzsxhXlIcInBAzUXF+lsW3v1LOlrr9UfmPGDuUIzK026VSg0VSb4qKyFVAJfBP8bYbYx4GHgaorKzs47BKNp8T0NtjutU1t3W+Sj3lkBH88bvTALj39fX86vV1NLaE2svth1RKh+Xy4DeP63V5LjimlAUra2hzytNV84sxJtzs0djqp9Uf5OtTy7hx5qSofP/9ymr+uHhzRL3sfe666OheTUQsItx+wRE93k8pNfAlEtC3AeMi1suctCgiMhO4FfgnY0xrcop3YKHBsx5573Meee9zLI8Q6GIIvlxfRy+YfGfeyPve3ECuz+o0NG5v5cfMR/nehl2c9au3WXDTKfzkryt4eon9x44xJjz0a/We5rj7AuRneWlpD3LQj18BOpqWuro6V0oNXolEhSXAJBGpwA7klwFXRGYQkanAQ8AsY0xt0kvZjdjA9rWppTy3tBqAS44r41lneUJxHtef1nH1e8GUsTS2+vEHDIeMLkxaeaZXFPPvZx9KS3uA8uJ8nv+4mvc31tHqD/LZtnrGFeXy1aPHAmB5hOkVw1m8qQ7LI1wwZWyn411SWUbAGIIRP1KjhmQzcoh2IVRKRTtgQDfG+EXkemABdrfFx4wxK0XkZ0CVMWY+cDdQADwr9k3KL4wxF/RXoUNPb+bEuYk396SKcEC/+5JjWLypjuo9zdx32VSOiuh6N7Iwh5tmHpL0suVmWVx32sHh9YaWdt7fWMfy6nr2NrVzZOkQfnj2oVH7fOXgki6PN3ZYLj84M8nlbKyF/Tt7vp/HC8HuexOF8xVPAk/EcwGtDbD3C8guhGHJuX+ilIqW0N/txphXgFdi0m6LWJ7Zaad+dNh/vkZJQRZVPzkTiWkpGZbn45BRBayrsUdAnDGxmOeWVlOU1/kBm1QIPdjzjYcWA3DSpK6Dd0oE2uG+qdDWzyNEfvXXUPmdjvWnLofN79rL130EIw6Nu5tSqvdc2xAberIzELCbIu666ChGD81lzNBcnrv2K+x2tv/X+ZO55LgyxqdpHJBZR47m0TmV4ZukaZ8Io7XBDuZTr4JJZyW+3zPfst/PuqP7K2xj4Nk50LA9Or1hO+QWQfMee1kDulJJ59qAHrJs614ALj2+I8gMyfExJMd+ZL8wx8f0mK57qZTttTjj8DTNVdnaAGtfs5tJRKBwNNQ4jw+UTYPJs3t+zEPOgZKDu8/zQi5s+xiWPdWR1lQHQ8rsgL7u77Bve9f794UJwJ7NMPyg/jl+PGOOgVGTU3c+pbrg+oBetWVPuoswcH3yBLz2o/jbhsbvG9+lw8+H1X+D/ASajIaNgw0L7VekyRdC7Sr44P6enXugG30U/PN76S6FUu4P6Kobzc6P3b9+DI+eBU274MiL7WaTIWN6dqyL/wCt+yB32IHzXrOo801XERg6Hk7/T/s4/eWhU+zjX/40jDis/84T8vefwPbl/X8epRLg6oAe+cBNxvr8HVgb98HbA/tiMXhzofggyMq3A3rBqJ4HcwDLC3kJtv9nF9ivePKL7Vd/yS2yA3rxwTC8ov/OE1Iw0v7xeu2WrvN4s+ErNyT++SnVS64O6Lc5g2hltHfuhs3vQVbPnwoFoPxE5/0kWLkTSo9NXtkGopm3w5t3QGEvfrR6o+x4WPE8fPJ/8beboH0TesThcExKxqtTg5irA3rIvZdNSXcR+k9rIxx0Olz1fN+Oc+ED9ivTHfl1+5UqU66wX13Ztx3uOaz/u4kqBbh2yvQfeZ/iGNkAkJmTNK95GX5/LuxcAz6des21spx/u/fvs/89P3gwveVRGc2VAV0Icq33b7yYbT/blONzZTW6t/IFu+tf6XFw5EXpLo3qrewhMPWbMHQc1K6GZU+ku0Qqg7myycUiemTFjLxCb2uC4RPh2y+luySqL0Rg9v/ay89+B3ZojxjVf1wf0IupJzfL5QG9djU8fgG0N3ektTfZV+cqc2QXQN0G+O8ePgPQEyIw63/sJ4EjVS+FJy8Bf1v8/ULlm7vQfo5AuZIrA7qXjjHGzxpeyxFjh6SxNElQuwr218IxV9jd7kImpXSIHNXfps2zm2BMUqYCiG/p76G6qnNAr/nMflr3uG+DL7/zfg3bYeVf7B8cDeiu5bqAHgyaqCv0r1VOwGe5qA3dGKivhmB7R9reL+z3U2+Gognx91PuN/oo+9Wf1vzN7he/e1N0+l5n0rGZt0dfNIR8ucwO6Ls3dXwHh5TafegT0drQ+WEyj9e+dxA7gp7qN64L6AFjsCKu0APisiqseB6enxtng0COy//SUOmXWwRrXrJfsTzerp9nCAX5l3/QkXbouXD5U/Hzx3r4NKhb3zn9wt92361TJZXLoiEEggZvxBV6SZ6Lrs7BvjoHuOA3YEUM6Vs4Ov6Vk1I98bWHYfuy+NuGjQfLF39b0QS48jm7WQbggwc6vquJqN8Kk87ueAbABOGv1/bsGKrPXBnQp3g2hNfzBvr90OY99uP7oXbT0H+2KVdFTwChVDKMPMx+9cakMzuW1/8dNv8DVv41gR0N+Fvsp5CPuawj+W83wvZPEzzGADW0DMoqe7//rvVQE+eJ9jFH273Yksx9Ad0Yfpd1T3g93xt/EuYB471fwT/ujU4rHKPBXA1shWOgcYc9tn3C+4yOXh8ytuvmH7fweOGWbeDr5ZSPz34balZ0Tj/vHg3oAAF/dB/0wi7+ghwwmnZD/gj41vyOtNgvvlIDzczb7Z4yifbI8XihZFJ02jWLoGFH0ouWMiueg3d/aQ/b0NuA3lRnDz196o+j0/spBrgvoMfMaWmZ9i5y9pO1r0H1RyAemHKlPaJfzUr4/F2Y/r3oO/r11fDJn+xfYp0AQbmJ5YORh/ftGHnD3T3C5JfOFfTbd9lz4fZG8x67t1CK/v8nFNBFZBZwL/Yk0Y8YY+6M2X4K8GvgaOAyY8xzSS5nWLC9NTohkMCkxcn02o/sGXHAnp/zzJ/C81fbfckPPSe62+GnTg+BsVNTW0alVN+NONzuFVT1WO+PIR4YfXTyynQABwzoImIB9wNnAtXAEhGZb4xZFZHtC+DbwA/7o5CRAv6YK3KT4jb01gaonGt3Pww92Vm72n6PHVGvtdHuyXJxH74QSqn0KDsOfrwt3aXokUSu0KcBG4wxmwBE5GlgNhAO6MaYzc62YLwDJFOnK/S//gscfVnPbjJ+8YHdNma6KO7IyfYkCfXVdtvgGbfZbeH/uBea99oj6Hlz7Dv7AB7LnrfzbzdF9yXfuU5HSlRKpUwiAb0U2BqxXg1M783JRGQeMA9g/PhuZo7vhgnEtpkb+278kLGJH2TVfNjwevymkH1f2tvAnjKt/gsYf4L9NOemRXbXrINn2scIBfSSQ6F2pf3XQnPEHKcFI+wbIkoplQIpvSlqjHkYeBigsrKyVwNaBOK1mbc19ewg7fshrwSuebPztnfvgTd+ai+fdov9cER7k/0qHAtXO8Hel9sR0LMLoeIUmPO3npVDKaWSKJGAvg2IHK2nzElLi2CnK3TgtyfAv63t+o76Hy+M7tzfuq/rbkORj0Zn5dsDGb3/GwgGoueo9ObYc33ePQmad9uzCimlVBolEtCXAJNEpAI7kF8GpG1whmC8K/RAmz2oULyAHgzYTSVjp8KYKR3p5SfFP8Fh59kjznm89lX32Xd0zOo+8dSOfKf8e0fTDMARX+tpVZRSKqkOGNCNMX4RuR5YgN1t8TFjzEoR+RlQZYyZLyLHAy8ARcD5IvJTY8wR/VHgUBv6Z8f/D0ctiZhpvW1//B32fWm/H/F1OPGGA59gaCmc+4uO9crvxs93+Fftl1JKDRAJtaEbY14BXolJuy1ieQl2U0y/C4a6LVrOk1sFo+2boqtehIn/FJ15/UJ44mJ7OWdoKoqnlFJp47oBRUJNLsaXC9/8K3zvHXuDJ85vU+gBoHN+kdqZ4JVSKg1c9+i/cSaGEMsHB51mJxaMgs3v2s0uWc5sLMGgPWIc2GNSZMWZpUUppTKI+67Q/fYVulgRv0WNNfaj93//SUfa5nc7Aro3N4UlVEqp9HBdQDfO4FwSr4ll29KO5aZd9vtVz+tQtUqpQcF1kc7TUm+/e+ME9BpnNAJj4MOH7OXiSZ3zKaVUBnJdQJfWffZC5PRtIaGBunZvgq0f2sv5I1JTMKWUSjP3BfRgGwDB/DhPeoa6JoaC/qVP2ANpKaXUIOC+gO4MWeuJ12uleY89pO2nT9vr2V3McK6UUhnIdQG9zcpnfbA0+sr7nIgnO2tWwbaP7eWROkuQUmrwcF1A/2LCRZzZdjeeyDn+pn8P5jgT0bbvt0dGPPRcKBiZnkIqpVQauO7BooAzaa0V+1MUaoJ5/Hz7qdERh6W2YEoplWauu0IPBO1ZhqzYvuWjIsYCG39C14NqKaVUhnJhQLffLZHoDd7sjuWTboLyE1NWJqWUGghcF9CDQbvJpduHP7OHdLNRKaUyk2vb0L3xIvq3X4a6jVB6XIpLpZRS6ee6gO7v7gq9/KSuZyJSSqkM59oml05t6EopNci5LqAHgt00uSil1CDmuqgYSOSmqFJKDUIJhUURmSUia0Vkg4jcHGd7toj82dn+oYiUJ72kjo4Hi7TJRSmlIh0woIuIBdwPnANMBi4XkdhBUuYCe4wxBwO/Au5KdkFDvntiBctvP4tcn9Vfp1BKKVdK5Ap9GrDBGLPJGNMGPA3MjskzG3jcWX4OOEOkf+5aZnk9DMnx0U+HV0op10okoJcCWyPWq520uHmMMX6gHiiOPZCIzBORKhGp2rlzZ+9KrJRSKq6U3lo0xjxsjKk0xlSOGKEzCSmlVDIl8mDRNmBcxHqZkxYvT7WIeIGhQF13B126dOkuEdnSg7JGKgF29XJft9I6Dw5a58GhL3We0NWGRAL6EmCSiFRgB+7LgCti8swH5gCLgYuBN41xuqN0wRjT60t0EakyxlT2dn830joPDlrnwaG/6nzAgG6M8YvI9cACwAIeM8asFJGfAVXGmPnAo8CfRGQDsBs76CullEqhhMZyMca8ArwSk3ZbxHILcElyi6aUUqon3Pq85cPpLkAaaJ0HB63z4NAvdZYDNHUrpZRyCbdeoSullIqhAV0ppTKE6wL6gQYKcxMReUxEakVkRUTacBFZKCLrnfciJ11E5D6n3stF5NiIfeY4+deLyJx01CURIjJORBaJyCoRWSkiNzrpmVznHBH5SEQ+der8Uye9whnIboMzsF2Wk97lQHcicouTvlZEzk5TlRImIpaIfCIiLznrGV1nEdksIp+JyDIRqXLSUvvdNsa45oXdbXIjMBHIAj4FJqe7XH2ozynAscCKiLRfADc7yzcDdznL5wKvAgLMAD500ocDm5z3Ime5KN1166K+Y4BjneVCYB32gG+ZXGcBCpxlH/ChU5dngMuc9AeBa53lfwEedJYvA/7sLE92vu/ZQIXz/8BKd/0OUPcfAE8CLznrGV1nYDNQEpOW0u922j+EHn5gJwALItZvAW5Jd7n6WKfymIC+FhjjLI8B1jrLDwGXx+YDLgceikiPyjeQX8CLwJmDpc5AHvAxMB37KUGvkx7+XmM/73GCs+x18knsdz0y30B8YT9R/gZwOvCSU4dMr3O8gJ7S77bbmlwSGSjM7UYZY7Y7yzuAUc5yV3V35Wfi/Fk9FfuKNaPr7DQ9LANqgYXYV5p7jT2QHUSXv6uB7lxVZ+DXwH8AQWe9mMyvswH+LiJLRWSek5bS77brJokeTIwxRkQyrl+piBQAzwM3GWP2ScRQyJlYZ2NMAJgiIsOAF4DD0lui/iUiXwVqjTFLReTUNBcnlU4yxmwTkZHAQhFZE7kxFd9tt12hJzJQmNvViMgYAOe91knvqu6u+kxExIcdzJ8wxvzFSc7oOocYY/YCi7CbG4aJPZAdRJc/XDeJHujOTXU+EbhARDZjz59wOnAvmV1njDHbnPda7B/uaaT4u+22gB4eKMy5Q34Z9sBgmSQ00BnO+4sR6d9y7o7PAOqdP+UWAGeJSJFzB/0sJ23AEftS/FFgtTHmnohNmVznEc6VOSKSi33PYDV2YL/YyRZb59BnETnQ3XzgMqdHSAUwCfgoJZXoIWPMLcaYMmNMOfb/0TeNMVeSwXUWkXwRKQwtY38nV5Dq73a6byT04sbDudi9IzYCt6a7PH2sy1PAdqAdu61sLnbb4RvAeuB1YLiTV7CnAtwIfAZURhznu8AG5/WddNerm/qehN3OuBxY5rzOzfA6Hw184tR5BXCbkz4ROzhtAJ4Fsp30HGd9g7N9YsSxbnU+i7XAOemuW4L1P5WOXi4ZW2enbp86r5Wh2JTq77Y++q+UUhnCbU0uSimluqABXSmlMoQGdKWUyhAa0JVSKkNoQFdKqQyhAV0ppTKEBnSllMoQ/x9vuTO6QqUhWgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure()\n",
    "plt.subplot(2,1,1)\n",
    "plt.plot(losses[:,0])\n",
    "plt.plot(losses[:,1])\n",
    "plt.legend([\"train\", \"val\"])\n",
    "\n",
    "plt.subplot(2,1,2)\n",
    "plt.plot(accuracies[:,0])\n",
    "plt.plot(accuracies[:,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the basics, we can experiment with more complicated networks. Rather than implement these all by hand, we will move to using existing Python packages next week. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are variants of \"universal approximation theorems\" roughly stating that there exists a nonlinear neural network with one hidden layer (possibly very wide) can fit an \"arbitrary\" nice/smooth function arbitrarily well. However, we can make the optimizaiton easier with fancier layers than \"fully connected,\" like convolutional layers, which we will learn about next week. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda/2022-07-01",
   "language": "python",
   "name": "conda-2022-07-01"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}