sec05-sgd.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stochastic Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Stochastic Gradient Descent (SGD)</b>, a simple modification applied to the <b>gradient descent</b> algorithm that we saw from previous week's section. SGD also computes the gradient but updates the weight matrix W on small batches of training data, instead of the entire training set itself.\n",
    "\n",
    "While this leads to “noiser” weight updates, it also allows us to take more steps along the gradient (1 step for each batch vs. 1 step per epoch), ultimately leading to faster convergence and no negative affects to loss and classification accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a quick <u>comparison table</u> between Stochastic Gradient Descent and gradient descent:\n",
    "\n",
    "| Algorithm | Time per iter | Total time to convergence<br>for large data in theory | Total time to convergence<br>for large data in practice | Sensitivity to params |\n",
    "|------------------------|---------------------|-------------------------------------------------------|---------------------------------------------------------|-----------------------|\n",
    "| Gradient<br>Descent | Slow for large data | Slower | Usually slower | Moderate |\n",
    "| Stochastic<br>Gradient | Always fast | Faster | Usually faster | Very high |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's consider an example where we illustrate the use of stochastic gradient descent for linear regression. \n",
    "\n",
    "Our goal is to find the equation of the straight line $h_\\theta(x) = \\theta_0 + \\theta_1 x$ that best fits our data points. The function that we are trying to minimize in this case is:\n",
    "\n",
    "$J(\\theta_0,\\theta_1) = {1 \\over 2m} \\sum\\limits_{i=1}^m (h_\\theta(x_i)-y_i)^2$\n",
    "\n",
    "In this case, our gradient will be defined in two dimensions:\n",
    "\n",
    "$\\frac{\\partial}{\\partial \\theta_0} J(\\theta_0,\\theta_1) = \\frac{1}{m}  \\sum\\limits_{i=1}^m (h_\\theta(x_i)-y_i)$\n",
    "\n",
    "$\\frac{\\partial}{\\partial \\theta_1} J(\\theta_0,\\theta_1) = \\frac{1}{m}  \\sum\\limits_{i=1}^m ((h_\\theta(x_i)-y_i) \\cdot x_i)$\n",
    "\n",
    "Then we followed this algorithm (where $\\alpha$ was a non-adapting stepsize):\n",
    "\n",
    "&nbsp;&nbsp;&nbsp; 1: &nbsp; Choose initial guess $x_0$ <br>\n",
    "&nbsp;&nbsp;&nbsp;    2: &nbsp; <b>for</b> k = 0, 1, 2, ... <b>do</b> <br>\n",
    "&nbsp;&nbsp;&nbsp;    3:   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $s_k$ = -$\\nabla f(x_k)$ <br>\n",
    "&nbsp;&nbsp;&nbsp;    4:   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $x_{k+1} = x_k + \\alpha s_k$ <br>\n",
    "&nbsp;&nbsp;&nbsp;    5: &nbsp;  <b>end for</b>\n",
    "\n",
    "Here is the stochastic gradient descent algorithm for simple linear regression, where m is the size of the data set:\n",
    "\n",
    "&nbsp;&nbsp;&nbsp; 1: &nbsp; Randomly shuffle the data set <br>\n",
    "&nbsp;&nbsp;&nbsp;    2: &nbsp; <b>for</b> k = 0, 1, 2, ... <b>do</b> <br>\n",
    "&nbsp;&nbsp;&nbsp;    3: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <b>for</b> i = 1 to m <b>do</b> <br>\n",
    "&nbsp;&nbsp;&nbsp;    4:   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $\\begin{bmatrix}\n",
    " \\theta_0 \\\\ \n",
    " \\theta_1 \\\\ \n",
    " \\end{bmatrix}=\\begin{bmatrix}\n",
    " \\theta_0 \\\\ \n",
    " \\theta_1 \\\\ \n",
    " \\end{bmatrix}-\\alpha\\begin{bmatrix}\n",
    " 2(h_\\theta(x_i)-y_i) \\\\ \n",
    " 2x_i(h_\\theta(x_i)-y_i) \\\\ \n",
    " \\end{bmatrix}$ <br>\n",
    "&nbsp;&nbsp;&nbsp;    5: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <b>end for</b> <br> \n",
    "&nbsp;&nbsp;&nbsp;    6: &nbsp;  <b>end for</b>\n",
    "\n",
    "Typically, with stochastic gradient descent, you will run through the entire data set 1 to 10 times (see value for k in line 2 of the pseudocode above), depending on how fast the data is converging and how large the data set is.\n",
    "\n",
    "In the example below, we'll create a set of 50,000 points around the line 𝑦 = 7+ 3𝑥 + 𝜖 , for values of x between 0 and 100:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "x = np.random.random(50000) * 100\n",
    "noise = np.random.randn(len(x)) * 10\n",
    "f = lambda x: 7 + x * 3 + noise\n",
    "\n",
    "y = f(x) \n",
    "m = len(y)\n",
    "batch = 100 # batch size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's randomly shuffle around our dataset. Note that in this example, this step isn't strictly necessary since the data is already in a random order. However, that obviously may not always be the case:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll setup our h function and our cost function, which we will use to check how the value is improving."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "h = lambda theta_0, theta_1, x: theta_0 + theta_1 * x\n",
    "cost = lambda theta_0, theta_1, x_i, y_i: 0.5 * (h(theta_0, theta_1, x_i) - y_i) ** 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll run our stochastic gradient descent algorithm. To see it's progress, we'll take a cost measurement at every step. Every 10,000 steps, we'll get an average cost from the last 10,000 steps and then append that to our cost_list variable. We will run through the entire list 10 times here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Local minimum occurs where:\n",
      "theta_0 = 7.07789735537464\n",
      "theta_1 = 3.0368460823859147\n"
     ]
    }
   ],
   "source": [
    "theta_old = np.array([0.,0.])\n",
    "theta_new = np.array([1.,1.]) # The algorithm starts at [1,1]\n",
    "lr = 0.00005 # learning rate\n",
    "batch = 1 # batch size\n",
    "\n",
    "iter_num = 0\n",
    "s_k = np.array([float(\"inf\"), float(\"inf\")])\n",
    "sum_cost = 0\n",
    "cost_list = []\n",
    "\n",
    "for i in range(m * 20): # In reality, you want while => condition to be gradient is sufficiently small\n",
    "    iter_num += 1\n",
    "    # random choice of matching x and y\n",
    "    idx = np.random.randint(0, m, size=batch) # randomly choose some i \n",
    "    x_i = x[idx]  # random choice of x\n",
    "    y_i = y[idx]  # random choice of y\n",
    "    \n",
    "    theta_old = theta_new\n",
    "    pred_y = h(theta_old[0], theta_old[1], x_i)\n",
    "    s_k[0] = pred_y - y_i\n",
    "    s_k[1] = (pred_y - y_i) * x_i\n",
    "    s_k = (-1) * s_k\n",
    "    theta_new = theta_old + lr * s_k\n",
    "    \n",
    "    # saved for plot\n",
    "    sum_cost += cost(theta_old[0], theta_old[1], x_i, y_i)\n",
    "    if (i + 1) % 10000 == 0:\n",
    "        cost_list.append(sum_cost / 10000.0)\n",
    "        sum_cost = 0   \n",
    "            \n",
    "print(\"Local minimum occurs where:\")\n",
    "print(\"theta_0 =\", theta_new[0])\n",
    "print(\"theta_1 =\", theta_new[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, our values for $\\theta_0$ and $\\theta_1$ are close to their true values of 7 and 3.\n",
    "\n",
    "Now, we plot our cost versus the number of iterations. As you can see, the cost goes down quickly at first, but starts to level off as we go through more iterations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO29d5xjZ33v//mqj9rMSFN2Zmd3Zmert9q764a7wTbYgIEkpJE4kHsNJCGFC1xabm5yye9HCK8EuEkA08IvlGCMjR2KC7axMeBd79rbvN2zbXoftVF/fn+c8xwdSUcaSSONZkff9+s1r5GOynlO0fN9vp2EEGAYhmGYXEz1HgDDMAyzPGEBwTAMwxjCAoJhGIYxhAUEwzAMYwgLCIZhGMYQS70HUAptbW2ir6+v3sNgGIa5rDh48OCkEKK90s9fFgKir68PBw4cqPcwGIZhLiuI6MJiPs8mJoZhGMYQFhAMwzCMISwgGIZhGENYQDAMwzCGsIBgGIZhDGEBwTAMwxjCAoJhGIYxZEULiKdPjOHffn623sNgGIa5LFnRAuK50xN44PmBeg+DYRjmsqSmAoKIWojoISI6SUQniOh6IvpH9fkRInqEiFpqtX+H1YxoIlWrr2cYhlnR1FqD+DyAx4UQWwDsAnACwFMAtgshdgI4DeBjtdq5w2JCNJEGd81jGIYpn5oJCCLyArgZwNcAQAgRF0LMCiGeFEIk1be9CKCnVmOwW80AgFgyXatdMAzDrFhqqUH0A5gA8A0ieoWIvkpErpz3vAfAT2s1AIcUEAkWEAzDMOVSSwFhAbAbwBeFEFcBCAP4qHyRiD4BIAng20YfJqL7iegAER2YmJioaAAOq3J40ST7IRiGYcqllgJiEMCgEGKf+vwhKAIDRHQfgDcD+H1RwEEghHhACLFXCLG3vb2ycuYOi6JBsKOaYRimfGomIIQQowAuEdFmddPrARwnojcC+J8A3iqEiNRq/0DGxBRlExPDMEzZ1Lph0AcAfJuIbAAGALwbwEsA7ACeIiIAeFEI8b5a7FwzMbEGwTAMUzY1FRBCiEMA9uZs3lDLferJaBAsIBiGYcplRWdSZ5zUbGJiGIYplxUtIOzspGYYhqmYFS0g2AfBMAxTOStaQEgNghPlGIZhymdFCwjNSc2JcgzDMGWzwgUEm5gYhmEqZYULCE6UYxiGqZQVLSCsZhPMJmINgmEYpgJWtIAAMj0hGIZhmPJY+QLCamYnNcMwTAU0hoBgExPDMEzZrHgBYbeaOA+CYRimAla8gHBYWINgGIaphJUvIKwm9kEwDMNUQAMICDNHMTEMw1RAgwgI1iAYhmHKpQEEhIkFBMMwTAWsfAFhYRMTwzBMJax4AWG3mhFjJzXDMEzZrHgBoZiYWINgGIYplwYQEOykZhiGqYSVLyAsZiTTAskUaxEMwzDlsPIFhGwalGQBwTAMUw4NICBkX2o2MzEMw5RDAwgI1iAYhmEqoQEEhGw7yhoEwzBMOax4AWG3sIBgGIaphBUvIDQTE+dCMAzDlEUDCAh2UjMMw1RCwwgI7gnBMAxTHg0gINjExDAMUwkrX0Cwk5phGKYiVr6A0MJcWYNgGIYphwYQENLExBoEwzBMOTSAgGAnNcMwTCWseAFht7CTmmEYphJqKiCIqIWIHiKik0R0goiuJyIfET1FRGfU/601HgPsFhPnQTAMw5RJrTWIzwN4XAixBcAuACcAfBTA00KIjQCeVp/XFG4axDAMUz41ExBE5AVwM4CvAYAQIi6EmAVwL4Bvqm/7JoC31WoMEm47yjAMUz611CD6AUwA+AYRvUJEXyUiF4BOIcQIAKj/O4w+TET3E9EBIjowMTGxqIE4rGZ2UjMMw5RJLQWEBcBuAF8UQlwFIIwyzElCiAeEEHuFEHvb29sXNRC7xcQmJoZhmDKppYAYBDAohNinPn8IisAYI6IuAFD/j9dwDACkD4JNTAzDMOVQMwEhhBgFcImINqubXg/gOIDHANynbrsPwKO1GoPEYWEnNcMwTLlYavz9HwDwbSKyARgA8G4oQulBIvpjABcB/FaNxwC71YRANFnr3TAMw6woaioghBCHAOw1eOn1tdxvLg6rGRPB2FLukmEY5rJnxWdSA5wHwTAMUwmNISAsnAfBMAxTLo0hIDgPgmEYpmwaREBwHgTDMEy5NIiAUPIghBD1HgrDMMxlQ8MICACIJdkPwTAMUyoNISBkT4gYO6oZhmFKpiEEBHeVYxiGKZ/GEhDsqGYYhimZBhEQ3HaUYRimXBpDQFhYg2AYhimXxhAQbGJiGIYpmwYREKqJicNcGYZhSqZBBARrEAzDMOXSIAJCOqlZQDAMw5RKQwgIu+qk5kQ5hmGY0mkIAcGJcgzDMOXTIAKCTUwMwzDl0iACQjqp2cTEMAxTKg0hIKxmE8wmYg2CYRimDBpCQADcdpRhGKZcGkdAWM2IsZOaYRimZBpKQLAGwTAMUzoNIyDsVhOHuTIMw5RBwwgIh8WMGDupGYZhSmZBAUFET5eybbnjsLKTmmEYphwshV4gIgcAJ4A2ImoFQOpLXgDdSzC2qqL4IFiDYBiGKZWCAgLAewH8JRRhcBAZAREA8K81HlfVcVjNCEQT9R4GwzDMZUNBASGE+DyAzxPRB4QQ/3cJx1QT2MTEMAxTHqU4qUeJyAMARPRJInqYiHbXeFxVx2FhExPDMEw5lCIg/loIESSiGwHcBeCbAL5Y22FVHzvnQTAMw5RFKQJCLrvvAfBFIcSjAGy1G1JtcFhNHObKMAxTBqUIiCEi+jKAdwL4CRHZS/zcssJuMXOiHMMwTBmUMtG/E8ATAN4ohJgF4APw4ZqOqgZ4HBYkUgLz8WwhIYTAWCBap1ExDMMsXxYUEEKICIDXANxFRH8GoEMI8WTNR1Zl/C7FKjYdiWdtf+bkOG749DMsJBiGYXIoJZP6LwB8G0CH+vctIvpArQdWbVpVATETzhYQ5ybDSKYFBmfm6zEshmGYZUuxRDnJHwO4VggRBgAi+gcAvwawYG4EEZ0HEITi6E4KIfYS0ZUAvgTAASAJ4E+EEPsrG37pSA1iKkdAyOfTOdsZhmEanVIEBCETyQT1MRV4rxG3CSEmdc8/A+BvhRA/JaK71ee3lvF9FVFIg5gOSQERq/UQGIZhLitKERDfALCPiB5Rn78NwNcWsU8BpZ4TADQDGF7Ed5VMYQ0iZridYRim0VlQQAgh/omIfg7gRiiaw7uFEK+U+P0CwJNEJAB8WQjxAJT6Tk8Q0Weh+EBeZ/RBIrofwP0AsHbt2hJ3VxivwwqzifI0CCkYcrczDMM0OgsKCCK6DsCrQoiX1eceIrpWCLGvhO+/QQgxTEQdAJ4iopMAfhPAXwkhfkBE74Sijbwh94OqMHkAAPbu3StKPyRjTCZCq9OapylI3wNrEAzDMNmUkgfxRQAh3fMwSiy1IYQYVv+PA3gEwDUA7gPwsPqW76vbloRWpy1fgwixk5phGMaIUgQECSG0FbwQIo3SNA+XrsifC8CdAI5B8Tncor7tdgBnyh10pbS6bFl5ELFkCqFYEgALCIZhmFxKcVIPENGfI6M1/AmAgRI+1wngESKS+/mOEOJxIgpBKSNuARCF6mdYCvwuG86MZ5QhKRSIWEAwDMPkUoqAeB+ALwD4JBSn89MoYVIXQgwA2GWw/QUAe8obZnVodWWbmKR5qae1SXvMMAzDKJQSxTQO4HeWYCw1x++yYSYSRzotYDKR5pje2OHBpelxRBMpOKzmOo+SYRhmeXDZVWVdDK1OG9ICmJtXWo/K5LiNHW4AHMnEMAyjp6EEhN+dnSwnzUobVAHBuRAMwzAZGkpAtDrVchuRTO6DxUToa3NpzxmGYRiFUsJVP2iweQ7AQSHEoeoPqXb4ZMlvWaAvFIfPZcuUAud6TAzDMBqlaBB7oUQyrVb/7odSXO8rRPSR2g2t+uQKiKlwDD6XTdvOkUwMwzAZSglz9QPYLYQIAQAR/Q2AhwDcDOAglGqslwX5AiKONrc9U6cpwgKCYRhGUooGsRaAfuZMAOgVQswDuKxsMg6rGU6bOWNiCismJqVOk42T5RiGYXSUokF8B8CLRPSo+vwtAL6rls84XrOR1Qh9PSbpgwCUHAk2MTEMw2QoJVHu/xDRT5Ap9/0+IcQB9eXfr+XgaoHfbcNUOI5YMoVgLIk2NfS11WVlDYJhGEZHKVFMnwfwPSHE55dgPDWn1alkU0th4HPZAQB+lx0nRgP1HBrDMMyyohQfxMsAPklEZ4noH4lob60HVUv8LsXXIM1J0sTkc7EPgmEYRs+CAkII8U0hxN1Q+jacBvAPRLRkJbqrTasUEKowkCYmn8uG2UgCyVS6nsNbVhy+NIunjo/VexgMw9SJcjKpNwDYAqAPwMmajGYJ8LlsiMRTGJmd157r/89EEnUb23LjX549i0/+8Gi9h8EwTJ1YUEAQkdQY/g7AqwD2CCHeUvOR1QgpCGRfCL/qg8gICDYzScYDUYwHY0iwVsVUESEEPvjgIfz6tal6D4VZgFI0iHMArhdCvFEI8XUhxGytB1VL9ALCYiJ4mxQ/vZ+zqfMYD8YgBDA6F633UJgl5F+fPYt7//WXNfv++UQKD788hBfOTtRsH0x1KCXM9UtE1EpE1wBw6LY/X9OR1QgpIF4bD8HnskHteAefOzvLutFJpwXGg0oe5MhcFGt8zjqPiFkqDl2axeFLs5iNxNGiFrisJqGo0uZ3Ps6a6XKnFBPTfwPwPIAnAPyt+v9/13ZYtUNWdB2andeEBQD4nFywT89UOI5UWmlFPjI3X+fRMEvJeEDRGE+MBGvy/UG1D/x8IlWT72eqRykmpr8AcDWAC0KI2wBcBeCy1Q39OqHQ5rZrj1u1Ok3spAaA8WDGrDQ0ywKikRjVBERt8oKCqgYRZQGRRzSRwld/MbBsoilLERBRIUQUAIjILoQ4CWBzbYdVO5qbrDApVqUsDcJqNsHrsLAGoTIeyJyHkVn2QTQKqbTAhGpaPFmjxNGMiYkFRC7PnZ7Ap358Aq9cWh6u3lJqMQ0SUQuAHwJ4iohmAAzXdli1QxbmmwrHswQEAPjddm4apDKmriK9DgubmBqIqVAMqmWxZiamUEzR0tnElE+mHfLymIdKcVK/XX34v4noWQDNAB6v6ahqTKvLppb6zhYQrU6uxySRDupda1owxBpEwyDNS31+J06NBZFMpWExV7fxpDQxsYDIR56b5dL+uKwrL4R4TgjxmBBieYy+QjLJcfac7XYWECpjgSh8Lht6/U7WIBqIMdW0eOvmDsSTaZybDFd9H+yDKEwwqmoQyyQfq6F6UktkxFKeiYnrMWmMBWLo8NjR1dyE2UgCkXiy3kNaEtJpga/+YkD7oTYa0rR46+Z2AMDxGjiqQzH2QRRCCs/ZZVLRoTEFhGpayjUx+dxKpVchxKL38dL5aZwdr40NdymYCEbR4XWgu0VJfRluEDPTkaE5fOrHJ/D0ifF6D6UujAWiMBFwXb8fVjPVxA8R4jDXgmgaxDJZqDamgCigQficNiRSAoHo4lbL0UQK7/n3l/DRH1y+dYzGAjF0euzobm4C0Di5EBenIwDQ0BpEu8cOh9WMDR2emoS6ynPLJqZ8LmsfxEqh1+9Ek9WMDq8ja7tWj2mRF+fJ42MIRpM4eHEGk6HLL2w2lRaYCMXQ6XWgu0UVEA2iQVySAiLWGCa1XMYCMaxSfxdXdNVKQLCJqRDy3LAPoo68Y3cPnvvwrXDbs4O42j2K03okp/ZQMJrAO7/0axwfLu3H8oODg3DZzBACeCbHVHFiJICDF2YWMfraM61mUXd47ej0OkDUOMlyF6cUARFapBZ5uTIWiGoLp61dXowHY5iq8iJHb2Kqhjl3JSG1K9Yg6ojZRHnaAwBc0eUFALw6PJe1/cD5Gew/P4395xauPjkWiOIXZybw7hvWYXVLE57U9VNIpQXe/62D+PjDy9v0JB2VHR4HbBYT2t32hjMxhRpWg4ii06sslOTvodp+CLlKTgsgvkwyhpcLmgbBAmL50e6xo7vZgSOD2QLikJrVOFlCpddHXhlCWgC/sacHd2ztxAtnJzRV+ukTYzg/FcHEMjc7yTIbcqLoamnK06pWKpqAaEANIpZMYSaS0JmYpICorplJf26jXLAvC+n/DESTy6LcBguIHHb0NOPIYHaa++FBKSCKT+xCCDx0cBB7e1uxrs2FO7Z2IppI4/kzSumqr71wDoDSc2I5XPxCyDIbUsvqbnY0hIkpkUprmlIj+iByr7vPZUOn1159AaE7txzJlE0wmkCT1QwAmJ2vf6AEC4gcdva04PxUBHNqHLIQAocvlSYgjgzO4ex4CL+xpwcAcM06H7wOC546PoZjQ3PYd24a/W0uCLF8nFBGyGSpdrWYYXdLE0ZmoyveXjw8O6+VmWjEKCZpWuzUmV+v6PJWPRciEE2g1WkFgIbJrymFWDKFWDKNXr9SWn85+CFYQOSws6cZAHB0SDEzXZqe19qQLmRieujgIOwWE+7Z2QVAKQB4+5YOPHNyHA88PwCXzYz33tIPYHk3JhoLRuF32WCzKLdHV7MD84mUVidmpSLNS267pSF9EHJhIE2LALCp04OBiXDVFgdCCIRiSS0ghDWIDNL/sFbtvbIc/BAsIHLYsVoREEeGFK1Bmpc2driLahDRRAqPHR7GXdtWweuwatvv2LoK0+E4Hjs8jN/auwbr2twAKhMQo3NR/I8HD9d81TUeiGk/YABaqOtKNzNJAbG1y2vog/j1a1MrWkjKOkyrdBpEc5MV8VQasWR1TKKReApCZCIGGzUX4uWLM3jvfxzIMjXnCojl0P6YBUQOLU6l/tBR1VF9+NIs7BYTbtjQhslQrOBK6mcnxjA3n8A7967J2n7L5nbYzCYQAe+5YR38avZ2JfkRjx8bwQ9eHsRL52sbJjsejGaZGRolF+LidAQ2swnrO1x5GkQsmcIffG0fPvez00s+rgtTYdz0mWcwOBOp6X7GA1HYLCY0N2UWODIUPFwljUqe1w6Pcn81ale5F85M4olXx7ICVqRZU5qYlkNvGhYQBuxY3axFMh0enMX21c1Y1exANJFGuEByz4MHBrG6pQmvW+/P2u62W/Cbe3vw+9euxVq/E21qgcBKBIQMNzwzVtsSHvpQR0BxUgMrP5v60nQEPb4meB1WbTUnmYskkEwLPHty6UtwnBoN4tL0fM3zZ+R1l214AcClCYjqrPTlJNjoJiapHegtCZoG4Xdlvaee1FRAENF5IjpKRIeI6IBu+weI6BQRvUpEn6nlGCphZ08zhmbnMRaI4ujQHHb1tGjd5yaD+RP78Ow8fnFmAr+xpwcmE+W9/v+8fQc+9bYdAABvkwUWE1XUd+KE2sDldA0FhGwYo9cg2tx2WM204st+X5yOYK3PCbfdglgyjbjOrCJNS+enIjWpcFoMuep+bTxU0/2MBqJZ5iUAcNvNWWNYLHISlAEQjSogZDE+vZ9BCs82tw1Om7lhnNS3CSGuFELsBQAiug3AvQB2CiG2AfjsEoyhLHb2tAAAfvDyIKKJNHatadZMQ1MGHecefnkQQgC/pUYvFYOI4Hfbys5OTabSODmqCIbTY7WbKKbCSsOYDp0PwmQirGp2LDsN4vM/O4OPP1K9pMOLU6qAcOSbVfS+h6XWIuQ4XpuonmCKJVP4j1+fx1O6RM7xQCwvgVTTIKrk99JMTKqGGm3QchuzqnagFxAyB8LrsKLVaVsWkY71MDG9H8CnhRAxABBCLLuymdtXN4MI+PaLFwEAu3patBXPRDD7ogkh8P2Dg7i+3481qnNpIdrc9pKS7vScmwwjnkyj1WnF2fFQzUJOc2PhJV3NTcvKBxFNpPDVFwbww1eGkEov/lzMRRIIRJOaBgFkr5rlis9qJjx7amlv2ZBq3nltYvELg3Ra4OGXB3H7Z5/DXz/6Kj70/cOao3gsEEWnx1hAsAZRXWRk5FQ438TkdVjhc9kaQoMQAJ4kooNEdL+6bROAm4hoHxE9R0RXG32QiO4nogNEdGBiYqLGw8zGbbdgfbsbQ7PzaG6yotfvzJiYclb++89N48JUBL+1d2HtQeJ328vWIGQs+j07uxCKJTFco8xmo1h4QPFDXJgOV2UyrgY/PzWBYDSJSDxVFZOPjGDqaXXCo2oQej+E1CBu3dyBfQPTVXPaloJs0Tkwufjz/6GHDuODDx5Gq8uKj7xxM+bmE/ivw8MIRhMIx1NY1ZzdRKvqTmopIBrcB5HRIPKd1G6HBa0uG6aXQU+IWguIG4QQuwG8CcCfEtHNUNqctgK4DsCHATxIeq+YihDiASHEXiHE3vb29hoPM5+darjrrjUtmlkIyBcQDx4YhNtuwZu2d5X83W0uW9kaxPGRAKxm0vZTKz+EbDWqNzEBwG1bOjAWiOFLz71Wk/2Wy6OHhmBTW2Hm1s6qBCkgFA1CieLRJ8tJAfH2q1YjnkrjV68tXJerWkgHcTyZxvAiQ433DUzjjq2deOxPb8T7b1mPjR1ufOvFC7ociAImpmppEOr3yAXXcqroemIkgMQSVTiYNeg9HYwm4bKZYTYRfE7rytcghBDD6v9xAI8AuAbAIICHhcJ+AGkAbbUcRyXIhLkr1f9WswktTmuWgEilBX56bAT37OhCk81c8nf73TZMhQuHzBpxYiSIDR0ebFXr49QqkklqEO05AuKtu7rxll3d+KenTmvRNEIIfOX5Afz+V19c0ozYQDSBp0+O451X98BmMeHVEqvsFkMKiDW+Jk2DCBn4IG7f0gG33bKkZia9JnN2kWam6XAcvT4nTCYCEeFd1/Xi8OCc5ovoyDExuW3yXFQ3isnjsMBhNS2bPIixQBT3fOEX+MnRkZrvK5UW2v00Gcp2UnvUHKoW5wo3MRGRi4g88jGAOwEcA/BDALer2zcBsAGYrNU4KuXqdT4AwLX9mbBVv8uWFZY2NDOPSDyF3b0tZX23321HNJFGpIzV04mRALZ2edHqsqHNba+Zo3osEEOb2wZrTqN6IsLfv307upod+PPvvoLBmQje+x8H8fc/OYFfnp3CE6+O1mQ8RjxxbBTxZBq/sbsHW1Z5StYgLk1H8MkfHjWsg3VxOgKfywaPw6o5qXMFhDKpmXHjhjY8e3K86n6gY0Nz+JNvH8xbxYZ1mceLiWSKJlKYT6TQqmuU9fbdq+G0mfHl5xXNcFVztoBwqlFM1TQxNVnNsJhNaLKal42JaXAmgrQAJgyiFKtNYD4BeevkahByceJz2RCMJbMi6epBLTWITgAvENFhAPsB/FgI8TiArwPoJ6JjAP4TwH1iGRb52dbdjF999HbcsCGj3CjO5cwNNDCp/FhldnSpFPJnFGIiGMNEMIYrujwAgE2d7pppEBPBKNo9+aXQAcV59oXfvQqjgShu++zP8czJcfz1m7didUsTHn55qKTvlwUNF+P0fOzwMNb6nLhyTQu2dTfj2FCgpMn66RNj+NaLF3F+Kj/hbHAmogUZeOzGPgiZQHbblnaMzEVxqsrX4NmT4/jJ0VHNzCcJxZJY09qEVqd1UZFM0tHe4swkwnkdVrztqtXaa7mmRavZBJvFVNVEOTkJNlnNy8bENDqnnPPc/JdaIM1LJsqNYkpo50YK8dn5+moRNRMQQogBIcQu9W+bEOLv1e1xIcS7hBDbhRC7hRDP1GoMi0VmEEvaPNnRR9I5uq7NVdb3ZvwZpV18WU1Tmpc2dXpwpkaRTFPheF6vbj2717bik/dcgV6/C9977/X44xvX4e1XrcYvz05iPLCw4/y1iTA+9P3D+K/DwxWNbzwYxS/PTuLeK7tBRNjW7cXcfKKkMiDS6WeUgCRzIAAU1CDkxHrr5g4AwLMnqxs8MaaWWQ/klPMIxZJwqYETi4lkkpORbLkrede1vQAUwejKaaIFVLc2VTCa1M6vw1Y7DWI8EMVXfzFQ8m9ElhlZihpc8v5b43NmBasoGoRyj8lrNFPnbGrOpC6Ddrc9K1Hu/GQYHrul6IRqhMymLjWSSQoIWZ9/Y6cbkXiqJrWRZsJxtDiLH8+7b1iHn33wFuzpbQWgmCnSAnj00MKTvsylGK0wCutHh0eQFsC9V3YDUEKSAeDY0MJ+CGnTzS2ClkylMTQzj7U+ZUHQZDXDRNl9C2YjcU2D6PQ6sK7NhaND2WXhF4t0FOcKiLC66l7f7sbAojQI5bhzr+/Wbi+u6fMVDNN22c1laRCnRoMFJ+ZgLKlpaE1Wc818EF994Rw+9eMTJUe4ycXNUvQBkddhfbsbgWhSMynqTUytLuVeq3fBPhYQZdDmVuyC8qYemAxjXbsLBkFYRckk3ZWuQXQ1OzS1c1OnYmo6UwM/xEwkAZ/OBFEK69vd2NXTjIdfWdjMJBsPyaZE5fL0yTFsWeXBhg7lHGxZ5YHZRDhegh9CrtxynX8jc1Ek00LTIIgob9WsNzEBSvHGavuBZIBArpkjFEvCZbNgfYcLk6GYVoq+XGTsvZx89Pzbu3bjy3+wx/BzLpulZCf1saE53PW55/HcaWPtKqRzxNbSB/H0CcXpfmG6tPpVS6lBSHNev2p5kPej3kntU3/r9S63wQKiDKTvQE7s5ybDZZuXAJ2JqUSH2PGRgKY9AMAmdXKsdqhrMpVGIJpYUIMw4h27e3BiJLBgcxmpOcjVcrmMzEXR35455w6rGRva3ThWQiST/LHlZqgOzihazeqWzArak1OPaW4+mSUgNnV6cG4yjFiyehOcFBCBaL6Jye2woF/1db02WZlgksefa2IClHu7kAbhtltK1iDkMbxwxjjuJBRLarkVTbba+CDOT4Y1X835EjUIeV8uRaMoKajXd6iVndX5JBBNwiud1Oo1Yg3iMsKvq8cUTSgmnkoEhN1ihsdhKUmDiCZSeG0irPkfAKDZaUWHp/qRTHNqdEVrmRoEALxlVzcsJsIjC2gRI5qAqEyDmA7H4XdlO1K3dXtLimSS1TFzNQgZLNChK1CoaBCZplGB+QSamzIT66ZVHqTSomp1mWQNLCBbgxBCIKxOqnJCqTSSSR53uQsAl91ScqkNKdxeLNC/PcsHYTVjPlF+lM539l3Evf/yAv7Pj47jZ8fH8lb9T6ulUCwmwgWDgAQjZGBAaL83DNEAACAASURBVAkaRc1G4jCRvmprHLFkCvFkGt6mTJgrUP+mQfkeKaYgbbpkuaZpM4Qo30Gd+S57SVFMZ8dDSKVFlgYBSEd1cQ0imUrj7i/8AjORBDq9dqzyOvD+WzdovoNcMiaI8jUIn8uGWze349FDQ/ifb9wCs0HRQgAYVX0QlWgQiVQas5GEpn5Ltq1WzFvjwWheHL+eTPZq9iQgr4Nf970eh0WbqOcTKcRT6RwNQpmsT40GsWVV9rWphKlQTOtmp/dBzCdSSAtlkl7T2gSrmSqOZJqJJOCymbVGUKXitltwqcRS44F55ZwdHw7kmeUAxcbvtuujmMpfsT96aAhnxkM4MRrE1144h+5mB5784C3a9z59YgybOt2wmEw4P7XwuRJCaBrEUjmpW5w2rdzIVDiu3WvSB2GzmOCxW7TfZL1gDaIM9OGp0lnYX2aIqyQ3p6IQcmUsQ1wlGzvdODMWQrpI6YXRQBSnx0Lo8zvR7rbj+dOT+MHLgwXfLyfQ1gpMTIBSBmQsECtq+pIaxFQ4VnbWqjSR5AYFbOtWJuhiCXNCCE1dn84puDgVisNsoqzjdjsyPgiZ1KSf7Prb3LCYqGp+IL3A1Js55BjcdgssZhP6/K6KI5lmIwsHIBhRjpNanqu0AA6cn856LZ0WCMUzZpRKfRBnxkN4665uHPmbO/Gvv7cbw3NRfF3t9x6IJrD/3DRu39KJvjZnSRpEIJrUxrE0TuoEWpqs2kJnOhTLExAA0OKysg/ickImK02G4pppoa+ttAJ9ufjdtpI0iEOX5uBxWNDnz9ZUNnV6MJ9IFV3ZSdv6n79+I77x7muwrs1VNHJKTqCVCoiNqm/kQpFV22ggCrvFBCHK74khBaovx8S0VRUQx4sIiPlESuuKllvjZjIUg89lyyrV7rZbtMnCSEDYLCb0tbmqlgsxqjO56TUIOQa5OlYimSoTENOReJ72VQouu6XkfhCB+QRsFhNsZhP2ncsWEOF4EkJkwogr8UFMhmKYDsexsdMDh9WMe3Z24a5tnXjg+QFMh+N4/vQEkmmBN1zRgT6/C5emI4aJkXpkBFOb2151DUIIkRfRNRtRQqZbnDaQmguhZZjbM/eYz2ljH8TlhMNqhttuwWQohnOTIbR77FrUQbm0ue0l+SAOXZrFlWta8vpM7OltBRHwvm+9jEsFIjWkgOhpVYSYUma88D5ni0S5lMJa1aZaaNU2H09hNpLQQlPLNTPJH4s/R4PwOpSCiseGCvshpKpOZOyD8OdMnB6HRVvJzxkkmAHVTViUPhmP3ZLlpJYTsyYgOly4MBWpqGbQTCSRdwyl4FZ9EKXkFASiCficNly5pgUvDmT7ITLakDIGh9WMaJk+CKmdShMfAHzozs2IxJP4t2fP4pkT42h1WnHV2lb0+V1IpgWGF6hCLIXz+nalk2A184v++anTuOcLL2Rtm4nE0eq0aVrrpIGJCVBMvaxBXGYoK/84zk9GKvY/KN9jx0wkXnR1E4kncWo0gCvX5Jfy2NTpwdf/6GoMzUTwln95wTBqRLao7G5xaPsstmqfWaSJSaljby0YWih/iLvUfhvlOqqNfAWSbd1ereKtEVIodDc3GQiIeF7tKb0GMWugQQDKNbgwHalKLP94IAoioL/dleWkDqqOcpdOg0imhVY7qhxm1YmpXFx2C4RASaVhAvNJeJssuK7fh2NDc1kFD0M5k2CT1Yx4Kr3gCl+PNOnJUG8A2NjpwTt29+D/e/ECnjoxhts2d8BsIs0JvJAfQvof1ne4kRbVrTD75PExHB8JZN0js5EEmp2ZcNbpkE6DcLAGcVnTpibLDUyGsc5fuYBoc9sgBIo6oY4OziEtYCggAOC2zR147M9uRIfHjj/8+j4cupSduDU4M49Orx12i1nbZzENYjoSh81sgrOMwoO59PpdBU1M8oe4a42iQZSSeZ01Pk2DsOe9trqlCWOBaMHVn/zs+g53Xo2byVBM8y9J3HYr5hMpJFNpQxMToExSQiiBBItFqYFlR4vTlmViytMg2pWV8+nR8jWXmXDlJiZlLAubXwLRBLwOK67t96t+iBnda6oGoQoIeZ9Fy6g3dHosCK/DklcS5C/fsBEQSpTU7Vcome596gKumMkTyEQwyXNbLT/EbCSuNfmS2rzcLgW1z6UIgUAhDYIFxOVFm9uG81NhTIZiWNe+CA2ihN7UcsIvJCAA5Ufwnf9+HdIC2J8TWjg4E9HMS8rY7VmJfrnMhhUTRLmJf3p6/YUdg6MB5UeyrbsZZhOVbWKaCinhgS1N+WaShQogzmjZq8o1kw55IYShiSnTVS6lTdheAwEBVCcfRbb79DZl51/IUFs5ni1dHtgsJrxyqbwsbiXHJVmhian0tqOBaALeJit2r22F1UxZ4a7y8zKT2qEKiHL8EGfGQtjY6cm7R3tanbjvdb1w2sy4eZPSHqDDY4fDajKsvaVndC6K5iarFvxQrVyI/TofjPQVxpNKX3sZSu53KZWdtXtMr0G4bAjHU1XNtSkXFhBl0ua2a5E4izExyZux2Ir+0KVZrPE1Ga6Yc8fU5rbhtfHsldLgzDx6WjP1pOQkWMj3MVOhE1NPr8+J4dl5wyqU8rytbmlCu9tetolpSl0BG/X91iJCCh2b1CDUVaJMlgvHU4gm0mjLWZFqBftiCczNJ2CizDZJn98Jm9lUFUf1WCCKTq8dHke2D0JmMLvUSdpuMWPn6ua8CKGFkGayikxMtoywXIi5+QS8DguabGbs6mnBiwOZcWZMTJlMagAlm+iEEDg9HszyP+j56JuuwM8/dKs2yRIR+gw02tzkOSmctTLvORrEn37nZfzbz8+W7ZvYf24a8laVfkJZfK85R4MI5mhXQOZazdYx1JUFRJnoTRH9i/RBAMY9riWKg9o4ZyGX/pxCbslUGqNz0SwBoWWCF9BalPjsyhzUkl6/C2kBwzpRo3NRtDitaLKZ0em1Y6zM0spTarSREW0FGjpJpiMJEGWE+rQqmOW5yDUx6XtCzEaUVXGuYLKYTehvd1Ul1HU8qPSD9jqsWi4BkDHr6KNb9vS24thQoCzfR6YOU2VOaqBEDUKXcX5dvx/Hhua0z+VqQ1JAlGrznwzFMRtJaNFyuZhNlNcqt9fvzEpmfO70BG797M+zBOxYIIrOZofmPNcfZyKVxo+PjOAzj5/CX37vUFnnfN+5aezt88FuMeGiqsVogSA6DWJ2XlmEyGZBEt8yqMfEAqJM5EqTKBO1U9H3LFDRdSwQxchctKh5Sc+GjmwBMRaMIZkWWSYm/wJay0wkUbGDWlLMMTgyp6zUAKXndSU+iNwsaokMfS2mQTQ3WTVntNQgJjUBYWxiCkaThglfkk2dHpyqwB+gJ5ZMYToc11ax8VRam4hC0SRMBDismZ/qnt5WxFPpolFbucjkwFr6INJpgaBqYgKAa/t9SKUFXlIn42BOyG6TTTmmUk1MZ7QIJmMBYYQS6jqvtWp9TC0oqe8IqPTitmvj0pv45IS+q6cZjx4axu9+5cWSekYEowm8OjyH69b5sNbn1ExMMzmh5D6X4ou8NB3Ji4hsXQbZ1CwgyqRN/YH1tDZpzt9K8DqssJio4Gr+lYsL+x/0rG93YyaS0CbIQa3Hcr4GMVFgn7OReEVZ1Hqk0LxoYPcdnYtqDWk6vfkmpgdfulQ0xn8qHIevQOVczXxWUPjF4XPa8n50E0GZfGfcizm0oIBQepcvpl+CnHA6vXZtcpVmJlm7SG9z361mwsvOfqWwmAg1TUAskPUcjieRFhk7+t5eH2wWkxZhlysgHGVqEEYhrgvR63chnkpjZG4eiVQaP1OL+B1Qz10ylcZEMIZVzQ7DToJS8/rjm/rxpXftxomRAP7+x8cX3O/BCzNIC6Xh2FqfExenFY1aBqXI+8mn3nfnp8LwNmWbMNf6nbj/5v48rWgpYQFRJlKDKLdJUC4mE8HnKpwsd+jSLKxm0rKEF0I6X6UWkZsDARTXIIQQqgaxOBNTu9sOp81s6KgemYuiSwoIjwMzkYTmgJsJx/GRHxzBd/ZdLPjdU6GYJqBz0QogFjDZSfOZNLHIFbU08RUyMQVjScwuoEEASnZvpUhBqZiYslex+uJ2kja3HevaXNokVwrVNjEFowl8/8ClLLu8jMSRE12TzYxr+nx4Xq3sqlSlzZhRyjUxnR4PZWmBpdCny8359WtTmJtPoKe1Ca9cmEE6LTAZiiMtlBLumUVBxuY/ozMJvXF7F+7dtRpPnxhfsNPbvnPTsJgIV61twRqfE5emIxBCYE71QciFmLyfL03P52kQXc1N+PjdV2BDx+LmmsXAAqJM5ESyGP+D/rsKrXgPXZrB1i6vtspaCOl8lYXcpICQORAA4LRZ4LSZDYVSIJpEKi0WbWIiIqz1OfMcg/FkGpOhGFZ5FY2mU10VjauRTIcGFY2pkAM9nlSicHKzqCVOmwVNVrPmW8hlOqzUcLKaTfA6LNqKejIos7NzTEzSHh1NqoX6iguIhcJOf3p0BF94+ozhazKaq9Pj0FbfMqolrDYLymVPbytevjBTsuN0cSam/Laj/3V4BB9+6EhWhJBRJM7Nm9pwZjyE4dl5BKOJLCes7OMeLcPEtKnTXVaUXa/6Oz0/FcZPj43CZTPjT27dgGAsidPjQU04d3od2nnWC8JczesNWzsRjCWzIpSM2H9uGjt6muG0WbDG50QolsRMJJElcABoGnE8lc4KcV0usIAok1VeB1qcVk3NXwx+t5JFmUsqLXB0cK5k8xKgRAbZLSYtJn9wJpKVAyFRhFK+gFhsHSY9vX5nXrKc/CFKDUJWTpV9IQ6rYZuFNCr5Q83Notbjd9sKChij2HO5v+Yma14Bu0xXuURRE9ManxN2y8KRTN/adwGf+9lpwz4Y8tysanZoq+8sDcJg4tjT24qpcHzBEE7JbCQOm8WkrdrLQUYx6XtCyKTHcYMSIfpw4Fs2KTkJvzgzobYbzbxWjgYhhMBpNcS1HLq8DtgsJgxMhPHkq6O4bUsHbtig9Jk/cH5GO45V6vvsFlNWmKv2u1AF640b2uCwmvDU8cI92OfjKRwZnMW165T9yD4jl6YjmFFzjeSx6wV2pVUZagkLiDJpsplx8JN34C07uxb9XTLpLpcz40GE4ylcubZ0AWEyUVYkkxLimu9ELzSJanWYKiyzoafX78LF6UhWIcFR3SQIZDQIuXqWOR+FNKpiWdQSJabc2Hw2rUsSa9UJiKlwzLAjoNNqBtHCTmqzibB9dfOC5p6z4yGkBfDjIyN5r40FYrCaCa1OqzZJ5PogctlbxA8hhMC9//ICvvdSxlynlHeoLMfFZCI4bdkF+yZUQacPsjBKKNzU6cYqrwPPn55USn3rjqUcATERjGFuPoGNZZpbTCZFo3300DCmwnG8aXsX1vqcaHPb8fKFmYwG0awsWDwOS1aYq9S85Iq/yWbGTRvb8dTxsYLa2ysXZ5BICVy7zgcAWKN2Krw4HcFcJDvXSL8gYw1ihWA20aKSySSdXgfGg9G8iqyHNAd1eVrK+naXVgp6cDaS5aCW+F12wyiMTEP7xWsQa31OxJNprccykMmB0DQI1Y4ss58X0iCKZVFL/AW0I1moTx6bvoTBZDCe538AlInFbbNgdC6KVFoUtd3ftLENRwZnC0abzM0nNEH4mEEv7rGAUqaciHQmJmWSChcQEOvb3fA6LDh4Id/UMRGM4fDgHJ5R+yIAi49Qc+U0DZKmQf310nwQupUwEeGmjW144ewk5uYTWZNgOYlypw1KbJRKn1/pxGe3mHDr5nYQEfb0tuCAKiDMJtLaAOd2EjTSvO64ohPDc9GCpV32qfkPe/qU3+8adaF2UdUg9NfBajZpApUFBJNFV7MDiZTIW/WemwzDZjGht0CHr0Ksb3fj0kwEkXgSI7NRQwHR7jFeZS+2DpMeWXn2/GTG/CH7QEgNotVpg9WsZFMrP5wE/OrK3qiEeaaS6wIahIEGIu2+Mq5cXwTNqMyGxOOwaPkchTQIALhpYzuEAH75mnEXNWn229vbilcuzuYVV5RJcnKfALTaPKGosQ/CZCLs6W011CDOqlrkiZGM2WsmXFkdJknuxCkj4fSLjYyJKXu8N29qx9x8Aq8OBww1iFJyC2QE08YyIpgk0lF96+Z27Vzu7fXh4nQEx4YC6PDYtRwXd44GYaR53X5FB4iAp46PGe5v/7lpXNHl1QSlS+1bPzij3OfNOYsNqRV72cTE6JGTpaxRJBlWo32MMoaLsb7DDSGAFwem8nIgJH6X3XASlitqo3aU5SJzIS5OZxzVI3NRuO0WzYRiMhE6PEouhDQv3bq5A8m0yGu5CWSc10bmIInPrQiYXNU/N/ZcCiJZZqPQd7pLFBC7eprhdVi0aJ1cZODAB+/YBCBfixgLZMJ/nWqUz0ImJkDxQ5weC+X1qJZa5MXpiDapz0TiizIf5vaEMNYg1ES4nPHeuKENRIpvTb9KtppNsJioJBPTmfEgWpxWrclOOUhH9Zu2Z8zC0of4y7OTmrkTUPwtwSwndb7m1ea2Y8/aVkMBkUilcejSLK7u82VtX+Nzaiam3EhBuehhDYLJQppbRuays45HZufR3Zy/+l8IGer63CllojI0MbltSKWFVnpBMhtRy0lU4SbtanbktXvU50BIOrx2jAWjeOXiLJqsZrxuveLUM0oenArFYDZR0VVWm8uOeCqdV0sn41/J+CBiSaUIXyCaLKhBuO0WDKsCIrcOkx6L2YQbN7bh+dOThnbpM+NB2C0mXNvvx57eVvxXjoAYD8S0TniKmUnpZieEWEBAKJPQyxeztQh9S9JTo4oZROlBsAgTky3TE0IJDzUQEPNJrbGRnlaXDTvVCr5ue/Z5VLrKFQ8ZnQ7H8ZOjo9jb21qRafeOKzrxu9esxZ3bOrVt21d7YbOYkEwLLXkTUO7/cI6Jyci8+IatnXh1OKDdH5LjwwHMJ1J5XRvXtDoNTUwACwimAJoGkZMwNjw7j66W8pNj+tvcIFLKCQAw1CAKlduQbRDL1VqMsJhN6GltyhIQ+hwISafHgbFADIcuzWLH6mZtJWfkR5BO5mLjy3ToyhYwueYzqSXJ3IVCfg23w4pESpnwW5qKT643bWzHaCBqWNn17HgI/e1umE2Et+7qxsnRoGYyCceSCMaSWatYj8OKwHwC0URaazdqxM6eZhABhwezC/cNTIa163xiJIh0WmiJgpWiNzFNR+JIqhrohO5cK5Vcjcd6y8Y25XtyXnfYzJhPFE/A+8cnTiIUS+LDd22paOyrmh34f9+xA05bZt+yphUAzbwH5JvS9MENeu7YqggbmXgnkcEKe/uyBcRanxPDs1HMROL5JiZVg/XY2cTE6Ghz2WExkebABZTMzrFgDKtbytcgmmxmrG5p0kIfuw2EjLwZc7Oppa21WvT6XbigMzGNzkWzJkFA+WGOzM7j+HAAV65tySTyGfhIJkPxohFMgC4RMCdZTpqY9FFMQMauXcjEpC/Ol/ujzkVWEH3OwMx0ZjykRd/cvaMLJsqUfMiEuGYmKW+TokEEc2oX5eKyW7C+3Z1XcuO18RBet94Pr8OCEyMBBKNKhvNi6my51KZBQMbv4LSZs6Lw5uYTBTWtWzYr5ye34KGiQRQ2MR2+NIv/fOkS/uh1fdi8qnwHdTGkE7lTt3DJ9UEU0rzWt7vR3+7C48eyw10PnJ/G6pYmdOVYANb6nEilBRKp/FwjeV8W01LrBQuIOmIyETq9jiwfxHgwhlRa5N1gpSIT5oxyIABkGqXnrrLDi6/DpEeW/RZCKOUMQrE8DaLD60A4nkI8lcaunpaixQSnw7GiORBApoR67rHJQn1aeQPVFi+L7OVWcpXoTTvFfBCAkoeyvt2FX+Q0borEkxiandeyYds9dtywoQ3fO3AJRwZns5LkJB67FYFoQtcLonDuwvZuL44NZaJp5uMpbX9burw4MRKoSgCCPopJ9k+4osuLiVBMM6sFigiIK9e04s9u24C7tq3K2l6sL3U6LfC/Hj2GNrdd6fdQZfasVQSE3sTktls1E6UQiim20MLpnh1deHFgShOYQggcuDCDq/vyow97fJnfc74PIjtAYTnBAqLOdDU7snwQ8nElJiYgIyCMzEuAropsARNTtVjrcyIYTeLXA1N46vgYUmmR54PQaxRXrm1Ro0UK+CCKFOqTFNJAZKE+WeJBTpRSgyjk+JQrd7OJ4CqhidJNG9ux79xUVlTOwEQYQiArfv/Dd22GmQhv/7df4fNPnwaArHo73iYLAvNJbUJ22QpPHNtXN2M0ENUmqYFJReitb3dja5cXp0aD2vlYjJPabTdrpheZHLe1y4t4MuPzCUSTBX1EZhPhQ3dtzitwqZiYjH0QDx64hMODc/j43VtqkkR286Z2vPuGPty6uUPb5nFYEE+mEUumFqwu8Oad3UgL4KfHlNyWS9PzmAjGsCfHQQ1kkuUAoDnHXLmrpxndzQ50V2A1qDUsIOrMquZsDWJoNtMzoRLWdyiOaiMHNaA02zGbKG8SVnpBVO9HuF6dEH/vK/vw/m+/DEDxkeiRtt92jx3dzQ5YzCa0Om2GJdCnQwv3qpCvGwk/vf3dp5mYpA+iQBSTqkG0NJWWYHbLpnZEE2mteimgOKgBZNXT2dnTgif+6mbcu6tb65egt4N7HFYEownDHgG57FDt6NLMJCOY1ne4cEWXB+F4CkdVH8ViNYhoQmkPqtcgAGhmJkWDKG8V3GQ1FSy18e+/Oo9da1rwtitXVzzuYjisZvzNW7Zl3VfymodjKV39KuPztnmVBxs73PjRYUVAyOtupEF0NTfBoi1Qsn9ne/t8+NXHXr+glloPlp9O02B0tzRpWZlEhBE1KiLXHFMqGQ3CWEDIIoH6SThTqK96GsTNG9vx5T/YAxMRPA4L2tz2vKJjUoO4ck2LNgEb5TLEkikEY8miIa6A8oN32y35GkROlVqvQwrJGJw2c5bzUo9U+Uv94V7b74PNbMLzpydw00bF5n52PASLidCb0562ucmKf/rtK/GmHV14bSKUtUL2OqwIRDMaRKEoJgDYqhZzPDY0h9u2dGBgIgQiJRclpq7MZWnrxeZBAEqDpYlgDB67RcsQngjG0N/u1tqNlkOT1WyoMUbiSZweC+LPbt9YlaTUUtFX8c3NnzHiLbu68c8/O43RuSgOXJiBx2HBJoN+FWYTYbUauLHYislLCWsQdWaV14FYMq1lMo/MReFxWCpWqbes8sBpM2N7d3PB9/hdNq3MNaBkGsd1mcbVwGwi3LVtFe7Y2onr+v2GFSlXNTtgNVPWistv0Ddby9FYwMRU+PPZdmSTibTnhUJcgYyAKNV56LRZcMMGPx49NKxVqT0zFkKv35lX60lyx9ZOvO+W9VnbvE1KJI0sXVFMQHgcVvS3uXBUp0GsaXXCYTVj8yoPTKTkxQCL1yAAJepqIhhDu9euVVWdDCl5NaFYsmxHa5PN2AdxYiSAtMhoSEuF1gckltCCG4r9Lt68swtCAD8+OoID56exp7e1YKSdNDMZtcxdrrCAqDOZXAjFtDRUYQ6EpMVpw0ufeAPeuH1Vwfe0ue1ZGkRmAl7aG9frsOLHf34T7ntdn7bN77bnleyWE/5CTmoguxCfZNYg9lxLmivynTJmvxzV/z03rsN4MIZH1SilsxOhgh3QCiEXB7JUSTEBASh+CM3ENB5Cv5oP47Casa7NhUA0CbOJFuUE1QuI8WAU7W67JlwnQzEEY0kIgYJhroVwFIhiOjKoHM/OnqUVEJ4sDWJh536/6uv57v6LODMe0mpkGbFGCogqLsRqDQuIOpPJhVBMSyNzleVA6HHlNJjJpS1nlV3NOkzlsqnTkxVt1WZgYpImo4XCXJX32LOSt3IL9Um0evxFNAi5miwnPPTGDW3Y2uXFV54fQDSRwoWpSNn1/OUkO6L6owrlQUi2r/ZieE5xVA9MhjQzIwBsUf0ELQYtU8tBRlKFYkmtPWqr0wYTKQIiYFCorxSarGbDUhtHh+bQ7rHnhUbXmkwV32Reae5CvHlXV6acioGDWvLWXd247/regtrkcuTyGekKRYazSg1ieDZa82gGvzt7Eq1mHabF4nfbMTefyGrIIp3OxQr1SdpyqtXKQn25dl/ptC4qIOzl+SAAJRP6/pv7cWY8hH//1Xmk0qLs+kFSgxienQeRkm9QjO2qGeap42OIJtJZAmKrFBCLzHGRkVThmOKD6PDYYTYRfGrxxzmDUt+l4CxgYjo6OKclsi0l+uZIs5E4TLRwjaQ37+gGAFhMhF09hSswX9fvx9/eu716g10CWEDUmXb1hzYyG0U0ofQm7q7QQV0qbW47IvEUImrik1aKooqJcpUiTT5SaAGlFeqT+Fw2zOhqTRVaBWY0iMLfWa6TWnLPzi50NzvwuZ8pIaz6CbsUZCTQ8FwUbltxbRAAtqn+pkcPDan7yzjEr+hSzFuLFf5SixkPRhGJp7RqvG1upSuirMNUiZN6PpHKKlESjiXx2kQIO5bYvARk9yIvtbrAWr8Te3tbsbu3VWuCtFJgAVFnzCZCh8eOkbmopkXUXoPIbj0qTUzLIbpC5jroNZypcBxWM5Vk3/a77VkF/3IL9Umkv6XaGgSgFKF7z43rEE2kQVSBgFAn2ZG5+aIhrpLmJit6/U7sU7ucrdeZtLasUjSIxV5beS7OTSphtB26EOWJUFwrT15umKvDZoYQQEynMR6vk4MayNYgZsKJkjWvr/zhXnzpXXtqObS6UFMBQUTniegoER0iogM5r32IiAQRtdVyDJcDq5odGA3Ma4W/Ks2iLhW5apaTsFytL4foiszY9BpEDD6XraRwR+mnkGam6bCx9tFagompq9mB/3HHJty9o/zmUL9zzVp4HBasaXWWvaqUAmI2kljQ/yCRZqbmJmuWr6ar2YE2ty0rW7gS5DgGVAHR7nao/5WmV4vRIIDskt/SQV0PAdFkbUe+sgAADpJJREFUNcNEGSd1qZpXq8tWUTvX5c5S5EHcJoTIqj9ARGsA3AGgcIf6BqKr2YGTo0FNQBjVUKombTnlNmbCcXgc+VU464FRpvd0CVnUmc9ntKP17Trhl/NDl+8rZmIiInzg9ZWVeHDbLfj0O3Zq4a7loF+FLxTBJNmxuhk/PjKC9e2uLEFKRPjP+69btIlJ0yAmsjWINo89y0lddpirrquctN4fG5pDp9eelV2+VBCRVrBvJpLA6hr/Fpc79ZoR/hnARwCU1nF9hbPK24TRuSiGZ7PbctYKvzvbjDMTSSyb1U+u+QsAJsPxkkJcAX09pljW9+Qe3zXr/Lh9S4eWaFYL7tnZhXfs7in7c3qhUKqAkHkv/QbmrA0dnpIc/MVwWE0wEXB+ShUQOh9ELJnGkOpQzy3GtxBNBl3ljgzOYsfq0tvtVhslkz2plvpeHr+LelFrASEAPElEB4nofgAgorcCGBJCHK7xvi8bupodiMRTODUWQJvbuMheNdGbYWQY6HL5IXjsFtjMJi0XIpUWODcRKpgZnktuPabHXx3FGl9TnpN6dUsTvv5HVy/LRvEWs0mr/eQqUqhPz47VzbBZTNhWI4FHRHDZLYjEU7Dp2mTKZLnXJsLw2C1lh9I6cvpSh2JJDEyG62JekigaRKLqFY4vR2ptYrpBCDFMRB0AniKikwA+AeDOhT6oCpT7AWDt2rW1HWWdkXkPL1+YXRKV1mE1w2O34Lv7L+I/fn0Bo4Eo7tzaufAHlwAiysqGPjUaRCCaxDXrCseX65GmlKlQHK8Oz2H/uWl84u4rlrRcQzXwOKwIx1N5DXYK0ey04ukP3lLTvAG3XSlD3u6xa+dTmisHJkIVlavO9UG8OjQHIZY+QU6P22HBZCiOaCI/PLrRqKkGIYQYVv+PA3gEwC0A1gE4TETnAfQAeJmI8tJ+hRAPCCH2CiH2tre313KYdadL1zio1g5qyZ6+ViRTAlev8+Hv7t2GT71t+cRnKwJC0SD2n1PKRFyzzl/SZ20WE7wOC6bDMXzzV+fRZDXjnXvX1GystUL6IYqV+s5lja9wSY9qIB3V7bry6FJADM3OV9RTOWNiUqKYZMmQ7XXWIGTf8OWQG1RPaqZBEJELgEkIEVQf3wng74QQHbr3nAewN9eJ3Wis0gmFxWZRl8q/v/uaJdlPJfhdds1EtF9twFJOdds2tx1nxkM4eGEGv7GnZ8FmP8sRafoqJcx1qZACosNAQAhRfogrkO2kBhQB0dXsyBJCS43bYdEq1rKJqXZ0AnhEVUUtAL4jhHi8hvu7bOnw2EGk/MgqLfO9kvC7bTg7HoIQAvvPTePmjeVpkH63Tatget/1fTUYYe2ROR+lhrkuBVKb6dCVJve5lHIbaVF+iCuQ8UFE4knEkinsG5iuq3kJyHa0LxffXL2o2d0nhBgAsGuB9/TVav+XE1azCe1uO8aDsSUzMS1n2tRSIAOTYUyG4iX7HyQyYul16/1Vb1O5VEh7fqlRTEuBLLchcyAAaOU2JkOxivoZSBNTNJHS/GGfuW5ndQZcIfpz3ugmpvoHvjMAMn6IWudAXA74XUro5LMnxwEAV5cpIGRIp75K7OWGLPOxnASEHItegwAyuSSLcVKPzsXwf585i5s3tWv9veuF3qy3mC58KwEWEMuEVZqAYA1CTvA/PTaKNrcN/W2uBT6RzY0b2nDr5na84YrlEZlVCdJcs5xMTEY+CCDjtK7ISa0KiK+9MIBANIGPvWnLIke5ePRCuaWpsTWI5XP3NTirW5QIlGKlHxoFmctw8MIM7t6xquwQ1bt3dFVUHmM5IZ3U5Sae1ZKMgMjWcuU9W4mT2q5GXQWiSbxzb4/WxrSe6LW3y6k0dy1YPndfg3P/zf14wxUdMC+iZv9KoU1XVuOaIvX1VzJysl1OGoSRkxpYnAZhMhEcVmUS/uAdmxc5wuogz/liS6SvBJbP3dfgrGp21LzExuWCvqxGqfkPK41enwsWEy2re+K2LR0YDSjd5PQsxgcBANf3+3HDhrZlc6zSxNToDmqABQSzDJFRSF6H5bKNQlosN2zwY9/HX7/oGkrVZFt3Mz71th1526WJqZIoJgD4xjLLyfFU0ElwpdLYBjZmWSJLgVzd52tYk5tScmT5CIdiXN3nwzXrfNhUZue85Yosb7JcCljWE9YgmGXJp96+vexGO0x9WONz4sH3Xl/vYVQNGebKJiYWEMwy5d4rV9d7CEyD4mYntQYLCIZhGB1ehwUfvmsz3rQ9r4Zow8ECgmEYRgcR4U9v21DvYSwL2EnNMAzDGMICgmEYhjGEBQTDMAxjCAsIhmEYxhAWEAzDMIwhLCAYhmEYQ1hAMAzDMIawgGAYhmEMISFEvcewIEQ0AeBChR9vAzBZxeFcLjTicTfiMQONedyNeMxA+cfdK4SouIfrZSEgFgMRHRBC7K33OJaaRjzuRjxmoDGPuxGPGVj642YTE8MwDGMICwiGYRjGkEYQEA/UewB1ohGPuxGPGWjM427EYwaW+LhXvA+CYRiGqYxG0CAYhmGYCmABwTAMwxiyogUEEb2RiE4R0Vki+mi9x1MKRLSGiJ4lohNE9CoR/YW63UdETxHRGfV/q7qdiOgL6jEeIaLduu+6T33/GSK6T7d9DxEdVT/zBSKiYvtYwmM3E9ErRPQj9fk6Itqnjud7RGRTt9vV52fV1/t03/ExdfspIrpLt93wXii0j6WCiFqI6CEiOqle8+tX+rUmor9S7+1jRPRdInKsxGtNRF8nonEiOqbbVrdrW2wfBRFCrMg/AGYArwHoB2ADcBjA1nqPq4RxdwHYrT72ADgNYCuAzwD4qLr9owD+QX18N4CfAiAA1wHYp273ARhQ/7eqj1vV1/YDuF79zE8BvEndbriPJTz2DwL4DoAfqc8fBPA76uMvAXi/+vhPAHxJffw7AL6nPt6qXmc7gHXq9TcXuxcK7WMJj/mbAP6b+tgGoGUlX2sAqwGcA9CkO/9/tBKvNYCbAewGcEy3rW7XttA+ih7DUv4YlvJPPXFP6J5/DMDH6j2uCo7jUQB3ADgFoEvd1gXglPr4ywB+V/f+U+rrvwvgy7rtX1a3dQE4qduuva/QPpboOHsAPA3gdgA/Um/iSQCW3OsJ4AkA16uPLer7KPcay/cVuheK7WOJjtkLZbKknO0r9lpDERCX1AnPol7ru1bqtQbQh2wBUbdrW2gfxca/kk1M8kaUDKrbLhtUdfoqAPsAdAohRgBA/d+hvq3QcRbbPmiwHUX2sRR8DsBHAKTV534As0KIpME4tWNTX59T31/uuSi2j6WgH8AEgG+QYlr7KhG5sIKvtRBiCMBnAVwEMALl2h3Eyr/Wknpe27LnxJUsIMhg22UT00tEbgA/APCXQohAsbcabBMVbK8bRPRmAONCiIP6zQZvFQu8drmdCwsUE8QXhRBXAQhDMQkU4nI7vjxUe/i9UMxC3QBcAN5k8NaVdq0XYimOp+zPrGQBMQhgje55D4DhOo2lLIjICkU4fFsI8bC6eYyIutTXuwCMq9sLHWex7T0G24vto9bcAOCtRHQewH9CMTN9DkALEVkMxqkdm/p6M4BplH8uJovsYykYBDAohNinPn8IisBYydf6DQDOCSEmhBAJAA8DeB1W/rWW1PPalj0nrmQB8RKAjWrkgg2Kg+uxOo9pQdRIhK8BOCGE+CfdS48BkBEM90HxTcjtf6hGKFwHYE5VK58AcCcRtaqrtjuh2FxHAASJ6Dp1X3+Y811G+6gpQoiPCSF6hBB9UK7TM0KI3wfwLIDfNBiPfpy/qb5fqNt/R418WQdgIxRHnuG9oH6m0D5qjhBiFMAlItqsbno9gONYwdcaimnpOiJyqmOSx7yir7WOel7bQvsozFI4pur1B8VrfxpKVMMn6j2eEsd8IxS17wiAQ+rf3VBsqE8DOKP+96nvJwD/qh7jUQB7dd/1HgBn1b9367bvBXBM/cy/IJNRb7iPJT7+W5GJYuqH8qM/C+D7AOzqdof6/Kz6er/u859Qj+sU1KiOYvdCoX0s4fFeCeCAer1/CCVSZUVfawB/C+CkOq7/gBKJtOKuNYDvQvGzJKCs3v+4nte22D4K/XGpDYZhGMaQlWxiYhiGYRYBCwiGYRjGEBYQDMMwjCEsIBiGYRhDWEAwDMMwhrCAYBoCIvqV+r+PiH6vyt/9caN9MczlDoe5Mg0FEd0K4ENCiDeX8RmzECJV5PWQEMJdjfExzHKCNQimISCikPrw0wBuIqJDpPQlMBPRPxLRS2qN/Peq77+VlL4c34GSVAQi+iERHSSll8H96rZPA2hSv+/b+n2pGav/SErfg6NE9Nu67/45ZfpAfFvNhgURfZqIjqtj+exSniOGycWy8FsYZkXxUeg0CHWinxNCXE1EdgC/JKIn1fdeA2C7EOKc+vw9QohpImoC8BIR/UAI8VEi+jMhxJUG+3oHlEzpXQDa1M88r752FYBtUGrh/BLADUR0HMDbAWwRQggiaqn60TNMGbAGwTQ6d0KpT3MISll1P5S6PgCwXyccAODPiegwgBehFD3biOLcCOC7QoiUEGIMwHMArtZ996AQIg2lnEofgACAKICvEtE7AEQWfXQMswhYQDCNDgH4gBDiSvVvnRBCahBh7U2K7+INUJrX7ALwCpQ6QQt9dyFiuscpKI1sklC0lh8AeBuAx8s6EoapMiwgmEYjCKWVq+QJAO8npcQ6iGgTKU17cmkGMCOEiBDRFigtGyUJ+fkcngfw26qfox1KC8r9hQZGSg+QZiHETwD8JRTzFMPUDfZBMI3GEQBJ1VT07wA+D8W887LqKJ6AsnrP5XEA7yOiI1Cqh76oe+0BAEeI6GWhlCmXPAKlteVhKBV6PyKEGFUFjBEeAI8SkQOK9vFXlR0iw1QHDnNlGIZhDGETE8MwDGMICwiGYRjGEBYQDMMwjCEsIBiGYRhDWEAwDMMwhrCAYBiGYQxhAcEwDMMY8v8Dwbr22LMKsgIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "iterations = np.arange(len(cost_list)) * 10000\n",
    "plt.plot(iterations,cost_list)\n",
    "plt.xlabel(\"iterations\")\n",
    "plt.ylabel(\"avg cost\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, that was kind of cool, but I wish that I didn't need to calculate the gradients myself before I can apply gradient descent or stochastic gradient descent. I wonder if there is some tool that can do that for whichever functions that I am interested in... <b>Pytorch</b> comes to help!\n",
    "\n",
    "So let's use the same example from above and use Pytorch to help us calculate gradients automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Local minimum occurs where:\n",
      "theta_0 = 7.0001702308654785\n",
      "theta_1 = 2.9323341846466064\n"
     ]
    }
   ],
   "source": [
    "# conda install pytorch\n",
    "import torch\n",
    "\n",
    "torch.manual_seed(42)\n",
    "\n",
    "lr = 0.00005 # learning rate or step size\n",
    "\n",
    "# let's get our data points x and y into the expected format\n",
    "x_tensor = torch.from_numpy(x).float()\n",
    "y_tensor = torch.from_numpy(y).float()\n",
    "\n",
    "# theta_0, theta_1\n",
    "theta = torch.tensor([1., 1.], requires_grad=True, dtype=torch.float)\n",
    "\n",
    "iter_num = 0\n",
    "cost_list = []\n",
    "\n",
    "# stopping condition based on iteration count\n",
    "for i in range(m * 5):\n",
    "    iter_num += 1\n",
    "    # random choice of matching x and y\n",
    "    idx = np.random.randint(0, m, size=batch)\n",
    "    x_i = x_tensor[idx]\n",
    "    y_i = y_tensor[idx]\n",
    "    \n",
    "    yhat = theta[0] + theta[1] * x_i\n",
    "    error = y_i - yhat\n",
    "    loss = (error ** 2).mean() #J\n",
    "    sum_cost += float(loss.data)\n",
    "\n",
    "    loss.backward() # this is the magic\n",
    "\n",
    "    theta.data -= lr * theta.grad\n",
    "#     theta.grad.detach()\n",
    "    theta.grad.zero_() # Fills self tensor with zeros\n",
    "    \n",
    "    # saved for plot\n",
    "    if (i + 1) % 10000 == 0:\n",
    "        cost_list.append(sum_cost / 10000.0)\n",
    "        sum_cost = 0\n",
    "            \n",
    "print(\"Local minimum occurs where:\")\n",
    "print(\"theta_0 =\", float(theta[0].data))\n",
    "print(\"theta_1 =\", float(theta[1].data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZ8AAAEGCAYAAAC6i5gfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU5fX48c/JToawJBl2yAKIsi8BRKyiVsF9ryJVFPxabW3tZtXaajd/rbXWtlqtVikuQK0o4obibqsCBgghIPsaEiBhSQIhZDu/P+YGh5hJJsksGXLer1demXnm3jvPZUJO7nPPcx5RVYwxxphQigp3B4wxxrQ/FnyMMcaEnAUfY4wxIWfBxxhjTMhZ8DHGGBNyMeHuQCikpqZqenp6uLthjDERZfny5cWq6g7GsdtF8ElPTyc7Ozvc3TDGmIgiItuDdWwbdjPGGBNyFnyMMcaEnAUfY4wxIWfBxxhjTMhZ8DHGGBNyFnyMMcaEnAUfY4wxIRe04CMis0Rkr4jkebX9VkRyRSRHRBaLSC+n/U6nLUdE8kSkRkSSGzjmbBHZ6rXtyGD1H+CDdXt4/KNNwXwLY4xpl4J55TMbmFKv7SFVHa6qI4E3gPsAVPUhVR3ptN8DfKyq+30c9866bVU1J1idB/h00z4efX8TtuaRMcYEVtCCj6p+Auyv11bq9dQFNPRbfSowL1j9ao70lESOVNWwt+xouLtijDEnlJDf8xGRB0RkJzAN58rH67VEPFdLLzdyiAecobtHRCS+kfe5RUSyRSS7qKioRX1NS3EBsK34cIv2N8YY07CQBx9VvVdV+wJzgNvrvXwx8GkjQ273ACcDY4Fk4K5G3ucpVc1S1Sy3u2V18dKd4LN9X3mL9jfGGNOwcGa7zQWurNd2LY0MualqoXocBf4FjAti/+jVJYHYaGHrPrvyMcaYQApp8BGRgV5PLwHWeb3WGTgTWNjI/j2d7wJcBuT52jYQYqKj6Ns1ke0WfIwxJqCCtqSCiMwDJgGpIpIP3A9cICKDgFpgO3Cr1y6XA4tV9XC947wF3KyqBcAcEXEDAuTU2z8o0lIS2VZsw27GGBNIQQs+qjq1geZnGtl+Np707PrtF3g9PjsQfWuOtBQXy7buR1XxXHAZY4xpLatw0ISMVBeHK2soOmTp1sYYEygWfJqQlpIIWMabMcYEkgWfJqTbXB9jjAk4Cz5N6N21A9FRYlc+xhgTQBZ8mhAbHUXfrh3YZunWxhgTMBZ8/JCW4rLgY4wxAWTBxw/pKYlsLy636tbGGBMgFnz8kJbiouxoNfsPV4a7K8YYc0Kw4OOH9FRPuvU2SzowxpiAsODjB0u3NsaYwLLg44c+XROJEqzAqDHGBIgFHz/ExUTRu2sHG3YzxpgAseDjp/QUl135GGNMgFjw8VN6ioutxYct3doYYwLAgo+f0lISKa2o5mB5Vbi7YowxEc+Cj5+OZbzZ0JsxxrSaBR8/1c31sQKjxhjTekENPiIyS0T2ikieV9tvRSRXRHJEZLGI9HLaJ4lIidOeIyL3+ThmhogsFZGNIvKiiMQF8xzq9E1ORAS22lwfY4xptWBf+cwGptRre0hVh6vqSOANwDvI/FdVRzpfv/FxzAeBR1R1IHAAmBnoTjckPiaaXp07WMabMcYEQFCDj6p+Auyv11bq9dQF+J0+JiICnA3Md5qeBS5rZTf9lp6aaHN9jDEmAMJyz0dEHhCRncA0jr/ymSAiq0RkkYgMaWDXFOCgqlY7z/OB3j7e4xYRyRaR7KKiooD0O83m+hhjTECEJfio6r2q2heYA9zuNK8A0lR1BPAo8GoDu0pDh/PxHk+papaqZrnd7kB0m4wUFwfKqyixdGtjjGmVcGe7zQWuBM9wnKoech6/BcSKSGq97YuBLiIS4zzvAxSEqrNpKXXVre3qxxhjWiPkwUdEBno9vQRY57T3cO7pICLjnL7t895XPeUFPgSucpqmAwuD3ec66ak218cYYwIhpulNWk5E5gGTgFQRyQfuBy4QkUFALbAduNXZ/CrgNhGpBo4A1zrBBhF5C7hZVQuAu4B/i8jvgJXAM8E8B2/9km2ujzHGBEJQg4+qTm2gucFgoaqPAY/5eO0Cr8dbgHEB6WAzJcRG07Nzgl35GGNMK4X7nk/ESU9x2aJyxhjTShZ8mik9NdGG3YwxppUs+DRTWoqLfYcrKa2wdGtjjGkpCz7NlO6kW++wqx9jjGkxCz7NVJdubQVGjTGm5Sz4NNNX6dYWfIwxpqUs+DRTYlwM3TvFW4FRY4xpBQs+LWAFRo0xpnUs+LRARoqLrcV25WOMMS1lwacF0lITKT50lENHq5ve2BhjzNdY8GmB9BRPxpsNvRljTMtY8GmBuqUVrNKBMca0jAWfFkhLsaUVjDGmNSz4tEDH+BjcSfFWYNQYY1rIgk8Lpack2lwfY4xpIQs+LWRzfYwxpuUs+LRQekoie0qPUl5p6dbGGNNcQQs+IjJLRPaKSJ5X229FJFdEckRksYj0ctqnOe25IvKZiIzwcczZIrLV2T9HREYGq/9NqSswahlvxhjTfMG88pkNTKnX9pCqDlfVkcAbwH1O+1bgTFUdDvwWeKqR496pqiOdr5xAd9pfNtfHGGNaLiZYB1bVT0QkvV5bqddTF6BO+2de7UuAPsHqV6D0c+b6WNKBMcY0X8jv+YjIAyKyE5jGV1c+3mYCixo5xAPO8NwjIhLfyPvcIiLZIpJdVFTUyl5/XaeEWFJccXblY4wxLRDy4KOq96pqX2AOcLv3ayJyFp7gc5eP3e8BTgbGAsmNbIeqPqWqWaqa5Xa7A9L3+tJTXbaonDHGtEA4s93mAlfWPRGR4cDTwKWquq+hHVS1UD2OAv8CxoWkpz6kpSRawoExxrRASIOPiAz0enoJsM5p7we8Alyvqhsa2b+n812Ay4A8X9uGQnqKi8KSCiqqasLZDWOMiThBSzgQkXnAJCBVRPKB+4ELRGQQUAtsB251Nr8PSAEe98QVqlU1yznOW8DNqloAzBERNyBAjtf+YVFXYHTH/nJO6p4Uzq4YY0xECWa229QGmp/xse3NwM0+XrvA6/HZgeldYNSlW28tPmzBxxhjmsEqHLSCzfUxxpiWseDTCp0TY+maGGtzfYwxppks+LSSFRg1xpjms+DTSukpiWwrtisfY4xpDgs+rZSe6qKg5IilWxtjTDNY8Gml9BQXqpB/wK5+jDHGXxZ8Wqluro8NvRljjP8s+LRSXbr1Nks6MMYYv1nwaaWurjg6d4i14GOMMc1gwScA0q3AqDHGNIsFnwBIS3HZlY8xxjSDBZ8ASE9JZNeBI1RW14a7K8YYExEs+ARAWoqLWoWdlm5tjDF+seATAOmpVmDUGGOaw4JPAKTbXB9jjGkWCz4BkOyKIyk+xq58jDHGTxZ8AkBESEtNZKulWxtjjF+CGnxEZJaI7BWRPK+234pIrojkiMhiEenltIuI/E1ENjmvj/ZxzDEistrZ7m/irLsdbum2tIIxxvgt2Fc+s4Ep9doeUtXhqjoSeAO4z2k/HxjofN0CPOHjmE84r9dtW//4YZGe4iL/wBGqaizd2hhjmhLU4KOqnwD767WVej11Aeo8vhR4Tj2WAF1EpKf3vs7zTqr6uaoq8BxwWdBOoBnSUhKpqVV2HTgS7q4YY0yb12TwEZH3/WlrDhF5QER2AtP46sqnN7DTa7N8p81bb6e9sW3Coi7d2iodGGNM03wGHxFJEJFkIFVEuopIsvOVDvRqzZuq6r2q2heYA9xe95YNbVq/W35s49lQ5BYRyRaR7KKiopZ31k/HqlsXW/AxxpimNHbl8x1gOXCy873uayHw9wC9/1zgSudxPtDX67U+QEG97fOd9sa2AUBVn1LVLFXNcrvdAequb6kd43DFRbPNMt6MMaZJPoOPqv5VVTOAn6pqpqpmOF8jVPWxlr6hiAz0enoJsM55/Bpwg5P1dipQoqqF9fpUCJSJyKlOltsNeIJh2IkIaZbxZowxfonxY5vdIpKkqmUi8gtgNPA7VV3R1I4iMg+YhGfoLh+4H7hARAYBtcB24FZn87eAC4BNQDlwk9dxcpzsOIDb8GTRdQAWOV9tQnpqIusKy8LdDWOMafP8CT6/VNWXROR0YDLwJzzpzuOb2lFVpzbQ/IyPbRX4no/XRno9zgaG+tHvkEtLcbF4zR6qa2qJibb5u8YY44s/vyFrnO8XAk+o6kIgLnhdilwZKS6qa5WCgxXh7ooxxrRp/gSfXSLyJPAt4C0Rifdzv3Ynra7AqN33McaYRvkTRL4FvANMUdWDQDJwZ1B7FaFsaQVjjPFPk8FHVcuBzcBkEbkd6Kaqi4PeswjULSmeDrHRbLWlFYwxplH+VDi4A89k0G7O1wsi8v1gdywSiQjpqS62Fh8Kd1eMMaZN8yfbbSYwXlUPA4jIg8DnwKPB7FikynS7yNtVEu5uGGNMm+bPPR/hq4w3nMdtYhmDtqh/qoud+8s5Wl3T9MbGGNNO+XPl8y9gqYgscJ5fho+5OgYy3C5qFXbsK2dg96Rwd8cYY9okfxIO/oyn2sB+4ABwk6r+Jdgdi1SZqR0B2FxkGW/GGONLk1c+Tp21NXXldEQkSUTGq+rSoPcuAmW6PenWWyzpwBhjfPLnns8TgPdv0sP4XmW03UtKiMWdFM8Wu/Ixxhif/Eo4cOquAaCqtfh3r6jdykx1sdXW9THGGJ/8CT5bROQHIhLrfN0BbAl2xyJZprsjW4ps2M0YY3zxJ/jcCpwG7MKzmNt44JZgdirS9Xe7OFBexYHDleHuijHGtElNDp+p6l7g2hD05YThnXQwxpUc5t4YY0zbY9Wpg8DSrY0xpnEWfIKgT9cOxEaLZbwZY4wPFnyCICY6in7JiZZ0YIwxPvgzyfTHDTSXAMtVNaeR/WYBFwF7VXWo0/YQcDFQiWeZhptU9aCITOP4NYKGA6PrH19EfgX8H1DkNP1cVd9q6hzCIdPdkS2Wbm2MMQ3y58onC0/GW2/n6xZgEvBPEflZI/vNBqbUa3sXGKqqw4ENwD0AqjpHVUeq6kjgemBbI4Htkbpt22rgAU/SwfZ9h6muqQ13V4wxps3xJ/ik4LkK+Ymq/gRPMHIDZwA3+tpJVT/BUw/Ou22xqlY7T5cAfRrYdSowz49+tWn9UztSVaPkHzgS7q4YY0yb40/w6YdnmKxOFZCmqkeAo6147xnAogbar6Hx4HO7iOSKyCwR6eprIxG5RUSyRSS7qKjI12ZBU5dubZUOjDHm6/wJPnOBJSJyv4jcD3wKzBMRF7C2JW8qIvcC1XhWSPVuHw+Uq2qej12fAPoDI4FC4GFf76GqT6lqlqpmud3ulnSzVTLddenWlnRgjDH1+TPJ9Lci8hZwOp5F5G5V1Wzn5WnNfUMRmY4nEeEc75pxjmtp5KpHVfd4HeefwBvNff9QSXbF0SUx1pIOjDGmAf5ku/0VeFFV/9raNxORKcBdwJmqWl7vtSjgajz3knzt31NVC52nlwO+rpDahMxUl6VbG2NMA/wZdlsB/EJENonIQyKS5c+BRWQe8DkwSETyRWQm8BiQBLwrIjki8g+vXc4A8lV1S73jPO31nn8UkdUikgucBfzIn76Ei6fAqF35GGNMff4Muz0LPCsiycCVwIMi0k9VBzax39QGmn0uv62qHwGnNtB+s9fj65vqb1uSkepi/vJ8yiqqSEqIDXd3jDGmzWhOhYMBwMlAOrAuKL05wfS3jDdjjGlQk8FHRB4UkY3Ab4A1wBhVvTjoPTsB1GW82dCbMcYcz58VSbcCE1S1ONidOdGkpSQSJVjSgTHG1OPPPZ9/iEhXERkHJHi1fxLUnp0A4mOi6dM1kc027GaMMcfxJ9X6ZuAOPKVwcvAkBXwOnB3crp0YMt0uttqwm4kgqkrRoaN0S0poemNjWsifhIM7gLHAdlU9CxjFV1WlTRMyUzuytfgwtbX159Ma0/bU1Co/+c8qJvz+A/J2lYS7O+YE5k/wqVDVCgARiVfVdcCg4HbrxJHpdnGkqobdpRXh7ooxjaquqeXH/8nhlZW7EGDW/7aGu0vmBOZP8MkXkS7Aq3gmhy4ECoLbrRNHXYFRy3gzbVlVTS13vJjDwpwC7ppyMt8+NY3XcwvY2w7/aCo+dJSj1TXh7sYJr8ngo6qXq+pBVf0V8Es8E0UvC3bHThT969Ktiy3jzbRNldW1fH/uSt7MLeTeC07htkn9uWliOtW1yvNLtoe7eyFVXlnNuX/+mN+/ZVMZg61Zy2ir6seq+pqqVja9tQHolhSPKy7arnxMm3S0uobvzV3B22t2c99Fg/m/MzIBSEtx8c1TujNn6Q4qqtrPVcDbebs5UF7Fyyvy29V5h0Ozgo9pPhEhw+2ypRVMm1NRVcNtL6zg3bV7+M2lQ5hxesZxr8+YmMH+w5UszNkVph6G3kvZ+STERlFWUc27a/c0vYNpMQs+IZCZagVGQ62soorc/IMszNnFn9/dwN/e38jXV/BovyqqavjO88v5YN1eHrh8KDdMSP/aNqdmJnNKz07M+t+2dvFvt3N/OZ9v2cetZ/anV+cE5i/PD3eXTmj+VDgwrZTpdvF6bgEVVTUkxEaHuzsnjKqaWvIPHGFL0SG2FB1mS/Fhz+PiwxSVfX2R3TNPcjOib5cw9LRtOVJZwy3PZ/O/TcU8eOUwrhnbr8HtRIQZE9O5c34un23ex8QBqSHuaWjNX56PCFyd1ZfqGuXxjzaxu6SCHp1tvlMwWPAJgUx3R1Q9BUZP6dkp3N2JWKrK0q37ef7z7Xy5u5Qd+8qp9po/1TUxlkx3R848yU2m20Vmakf6u110TozltN9/wKK83e0++JRXVjNzdjZLtu7joatGcNWYPo1uf/GIXjz49jpm/W/rCR18amuV+cvzmdg/ld5dOnDlmD489uEmFqzcxW2T+oe7ey1ypLKGhxev545vDmyTVfUt+IRAZupX6dYWfJpPVflg3V7+/uEmVuw4SIorjrHpyUwe0oPMVBeZ7o5kprro6orzeYwJ/VN4O6+Qu6YMQkRC2Pu24/DRam6a/QXZ2/bzyLdGctmo3k3ukxAbzbTxafz1/Y1sLT5MhvOzfKJZsnUfuw4e4WdTPFMYM1JdZKV1Zf7yndx6ZmbE/cxUVNXwf89l8+nmYib0T+GcU7qHu0tfY8EnBDKPLa1gSQfNUVOrvLm6kMc/3MS63WX07tKB3146hKuz+jZ7+HLykB784tU8Nuw5xKAeSUHqcdtVVlHFTf/6gpU7D/LXa0dx8Yhefu877dR+PPHRZmZ/upVfXzo0iL0Mn/nZ+SQlxDB5SI9jbVeN6cPdr6wmZ+dBRvXrGsbeNU/d/bxPNxfzxyuHt8nAA5ZwEBKJcTH07JxgSQd+Olpdw7xlOzjn4Y/4wbyVVNXU8vDVI/jozklcPyG9RffNzhvSHRFPKm17U1pRxQ2zlpGz8yCPTm1e4AHolpTAxSN68dLyfEqOVAWpl+FTVlHFW3mFXDyi13E/WxcM70lCbBQvr4icxIOj1TV8d84KPt5QxO8vH8bVWX3D3SWfghZ8RGSWiOwVkTyvtodEZJ2I5IrIAqdyAiKSLiJHnKW16y+v7X3MZBF5V0Q2Ot8j5s+RTLfLqls3obyymqf/u4Uz/vgh97yymqSEWP7x7TG8+6MzuXJMH2KjW/7j2i0pgay0rizKKwxgj9u+2lrlxlnLyNtVwmPXjeaCYT1bdJybJqZTXlnDi1/sCHAPw+/N3EIqqmq5ut79r04JsUwZ0oPXcgoiYs5PZXUt35uz4lgG47XjGk4kaSuCeeUzG5hSr+1dYKiqDgc2APd4vbZZVUc6X7f6OObdwPvOEt7vO88jgifd+lC7SFltrpLyKv72/kYm/uEDfvfml2Skunh+5jheu30iU4b2ICoqMOPtk4f0YN3uMra1oz8Ctu07zIodB7lryslMGdqj6R18GNq7M+Mzknn2s+1U19QGsIfh99LyfPq7XYxsIBnlqjF9Ka2o5r0v2/acn6qaWm6fu4L3vtzLby8dwrTxaeHuUpOCFnyc9X7212tbrKrVztMleJZpaI5LgWedx88SQWV+MlJdlFVUU3zIikPUUVX+8fFmTvvD+/z53Q2M7teVl2+bwL9vmcA3BroDfpO3bjz/nTWBH3qrqKrh9rkrWLXzYMCP3RprC0sBODUzpdXHmnl6BrsOHmHxCTT5ckvRIZZvP8DVWX0b/Hmb0D+lzc/5qaqp5QfzVrJ47R5+dfFgrm9gzlZbFM57PjOARV7PM0RkpYh8LCLf8LFPd1UtBHC+d/N1cBG5RUSyRSS7qCj8K0B8VWDUkg7qPP7RZv6waB0T+qey6I5v8MyNYxmTlhy09+ubnMiw3p1ZFIT7PovyCnkjt5C/vLch4MdujTUFpcRGCyd1b32SxTmndKdfcuIJVe16/vJ8oqOEK3xk/kVHCVeM7sMnG4rY0waLrFbX1PKjF3NYlLebX1x4CjdOzGh6pzYiLMFHRO4FqoE5TlMh0E9VRwE/BuaKSKtyklX1KVXNUtUst9vdug4HwFcFRtvPkE9jnvt8Gw+9s57LR/XmqevHhCwFfcrQHuTsPEhhyZGAHnfe0p0AfLShiO372s5nvLaglAHdkoiLaf1/9ego4cbT0snefqDVV3jlldX88tU83g/jcFZNrfLKil2ceZKbbp18TyS9ckwfahUWrGxbZYZqapWfvLSKN3IL+fkFJ3PzNzLD3aVmCXnwEZHpwEXANHVugKjqUVXd5zxeDmwGTmpg9z0i0tM5Tk9gb2h63Xq9unQgLibKrnyABSvzuW/hGs4d3J2HrhoesHs6/qgbelu8JnC/9DbuKWPZtv3MmJhBtAgvtKFK0GsKShnSK3CB/eqsPnSMj+Ffn7b86ufw0Wpu+tcXPL9kO7c8v5zXVoVnhZb/bixid2nF1xIN6vtqzk9+m7lnW1Or3PnSKhbmFPCzKYO45YzImwgb0uAjIlOAu4BLVLXcq90tItHO40xgILClgUO8Bkx3Hk8HFga3x4ETHSVkpLjafbr1O2t289OXcjmtfwqPTh1FTCsy2FpiQLeODOzWMaAp13OX7SA2WvjeWf2ZPLQHL36xkyOV4c+O2ltWQfGhowwO4FVlUkIs38rqyxu5hS0ahjp0tJob/7WM7O0H+MMVwxjTrys//PdKXg7DPZWXlufTNTHWr3kwV43pw6a9h1iVH5jVXXPzD/LX9zby8YYiyiqal75eW6vc9XIur6zcxU/PO4nvThoQkD6FWjBTrecBnwODRCRfRGYCjwFJeBal806pPgPIFZFVwHzgVlXd7xznaRHJcrb7A3CuiGwEznWeR4xMt6tdD7t9uqmY789dybDenfnnDVlhq3M3ZWgPlm7dx75DX6//1lwVVTW8smIXk4f0IKVjPNMnpFNaUd0mKkGvKfAkGwTyygfgxtPSqVHl+c+bd4VXWlHFDc8sZcWOg/zt2lFcO64fs2eMZUL/FH46fxXzloUujftgeSXvrtnDpSN7+zUkWTfnZ/7yna1+7wOHK7n52WweeW8D02ctY8SvF3P+X//LfQvzWJizi4KDvoeEa2uVny9Yzfzl+fzwmwO5/eyBre5PuAStwoGqTm2g+Rkf274MvOzjtZu9Hu8DzglIB8Mg0+1i8do9VNXUtmrOSiRaseMA//dcNpluF7NvGosrPnzFNSYP6cGjH2zivS/3+Cyq6a+3VhdScqSK68Z7jjM2vSsn90ji2c+3c83YhjOoQmWtE3xOCXDw6ZeSyLmndGfO0u3cfvYAv/6IKDnimei6ZlcJf79uFFOGeuYbJcbF8Mz0sdz6wnLueWU1ldW1TD8tPaD9bcjrqwqorKltsrZdHe85P7+4cHCL/3BSVX7xah4Hyit56dYJHK2qJXv7frK3HeDl5fk85wT0Xp0TyEpPJiu9K1lpyQzqkYQAv1iYx7+/2Mn3zx7AHedEbuABK68TUpmpHampVXbsLz+WgNAefFlYyo2zluFOiue5mePokui7BlsoDOnViT5dO/B23u5WB5+5S3eQkepigpPKLCLcMCGdny9YzfLtB8hKD172XlPWFpTSLzmRTkEoKjnj9AwWr93DgpW7mNrEZMaD5ZVc/8wy1u0u5fFpozlvyPHzjRJio3ny+jHcPncl97+2hsrq2mOL2gXLS8vzOaVnJ4b27uz3PleN6curOQW89+UeLhrevCoRdV7N2cWbqwv52ZRBjHV+Nk4f6CnYWl1Ty7rdZWRv288X2w+wdOu+Y/fDkuJj6JOcyJeFpXx3Un9+fO5JEVdvrr729ed3mH2Vbt1+ht62FR/m+meWkRgXwwszx9MtKfzl6UWE84f24NNN+yht5ni7tw17ysjefoCp446/wrlsVC+SEmJ4tpnDUoG2trA0oPd7vI3PSGZwz07M+t/WRm/CHzhcyXX/XMr63WU8ef2YrwWeOvEx0Tw+bTQXDuvJA299yWMfbAxKvwHW7y4jN7+kyUSD+lo75yf/QDn3vbqGseld+U4DCQIx0VEM7d2ZGydm8PfrRrPknnP478/O4i/XjOSSkb2IiRJ+fO5J3Dn5xCiOa8EnhDJTnXTrEGe8VVbXMmP2F9z50qqQlgkpOHiEaU8vpVaVF24eR9/kxJC9d1OmDO1BZU0tH65recLk3KU7iIuO4qoxx9fPSoyL4VtZfVm0upC9YZobcuhoNVuLDwf8fk8dEWHm6Rls3HuI/20qbnCbfYeOMvWfS9hUdIinbhjD2Sc3fmM/NjqKv147kstH9eZPizfw8OL1Qckueyl7J7HR4ldVb2+tmfNTW6v85D+rUODP3xpJtB8ZniJC3+RELhvVmwcuH8br3z+dH5wz8IQIPGDBJ6Q6J8aS4ooL+ZXPw4vX88G6vby0PJ9vP72U/YeDX2Vh36GjfPuZpZQeqeK5GeMY0K1tVZIe1bcr3ZLiW5z15kk0yGfy0B4kN7CUw/WnplFdq8wN4U10b+ucygaDgxR8AC4a0ZPUjvENTjotKvMEnq3Fh5k1fSyTBvmcD36cmOgo/nT1CK7J6sujH2ziD4vWBTQAVdXU8mrOLs45uXuDn1tTWjrn5+n/bWHp1v3cf/HgNvVHWDhZ8AkxT8Zb6K58Plq/lyc/2V1b+ssAABptSURBVMK08f14dOoocneVcMXjn7I1iFl3dVWUCw4e4ZkbxzZrXD1UoqKEyUN68NH6ohalRb+ZW0hpRTXX+bjfkZ7q4syT3MxduoOqMNRC+yrTLXj/9vEx0Vx/ahofri9is9fV/N7SCq596nN27j/Cv24ae+yehr+io4TfXzGM609N48lPtvDr19cGLAB9tL6I4kOVfica1NeSOT9fFpbyp3c2MHlI9xa/74nIgk+IeQqMhubKZ29ZBT99aRWDuifxy4sGc/GIXsy9eTwlR6q44vFP+WLb/qYP0kxHKmuYOfsLNuwp4x/fHsO4jPDdcG/KlKE9OFJVwycbm19+ae6yHWSmujg10/f5TT8tjb1lR4NSS64pawtKSXbF0b1TfFDfZ9qp/YiLjmL2p9sA2F1SwbVPLaGwpILZN43ltP4tW/00Kkr4zaVDmHl6BrM/28bPF+RRW9v6APRS9k5SO8YzaVDLq540Z85PRVUNP3oxh04dYvn9FcNPmCGzQLDgE2KZbhf7DldSUh7cdVFqa5Ufv7iKQ0erefS6UcdSQ7PSk1nw3Yl0TYxj2j+XBnR2eWV1Lbe+sJzl2w/wl2tG+T3UEi7jMpLpkhjb7KG39bvLWL79AFPH9Wv0l8mZJ3WjX3Iiz30W+sSDNYUlDOnVKei/7FI7xnPpyF7MX57Put2lXPvU5+wpreC5GeMY38pipiLCLy48he9O6s+8ZTu4c34uNa0IQMWHjvLBur1cMbp3qyY3N2fOz8OL17NudxkPXTW8RcN8JzILPiGW6aRYbw7y0NuTn2zhf5uKuf/iIV8rKpme6uLl205jZN8u/GDeSv7+4aZWDWtU1dTyn+ydnPfIx55FrK4YxoXDW7ZuTCjFRkdx7indee/LPVRW+z80Nm+ZJ9HgyiaGUKKjhOtPTWPZtv3H5tyEQlVNLRt2Hwpaplt9N03M4EhVDZc8+in7DlXy/M3jA5ZiLiLcOXkQPz73JF5ekc8PX8xpcfWIV1fuorpWm53lVp+/6/x8trmYp/+3lW+f2o+zTm7bf4iFgwWfEAtFuvWKHQf40+L1XDisJ9eObXglw66uOJ6/eRyXjuzFQ++s5+6XVzf73sTR6hpeWLKdSQ99xM/m5+KKj+GZ6VmtnjsTSlOG9qCsoprPt+zza/sjlTW8vCKfKT4SDeq7OqsPCbFRPL9kWyt76r9New9RWVMb1GQDb4N7deIbA1NJiI3ihZvHMzrAS06LCD84ZyB3n38yr68q4NxHPm52QVJVZf7yfEb07cLAAFT4bmqdn5IjVfz0P6tIT3Hx8wtOafX7nYhskmmI9UtOJCZK2BqkK5+SI1X8YN5KenRK4P9dMazRYZf4mGj+cs1I0pIT+dsHm9h18AiPf3t0k5MSj1R6lrl+8pPN7Ck9yqh+XfjdZUOZNCjwa/AE28QBqbjionk7bzdnntT0fYA3VxdSVlF9rKJBU7okxnHpiN68urKAu6ecQufEwE/4rG9tkMrqNOaJb4+hqrqWrkEcWrr1zP6M7NuFX76ax8xnszlvcHfuv2QIvbt0aHLfvF2lrNtdxu8uGxqQvnjP+WlowumvXlvDnrKjvHzbaSTG2a/ZhtiVT4jFRkfRLzkxKFc+qsrPX1lNYUkFj143is4dmv5FJyL8+LxB/PGq4SzZso+rnviM/APlDW576Gg1T3y0mdMf/IDfvLGW9BQXc24ezyu3ncZZJ3eLuMADntn1Z53cjXfX7vbrfsLcpdvJdLsY34xEiusnpHGkqoaXAlAXzB9rCkpJiI0iIzV0VTQ6xscENfDUOTUzhbfu+AZ3n38y/91YzDcf/pgnPtrc5LDp/OU7iYuJ4uIWViaoz3vOT/25XG/kFrBg5S6+f/aABldHNR4WfMIg0x2c6tb//mInb64u5CfnndTsoY9vZfXl2RnjKCyp4PLHPyM3/6v1WkrKq/jre55lrh98ex1DenfmP9+ZwIvfmcDEAakRGXS8nT+0J8WHKsluIvtv3e5SVuw4yHVNJBrUN7R3Z7LSuvL8ku0BydhqytrCEk7u0cmviYyRKDY6ilvP7M97PzmTM05K5cG313Hh3/7LEh9Dp0era1i4qoDJQ3oE9MqzoTk/u0squHdBHiP6duH2syKz2nSoWPAJg4xUF1v3HW5V5k59G/eU8evX13D6gFRubeHaHhMHpPLKbacRFx3FNU8uYcHKfB56Zx2nP/gBj7y3gbHpybz6vYk8N2Ncm06hbq5Jg9zExUTxdhMp0fOcigZXjm7+DesbTktn+75yPm5BWndzqCprA7yGT1vVu0sHnrw+i2emZ3GkqoZrn1rCj1/Moajs+Grl763dy8HyqlYnGtSXkepijNecn9pa5c75q6isruWRb40I+XIhkcb+dcIg092RyuraRkunN0dFVQ23z12JKy6GP39rRKsWZxvYPYkF3zuNk7p35EcvruLxjzZzxklu3vzB6Tw9PeuEHEZwxcdwxkA37+Tt9pn1d6SyhldW7uL8YT1aNLw0ZUgP3EnxzV6GoLnyDxyhtKI6ZMkGbcE5p3Tn3R+dyffPHsDruQWc/fBHPP/5tmN/3L20fCc9OycwcUDL5hw15qoxfdi49xC5+SU89/k2/ruxmHsvPOVYVqvxzYJPGGSmejLeNgeoxtvv3lzL+j1lPPytEY0uB+yvbkkJ/PuWCdx30WAW//AM/j5tdFBnyrcF5w/tQUFJBbk+Jg6+nlvgSTRoooKzL3ExUVw3rh8frt8b1GW2Q1HZoC3qEBfNT84bxNs/PIPhfTrzy4VruPzxT3lv7R4+2VDElaP7BGUY8sLhPYmPieLhdzfw+0XrOGuQm2l+JqO0dxZ8wqDur6JA3PdZtLqQF5bs4JYzMgM6qbNDXDQzTs8ISFpqJDjnlG7ERInPobd5y3bQ3+1q1XDjdeP7BX2Z7bWFpUQJDGonn1t9/d0deWHmeP42dRSFJRXc/Fw2tUrQytp0SohlytAefLKhCFd8DA9eZVUM/GXBJwxSO8aRlBDT6hpv+QfKuevlXEb06cxPzxsUoN61T10S45jQP4W3Gxh6+7KwlJU7DjZZ0aAp3TslBH2Z7bUFJfR3d6RDXHhWiW0LRIRLRvTi/Z+cyc2nZzDz9AzSndGGYLhuXL9j9ejawpIhkSKYy2jPEpG9IpLn1faQiKwTkVwRWSAiXZz2c0VkuYisdr6f7eOYvxKRXc4S3DkickGw+h9MIkKmu3U13qprarnj3znUKvxt6ii/lgI2jZs8pAdbiw+zce/xfxTMXbqDuJiWJRrUV7fM9murgrPM9pqC0nZ1v6cxnRJi+cVFg/nlRYOD+j7jM1PIvf88JvtYq8g0LJi/sWYDU+q1vQsMVdXhwAbgHqe9GLhYVYcB04HnGznuI6o60vl6K8B9Dpn+qa1Lt/7LextZvv0AD1w+lLSU4P1V156cN6Q7IrBo9VdDb+WV1by6chcXDG1ZokF9x5bZ/mx7wNeq2X+4ksKSinaR6dbWhHNZ+EgVtOCjqp8A++u1LVbVaufpEqCP075SVesqXK4BEkQkuOV4wyzT7WJ3aQWHj1Y3vXE9/9tYzN8/2sTVY/pw6cjmLYhlfOuWlEBWWtfj7vu8saqQsqPVXDc+LSDvISJMPy2dtYWlLN9+ICDHrFNX2WBwz/aVbGAiUzjHamYAixpovxJYqapHG3gN4HZn2G6WiPicSSkit4hItohkFxUFd25FS9QlHTR3XZ11u0u5bc5yBrg78utLhwSja+3a5CE9+LKw9FhG2pxlOxjQrSNj0wNXr+zSkcFZZnttoSdTz4bdTCQIS/ARkXuBamBOvfYhwIPAd3zs+gTQHxgJFAIP+3oPVX1KVbNUNcvtbvnaHcFyrMBoM4LProNHmD5rGYlx0cyeMc5qRgXBlKGecft31uxmTUEJq3a2PtGgvmAts72moJSenROsdL+JCCEPPiIyHbgImKZeg94i0gdYANygqpsb2ldV96hqjarWAv8ExoWiz8GQnuJCBLb4OdfnYHkl02cto7yyhmdnjPOrmKJpvj5dExnWuzOL8nZ7lk6IieLK0YEf2qxbZnvessDVe2svlQ3MiSGkwUdEpgB3AZeoarlXexfgTeAeVf20kf29F4m5HMjztW1blxAbTa/OHfxKOqioqmHms9ns2FfOU9dncXIP+wUTTFOG9mDljoO8smIXFw7rSZfEwF9JpKe6mDTIzZyl2wOyzPaRyho2F4VuDR9jWiuYqdbzgM+BQSKSLyIzgceAJOBdJ1X6H87mtwMDgF96pVF3c47ztIhkOdv90UnHzgXOAn4UrP6HQqbb1eRcn5pa5QfzVrJixwEeuWYkE/q3bnVI07S6lNnyyhq/l05oiWnjPctsf7Kh9fck1+8po1ZhcDurbGAiV9BuGqjq1Aaan/Gx7e+A3/l47Wavx9cHpndtQ393R17K3omqNnhPQVW5b2Eei9fu4VcXD46I1UFPBAO6dWRQ9yQUJSstsAujeZs0yE2yK45XVu7inFO6t+pYawo8yQY27GYihd2xDqNMt4vDlTXsKT1Kj85fnxn92AebmLN0B7ee2Z8bJ2aEoYft19PTsxAhqKVSYqOjuGREL+Yu20HJkSq/1l/yZW1BKUkJMfTpavcCTWSwafFhlJlaV+Pt60NvL36xg4ff3cAVo3pz1xQrnRNqfZMT6dM1Mejvc/mo3lRW17JodWGrjrOmoJTBPTtZXTETMSz4hFFduvXmeunWH6zbw88X5HHGSW4rVHiCG96nM5luF6+sbHm5nZpaZd3u0nZXydpENgs+YdSjUwIdYqOPu/JZueMA352zgsE9O/HEtNHE2oJUJzQR4YpRvVm2dT879ze8fHlTthYfoqKq1iaXmohiv9nCKCpKPKuaOlc+m4sOMWP2F3TvlMCsG8davah2oq5E0sKcll39fLWGjwUfEzks+IRZpttTYHRvaQXTZy0jSoRnbxqHO+mELm1nvPRNTmR8RjKvrNjVomKjawtKiYuOYkA3Wz3TRA4LPmGWmeoi/0A50//1BfsPVzLrxrFBXXvEtE1XjO7NluLDrPKxkmpj1haWclKPjjZEayKK/bSGWaa7I7UKG/eU8fi00Yzo2yXcXTJhcP4wz3LMC1bkN2s/VWVNQSlDrJK1iTAWfMJsZN8uuOKiefDK4QFdBttElk4JsXxzcHdezy2kstr/cjt7So+y/3ClJRuYiGPBJ8zSU13k/moyVwZpjXkTOa4c3Zv9hyv5uBnldqyygYlUFnzagOgom8dj4BsD3aS44liw0v+ht7UFpYjAyVZQ1EQYCz7GtBGx0VFcPKIX7325l5IjVX7ts6aglPQUFx0tLd9EGAs+xrQhV4z2lNt5y89yO2sLS20ZBRORLPgY04YM692Z/m4XC1Y0PeG0tKKKHfvLLdnARCQLPsa0ISLCFaP7sGxb0+V2vnQqG1jwMZHIgo8xbcxlozzldhY0UWzUyuqYSGbBx5g2pneXDpyamcyClY2X21lbWEpqx3i6JX19LShj2rqgBh8RmSUie0Ukz6vtIRFZJyK5IrJARLp4vXaPiGwSkfUiMtnHMTNEZKmIbBSRF0UkLpjnYEw4XDGqD1uLD5Oz86DPbdYUlNpVj4lYwb7ymQ1Mqdf2LjBUVYcDG4B7AERkMHAtMMTZ53ERiW7gmA8Cj6jqQOAAMDM4XTcmfM4f1oP4mChe8ZF4UFldy6a9ZXa/x0SsoAYfVf0E2F+vbbGqVjtPlwB1U/svBf6tqkdVdSuwCRjnva94VlU7G5jvND0LXBak7hsTNkkJsZw3pAev5xY0WG5nw54yqmrUrnxMxAr3PZ8ZwCLncW9gp9dr+U6btxTgoFfwamgbAETkFhHJFpHsoiL/y5UY01ZcMao3B8ur+Gj93q+9trbQyXSzOT4mQoUt+IjIvUA1MKeuqYHN6t9t9WcbT6PqU6qapapZbre75R01Jky+MTCV1I5xDWa9rS0oJTEumvQUW37DRKawBB8RmQ5cBEzTr9J58oG+Xpv1AQrq7VoMdBGRmEa2MeaEEOOU23n/y72UlB9fbmdtQSmn9OxElNUFNBEq5MFHRKYAdwGXqKr3LLrXgGtFJF5EMoCBwDLvfZ1A9SFwldM0HVgY/F4bEx5Xju5DZU0tb6z+6m+s2lplbaFlupnIFuxU63nA58AgEckXkZnAY0AS8K6I5IjIPwBUdQ3wH2At8DbwPVWtcY7zloj0cg57F/BjEdmE5x7QM8E8B2PCaUivTgzs1vG4cjs7D5Rz6Gi13e8xES2opXBVdWoDzT6Dhao+ADzQQPsFXo+3UC8LzpgTlYhw+eje/PHt9ezYV06/lESvyga2eqmJXOHOdjPGNOGykb0R+arcztqCUqKjhIHdO4a5Z8a0nAUfY9q4Xl06cGpGCq+szEdVWVNQwsBuHUmIbWgOtjGRwYKPMRHgitG92b6vnBU7DtoaPuaEYMHHmAhw/rCeJMRG8fR/t7Cn9KiV1TERz4KPMRGgY3wM5w3uwaK83YCt4WMinwUfYyLE5aO/qiQ1pKdlupnIFtRUa2NM4HxjQCqpHeOJj4mic2JsuLtjTKtY8DEmQsRER/HbS4dQWfP1KtfGRBoLPsZEkPOH9Qx3F4wJCLvnY4wxJuQs+BhjjAk5Cz7GGGNCzoKPMcaYkLPgY4wxJuQs+BhjjAk5Cz7GGGNCzoKPMcaYkBNVDXcfgk5EioDtLdw9FSgOYHciTXs+fzv39qs9n7/3uaepqjsYb9Iugk9riEi2qmaFux/h0p7P3869fZ47tO/zD9W527CbMcaYkLPgY4wxJuQs+DTtqXB3IMza8/nbubdf7fn8Q3Luds/HGGNMyNmVjzHGmJCz4GOMMSbkLPg0QkSmiMh6EdkkIneHuz+tISLbRGS1iOSISLbTliwi74rIRud7V6ddRORvznnnishor+NMd7bfKCLTvdrHOMff5OwroT/LY32ZJSJ7RSTPqy3o5+rrPULNx/n/SkR2OZ9/johc4PXaPc65rBeRyV7tDf78i0iGiCx1zvNFEYlz2uOd55uc19NDc8ZfEZG+IvKhiHwpImtE5A6n/YT//Bs597b52auqfTXwBUQDm4FMIA5YBQwOd79acT7bgNR6bX8E7nYe3w086Dy+AFgECHAqsNRpTwa2ON+7Oo+7Oq8tAyY4+ywCzg/juZ4BjAbyQnmuvt6jjZz/r4CfNrDtYOdnOx7IcH7moxv7+Qf+A1zrPP4HcJvz+LvAP5zH1wIvhuHcewKjncdJwAbnHE/4z7+Rc2+Tn31YfjlEwpfzw/WO1/N7gHvC3a9WnM82vh581gM9ncc9gfXO4yeBqfW3A6YCT3q1P+m09QTWebUft12Yzjed43/5Bv1cfb1HGzl/X7+Ajvu5Bt5xfvYb/Pl3fuEWAzFO+7Ht6vZ1Hsc420mYfw4WAue2t8+/3rm3yc/eht186w3s9Hqe77RFKgUWi8hyEbnFaeuuqoUAzvduTruvc2+sPb+B9rYkFOfq6z3aitudoaVZXkNCzT3/FOCgqlbXaz/uWM7rJc72YeEM/YwCltLOPv965w5t8LO34ONbQ/csIjkvfaKqjgbOB74nImc0sq2vc29ueyRoL+f6BNAfGAkUAg877YE8/zbzbyMiHYGXgR+qamljmzbQFtGffwPn3iY/ews+vuUDfb2e9wEKwtSXVlPVAuf7XmABMA7YIyI9AZzve53NfZ17Y+19GmhvS0Jxrr7eI+xUdY+q1qhqLfBPPJ8/NP/8i4EuIhJTr/24Yzmvdwb2B/5sGicisXh++c5R1Vec5nbx+Td07m31s7fg49sXwEAnuyMOz02018LcpxYREZeIJNU9Bs4D8vCcT10Wz3Q8Y8Q47Tc4mUCnAiXOMMI7wHki0tW5dD8Pz5hvIVAmIqc6mT83eB2rrQjFufp6j7Cr+6XouBzP5w+ePl/rZCtlAAPx3FBv8OdfPYP6HwJXOfvX/7esO/+rgA+c7UPG+UyeAb5U1T97vXTCf/6+zr3NfvbhvCHW1r/wZMJswJP5cW+4+9OK88jEk7GyClhTdy54xmTfBzY635OddgH+7pz3aiDL61gzgE3O101e7VnOD/Vm4DHCeKMZmIdneKEKz19kM0Nxrr7eo42c//PO+eU6vyh6em1/r3Mu6/HKUvT18+/8PC1z/l1eAuKd9gTn+Sbn9cwwnPvpeIZ7coEc5+uC9vD5N3LubfKzt/I6xhhjQs6G3YwxxoScBR9jjDEhZ8HHGGNMyFnwMcYYE3IWfIwxxoScBR9j/CAinznf00XkugAf++cNvZcxJzJLtTamGURkEp4ijRc1Y59oVa1p5PVDqtoxEP0zJlLYlY8xfhCRQ87DPwDfcNZF+ZGIRIvIQyLyhVO48TvO9pOctVXm4pngh4i86hR2XVNX3FVE/gB0cI43x/u9nFn3D4lInnjWj7nG69gfich8EVknInOc2e2IyB9EZK3Tlz+F8t/ImOaIaXoTY4yXu/G68nGCSImqjhWReOBTEVnsbDsOGKqqW53nM1R1v4h0AL4QkZdV9W4RuV1VRzbwXlfgKQY5Akh19vnEeW0UMARPba1PgYkishZP+ZSTVVVFpEvAz96YALErH2Na5zw8tcFy8JSvT8FTIwtgmVfgAfiBiKwCluApwjiQxp0OzFNPUcg9wMfAWK9j56unWGQOnvV7SoEK4GkRuQIob/XZGRMkFnyMaR0Bvq+qI52vDFWtu/I5fGwjz72ib+JZcGsEsBJPPaymju3LUa/HNXgW+KrGc7X1MnAZ8HazzsSYELLgY0zzlOFZorjOO8BtTil7ROQkp3J4fZ2BA6paLiIn41myuU5V3f71fAJc49xXcuNZHnuZr46JZx2Xzqr6FvBDPEN2xrRJds/HmObJBaqd4bPZwF/xDHmtcG76F+G56qjvbeBWEcnFU0F4iddrTwG5IrJCVad5tS/As1TxKjzVin+mqrud4NWQJGChiCTguWr6UctO0Zjgs1RrY4wxIWfDbsYYY0LOgo8xxpiQs+BjjDEm5Cz4GGOMCTkLPsYYY0LOgo8xxpiQs+BjjDEm5P4/CDucVJpvA5cAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "iterations = np.arange(len(cost_list)) * 10000\n",
    "plt.plot(iterations,cost_list)\n",
    "plt.xlabel(\"iterations\")\n",
    "plt.ylabel(\"avg cost\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The loss above really fluctuates (noisy) due to random choices of x and y at each iteration (epoch)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you would like to learn about Pytorch, feel free to check out its documentation and more tutorials [here](https://pytorch.org/). We will be discussing more Pytorch in depth in later weeks of the quarter when we talk about neural networks, so stay tuned and excited!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}