intro_deep_learning.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "44be838e-0fcd-427b-8057-6c71ec1055b6",
      "metadata": {
        "id": "44be838e-0fcd-427b-8057-6c71ec1055b6"
      },
      "source": [
        "# Recitation 11\n",
        "\n",
        "This week we'll do something a bit different. This notebook is meant to be a review of numpy, however most people learn how to use numpy for a different reason, machine learning, it wouldn't do justice to the library if we skip that now would it?\n",
        "\n",
        "<img src=\"https://media.tenor.com/Vz-Vjh01PTYAAAAM/is-it-ai-ai.gif\" alt=\"meme\" width=\"200\"/>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6aeafdc1-f0bd-437f-a385-4106d2ab7341",
      "metadata": {
        "id": "6aeafdc1-f0bd-437f-a385-4106d2ab7341",
        "tags": []
      },
      "source": [
        "## A Brief Primer\n",
        "\n",
        "What we'll be doing today is an extremely simplified version of a \"fully connected neural network.\" Neural networks are pretty self explanatory (they're networks of neurons), however what is a neuron exactly? Well that's simple too, a neuron is just a weight that you multiply to some vector, in other words it's a simple function. Some very smart people in the 90s figured out that by layering this neurons on top of one another you can learn really precise details about your datasets and even learn \"non-linear\" featured, thus this method of learning was dubbed \"deep learning\". Alas that little intro doesn't do the subject nearly enough justice, so I'd recommend watching some videos on youtube so that you can learn from people way smarter than myself."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Mnq5l17bfWWj",
      "metadata": {
        "id": "Mnq5l17bfWWj"
      },
      "source": [
        "## Data Setup\n",
        "\n",
        "We'll start by using the famous mnist dataset that we'll convert into binary values 0 or 1 to make our life easier. These libraries come pre installed on colab, however please install them on your machine if you need them. The Binarized MNIST dataset here are digits from 0-9 in a 20x16 pixel grid, instructions on how to visualize them is provided below.\n",
        "\n",
        "For now use x and Y arrays without splitting into test/train where x are the actual features and y are the labels. If you are unfamiliar with test/train splits I'd recommend looking that up and being comfortable with the subject."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "id": "kpPrtEmms0Es",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kpPrtEmms0Es",
        "outputId": "a093ba87-5f07-4d51-a430-90ca52896448"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "100  519k  100  519k    0     0  1359k      0 --:--:-- --:--:-- --:--:-- 1357k\n"
          ]
        }
      ],
      "source": [
        "!curl -L  https://cs.nyu.edu/~roweis/data/binaryalphadigs.mat --output binaryalphadigs.mat"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 73,
      "id": "8UYQE4W1smth",
      "metadata": {
        "id": "8UYQE4W1smth"
      },
      "outputs": [],
      "source": [
        "import scipy.io\n",
        "import numpy as np\n",
        "\n",
        "x = []\n",
        "Y = []\n",
        "\n",
        "data = scipy.io.loadmat('binaryalphadigs.mat')\n",
        "for i in range(10):\n",
        "      label = data['classlabels'][0][i].item()\n",
        "      for j in range(len(data['dat'][i])):\n",
        "        image = data['dat'][i][j].reshape(20, 16, 1)\n",
        "        image = np.array(image)\n",
        "        reshaped_image = np.reshape(image, (20, 16))\n",
        "        x.append(reshaped_image)\n",
        "        Y.append(int(label))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 74,
      "id": "rbpCA4EnvGZt",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 448
        },
        "id": "rbpCA4EnvGZt",
        "outputId": "3c5bc4b9-edb6-4ffb-dcd5-5ea5c71c8863"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "9\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAGdCAYAAAA7TzlCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAiEUlEQVR4nO3df2xV9f3H8dct0Ftj6K0K9PZqKT8UUMRGme2KMr8bHaUjCG4TbJwUf6AhJdEwDbKIRV1Wf23ZhAaXKVTDBDURyNThoPJjSCtKaYY/RiirLQRuCUTubctom97P94+Nu91xb+ml97afe3k+kk/COefzOX3fT+99cTj3cI7DGGMEALBWykAXAADoGUENAJYjqAHAcgQ1AFiOoAYAyxHUAGA5ghoALEdQA4DlBg90AbEQCAR07NgxDR06VA6HY6DLAYAgY4xaW1vl8XiUknJxx8ZJEdTHjh1Tdnb2QJcBABEdOXJE11xzzUWNTYqgHjp06ECXgCTj8/kGuoTzuFyugS4BfdCXnEqKoOZ0B2ItPT19oEtAkulLTvFlIgBYjqAGAMvFLagrKys1atQopaWlKT8/X3v37u2x/7vvvqsJEyYoLS1NkyZN0ocffhiv0gAgsZg42LBhg0lNTTVr1qwxX375pVm4cKHJyMgwLS0tYft/8sknZtCgQebFF180X331lXnqqafMkCFDzIEDB3r183w+n5FEo8Ws2Wig54TWt+bz+S7+dx/D91FQXl6eKSsrCy53d3cbj8djKioqwvafO3eumTlzZsi6/Px888gjj/Tq5xHUtFg3Gw30nND61voS1DE/9dHZ2al9+/apsLAwuC4lJUWFhYWqqakJO6ampiakvyQVFRVF7N/R0SG/3x/SACBZxTyoT548qe7ubmVmZoasz8zMlNfrDTvG6/VG1b+iokIulyvY+M8uAJJZQl71sWzZMvl8vmA7cuTIQJcEAHET8//wMmzYMA0aNEgtLS0h61taWuR2u8OOcbvdUfV3Op1yOp2xKRgALBfzI+rU1FRNnjxZ1dXVwXWBQEDV1dUqKCgIO6agoCCkvyRt3bo1Yn8AuKTE8EvpoA0bNhin02mqqqrMV199ZR5++GGTkZFhvF6vMcaY++67zzz55JPB/p988okZPHiwefnll83XX39tysvLuTyPNqDNRgM9J7S+NesuzzPGmJUrV5qRI0ea1NRUk5eXZ2pra4Pb7rjjDlNaWhrS/5133jHjxo0zqampZuLEieaDDz7o9c8iqGmxbjYa6Dmh9a31Jagd/34DJDS/38+dxRBTNn4suPlYYvP5fBd9s6+EvOoDAC4lSXGbU/Q/G484k10yzzn/WugZR9QAYDmCGgAsR1ADgOUIagCwHEENAJYjqAHAcgQ1AFiOoAYAyxHUAGA5ghoALEdQA4DlCGoAsBxBDQCWI6gBwHIENQBYjqAGAMsR1ABgOYIaACzHo7guMcn8OCcgWXFEDQCWI6gBwHIENQBYjqAGAMsR1ABgOYIaACxHUAOA5QhqALAcQQ0AliOoAcByBDUAWI6gBgDLEdQAYDmCGgAsF/Ogrqio0K233qqhQ4dqxIgRmjNnjg4ePNjjmKqqKjkcjpCWlpYW69IAICHFPKh37typsrIy1dbWauvWrerq6tL06dPV3t7e47j09HQdP3482JqammJdGgAkpJg/OGDLli0hy1VVVRoxYoT27dun733vexHHORwOud3uWJcDAAkv7k948fl8kqQrr7yyx35tbW3KyclRIBDQLbfcol/96leaOHFi2L4dHR3q6OgILvv9/tgVbCGeytI7DodjoEuIq2R+HyTza/P7/XK5XH3aR1y/TAwEAnrsscd022236cYbb4zYb/z48VqzZo02b96sdevWKRAIaMqUKTp69GjY/hUVFXK5XMGWnZ0dr5cAAAPOYeL4V9miRYv05z//Wbt379Y111zT63FdXV26/vrrVVJSoueee+687eGOqJM5rJP5aCOWOKKGjc4dUft8PqWnp1/UPuJ26mPx4sV6//33tWvXrqhCWpKGDBmim2++WQ0NDWG3O51OOZ3OWJQJANaL+akPY4wWL16sjRs36uOPP9bo0aOj3kd3d7cOHDigrKysWJcHAAkn5kfUZWVleuutt7R582YNHTpUXq9XkuRyuXTZZZdJkubPn6+rr75aFRUVkqRnn31W3/3ud3Xttdfq9OnTeumll9TU1KSHHnoo1uUBQMKJeVCvXr1akvR///d/IevXrl2rBQsWSJKam5uVkvKfg/lvv/1WCxculNfr1RVXXKHJkydrz549uuGGG2JdHgAknLh+mdhfYnH5i82S4FfUL/gyETaKxZeJ3OsDACxHUAOA5QhqALAcQQ0AliOoAcByBDUAWI6gBgDLEdQAYDmCGgAsR1ADgOUIagCwXNwfxZVouJ9C/0v2e3TEio3zxOelf3BEDQCWI6gBwHIENQBYjqAGAMsR1ABgOYIaACxHUAOA5QhqALAcQQ0AliOoAcByBDUAWI6gBgDLEdQAYDmCGgAsR1ADgOUIagCwHEENAJbjCS+XGBufEoLEFav3E0+K6RlH1ABgOYIaACxHUAOA5QhqALAcQQ0Alot5UK9YsUIOhyOkTZgwoccx7777riZMmKC0tDRNmjRJH374YazLAoCEFZcj6okTJ+r48ePBtnv37oh99+zZo5KSEj344IPav3+/5syZozlz5uiLL76IR2kAkHDiEtSDBw+W2+0OtmHDhkXs+7vf/U4zZszQE088oeuvv17PPfecbrnlFq1atSoepQFAwolLUB86dEgej0djxozRvffeq+bm5oh9a2pqVFhYGLKuqKhINTU1Ecd0dHTI7/eHNABIVjEP6vz8fFVVVWnLli1avXq1GhsbNXXqVLW2tobt7/V6lZmZGbIuMzNTXq834s+oqKiQy+UKtuzs7Ji+BgCwScyDuri4WHfffbduuukmFRUV6cMPP9Tp06f1zjvvxOxnLFu2TD6fL9iOHDkSs30DgG3ifq+PjIwMjRs3Tg0NDWG3u91utbS0hKxraWmR2+2OuE+n0ymn0xnTOgHAVnG/jrqtrU2HDx9WVlZW2O0FBQWqrq4OWbd161YVFBTEuzQASAgxD+rHH39cO3fu1DfffKM9e/borrvu0qBBg1RSUiJJmj9/vpYtWxbs/+ijj2rLli369a9/rb///e9asWKFPv/8cy1evDjWpQFAQor5qY+jR4+qpKREp06d0vDhw3X77bertrZWw4cPlyQ1NzcrJeU/fz9MmTJFb731lp566in94he/0HXXXadNmzbpxhtvjHVpAJCQHCYJbgTr9/vlcrlisq8kmI4ecT9q2CiZP3fn8snn8yk9Pf2i9sG9PgDAcgQ1AFiOoAYAyxHUAGA5ghoALEdQA4DlCGoAsBxBDQCWI6gBwHIENQBYjqAGAMsR1ABgOYIaACxHUAOA5QhqALAcQQ0AliOoAcByBDUAWI6gBgDLEdQAYLmYP4V8IPXl4ZE244G0sFUyP5TWJhxRA4DlCGoAsBxBDQCWI6gBwHIENQBYjqAGAMsR1ABgOYIaACxHUAOA5QhqALAcQQ0AliOoAcByBDUAWI6gBgDLxTyoR40aJYfDcV4rKysL27+qquq8vmlpabEuCwASVszvR/3ZZ5+pu7s7uPzFF1/ohz/8oe6+++6IY9LT03Xw4MHgMvdfBoD/iHlQDx8+PGT5+eef19ixY3XHHXdEHONwOOR2u2NdCgAkhbieo+7s7NS6dev0wAMP9HiU3NbWppycHGVnZ2v27Nn68ssv41kWACSUuAb1pk2bdPr0aS1YsCBin/Hjx2vNmjXavHmz1q1bp0AgoClTpujo0aMRx3R0dMjv94c0AEhWDhPHh54VFRUpNTVVf/rTn3o9pqurS9dff71KSkr03HPPhe2zYsUKPfPMM+et55mJQP/imYkX5vf75XK5+pRPcTuibmpq0rZt2/TQQw9FNW7IkCG6+eab1dDQELHPsmXL5PP5gu3IkSN9LRcArBW3oF67dq1GjBihmTNnRjWuu7tbBw4cUFZWVsQ+TqdT6enpIQ0AklVcgjoQCGjt2rUqLS3V4MGhF5bMnz9fy5YtCy4/++yz+stf/qJ//OMfqqur089+9jM1NTVFfSQOAMkq5pfnSdK2bdvU3NysBx544Lxtzc3NSkn5z98P3377rRYuXCiv16srrrhCkydP1p49e3TDDTfEozQASDhx/TKxv8TiZL3N+DIRtkqC+Ig7q79MBADEBkENAJYjqAHAcgQ1AFiOoAYAyxHUAGA5ghoALEdQA4DlCGoAsBxBDQCWI6gBwHJxuSkTYiuW91PgviHg/hyJhyNqALAcQQ0AliOoAcByBDUAWI6gBgDLEdQAYDmCGgAsR1ADgOUIagCwHEENAJYjqAHAcgQ1AFiOoAYAyxHUAGA5ghoALEdQA4DlCGoAsBxBDQCW41FcQILgEVq9k4yPm+OIGgAsR1ADgOUIagCwHEENAJYjqAHAclEH9a5duzRr1ix5PB45HA5t2rQpZLsxRk8//bSysrJ02WWXqbCwUIcOHbrgfisrKzVq1CilpaUpPz9fe/fujbY0AEhKUQd1e3u7cnNzVVlZGXb7iy++qFdeeUWvvvqqPv30U11++eUqKirS2bNnI+7z7bff1pIlS1ReXq66ujrl5uaqqKhIJ06ciLY8AEg+pg8kmY0bNwaXA4GAcbvd5qWXXgquO336tHE6nWb9+vUR95OXl2fKysqCy93d3cbj8ZiKiope1eHz+Ywk4/P5on8RlxhJtARt6J2B/j1Fan3Jp5ieo25sbJTX61VhYWFwncvlUn5+vmpqasKO6ezs1L59+0LGpKSkqLCwMOKYjo4O+f3+kAYAySqmQe31eiVJmZmZIeszMzOD2/7XyZMn1d3dHdWYiooKuVyuYMvOzo5B9QBgp4S86mPZsmXy+XzBduTIkYEuCQDiJqZB7Xa7JUktLS0h61taWoLb/tewYcM0aNCgqMY4nU6lp6eHNABIVjEN6tGjR8vtdqu6ujq4zu/369NPP1VBQUHYMampqZo8eXLImEAgoOrq6ohjAOBSEvXd89ra2tTQ0BBcbmxsVH19va688kqNHDlSjz32mH75y1/quuuu0+jRo7V8+XJ5PB7NmTMnOGbatGm66667tHjxYknSkiVLVFpaqu985zvKy8vTb3/7W7W3t+v+++/v+ysEgEQX7WUi27dvD3vpSWlpqTHmX5foLV++3GRmZhqn02mmTZtmDh48GLKPnJwcU15eHrJu5cqVZuTIkSY1NdXk5eWZ2traXtfE5Xm9F+53R0uMht4Z6N9TpNaXfHL8+4UlNL/fL5fLJZ/Px/nqC0jGe/VeKpLgo9ovbH2P9yWfEvKqDwC4lCTVE15cLlef95HsRy3J/vqAZMQRNQBYjqAGAMsR1ABgOYIaACxHUAOA5QhqALAcQQ0AliOoAcByBDUAWI6gBgDLEdQAYDmCGgAsR1ADgOUIagCwHEENAJYjqAHAcgQ1AFiOoAYAyyXVo7hiIZYPxuSxV4nLxgekJvP7ycb5tglH1ABgOYIaACxHUAOA5QhqALAcQQ0AliOoAcByBDUAWI6gBgDLEdQAYDmCGgAsR1ADgOUIagCwHEENAJYjqAHAclEH9a5duzRr1ix5PB45HA5t2rQpuK2rq0tLly7VpEmTdPnll8vj8Wj+/Pk6duxYj/tcsWKFHA5HSJswYULULwYAklHUQd3e3q7c3FxVVlaet+3MmTOqq6vT8uXLVVdXp/fee08HDx7UnXfeecH9Tpw4UcePHw+23bt3R1saACSlqB8cUFxcrOLi4rDbXC6Xtm7dGrJu1apVysvLU3Nzs0aOHBm5kMGD5Xa7oy0HAJJe3J/w4vP55HA4lJGR0WO/Q4cOyePxKC0tTQUFBaqoqIgY7B0dHero6Agu+/3+WJYcMzy1AkAsxPXLxLNnz2rp0qUqKSlRenp6xH75+fmqqqrSli1btHr1ajU2Nmrq1KlqbW0N27+iokIulyvYsrOz4/USAGDgmT6QZDZu3Bh2W2dnp5k1a5a5+eabjc/ni2q/3377rUlPTzevvfZa2O1nz541Pp8v2I4cOWIk0WhJ3ZLZQM9tf7Roc/C/xeXUR1dXl+bOnaumpiZ9/PHHPR5Nh5ORkaFx48apoaEh7Han0ymn0xmLUgHAejE/9XEupA8dOqRt27bpqquuinofbW1tOnz4sLKysmJdHgAknKiDuq2tTfX19aqvr5ckNTY2qr6+Xs3Nzerq6tJPf/pTff755/rjH/+o7u5ueb1eeb1edXZ2Bvcxbdo0rVq1Krj8+OOPa+fOnfrmm2+0Z88e3XXXXRo0aJBKSkr6/goBINFFe65k+/btYc+/lJaWmsbGxojnZ7Zv3x7cR05OjikvLw8uz5s3z2RlZZnU1FRz9dVXm3nz5pmGhoZe1+Tz+Qb8/BONFu+WzAZ6bvuj9eUctePfk5TQ/H6/XC7XQJcBxFUSfFQjuhQuZfX5fFF/X3cO9/oAAMsR1ABgOYIaACxHUAOA5QhqALAcQQ0AliOoAcByBDUAWI6gBgDLEdQAYDmCGgAsF/dHcQGXsmS+Pwf6D0fUAGA5ghoALEdQA4DlCGoAsBxBDQCWI6gBwHIENQBYjqAGAMsR1ABgOYIaACxHUAOA5QhqALAcQQ0AliOoAcByBDUAWI6gBgDLEdQAYDme8AKEwZNZesfhcAx0CZcEjqgBwHIENQBYjqAGAMsR1ABgOYIaACwXdVDv2rVLs2bNksfjkcPh0KZNm0K2L1iwQA6HI6TNmDHjgvutrKzUqFGjlJaWpvz8fO3duzfa0gAgKUUd1O3t7crNzVVlZWXEPjNmzNDx48eDbf369T3u8+2339aSJUtUXl6uuro65ebmqqioSCdOnIi2PABIPqYPJJmNGzeGrCstLTWzZ8+Oaj95eXmmrKwsuNzd3W08Ho+pqKjo1Xifz2ck0Wgxa+idgf49JVLz+XwXPc9xOUe9Y8cOjRgxQuPHj9eiRYt06tSpiH07Ozu1b98+FRYWBtelpKSosLBQNTU1Ycd0dHTI7/eHNABIVjEP6hkzZujNN99UdXW1XnjhBe3cuVPFxcXq7u4O2//kyZPq7u5WZmZmyPrMzEx5vd6wYyoqKuRyuYItOzs71i8DAKwR8/9Cfs899wT/PGnSJN10000aO3asduzYoWnTpsXkZyxbtkxLliwJLvv9fsIaQNKK++V5Y8aM0bBhw9TQ0BB2+7BhwzRo0CC1tLSErG9paZHb7Q47xul0Kj09PaQBQLKKe1AfPXpUp06dUlZWVtjtqampmjx5sqqrq4PrAoGAqqurVVBQEO/yAMB6UQd1W1ub6uvrVV9fL0lqbGxUfX29mpub1dbWpieeeEK1tbX65ptvVF1drdmzZ+vaa69VUVFRcB/Tpk3TqlWrgstLlizRH/7wB73xxhv6+uuvtWjRIrW3t+v+++/v+ysEgEQX7WUi27dvD3vpSWlpqTlz5oyZPn26GT58uBkyZIjJyckxCxcuNF6vN2QfOTk5pry8PGTdypUrzciRI01qaqrJy8sztbW1va6Jy/NosW7onYH+PSVS68vleY5/T3ZC8/v9crlcA10GkkgSfCz6Bfej7j2fz3fR36dxrw8AsBxBDQCW41FcSBqcrugdTlckHo6oAcByBDUAWI6gBgDLEdQAYDmCGgAsR1ADgOUIagCwHEENAJYjqAHAcgQ1AFiOoAYAyxHUAGA5ghoALEdQA4DlCGoAsBxBDQCWI6gBwHI84QVIEDyZ5dLFETUAWI6gBgDLEdQAYDmCGgAsR1ADgOUIagCwHEENAJYjqAHAcgQ1AFiOoAYAyxHUAGA5ghoALEdQA4DlCGoAsFzUQb1r1y7NmjVLHo9HDodDmzZtCtnucDjCtpdeeiniPlesWHFe/wkTJkT9YgAgGUUd1O3t7crNzVVlZWXY7cePHw9pa9askcPh0E9+8pMe9ztx4sSQcbt37462NABISlE/OKC4uFjFxcURt7vd7pDlzZs36/vf/77GjBnTcyGDB583FgAQ53PULS0t+uCDD/Tggw9esO+hQ4fk8Xg0ZswY3XvvvWpubo7Yt6OjQ36/P6QBQLKKa1C/8cYbGjp0qH784x/32C8/P19VVVXasmWLVq9ercbGRk2dOlWtra1h+1dUVMjlcgVbdnZ2PMpHPzHGxKQBScv0gSSzcePGiNvHjx9vFi9eHPV+v/32W5Oenm5ee+21sNvPnj1rfD5fsB05csRIoiVoQ+8M9O+J1rfm8/ku+ncft4fb/vWvf9XBgwf19ttvRz02IyND48aNU0NDQ9jtTqdTTqezryUCQEKI26mP119/XZMnT1Zubm7UY9va2nT48GFlZWXFoTIASCxRB3VbW5vq6+tVX18vSWpsbFR9fX3Il39+v1/vvvuuHnroobD7mDZtmlatWhVcfvzxx7Vz505988032rNnj+666y4NGjRIJSUl0ZYHAEkn6lMfn3/+ub7//e8Hl5csWSJJKi0tVVVVlSRpw4YNMsZEDNrDhw/r5MmTweWjR4+qpKREp06d0vDhw3X77bertrZWw4cPj7Y8AEg6jn9/SZHQ/H6/XC7XQJeBi5QEb8F+4XA4BroE9IHP51N6evpFjeVeHwBgOYIaACxHUAOA5QhqALAcQQ0AliOoAcByBDUAWI6gBgDLEdQAYDmCGgAsR1ADgOXidj9q2In7avQv7s+BWOCIGgAsR1ADgOUIagCwHEENAJYjqAHAcgQ1AFiOoAYAyxHUAGA5ghoALEdQA4DlCGoAsBxBDQCWI6gBwHIENQBYjqAGAMsR1ABgOYIaACyXFE944aklvef3+we6BOCS1JecSoqgbm1tHegSEobL5RroEoBLUmtr60V//hwmCQ5HA4GAjh07pqFDh/b4jDq/36/s7GwdOXJE6enp/Vhh31B3/0rUuqXErT2Z6zbGqLW1VR6PRykpF3e2OSmOqFNSUnTNNdf0un96enpCvRnOoe7+lah1S4lbe7LW3dd/yfJlIgBYjqAGAMtdUkHtdDpVXl4up9M50KVEhbr7V6LWLSVu7dTds6T4MhEAktkldUQNAImIoAYAyxHUAGA5ghoALJd0QV1ZWalRo0YpLS1N+fn52rt3b4/93333XU2YMEFpaWmaNGmSPvzww36q9F8qKip06623aujQoRoxYoTmzJmjgwcP9jimqqpKDocjpKWlpfVTxf+yYsWK82qYMGFCj2MGeq4ladSoUefV7XA4VFZWFrb/QM71rl27NGvWLHk8HjkcDm3atClkuzFGTz/9tLKysnTZZZepsLBQhw4duuB+o/2MxLLurq4uLV26VJMmTdLll18uj8ej+fPn69ixYz3u82Leb7GsW5IWLFhwXg0zZsy44H5jMd9JFdRvv/22lixZovLyctXV1Sk3N1dFRUU6ceJE2P579uxRSUmJHnzwQe3fv19z5szRnDlz9MUXX/RbzTt37lRZWZlqa2u1detWdXV1afr06Wpvb+9xXHp6uo4fPx5sTU1N/VTxf0ycODGkht27d0fsa8NcS9Jnn30WUvPWrVslSXfffXfEMQM11+3t7crNzVVlZWXY7S+++KJeeeUVvfrqq/r00091+eWXq6ioSGfPno24z2g/I7Gu+8yZM6qrq9Py5ctVV1en9957TwcPHtSdd955wf1G836Ldd3nzJgxI6SG9evX97jPmM23SSJ5eXmmrKwsuNzd3W08Ho+pqKgI23/u3Llm5syZIevy8/PNI488Etc6e3LixAkjyezcuTNin7Vr1xqXy9V/RYVRXl5ucnNze93fxrk2xphHH33UjB071gQCgbDbbZhrY4yRZDZu3BhcDgQCxu12m5deeim47vTp08bpdJr169dH3E+0n5FY1x3O3r17jSTT1NQUsU+077e+Cld3aWmpmT17dlT7idV8J80RdWdnp/bt26fCwsLgupSUFBUWFqqmpibsmJqampD+klRUVBSxf3/w+XySpCuvvLLHfm1tbcrJyVF2drZmz56tL7/8sj/KC3Ho0CF5PB6NGTNG9957r5qbmyP2tXGuOzs7tW7dOj3wwAM93szLhrn+X42NjfJ6vSFz6nK5lJ+fH3FOL+Yz0h98Pp8cDocyMjJ67BfN+y1eduzYoREjRmj8+PFatGiRTp06FbFvLOc7aYL65MmT6u7uVmZmZsj6zMxMeb3esGO8Xm9U/eMtEAjoscce02233aYbb7wxYr/x48drzZo12rx5s9atW6dAIKApU6bo6NGj/VZrfn6+qqqqtGXLFq1evVqNjY2aOnVqxFvO2jbXkrRp0yadPn1aCxYsiNjHhrkO59y8RTOnF/MZibezZ89q6dKlKikp6fGmRtG+3+JhxowZevPNN1VdXa0XXnhBO3fuVHFxsbq7u8P2j+V8J8Xd85JFWVmZvvjiiwueeysoKFBBQUFwecqUKbr++uv1+9//Xs8991y8y5QkFRcXB/980003KT8/Xzk5OXrnnXf04IMP9ksNffX666+ruLhYHo8nYh8b5jpZdXV1ae7cuTLGaPXq1T32teH9ds899wT/PGnSJN10000aO3asduzYoWnTpsX1ZyfNEfWwYcM0aNAgtbS0hKxvaWmR2+0OO8btdkfVP54WL16s999/X9u3b4/qlq2SNGTIEN18881qaGiIU3UXlpGRoXHjxkWswaa5lqSmpiZt27ZNDz30UFTjbJhrScF5i2ZOL+YzEi/nQrqpqUlbt26N+tamF3q/9YcxY8Zo2LBhEWuI5XwnTVCnpqZq8uTJqq6uDq4LBAKqrq4OOSL6bwUFBSH9JWnr1q0R+8eDMUaLFy/Wxo0b9fHHH2v06NFR76O7u1sHDhxQVlZWHCrsnba2Nh0+fDhiDTbM9X9bu3atRowYoZkzZ0Y1zoa5lqTRo0fL7XaHzKnf79enn34acU4v5jMSD+dC+tChQ9q2bZuuuuqqqPdxofdbfzh69KhOnToVsYaYzndUXz1absOGDcbpdJqqqirz1VdfmYcffthkZGQYr9drjDHmvvvuM08++WSw/yeffGIGDx5sXn75ZfP111+b8vJyM2TIEHPgwIF+q3nRokXG5XKZHTt2mOPHjwfbmTNngn3+t+5nnnnGfPTRR+bw4cNm37595p577jFpaWnmyy+/7Le6f/7zn5sdO3aYxsZG88knn5jCwkIzbNgwc+LEibA12zDX53R3d5uRI0eapUuXnrfNprlubW01+/fvN/v37zeSzG9+8xuzf//+4NURzz//vMnIyDCbN282f/vb38zs2bPN6NGjzT//+c/gPn7wgx+YlStXBpcv9BmJd92dnZ3mzjvvNNdcc42pr68Pec93dHRErPtC77d4193a2moef/xxU1NTYxobG822bdvMLbfcYq677jpz9uzZiHXHar6TKqiNMWblypVm5MiRJjU11eTl5Zna2trgtjvuuMOUlpaG9H/nnXfMuHHjTGpqqpk4caL54IMP+rVeSWHb2rVrI9b92GOPBV9jZmam+dGPfmTq6ur6te558+aZrKwsk5qaaq6++mozb94809DQELFmYwZ+rs/56KOPjCRz8ODB87bZNNfbt28P+944V18gEDDLly83mZmZxul0mmnTpp33mnJyckx5eXnIup4+I/Guu7GxMeJ7fvv27RHrvtD7Ld51nzlzxkyfPt0MHz7cDBkyxOTk5JiFCxeeF7jxmm9ucwoAlkuac9QAkKwIagCwHEENAJYjqAHAcgQ1AFiOoAYAyxHUAGA5ghoALEdQA4DlCGoAsBxBDQCWI6gBwHL/D8h4+jp8OQlNAAAAAElFTkSuQmCC",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.imshow(x[389], cmap='gray', interpolation='nearest')\n",
        "print(Y[389])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6xiZQbLrxZXX",
      "metadata": {
        "id": "6xiZQbLrxZXX"
      },
      "source": [
        "## Our First Problem!\n",
        "\n",
        "Though we have our 390 images of binarized digits from 0-9, how am I supposed to work with a two dimensional image?!? Think of other \"regression\" problems such as predicting house prices or test scores. We always get the features of an item as a \"vector\" and not as a matrix. This is because classifying a matrix is non-trivial.\n",
        "\n",
        "*An Aside* While there is absolutely a way we can classify these two dimensional matrix images directly, these are called convolutional neural networks, that's way beyond the scope of this class. However you're welcome to look into the subject in your own time.\n",
        "\n",
        "Therefore we now need to take our matrix inputs and convert them into vector inputs, if only there was an easy way to \"flatten\" all of our matrices... however we need to make them columns not rows so after flattening we can flip them"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 105,
      "id": "M11_8ehwyTSU",
      "metadata": {
        "id": "M11_8ehwyTSU"
      },
      "outputs": [],
      "source": [
        "flattned_x = [matrix.flatten().reshape(1, -1) for matrix in x]\n",
        "flattened_x = np.array(flattned_x)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "J8DqrnRX_Hg-",
      "metadata": {
        "id": "J8DqrnRX_Hg-"
      },
      "source": [
        "On that note we also need to convert our labels which are digits from 0-9 into a one hot encoding that look like the following:\n",
        "\n",
        "0 -> [1, 0, 0, 0, ...]\n",
        "\n",
        "...\n",
        "\n",
        "9-> [..., 1]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 101,
      "id": "xub8K7of_Ylx",
      "metadata": {
        "id": "xub8K7of_Ylx"
      },
      "outputs": [],
      "source": [
        "one_hot_encoding = np.eye(10) # NOTE: Look into why this line is useful\n",
        "one_hot_array = np.array(one_hot_encoding)\n",
        "\n",
        "one_hot_Y = np.array([one_hot_array[i] for i in Y])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "pTj3oIZ5zHRJ",
      "metadata": {
        "id": "pTj3oIZ5zHRJ"
      },
      "source": [
        "## The Architecture\n",
        "\n",
        "Just like designing a house, a strong data scientist designs the best model for their needs. Different business needs would dictate different model designs, however in an effort to keep this task simple we will work with the bare minimum.\n",
        "\n",
        "We'll build a neural network model with three layers, an input, a hidden, and an output layer. Each layer will have a certain inputs and outputs, any guesses what the inputs of the first and the outputs of the final layer will be??\n",
        "\n",
        "**Input Layer** Here we simply take each pixel value from our digit input... that's literally it there's no math in this part. Our input layer is going to be a 1x320 input since there are 320 pixels in total.\n",
        "\n",
        "**W1** The weights in between the layers take the values from each neuron (there are 320 neurons in the first layer) and multiply it with some set of weights that we learn iteratively. The result of this matrix multiplication needs to be the size of the hidden layer.\n",
        "\n",
        "**Hidden Layer** It's called a hidden layer because, well it's hidden! After multiplying the first layer by the weights I want to be left with 120 neurons or values in this layer for each image. This means that the weights combine certain pixel values together in some smart way that we don't really need to worry about.\n",
        "\n",
        "**W2** A second set of weights need to transform the 120 neurons of the hidden layer into exactly 10 values\n",
        "\n",
        "**Output Layer** There are exactly 10 possibilities for out image, this layer should consist of 10 values which should be the probabilities/likelihoods of the given digits from 0-9.\n",
        "\n",
        "**Non Linearity** You'll also see that we use a sigmoid function below, this helps us understand the nonlinear relationship between the pixels and its class much better. The important part for now is understanding how to implement it, please look into why it's important in your own time. Without non linearity what we're doing simply becomes your good ole fashioned linear regression\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/Codecademy/docs/main/media/sigmoid-function.png\" alt=\"meme\" width=\"400\"/>\n",
        "\n",
        "If this was a bit too much to understand please ask your TA for help!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 77,
      "id": "f4Y0zsVV2MlH",
      "metadata": {
        "id": "f4Y0zsVV2MlH"
      },
      "outputs": [],
      "source": [
        "# Creating the Feed forward neural network\n",
        "# 1 Input layer(1, 320)\n",
        "# 1 hidden layer (1, 120)\n",
        "# 1 output layer(1, 10)\n",
        "\n",
        "def sigmoid(x):\n",
        "    return(1/(1 + np.exp(-x)))\n",
        "\n",
        "def f_forward(x, w1, w2):\n",
        "    # hidden\n",
        "    z1 = x.dot(w1)# input from layer 1\n",
        "    a1 = sigmoid(z1)# output of layer 2\n",
        "\n",
        "    # Output layer\n",
        "    z2 = a1.dot(w2)# input of out layer\n",
        "    a2 = sigmoid(z2)# output of out layer\n",
        "    return(a2)\n",
        "\n",
        "# The weights are the super important part, however at the start we can just\n",
        "# assign them random values since we don't know what they'll be\n",
        "def generate_wt(x, y):\n",
        "    l =[]\n",
        "    for i in range(x * y):\n",
        "        l.append(np.random.randn())\n",
        "    return(np.array(l).reshape(x, y))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "XdCcMJsX4Mzh",
      "metadata": {
        "id": "XdCcMJsX4Mzh"
      },
      "source": [
        "## Loss\n",
        "\n",
        "A loss function tells our model how well it has done in predicting classes, in our case it tells the model how well it has classified our digits into 0-9. To keep things simple we use the Mean Square Error loss, basically we look at the distance between our prediction and label and take the square of it then mean of all the values to figure out how good/bad our model is"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 78,
      "id": "Onhe7BO34sqC",
      "metadata": {
        "id": "Onhe7BO34sqC"
      },
      "outputs": [],
      "source": [
        "def loss(out, Y):\n",
        "    s =(np.square(out-Y))\n",
        "    s = np.sum(s)/len(Y)\n",
        "    return(s)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "oJAClmB64NIy",
      "metadata": {
        "id": "oJAClmB64NIy"
      },
      "source": [
        "## Backpropogation\n",
        "\n",
        "Understanding back propogation is the key to understanding how neural networks learn. Therefore, I've used the help of my handy AI friend to help explain.\n",
        "\n",
        "Imagine you're playing a video game, and your goal is to get the highest score. Now, let's say after each game, your friend gives you feedback on your performance. They point out where you made mistakes and suggest how you can improve.\n",
        "\n",
        "Backpropagation in the world of computers is a bit like that feedback process for a computer learning how to do a task, such as recognizing pictures of cats and dogs.\n",
        "\n",
        "Here's how it works:\n",
        "\n",
        "1. **Making Predictions:**\n",
        "   - Just like you're trying to predict the best moves in the game, a computer model makes predictions based on the input it gets. For example, it looks at a vector and tries to decide if it's a 0 or a 5.\n",
        "\n",
        "2. **Comparing Predictions to Reality:**\n",
        "   - After making a prediction, the computer checks it against the correct answer (like checking if you correctly identified the picture). If it got it wrong, it needs to figure out what went wrong.\n",
        "\n",
        "3. **Calculating Mistakes (Loss):**\n",
        "   - The computer calculates how far off its prediction was from the correct answer using the loss function we talked about earlier. This is like your friend giving you a score after each game.\n",
        "\n",
        "4. **Adjusting to Improve:**\n",
        "   - Now comes the cool part, backpropagation! The computer looks at the mistakes it made and figures out how it can adjust its \"strategies\" to do better next time. It's like you learning from your mistakes and trying a different approach in the game.\n",
        "\n",
        "5. **Updating the Model:**\n",
        "   - The computer updates its weights based on the mistakes it made. It's like tweaking the way it makes decisions so that it gets closer to the correct answer the next time.\n",
        "\n",
        "6. **Repeating the Process:**\n",
        "   - The computer repeats this process over and over. Each time it plays a \"game\" (makes predictions on a set of data), it learns from its mistakes and gets better.\n",
        "\n",
        "So, backpropagation is like a computer learning from its errors, adjusting its approach, and getting smarter and more accurate over time just like you do when playing a video game and trying to improve your score!\n",
        "\n",
        "The way it improves is by tweaking the weights repeatedly until it has reached some local maxima at which point it proclaims that it has done the best it can! This process is called gradient descent since we calculate the gradient of the given input and follow that gradient by taking steps until we reach the optimum value for the weights.\n",
        "\n",
        "At the end of the day, all we want to learn is the best way that we can transform a vector of 320 into a vector of 10."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 79,
      "id": "b930XDTL4Nf3",
      "metadata": {
        "id": "b930XDTL4Nf3"
      },
      "outputs": [],
      "source": [
        "def back_prop(x, Y, w1, w2, alpha):\n",
        "\n",
        "    # hidden layer\n",
        "    z1 = x.dot(w1)# input from layer 1\n",
        "    a1 = sigmoid(z1)# output of layer 2\n",
        "\n",
        "    # Output layer\n",
        "    z2 = a1.dot(w2)# input of out layer\n",
        "    a2 = sigmoid(z2)# output of out layer\n",
        "\n",
        "    # error in output layer\n",
        "    d2 =(a2-Y)\n",
        "\n",
        "    # The weights for the hidden layer are the more complex part, spend some time trying to\n",
        "    # understand this code basically we try to figure out how much error each neuron\n",
        "    # in the hidden layer is responsible for\n",
        "\n",
        "    # The second part is to make sure that we correctly identify both it being a certain\n",
        "    # class and also NOT being a certain class, the first is the assignment of blame,\n",
        "    # your TA will go into more detail\n",
        "    d1 = np.multiply((w2.dot((d2.transpose()))).transpose(), (np.multiply(a1, 1-a1)))\n",
        "\n",
        "    # Gradient for w1 and w2\n",
        "    w1_adj = x.transpose().dot(d1)\n",
        "    w2_adj = a1.transpose().dot(d2)\n",
        "\n",
        "    # Updating parameters\n",
        "    w1 = w1-(alpha*(w1_adj))\n",
        "    w2 = w2-(alpha*(w2_adj))\n",
        "\n",
        "    return(w1, w2)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "t-1PSPqs94XJ",
      "metadata": {
        "id": "t-1PSPqs94XJ"
      },
      "source": [
        "## Wrapping Up\n",
        "\n",
        "We finally need to bring it all together, let's write a train function, we set a limit for how many iterations we want to try to train our model (epochs)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 111,
      "id": "wnXACrFU93OF",
      "metadata": {
        "id": "wnXACrFU93OF"
      },
      "outputs": [],
      "source": [
        "def train(x, Y, w1, w2, alpha = 0.01, epoch = 10):\n",
        "    acc =[]\n",
        "    losss =[]\n",
        "    for j in range(epoch):\n",
        "        l =[]\n",
        "        for i in range(len(x)):\n",
        "            out = f_forward(x[i], w1, w2)\n",
        "            l.append((loss(out, Y[i])))\n",
        "            w1, w2 = back_prop(x[i], Y[i], w1, w2, alpha)\n",
        "        print(\"epochs:\", j + 1, \"======== acc:\", (1-(sum(l)/len(x)))*100)\n",
        "        acc.append((1-(sum(l)/len(x)))*100)\n",
        "        losss.append(sum(l)/len(x))\n",
        "    return(acc, losss, w1, w2)\n",
        "\n",
        "def predict(x, w1, w2):\n",
        "    Out = f_forward(x, w1, w2)\n",
        "    maxm = 0\n",
        "    k = 0\n",
        "    for i in range(len(Out[0])):\n",
        "        if(maxm < Out[0][i]):\n",
        "            maxm = Out[0][i]\n",
        "            k = i\n",
        "    print(k)\n",
        "    plt.imshow(x.reshape(20, 16), cmap='gray', interpolation='nearest')\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 117,
      "id": "Y6MpDL6M-g4P",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Y6MpDL6M-g4P",
        "outputId": "a90d2cca-602c-46eb-d261-fd4888775350"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "epochs: 1 ======== acc: 97.98068816727906\n",
            "epochs: 2 ======== acc: 98.7395418515099\n",
            "epochs: 3 ======== acc: 99.14729848939221\n",
            "epochs: 4 ======== acc: 99.40902321814347\n",
            "epochs: 5 ======== acc: 99.55458427888809\n",
            "epochs: 6 ======== acc: 99.64894206408991\n",
            "epochs: 7 ======== acc: 99.7402402768996\n",
            "epochs: 8 ======== acc: 99.82838381688326\n",
            "epochs: 9 ======== acc: 99.867182373644\n",
            "epochs: 10 ======== acc: 99.89899991050717\n"
          ]
        }
      ],
      "source": [
        "w1 = generate_wt(320, 120)\n",
        "w2 = generate_wt(120, 10)\n",
        "\n",
        "acc, losss, w1, w2 = train(flattened_x, one_hot_Y, w1, w2, 0.1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 114,
      "id": "VFMI4hMOCITh",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 448
        },
        "id": "VFMI4hMOCITh",
        "outputId": "967e32d9-cb47-4b95-f994-e5e18f9f18f5"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "8\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAGdCAYAAAA7TzlCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAiqklEQVR4nO3df2xV9f3H8dct0Ftj6K0K9PZKKaACitAos11R5zYqpTNInfNH46QooiEl0TANsohFXVZ/zWxCg8siVMPEH4lApg4HlR9DWlFKM0BHKKstBG4JRO6lZZSm/Xz/2Jc777i3cOm57edeno/kk3DO+XxO3+f03BeHcy7nuIwxRgAAa6X0dwEAgJ4R1ABgOYIaACxHUAOA5QhqALAcQQ0AliOoAcByBDUAWG5gfxfghO7ubh06dEiDBw+Wy+Xq73IAIMQYoxMnTsjn8ykl5cLOjZMiqA8dOqTs7Oz+LgMAojpw4ICGDx9+QWOTIqgHDx7c3yUACSMQCDi2Lo/H49i6nOLU9jm9bb3JqaQIai53AOcvPT29v0uIK1u3rzc5xc1EALAcQQ0AlotbUFdVVWnkyJFKS0tTfn6+tm/f3mP/Dz74QOPGjVNaWpomTJigTz75JF6lAUBiMXHw7rvvmtTUVLN8+XKzZ88eM2fOHJORkWFaW1sj9v/888/NgAEDzMsvv2y+/vpr88wzz5hBgwaZXbt2ndfPCwQCRhKNRjuP5qT+3pZ4bp/TdQUCgQuvxbGt+p68vDxTXl4emu7q6jI+n89UVlZG7H/vvfeaO+64I2xefn6+eeyxx87r5xHUNNr5Nyf197bEc/ucrqs3Qe34pY/Tp09rx44dKiwsDM1LSUlRYWGhamtrI46pra0N6y9JRUVFUft3dHQoGAyGNQBIVo4H9dGjR9XV1aXMzMyw+ZmZmfL7/RHH+P3+mPpXVlbK4/GEGv/ZBUAyS8hvfSxcuFCBQCDUDhw40N8lAUDcOP4fXoYMGaIBAwaotbU1bH5ra6u8Xm/EMV6vN6b+brdbbrfbmYIBwHKOn1GnpqZq0qRJqqmpCc3r7u5WTU2NCgoKIo4pKCgI6y9J69evj9ofAC4qjt0i/Z53333XuN1uU11dbb7++mvz6KOPmoyMDOP3+40xxjz44IPm6aefDvX//PPPzcCBA82rr75qvvnmG1NRUcHX82i0ODUn9fe2xHP7nK7Luq/nGWPMkiVLzIgRI0xqaqrJy8szdXV1oWW33XabKSsrC+v//vvvmzFjxpjU1FQzfvx48/HHH5/3zyKoabTzb07q722J5/Y5XVdvgtr1/wUltGAwaOVTvAAbOfmRt/GBaE5tn9PbFggELviBUQn5rQ8AuJgkxWNOAaclwT80+0Qy7yents2Jf/FzRg0AliOoAcByBDUAWI6gBgDLEdQAYDmCGgAsR1ADgOUIagCwHEENAJYjqAHAcgQ1AFiOoAYAyxHUAGA5ghoALEdQA4DlCGoAsBxBDQCWI6gBwHK8igtJI5lfC2WrZH65rU04owYAyxHUAGA5ghoALEdQA4DlCGoAsBxBDQCWI6gBwHIENQBYjqAGAMsR1ABgOYIaACxHUAOA5QhqALAcQQ0AlnM8qCsrK3XTTTdp8ODBGjZsmEpKSrR3794ex1RXV8vlcoW1tLQ0p0sDgITkeFBv3rxZ5eXlqqur0/r169XZ2ampU6eqvb29x3Hp6ek6fPhwqDU3NztdGgAkJMdfHLBu3bqw6erqag0bNkw7duzQj370o6jjXC6XvF6v0+UAQMKL+xteAoGAJOnyyy/vsV9bW5tycnLU3d2tG2+8Ub/97W81fvz4iH07OjrU0dERmg4Gg84VbKFkfGPF99n4lhCnJPO22cqpfW7T5y6uNxO7u7v1xBNP6Oabb9b1118ftd/YsWO1fPlyrV27VitXrlR3d7cmT56sgwcPRuxfWVkpj8cTatnZ2fHaBADody4Tx7825s6dq7/+9a/aunWrhg8fft7jOjs7de2116q0tFQvvPDCWcsjnVEnc1jb9Dd7PCTjGdAZnFEnLqeOp2AwKI/Ho0AgoPT09AtaR9wufcybN08fffSRtmzZElNIS9KgQYN0ww03qLGxMeJyt9stt9vtRJkAYD3HL30YYzRv3jytXr1an332mUaNGhXzOrq6urRr1y5lZWU5XR4AJBzHz6jLy8v1zjvvaO3atRo8eLD8fr8kyePx6JJLLpEkzZw5U1deeaUqKyslSc8//7x++MMf6uqrr9bx48f1yiuvqLm5WY888ojT5QFAwnE8qJctWyZJ+vGPfxw2f8WKFZo1a5YkqaWlRSkp/z2Z/+677zRnzhz5/X5ddtllmjRpkrZt26brrrvO6fIAIOHE9WZiXzlzsT5ZJcGvqEfcTISNbLqZyLM+AMByBDUAWI6gBgDLEdQAYDmCGgAsR1ADgOUIagCwHEENAJYjqAHAcgQ1AFiOoAYAy8X9VVxITsn+DItk375kZuMzX3qLM2oAsBxBDQCWI6gBwHIENQBYjqAGAMsR1ABgOYIaACxHUAOA5QhqALAcQQ0AliOoAcByBDUAWI6gBgDLEdQAYDmCGgAsR1ADgOUIagCwHG94SQBOvm3EqbdfJONbNABbcUYNAJYjqAHAcgQ1AFiOoAYAyxHUAGA5x4N68eLFcrlcYW3cuHE9jvnggw80btw4paWlacKECfrkk0+cLgsAElZczqjHjx+vw4cPh9rWrVuj9t22bZtKS0s1e/Zs7dy5UyUlJSopKdHu3bvjURoAJJy4BPXAgQPl9XpDbciQIVH7/uEPf9C0adP01FNP6dprr9ULL7ygG2+8UUuXLo1HaQCQcOIS1Pv27ZPP59Po0aP1wAMPqKWlJWrf2tpaFRYWhs0rKipSbW1t1DEdHR0KBoNhDQCSleNBnZ+fr+rqaq1bt07Lli1TU1OTbr31Vp04cSJif7/fr8zMzLB5mZmZ8vv9UX9GZWWlPB5PqGVnZzu6DQBgE8eDuri4WPfcc48mTpyooqIiffLJJzp+/Ljef/99x37GwoULFQgEQu3AgQOOrRsAbBP3Z31kZGRozJgxamxsjLjc6/WqtbU1bF5ra6u8Xm/UdbrdbrndbkfrBABbxf171G1tbdq/f7+ysrIiLi8oKFBNTU3YvPXr16ugoCDepQFAQnA8qJ988klt3rxZ3377rbZt26a77rpLAwYMUGlpqSRp5syZWrhwYaj/448/rnXr1ul3v/ud/vnPf2rx4sX66quvNG/ePKdLA4CE5Pilj4MHD6q0tFTHjh3T0KFDdcstt6iurk5Dhw6VJLW0tCgl5b9/P0yePFnvvPOOnnnmGf3617/WNddcozVr1uj66693ujQASEgukwQPFg4Gg/J4PP1dRkJIgl93n3DyGeDoW7Yd42fyKRAIKD09/YLWwbM+AMByBDUAWI5XcV1knPonvW3/vERi43jqGWfUAGA5ghoALEdQA4DlCGoAsBxBDQCWI6gBwHIENQBYjqAGAMsR1ABgOYIaACxHUAOA5QhqALAcQQ0AliOoAcByBDUAWI6gBgDLEdQAYDne8IILYuPLX518S4iNbxzh7Tx9y6ZjnDNqALAcQQ0AliOoAcByBDUAWI6gBgDLEdQAYDmCGgAsR1ADgOUIagCwHEENAJYjqAHAcgQ1AFiOoAYAyxHUAGA5x4N65MiRcrlcZ7Xy8vKI/aurq8/qm5aW5nRZAJCwHH8e9Zdffqmurq7Q9O7du3X77bfrnnvuiTomPT1de/fuDU3b9BxYAOhvjgf10KFDw6ZffPFFXXXVVbrtttuijnG5XPJ6vU6XAgBJIa7XqE+fPq2VK1fq4Ycf7vEsua2tTTk5OcrOztaMGTO0Z8+eeJYFAAklrq/iWrNmjY4fP65Zs2ZF7TN27FgtX75cEydOVCAQ0KuvvqrJkydrz549Gj58eMQxHR0d6ujoCE0Hg0GnS0cCcvKSmY2vq7KxJhsl46VTl4njb7+oqEipqan6y1/+ct5jOjs7de2116q0tFQvvPBCxD6LFy/Wc88951SZwFkIxcRla1AHAgGlp6df0Ni4Xfpobm7Whg0b9Mgjj8Q0btCgQbrhhhvU2NgYtc/ChQsVCARC7cCBA70tFwCsFbegXrFihYYNG6Y77rgjpnFdXV3atWuXsrKyovZxu91KT08PawCQrOIS1N3d3VqxYoXKyso0cGD4ZfCZM2dq4cKFoennn39ef/vb3/Svf/1L9fX1+uUvf6nm5uaYz8QBIFnF5Wbihg0b1NLSoocffvisZS0tLUpJ+e/fD999953mzJkjv9+vyy67TJMmTdK2bdt03XXXxaM0AEg4cb2Z2FeCwaA8Hk9/l4EkkgQfi4sWNxMBAH2OoAYAyxHUAGA5ghoALEdQA4DlCGoAsBxBDQCWI6gBwHIENQBYjqAGAMsR1ABgubi+4QVIVE49L4JnhpwfW5/PYQvOqAHAcgQ1AFiOoAYAyxHUAGA5ghoALEdQA4DlCGoAsBxBDQCWI6gBwHIENQBYjqAGAMsR1ABgOYIaACxHUAOA5QhqALAcQQ0AliOoAcByBDUAWI5XcSFp8NorJCvOqAHAcgQ1AFiOoAYAyxHUAGA5ghoALBdzUG/ZskXTp0+Xz+eTy+XSmjVrwpYbY/Tss88qKytLl1xyiQoLC7Vv375zrreqqkojR45UWlqa8vPztX379lhLA4CkFHNQt7e3Kzc3V1VVVRGXv/zyy3r99df1xhtv6IsvvtCll16qoqIinTp1Kuo633vvPc2fP18VFRWqr69Xbm6uioqKdOTIkVjLA4DkY3pBklm9enVouru723i9XvPKK6+E5h0/fty43W6zatWqqOvJy8sz5eXloemuri7j8/lMZWXledURCASMJNpF3pC4+vvY6YsWCAQueP84eo26qalJfr9fhYWFoXkej0f5+fmqra2NOOb06dPasWNH2JiUlBQVFhZGHdPR0aFgMBjWACBZORrUfr9fkpSZmRk2PzMzM7Tsfx09elRdXV0xjamsrJTH4wm17OxsB6oHADsl5Lc+Fi5cqEAgEGoHDhzo75IAIG4cDWqv1ytJam1tDZvf2toaWva/hgwZogEDBsQ0xu12Kz09PawBQLJyNKhHjRolr9ermpqa0LxgMKgvvvhCBQUFEcekpqZq0qRJYWO6u7tVU1MTdQwAXExifnpeW1ubGhsbQ9NNTU1qaGjQ5ZdfrhEjRuiJJ57Qb37zG11zzTUaNWqUFi1aJJ/Pp5KSktCYKVOm6K677tK8efMkSfPnz1dZWZl+8IMfKC8vT7///e/V3t6uhx56qPdbCACJLtaviWzcuDHiV0/KysqMMf/5it6iRYtMZmamcbvdZsqUKWbv3r1h68jJyTEVFRVh85YsWWJGjBhhUlNTTV5enqmrqzvvmvh6Hk3i63mJrL+Pnb5ovfl6nuv/d1JCCwaD8ng8/V0G+lkSHMoXLZfL1d8lxF0gELjg+2kJ+a0PALiY8IYXXBDOXs+PjWeKNv7unKzJxn3eW5xRA4DlCGoAsBxBDQCWI6gBwHIENQBYjqAGAMsR1ABgOYIaACxHUAOA5QhqALAcQQ0AliOoAcByBDUAWI6gBgDLEdQAYDmCGgAsR1ADgOUIagCwHK/iusjY+BompyTjK5jiwcb95ORx6dS6bNpPnFEDgOUIagCwHEENAJYjqAHAcgQ1AFiOoAYAyxHUAGA5ghoALEdQA4DlCGoAsBxBDQCWI6gBwHIENQBYjqAGAMvFHNRbtmzR9OnT5fP55HK5tGbNmtCyzs5OLViwQBMmTNCll14qn8+nmTNn6tChQz2uc/HixXK5XGFt3LhxMW8MACSjmIO6vb1dubm5qqqqOmvZyZMnVV9fr0WLFqm+vl4ffvih9u7dqzvvvPOc6x0/frwOHz4calu3bo21NABISjG/OKC4uFjFxcURl3k8Hq1fvz5s3tKlS5WXl6eWlhaNGDEieiEDB8rr9cZaDgAkvbhfow4EAnK5XMrIyOix3759++Tz+TR69Gg98MADamlpidq3o6NDwWAwrCUzY4xjDcD5ceozFwgEel1LXIP61KlTWrBggUpLS5Wenh61X35+vqqrq7Vu3TotW7ZMTU1NuvXWW3XixImI/SsrK+XxeEItOzs7XpsAAP3OZXpxmuVyubR69WqVlJSctayzs1N33323Dh48qE2bNvUY1P/r+PHjysnJ0WuvvabZs2eftbyjo0MdHR2h6WAwmNRhzZnw+bHpHXeITTIf48FgUB6PR4FAIKYc/L64vNy2s7NT9957r5qbm/XZZ5/FXFxGRobGjBmjxsbGiMvdbrfcbrcTpQKA9Ry/9HEmpPft26cNGzboiiuuiHkdbW1t2r9/v7KyspwuDwASTsxB3dbWpoaGBjU0NEiSmpqa1NDQoJaWFnV2duoXv/iFvvrqK/35z39WV1eX/H6//H6/Tp8+HVrHlClTtHTp0tD0k08+qc2bN+vbb7/Vtm3bdNddd2nAgAEqLS3t/RYCQKIzMdq4caORdFYrKyszTU1NEZdJMhs3bgytIycnx1RUVISm77vvPpOVlWVSU1PNlVdeae677z7T2Nh43jUFAoGoPzcZGs5Pf/+eaBzjkZzJp0AgcMHr6NXNRFucuVifrJLgV9QnuJmYuJL5GHfiZiLP+gAAyxHUAGA5ghoALEdQA4DlCGoAsBxBDQCWI6gBwHIENQBYjqAGAMsR1ABgOYIaACwXl+dRJ7JkfuaAk3iuBpzk5PGUjJ9hzqgBwHIENQBYjqAGAMsR1ABgOYIaACxHUAOA5QhqALAcQQ0AliOoAcByBDUAWI6gBgDLEdQAYDmCGgAsR1ADgOUIagCwHEENAJYjqAHAcrzhJQHwNhXg4sYZNQBYjqAGAMsR1ABgOYIaACxHUAOA5WIO6i1btmj69Ony+XxyuVxas2ZN2PJZs2bJ5XKFtWnTpp1zvVVVVRo5cqTS0tKUn5+v7du3x1oaACSlmIO6vb1dubm5qqqqitpn2rRpOnz4cKitWrWqx3W+9957mj9/vioqKlRfX6/c3FwVFRXpyJEjsZYHAMnH9IIks3r16rB5ZWVlZsaMGTGtJy8vz5SXl4emu7q6jM/nM5WVlec1PhAIGEmONBs5tW002sXQbHMmnwKBwAWvIy7XqDdt2qRhw4Zp7Nixmjt3ro4dOxa17+nTp7Vjxw4VFhaG5qWkpKiwsFC1tbURx3R0dCgYDIY1AEhWjgf1tGnT9Pbbb6umpkYvvfSSNm/erOLiYnV1dUXsf/ToUXV1dSkzMzNsfmZmpvx+f8QxlZWV8ng8oZadne30ZgCANRz/L+T3339/6M8TJkzQxIkTddVVV2nTpk2aMmWKIz9j4cKFmj9/fmg6GAwS1gCSVty/njd69GgNGTJEjY2NEZcPGTJEAwYMUGtra9j81tZWeb3eiGPcbrfS09PDGgAkq7gH9cGDB3Xs2DFlZWVFXJ6amqpJkyappqYmNK+7u1s1NTUqKCiId3kAYL2Yg7qtrU0NDQ1qaGiQJDU1NamhoUEtLS1qa2vTU089pbq6On377beqqanRjBkzdPXVV6uoqCi0jilTpmjp0qWh6fnz5+tPf/qT3nrrLX3zzTeaO3eu2tvb9dBDD/V+CwEg0cX6NZGNGzdG/EpMWVmZOXnypJk6daoZOnSoGTRokMnJyTFz5swxfr8/bB05OTmmoqIibN6SJUvMiBEjTGpqqsnLyzN1dXXnXRNfz6PRaGeabZz4ep7LGGOU4ILBoDwejyPrsnF38Dxq4PzZ9hk+k0+BQOCC76fxrA8AsBxBDQCWS6pXcfXmnxbxwCULoO859bmz6RIKZ9QAYDmCGgAsR1ADgOUIagCwHEENAJYjqAHAcgQ1AFiOoAYAyxHUAGA5ghoALEdQA4DlCGoAsBxBDQCWI6gBwHIENQBYjqAGAMsR1ABguaR6wwsA2PRmFqdwRg0AliOoAcByBDUAWI6gBgDLEdQAYDmCGgAsR1ADgOUIagCwHEENAJYjqAHAcgQ1AFiOoAYAyxHUAGA5ghoALBdzUG/ZskXTp0+Xz+eTy+XSmjVrwpa7XK6I7ZVXXom6zsWLF5/Vf9y4cTFvDAAko5iDur29Xbm5uaqqqoq4/PDhw2Ft+fLlcrlcuvvuu3tc7/jx48PGbd26NdbSACApxfzigOLiYhUXF0dd7vV6w6bXrl2rn/zkJxo9enTPhQwceNZYAECcr1G3trbq448/1uzZs8/Zd9++ffL5fBo9erQeeOABtbS0RO3b0dGhYDAY1gAgWcX1VVxvvfWWBg8erJ///Oc99svPz1d1dbXGjh2rw4cP67nnntOtt96q3bt3a/DgwWf1r6ys1HPPPXfWfI/H0+uanXyNj1PrcrlcjqwHkJLzVVXxYNPnzmV68VtzuVxavXq1SkpKIi4fN26cbr/9di1ZsiSm9R4/flw5OTl67bXXIp6Nd3R0qKOjIzQdDAaVnZ0d08+IxsaD2KYDBonPxmPcRk5/7gKBgNLT0y9obNzOqP/+979r7969eu+992Iem5GRoTFjxqixsTHicrfbLbfb3dsSASAhxO0a9ZtvvqlJkyYpNzc35rFtbW3av3+/srKy4lAZACSWmIO6ra1NDQ0NamhokCQ1NTWpoaEh7OZfMBjUBx98oEceeSTiOqZMmaKlS5eGpp988klt3rxZ3377rbZt26a77rpLAwYMUGlpaazlAUDSifnSx1dffaWf/OQnoen58+dLksrKylRdXS1Jevfdd2WMiRq0+/fv19GjR0PTBw8eVGlpqY4dO6ahQ4fqlltuUV1dnYYOHRpreQCQdHp1M9EWwWDQkW98SHbeaOFmIpxk4zFuI5tuJvKsDwCwHEENAJYjqAHAcgQ1AFiOoAYAyxHUAGA5ghoALEdQA4DlCGoAsBxBDQCWI6gBwHJxfcNLInLy//c79UwFns2AZMfzbHrGGTUAWI6gBgDLEdQAYDmCGgAsR1ADgOUIagCwHEENAJYjqAHAcgQ1AFiOoAYAyxHUAGA5ghoALEdQA4DlCGoAsBxBDQCWI6gBwHIENQBYLine8GLrG1CCwWB/lwDAEr3JqaQI6hMnTvR3CRF5PJ7+LgGAJU6cOHHBmeAytp6OxqC7u1uHDh3S4MGDe3z3WjAYVHZ2tg4cOKD09PQ+rLB3qLtvJWrdUuLWnsx1G2N04sQJ+Xw+paRc2NXmpDijTklJ0fDhw8+7f3p6ekIdDGdQd99K1LqlxK09Wevu7b+uuZkIAJYjqAHAchdVULvdblVUVMjtdvd3KTGh7r6VqHVLiVs7dfcsKW4mAkAyu6jOqAEgERHUAGA5ghoALEdQA4Dlki6oq6qqNHLkSKWlpSk/P1/bt2/vsf8HH3ygcePGKS0tTRMmTNAnn3zSR5X+R2VlpW666SYNHjxYw4YNU0lJifbu3dvjmOrqarlcrrCWlpbWRxX/x+LFi8+qYdy4cT2O6e99LUkjR448q26Xy6Xy8vKI/ftzX2/ZskXTp0+Xz+eTy+XSmjVrwpYbY/Tss88qKytLl1xyiQoLC7Vv375zrjfWz4iTdXd2dmrBggWaMGGCLr30Uvl8Ps2cOVOHDh3qcZ0Xcrw5WbckzZo166wapk2bds71OrG/kyqo33vvPc2fP18VFRWqr69Xbm6uioqKdOTIkYj9t23bptLSUs2ePVs7d+5USUmJSkpKtHv37j6refPmzSovL1ddXZ3Wr1+vzs5OTZ06Ve3t7T2OS09P1+HDh0Otubm5jyr+r/Hjx4fVsHXr1qh9bdjXkvTll1+G1bx+/XpJ0j333BN1TH/t6/b2duXm5qqqqiri8pdfflmvv/663njjDX3xxRe69NJLVVRUpFOnTkVdZ6yfEafrPnnypOrr67Vo0SLV19frww8/1N69e3XnnXeec72xHG9O133GtGnTwmpYtWpVj+t0bH+bJJKXl2fKy8tD011dXcbn85nKysqI/e+9915zxx13hM3Lz883jz32WFzr7MmRI0eMJLN58+aofVasWGE8Hk/fFRVBRUWFyc3NPe/+Nu5rY4x5/PHHzVVXXWW6u7sjLrdhXxtjjCSzevXq0HR3d7fxer3mlVdeCc07fvy4cbvdZtWqVVHXE+tnxOm6I9m+fbuRZJqbm6P2ifV4661IdZeVlZkZM2bEtB6n9nfSnFGfPn1aO3bsUGFhYWheSkqKCgsLVVtbG3FMbW1tWH9JKioqitq/LwQCAUnS5Zdf3mO/trY25eTkKDs7WzNmzNCePXv6orww+/btk8/n0+jRo/XAAw+opaUlal8b9/Xp06e1cuVKPfzwwz0+zMuGff2/mpqa5Pf7w/apx+NRfn5+1H16IZ+RvhAIBORyuZSRkdFjv1iOt3jZtGmThg0bprFjx2ru3Lk6duxY1L5O7u+kCeqjR4+qq6tLmZmZYfMzMzPl9/sjjvH7/TH1j7fu7m498cQTuvnmm3X99ddH7Td27FgtX75ca9eu1cqVK9Xd3a3Jkyfr4MGDfVZrfn6+qqurtW7dOi1btkxNTU269dZboz5y1rZ9LUlr1qzR8ePHNWvWrKh9bNjXkZzZb7Hs0wv5jMTbqVOntGDBApWWlvb4UKNYj7d4mDZtmt5++23V1NTopZde0ubNm1VcXKyurq6I/Z3c30nx9LxkUV5ert27d5/z2ltBQYEKCgpC05MnT9a1116rP/7xj3rhhRfiXaYkqbi4OPTniRMnKj8/Xzk5OXr//fc1e/bsPqmht958800VFxfL5/NF7WPDvk5WnZ2duvfee2WM0bJly3rsa8Pxdv/994f+PGHCBE2cOFFXXXWVNm3apClTpsT1ZyfNGfWQIUM0YMAAtba2hs1vbW2V1+uNOMbr9cbUP57mzZunjz76SBs3bozpka2SNGjQIN1www1qbGyMU3XnlpGRoTFjxkStwaZ9LUnNzc3asGGDHnnkkZjG2bCvJYX2Wyz79EI+I/FyJqSbm5u1fv36mB9teq7jrS+MHj1aQ4YMiVqDk/s7aYI6NTVVkyZNUk1NTWhed3e3ampqws6Ivq+goCCsvyStX78+av94MMZo3rx5Wr16tT777DONGjUq5nV0dXVp165dysrKikOF56etrU379++PWoMN+/r7VqxYoWHDhumOO+6IaZwN+1qSRo0aJa/XG7ZPg8Ggvvjii6j79EI+I/FwJqT37dunDRs26Iorroh5Hec63vrCwYMHdezYsag1OLq/Y7r1aLl3333XuN1uU11dbb7++mvz6KOPmoyMDOP3+40xxjz44IPm6aefDvX//PPPzcCBA82rr75qvvnmG1NRUWEGDRpkdu3a1Wc1z50713g8HrNp0yZz+PDhUDt58mSoz//W/dxzz5lPP/3U7N+/3+zYscPcf//9Ji0tzezZs6fP6v7Vr35lNm3aZJqamsznn39uCgsLzZAhQ8yRI0ci1mzDvj6jq6vLjBgxwixYsOCsZTbt6xMnTpidO3eanTt3GknmtddeMzt37gx9O+LFF180GRkZZu3ateYf//iHmTFjhhk1apT597//HVrHT3/6U7NkyZLQ9Lk+I/Gu+/Tp0+bOO+80w4cPNw0NDWHHfEdHR9S6z3W8xbvuEydOmCeffNLU1taapqYms2HDBnPjjTeaa665xpw6dSpq3U7t76QKamOMWbJkiRkxYoRJTU01eXl5pq6uLrTstttuM2VlZWH933//fTNmzBiTmppqxo8fbz7++OM+rVdSxLZixYqodT/xxBOhbczMzDQ/+9nPTH19fZ/Wfd9995msrCyTmppqrrzySnPfffeZxsbGqDUb0//7+oxPP/3USDJ79+49a5lN+3rjxo0Rj40z9XV3d5tFixaZzMxM43a7zZQpU87appycHFNRURE2r6fPSLzrbmpqinrMb9y4MWrd5zre4l33yZMnzdSpU83QoUPNoEGDTE5OjpkzZ85ZgRuv/c1jTgHAcklzjRoAkhVBDQCWI6gBwHIENQBYjqAGAMsR1ABgOYIaACxHUAOA5QhqALAcQQ0AliOoAcByBDUAWO7/ACp1AOE+76cVAAAAAElFTkSuQmCC",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "predict(flattened_x[320], w1, w2)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "cy1ByUitDa4X",
      "metadata": {
        "id": "cy1ByUitDa4X"
      },
      "source": [
        "## Credits\n",
        "\n",
        "Most of this code was taken from here: https://www.geeksforgeeks.org/implementation-of-neural-network-from-scratch-using-numpy/"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}