backprop.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Yet another explanation of backprop\n",
    "\n",
    "There are many tutorials on backpropagation out there. I've skimmed through a bunch of them, and overall my favorite was [this one](https://www.ritchievink.com/blog/2017/07/10/programming-a-neural-network-from-scratch/) by Ritchie Vink. I preferred because the code examples are of good quality and give a lot of leeway for improvement. [This](https://victorzhou.com/blog/intro-to-neural-networks/) blogpost by Victor Zhou also helped me develop a mental model of what's going on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural networks in a nutshell\n",
    "\n",
    "A neural network is a sequence of layers. Every layer takes as input $x$ and outputs $z$. We can denote this by a function which we call $f$:\n",
    "\n",
    "$$z = f(x)$$\n",
    "\n",
    "Note that the input $x$ can be a set of features, as well as the output from another layer. In the case of a dense layer, $f$ is an affine transformation:\n",
    "\n",
    "$$z = w x + b$$\n",
    "\n",
    "When we stack layers, we are simply chaining functions:\n",
    "\n",
    "$$\\hat{y} = f(f(f(\\dots(f(x)))))$$\n",
    "\n",
    "In the case of dense layers, which are linear, chaining them essentially results in a linear function. This means that even if we have a million dense layers stacked together, we still won't be able to learn non-linear patterns such as the XOR function. To add non-linearity, we add an *activation function* after each layer. Let's call these activation functions $g$. The output from the activation functions will be called $a$.\n",
    "\n",
    "$$a = g(f(x))$$\n",
    "\n",
    "When we stack layers, our final output is:\n",
    "\n",
    "$$\\hat{y} = g(f(g(f(\\dots(g(f(x)))))))$$\n",
    "\n",
    "Of course there are many more flavors of neural networks but that's the general idea. In the case of using dense layers, we're looking to tune the weights $w$ and biases $b$. That's where backpropagation comes in.\n",
    "\n",
    "## Backpropagation\n",
    "\n",
    "First of all, let's get the chain rule out of the way. Say you have a function $f$, a function $g$, and an input $x$. If we compose our functions and apply them to $x$ we get $g(f(x))$. Now say we want to find the derivative of $g$ with respect to $x$. The trick is that there the function $f$ in between $g$ and $x$. In this case we use the chain rule, which gives us:\n",
    "\n",
    "$$\\frac{\\partial g}{\\partial x} = \\frac{\\partial g}{\\partial f} \\times \\frac{\\partial f}{\\partial x}$$\n",
    "\n",
    "In other words, in order to compute $\\frac{\\partial g}{\\partial x}$, we have to compute $\\frac{\\partial g}{\\partial f}$ and $\\frac{\\partial f}{\\partial x}$ and multiply them together. The chain rule is thus just a tool that we can add to our toolkit. In the case of neural networks it's super useful because we're basically just chaining functions. \n",
    "\n",
    "Let's say we're looking at the weights of the final layer. We'll call them $w$. The output of the network is denoted as $\\hat{y}$ whilst the ground truth is $y$. We have a loss function $L$ which indicates the error between $y$ and $\\hat{y}$. To update the weights, we need to calculate the gradient of the loss function with respect to the weights:\n",
    "\n",
    "$$\\frac{\\partial L}{\\partial w}$$\n",
    "\n",
    "In between $w_i$ and $L$, there is the application of the dense layer and the activation function. We can thus apply the chain rule:\n",
    "\n",
    "$$\\frac{\\partial L}{\\partial w} = \\frac{\\partial L}{\\partial a} \\times \\frac{\\partial a}{\\partial z} \\times \\frac{\\partial z}{\\partial w}$$\n",
    "\n",
    "In the case where our loss function is the mean squared error, the derivative is:\n",
    "\n",
    "$$\\frac{\\partial L}{\\partial a} = 2 \\times (a - y)$$\n",
    "\n",
    "For a sigmoid activation function, the derivative is:\n",
    "\n",
    "$$\\frac{\\partial a}{\\partial z} = \\sigma(z) (1 - \\sigma(z))$$\n",
    "\n",
    "where $\\sigma$ is in fact the sigmoid function. In the case of a dense layer, the derivative is:\n",
    "\n",
    "$$\\frac{\\partial z}{\\partial w} = x$$\n",
    "\n",
    "We simply have to multiply all these elements together in order to obtain $\\frac{\\partial L}{\\partial w}$:\n",
    "\n",
    "$$\\frac{\\partial L}{\\partial w} = (2 \\times (a - y)) \\times (\\sigma(z) (1 - \\sigma(z))) \\times x$$\n",
    "\n",
    "Recall that $a$ is the output of the network after having been processed by the activation function. We could have as well called it $\\hat{y}$ because we're looking at the final layer, but we use $a$ because it's more generic and applies to each layer in the network. $z$ is the output of the network *before* being processed by the activation function. Note that implementation wise we thus have to keep both in memory. We can't just obtain $a$ and erase $z$.\n",
    "\n",
    "If we plug in a different activation function and/or a different loss function, then everything will still work as long as each element is differentiable. Note that if we use the identity activation function (which doesn't change the input and has a derivative of 1), then we're simply doing linear regression!\n",
    "\n",
    "Now how about the weights of the penultimate layer (the one just before the last one). Well we \"just\" have write it down using the chain rule. Here goes:\n",
    "\n",
    "$$\\frac{\\partial L}{\\partial w_2} = \\frac{\\partial L}{\\partial a_3} \\times \\frac{\\partial a_3}{\\partial z_3} \\times \\frac{\\partial z_3}{\\partial a_2} \\times \\frac{\\partial a_2}{\\partial z_2} \\times \\frac{\\partial z_2}{\\partial w_2}$$\n",
    "\n",
    "We've indexed the $a$s and $z$s because we're looking at multiple layer. In this case $a_3$ is the output of the 3rd layer (we called it $a$ before) whilst $a_2$ is the output of the 2nd layer. An important thing to notice is that we're using $\\frac{\\partial L}{\\partial a_3} \\times \\frac{\\partial a_3}{\\partial z_3}$, which we already calculated previously. We can exploit this when we implement backpropagation in order to speed up our code but also make it shorter.\n",
    "\n",
    "Here is the gradients for the weights of the 1st layer:\n",
    "\n",
    "$$\\frac{\\partial L}{\\partial w_2} = \\frac{\\partial L}{\\partial a_3} \\times \\frac{\\partial a_3}{\\partial z_3} \\times \\frac{\\partial z_3}{\\partial a_2} \\times \\frac{\\partial a_2}{\\partial z_2} \\times \\frac{\\partial z_2}{\\partial a_1} \\times \\frac{\\partial a_1}{\\partial z_1} \\times \\frac{\\partial z_1}{\\partial w_1}$$\n",
    "\n",
    "Again the first four elements of the product have already been computed.\n",
    "\n",
    "How about the biases $b_i$? Well in a dense layer the derivative with respect to the biases is 1 (it was $x$ with respect to the weights). For the 3rd layer this will result in:\n",
    "\n",
    "$$\\frac{\\partial L}{\\partial b} = (2 \\times (a - y)) \\times (\\sigma(z) (1 - \\sigma(z))) \\times 1$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "out = 30\n",
    "\n",
    "x = {\n",
    "    'user_x': 1,\n",
    "    'item_y': 1\n",
    "}\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.25449581,  0.89660666, -0.33379657, -0.58805437, -0.77278999,\n",
       "        -0.11170595,  0.36491713],\n",
       "       [ 0.12163044,  0.18683425,  0.43287396, -0.43255627, -0.02451537,\n",
       "         0.308671  , -1.69045271],\n",
       "       [ 0.13830588, -0.44750209,  0.63187086,  0.07793882,  0.10308918,\n",
       "        -0.73050789,  0.2066034 ]])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dimensions = (3, 7)\n",
    "W = np.random.randn(dimensions[0], dimensions[1]) / np.sqrt(dimensions[0])\n",
    "W"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.7320508075688772"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sqrt(dimensions[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn import datasets\n",
    "from sklearn import metrics\n",
    "from sklearn import model_selection\n",
    "from sklearn import preprocessing\n",
    "\n",
    "\n",
    "class ReLU:\n",
    "    \"\"\"Rectified Linear Unit (ReLU) activation function.\"\"\"\n",
    "\n",
    "    @staticmethod\n",
    "    def activation(z):\n",
    "        z[z < 0] = 0\n",
    "        return z\n",
    "\n",
    "    @staticmethod\n",
    "    def gradient(z):\n",
    "        z[z < 0] = 0\n",
    "        z[z > 0] = 1\n",
    "        return z\n",
    "\n",
    "\n",
    "class Sigmoid:\n",
    "    \"\"\"Sigmoid activation function.\"\"\"\n",
    "\n",
    "    @staticmethod\n",
    "    def activation(z):\n",
    "        return 1 / (1 + np.exp(-z))\n",
    "\n",
    "    @staticmethod\n",
    "    def gradient(z):\n",
    "        s = Sigmoid.activation(z)\n",
    "        return s * (1 - s)\n",
    "\n",
    "\n",
    "class Identity:\n",
    "    \"\"\"Identity activation function.\"\"\"\n",
    "\n",
    "    @staticmethod\n",
    "    def activation(z):\n",
    "        return z\n",
    "\n",
    "    @staticmethod\n",
    "    def gradient(z):\n",
    "        return np.ones_like(z)\n",
    "\n",
    "\n",
    "class MSE:\n",
    "    \"\"\"Mean Squared Error (MSE) loss function.\"\"\"\n",
    "\n",
    "    @staticmethod\n",
    "    def loss(y_true, y_pred):\n",
    "        return np.mean((y_pred - y_true) ** 2)\n",
    "\n",
    "    @staticmethod\n",
    "    def gradient(y_true, y_pred):\n",
    "        return 2 * (y_pred - y_true)\n",
    "\n",
    "\n",
    "class SGD:\n",
    "    \"\"\"Stochastic Gradient Descent (SGD).\"\"\"\n",
    "\n",
    "    def __init__(self, learning_rate):\n",
    "        self.learning_rate = learning_rate\n",
    "\n",
    "    def step(self, weights, gradients):\n",
    "        weights -= self.learning_rate * gradients\n",
    "\n",
    "\n",
    "class NN:\n",
    "    \"\"\"\n",
    "\n",
    "    Parameters:\n",
    "        dimensions (tuples of ints of length n_layers)\n",
    "\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, dimensions, activations, loss, optimizer):\n",
    "        self.n_layers = len(dimensions)\n",
    "        self.loss = loss\n",
    "        self.optimizer = optimizer\n",
    "\n",
    "        # Weights and biases are initiated by index. For a one hidden layer net you will have a w[1] and w[2]\n",
    "        self.w = {}\n",
    "        self.b = {}\n",
    "\n",
    "        # Activations are also initiated by index. For the example we will have activations[2] and activations[3]\n",
    "        self.activations = {}\n",
    "        for i in range(len(dimensions) - 1):\n",
    "            self.w[i + 1] = np.random.randn(dimensions[i], dimensions[i + 1]) / np.sqrt(dimensions[i])\n",
    "            self.b[i + 1] = np.zeros(dimensions[i + 1])\n",
    "            self.activations[i + 2] = activations[i]\n",
    "\n",
    "    def _feed_forward(self, X):\n",
    "        \"\"\"Executes a forward pass through the neural network.\n",
    "\n",
    "        This will return the state at each layer of the network, which includes the output of the\n",
    "        network.\n",
    "\n",
    "        Parameters:\n",
    "            X (array of shape (batch_size, n_features))\n",
    "\n",
    "        \"\"\"\n",
    "\n",
    "        # z = w(x) + b\n",
    "        z = {}\n",
    "\n",
    "        # a = f(z)\n",
    "        a = {1: X}  # First layer has no activations as input\n",
    "\n",
    "        for i in range(2, self.n_layers + 1):\n",
    "            z[i] = np.dot(a[i - 1], self.w[i - 1]) + self.b[i - 1]\n",
    "            a[i] = self.activations[i].activation(z[i])\n",
    "\n",
    "        return z, a\n",
    "\n",
    "    def _backprop(self, z, a, y_true):\n",
    "        \"\"\"Backpropagation.\n",
    "\n",
    "        Parameters:\n",
    "            z (dict of length n_layers - 1):\n",
    "\n",
    "                z = {\n",
    "                    2: w1 * x + b1\n",
    "                    3: w2 * (w1 * x + b1) + b2\n",
    "                    4: w3 * (w2 * (w1 * x + b1) + b2) + b3\n",
    "                    ...\n",
    "                }\n",
    "\n",
    "            a (dict of length n_layers):\n",
    "\n",
    "                a = {\n",
    "                    1: x,\n",
    "                    2: f(w1 * x + b1)\n",
    "                    3: f(w2 * (w1 * x + b1) + b2)\n",
    "                    4: f(w3 * (w2 * (w1 * x + b1) + b2) + b3)\n",
    "                    ...\n",
    "                }\n",
    "\n",
    "            y_true (array of shape (batch_size, n_targets))\n",
    "\n",
    "        \"\"\"\n",
    "\n",
    "        # Determine the partial derivative and delta for the output layer\n",
    "        y_pred = a[self.n_layers]\n",
    "        final_activation = self.activations[self.n_layers]\n",
    "        delta = self.loss.gradient(y_true, y_pred) * final_activation.gradient(y_pred)\n",
    "        dw = np.dot(a[self.n_layers - 1].T, delta)\n",
    "\n",
    "        update_params = {\n",
    "            self.n_layers - 1: (dw, delta)\n",
    "        }\n",
    "\n",
    "        # Go through the layers in reverse order\n",
    "        for i in range(self.n_layers - 2, 0, -1):\n",
    "            delta = np.dot(delta, self.w[i + 1].T) * self.activations[i + 1].gradient(z[i + 1])\n",
    "            dw = np.dot(a[i].T, delta)\n",
    "            update_params[i] = (dw, delta)\n",
    "\n",
    "        # Update the parameters\n",
    "        for k, (dw, delta) in update_params.items():\n",
    "            self.optimizer.step(weights=self.w[k], gradients=dw)\n",
    "            self.optimizer.step(weights=self.b[k], gradients=np.mean(delta, axis=0))\n",
    "\n",
    "    def fit(self, X, y, epochs, batch_size, print_every=np.inf):\n",
    "        \"\"\"Trains the neural network.\n",
    "\n",
    "        Parameters:\n",
    "            X (array of shape (n_samples, n_features))\n",
    "            y (array of shape (n_samples, n_targets))\n",
    "            epochs (int)\n",
    "            batch_size (int)\n",
    "\n",
    "        \"\"\"\n",
    "\n",
    "        # As a convention we expect y to be 2D, even if there is only one target to predict\n",
    "        if y.ndim == 1:\n",
    "            y = np.expand_dims(y, axis=1)\n",
    "\n",
    "        # Go through the epochs\n",
    "        for i in range(epochs):\n",
    "\n",
    "            # Shuffle the data\n",
    "            idx = np.arange(X.shape[0])\n",
    "            np.random.shuffle(idx)\n",
    "            x_ = X[idx]\n",
    "            y_ = y[idx]\n",
    "\n",
    "            # Iterate over the training data in mini-batches\n",
    "            for j in range(X.shape[0] // batch_size):\n",
    "                start = j * batch_size\n",
    "                stop = (j + 1) * batch_size\n",
    "                z, a = self._feed_forward(x_[start:stop])\n",
    "                self._backprop(z, a, y_[start:stop])\n",
    "\n",
    "            # Display the performance every print_every eooch\n",
    "            if (i + 1) % print_every == 0:\n",
    "                y_pred = self.predict(X)\n",
    "                print(f'[{i+1}] train loss: {self.loss.loss(y, y_pred)}')\n",
    "\n",
    "    def predict(self, X):\n",
    "        \"\"\"Predicts an output for each sample in X.\n",
    "\n",
    "        Parameters:\n",
    "            X (array of shape (n_samples, n_features))\n",
    "\n",
    "        \"\"\"\n",
    "        _, a = self._feed_forward(X)\n",
    "        return a[self.n_layers]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Credit card."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "X_train = pd.read_csv('train.csv')\n",
    "y_train = X_train.pop('Class').to_numpy()\n",
    "X_train = X_train.to_numpy()\n",
    "X_train = preprocessing.scale(X_train)\n",
    "\n",
    "X_test = pd.read_csv('test.csv')\n",
    "y_test = X_test.pop('Class').to_numpy()\n",
    "X_test = X_test.to_numpy()\n",
    "X_test = preprocessing.scale(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "0.9482957993578214, 10.250 seconds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] train loss: 0.0008219300025216478\n",
      "[2] train loss: 0.000721527485585961\n",
      "[3] train loss: 0.0006945727754202831\n",
      "[4] train loss: 0.000671939618738007\n",
      "[5] train loss: 0.0006518139949863301\n",
      "0.9482957993578214\n",
      " "
     ]
    },
    {
     "data": {
      "text/plain": [
       "         3376532 function calls (3376526 primitive calls) in 11.026 seconds\n",
       "\n",
       "   Ordered by: internal time\n",
       "\n",
       "   ncalls  tottime  percall  cumtime  percall filename:lineno(function)\n",
       "437580/437574    1.337    0.000    3.656    0.000 {built-in method numpy.core._multiarray_umath.implement_array_function}\n",
       "    62500    1.075    0.000    6.976    0.000 <ipython-input-3-1e1ec5bcfaa3>:117(_backprop)\n",
       "   250000    1.048    0.000    1.048    0.000 <ipython-input-3-1e1ec5bcfaa3>:66(step)\n",
       "    62506    1.008    0.000    1.008    0.000 <ipython-input-3-1e1ec5bcfaa3>:11(activation)\n",
       "   125005    1.004    0.000    2.049    0.000 _methods.py:134(_mean)\n",
       "    62506    0.911    0.000    3.099    0.000 <ipython-input-3-1e1ec5bcfaa3>:94(_feed_forward)\n",
       "        1    0.846    0.846   10.928   10.928 <ipython-input-3-1e1ec5bcfaa3>:165(fit)\n",
       "   125006    0.806    0.000    0.806    0.000 <ipython-input-3-1e1ec5bcfaa3>:26(activation)\n",
       "    62500    0.698    0.000    0.698    0.000 <ipython-input-3-1e1ec5bcfaa3>:16(gradient)\n",
       "   125019    0.652    0.000    0.652    0.000 {method 'reduce' of 'numpy.ufunc' objects}\n",
       "    62500    0.293    0.000    0.293    0.000 <ipython-input-3-1e1ec5bcfaa3>:55(gradient)\n",
       "   125005    0.236    0.000    2.284    0.000 fromnumeric.py:3153(mean)\n",
       "   312512    0.214    0.000    1.503    0.000 <__array_function__ internals>:2(dot)\n",
       "    62500    0.194    0.000    0.534    0.000 <ipython-input-3-1e1ec5bcfaa3>:30(gradient)\n",
       "   125005    0.153    0.000    0.199    0.000 _methods.py:50(_count_reduce_items)\n",
       "   125005    0.120    0.000    2.525    0.000 <__array_function__ internals>:2(mean)\n",
       "   250105    0.069    0.000    0.069    0.000 {built-in method builtins.isinstance}\n",
       "   125100    0.061    0.000    0.061    0.000 {built-in method numpy.array}\n",
       "   125046    0.058    0.000    0.118    0.000 _asarray.py:88(asanyarray)\n",
       "   250025    0.052    0.000    0.052    0.000 {built-in method builtins.issubclass}\n",
       "   312512    0.052    0.000    0.052    0.000 multiarray.py:707(dot)\n",
       "        5    0.049    0.010    0.050    0.010 {method 'shuffle' of 'numpy.random.mtrand.RandomState' objects}\n",
       "   125005    0.023    0.000    0.023    0.000 fromnumeric.py:3149(_mean_dispatcher)\n",
       "        1    0.017    0.017    0.017    0.017 {method 'argsort' of 'numpy.ndarray' objects}\n",
       "    62508    0.014    0.000    0.014    0.000 {method 'items' of 'dict' objects}\n",
       "       10    0.013    0.001    0.013    0.001 {method 'sort' of 'numpy.ndarray' objects}\n",
       "        5    0.005    0.001    0.007    0.001 <ipython-input-3-1e1ec5bcfaa3>:51(loss)\n",
       "        5    0.003    0.001    0.003    0.001 {built-in method numpy.arange}\n",
       "        1    0.002    0.002    0.026    0.026 _ranking.py:478(_binary_clf_curve)\n",
       "        9    0.001    0.000    0.015    0.002 arraysetops.py:297(_unique1d)\n",
       "        2    0.001    0.000    0.001    0.000 {method 'cumsum' of 'numpy.ndarray' objects}\n",
       "        9    0.001    0.000    0.001    0.000 {method 'flatten' of 'numpy.ndarray' objects}\n",
       "       17    0.001    0.000    0.001    0.000 socket.py:337(send)\n",
       "        5    0.001    0.000    0.001    0.000 function_base.py:1147(diff)\n",
       "        1    0.001    0.001    0.001    0.001 {method 'searchsorted' of 'numpy.ndarray' objects}\n",
       "        1    0.001    0.001   11.026   11.026 <string>:3(<module>)\n",
       "        1    0.000    0.000    0.027    0.027 _ranking.py:666(roc_curve)\n",
       "        1    0.000    0.000   11.026   11.026 {built-in method builtins.exec}\n",
       "        1    0.000    0.000    0.000    0.000 {built-in method scipy.sparse._sparsetools.csr_todense}\n",
       "        1    0.000    0.000    0.006    0.006 _label.py:541(label_binarize)\n",
       "        2    0.000    0.000    0.000    0.000 {method 'randn' of 'numpy.random.mtrand.RandomState' objects}\n",
       "        1    0.000    0.000    0.000    0.000 arraysetops.py:483(in1d)\n",
       "       10    0.000    0.000    0.000    0.000 _internal.py:262(__array_interface__)\n",
       "        1    0.000    0.000    0.000    0.000 <ipython-input-3-1e1ec5bcfaa3>:78(__init__)\n",
       "       10    0.000    0.000    0.001    0.000 _internal.py:274(_get_void_ptr)\n",
       "        4    0.000    0.000    0.000    0.000 {built-in method numpy.zeros}\n",
       "        5    0.000    0.000    0.000    0.000 index_tricks.py:318(__getitem__)\n",
       "       17    0.000    0.000    0.001    0.000 iostream.py:195(schedule)\n",
       "       10    0.000    0.000    0.000    0.000 __init__.py:497(cast)\n",
       "       12    0.000    0.000    0.001    0.000 iostream.py:382(write)\n",
       "        7    0.000    0.000    0.000    0.000 {method 'ravel' of 'numpy.ndarray' objects}\n",
       "        5    0.000    0.000    0.000    0.000 validation.py:37(_assert_all_finite)\n",
       "        4    0.000    0.000    0.009    0.002 multiclass.py:172(type_of_target)\n",
       "        3    0.000    0.000    0.001    0.000 validation.py:339(check_array)\n",
       "        7    0.000    0.000    0.000    0.000 validation.py:136(_num_samples)\n",
       "        6    0.000    0.000    0.001    0.000 {built-in method builtins.print}\n",
       "       10    0.000    0.000    0.000    0.000 {method 'from_buffer' of '_ctypes.PyCArrayType' objects}\n",
       "       10    0.000    0.000    0.001    0.000 _internal.py:291(__init__)\n",
       "        2    0.000    0.000    0.001    0.000 sputils.py:120(get_index_dtype)\n",
       "       60    0.000    0.000    0.000    0.000 {built-in method builtins.hasattr}\n",
       "       17    0.000    0.000    0.000    0.000 threading.py:1092(is_alive)\n",
       "        1    0.000    0.000    0.000    0.000 {method 'fill' of 'numpy.ndarray' objects}\n",
       "        8    0.000    0.000    0.000    0.000 fromnumeric.py:73(_wrapreduction)\n",
       "        1    0.000    0.000    0.000    0.000 compressed.py:138(check_format)\n",
       "       10    0.000    0.000    0.000    0.000 {built-in method _ctypes.pointer}\n",
       "        9    0.000    0.000    0.000    0.000 {built-in method numpy.empty}\n",
       "       12    0.000    0.000    0.000    0.000 iostream.py:307(_is_master_process)\n",
       "        9    0.000    0.000    0.015    0.002 arraysetops.py:151(unique)\n",
       "        5    0.000    0.000    0.000    0.000 numerictypes.py:602(find_common_type)\n",
       "       39    0.000    0.000    0.000    0.000 _asarray.py:16(asarray)\n",
       "       17    0.000    0.000    0.000    0.000 threading.py:1050(_wait_for_tstate_lock)\n",
       "       11    0.000    0.000    0.000    0.000 {built-in method _abc._abc_instancecheck}\n",
       "        1    0.000    0.000    0.000    0.000 function_base.py:3987(trapz)\n",
       "        1    0.000    0.000    0.000    0.000 numeric.py:2256(within_tol)\n",
       "       17    0.000    0.000    0.000    0.000 {method 'acquire' of '_thread.lock' objects}\n",
       "       12    0.000    0.000    0.000    0.000 iostream.py:320(_schedule_flush)\n",
       "        1    0.000    0.000    0.042    0.042 _ranking.py:246(roc_auc_score)\n",
       "        1    0.000    0.000    0.000    0.000 _ranking.py:41(auc)\n",
       "        1    0.000    0.000    0.001    0.001 compressed.py:30(__init__)\n",
       "       17    0.000    0.000    0.000    0.000 iostream.py:93(_event_pipe)\n",
       "        5    0.000    0.000    0.000    0.000 validation.py:754(column_or_1d)\n",
       "       74    0.000    0.000    0.000    0.000 {built-in method builtins.len}\n",
       "       12    0.000    0.000    0.000    0.000 {built-in method posix.getpid}\n",
       "        1    0.000    0.000    0.000    0.000 shape_base.py:512(expand_dims)\n",
       "        1    0.000    0.000    0.001    0.001 extmath.py:810(stable_cumsum)\n",
       "        4    0.000    0.000    0.000    0.000 getlimits.py:497(__init__)\n",
       "        1    0.000    0.000    0.000    0.000 numeric.py:2179(isclose)\n",
       "        3    0.000    0.000    0.000    0.000 warnings.py:181(_add_filter)\n",
       "        3    0.000    0.000    0.000    0.000 warnings.py:474(__enter__)\n",
       "        1    0.000    0.000    0.000    0.000 compressed.py:1139(prune)\n",
       "        4    0.000    0.000    0.000    0.000 _internal.py:865(npy_ctypes_check)\n",
       "        4    0.000    0.000    0.018    0.005 fromnumeric.py:55(_wrapfunc)\n",
       "        7    0.000    0.000    0.000    0.000 {method 'astype' of 'numpy.ndarray' objects}\n",
       "        4    0.000    0.000    0.000    0.000 {method 'reshape' of 'numpy.ndarray' objects}\n",
       "       11    0.000    0.000    0.000    0.000 abc.py:137(__instancecheck__)\n",
       "        6    0.000    0.000    0.693    0.115 <ipython-input-3-1e1ec5bcfaa3>:201(predict)\n",
       "        9    0.000    0.000    0.015    0.002 <__array_function__ internals>:2(unique)\n",
       "        2    0.000    0.000    0.000    0.000 _ufunc_config.py:39(seterr)\n",
       "        3    0.000    0.000    0.000    0.000 fromnumeric.py:2045(sum)\n",
       "        5    0.000    0.000    0.000    0.000 numerictypes.py:365(issubdtype)\n",
       "        1    0.000    0.000    0.000    0.000 compressed.py:1021(toarray)\n",
       "        5    0.000    0.000    0.000    0.000 fromnumeric.py:1648(ravel)\n",
       "       10    0.000    0.000    0.000    0.000 numerictypes.py:293(issubclass_)\n",
       "        9    0.000    0.000    0.000    0.000 arraysetops.py:138(_unpack_tuple)\n",
       "        7    0.000    0.000    0.000    0.000 {built-in method _abc._abc_subclasscheck}\n",
       "        4    0.000    0.000    0.000    0.000 multiclass.py:109(is_multilabel)\n",
       "       10    0.000    0.000    0.000    0.000 numerictypes.py:578(_can_coerce_all)\n",
       "        2    0.000    0.000    0.000    0.000 extmath.py:681(_safe_accumulator_op)\n",
       "        1    0.000    0.000    0.000    0.000 numeric.py:2299(array_equal)\n",
       "       35    0.000    0.000    0.000    0.000 {method 'append' of 'list' objects}\n",
       "        3    0.000    0.000    0.000    0.000 {method 'all' of 'numpy.generic' objects}\n",
       "       17    0.000    0.000    0.000    0.000 threading.py:507(is_set)\n",
       "        5    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:416(parent)\n",
       "        1    0.000    0.000    0.000    0.000 sputils.py:266(check_shape)\n",
       "        1    0.000    0.000    0.000    0.000 <__array_function__ internals>:2(expand_dims)\n",
       "        2    0.000    0.000    0.000    0.000 validation.py:197(check_consistent_length)\n",
       "       10    0.000    0.000    0.000    0.000 _internal.py:340(data)\n",
       "        2    0.000    0.000    0.000    0.000 _ufunc_config.py:139(geterr)\n",
       "        3    0.000    0.000    0.000    0.000 {method 'remove' of 'list' objects}\n",
       "        1    0.000    0.000    0.030    0.030 _ranking.py:218(_binary_roc_auc_score)\n",
       "        4    0.000    0.000    0.000    0.000 compressed.py:110(getnnz)\n",
       "       10    0.000    0.000    0.000    0.000 _internal.py:259(__init__)\n",
       "        8    0.000    0.000    0.000    0.000 fromnumeric.py:74(<dictcomp>)\n",
       "        6    0.000    0.000    0.000    0.000 <__array_function__ internals>:2(concatenate)\n",
       "        3    0.000    0.000    0.000    0.000 warnings.py:453(__init__)\n",
       "        1    0.000    0.000    0.032    0.032 _base.py:23(_average_binary_score)\n",
       "        1    0.000    0.000    0.000    0.000 shape_base.py:24(atleast_1d)\n",
       "        5    0.000    0.000    0.000    0.000 <__array_function__ internals>:2(ravel)\n",
       "        5    0.000    0.000    0.000    0.000 <__array_function__ internals>:2(shape)\n",
       "        5    0.000    0.000    0.000    0.000 <__array_function__ internals>:2(ndim)\n",
       "       10    0.000    0.000    0.000    0.000 {built-in method builtins.getattr}\n",
       "       10    0.000    0.000    0.000    0.000 {method 'index' of 'list' objects}\n",
       "        7    0.000    0.000    0.000    0.000 base.py:1189(isspmatrix)\n",
       "        3    0.000    0.000    0.000    0.000 sputils.py:279(<genexpr>)\n",
       "        5    0.000    0.000    0.001    0.000 <__array_function__ internals>:2(diff)\n",
       "        1    0.000    0.000    0.000    0.000 shape_base.py:285(hstack)\n",
       "        2    0.000    0.000    0.000    0.000 fromnumeric.py:2189(any)\n",
       "        3    0.000    0.000    0.000    0.000 fromnumeric.py:2277(all)\n",
       "        5    0.000    0.000    0.000    0.000 numerictypes.py:654(<listcomp>)\n",
       "        3    0.000    0.000    0.000    0.000 warnings.py:165(simplefilter)\n",
       "        3    0.000    0.000    0.000    0.000 warnings.py:493(__exit__)\n",
       "        2    0.000    0.000    0.000    0.000 validation.py:208(<listcomp>)\n",
       "        1    0.000    0.000    0.000    0.000 _ufunc_config.py:437(__init__)\n",
       "        7    0.000    0.000    0.000    0.000 abc.py:141(__subclasscheck__)\n",
       "       17    0.000    0.000    0.000    0.000 {method 'append' of 'collections.deque' objects}\n",
       "        1    0.000    0.000    0.000    0.000 base.py:70(__init__)\n",
       "        1    0.000    0.000    0.000    0.000 sputils.py:209(isshape)\n",
       "        5    0.000    0.000    0.000    0.000 _config.py:13(get_config)\n",
       "        3    0.000    0.000    0.000    0.000 <__array_function__ internals>:2(all)\n",
       "        1    0.000    0.000    0.000    0.000 {method 'take' of 'numpy.ndarray' objects}\n",
       "        5    0.000    0.000    0.000    0.000 {method 'rpartition' of 'str' objects}\n",
       "        3    0.000    0.000    0.000    0.000 {method 'insert' of 'list' objects}\n",
       "        3    0.000    0.000    0.000    0.000 validation.py:332(_ensure_no_complex_data)\n",
       "        3    0.000    0.000    0.000    0.000 <__array_function__ internals>:2(sum)\n",
       "        2    0.000    0.000    0.000    0.000 <__array_function__ internals>:2(any)\n",
       "        2    0.000    0.000    0.001    0.000 <__array_function__ internals>:2(cumsum)\n",
       "        1    0.000    0.000    0.000    0.000 fromnumeric.py:826(sort)\n",
       "        2    0.000    0.000    0.001    0.000 fromnumeric.py:2358(cumsum)\n",
       "        5    0.000    0.000    0.000    0.000 {built-in method numpy.core._multiarray_umath.normalize_axis_index}\n",
       "        2    0.000    0.000    0.000    0.000 {built-in method builtins.abs}\n",
       "        1    0.000    0.000    0.000    0.000 base.py:1175(_process_toarray_args)\n",
       "        1    0.000    0.000    0.000    0.000 sputils.py:92(to_native)\n",
       "        9    0.000    0.000    0.000    0.000 arraysetops.py:146(_unique_dispatcher)\n",
       "        2    0.000    0.000    0.000    0.000 validation.py:68(assert_all_finite)\n",
       "        2    0.000    0.000    0.000    0.000 getlimits.py:508(min)\n",
       "        5    0.000    0.000    0.000    0.000 fromnumeric.py:1856(shape)\n",
       "        4    0.000    0.000    0.000    0.000 _methods.py:47(_all)\n",
       "        1    0.000    0.000    0.000    0.000 {method 'copy' of 'numpy.ndarray' objects}\n",
       "        2    0.000    0.000    0.000    0.000 {method 'max' of 'numpy.ndarray' objects}\n",
       "        2    0.000    0.000    0.000    0.000 {built-in method numpy.seterrobj}\n",
       "        2    0.000    0.000    0.001    0.001 <__array_function__ internals>:2(where)\n",
       "        5    0.000    0.000    0.000    0.000 numerictypes.py:655(<listcomp>)\n",
       "        5    0.000    0.000    0.000    0.000 {method 'copy' of 'dict' objects}\n",
       "        2    0.000    0.000    0.000    0.000 _util.py:129(_prune_array)\n",
       "        1    0.000    0.000    0.000    0.000 <__array_function__ internals>:2(atleast_1d)\n",
       "        1    0.000    0.000    0.000    0.000 <__array_function__ internals>:2(sort)\n",
       "        1    0.000    0.000    0.000    0.000 <__array_function__ internals>:2(isclose)\n",
       "        1    0.000    0.000    0.001    0.001 fromnumeric.py:1229(searchsorted)\n",
       "        2    0.000    0.000    0.000    0.000 _methods.py:32(_amin)\n",
       "        2    0.000    0.000    0.000    0.000 {method 'min' of 'numpy.ndarray' objects}\n",
       "        4    0.000    0.000    0.000    0.000 <__array_function__ internals>:2(can_cast)\n",
       "        5    0.000    0.000    0.000    0.000 <frozen importlib._bootstrap>:1009(_handle_fromlist)\n",
       "        1    0.000    0.000    0.000    0.000 <__array_function__ internals>:2(hstack)\n",
       "        1    0.000    0.000    0.017    0.017 <__array_function__ internals>:2(argsort)\n",
       "        1    0.000    0.000    0.000    0.000 _ufunc_config.py:441(__enter__)\n",
       "        1    0.000    0.000    0.000    0.000 _ufunc_config.py:446(__exit__)\n",
       "        1    0.000    0.000    0.017    0.017 fromnumeric.py:978(argsort)\n",
       "        5    0.000    0.000    0.000    0.000 fromnumeric.py:2986(ndim)\n",
       "        1    0.000    0.000    0.000    0.000 {method 'newbyteorder' of 'numpy.dtype' objects}\n",
       "        4    0.000    0.000    0.000    0.000 {built-in method numpy.geterrobj}\n",
       "        1    0.000    0.000    0.000    0.000 <__array_function__ internals>:2(result_type)\n",
       "        4    0.000    0.000    0.000    0.000 multiarray.py:469(can_cast)\n",
       "        1    0.000    0.000    0.000    0.000 {built-in method builtins.max}\n",
       "        1    0.000    0.000    0.000    0.000 <ipython-input-3-1e1ec5bcfaa3>:63(__init__)\n",
       "        1    0.000    0.000    0.000    0.000 data.py:22(__init__)\n",
       "        1    0.000    0.000    0.000    0.000 <__array_function__ internals>:2(in1d)\n",
       "        5    0.000    0.000    0.000    0.000 function_base.py:1143(_diff_dispatcher)\n",
       "        1    0.000    0.000    0.000    0.000 shape_base.py:208(_arrays_for_stack_dispatcher)\n",
       "        1    0.000    0.000    0.000    0.000 shape_base.py:219(_vhstack_dispatcher)\n",
       "        1    0.000    0.000    0.001    0.001 <__array_function__ internals>:2(searchsorted)\n",
       "        1    0.000    0.000    0.000    0.000 <__array_function__ internals>:2(array_equal)\n",
       "        5    0.000    0.000    0.000    0.000 fromnumeric.py:1852(_shape_dispatcher)\n",
       "        2    0.000    0.000    0.000    0.000 _methods.py:28(_amax)\n",
       "        1    0.000    0.000    0.000    0.000 {method 'all' of 'numpy.ndarray' objects}\n",
       "        1    0.000    0.000    0.000    0.000 {method 'sum' of 'numpy.ndarray' objects}\n",
       "        1    0.000    0.000    0.000    0.000 <__array_function__ internals>:2(empty_like)\n",
       "        6    0.000    0.000    0.000    0.000 multiarray.py:145(concatenate)\n",
       "        9    0.000    0.000    0.000    0.000 {built-in method _warnings._filters_mutated}\n",
       "        1    0.000    0.000    0.000    0.000 {method 'pop' of 'dict' objects}\n",
       "        4    0.000    0.000    0.000    0.000 base.py:84(get_shape)\n",
       "        4    0.000    0.000    0.000    0.000 base.py:242(nnz)\n",
       "        1    0.000    0.000    0.000    0.000 arraysetops.py:479(_in1d_dispatcher)\n",
       "        1    0.000    0.000    0.000    0.000 shape_base.py:508(_expand_dims_dispatcher)\n",
       "        1    0.000    0.000    0.000    0.000 function_base.py:3983(_trapz_dispatcher)\n",
       "        1    0.000    0.000    0.000    0.000 <__array_function__ internals>:2(trapz)\n",
       "        2    0.000    0.000    0.000    0.000 getlimits.py:521(max)\n",
       "        1    0.000    0.000    0.000    0.000 numeric.py:2175(_isclose_dispatcher)\n",
       "        1    0.000    0.000    0.000    0.000 numeric.py:2295(_array_equal_dispatcher)\n",
       "        1    0.000    0.000    0.000    0.000 fromnumeric.py:974(_argsort_dispatcher)\n",
       "        1    0.000    0.000    0.000    0.000 fromnumeric.py:1225(_searchsorted_dispatcher)\n",
       "        3    0.000    0.000    0.000    0.000 fromnumeric.py:2040(_sum_dispatcher)\n",
       "        2    0.000    0.000    0.000    0.000 fromnumeric.py:2185(_any_dispatcher)\n",
       "        3    0.000    0.000    0.000    0.000 fromnumeric.py:2273(_all_dispatcher)\n",
       "        5    0.000    0.000    0.000    0.000 fromnumeric.py:2982(_ndim_dispatcher)\n",
       "        1    0.000    0.000    0.000    0.000 _methods.py:36(_sum)\n",
       "        1    0.000    0.000    0.000    0.000 numerictypes.py:587(<listcomp>)\n",
       "        1    0.000    0.000    0.000    0.000 multiarray.py:77(empty_like)\n",
       "        2    0.000    0.000    0.000    0.000 multiarray.py:312(where)\n",
       "        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}\n",
       "        2    0.000    0.000    0.000    0.000 {built-in method _operator.index}\n",
       "        1    0.000    0.000    0.000    0.000 data.py:25(_get_dtype)\n",
       "        1    0.000    0.000    0.000    0.000 csr.py:160(tocsr)\n",
       "        5    0.000    0.000    0.000    0.000 csr.py:228(_swap)\n",
       "        1    0.000    0.000    0.000    0.000 shape_base.py:20(_atleast_1d_dispatcher)\n",
       "        1    0.000    0.000    0.000    0.000 fromnumeric.py:822(_sort_dispatcher)\n",
       "        5    0.000    0.000    0.000    0.000 fromnumeric.py:1644(_ravel_dispatcher)\n",
       "        2    0.000    0.000    0.000    0.000 fromnumeric.py:2354(_cumsum_dispatcher)\n",
       "        1    0.000    0.000    0.000    0.000 multiarray.py:635(result_type)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%prun\n",
    "\n",
    "nn = NN(\n",
    "    dimensions=(30, 30, 1),\n",
    "    activations=(ReLU, Sigmoid),\n",
    "    loss=MSE,\n",
    "    optimizer=SGD(learning_rate=1e-3)\n",
    ")\n",
    "nn.fit(X_train, y_train, epochs=5, batch_size=16, print_every=1)\n",
    "y_pred = nn.predict(X_test)\n",
    "print(metrics.roc_auc_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Boston."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[10] train loss: 11.796707532482444\n",
      "[20] train loss: 9.700941500985953\n",
      "[30] train loss: 9.023612069639709\n",
      "2.505495393489851\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(1)\n",
    "\n",
    "X, y = datasets.load_boston(return_X_y=True)\n",
    "X = preprocessing.scale(X)\n",
    "\n",
    "# Split into train and test\n",
    "X_train, X_test, y_train, y_test = model_selection.train_test_split(\n",
    "    X, y,\n",
    "    test_size=.3,\n",
    "    shuffle=True,\n",
    "    random_state=42\n",
    ")\n",
    "\n",
    "nn = NN(\n",
    "    dimensions=(13, 10, 1),\n",
    "    activations=(ReLU, Identity),\n",
    "    loss=MSE,\n",
    "    optimizer=SGD(learning_rate=1e-3)\n",
    ")\n",
    "nn.fit(X_train, y_train, epochs=30, batch_size=8, print_every=10)\n",
    "\n",
    "y_pred = nn.predict(X_test)\n",
    "\n",
    "print(metrics.mean_absolute_error(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Digits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[10] train loss: 0.008308476136280957\n",
      "[20] train loss: 0.004984925198988307\n",
      "[30] train loss: 0.004102445263740696\n",
      "[40] train loss: 0.0029634369443098745\n",
      "[50] train loss: 0.0018708680417568045\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00        53\n",
      "           1       0.96      0.98      0.97        50\n",
      "           2       0.94      1.00      0.97        47\n",
      "           3       0.96      0.96      0.96        54\n",
      "           4       0.98      1.00      0.99        60\n",
      "           5       0.94      0.97      0.96        66\n",
      "           6       0.98      0.98      0.98        53\n",
      "           7       1.00      0.98      0.99        55\n",
      "           8       1.00      0.93      0.96        43\n",
      "           9       0.98      0.93      0.96        59\n",
      "\n",
      "    accuracy                           0.97       540\n",
      "   macro avg       0.98      0.97      0.97       540\n",
      "weighted avg       0.97      0.97      0.97       540\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(1)\n",
    "\n",
    "X, y = datasets.load_digits(return_X_y=True)\n",
    "\n",
    "# One-hot encode y\n",
    "y = np.eye(10)[y]\n",
    "\n",
    "# Split into train and test\n",
    "X_train, X_test, y_train, y_test = model_selection.train_test_split(\n",
    "    X, y,\n",
    "    test_size=.3,\n",
    "    shuffle=True,\n",
    "    random_state=42\n",
    ")\n",
    "\n",
    "nn = NN(\n",
    "    dimensions=(64, 15, 10),\n",
    "    activations=(ReLU, Sigmoid),\n",
    "    loss=MSE,\n",
    "    optimizer=SGD(learning_rate=1e-3)\n",
    ")\n",
    "nn.fit(X_train, y_train, epochs=50, batch_size=16, print_every=10)\n",
    "\n",
    "y_pred = nn.predict(X_test)\n",
    "\n",
    "print(metrics.classification_report(y_test.argmax(1), y_pred.argmax(1)))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}