assignment2/BatchNormalization.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This mounts your Google Drive to the Colab VM.\n",
    "from google.colab import drive\n",
    "drive.mount('/content/drive')\n",
    "\n",
    "# TODO: Enter the foldername in your Drive where you have saved the unzipped\n",
    "# assignment folder, e.g. 'cs231n/assignments/assignment2/'\n",
    "FOLDERNAME = None\n",
    "assert FOLDERNAME is not None, \"[!] Enter the foldername.\"\n",
    "\n",
    "# Now that we've mounted your Drive, this ensures that\n",
    "# the Python interpreter of the Colab VM can load\n",
    "# python files from within it.\n",
    "import sys\n",
    "sys.path.append('/content/drive/My Drive/{}'.format(FOLDERNAME))\n",
    "\n",
    "# This downloads the CIFAR-10 dataset to your Drive\n",
    "# if it doesn't already exist.\n",
    "%cd /content/drive/My\\ Drive/$FOLDERNAME/cs231n/datasets/\n",
    "!bash get_datasets.sh\n",
    "%cd /content/drive/My\\ Drive/$FOLDERNAME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-title"
    ]
   },
   "source": [
    "# Batch Normalization\n",
    "One way to make deep networks easier to train is to use more sophisticated optimization procedures such as SGD+momentum, RMSProp, or Adam. Another strategy is to change the architecture of the network to make it easier to train. One idea along these lines is batch normalization, proposed by [1] in 2015.\n",
    "\n",
    "To understand the goal of batch normalization, it is important to first recognize that machine learning methods tend to perform better with input data consisting of uncorrelated features with zero mean and unit variance. When training a neural network, we can preprocess the data before feeding it to the network to explicitly decorrelate its features. This will ensure that the first layer of the network sees data that follows a nice distribution. However, even if we preprocess the input data, the activations at deeper layers of the network will likely no longer be decorrelated and will no longer have zero mean or unit variance, since they are output from earlier layers in the network. Even worse, during the training process the distribution of features at each layer of the network will shift as the weights of each layer are updated.\n",
    "\n",
    "The authors of [1] hypothesize that the shifting distribution of features inside deep neural networks may make training deep networks more difficult. To overcome this problem, they propose to insert into the network layers that normalize batches. At training time, such a layer uses a minibatch of data to estimate the mean and standard deviation of each feature. These estimated means and standard deviations are then used to center and normalize the features of the minibatch. A running average of these means and standard deviations is kept during training, and at test time these running averages are used to center and normalize features.\n",
    "\n",
    "It is possible that this normalization strategy could reduce the representational power of the network, since it may sometimes be optimal for certain layers to have features that are not zero-mean or unit variance. To this end, the batch normalization layer includes learnable shift and scale parameters for each feature dimension.\n",
    "\n",
    "[1] [Sergey Ioffe and Christian Szegedy, \"Batch Normalization: Accelerating Deep Network Training by Reducing\n",
    "Internal Covariate Shift\", ICML 2015.](https://arxiv.org/abs/1502.03167)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=========== You can safely ignore the message below if you are NOT working on ConvolutionalNetworks.ipynb ===========\n",
      "\tYou will need to compile a Cython extension for a portion of this assignment.\n",
      "\tThe instructions to do this will be given in a section of the notebook below.\n"
     ]
    }
   ],
   "source": [
    "# Setup cell.\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from cs231n.classifiers.fc_net import *\n",
    "from cs231n.data_utils import get_CIFAR10_data\n",
    "from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array\n",
    "from cs231n.solver import Solver\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams[\"figure.figsize\"] = (10.0, 8.0)  # Set default size of plots.\n",
    "plt.rcParams[\"image.interpolation\"] = \"nearest\"\n",
    "plt.rcParams[\"image.cmap\"] = \"gray\"\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "def rel_error(x, y):\n",
    "    \"\"\"Returns relative error.\"\"\"\n",
    "    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))\n",
    "\n",
    "def print_mean_std(x,axis=0):\n",
    "    print(f\"  means: {x.mean(axis=axis)}\")\n",
    "    print(f\"  stds:  {x.std(axis=axis)}\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train: (49000, 3, 32, 32)\n",
      "y_train: (49000,)\n",
      "X_val: (1000, 3, 32, 32)\n",
      "y_val: (1000,)\n",
      "X_test: (1000, 3, 32, 32)\n",
      "y_test: (1000,)\n"
     ]
    }
   ],
   "source": [
    "# Load the (preprocessed) CIFAR-10 data.\n",
    "data = get_CIFAR10_data()\n",
    "for k, v in list(data.items()):\n",
    "    print(f\"{k}: {v.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Batch Normalization: Forward Pass\n",
    "In the file `cs231n/layers.py`, implement the batch normalization forward pass in the function `batchnorm_forward`. Once you have done so, run the following to test your implementation.\n",
    "\n",
    "Referencing the paper linked to above in [1] may be helpful!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before batch normalization:\n",
      "  means: [ -2.3814598  -13.18038246   1.91780462]\n",
      "  stds:  [27.18502186 34.21455511 37.68611762]\n",
      "\n",
      "After batch normalization (gamma=1, beta=0)\n",
      "  means: [2.22044605e-17 8.16013923e-17 4.57966998e-17]\n",
      "  stds:  [0.99999999 1.         1.        ]\n",
      "\n",
      "After batch normalization (gamma= [1. 2. 3.] , beta= [11. 12. 13.] )\n",
      "  means: [11. 12. 13.]\n",
      "  stds:  [0.99999999 1.99999999 2.99999999]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Check the training-time forward pass by checking means and variances\n",
    "# of features both before and after batch normalization   \n",
    "\n",
    "# Simulate the forward pass for a two-layer network.\n",
    "np.random.seed(231)\n",
    "N, D1, D2, D3 = 200, 50, 60, 3\n",
    "X = np.random.randn(N, D1)\n",
    "W1 = np.random.randn(D1, D2)\n",
    "W2 = np.random.randn(D2, D3)\n",
    "a = np.maximum(0, X.dot(W1)).dot(W2)\n",
    "\n",
    "print('Before batch normalization:')\n",
    "print_mean_std(a,axis=0)\n",
    "\n",
    "gamma = np.ones((D3,))\n",
    "beta = np.zeros((D3,))\n",
    "\n",
    "# Means should be close to zero and stds close to one.\n",
    "print('After batch normalization (gamma=1, beta=0)')\n",
    "a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})\n",
    "print_mean_std(a_norm,axis=0)\n",
    "\n",
    "gamma = np.asarray([1.0, 2.0, 3.0])\n",
    "beta = np.asarray([11.0, 12.0, 13.0])\n",
    "\n",
    "# Now means should be close to beta and stds close to gamma.\n",
    "print('After batch normalization (gamma=', gamma, ', beta=', beta, ')')\n",
    "a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})\n",
    "print_mean_std(a_norm,axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "After batch normalization (test-time):\n",
      "  means: [-0.03927354 -0.04349152 -0.10452688]\n",
      "  stds:  [1.01531428 1.01238373 0.97819988]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Check the test-time forward pass by running the training-time\n",
    "# forward pass many times to warm up the running averages, and then\n",
    "# checking the means and variances of activations after a test-time\n",
    "# forward pass.\n",
    "\n",
    "np.random.seed(231)\n",
    "N, D1, D2, D3 = 200, 50, 60, 3\n",
    "W1 = np.random.randn(D1, D2)\n",
    "W2 = np.random.randn(D2, D3)\n",
    "\n",
    "bn_param = {'mode': 'train'}\n",
    "gamma = np.ones(D3)\n",
    "beta = np.zeros(D3)\n",
    "\n",
    "for t in range(50):\n",
    "  X = np.random.randn(N, D1)\n",
    "  a = np.maximum(0, X.dot(W1)).dot(W2)\n",
    "  batchnorm_forward(a, gamma, beta, bn_param)\n",
    "\n",
    "bn_param['mode'] = 'test'\n",
    "X = np.random.randn(N, D1)\n",
    "a = np.maximum(0, X.dot(W1)).dot(W2)\n",
    "a_norm, _ = batchnorm_forward(a, gamma, beta, bn_param)\n",
    "\n",
    "# Means should be close to zero and stds close to one, but will be\n",
    "# noisier than training-time forward passes.\n",
    "print('After batch normalization (test-time):')\n",
    "print_mean_std(a_norm,axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Batch Normalization: Backward Pass\n",
    "Now implement the backward pass for batch normalization in the function `batchnorm_backward`.\n",
    "\n",
    "To derive the backward pass you should write out the computation graph for batch normalization and backprop through each of the intermediate nodes. Some intermediates may have multiple outgoing branches; make sure to sum gradients across these branches in the backward pass.\n",
    "\n",
    "Once you have finished, run the following to numerically check your backward pass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dx error:  1.7029235612572515e-09\n",
      "dgamma error:  7.420414216247087e-13\n",
      "dbeta error:  2.8795057655839487e-12\n"
     ]
    }
   ],
   "source": [
    "# Gradient check batchnorm backward pass.\n",
    "np.random.seed(231)\n",
    "N, D = 4, 5\n",
    "x = 5 * np.random.randn(N, D) + 12\n",
    "gamma = np.random.randn(D)\n",
    "beta = np.random.randn(D)\n",
    "dout = np.random.randn(N, D)\n",
    "\n",
    "bn_param = {'mode': 'train'}\n",
    "fx = lambda x: batchnorm_forward(x, gamma, beta, bn_param)[0]\n",
    "fg = lambda a: batchnorm_forward(x, a, beta, bn_param)[0]\n",
    "fb = lambda b: batchnorm_forward(x, gamma, b, bn_param)[0]\n",
    "\n",
    "dx_num = eval_numerical_gradient_array(fx, x, dout)\n",
    "da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)\n",
    "db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)\n",
    "\n",
    "_, cache = batchnorm_forward(x, gamma, beta, bn_param)\n",
    "dx, dgamma, dbeta = batchnorm_backward(dout, cache)\n",
    "\n",
    "# You should expect to see relative errors between 1e-13 and 1e-8.\n",
    "print('dx error: ', rel_error(dx_num, dx))\n",
    "print('dgamma error: ', rel_error(da_num, dgamma))\n",
    "print('dbeta error: ', rel_error(db_num, dbeta))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Batch Normalization: Alternative Backward Pass\n",
    "In class we talked about two different implementations for the sigmoid backward pass. One strategy is to write out a computation graph composed of simple operations and backprop through all intermediate values. Another strategy is to work out the derivatives on paper. For example, you can derive a very simple formula for the sigmoid function's backward pass by simplifying gradients on paper.\n",
    "\n",
    "Surprisingly, it turns out that you can do a similar simplification for the batch normalization backward pass too!  \n",
    "\n",
    "In the forward pass, given a set of inputs $X=\\begin{bmatrix}x_1\\\\x_2\\\\...\\\\x_N\\end{bmatrix}$, \n",
    "\n",
    "we first calculate the mean $\\mu$ and variance $v$.\n",
    "With $\\mu$ and $v$ calculated, we can calculate the standard deviation $\\sigma$  and normalized data $Y$.\n",
    "The equations and graph illustration below describe the computation ($y_i$ is the i-th element of the vector $Y$).\n",
    "\n",
    "\\begin{align}\n",
    "& \\mu=\\frac{1}{N}\\sum_{k=1}^N x_k  &  v=\\frac{1}{N}\\sum_{k=1}^N (x_k-\\mu)^2 \\\\\n",
    "& \\sigma=\\sqrt{v+\\epsilon}         &  y_i=\\frac{x_i-\\mu}{\\sigma}\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/cs231n/cs231n.github.io/master/assets/a2/batchnorm_graph.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "source": [
    "The meat of our problem during backpropagation is to compute $\\frac{\\partial L}{\\partial X}$, given the upstream gradient we receive, $\\frac{\\partial L}{\\partial Y}.$ To do this, recall the chain rule in calculus gives us $\\frac{\\partial L}{\\partial X} = \\frac{\\partial L}{\\partial Y} \\cdot \\frac{\\partial Y}{\\partial X}$.\n",
    "\n",
    "The unknown/hard part is $\\frac{\\partial Y}{\\partial X}$. We can find this by first deriving step-by-step our local gradients at \n",
    "$\\frac{\\partial v}{\\partial X}$, $\\frac{\\partial \\mu}{\\partial X}$,\n",
    "$\\frac{\\partial \\sigma}{\\partial v}$, \n",
    "$\\frac{\\partial Y}{\\partial \\sigma}$, and $\\frac{\\partial Y}{\\partial \\mu}$,\n",
    "and then use the chain rule to compose these gradients (which appear in the form of vectors!) appropriately to compute $\\frac{\\partial Y}{\\partial X}$.\n",
    "\n",
    "If it's challenging to directly reason about the gradients over $X$ and $Y$ which require matrix multiplication, try reasoning about the gradients in terms of individual elements $x_i$ and $y_i$ first: in that case, you will need to come up with the derivations for $\\frac{\\partial L}{\\partial x_i}$, by relying on the Chain Rule to first calculate the intermediate $\\frac{\\partial \\mu}{\\partial x_i}, \\frac{\\partial v}{\\partial x_i}, \\frac{\\partial \\sigma}{\\partial x_i},$ then assemble these pieces to calculate $\\frac{\\partial y_i}{\\partial x_i}$. \n",
    "\n",
    "You should make sure each of the intermediary gradient derivations are all as simplified as possible, for ease of implementation. \n",
    "\n",
    "After doing so, implement the simplified batch normalization backward pass in the function `batchnorm_backward_alt` and compare the two implementations by running the following. Your two implementations should compute nearly identical results, but the alternative implementation should be a bit faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dx difference:  5.344522019088175e-13\n",
      "dgamma difference:  0.0\n",
      "dbeta difference:  0.0\n",
      "speedup: 0.72x\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "N, D = 100, 500\n",
    "x = 5 * np.random.randn(N, D) + 12\n",
    "gamma = np.random.randn(D)\n",
    "beta = np.random.randn(D)\n",
    "dout = np.random.randn(N, D)\n",
    "\n",
    "bn_param = {'mode': 'train'}\n",
    "out, cache = batchnorm_forward(x, gamma, beta, bn_param)\n",
    "\n",
    "t1 = time.time()\n",
    "dx1, dgamma1, dbeta1 = batchnorm_backward(dout, cache)\n",
    "t2 = time.time()\n",
    "dx2, dgamma2, dbeta2 = batchnorm_backward_alt(dout, cache)\n",
    "t3 = time.time()\n",
    "\n",
    "print('dx difference: ', rel_error(dx1, dx2))\n",
    "print('dgamma difference: ', rel_error(dgamma1, dgamma2))\n",
    "print('dbeta difference: ', rel_error(dbeta1, dbeta2))\n",
    "print('speedup: %.2fx' % ((t2 - t1) / (t3 - t2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fully Connected Networks with Batch Normalization\n",
    "Now that you have a working implementation for batch normalization, go back to your `FullyConnectedNet` in the file `cs231n/classifiers/fc_net.py`. Modify your implementation to add batch normalization.\n",
    "\n",
    "Concretely, when the `normalization` flag is set to `\"batchnorm\"` in the constructor, you should insert a batch normalization layer before each ReLU nonlinearity. The outputs from the last layer of the network should not be normalized. Once you are done, run the following to gradient-check your implementation.\n",
    "\n",
    "**Hint:** You might find it useful to define an additional helper layer similar to those in the file `cs231n/layer_utils.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running check with reg =  0\n",
      "Initial loss:  2.2611955101340957\n",
      "W1 relative error: 1.10e-04\n",
      "W2 relative error: 3.07e-06\n",
      "W3 relative error: 3.92e-10\n",
      "b1 relative error: 8.33e-09\n",
      "b2 relative error: 2.00e-07\n",
      "b3 relative error: 9.06e-11\n",
      "beta1 relative error: 7.85e-09\n",
      "beta2 relative error: 1.89e-09\n",
      "gamma1 relative error: 7.47e-09\n",
      "gamma2 relative error: 3.35e-09\n",
      "\n",
      "Running check with reg =  3.14\n",
      "Initial loss:  6.996533220108303\n",
      "W1 relative error: 1.98e-06\n",
      "W2 relative error: 2.29e-06\n",
      "W3 relative error: 2.79e-08\n",
      "b1 relative error: 1.11e-08\n",
      "b2 relative error: 2.36e-08\n",
      "b3 relative error: 2.10e-10\n",
      "beta1 relative error: 6.65e-09\n",
      "beta2 relative error: 4.23e-09\n",
      "gamma1 relative error: 5.94e-09\n",
      "gamma2 relative error: 3.72e-09\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "N, D, H1, H2, C = 2, 15, 20, 30, 10\n",
    "X = np.random.randn(N, D)\n",
    "y = np.random.randint(C, size=(N,))\n",
    "\n",
    "# You should expect losses between 1e-4~1e-10 for W, \n",
    "# losses between 1e-08~1e-10 for b,\n",
    "# and losses between 1e-08~1e-09 for beta and gammas.\n",
    "for reg in [0, 3.14]:\n",
    "  print('Running check with reg = ', reg)\n",
    "  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,\n",
    "                            reg=reg, weight_scale=5e-2, dtype=np.float64,\n",
    "                            normalization='batchnorm')\n",
    "\n",
    "  loss, grads = model.loss(X, y)\n",
    "  print('Initial loss: ', loss)\n",
    "\n",
    "  for name in sorted(grads):\n",
    "    f = lambda _: model.loss(X, y)[0]\n",
    "    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)\n",
    "    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))\n",
    "  if reg == 0: print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Batch Normalization for Deep Networks\n",
    "Run the following to train a six-layer network on a subset of 1000 training examples both with and without batch normalization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Solver with batch norm:\n",
      "(Iteration 1 / 200) loss: 2.340974\n"
     ]
    },
    {
     "ename": "ValueError",
     "evalue": "matmul: Input operand 1 does not have enough dimensions (has 0, gufunc core with signature (n?,k),(k,m?)->(n?,m?) requires 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "Input \u001b[1;32mIn [8]\u001b[0m, in \u001b[0;36m<cell line: 26>\u001b[1;34m()\u001b[0m\n\u001b[0;32m     18\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mSolver with batch norm:\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[0;32m     19\u001b[0m bn_solver \u001b[38;5;241m=\u001b[39m Solver(bn_model, small_data,\n\u001b[0;32m     20\u001b[0m                 num_epochs\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m10\u001b[39m, batch_size\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m50\u001b[39m,\n\u001b[0;32m     21\u001b[0m                 update_rule\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124madam\u001b[39m\u001b[38;5;124m'\u001b[39m,\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m     24\u001b[0m                 },\n\u001b[0;32m     25\u001b[0m                 verbose\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m,print_every\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m20\u001b[39m)\n\u001b[1;32m---> 26\u001b[0m \u001b[43mbn_solver\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtrain\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m     28\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124mSolver without batch norm:\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[0;32m     29\u001b[0m solver \u001b[38;5;241m=\u001b[39m Solver(model, small_data,\n\u001b[0;32m     30\u001b[0m                 num_epochs\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m10\u001b[39m, batch_size\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m50\u001b[39m,\n\u001b[0;32m     31\u001b[0m                 update_rule\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124madam\u001b[39m\u001b[38;5;124m'\u001b[39m,\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m     34\u001b[0m                 },\n\u001b[0;32m     35\u001b[0m                 verbose\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m, print_every\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m20\u001b[39m)\n",
      "File \u001b[1;32mc:\\Users\\I542634\\Documents\\Learning\\cs231n\\cs231n-answers\\assignment2\\cs231n\\solver.py:285\u001b[0m, in \u001b[0;36mSolver.train\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    283\u001b[0m last_it \u001b[38;5;241m=\u001b[39m t \u001b[38;5;241m==\u001b[39m num_iterations \u001b[38;5;241m-\u001b[39m \u001b[38;5;241m1\u001b[39m\n\u001b[0;32m    284\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m first_it \u001b[38;5;129;01mor\u001b[39;00m last_it \u001b[38;5;129;01mor\u001b[39;00m epoch_end:\n\u001b[1;32m--> 285\u001b[0m     train_acc \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcheck_accuracy\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m    286\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mX_train\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43my_train\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mnum_samples\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mnum_train_samples\u001b[49m\n\u001b[0;32m    287\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m    288\u001b[0m     val_acc \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcheck_accuracy(\n\u001b[0;32m    289\u001b[0m         \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mX_val, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39my_val, num_samples\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mnum_val_samples\n\u001b[0;32m    290\u001b[0m     )\n\u001b[0;32m    291\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mtrain_acc_history\u001b[38;5;241m.\u001b[39mappend(train_acc)\n",
      "File \u001b[1;32mc:\\Users\\I542634\\Documents\\Learning\\cs231n\\cs231n-answers\\assignment2\\cs231n\\solver.py:247\u001b[0m, in \u001b[0;36mSolver.check_accuracy\u001b[1;34m(self, X, y, num_samples, batch_size)\u001b[0m\n\u001b[0;32m    245\u001b[0m     start \u001b[38;5;241m=\u001b[39m i \u001b[38;5;241m*\u001b[39m batch_size\n\u001b[0;32m    246\u001b[0m     end \u001b[38;5;241m=\u001b[39m (i \u001b[38;5;241m+\u001b[39m \u001b[38;5;241m1\u001b[39m) \u001b[38;5;241m*\u001b[39m batch_size\n\u001b[1;32m--> 247\u001b[0m     scores \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mmodel\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mloss\u001b[49m\u001b[43m(\u001b[49m\u001b[43mX\u001b[49m\u001b[43m[\u001b[49m\u001b[43mstart\u001b[49m\u001b[43m:\u001b[49m\u001b[43mend\u001b[49m\u001b[43m]\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m    248\u001b[0m     y_pred\u001b[38;5;241m.\u001b[39mappend(np\u001b[38;5;241m.\u001b[39margmax(scores, axis\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1\u001b[39m))\n\u001b[0;32m    249\u001b[0m y_pred \u001b[38;5;241m=\u001b[39m np\u001b[38;5;241m.\u001b[39mhstack(y_pred)\n",
      "File \u001b[1;32mc:\\Users\\I542634\\Documents\\Learning\\cs231n\\cs231n-answers\\assignment2\\cs231n\\classifiers\\fc_net.py:172\u001b[0m, in \u001b[0;36mFullyConnectedNet.loss\u001b[1;34m(self, X, y)\u001b[0m\n\u001b[0;32m    169\u001b[0m     bn \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mbn_params[l] \u001b[38;5;28;01mif\u001b[39;00m gamma \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m  \u001b[38;5;66;03m# bn params if exist\u001b[39;00m\n\u001b[0;32m    170\u001b[0m     do \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mdropout_param \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39muse_dropout \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m  \u001b[38;5;66;03m# do params if exist\u001b[39;00m\n\u001b[1;32m--> 172\u001b[0m     X, cache[l] \u001b[38;5;241m=\u001b[39m \u001b[43mgeneric_forward\u001b[49m\u001b[43m(\u001b[49m\u001b[43mX\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mw\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mb\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mgamma\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbeta\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbn\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdo\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43ml\u001b[49m\u001b[38;5;241;43m==\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mnum_layers\u001b[49m\u001b[38;5;241;43m-\u001b[39;49m\u001b[38;5;241;43m1\u001b[39;49m\u001b[43m)\u001b[49m \u001b[38;5;66;03m# generic forward pass\u001b[39;00m\n\u001b[0;32m    174\u001b[0m scores \u001b[38;5;241m=\u001b[39m X\n\u001b[0;32m    176\u001b[0m \u001b[38;5;66;03m# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****\u001b[39;00m\n\u001b[0;32m    177\u001b[0m \u001b[38;5;66;03m############################################################################\u001b[39;00m\n\u001b[0;32m    178\u001b[0m \u001b[38;5;66;03m#                             END OF YOUR CODE                             #\u001b[39;00m\n\u001b[0;32m    179\u001b[0m \u001b[38;5;66;03m############################################################################\u001b[39;00m\n\u001b[0;32m    180\u001b[0m \n\u001b[0;32m    181\u001b[0m \u001b[38;5;66;03m# If test mode return early.\u001b[39;00m\n",
      "File \u001b[1;32mc:\\Users\\I542634\\Documents\\Learning\\cs231n\\cs231n-answers\\assignment2\\cs231n\\layer_utils.py:50\u001b[0m, in \u001b[0;36mgeneric_forward\u001b[1;34m(x, w, b, gamma, beta, bn_param, dropout_param, last)\u001b[0m\n\u001b[0;32m     47\u001b[0m \u001b[38;5;66;03m# Initialize empty optional caches\u001b[39;00m\n\u001b[0;32m     48\u001b[0m bn_cache, ln_cache, relu_cache, dropout_cache \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m, \u001b[38;5;28;01mNone\u001b[39;00m, \u001b[38;5;28;01mNone\u001b[39;00m, \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m---> 50\u001b[0m out, fc_cache \u001b[38;5;241m=\u001b[39m \u001b[43maffine_forward\u001b[49m\u001b[43m(\u001b[49m\u001b[43mx\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mw\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mb\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m     52\u001b[0m \u001b[38;5;66;03m# If the the layer is not last\u001b[39;00m\n\u001b[0;32m     53\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m last:\n\u001b[0;32m     54\u001b[0m     \u001b[38;5;66;03m# normalize outputs if bn_param exists\u001b[39;00m\n\u001b[0;32m     55\u001b[0m     \u001b[38;5;66;03m# train - batchnorm \u001b[39;00m\n\u001b[0;32m     56\u001b[0m     \u001b[38;5;66;03m# test - layernorm\u001b[39;00m\n",
      "File \u001b[1;32mc:\\Users\\I542634\\Documents\\Learning\\cs231n\\cs231n-answers\\assignment2\\cs231n\\layers.py:29\u001b[0m, in \u001b[0;36maffine_forward\u001b[1;34m(x, w, b)\u001b[0m\n\u001b[0;32m     22\u001b[0m out \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[0;32m     23\u001b[0m \u001b[38;5;66;03m###########################################################################\u001b[39;00m\n\u001b[0;32m     24\u001b[0m \u001b[38;5;66;03m# TODO: Copy over your solution from Assignment 1.                        #\u001b[39;00m\n\u001b[0;32m     25\u001b[0m \u001b[38;5;66;03m###########################################################################\u001b[39;00m\n\u001b[0;32m     26\u001b[0m \u001b[38;5;66;03m# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****\u001b[39;00m\n\u001b[0;32m     27\u001b[0m \n\u001b[0;32m     28\u001b[0m \u001b[38;5;66;03m# reshape to rows\u001b[39;00m\n\u001b[1;32m---> 29\u001b[0m out \u001b[38;5;241m=\u001b[39m \u001b[43mx\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mreshape\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mlen\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mx\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m-\u001b[39;49m\u001b[38;5;241;43m1\u001b[39;49m\u001b[43m)\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m@\u001b[39;49m\u001b[43m \u001b[49m\u001b[43mw\u001b[49m \u001b[38;5;241m+\u001b[39m b\n\u001b[0;32m     31\u001b[0m \u001b[38;5;66;03m# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****\u001b[39;00m\n\u001b[0;32m     32\u001b[0m \u001b[38;5;66;03m###########################################################################\u001b[39;00m\n\u001b[0;32m     33\u001b[0m \u001b[38;5;66;03m#                             END OF YOUR CODE                            #\u001b[39;00m\n\u001b[0;32m     34\u001b[0m \u001b[38;5;66;03m###########################################################################\u001b[39;00m\n\u001b[0;32m     35\u001b[0m cache \u001b[38;5;241m=\u001b[39m (x, w, b)\n",
      "\u001b[1;31mValueError\u001b[0m: matmul: Input operand 1 does not have enough dimensions (has 0, gufunc core with signature (n?,k),(k,m?)->(n?,m?) requires 1)"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "\n",
    "# Try training a very deep net with batchnorm.\n",
    "hidden_dims = [100, 100, 100, 100, 100]\n",
    "\n",
    "num_train = 1000\n",
    "small_data = {\n",
    "  'X_train': data['X_train'][:num_train],\n",
    "  'y_train': data['y_train'][:num_train],\n",
    "  'X_val': data['X_val'],\n",
    "  'y_val': data['y_val'],\n",
    "}\n",
    "\n",
    "weight_scale = 2e-2\n",
    "bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')\n",
    "model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)\n",
    "\n",
    "print('Solver with batch norm:')\n",
    "bn_solver = Solver(bn_model, small_data,\n",
    "                num_epochs=10, batch_size=50,\n",
    "                update_rule='adam',\n",
    "                optim_config={\n",
    "                  'learning_rate': 1e-3,\n",
    "                },\n",
    "                verbose=True,print_every=20)\n",
    "bn_solver.train()\n",
    "\n",
    "print('\\nSolver without batch norm:')\n",
    "solver = Solver(model, small_data,\n",
    "                num_epochs=10, batch_size=50,\n",
    "                update_rule='adam',\n",
    "                optim_config={\n",
    "                  'learning_rate': 1e-3,\n",
    "                },\n",
    "                verbose=True, print_every=20)\n",
    "solver.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following to visualize the results from two networks trained above. You should find that using batch normalization helps the network to converge much faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pdf-ignore-input"
    ]
   },
   "outputs": [],
   "source": [
    "def plot_training_history(title, label, baseline, bn_solvers, plot_fn, bl_marker='.', bn_marker='.', labels=None):\n",
    "    \"\"\"utility function for plotting training history\"\"\"\n",
    "    plt.title(title)\n",
    "    plt.xlabel(label)\n",
    "    bn_plots = [plot_fn(bn_solver) for bn_solver in bn_solvers]\n",
    "    bl_plot = plot_fn(baseline)\n",
    "    num_bn = len(bn_plots)\n",
    "    for i in range(num_bn):\n",
    "        label='with_norm'\n",
    "        if labels is not None:\n",
    "            label += str(labels[i])\n",
    "        plt.plot(bn_plots[i], bn_marker, label=label)\n",
    "    label='baseline'\n",
    "    if labels is not None:\n",
    "        label += str(labels[0])\n",
    "    plt.plot(bl_plot, bl_marker, label=label)\n",
    "    plt.legend(loc='lower center', ncol=num_bn+1) \n",
    "\n",
    "    \n",
    "plt.subplot(3, 1, 1)\n",
    "plot_training_history('Training loss','Iteration', solver, [bn_solver], \\\n",
    "                      lambda x: x.loss_history, bl_marker='o', bn_marker='o')\n",
    "plt.subplot(3, 1, 2)\n",
    "plot_training_history('Training accuracy','Epoch', solver, [bn_solver], \\\n",
    "                      lambda x: x.train_acc_history, bl_marker='-o', bn_marker='-o')\n",
    "plt.subplot(3, 1, 3)\n",
    "plot_training_history('Validation accuracy','Epoch', solver, [bn_solver], \\\n",
    "                      lambda x: x.val_acc_history, bl_marker='-o', bn_marker='-o')\n",
    "\n",
    "plt.gcf().set_size_inches(15, 15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Batch Normalization and Initialization\n",
    "We will now run a small experiment to study the interaction of batch normalization and weight initialization.\n",
    "\n",
    "The first cell will train eight-layer networks both with and without batch normalization using different scales for weight initialization. The second layer will plot training accuracy, validation set accuracy, and training loss as a function of the weight initialization scale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pdf-ignore-input"
    ]
   },
   "outputs": [],
   "source": [
    "np.random.seed(231)\n",
    "\n",
    "# Try training a very deep net with batchnorm.\n",
    "hidden_dims = [50, 50, 50, 50, 50, 50, 50]\n",
    "num_train = 1000\n",
    "small_data = {\n",
    "  'X_train': data['X_train'][:num_train],\n",
    "  'y_train': data['y_train'][:num_train],\n",
    "  'X_val': data['X_val'],\n",
    "  'y_val': data['y_val'],\n",
    "}\n",
    "\n",
    "bn_solvers_ws = {}\n",
    "solvers_ws = {}\n",
    "weight_scales = np.logspace(-4, 0, num=20)\n",
    "for i, weight_scale in enumerate(weight_scales):\n",
    "    print('Running weight scale %d / %d' % (i + 1, len(weight_scales)))\n",
    "    bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')\n",
    "    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)\n",
    "\n",
    "    bn_solver = Solver(bn_model, small_data,\n",
    "                  num_epochs=10, batch_size=50,\n",
    "                  update_rule='adam',\n",
    "                  optim_config={\n",
    "                    'learning_rate': 1e-3,\n",
    "                  },\n",
    "                  verbose=False, print_every=200)\n",
    "    bn_solver.train()\n",
    "    bn_solvers_ws[weight_scale] = bn_solver\n",
    "\n",
    "    solver = Solver(model, small_data,\n",
    "                  num_epochs=10, batch_size=50,\n",
    "                  update_rule='adam',\n",
    "                  optim_config={\n",
    "                    'learning_rate': 1e-3,\n",
    "                  },\n",
    "                  verbose=False, print_every=200)\n",
    "    solver.train()\n",
    "    solvers_ws[weight_scale] = solver"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pdf-ignore-input"
    ]
   },
   "outputs": [],
   "source": [
    "# Plot results of weight scale experiment.\n",
    "best_train_accs, bn_best_train_accs = [], []\n",
    "best_val_accs, bn_best_val_accs = [], []\n",
    "final_train_loss, bn_final_train_loss = [], []\n",
    "\n",
    "for ws in weight_scales:\n",
    "  best_train_accs.append(max(solvers_ws[ws].train_acc_history))\n",
    "  bn_best_train_accs.append(max(bn_solvers_ws[ws].train_acc_history))\n",
    "  \n",
    "  best_val_accs.append(max(solvers_ws[ws].val_acc_history))\n",
    "  bn_best_val_accs.append(max(bn_solvers_ws[ws].val_acc_history))\n",
    "  \n",
    "  final_train_loss.append(np.mean(solvers_ws[ws].loss_history[-100:]))\n",
    "  bn_final_train_loss.append(np.mean(bn_solvers_ws[ws].loss_history[-100:]))\n",
    "  \n",
    "plt.subplot(3, 1, 1)\n",
    "plt.title('Best val accuracy vs. weight initialization scale')\n",
    "plt.xlabel('Weight initialization scale')\n",
    "plt.ylabel('Best val accuracy')\n",
    "plt.semilogx(weight_scales, best_val_accs, '-o', label='baseline')\n",
    "plt.semilogx(weight_scales, bn_best_val_accs, '-o', label='batchnorm')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "\n",
    "plt.subplot(3, 1, 2)\n",
    "plt.title('Best train accuracy vs. weight initialization scale')\n",
    "plt.xlabel('Weight initialization scale')\n",
    "plt.ylabel('Best training accuracy')\n",
    "plt.semilogx(weight_scales, best_train_accs, '-o', label='baseline')\n",
    "plt.semilogx(weight_scales, bn_best_train_accs, '-o', label='batchnorm')\n",
    "plt.legend()\n",
    "\n",
    "plt.subplot(3, 1, 3)\n",
    "plt.title('Final training loss vs. weight initialization scale')\n",
    "plt.xlabel('Weight initialization scale')\n",
    "plt.ylabel('Final training loss')\n",
    "plt.semilogx(weight_scales, final_train_loss, '-o', label='baseline')\n",
    "plt.semilogx(weight_scales, bn_final_train_loss, '-o', label='batchnorm')\n",
    "plt.legend()\n",
    "plt.gca().set_ylim(1.0, 3.5)\n",
    "\n",
    "plt.gcf().set_size_inches(15, 15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 1:\n",
    "Describe the results of this experiment. How does the weight initialization scale affect models with/without batch normalization differently, and why?\n",
    "\n",
    "## Answer:\n",
    "[FILL THIS IN]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Batch Normalization and Batch Size\n",
    "We will now run a small experiment to study the interaction of batch normalization and batch size.\n",
    "\n",
    "The first cell will train 6-layer networks both with and without batch normalization using different batch sizes. The second layer will plot training accuracy and validation set accuracy over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pdf-ignore-input"
    ]
   },
   "outputs": [],
   "source": [
    "def run_batchsize_experiments(normalization_mode):\n",
    "    np.random.seed(231)\n",
    "    \n",
    "    # Try training a very deep net with batchnorm.\n",
    "    hidden_dims = [100, 100, 100, 100, 100]\n",
    "    num_train = 1000\n",
    "    small_data = {\n",
    "      'X_train': data['X_train'][:num_train],\n",
    "      'y_train': data['y_train'][:num_train],\n",
    "      'X_val': data['X_val'],\n",
    "      'y_val': data['y_val'],\n",
    "    }\n",
    "    n_epochs=10\n",
    "    weight_scale = 2e-2\n",
    "    batch_sizes = [5,10,50]\n",
    "    lr = 10**(-3.5)\n",
    "    solver_bsize = batch_sizes[0]\n",
    "\n",
    "    print('No normalization: batch size = ',solver_bsize)\n",
    "    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)\n",
    "    solver = Solver(model, small_data,\n",
    "                    num_epochs=n_epochs, batch_size=solver_bsize,\n",
    "                    update_rule='adam',\n",
    "                    optim_config={\n",
    "                      'learning_rate': lr,\n",
    "                    },\n",
    "                    verbose=False)\n",
    "    solver.train()\n",
    "    \n",
    "    bn_solvers = []\n",
    "    for i in range(len(batch_sizes)):\n",
    "        b_size=batch_sizes[i]\n",
    "        print('Normalization: batch size = ',b_size)\n",
    "        bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=normalization_mode)\n",
    "        bn_solver = Solver(bn_model, small_data,\n",
    "                        num_epochs=n_epochs, batch_size=b_size,\n",
    "                        update_rule='adam',\n",
    "                        optim_config={\n",
    "                          'learning_rate': lr,\n",
    "                        },\n",
    "                        verbose=False)\n",
    "        bn_solver.train()\n",
    "        bn_solvers.append(bn_solver)\n",
    "        \n",
    "    return bn_solvers, solver, batch_sizes\n",
    "\n",
    "batch_sizes = [5,10,50]\n",
    "bn_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('batchnorm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.subplot(2, 1, 1)\n",
    "plot_training_history('Training accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \\\n",
    "                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)\n",
    "plt.subplot(2, 1, 2)\n",
    "plot_training_history('Validation accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \\\n",
    "                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)\n",
    "\n",
    "plt.gcf().set_size_inches(15, 10)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 2:\n",
    "Describe the results of this experiment. What does this imply about the relationship between batch normalization and batch size? Why is this relationship observed?\n",
    "\n",
    "## Answer:\n",
    "[FILL THIS IN]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Layer Normalization\n",
    "Batch normalization has proved to be effective in making networks easier to train, but the dependency on batch size makes it less useful in complex networks which have a cap on the input batch size due to hardware limitations. \n",
    "\n",
    "Several alternatives to batch normalization have been proposed to mitigate this problem; one such technique is Layer Normalization [2]. Instead of normalizing over the batch, we normalize over the features. In other words, when using Layer Normalization, each feature vector corresponding to a single datapoint is normalized based on the sum of all terms within that feature vector.\n",
    "\n",
    "[2] [Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. \"Layer Normalization.\" stat 1050 (2016): 21.](https://arxiv.org/pdf/1607.06450.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 3:\n",
    "Which of these data preprocessing steps is analogous to batch normalization, and which is analogous to layer normalization?\n",
    "\n",
    "1. Scaling each image in the dataset, so that the RGB channels for each row of pixels within an image sums up to 1.\n",
    "2. Scaling each image in the dataset, so that the RGB channels for all pixels within an image sums up to 1.  \n",
    "3. Subtracting the mean image of the dataset from each image in the dataset.\n",
    "4. Setting all RGB values to either 0 or 1 depending on a given threshold.\n",
    "\n",
    "## Answer:\n",
    "[FILL THIS IN]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Layer Normalization: Implementation\n",
    "\n",
    "Now you'll implement layer normalization. This step should be relatively straightforward, as conceptually the implementation is almost identical to that of batch normalization. One significant difference though is that for layer normalization, we do not keep track of the moving moments, and the testing phase is identical to the training phase, where the mean and variance are directly calculated per datapoint.\n",
    "\n",
    "Here's what you need to do:\n",
    "\n",
    "* In `cs231n/layers.py`, implement the forward pass for layer normalization in the function `layernorm_forward`. \n",
    "\n",
    "Run the cell below to check your results.\n",
    "* In `cs231n/layers.py`, implement the backward pass for layer normalization in the function `layernorm_backward`. \n",
    "\n",
    "Run the second cell below to check your results.\n",
    "* Modify `cs231n/classifiers/fc_net.py` to add layer normalization to the `FullyConnectedNet`. When the `normalization` flag is set to `\"layernorm\"` in the constructor, you should insert a layer normalization layer before each ReLU nonlinearity. \n",
    "\n",
    "Run the third cell below to run the batch size experiment on layer normalization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the training-time forward pass by checking means and variances\n",
    "# of features both before and after layer normalization.\n",
    "\n",
    "# Simulate the forward pass for a two-layer network.\n",
    "np.random.seed(231)\n",
    "N, D1, D2, D3 =4, 50, 60, 3\n",
    "X = np.random.randn(N, D1)\n",
    "W1 = np.random.randn(D1, D2)\n",
    "W2 = np.random.randn(D2, D3)\n",
    "a = np.maximum(0, X.dot(W1)).dot(W2)\n",
    "\n",
    "print('Before layer normalization:')\n",
    "print_mean_std(a,axis=1)\n",
    "\n",
    "gamma = np.ones(D3)\n",
    "beta = np.zeros(D3)\n",
    "\n",
    "# Means should be close to zero and stds close to one.\n",
    "print('After layer normalization (gamma=1, beta=0)')\n",
    "a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})\n",
    "print_mean_std(a_norm,axis=1)\n",
    "\n",
    "gamma = np.asarray([3.0,3.0,3.0])\n",
    "beta = np.asarray([5.0,5.0,5.0])\n",
    "\n",
    "# Now means should be close to beta and stds close to gamma.\n",
    "print('After layer normalization (gamma=', gamma, ', beta=', beta, ')')\n",
    "a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})\n",
    "print_mean_std(a_norm,axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gradient check batchnorm backward pass.\n",
    "np.random.seed(231)\n",
    "N, D = 4, 5\n",
    "x = 5 * np.random.randn(N, D) + 12\n",
    "gamma = np.random.randn(D)\n",
    "beta = np.random.randn(D)\n",
    "dout = np.random.randn(N, D)\n",
    "\n",
    "ln_param = {}\n",
    "fx = lambda x: layernorm_forward(x, gamma, beta, ln_param)[0]\n",
    "fg = lambda a: layernorm_forward(x, a, beta, ln_param)[0]\n",
    "fb = lambda b: layernorm_forward(x, gamma, b, ln_param)[0]\n",
    "\n",
    "dx_num = eval_numerical_gradient_array(fx, x, dout)\n",
    "da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)\n",
    "db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)\n",
    "\n",
    "_, cache = layernorm_forward(x, gamma, beta, ln_param)\n",
    "dx, dgamma, dbeta = layernorm_backward(dout, cache)\n",
    "\n",
    "# You should expect to see relative errors between 1e-12 and 1e-8.\n",
    "print('dx error: ', rel_error(dx_num, dx))\n",
    "print('dgamma error: ', rel_error(da_num, dgamma))\n",
    "print('dbeta error: ', rel_error(db_num, dbeta))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Layer Normalization and Batch Size\n",
    "\n",
    "We will now run the previous batch size experiment with layer normalization instead of batch normalization. Compared to the previous experiment, you should see a markedly smaller influence of batch size on the training history!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ln_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('layernorm')\n",
    "\n",
    "plt.subplot(2, 1, 1)\n",
    "plot_training_history('Training accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \\\n",
    "                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)\n",
    "plt.subplot(2, 1, 2)\n",
    "plot_training_history('Validation accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \\\n",
    "                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)\n",
    "\n",
    "plt.gcf().set_size_inches(15, 10)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 4:\n",
    "When is layer normalization likely to not work well, and why?\n",
    "\n",
    "1. Using it in a very deep network\n",
    "2. Having a very small dimension of features\n",
    "3. Having a high regularization term\n",
    "\n",
    "\n",
    "## Answer:\n",
    "[FILL THIS IN]\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}