MNIST.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is an example on how to build a convolutional neural network (CNN) to recognize handwritten digits in the MNIST data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.autograd import Variable\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "from torchvision import datasets, transforms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convolutional Neural Network\n",
    "\n",
    "Convolutional neural networks take biological inspiration from the animal visual cortex. The visual cortex consists of a complex arrangement of cells. These cells are sensitive to specific regions of the visual field, called a receptive field. The different receptive fields partially overlap such that they cover the entire visual field. CNNs use this idea of specialized components having specific tasks similar to how neurons in the visual cortex look for specific characteristics.\n",
    "\n",
    "In a CNN, an input in passed through a series of <b>convolution layers</b>, <b>nonlinear layers</b>, <b>pooling layers</b>, and <b>fully-connected layers</b>, and we get an output. In image classification, the input is an image that can be represented as an array of pixel values, and t eoutput can be single class or a probability of classes that best describe the image. \n",
    "\n",
    "### Convolution layer \n",
    "\n",
    "The first layer in a CNN is the convolutional layer. This layer extracts features, such as edges, curves, etc., from the input image. It consists of a set of filters. We slide each filter across the width and height of the input image, computing element vise multiplications of the values in the filter and the original pixel values in the input. The multiplications are summed up, and the process is repeated by moving the filter to the right on the input volume. The output produced by sliding the filter over all the locations is called an activation or feature map. The depth of the output array is the same as the depth of the input array.\n",
    "\n",
    "Every image can be considered as a matrix of pixel values. Consider a 5x5 image whose pixel values are only 0 and 1 (for a grayscale image, the pixel values range from 0 to 255, the green matrix below is a special case). Also, consider another 3x3 matrix which is the filter. The convolution of the 5x5 image and the 3x3 filter can computed as shown in the animation below:   \n",
    "<img src=\"https://i.stack.imgur.com/I7DBr.gif\" style=\"width: 400px;\">\n",
    "\n",
    "Three parameters control the size of the output:\n",
    "<ul>\n",
    "<li><b>Depth</b>: Depth corresponds to the number of filters used for the convolution operation.\n",
    "\n",
    "<li><b>Stride</b>: Stride is the number of pixels by which we slide our filter matrix over the input matrix. By default, the <b> stride </b> is 1, which results in the filter sliding by one pixel at a time. When the stride is 2, then the filter jumps two pixels at a time resulting in smaller output volumes.\n",
    "\n",
    "<li><b>Zero-padding</b>: Zero-padding is used to pad the input volume with zeros around the border. This allows us to control the spatial size of the output volume.\n",
    "</ul>\n",
    "\n",
    "### Non-Linear Layer (ReLU)\n",
    "\n",
    "ReLU (Rectified Linear Units) is an element-wise activation function, and replaces all negative pixel values in the feature map by zero. It implements the function $y = max(0, x)$, so the input and ouput sizes of this layer are the same.\n",
    "\n",
    "<img src=\"https://www.embedded-vision.com/sites/default/files/technical-articles/CadenceCNN/Figure8.jpg\" style=\"width: 600px;\">\n",
    "\n",
    "### Pooling Layer\n",
    "\n",
    "This layer reduces the spatial size of the representation. It controls overfitting by reducing the amount of parameters and computation in the network. The most common form of pooling uses the Max operation. The example shown below uses max pooling with a 2x2 window. We slide our window with a stride of 2 and take the maximum value in each region.\n",
    "\n",
    "<img src=\"https://qph.ec.quoracdn.net/main-qimg-8afedfb2f82f279781bfefa269bc6a90\" style=\"width: 600px;\">\n",
    "\n",
    "### Fully Connected Layer\n",
    "\n",
    "This layer is fully connected with the output of the previous layer. This layer performs classification on the features extracted by the convolutional layer and downsampled by the pooling layer by using a weighted sum of the features followed by a bias offset.\n",
    "\n",
    "<img src=\"https://cdn-images-1.medium.com/max/1600/1*Kdnux0Kw1yQ4D8dq__mYCA.png\" style=\"width: 300px;\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Model (\n",
       "  (conv1): Conv2d(1, 32, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))\n",
       "  (conv2): Conv2d(32, 64, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))\n",
       "  (fc1): Linear (3136 -> 1024)\n",
       "  (fc2): Linear (1024 -> 10)\n",
       ")"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class Model(nn.Module):\n",
    "  def __init__(self):\n",
    "    super(Model, self).__init__()\n",
    "    self.conv1 = nn.Conv2d(1, 32, 5, padding=2)\n",
    "    self.conv2 = nn.Conv2d(32, 64, 5, padding=2)\n",
    "    self.fc1 = nn.Linear(64*7*7, 1024)\n",
    "    self.fc2 = nn.Linear(1024, 10)\n",
    "\n",
    "  def forward(self, x):\n",
    "    x = F.relu(F.max_pool2d(self.conv1(x), 2))\n",
    "    x = F.relu(F.max_pool2d(self.conv2(x), 2))\n",
    "    x = x.view(-1, 64*7*7)\n",
    "    x = F.relu(self.fc1(x))\n",
    "    x = F.dropout(x, training=self.training)\n",
    "    x = self.fc2(x)\n",
    "    return F.log_softmax(x)\n",
    "\n",
    "model = Model()\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We load the training and test data from the MNIST dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "batch_size = 50\n",
    "train_loader = torch.utils.data.DataLoader(\n",
    "    datasets.MNIST('data', train=True, download=True, transform=transforms.ToTensor()),\n",
    "    batch_size=batch_size, shuffle=True)\n",
    "\n",
    "test_loader = torch.utils.data.DataLoader(\n",
    "    datasets.MNIST('data', train=False, transform=transforms.ToTensor()),\n",
    "    batch_size=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Optimizer:</b> This updates the parameters based on the computed gradients. Here, we have used Stochastic Gradient Descent (SGD)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "optimizer = optim.SGD(model.parameters(), lr=0.0003)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Loss Function:</b> For training and evaluation, we have to define a loss function that measures how closely the model's predictions match the target classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "criterion = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def train(epoch):\n",
    "  model.train()\n",
    "  i = 1\n",
    "  for data, target in train_loader:\n",
    "    data, target = Variable(data), Variable(target)\n",
    "    optimizer.zero_grad()\n",
    "    output = model(data)\n",
    "    # make_dot(output)\n",
    "    loss = criterion(output, target)\n",
    "    prediction = output.data.max(1)[1]\n",
    "    accuracy = prediction.eq(target.data).sum()/batch_size*100\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    if i % 1000 == 0:\n",
    "      print('\\nTrain Step: {}\\tLoss: {:.3f}'.format(epoch, loss.data[0]))\n",
    "    i += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def test():\n",
    "  model.eval()\n",
    "  correct = 0\n",
    "  for data, target in test_loader:\n",
    "    data, target = Variable(data), Variable(target)\n",
    "    output = model(data)\n",
    "    prediction = output.data.max(1)[1]\n",
    "    correct += prediction.eq(target.data).sum()\n",
    "\n",
    "  print('Test set: Accuracy: {:.2f}%'.format(100. * correct / len(test_loader.dataset)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Step: 0\tLoss: 2.289\n",
      "\n",
      "Test set: Accuracy: 24.83%\n",
      "Train Step: 1\tLoss: 2.268\n",
      "\n",
      "Test set: Accuracy: 37.09%\n",
      "Train Step: 2\tLoss: 2.254\n",
      "\n",
      "Test set: Accuracy: 49.42%\n",
      "Train Step: 3\tLoss: 2.188\n",
      "\n",
      "Test set: Accuracy: 58.45%\n",
      "Train Step: 4\tLoss: 2.116\n",
      "\n",
      "Test set: Accuracy: 65.52%\n",
      "Train Step: 5\tLoss: 1.998\n",
      "\n",
      "Test set: Accuracy: 70.11%\n",
      "Train Step: 6\tLoss: 1.648\n",
      "\n",
      "Test set: Accuracy: 72.65%\n",
      "Train Step: 7\tLoss: 1.211\n",
      "\n",
      "Test set: Accuracy: 77.72%\n",
      "Train Step: 8\tLoss: 0.850\n",
      "\n",
      "Test set: Accuracy: 81.59%\n",
      "Train Step: 9\tLoss: 0.868\n",
      "\n",
      "Test set: Accuracy: 83.75%\n",
      "Train Step: 10\tLoss: 0.512\n",
      "\n",
      "Test set: Accuracy: 85.32%\n",
      "Train Step: 11\tLoss: 0.672\n",
      "\n",
      "Test set: Accuracy: 86.81%\n",
      "Train Step: 12\tLoss: 0.386\n",
      "\n",
      "Test set: Accuracy: 88.04%\n",
      "Train Step: 13\tLoss: 0.790\n",
      "\n",
      "Test set: Accuracy: 88.84%\n",
      "Train Step: 14\tLoss: 0.337\n",
      "\n",
      "Test set: Accuracy: 89.41%\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(15):\n",
    "  train(epoch)\n",
    "  test()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}