day1_student_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3e3947be",
   "metadata": {},
   "source": [
    "# **Introduction to PyTorch** - Logistic Regression, MLP, and MNIST\n",
    "\n",
    "**Duke Machine Learning Winter School: Computer Vision 2022**\n",
    "\n",
    "**Code by Billy Carson**\n",
    "\n",
    "In this notebook, we're going to gain familiarity with the PyTorch deep learning framework by building PyTorch models to perform handwritten digit classification.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bff4abb6",
   "metadata": {},
   "source": [
    "## **1. Setting Up the Enivronment**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5579766a",
   "metadata": {},
   "source": [
    "### 1.1. Import libraries, modules, and functions\n",
    "\n",
    "First, we need to import necessary libraries, modules, and functions we'll use throughout this tutorial. To give ourselves access to the offerings of the PyTorch library, we'll import the [torch](https://pytorch.org/docs/stable/torch.html) package and the [torch.nn](https://pytorch.org/docs/stable/nn.html) module. Later, you can cut down on import bloat by only importing specific modules, functions, and objects that you'll need for your project. For now, we'll keep it simple by just using these two import statements.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6abec3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define random state\n",
    "random_state = 0\n",
    "\n",
    "# Import libraries, modules, and functions\n",
    "import time\n",
    "from idx2numpy import convert_from_file\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# PyTorch imports\n",
    "\n",
    "# ============================ #\n",
    "#  ----- YOUR CODE HERE -----  #\n",
    "# ============================ #\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7182003",
   "metadata": {},
   "source": [
    "### 1.2. GPU/CUDA availability\n",
    "\n",
    "When setting up your coding environment for a project or experiment, it's important that we are aware of the available computing resources. The resources we have access to will determine what models we can or can't use and the amount of data we can load into memory, among other considerations. For this tutorial, since we are dealing with simpler machine learning models, we won't need access to a [Graphics Processing Unit](https://en.wikipedia.org/wiki/Graphics_processing_unit) (GPU). GPUs help speed up the training of deep learning models by parallelizing many computations intead of computing sequentially. This parallelization is critical when training more complex models like [Convolutional Neural Networks](https://en.wikipedia.org/wiki/Convolutional_neural_network) (CNNs), and without it these models would take prohibitively long to train.\n",
    "\n",
    "We can check whether or not our environment has access to a GPU by calling the [`torch.cuda.is_available()`](https://pytorch.org/docs/stable/cuda.html) function. [CUDA](https://developer.nvidia.com/about-cuda) is a parallel computing platform and programming model developed by Nvidia for general computing on its own GPUs. This function will tell us if CUDA is available in this computing environment. Since we don't need/have access to a GPU, `torch.cuda.is_available()` should return `False`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7db8ea4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display GPU/CUDA availability\n",
    "\n",
    "# ============================ #\n",
    "#  ----- YOUR CODE HERE -----  #\n",
    "# ============================ #\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f96be876",
   "metadata": {},
   "source": [
    "## **2. Loading and Preparing the Data**\n",
    "\n",
    "One of the first (and often most important) steps in a machine learning pipeline is preparing the data. Here, we will load the MNIST dataset of images of handwritten digits and format this data so it can be used with PyTorch models.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a8dd177",
   "metadata": {},
   "source": [
    "### 2.1. Load MNIST dataset\n",
    "\n",
    "The [MNIST dataset](http://yann.lecun.com/exdb/mnist/) is very popular machine learning dataset, consisting of 70,000 grayscale, 28x28 images of handwritten digits.  We'll be using it as our example dataset for this section of the tutorial, with the goal being to predict which digit is depicted in each image.\n",
    "\n",
    "<img src=\"figures/mnist_examples.png\" alt=\"drawing\" width=\"500\"/>\n",
    "\n",
    "First, let's load the training and test sets of MNIST. The MNIST dataset needs to be converted to a type that can be operated on by PyTorch models. MNIST data will first be loaded from IDX file format and converted into NumPy arrays using the [idx2numpy](https://pypi.org/project/idx2numpy/) package. [NumPy](https://numpy.org) is a commonly-used Python library which provides support for multi-dimensional arrays and matrices, along with a variety of mathematical functions to operate on these arrays. In Python data science and machine learning pipelines, [NumPy arrays](https://numpy.org/doc/stable/reference/generated/numpy.array.html) are commonly used to store data and facilitate quick and efficient operations on array-like data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ac3e0ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load NumPy arrays of MNIST dataset\n",
    "mnist_data_dir = '/datasets/mnist/'\n",
    "X_raw_train_npy = convert_from_file(mnist_data_dir + 'train-images-idx3-ubyte')\n",
    "y_train_npy = convert_from_file(mnist_data_dir + 'train-labels-idx1-ubyte')\n",
    "X_raw_test_npy = convert_from_file(mnist_data_dir + 't10k-images-idx3-ubyte')\n",
    "y_test_npy = convert_from_file(mnist_data_dir + 't10k-labels-idx1-ubyte')\n",
    "\n",
    "# Display class type\n",
    "print('\\nClass types:')\n",
    "print('  Images:  %s' % (type(X_raw_train_npy)))\n",
    "print('  Labels:  %s' % (type(y_train_npy)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01218539",
   "metadata": {},
   "source": [
    "### 2.2 Convert data from NumPy arrays to PyTorch tensors"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e417ddc6",
   "metadata": {},
   "source": [
    "Now, MNIST data stored as NumPy arrays must be converted to [PyTorch tensors](https://pytorch.org/docs/stable/tensors.html) so that this data can be operated on by PyTorch models.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70995db3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert NumPy arrays to PyTorch tensors\n",
    "\n",
    "# ============================ #\n",
    "#  ----- YOUR CODE HERE -----  #\n",
    "# ============================ #\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc1a85e0",
   "metadata": {},
   "source": [
    "### 2.3. Explore data/PyTorch tensors\n",
    "\n",
    "Tensors are a specialized data structure very similar to arrays and matrices that house data in $N$ dimensions. Tensors can be thought of as generalizations of matrices (which are specifically 2-dimensional tensors) to $N$-dimensional space. \n",
    "\n",
    "In PyTorch, tensors are used to encode data, model inputs, model outputs, and model parameters. [PyTorch tensors](https://pytorch.org/docs/stable/tensors.html) are objects similar to [NumPy ndarrays](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html), except that tensors can run on GPUs or other hardware accelerators. Tensors are also optimized for automatic differentiation with [`autograd`](https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html), PyTorch’s automatic differentiation engine that powers neural network training. Here, where call the tensor method `shape()` to get an idea of the size and dimensionality of our data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e57fcbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display training data dimensionality\n",
    "print('\\nTraining data dimensionality:')\n",
    "print('  Images:  (%d, %d, %d)' % (X_raw_train.shape))\n",
    "print('  Labels:  (%s,)' % (y_train.shape))\n",
    "\n",
    "# Display test data dimensionality\n",
    "print('\\nTest data dimensionality:')\n",
    "print('  Images:  (%d, %d, %d)' % (X_raw_test.shape))\n",
    "print('  Labels:  (%d,)\\n' % (y_test.shape))\n",
    "\n",
    "# Display different labels\n",
    "print('Training labels:')\n",
    "print(np.unique(y_train.numpy()))\n",
    "print('\\nTest labels:')\n",
    "print(np.unique(y_test.numpy()))\n",
    "print()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7d6818c",
   "metadata": {},
   "source": [
    "### 2.4. Format tensors\n",
    "\n",
    "For our PyTorch models and loss functions to be able to operate on the tensors of MNIST data, they first need to be of a specific [tensor type](https://pytorch.org/docs/stable/tensors.html). Specifically, the image data tensors need to be 32-bit floating point tensors (tensor type `FloatTensor`) and the labels tensors need to be 64-bit signed integers (tensor type `LongTensor`).\n",
    "\n",
    "We can display the tensor type by calling the tensor method `type()`, which will return the tensor type. As we'll see, image data tensors need to be converted from type `ByteTensor` to type `FloatTensor` and labels from type `ByteTensor` to type `LongTensor`. We can make this conversion by calling the tensor method `float()`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20a80f88",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display original tensor type\n",
    "print('\\nTensor type (before conversion):')\n",
    "print('  Images: %s' % (X_raw_train.type()))\n",
    "print('  Labels: %s\\n' % (y_train.type()))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52a0c847",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert image tensor type to float\n",
    "\n",
    "# ============================ #\n",
    "#  ----- YOUR CODE HERE -----  #\n",
    "# ============================ #\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7ad7466",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display tensor type after conversion\n",
    "print('\\nTensor type (after conversion):')\n",
    "print('  Images:  %s' % (X_raw_train.type()))\n",
    "print('  Labels:  %s\\n' % (y_train.type()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bd75818",
   "metadata": {},
   "source": [
    "### 2.5. Scale data\n",
    "\n",
    "Often in many machine learning and data science applications, it helps to first scale the data. Scaling our data can ensure that all of our features are on approximately the same scale. Significant differences in the scales across input variables may increase the difficulty of our optimization problem [1].\n",
    "\n",
    "For this modelling problem, we'll scale our data to be between 0 and 1. Since MNIST data are black and white images, data elements correspond to pixel intensities with values between 0 and 255. Therefore, to scale our data between 0 and 1 we just need to divide our image data tensors by 255.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17aae656",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display data max and min\n",
    "print('\\nData range pre-scaling (min, max)):')\n",
    "print('  Train images:  %.1f, %.1f' % (torch.min(X_raw_train), torch.max(X_raw_train)))\n",
    "print('  Test images:  %.1f, %.1f\\n' % (torch.min(X_raw_test), torch.max(X_raw_test)))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f15ed629",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scale image pixel intensities between 0 and 1\n",
    "\n",
    "# ============================ #\n",
    "#  ----- YOUR CODE HERE -----  #\n",
    "# ============================ #\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "791c14c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display data max and min\n",
    "print('\\nData range post-scaling (min, max)):')\n",
    "print('  Train images:  %.1f, %.1f' % (torch.min(X_train), torch.max(X_train)))\n",
    "print('  Test images:  %.1f, %.1f\\n' % (torch.min(X_test), torch.max(X_test)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4133483b",
   "metadata": {},
   "source": [
    "### 2.6. Reshape data\n",
    "\n",
    "Sometimes we need to reshape the data so that our chosen model can operate on it. Computer vision models such as CNNs operate on directly 2-dimensional image data, so if we were using a CNN we would not need to reshape our MNIST data. However, for this tutorial we will be working with logistic regression and MLP models. Both of these models require input data to be a 1-dimensional vector, meanining that we'll have to \"flatten\" our image data.\n",
    "\n",
    "When flattening the image data, we want to preserve the first dimension of our tensors, which represents the samples or observations dimension. We can reshape our training and test data tensors using the [`view()`](https://pytorch.org/docs/stable/tensor_view.html) method. Additionally, this method avoids explicit data copy, thus facilitating fast and memory efficient reshaping, slicing, and element-wise operations on tensors [2].\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1c8ba3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reshape and \"flatten\" image data\n",
    "# Hint: -1 tells view() to infer the size of a given dimension\n",
    "\n",
    "# ============================ #\n",
    "#  ----- YOUR CODE HERE -----  #\n",
    "# ============================ #\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "884dfb93",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display training data dimensionality\n",
    "print('\\nTraining data dimensionality:')\n",
    "print('  Images:  (%d, %d)' % (X_train.shape))\n",
    "\n",
    "# Display test data dimensionality\n",
    "print('\\nTest data dimensionality:')\n",
    "print('  Images:  (%d, %d)\\n' % (X_test.shape))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b97b034",
   "metadata": {},
   "source": [
    "### 2.7. Define PyTorch dataset object\n",
    "\n",
    "[`torch.utils.data.Dataset`](https://pytorch.org/docs/stable/data.html#torch.utils.data.Dataset) is an abstract class representing a dataset. It allows us to treat the dataset as an object of a class, rather than a set of data and labels. Creating a custom dataset will allow us to have greater control over how our model loads and interacts with the data, as well as permit built-in batching, shuffling, and loading data in parallel later on [3]. A custom `Dataset` class must implement three functions: ` __init__`, `__len__`, and `__getitem__`:\n",
    "\n",
    "* The `__init__` function is run once when instantiating the `Dataset` object.\n",
    "* The `__len__` function returns the number of samples in our dataset.\n",
    "* The `__getitem__` function loads and returns a sample from the dataset at the given index idx.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e53c4414",
   "metadata": {},
   "outputs": [],
   "source": [
    "# MNIST PyTorch dataset object\n",
    "\n",
    "# ============================ #\n",
    "#  ----- YOUR CODE HERE -----  #\n",
    "# ============================ #\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5edd6db7",
   "metadata": {},
   "source": [
    "### 2.8. Instantiate PyTorch datasets\n",
    "\n",
    "Here, we instantiate a training and test version of the `MNISTDataset` class we defined above. Instantiating a class creates a copy or instance of the class which inherits all class attributes and methods. To instantiate an object in Python, we simply call the class as if it were a function, passing the requisite arguments to the `__init__` method. The newly created object will then be assigned to whatever variable name we choose.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a756756b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate MNIST training and test PyTorch dataset objects\n",
    "\n",
    "# ============================ #\n",
    "#  ----- YOUR CODE HERE -----  #\n",
    "# ============================ #\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0aac7434",
   "metadata": {},
   "source": [
    "### 2.9. Instantiate PyTorch data loader objects\n",
    "\n",
    "While we could interface directly with the data as a PyTorch `torch.utils.data.Dataset` object, it's often easier to use a PyTorch [`torch.utils.data.DataLoader`](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader) object. The PyTorch `DataLoader` is an iterator that automatically takes care of:\n",
    "* batching the data\n",
    "* shuffling the data\n",
    "* loading the data in parallel using multiprocessing workers\n",
    "\n",
    "Similar to how we created both training and test set instances of PyTorch `Dataset` objects, we will now create instances of training and test set `DataLoader` objects.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f95e7ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate MNIST training and test PyTorch data loader objects\n",
    "\n",
    "# ============================ #\n",
    "#  ----- YOUR CODE HERE -----  #\n",
    "# ============================ #\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d639df49",
   "metadata": {},
   "source": [
    "## **3. Building a Logistic Regression Model**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f9a0fc2",
   "metadata": {},
   "source": [
    "### 3.1. Define logistic regression model\n",
    "\n",
    "For the first part of this tutorial, we'll be building a logistic regression model, which is essentially a fully-connected neural network without any hidden layers. Although basic, logistic regression can perform quite well on many simple classification tasks.\n",
    "\n",
    "PyTorch has higher level abstractions to help speed up implementation and improve model organization. While there are many ways to organize PyTorch code, one common paradigm is the `torch.nn.Module`. PyTorch uses modules to represent neural networks. Modules are:\n",
    "\n",
    "* Building blocks of stateful computation. PyTorch provides a robust library of pre-defined modules and makes it simple to define new custom modules, facilitating construction of many different deep neural network architecturess.\n",
    "* Tightly integrated with PyTorch’s autograd system. Modules make it simple to specify learnable parameters for PyTorch’s Optimizers to update.\n",
    "* Easy to work with and transform. Modules are straightforward to save, restore, transfer between CPU/GPU/TPU devices, and more [4].\n",
    "\n",
    "For more on `torch.nn.Module`, see [this](https://pytorch.org/tutorials/beginner/nn_tutorial.html) tutorial. To create our LogisticRegression model that inherits from `torch.nn.Module`, we need to define two methods: `__init__()` and `forward()`. `__init__()` initializes the model parameters and assigns layers to object attributes. The `forward()` method computes the forward pass of the model by passing minibatches of input data through the different layers of the model.\n",
    "\n",
    "To get an idea of what types of PyTorch layers to include in our logistic regression models, let's take a look at the mathematical formulation of logistic regression. Multiclass logistic regression can be expressed in matrix form as:\n",
    "\n",
    "\\begin{align}\n",
    "y = x W + b \n",
    "\\end{align}\n",
    "\n",
    "To take advantage of parallel computation, we commonly process multiple inputs $x$ at once, in a minibatch. We can individual observations or samples of our data $x$ into a batch matrix of $X$. This can be expressed in matrix form as:\n",
    "\n",
    "\\begin{align}\n",
    "Y = X W + b \n",
    "\\end{align}\n",
    "\n",
    "In our specific example, the minibatch size $m$ is 100, the dimension of the data is 28 $\\times$ 28 = 784, and the number of classes $c$ is 10. Below is an image depicting the transformation of a batch of data $X$ into a set of corresponding predictions $Y$:\n",
    "\n",
    "<img src=\"figures/mnist_matmul.png\" width=\"500\"/>\n",
    "\n",
    "PyTorch offers a pre-defined [`torch.nn.Linear`](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) layer that computes a linear transformation of the input data. We can use this layer to define the parameters of our logistic regression model. Typically, after the `Linear` layer we would implement a softmax activation function that converts our output to a multinomial probability distribution. However, PyTorch has a way of building this calculation into its loss functions. This helps model stability during training. Therefore, we won't add a softmax activation function for our models and instead leave that calculation to PyTorch's loss functions.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7067381",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Logistic regression model\n",
    "\n",
    "# ============================ #\n",
    "#  ----- YOUR CODE HERE -----  #\n",
    "# ============================ #\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5054e6c0",
   "metadata": {},
   "source": [
    "### 3.2. Instantiate model\n",
    "\n",
    "To instantiate our logistic regression model, we need to pass two parameters to the instantiation method `__init__()`: the model `input_dim` and model `output_dim`. `input_dim` will be the dimensionality of our flattened image data, which is 784. `output_dim` corresponds to the number of digits we are trying to classify, which is 10.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "682bd425",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define model input and output dimensions from data\n",
    "input_dim = X_train.shape[1]             # second dimension is data dimensionality\n",
    "output_dim = len(np.unique(y_train.numpy()))  # output dim. is equal to number of unique labels\n",
    "\n",
    "# Instantiate logistic regression model\n",
    "\n",
    "# ============================ #\n",
    "#  ----- YOUR CODE HERE -----  #\n",
    "# ============================ #\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49c54c82",
   "metadata": {},
   "source": [
    "## **4. Training the Logistic Regression Model**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "403ff641",
   "metadata": {},
   "source": [
    "### 4.1. Defining a loss function\n",
    "\n",
    "We need a way to evaluate our model predictions. This evaluation is done via a loss function, which takes the model's predictions and returns a single number summarizing model performance. The magnitude of this loss will inform how much or how little we should update the model parameters. The loss we commonly use in classification is cross-entropy, a concept from information theory. Cross-entropy not only captures whether or not model predictions are correct, it also accounts for how *confident* these predictions are. This encourages the model to produce very high probabilities for correct answers while driving down the probabilities for the wrong answers.\n",
    "\n",
    "To define our loss function object, we can use PyTorch's implementation of cross entropy [`torch.nn.CrossEntropyLoss`](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss). In addition to computing the cross entropy loss, this loss function takes logits as inputs computes the log-softmax internally, which is why we did not have to add a softmax activation at the end of our logistic regression implementation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54bce19c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate cross-entropy loss function\n",
    "\n",
    "# ============================ #\n",
    "#  ----- YOUR CODE HERE -----  #\n",
    "# ============================ #\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe5be31d",
   "metadata": {},
   "source": [
    "### 4.2. Defining optimizer and learning rate\n",
    "\n",
    "Now that we've defined our loss function as a way of quantifying how good or bad our model predictions are, we can improve our model by changing the parameters in a way that minimizes the loss. For neural networks, this is done via [backpropagation](https://en.wikipedia.org/wiki/Backpropagation): we take the gradient of the loss with respect to model parameters and take a \"step\" in the direction that reduces our loss. If we were not using a deep learning framework like PyTorch, we would have to go through and derive all the gradients ourselves by hand, then code them into our training pipeline. Luckily, Pytorch has automated gradient calculation with [`torch.autograd`](https://pytorch.org/docs/stable/autograd.html): an \"automatic differentiation engine that powers neural network training\" [5].\n",
    "\n",
    "Next, we define our optimizer and the learning rate we'll use for training the model. [`torch.optim`](https://pytorch.org/docs/stable/optim.html) is a package of various, commonly-used optimization algorithms implementations. A `torch.optim` optimizer object will keep track of the current state of model parameters and update the parameters based on the gradients computed by `torch.autograd`. To construct an optimizer you must provide an iterable containing the model parameters to optimize. This can be done by calling the `parameters()` method of our model (a method that is inherited from `torch.nn.Module`). Then, you can specify optimizer-specific options such as the learning rate and weight decay. For many deep learning applications, standard [stochastic gradient descent](https://scikit-learn.org/stable/modules/sgd.html) (SGD) optimizes models during training quite well. We can use PyTorch's `torch.optim` implementation of SGD as our optimizer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aaab66d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate stochastic gradient descent (SGD) optimizer\n",
    "\n",
    "# ============================ #\n",
    "#  ----- YOUR CODE HERE -----  #\n",
    "# ============================ #\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaef993f",
   "metadata": {},
   "source": [
    "### 4.3. Define number of training epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3dbc4f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of training epochs (epochs = # of passes through data)\n",
    "n_epochs = 10\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "745b7afd",
   "metadata": {},
   "source": [
    "### Training logistic regression on MNIST\n",
    "\n",
    "To train our model, we need to repeat the following steps:\n",
    "\n",
    "1. Draw a minibatch.\n",
    "2. Reset all gradients to zero in the buffers.\n",
    "3. Perform the forward pass (compute prediction, calculate loss).\n",
    "4. Perform the backward pass (compute gradients with respects to loss, optimize parameters with SGD).\n",
    "\n",
    "Performing these steps for the entirety of a dataset once is referred to as an epoch. We'll accomplish the above steps using the following PyTorch functionalities:\n",
    "\n",
    "1. Draw a minibatches by iterating over our training `DataLoader` instance. Remember, our custom PyTorch data loader takes care of batching the data, shuffling the data, and loading the data in parallel.\n",
    "2. Reset all gradients to zero in the buffers using the `zero_grad()` method of our optimizer. We need to set the gradients to zero before performing backpropragation because PyTorch accumulates the gradients on subsequent backward passes if `zero_grad()` is not called. Accumulation of gradients is convenient for models such as recurrent neural networks (RNNs); however, for this tutorial we'll zero out gradients before processing each minibatch of data.\n",
    "3. Perform the forward pass by providing the minibatch of data to our model's `forward()` method.\n",
    "4. Perform the backward pass by calling the `backward()` method of our model's computed loss and the `step()` method of our optimizer. The `backward()` method of our model computes the parameter gradients with respects to the calculated loss, and the `step()` method of our optimizer updates the parameters according to the calculated gradients.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3781ef91",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get weights\n",
    "weights_before = lr_model.linear.weight.detach().numpy().copy()\n",
    "\n",
    "# Time model training\n",
    "print('\\nTraining logistic regression model...\\n')\n",
    "time_start = time.time()\n",
    "\n",
    "# Place model in training mode\n",
    "# .train() method affects operations such as dropout and batch normalization\n",
    "lr_model.train()\n",
    "\n",
    "# Train model/iterate over epochs\n",
    "for epoch in range(n_epochs):\n",
    "    # Initialize epoch metrics variables\n",
    "    n_obs = 0\n",
    "    loss_sum = 0\n",
    "    n_correct = 0\n",
    "    \n",
    "    # Iterate through training data mini-batches, zero out gradients,\n",
    "    # implement forward and backward pass through model, and update epoch metrics variables\n",
    "    \n",
    "    # ============================ #\n",
    "    #  ----- YOUR CODE HERE -----  #\n",
    "    # ============================ #\n",
    "        \n",
    "    # Calculate epoch training loss and training accuracy\n",
    "    loss_train = loss_sum / n_obs\n",
    "    acc_train = n_correct / n_obs\n",
    "    \n",
    "    # Display training progress\n",
    "    prog_disp_freq = 1   # frequency of training progress display\n",
    "    if (((epoch + 1) % prog_disp_freq) == 0) | (epoch == 0) | ((epoch + 1) == n_epochs):\n",
    "        print('  E%02d | train loss: %s | train acc.: %s' % \n",
    "              (epoch + 1, '{:.4f}'.format(loss_train), '{:.4f}'.format(acc_train)))\n",
    "\n",
    "# Time model training\n",
    "time_end = time.time()\n",
    "print('\\nLogistic regression model training complete.\\n')\n",
    "print('Time to train logistic regression model:  %.1f s\\n' % (time_end - time_start))\n",
    "\n",
    "# Get weights after training\n",
    "weights_after = lr_model.linear.weight.detach().numpy().copy()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d407a9ba",
   "metadata": {},
   "source": [
    "### 4.4. Visualize logistic regression weights\n",
    "\n",
    "One of the nice things about logistic regression is that since it is a linear model we can interpret what the model has learned by plotting the weights before and after training.\n",
    "\n",
    "Before training, the model weights for each digit/output look like a static TV screen. Since the weights were randomly initialized, it is expected that the model parameters will appear random before training on any data. After training, we can see the model learned a general template for each digit. Remember that logistic regression takes a dot product between the weights of each digit and input. Therefore, the more the input matches the template or weights for a given digit, the higher the value of the dot product for that digit will be which makes the model more likely to predict that digit.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "526a199f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot logistic regression weights before and after training\n",
    "fig1, ax1 = plt.subplots(2, 10, figsize=(14, 2.6))\n",
    "for digit in range(10):\n",
    "    ax1[0, digit].imshow(np.reshape(weights_before[digit, :], newshape=(28, 28)), cmap='gray')\n",
    "    ax1[1, digit].imshow(np.reshape(weights_after[digit, :], newshape=(28, 28)), cmap='gray')\n",
    "    ax1[0, digit].axis('off')\n",
    "    ax1[1, digit].axis('off')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b233d68a",
   "metadata": {},
   "source": [
    "## **5. Evaluating Logistic Regression Model on Test Set**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "612e1197",
   "metadata": {},
   "source": [
    "### 5.1. Evaluate model on test set\n",
    "\n",
    "Once our model has been trained, we now need to evaluate it on a set of hold-out data, or a test set. Evaluating trained models on a test set is a critical step of machine learning pipelines, as this gives us an idea of how well our model will generalize in the real world to new, never-seen-before data.\n",
    "\n",
    "First, we need to put our model in evaluation mode using the `eval()` method. This method affects operations with different training and test-time behavior such as dropout and batch normalization. Although we don't use any layers in this tutorial that would be affected by this method, it's a good thing to get into the habit of doing.\n",
    "\n",
    "Additionally, since we aren't training our model any more via backpropagation, we don't need to compute a loss or calculate gradients. We can use the [`torch.no_grad()`](https://pytorch.org/docs/stable/generated/torch.no_grad.html) context-manager that disables gradient calculation. Disabling gradient calculation is useful for model inference, when you are sure that you will not call Tensor.backward(). Using `torch.no_grad()` will help reduce memory consumption for computations that would otherwise require gradients during training [6].\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e7f59b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Place model in evaluation mode\n",
    "# .eval() method affects operations such as dropout and batch normalization\n",
    "lr_model.eval()\n",
    "\n",
    "# Initialize test set metrics variables\n",
    "n_obs = 0\n",
    "n_correct = 0\n",
    "\n",
    "# Disable gradient calculation, iterate through test data mini-batches,\n",
    "# implement model forward pass, and update test set metrics variables\n",
    "\n",
    "# ============================ #\n",
    "#  ----- YOUR CODE HERE -----  #\n",
    "# ============================ #\n",
    "\n",
    "# Calculate test accuracy\n",
    "acc_test = n_correct / n_obs\n",
    "\n",
    "# Display test accuracy\n",
    "print('\\nLogistic regression model MNIST test acc.:  %.4f\\n' % (acc_test))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38bab38a",
   "metadata": {},
   "source": [
    "## **6. Training and Evaluating Multi-Layer Perceptron on MNIST Dataset**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "281bcd1a",
   "metadata": {},
   "source": [
    "### 6.1. Defining MLP model\n",
    "\n",
    "Similarly to how we previously defined our logistic regression model by using PyTorch's object-oriented style, we can implement a neural network/[multi-layer perceptron](https://machinelearningmastery.com/neural-networks-crash-course/) (MLP) with just a few tweaks. Instead of just one `Linear` layer, this time we'll use two. Additionally, after the first `Linear` layer we'll apply a non-linear [activation function](https://machinelearningmastery.com/choose-an-activation-function-for-deep-learning/). Activation functions are a critical part of neural network design, and help give our neural network model added expressivity for modelling complex, non-linear relationships. The de facto nonlinearity used in neural networks is the [rectified linear unit](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)) (ReLU). We can implement this ReLU activation function as a layer in our MLP by using PyTorch's [`torch.nn.ReLU`](https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html) layer.\n",
    "\n",
    "\n",
    "*Note: multi-layer perceptron and artifical neural network (ANN) are often used interchangeably to describe a feed-forward, fully-connected neural network with many layers.*\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6906f7af",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Multi-layer perceptron model\n",
    "\n",
    "# ============================ #\n",
    "#  ----- YOUR CODE HERE -----  #\n",
    "# ============================ #\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1c1c934",
   "metadata": {},
   "source": [
    "### 6.2. Instantiate MLP model\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ea847af",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define model input and output dimensions from data\n",
    "input_dim = X_train.shape[1]                  # second dimension is data dimensionality\n",
    "output_dim = len(np.unique(y_train.numpy()))  # output dim. is equal to number of unique labels\n",
    "\n",
    "# Instantiate multi-layer perceptron model\n",
    "\n",
    "# ============================ #\n",
    "#  ----- YOUR CODE HERE -----  #\n",
    "# ============================ #\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37a5deda",
   "metadata": {},
   "source": [
    "### 6.3. Re-define optimizer with MLP model parameters\n",
    "\n",
    "Since a PyTorch optimizer is an object that keeps track of and updates model parameters, we need to re-define our optimizer with the parameters of our newly defined MLP model. We can keep the same learning rate of 0.1 as last time.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "709a051f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate stochastic gradient descent (SGD) optimizer\n",
    "\n",
    "# ============================ #\n",
    "#  ----- YOUR CODE HERE -----  #\n",
    "# ============================ #\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53c237d9",
   "metadata": {},
   "source": [
    "### 6.4. Train MLP model on training data\n",
    "\n",
    "Now we train our MLP model on MNIST. Ideally, since we have now defined a non-linear model with added expressivity by including a hidden layer and a non-linear activation function following this hidden layer, we would like to see better classification performance than that achieved by logistic regression. As we will see, these modifications do indeed result in better hand-written digit classification performance.\n",
    "\n",
    "Even though we changed the type of model, we can keep the same exact training loop we developed for training logistic regression on MNIST.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4621a030",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of training epochs (epochs = # of passes through data)\n",
    "n_epochs = 10\n",
    "\n",
    "# Time model training\n",
    "print('\\nTraining MLP model...\\n')\n",
    "time_start = time.time()\n",
    "\n",
    "# Place model in training mode\n",
    "# .train() method affects operations such as dropout and batch normalization\n",
    "mlp_model.train()\n",
    "\n",
    "# Train model/iterate over epochs\n",
    "for epoch in range(n_epochs):\n",
    "    # Initialize epoch metrics variables\n",
    "    n_obs = 0\n",
    "    loss_sum = 0\n",
    "    n_correct = 0\n",
    "    \n",
    "    # Iterate through training data mini-batches, zero out gradients,\n",
    "    # implement forward and backward pass through model, and update epoch metrics variables\n",
    "    \n",
    "    # ============================ #\n",
    "    #  ----- YOUR CODE HERE -----  #\n",
    "    # ============================ #\n",
    "    \n",
    "    # Calculate epoch training loss and training accuracy\n",
    "    loss_train = loss_sum / n_obs\n",
    "    acc_train = n_correct / n_obs\n",
    "    \n",
    "    # Display training progress\n",
    "    prog_disp_freq = 1   # frequency of training progress display\n",
    "    if (((epoch + 1) % prog_disp_freq) == 0) | (epoch == 0) | ((epoch + 1) == n_epochs):\n",
    "        print('  E%02d | train loss: %s | train acc.: %s' % \n",
    "              (epoch + 1, '{:.4f}'.format(loss_train), '{:.4f}'.format(acc_train)))\n",
    "\n",
    "# Time model training\n",
    "time_end = time.time()\n",
    "print('\\nMLP model training complete.\\n')\n",
    "print('Time to train MLP model:  %.1f s\\n' % (time_end - time_start))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fe73b7e",
   "metadata": {},
   "source": [
    "### 6.5. Evaluate MLP model on test set\n",
    "\n",
    "We'll now evaluate our MLP model on the test set. The added expressivity and complexity of our model can be a double-edged sword. While this added complexity gives our model the capacity to learn more complex, non-linear relationships, we must now also be weary of [overfitting](https://www.ibm.com/cloud/learn/overfitting). Overfitting occurs when a model learns the information represented in training data to the point where it's performance on hold-out data is negatively influenced. Deep learning models can have millions (sometimes [billions](https://developer.nvidia.com/blog/openai-presents-gpt-3-a-175-billion-parameters-language-model/)!) of trainable parameters, which means that the number of model parameters can be greater than the number of data points on which the model is trained. These \"over-parametrized\" models can be prone to overfitting, especially in cases of small data. Here, we're hoping that our test accuracy is similar to our training accuracy of around 96%-97%. As we'll see, our model did not overfit the training set and achieves a comparable test set accuracy.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "500c3277",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Place model in evaluation mode\n",
    "# .eval() method affects operations such as dropout and batch normalization\n",
    "mlp_model.eval()\n",
    "\n",
    "# Initialize test set metrics variables\n",
    "n_obs = 0\n",
    "n_correct = 0\n",
    "\n",
    "# Disable gradient calculation, iterate through test data mini-batches,\n",
    "# implement model forward pass, and update test set metrics variables\n",
    "\n",
    "# ============================ #\n",
    "#  ----- YOUR CODE HERE -----  #\n",
    "# ============================ #\n",
    "\n",
    "# Calculate test accuracy\n",
    "acc_test = n_correct / n_obs\n",
    "\n",
    "# Display test accuracy\n",
    "print('\\nMLP model MNIST test acc.:  %.4f\\n' % (acc_test))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b45a537",
   "metadata": {},
   "source": [
    "## **7. Full Pipeline** - Training PyTorch Model on MNIST Dataset\n",
    "\n",
    "Finally, we can combine our entire deep learning pipeline into a single cell.\n",
    "\n",
    "In practice, when using Jupyter notebooks you shouldn't combine all of your code into a single cell, as this defeats the purpose of the modular nature of Jupyter notebooks. However, it can be helpful to see the entirety of a basic deep learning pipeline from beginning to end in code (without the extra bloat of my writing).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49225f31",
   "metadata": {},
   "outputs": [],
   "source": [
    "# -----------------\n",
    "# IMPORT STATEMENTS\n",
    "# -----------------\n",
    "\n",
    "\n",
    "\n",
    "# -----------------------------------------------\n",
    "# CLASS DEFINITIONS - MNIST DATASET AND MLP MODEL\n",
    "# -----------------------------------------------\n",
    "\n",
    "\n",
    "\n",
    "# ---------------------------\n",
    "# LOADING AND FORMATTING DATA\n",
    "# ---------------------------\n",
    "\n",
    "\n",
    "\n",
    "# -------------------------------------------\n",
    "# INSTANTIATE DATASET AND DATA LOADER OBJECTS\n",
    "# -------------------------------------------\n",
    "\n",
    "\n",
    "\n",
    "# -----------------------------------------------\n",
    "# INSTANTIATE MODEL, LOSS FUNCTION, AND OPTIMIZER\n",
    "# -----------------------------------------------\n",
    "\n",
    "\n",
    "\n",
    "# ----------------------------\n",
    "# TRAIN MODEL ON TRAINING DATA\n",
    "# ----------------------------\n",
    "\n",
    "\n",
    "\n",
    "# ---------------------------\n",
    "# EVALUATE MODEL ON TEST DATA\n",
    "# ---------------------------\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7313b0c9",
   "metadata": {},
   "source": [
    "## **References**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9279f95c",
   "metadata": {},
   "source": [
    "[1] https://machinelearningmastery.com/how-to-improve-neural-network-stability-and-modeling-performance-with-data-scaling/\n",
    "\n",
    "[2] https://pytorch.org/docs/stable/tensor_view.html\n",
    "\n",
    "[3] https://pytorch.org/tutorials/beginner/data_loading_tutorial.html\n",
    "\n",
    "[4] https://pytorch.org/docs/stable/notes/modules.html\n",
    "\n",
    "[5] https://pytorch.org/docs/stable/autograd.html, https://pytorch.org/docs/stable/optim.html\n",
    "\n",
    "[6] https://pytorch.org/docs/stable/generated/torch.no_grad.html\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}