3B_PT_Convolutional_Neural_Networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Convolutional Neural Networks (CNNs) in PyTorch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Representing images digitally\n",
    "\n",
    "While convolutional neural networks (CNNs) see a wide variety of uses, they were originally designed for images, and CNNs are still most commonly used for vision-related tasks.\n",
    "For today, we'll primarily be focusing on CNNs for images.\n",
    "Before we dive into convolutions and neural networks, it's worth prefacing with how images are represented by a computer, as this understanding will inform some of our design choices.\n",
    "\n",
    "Previously, we saw an example of a digitized MNIST handwritten digit.\n",
    "Specifically, we represent it as an $H \\times W$ table, with the value of each element storing the intensity of the corresponding pixel.\n",
    "\n",
    "<img src=\"Figures/mnist_digital.PNG\" alt=\"mnist_digital\" style=\"width: 600px;\"/>\n",
    "\n",
    "With a 2D representation as above, we for the most part can only efficiently represent grayscale images.\n",
    "What if we want color?\n",
    "There are many schemes for storing color, but one of the most common ones is the [RGB color model](https://en.wikipedia.org/wiki/RGB_color_model).\n",
    "In such a system, we store 3 tables of pixel intensities (each called a *channel*), one each for the colors red, green, and blue (hence RGB), resulting in an $H \\times W \\times 3$ tensor.\n",
    "Pixel values for a particular channel indicate how much of the corresponding color the image has at a particular location.\n",
    "\n",
    "Let's load an image and see this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "# Read image\n",
    "\n",
    "print(\"Shape of the image tensor: {}\".format(None))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the image we loaded has height and width of $620 \\times 1175$, with 3 channels corresponding to RGB.\n",
    "\n",
    "We can easily slice out and view individual color channels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Red channel\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Blue channel\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While we have so far considered only 3 channel RGB images, there are many settings in which we may consider a different number of channels.\n",
    "For example, [hyperspectral imaging](https://en.wikipedia.org/wiki/Hyperspectral_imaging) uses a wide range of the electromagnetic spectrum to characterize a scene.\n",
    "Such modalities may have hundreds of channels or more.\n",
    "Additionally, we'll soon see that certain intermediate representations in a CNN can be considered images with many channels.\n",
    "However, we'll focus much of today's discussion to 1 channel grayscale and 3 channel RGB images as our inputs, as is most commonly the case in computer vision."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convolutions\n",
    "Convolutional neural networks (CNNs) are a class of neural networks that have convolutional layers.\n",
    "CNNs are particularly effective for data that have spatial structures and correlations (e.g. images).\n",
    "We'll focus on CNNs applied to images in this tutorial.\n",
    "Recall that a multilayer perceptron (MLP) is entirely composed of fully connected layers, which are each a matrix multiply operation (and addition of a bias) followed by a non-linearity (e.g. sigmoid, ReLU). \n",
    "A convolutional layer is similar, except the matrix multiply operation is replaced with a convolution operation (in practice a cross-correlation). \n",
    "Note that a CNN need not be entirely composed of convolutional layers; in fact, many popular CNN architectures end in fully connected layers.\n",
    "\n",
    "As before, since we're building neural networks, let's start by loading PyTorch. We'll find NumPy useful as well, so we'll also import that here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Review: Fully connected layer\n",
    "In a fully connected layer, the input $x \\in \\mathbb R^{M \\times C_{in}}$ is a vector (or, rather a batch of vectors), where $M$ is the minibatch size and $C_{in}$ is the dimensionality of the input. \n",
    "We first matrix multiply the input $x$ by a weight matrix $W$.\n",
    "This weight matrix has dimensions $W \\in \\mathbb R^{C_{in} \\times C_{out}}$, where $C_{out}$ is the number of output units.\n",
    "We then add a bias for each output, which we do by adding $b \\in \\mathbb{R}^{C_{out}}$.\n",
    "The output $y \\in \\mathbb{R}^{M \\times C_{out}}$ of the fully connected layer then:\n",
    "\n",
    "\\begin{align*}\n",
    "y = \\text{ReLU}(x W + b)\n",
    "\\end{align*}\n",
    "\n",
    "Remember, the values of $W$ and $b$ are variables that we are trying to learn for our model. \n",
    "Below we have a visualization of what the matrix operation looks like (bias term and activation function omitted).\n",
    "\n",
    "<img src=\"Figures/mnist_matmul.PNG\" width=\"500\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a random flat input vector\n",
    "\n",
    "\n",
    "# Create weight matrix variable\n",
    "\n",
    "\n",
    "# Create bias variable\n",
    "\n",
    "\n",
    "# Apply fully connected layer\n",
    "\n",
    "\n",
    "# Print input/output shape\n",
    "print(\"Input shape: {}\".format(None))\n",
    "print(\"Output shape: {}\".format(None))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convolutional layer\n",
    "\n",
    "In a convolutional layer, we convolve the input $x$ with a convolutional kernel (aka filter), which we also call $W$, producing output $y$:\n",
    "\n",
    "\\begin{align*}\n",
    "y = \\text{ReLU}(W*x + b)\n",
    "\\end{align*}\n",
    "\n",
    "In the context of CNNs, the output $y$ is often referred to as feature maps. As with a fully connected layer, the goal is to learn $W$ and $b$ for our model.\n",
    "\n",
    "Unlike the input of a fully connected layer, which is $x \\in \\mathbb R^{M\\times C_{in}}$, the dimensionality of an image input is 4D: $x \\in \\mathbb R^{M \\times C_{in} \\times H_{in} \\times W_{in}}$, where $M$ is still the batch size, $C_{in}$ is the number of channels of the input (e.g. 3 for RGB), and $H_{in}$ and $W_{in}$ are the height and width of the image.\n",
    "\n",
    "The weight parameter $W$ is also different in a convolutional layer.\n",
    "Unlike the 2-D weight matrix for fully connected layers, the kernel is 4-D with dimensions $W \\in \\mathbb R^{C_{out} \\times C_{in} \\times H_K \\times W_K }$, where $H_K$ and $W_K$ are the kernel height and weight, respectively.\n",
    "A common choice for $H_K$ and $W_K$ is $H_K = W_K = 3$ or $5$, but this tends to vary depending on the architecture.\n",
    "Convolving the input with the kernel and adding a bias then gives an output $y \\in \\mathbb R^{M \\times C_{out} \\times H_{out} \\times W_{out}}$.\n",
    "If we use \"same\" padding and a stride of $1$ in our convolution (more on this later), our output will have the same spatial dimensions as the input: $H_{out}=H_{in}$ and $W_{out}=W_{in}$.\n",
    "\n",
    "If you're having trouble visualizing this operation in 4D, it's easier to think about for a single member of the minibatch, one convolutional kernel at a time. \n",
    "Consider a stack of $C_{out}$ number of kernels, each of which are 3D ($C_{in} \\times H_K \\times W_K $). \n",
    "This 3D volume is then slid across the input (which is also 3D: $C_{in} \\times H_{in} \\times W_{in}$) in the two spatial dimensions (along $H_{in}$ and $W_{in}$). \n",
    "The outputs of the multiplication of the kernel and the input at every location creates a single feature map that is $H_{out} \\times W_{out}$. \n",
    "Stacking the feature maps generated by each kernel gives the 3D output $C_{out} \\times H_{out} \\times W_{out} $.\n",
    "Repeat the process for all $M$ inputs in the minibatch, and we get a 4D output $M  \\times C_{out} \\times H_{out} \\times W_{out}$.\n",
    "\n",
    "<img src=\"Figures/conv_filters.png\" alt=\"Convolutional filters\" style=\"width: 400px;\"/>\n",
    "\n",
    "A few more things to note:\n",
    "- Notice the ordering of the dimensions of the input (batch, channels in, height, width).\n",
    "This is commonly referred to as $NCHW$ ordering.\n",
    "Many other languages and libraries (e.g. MATLAB, TensorFlow, the image example at the beginning of this notebook) instead default to the slightly different $NHWC$ ordering.\n",
    "PyTorch defaults to $NCHW$, as it more efficient computationally, especially with CUDA. \n",
    "- An additional argument for the convolution is the *stride*, which controls the how far we slide the convolutional filter as we move it along the input image. \n",
    "The convolutional operator, from its signal processing roots, by default considers a stride length of 1 in all dimensions, but in some situations we would like to consider strides more than 1 (or even less than 1). \n",
    "More on this later.\n",
    "- In the context of signal processing, convolutions usually result in outputs that are larger than the input size, which results from when the kernel \"hangs off the edge\" of the input on both sides. \n",
    "This might not always be desirable.\n",
    "We can control this by controlling the padding of the input.\n",
    "Typically, we use pad the input to ensure the output has the same spatial dimensions as the input (assuming stride of 1); this makes it easier for us to keep track of what the size of our model is.\n",
    "\n",
    "Let's implement this convolution operator in code.\n",
    "There is a convolution implementation in `torch.nn.functional`, which we use here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a random image input tensor\n",
    "\n",
    "\n",
    "# Create convolutional kernel variable\n",
    "\n",
    "\n",
    "# Create bias variable\n",
    "\n",
    "\n",
    "# Apply convolutional layer\n",
    "\n",
    "\n",
    "# Print input/output shape\n",
    "print(\"Input shape: {}\".format(None))\n",
    "print(\"Convolution output shape: {}\".format(None))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just like in a MLP, we can stack multiple of these convolutional layers. \n",
    "In the *Representing Images Digitally* section, we briefly mentioned considering images with channels more than 3.\n",
    "Observe that the input to the second layer (i.e. the output of the first layer) can be viewed as an \"image\" with $C_{out}$ channels.\n",
    "Instead of each channel representing a color content though, each channel effectively represents how much the original input image activated a particular convolutional kernel.\n",
    "Given $C_{out}$ kernels that are each $C_{in} \\times H_K \\times W_K$, this results in $C_{out}$ channels for the output of the convolution.\n",
    "\n",
    "Note that we need to change the dimensions of the convolutional kernel such that its input channels matches the number of output channels of the previous layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2nd layer variables\n",
    "\n",
    "\n",
    "# Apply 2nd convolutional layer\n",
    "\n",
    "\n",
    "# Print output shape\n",
    "print(\"Second convolution output shape: {}\".format(None))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact, we typically perform these convolution operations many times. \n",
    "Popular CNN architectures for image analysis today can be 100+ layers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reshaping\n",
    "\n",
    "You'll commonly finding yourself needing to reshape tensors while building CNNs.\n",
    "The PyTorch function for doing so is `view()`. \n",
    "Anyone familiar with NumPy will find it very similar to `np.reshape()`.\n",
    "Importantly, the new dimensions must be chosen so that it is possible to rearrange the input into the shape of the output (i.e. the total number of elements must be the same).\n",
    "As with NumPy, you can optionally replace one of the dimensions with a `-1`, which tells `torch` to infer the missing dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get an idea of why reshaping is need in a CNN, let's look at a diagram of a simple CNN.\n",
    "\n",
    "<img src=\"Figures/mnist_cnn_ex.png\" alt=\"mnist_cnn_ex\" style=\"width: 500px;\"/>\n",
    "\n",
    "First of all, the CNN expects a 4D input, with the dimensions corresponding to `[batch, channel, height, width]`.\n",
    "Your data may not come in this format, so you may have to reshape it yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reshape flat input image into a 4D batched image input\n",
    "\n",
    "\n",
    "# Print input shape\n",
    "print(None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CNN architectures also commonly contain fully connected layers or a softmax, as we're often interested in classification.\n",
    "Both of these expect 2D inputs with dimensions `[batch, dim]`, so you have to \"flatten\" a CNN's 4D output to 2D.\n",
    "For example, to flatten the convolutional feature maps we created earlier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Flatten convolutional feature maps into a vector\n",
    "\n",
    "\n",
    "# Print output shape\n",
    "print(None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pooling and striding\n",
    "\n",
    "Almost all CNN architectures incorporate either pooling or striding. This is done for a number of reasons, including:\n",
    "- Dimensionality reduction: pooling and striding operations reduces computational complexity by shrinking the number of values passed to the next layer.\n",
    "For example, a 2x2 maxpool reduces the size of the feature maps by a factor of 4.\n",
    "- Translational invariance: Oftentimes in computer vision, we'd prefer that shifting the input by a few pixels doesn't change the output. Pooling and striding reduces sensitivity to exact pixel locations.\n",
    "- Increasing receptive field: by summarizing a window with a single value, subsequent convolutional kernels are seeing a wider swath of the original input image. For example, a max pool on some input followed by a 3x3 convolution results in a kernel \"seeing\" a 6x6 region instead of 3x3.\n",
    "\n",
    "#### Pooling\n",
    "The two most common forms of pooling are max pooling and average pooling. \n",
    "Both reduce values within a window to a single value, on a per-feature-map basis.\n",
    "Max pooling takes the maximum value of the window as the output value; average pooling takes the mean.\n",
    "\n",
    "<img src=\"Figures/maxpool.PNG\" alt=\"avg_vs_max\" style=\"width: 400px;\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Taking the output we've been working with so far, first print its current size\n",
    "print(\"Shape of conv2 feature maps before pooling: {0}\".format(None))\n",
    "\n",
    "# Max pool and then print new shape\n",
    "\n",
    "print(\"Shape of conv2 feature maps after max pooling: {0}\".format(None))\n",
    "\n",
    "# Average pool and then print new shape\n",
    "\n",
    "print(\"Shape of conv2 feature maps after avg pooling: {0}\".format(None))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can confirm that the two types of pooling behave how we'd expect:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Recreate values in pooling figure and make it 4D\n",
    "# feature_map_fig = torch.tensor(np.array([[1,1,2,4],\n",
    "#                                         [5,6,7,8],\n",
    "#                                         [3,2,1,0],\n",
    "#                                         [1,2,3,4]], dtype=np.float32))\n",
    "# fmap_fig = feature_map_fig.view(1,1,4,4)\n",
    "print(\"Feature map shape pre-pooling: {}\".format(None))\n",
    "\n",
    "# Maxpool\n",
    "\n",
    "print(\"\\nMax pool\")\n",
    "print(\"Shape: {}\".format(None))\n",
    "print(None)\n",
    "\n",
    "# Avgpool\n",
    "\n",
    "print(\"\\nAvg pool\")\n",
    "print(\"Shape: {}\".format(None))\n",
    "print(None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Striding\n",
    "One might expect that pixels in an image have high correlation with neighboring pixels, so we can save computation by skipping positions while sliding the convolutional kernel. \n",
    "By default, a CNN slides across the input one pixel at a time, which we call a stride of 1.\n",
    "By instead striding by 2, we skip calculating 75% of the values of the output feature map, which yields a feature map that's half the size in each spatial direction.\n",
    "Note, while pooling is an operation done after the convolution, striding is part of the convolution operation itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Since striding is part of the convolution operation, we'll start with the feature maps before the 2nd convolution\n",
    "print(\"Shape of conv1 feature maps: {0}\".format(None))\n",
    "\n",
    "# Apply 2nd convolutional layer, with striding of 2\n",
    "\n",
    "\n",
    "# Print output shape\n",
    "print(\"Shape of conv2 feature maps with stride of 2: {0}\".format(None))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building a custom CNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's revisit MNIST digit classification, but this time, we'll use the following CNN as our classifier: $5 \\times 5$ convolution -> $2 \\times 2$ max pool -> $5 \\times 5$ convolution -> $2 \\times 2$ max pool -> fully connected to $\\mathbb R^{256}$ -> fully connected to $\\mathbb R^{10}$ (prediction). \n",
    "ReLU activation functions will be used to impose non-linearities.\n",
    "Remember, convolutions produce 4-D outputs, and fully connected layers expect 2-D inputs, so tensors must be reshaped when transitioning from one to the other.\n",
    "\n",
    "We can build this CNN with the components introduced before, but as with the logistic regression example, it may prove helpful to instead organize our model with a `nn.Module`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how our `nn.Module` contains several operation chained together.\n",
    "The code for submodule initialization, which creates all the stateful parameters associated with each operation, is placed in the `__init__()` function, where it is run once during object instantiation.\n",
    "Meanwhile, the code describing the forward pass, which is used every time the model is run, is placed in the `forward()` method.\n",
    "Printing an instantiated model shows the model summary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can drop this model into our logistic regression training code, with few modifications beyond changing the model itself.\n",
    "A few other changes:\n",
    "- CNNs expect a 4-D input, so we no longer have to reshape the images before feeding them to our neural network.\n",
    "- Since CNNs are a little more complex than models we've worked with before, we're going to increase the number of epochs (complete passes through the training data) during training.\n",
    "- We switch from a vanilla stochastic gradient descent optimizer to the [Adam](https://arxiv.org/abs/1412.6980) optimizer, which tends to do well for neural networks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "###################################################################\n",
    "# Logistic Regression Model Code\n",
    "###################################################################\n",
    "\n",
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torchvision import datasets, transforms\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "class MNIST_Logistic_Regression(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.lin = nn.Linear(784, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.lin(x)\n",
    "\n",
    "# Load the data\n",
    "mnist_train = datasets.MNIST(root=\"./datasets\", train=True, transform=transforms.ToTensor(), download=True)\n",
    "mnist_test = datasets.MNIST(root=\"./datasets\", train=False, transform=transforms.ToTensor(), download=True)\n",
    "train_loader = torch.utils.data.DataLoader(mnist_train, batch_size=100, shuffle=True)\n",
    "test_loader = torch.utils.data.DataLoader(mnist_test, batch_size=100, shuffle=False)\n",
    "\n",
    "## Training\n",
    "# Instantiate model\n",
    "model = MNIST_Logistic_Regression()\n",
    "\n",
    "# Loss and Optimizer\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=0.1)\n",
    "\n",
    "# Iterate through train set minibatchs \n",
    "for images, labels in tqdm(train_loader):\n",
    "    # Zero out the gradients\n",
    "    optimizer.zero_grad()\n",
    "    \n",
    "    # Forward pass\n",
    "    x = images.view(-1, 28*28)\n",
    "    y = model(x)\n",
    "    loss = criterion(y, labels)\n",
    "    # Backward pass\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "## Testing\n",
    "correct = 0\n",
    "total = len(mnist_test)\n",
    "\n",
    "with torch.no_grad():\n",
    "    # Iterate through test set minibatchs \n",
    "    for images, labels in tqdm(test_loader):\n",
    "        # Forward pass\n",
    "        x = images.view(-1, 28*28)\n",
    "        y = model(x)\n",
    "        \n",
    "        predictions = torch.argmax(y, dim=1)\n",
    "        correct += torch.sum((predictions == labels).float())\n",
    "    \n",
    "print('Test accuracy: {}'.format(correct/total))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are running this notebook on CPU, training this CNN might take a while.\n",
    "On the other hand, if you use a GPU, this model should train in seconds.\n",
    "This is why we usually prefer to use GPUs when we have them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference on single images\n",
    "\n",
    "The accuracy of our CNN looks great, but let's dive a little deeper and look at what the prediction probabilities are for individual images.\n",
    "To do so, we're going to draw a single random image from our test set, feed it through our classifier, and observe the vector produced after the softmax.\n",
    "This is closer to how the model would behave when deployed.\n",
    "\n",
    "### Saving and loading models\n",
    "If you're training the above CNN on your personal machine, you're probably using your CPU only, in which case it might take a while to train (~15 minutes).\n",
    "That's not too bad, but for the sake of this demo, I'm going to load a model I trained beforehand.\n",
    "I then saved the model parameters using PyTorch's save functionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To load the model, I create a new model instance and then load my saved (and trained) parameters into it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just to check that the load was successful, I can re-run evaluation with the loaded model on the test set, which is much faster than training from scratch.\n",
    "The accuracy of the loaded model is the same as when the model was first trained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inference demo\n",
    "\n",
    "Now that we have a trained model, we can do our single image inference demo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Load the test data\n",
    "# mnist_test = datasets.MNIST(root=\"./datasets\", train=False, transform=transforms.ToTensor(), download=True)\n",
    "\n",
    "# # Grab a random image\n",
    "# random_image_index = np.random.randint(low=0,high=len(mnist_test))\n",
    "# random_image = mnist_test[random_image_index][0]\n",
    "# batched_random_image = random_image.unsqueeze(0)\n",
    "\n",
    "# # Model prediction\n",
    "# with torch.no_grad():\n",
    "#     y = loaded_model(batched_random_image)\n",
    "#     probabilities = F.softmax(y, dim=1)\n",
    "\n",
    "# # Show image\n",
    "# image_figure = plt.figure()\n",
    "# plt.imshow(random_image.reshape([28,28]), cmap='gray')\n",
    "# plt.title(mnist_test[random_image_index][1])\n",
    "\n",
    "# # Show predictions probabilities\n",
    "# bar_figure = plt.figure()\n",
    "# plt.bar(np.arange(0,10),probabilities[0,:]*100,bottom=0.1,log=True)\n",
    "# plt.yticks((0.1,1,10,100),('0.1','1','10','100'))\n",
    "# plt.xticks(np.arange(0,10))\n",
    "# plt.xlabel('Digit')\n",
    "# plt.ylabel('Probability(Digit) [Percent]')\n",
    "# plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Torchvision"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Datasets and transforms\n",
    "\n",
    "As any experienced ML practioner will say, data wrangling is often half (sometimes even 90%) of the battle when building a model.\n",
    "Often, we have to write significant code to handle downloading, organizing, formatting, shuffling, pre-processing, augmenting, and batching examples. \n",
    "For popular datasets, we'd like to standardize data handling so that the comparisons we make are specific to the models themselves.\n",
    "\n",
    "Enter [Torchvision](https://pytorch.org/docs/stable/torchvision/index.html).\n",
    "Torchvision includes easy-to-use APIs for downloading and loading many popular vision datasets.\n",
    "We've previously seen this in action for downloading the MNIST dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, there's [many more](https://pytorch.org/docs/stable/torchvision/datasets.html).\n",
    "Currently, datasets for image classification (e.g. MNIST, CIFAR, ImageNet), object detection (VOC, COCO, Cityscapes), and video action recognition (UCF101, Kinetics) are included.\n",
    "\n",
    "For formatting, pre-processing, and augmenting, [transforms](https://pytorch.org/docs/stable/torchvision/transforms.html) can come in handy.\n",
    "Again, we've seen this before (see above), when we used a transform to convert the MNIST data from PIL images to PyTorch tensors.\n",
    "However, transforms can be used for much more. \n",
    "Preprocessing steps like data whitening are common before feeding the data into the model.\n",
    "Also, in many cases, we use data augmentations to artificially inflate our dataset and learn invariances.\n",
    "Transforms are a versatile tool for all of these."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Leveraging popular convolutional neural networks\n",
    "\n",
    "While you certainly can build your own custom CNNs like we did above, more often than not, it's better to use one of the popular existing architectures. \n",
    "The Torchvision documentation has a [list of supported CNNs](https://pytorch.org/docs/stable/torchvision/models.html), as well as some performance characteristics. \n",
    "There's a number of reasons for using one of these CNNs instead of designing your own.\n",
    "\n",
    "First, for image datasets larger and more complex than MNIST (which is basically all of them), a fair amount network depth and width is often necessary.\n",
    "For example, some of the popular CNNs can be over 100 layers deep, with several tricks and details beyond what we've covered in this notebook.\n",
    "Coding all of this yourself has a high potential for error, especially when you're first getting started.\n",
    "Instead, you can create the CNN architecture using Torchvision, using a couple lines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading a working CNN architecture in a couple lines can save a significant amount of time both implementing and debugging.\n",
    "\n",
    "The second, perhaps even more important, reason to use one of these existing architectures is the ability to use pre-trained weights.\n",
    "Early on in the recent resurgence of deep learning, people discovered that the weights of a CNN trained for ImageNet classification were highly transferable. \n",
    "For example, it is common to use the weights of an ImageNet-trained CNN as a weight initialization for other vision tasks, or even to freeze the bulk of the weights and only re-train the final classification layer(s) on a new task.\n",
    "This is significant, as in most settings, we rarely have enough labeled data to train a powerful CNN from scratch without overfitting.\n",
    "Loading pre-trained CNN is also pretty simple, involving an additional argument to the previous cell block:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"1\">*Note that this line of code is commented out, as running it will initiate a download of the pre-trained weights, which is a fairly large file.*</font>\n",
    "\n",
    "A full tutorial on using pre-trained CNNs is a little beyond the scope of this notebook.\n",
    "See [this tutorial](https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html) for an example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Other computer vision tasks\n",
    "The base CNN architectures were often designed for image classification, but the same CNNs are often used as the backbone of most modern computer vision models.\n",
    "These other models often take this base CNN and include additional networks or make other architecture changes to adapt them to other tasks, such as object detection.\n",
    "Torchvision contains a few models (and pre-trained weights) for object detection, segmentation, and video action recognition.\n",
    "For example, to load a [Faster R-CNN](https://arxiv.org/abs/1506.01497) with a [ResNet50](https://arxiv.org/abs/1512.03385) convolutional feature extractor with [Feature Pyramid Networks](https://arxiv.org/abs/1612.03144) pre-trained on [MS COCO](http://cocodataset.org/#home):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"1\">*Again, this line has been commented out to prevent loading a large network for this demo.*</font>\n",
    "\n",
    "Torchvision's selection of non-classification models is relatively light, and not particularly flexible.\n",
    "A number of other libraries are available, depending on the task.\n",
    "For example, for object detection and segmentation, Facebook AI Research's [Detectron2](https://github.com/facebookresearch/detectron2) is highly recommend."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "03c821f7cc304647bd7dd129ece95022": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_a9254a18982444839701b4f3c20b1113",
       "max": 600,
       "style": "IPY_MODEL_feba469114dd47aaa7a13bc21b134099",
       "value": 600
      }
     },
     "06865ecae49c4d328421c474fea799a0": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "09e99596521e40948b044aea1dd49ceb": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "0a271956e4984a0a9ba69c1f66809bfb": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_6ca5eb76ff8441cc996cdbc8bcfb5ab3",
       "style": "IPY_MODEL_eb483872c21d43d4a42b34091b4eb8c1",
       "value": " 600/600 [05:31&lt;00:00,  1.81it/s]"
      }
     },
     "0edf80c192134246a102f49c1ac4a77e": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "13e3067366c34bfdbd817690ebeb5685": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "15ada75a5a524e0d80956489b169679b": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "1d856c2263ec4b1f97f46727b8d85f12": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "1e7d4fd24b054b3daa60ae7dd2726ec5": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_6933964202a74264b0ed487f706297a3",
        "IPY_MODEL_b8590f98675a453ba38b85163a285eb4"
       ],
       "layout": "IPY_MODEL_ba35f1bbfe5549f394fcb2a1b544584b"
      }
     },
     "234c9d9fbbaa4aaa84dfd3598983e96a": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "26e0d143239647a2832a3d656f56fd12": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "2779485b2912403f80088f678d5a01c0": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "28a7ac6f810e4b4a94503dbda7ebf78c": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "2d0476b37d43454585abb625c1c50181": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "2da8ddac80d74ca3ab5fd5efcefbd0b0": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_c047c4ef8c3c4f1fa8508267806dc5c2",
        "IPY_MODEL_35e061d7f6aa4817a081a3ce2416bc7d"
       ],
       "layout": "IPY_MODEL_62ef7f1b4c0f4a008c366a21cec33c17"
      }
     },
     "2e67a47c98ae4b19a913886efd194566": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_bb81062e385c414abc8c3f8a70716327",
       "max": 600,
       "style": "IPY_MODEL_731297e272154e4ab71e7fd6a6859309",
       "value": 600
      }
     },
     "3152dae7252a4ee3958217f48a9bb0c7": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "322ecba10f114730888b6b15a8694fea": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "34484a12bd9f494a9bb6c6e479524246": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "35e061d7f6aa4817a081a3ce2416bc7d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_5dfd065b0af94b72b0a797c1659d58e4",
       "style": "IPY_MODEL_13e3067366c34bfdbd817690ebeb5685",
       "value": " 600/600 [04:52&lt;00:00,  2.05it/s]"
      }
     },
     "371bf011ef8c488fac1fae1c63dc7dae": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_6f06679d39f84b75a08a05e478e0bc89",
       "style": "IPY_MODEL_cadc3c445e7444f5af275f20c674019b",
       "value": " 100/100 [00:18&lt;00:00,  5.43it/s]"
      }
     },
     "40143d1b96fa4cbdb4ea4f915ac0e7c7": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_34484a12bd9f494a9bb6c6e479524246",
       "style": "IPY_MODEL_d274246f37a643ba879a390b788b3c5b",
       "value": " 100/100 [00:18&lt;00:00,  5.32it/s]"
      }
     },
     "456c3368617d4acd9b81809d2643f847": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "526b7dd6def34ac68dfd667d9ecc7373": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "55697ad8b13f40eb900098274954edff": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_15ada75a5a524e0d80956489b169679b",
       "style": "IPY_MODEL_783576334fac49bbb171283f192725f2",
       "value": " 3/3 [28:40&lt;00:00, 573.52s/it]"
      }
     },
     "5c35d2a0736845aca0c743382fb21435": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_968de7cc30cc49219edb47c77ac19835",
       "style": "IPY_MODEL_26e0d143239647a2832a3d656f56fd12",
       "value": " 600/600 [06:08&lt;00:00,  1.63it/s]"
      }
     },
     "5c778265dd2e4f55990114a410be07e7": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_1d856c2263ec4b1f97f46727b8d85f12",
       "style": "IPY_MODEL_3152dae7252a4ee3958217f48a9bb0c7",
       "value": 100
      }
     },
     "5dfd065b0af94b72b0a797c1659d58e4": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "5dfea9beebf347babc96499b6ef6e55b": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "5fa779faf29448bda9ecb19f04db2363": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "62ef7f1b4c0f4a008c366a21cec33c17": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "68181c92926e4b209ebddc72e0a18a90": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "6933964202a74264b0ed487f706297a3": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "description": "  0%",
       "layout": "IPY_MODEL_88946e6e9e3d45c2a58c6712f7b78fef",
       "max": 3,
       "style": "IPY_MODEL_28a7ac6f810e4b4a94503dbda7ebf78c"
      }
     },
     "6ca5eb76ff8441cc996cdbc8bcfb5ab3": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "6ebc572251db40378e419fa603857e39": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "6f06679d39f84b75a08a05e478e0bc89": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "6faf5a48daed4ceb8b42b918f5890080": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "70bfccb7e1144fc3a04cd0ca72ae63eb": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_f67d8b42eb0b4156929a609efc6e9708",
        "IPY_MODEL_55697ad8b13f40eb900098274954edff"
       ],
       "layout": "IPY_MODEL_6faf5a48daed4ceb8b42b918f5890080"
      }
     },
     "731297e272154e4ab71e7fd6a6859309": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "783576334fac49bbb171283f192725f2": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "78f433d8b75e4a08a36465e145079cf1": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "7adead9aa0e54151b5bae6180e87eaf2": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "7cd916a53b1a42caa0d8fab7855e43bc": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_5c778265dd2e4f55990114a410be07e7",
        "IPY_MODEL_371bf011ef8c488fac1fae1c63dc7dae"
       ],
       "layout": "IPY_MODEL_d1536ee4c2b94f5d819c52777db82fd1"
      }
     },
     "7d6bdfc513274cbc997b980fd0bf6d4b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_b4d97b0af743462fa7ab900547f28cec",
       "max": 3,
       "style": "IPY_MODEL_b2374c1841914c56a1c0a0ce0fdd8f52",
       "value": 3
      }
     },
     "7e775f18ba894cad88b398c06546947d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "7f1e0c9662724b0a94f6412f23d4f48b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_526b7dd6def34ac68dfd667d9ecc7373",
       "style": "IPY_MODEL_7adead9aa0e54151b5bae6180e87eaf2",
       "value": 100
      }
     },
     "806fdae35ae54e4398bc198978f1143a": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_06865ecae49c4d328421c474fea799a0",
       "style": "IPY_MODEL_6ebc572251db40378e419fa603857e39",
       "value": " 600/600 [17:51&lt;00:00,  1.79s/it]"
      }
     },
     "8334eba58bc34ceeb9c15edb42401d3f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_2e67a47c98ae4b19a913886efd194566",
        "IPY_MODEL_9301e0239a174dfc9b221ec2a02af44f"
       ],
       "layout": "IPY_MODEL_5dfea9beebf347babc96499b6ef6e55b"
      }
     },
     "88946e6e9e3d45c2a58c6712f7b78fef": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "88e7221416214664b30773fed9a88614": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "8cecc0e1039c4eee90dfa2db1afab3c1": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "90722b21a44e4541a5333511f9e1e3c8": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "9301e0239a174dfc9b221ec2a02af44f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_d9151e052ab545159f94594f8d155fd7",
       "style": "IPY_MODEL_e45c675d8d044d7c9550a20e40b0539e",
       "value": " 600/600 [05:17&lt;00:00,  1.89it/s]"
      }
     },
     "968de7cc30cc49219edb47c77ac19835": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "96c234a72edc42ca80e536e3a54c2acc": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "a229f6b4ca20475ea3a5745d18d7131c": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_322ecba10f114730888b6b15a8694fea",
       "style": "IPY_MODEL_a8a4eb83499a453787b42fc46b93a0ba",
       "value": " 223/600 [01:52&lt;03:02,  2.06it/s]"
      }
     },
     "a83b638605be4a9ba18c4529729de544": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "a8a4eb83499a453787b42fc46b93a0ba": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "a9254a18982444839701b4f3c20b1113": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "abc221956ffb4578aec132dddb6ef921": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "acb0ac60dd0643c6b1ae204792f442d4": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "acc0927c8ae7415f9df9b49feeadfded": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "description": " 37%",
       "layout": "IPY_MODEL_2d0476b37d43454585abb625c1c50181",
       "max": 600,
       "style": "IPY_MODEL_88e7221416214664b30773fed9a88614",
       "value": 223
      }
     },
     "aff6325150904f36b4dae74956e87bff": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_03c821f7cc304647bd7dd129ece95022",
        "IPY_MODEL_806fdae35ae54e4398bc198978f1143a"
       ],
       "layout": "IPY_MODEL_78f433d8b75e4a08a36465e145079cf1"
      }
     },
     "b18a79c6165442b7ac686c25eb915836": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "b1ce822ea1f44e69b04dfabf5d6b176b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_7f1e0c9662724b0a94f6412f23d4f48b",
        "IPY_MODEL_40143d1b96fa4cbdb4ea4f915ac0e7c7"
       ],
       "layout": "IPY_MODEL_8cecc0e1039c4eee90dfa2db1afab3c1"
      }
     },
     "b2374c1841914c56a1c0a0ce0fdd8f52": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "b4d97b0af743462fa7ab900547f28cec": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "b55d6f340ec0428eac1c4ba7262fda73": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_7d6bdfc513274cbc997b980fd0bf6d4b",
        "IPY_MODEL_c4a75cbcd5784be48fdfb09dbacbd3a3"
       ],
       "layout": "IPY_MODEL_acb0ac60dd0643c6b1ae204792f442d4"
      }
     },
     "b7e465f909304b8da8f19b67743c925f": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "b8590f98675a453ba38b85163a285eb4": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_decefda30d7f4c59a15bfd79f082045b",
       "style": "IPY_MODEL_cf367f49b29745fe88c33e2b2f46f8e4",
       "value": " 0/3 [00:00&lt;?, ?it/s]"
      }
     },
     "b9c879ca534d4691a94bd015b53d7967": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "ba35f1bbfe5549f394fcb2a1b544584b": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "bb81062e385c414abc8c3f8a70716327": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "c047c4ef8c3c4f1fa8508267806dc5c2": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_456c3368617d4acd9b81809d2643f847",
       "max": 600,
       "style": "IPY_MODEL_dccbf0d594854417975e06c2251f4f92",
       "value": 600
      }
     },
     "c18eea10e4ff4974b07ff2253f5f779d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_09e99596521e40948b044aea1dd49ceb",
       "max": 600,
       "style": "IPY_MODEL_d6c628ea9a2347579e8b04a34dae7e83",
       "value": 600
      }
     },
     "c4a75cbcd5784be48fdfb09dbacbd3a3": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_90722b21a44e4541a5333511f9e1e3c8",
       "style": "IPY_MODEL_b18a79c6165442b7ac686c25eb915836",
       "value": " 3/3 [16:33&lt;00:00, 331.17s/it]"
      }
     },
     "c6a40c98b57d4d99ad89de894e76aafb": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_a83b638605be4a9ba18c4529729de544",
       "max": 600,
       "style": "IPY_MODEL_abc221956ffb4578aec132dddb6ef921",
       "value": 600
      }
     },
     "cadc3c445e7444f5af275f20c674019b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "cf367f49b29745fe88c33e2b2f46f8e4": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "d0160fcf3a1c4409aa417bc54dcd2029": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_f9ae08e19e6540a2ab502af5ff456067",
        "IPY_MODEL_fe492931e45e4f9995c1302483fde1ba"
       ],
       "layout": "IPY_MODEL_b9c879ca534d4691a94bd015b53d7967"
      }
     },
     "d1536ee4c2b94f5d819c52777db82fd1": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "d274246f37a643ba879a390b788b3c5b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "d34d970aa5e84bd8b30f3c0af6851044": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_acc0927c8ae7415f9df9b49feeadfded",
        "IPY_MODEL_a229f6b4ca20475ea3a5745d18d7131c"
       ],
       "layout": "IPY_MODEL_68181c92926e4b209ebddc72e0a18a90"
      }
     },
     "d6c628ea9a2347579e8b04a34dae7e83": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "d9151e052ab545159f94594f8d155fd7": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "dccbf0d594854417975e06c2251f4f92": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "de12b4d028d240d88666934d195132db": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "decefda30d7f4c59a15bfd79f082045b": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "e45c675d8d044d7c9550a20e40b0539e": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "e59571b38ab04469b7e1af29747dd20d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_c6a40c98b57d4d99ad89de894e76aafb",
        "IPY_MODEL_0a271956e4984a0a9ba69c1f66809bfb"
       ],
       "layout": "IPY_MODEL_0edf80c192134246a102f49c1ac4a77e"
      }
     },
     "eb483872c21d43d4a42b34091b4eb8c1": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "f67d8b42eb0b4156929a609efc6e9708": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_de12b4d028d240d88666934d195132db",
       "max": 3,
       "style": "IPY_MODEL_2779485b2912403f80088f678d5a01c0",
       "value": 3
      }
     },
     "f9ae08e19e6540a2ab502af5ff456067": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_234c9d9fbbaa4aaa84dfd3598983e96a",
       "max": 600,
       "style": "IPY_MODEL_7e775f18ba894cad88b398c06546947d",
       "value": 600
      }
     },
     "fce41ea4b3b74dc2ab55a27f6578cbbf": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_c18eea10e4ff4974b07ff2253f5f779d",
        "IPY_MODEL_5c35d2a0736845aca0c743382fb21435"
       ],
       "layout": "IPY_MODEL_b7e465f909304b8da8f19b67743c925f"
      }
     },
     "fe492931e45e4f9995c1302483fde1ba": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_96c234a72edc42ca80e536e3a54c2acc",
       "style": "IPY_MODEL_5fa779faf29448bda9ecb19f04db2363",
       "value": " 600/600 [05:32&lt;00:00,  1.81it/s]"
      }
     },
     "feba469114dd47aaa7a13bc21b134099": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}