TransferLearning.ipynb

{
 "nbformat_minor": 0, 
 "nbformat": 4, 
 "cells": [
  {
   "source": [
    "# Training a ConvNet PyTorch\n", 
    "\n", 
    "In this notebook, you'll learn how to use the powerful PyTorch framework to specify a conv net architecture and train it on the CIFAR-10 dataset."
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "execution_count": null, 
   "cell_type": "code", 
   "source": [
    "import torch\n", 
    "import torch.nn as nn\n", 
    "import torch.optim as optim\n", 
    "from torch.autograd import Variable\n", 
    "from torch.utils.data import DataLoader\n", 
    "from torch.utils.data import sampler\n", 
    "\n", 
    "import torchvision.datasets as dset\n", 
    "import torchvision.transforms as T\n", 
    "\n", 
    "import numpy as np\n", 
    "\n", 
    "import timeit"
   ], 
   "outputs": [], 
   "metadata": {
    "collapsed": false
   }
  }, 
  {
   "source": [
    "## What's this PyTorch business?\n", 
    "\n", 
    "You've written a lot of code in this assignment to provide a whole host of neural network functionality. Dropout, Batch Norm, and 2D convolutions are some of the workhorses of deep learning in computer vision. You've also worked hard to make your code efficient and vectorized.\n", 
    "\n", 
    "For the last part of this assignment, though, we're going to leave behind your beautiful codebase and instead migrate to one of two popular deep learning frameworks: in this instance, PyTorch (or TensorFlow, if you switch over to that notebook). \n", 
    "\n", 
    "Why?\n", 
    "\n", 
    "* Our code will now run on GPUs! Much faster training. When using a framework like PyTorch or TensorFlow you can harness the power of the GPU for your own custom neural network architectures without having to write CUDA code directly (which is beyond the scope of this class).\n", 
    "* We want you to be ready to use one of these frameworks for your project so you can experiment more efficiently than if you were writing every feature you want to use by hand. \n", 
    "* We want you to stand on the shoulders of giants! TensorFlow and PyTorch are both excellent frameworks that will make your lives a lot easier, and now that you understand their guts, you are free to use them :) \n", 
    "* We want you to be exposed to the sort of deep learning code you might run into in academia or industry. "
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "source": [
    "## How will I learn PyTorch?\n", 
    "\n", 
    "If you've used Torch before, but are new to PyTorch, this tutorial might be of use: http://pytorch.org/tutorials/beginner/former_torchies_tutorial.html\n", 
    "\n", 
    "Otherwise, this notebook will walk you through much of what you need to do to train models in Torch. See the end of the notebook for some links to helpful tutorials if you want to learn more or need further clarification on topics that aren't fully explained here."
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "source": [
    "## Load Datasets\n", 
    "\n", 
    "We load the CIFAR-10 dataset. This might take a couple minutes the first time you do it, but the files should stay cached after that."
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "execution_count": null, 
   "cell_type": "code", 
   "source": [
    "class ChunkSampler(sampler.Sampler):\n", 
    "    \"\"\"Samples elements sequentially from some offset. \n", 
    "    Arguments:\n", 
    "        num_samples: # of desired datapoints\n", 
    "        start: offset where we should start selecting from\n", 
    "    \"\"\"\n", 
    "    def __init__(self, num_samples, start = 0):\n", 
    "        self.num_samples = num_samples\n", 
    "        self.start = start\n", 
    "\n", 
    "    def __iter__(self):\n", 
    "        return iter(range(self.start, self.start + self.num_samples))\n", 
    "\n", 
    "    def __len__(self):\n", 
    "        return self.num_samples\n", 
    "\n", 
    "NUM_TRAIN = 49000\n", 
    "NUM_VAL = 1000\n", 
    "\n", 
    "cifar10_train = dset.CIFAR10('./cs231n/datasets', train=True, download=True,\n", 
    "                           transform=T.ToTensor())\n", 
    "loader_train = DataLoader(cifar10_train, batch_size=64, sampler=ChunkSampler(NUM_TRAIN, 0))\n", 
    "\n", 
    "cifar10_val = dset.CIFAR10('./cs231n/datasets', train=True, download=True,\n", 
    "                           transform=T.ToTensor())\n", 
    "loader_val = DataLoader(cifar10_val, batch_size=64, sampler=ChunkSampler(NUM_VAL, NUM_TRAIN))\n", 
    "\n", 
    "cifar10_test = dset.CIFAR10('./cs231n/datasets', train=False, download=True,\n", 
    "                          transform=T.ToTensor())\n", 
    "loader_test = DataLoader(cifar10_test, batch_size=64)\n"
   ], 
   "outputs": [], 
   "metadata": {
    "collapsed": false
   }
  }, 
  {
   "source": [
    "For now, we're going to use a CPU-friendly datatype. Later, we'll switch to a datatype that will move all our computations to the GPU and measure the speedup."
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "execution_count": null, 
   "cell_type": "code", 
   "source": [
    "dtype = torch.FloatTensor # the CPU datatype\n", 
    "\n", 
    "# Constant to control how frequently we print train loss\n", 
    "print_every = 100\n", 
    "\n", 
    "# This is a little utility that we'll use to reset the model\n", 
    "# if we want to re-initialize all our parameters\n", 
    "def reset(m):\n", 
    "    if hasattr(m, 'reset_parameters'):\n", 
    "        m.reset_parameters()"
   ], 
   "outputs": [], 
   "metadata": {
    "collapsed": true
   }
  }, 
  {
   "source": [
    "## Example Model\n", 
    "\n", 
    "### Some assorted tidbits\n", 
    "\n", 
    "Let's start by looking at a simple model. First, note that PyTorch operates on Tensors, which are n-dimensional arrays functionally analogous to numpy's ndarrays, with the additional feature that they can be used for computations on GPUs.\n", 
    "\n", 
    "We'll provide you with a Flatten function, which we explain here. Remember that our image data (and more relevantly, our intermediate feature maps) are initially N x C x H x W, where:\n", 
    "* N is the number of datapoints\n", 
    "* C is the number of channels\n", 
    "* H is the height of the intermediate feature map in pixels\n", 
    "* W is the height of the intermediate feature map in pixels\n", 
    "\n", 
    "This is the right way to represent the data when we are doing something like a 2D convolution, that needs spatial understanding of where the intermediate features are relative to each other. When we input  data into fully connected affine layers, however, we want each datapoint to be represented by a single vector -- it's no longer useful to segregate the different channels, rows, and columns of the data. So, we use a \"Flatten\" operation to collapse the C x H x W values per representation into a single long vector. The Flatten function below first reads in the N, C, H, and W values from a given batch of data, and then returns a \"view\" of that data. \"View\" is analogous to numpy's \"reshape\" method: it reshapes x's dimensions to be N x ??, where ?? is allowed to be anything (in this case, it will be C x H x W, but we don't need to specify that explicitly). "
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "execution_count": null, 
   "cell_type": "code", 
   "source": [
    "class Flatten(nn.Module):\n", 
    "    def forward(self, x):\n", 
    "        N, C, H, W = x.size() # read in N, C, H, W\n", 
    "        return x.view(N, -1)  # \"flatten\" the C * H * W values into a single vector per image"
   ], 
   "outputs": [], 
   "metadata": {
    "collapsed": true
   }
  }, 
  {
   "source": [
    "### The example model itself\n", 
    "\n", 
    "The first step to training your own model is defining its architecture.\n", 
    "\n", 
    "Here's an example of a convolutional neural network defined in PyTorch -- try to understand what each line is doing, remembering that each layer is composed upon the previous layer. We haven't trained anything yet - that'll come next - for now, we want you to understand how everything gets set up.  nn.Sequential is a container which applies each layer\n", 
    "one after the other.\n", 
    "\n", 
    "In that example, you see 2D convolutional layers (Conv2d), ReLU activations, and fully-connected layers (Linear). You also see the Cross-Entropy loss function, and the Adam optimizer being used. \n", 
    "\n", 
    "Make sure you understand why the parameters of the Linear layer are 5408 and 10.\n"
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "execution_count": null, 
   "cell_type": "code", 
   "source": [
    "# Here's where we define the architecture of the model... \n", 
    "simple_model = nn.Sequential(\n", 
    "                nn.Conv2d(3, 32, kernel_size=7, stride=2),\n", 
    "                nn.ReLU(inplace=True),\n", 
    "                Flatten(), # see above for explanation\n", 
    "                nn.Linear(5408, 10), # affine layer\n", 
    "              )\n", 
    "\n", 
    "# Set the type of all data in this model to be FloatTensor \n", 
    "simple_model.type(dtype)\n", 
    "\n", 
    "loss_fn = nn.CrossEntropyLoss().type(dtype)\n", 
    "optimizer = optim.Adam(simple_model.parameters(), lr=1e-2) # lr sets the learning rate of the optimizer"
   ], 
   "outputs": [], 
   "metadata": {
    "collapsed": false
   }
  }, 
  {
   "source": [
    "PyTorch supports many other layer types, loss functions, and optimizers - you will experiment with these next. Here's the official API documentation for these (if any of the parameters used above were unclear, this resource will also be helpful). One note: what we call in the class \"spatial batch norm\" is called \"BatchNorm2D\" in PyTorch.\n", 
    "\n", 
    "* Layers: http://pytorch.org/docs/nn.html\n", 
    "* Activations: http://pytorch.org/docs/nn.html#non-linear-activations\n", 
    "* Loss functions: http://pytorch.org/docs/nn.html#loss-functions\n", 
    "* Optimizers: http://pytorch.org/docs/optim.html#algorithms"
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "source": [
    "## Training a specific model\n", 
    "\n", 
    "In this section, we're going to specify a model for you to construct. The goal here isn't to get good performance (that'll be next), but instead to get comfortable with understanding the PyTorch documentation and configuring your own model. \n", 
    "\n", 
    "Using the code provided above as guidance, and using the following PyTorch documentation, specify a model with the following architecture:\n", 
    "\n", 
    "* 7x7 Convolutional Layer with 32 filters and stride of 1\n", 
    "* ReLU Activation Layer\n", 
    "* Spatial Batch Normalization Layer\n", 
    "* 2x2 Max Pooling layer with a stride of 2\n", 
    "* Affine layer with 1024 output units\n", 
    "* ReLU Activation Layer\n", 
    "* Affine layer from 1024 input units to 10 outputs\n", 
    "\n", 
    "And finally, set up a **cross-entropy** loss function and the **RMSprop** learning rule."
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "execution_count": null, 
   "cell_type": "code", 
   "source": [
    "fixed_model_base = nn.Sequential( # You fill this in!\n", 
    "            )\n", 
    "\n", 
    "fixed_model = fixed_model_base.type(dtype)"
   ], 
   "outputs": [], 
   "metadata": {
    "collapsed": false
   }
  }, 
  {
   "source": [
    "To make sure you're doing the right thing, use the following tool to check the dimensionality of your output (it should be 64 x 10, since our batches have size 64 and the output of the final affine layer should be 10, corresponding to our 10 classes):"
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "execution_count": null, 
   "cell_type": "code", 
   "source": [
    "## Now we're going to feed a random batch into the model you defined and make sure the output is the right size\n", 
    "x = torch.randn(64, 3, 32, 32).type(dtype)\n", 
    "x_var = Variable(x.type(dtype)) # Construct a PyTorch Variable out of your input data\n", 
    "ans = fixed_model(x_var)        # Feed it through the model! \n", 
    "\n", 
    "# Check to make sure what comes out of your model\n", 
    "# is the right dimensionality... this should be True\n", 
    "# if you've done everything correctly\n", 
    "np.array_equal(np.array(ans.size()), np.array([64, 10]))       "
   ], 
   "outputs": [], 
   "metadata": {
    "collapsed": false
   }
  }, 
  {
   "source": [
    "### GPU!\n", 
    "\n", 
    "Now, we're going to switch the dtype of the model and our data to the GPU-friendly tensors, and see what happens... everything is the same, except we are casting our model and input tensors as this new dtype instead of the old one.\n", 
    "\n", 
    "If this returns false, or otherwise fails in a not-graceful way (i.e., with some error message), you may not have an NVIDIA GPU available on your machine. If you're running locally, we recommend you switch to Google Cloud and follow the instructions to set up a GPU there. If you're already on Google Cloud, something is wrong -- make sure you followed the instructions on how to request and use a GPU on your instance. If you did, post on Piazza or come to Office Hours so we can help you debug."
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "execution_count": null, 
   "cell_type": "code", 
   "source": [
    "# Verify that CUDA is properly configured and you have a GPU available\n", 
    "\n", 
    "torch.cuda.is_available()"
   ], 
   "outputs": [], 
   "metadata": {
    "collapsed": false
   }
  }, 
  {
   "execution_count": null, 
   "cell_type": "code", 
   "source": [
    "import copy\n", 
    "gpu_dtype = torch.cuda.FloatTensor\n", 
    "\n", 
    "fixed_model_gpu = copy.deepcopy(fixed_model_base).type(gpu_dtype)\n", 
    "\n", 
    "x_gpu = torch.randn(64, 3, 32, 32).type(gpu_dtype)\n", 
    "x_var_gpu = Variable(x.type(gpu_dtype)) # Construct a PyTorch Variable out of your input data\n", 
    "ans = fixed_model_gpu(x_var_gpu)        # Feed it through the model! \n", 
    "\n", 
    "# Check to make sure what comes out of your model\n", 
    "# is the right dimensionality... this should be True\n", 
    "# if you've done everything correctly\n", 
    "np.array_equal(np.array(ans.size()), np.array([64, 10]))"
   ], 
   "outputs": [], 
   "metadata": {
    "collapsed": false
   }
  }, 
  {
   "source": [
    "Run the following cell to evaluate the performance of the forward pass running on the CPU:"
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "execution_count": null, 
   "cell_type": "code", 
   "source": [
    "%%timeit \n", 
    "ans = fixed_model(x_var)"
   ], 
   "outputs": [], 
   "metadata": {
    "collapsed": false
   }
  }, 
  {
   "source": [
    "... and now the GPU:"
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "execution_count": null, 
   "cell_type": "code", 
   "source": [
    "%%timeit \n", 
    "torch.cuda.synchronize() # Make sure there are no pending GPU computations\n", 
    "ans = fixed_model_gpu(x_var_gpu)        # Feed it through the model! \n", 
    "torch.cuda.synchronize() # Make sure there are no pending GPU computations"
   ], 
   "outputs": [], 
   "metadata": {
    "collapsed": false
   }
  }, 
  {
   "source": [
    "You should observe that even a simple forward pass like this is significantly faster on the GPU. So for the rest of the assignment (and when you go train your models in assignment 3 and your project!), you should use the GPU datatype for your model and your tensors: as a reminder that is *torch.cuda.FloatTensor* (in our notebook here as *gpu_dtype*)"
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "source": [
    "### Train the model.\n", 
    "\n", 
    "Now that you've seen how to define a model and do a single forward pass of some data through it, let's  walk through how you'd actually train one whole epoch over your training data (using the simple_model we provided above).\n", 
    "\n", 
    "Make sure you understand how each PyTorch function used below corresponds to what you implemented in your custom neural network implementation.\n", 
    "\n", 
    "Note that because we are not resetting the weights anywhere below, if you run the cell multiple times, you are effectively training multiple epochs (so your performance should improve).\n", 
    "\n", 
    "First, set up an RMSprop optimizer (using a 1e-3 learning rate) and a cross-entropy loss function:"
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "execution_count": null, 
   "cell_type": "code", 
   "source": [
    "loss_fn = None\n", 
    "optimizer = None\n", 
    "pass\n"
   ], 
   "outputs": [], 
   "metadata": {
    "collapsed": true
   }
  }, 
  {
   "execution_count": null, 
   "cell_type": "code", 
   "source": [
    "# This sets the model in \"training\" mode. This is relevant for some layers that may have different behavior\n", 
    "# in training mode vs testing mode, such as Dropout and BatchNorm. \n", 
    "fixed_model_gpu.train()\n", 
    "\n", 
    "# Load one batch at a time.\n", 
    "for t, (x, y) in enumerate(loader_train):\n", 
    "    x_var = Variable(x.type(gpu_dtype))\n", 
    "    y_var = Variable(y.type(gpu_dtype).long())\n", 
    "\n", 
    "    # This is the forward pass: predict the scores for each class, for each x in the batch.\n", 
    "    scores = fixed_model_gpu(x_var)\n", 
    "    \n", 
    "    # Use the correct y values and the predicted y values to compute the loss.\n", 
    "    loss = loss_fn(scores, y_var)\n", 
    "    \n", 
    "    if (t + 1) % print_every == 0:\n", 
    "        print('t = %d, loss = %.4f' % (t + 1, loss.data[0]))\n", 
    "\n", 
    "    # Zero out all of the gradients for the variables which the optimizer will update.\n", 
    "    optimizer.zero_grad()\n", 
    "    \n", 
    "    # This is the backwards pass: compute the gradient of the loss with respect to each \n", 
    "    # parameter of the model.\n", 
    "    loss.backward()\n", 
    "    \n", 
    "    # Actually update the parameters of the model using the gradients computed by the backwards pass.\n", 
    "    optimizer.step()"
   ], 
   "outputs": [], 
   "metadata": {
    "collapsed": false
   }
  }, 
  {
   "source": [
    "Now you've seen how the training process works in PyTorch. To save you writing boilerplate code, we're providing the following helper functions to help you train for multiple epochs and check the accuracy of your model:"
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "execution_count": 17, 
   "cell_type": "code", 
   "source": [
    "def train(model, loss_fn, optimizer, num_epochs = 1):\n", 
    "    for epoch in range(num_epochs):\n", 
    "        print('Starting epoch %d / %d' % (epoch + 1, num_epochs))\n", 
    "        model.train()\n", 
    "        for t, (x, y) in enumerate(loader_train):\n", 
    "            x_var = Variable(x.type(gpu_dtype))\n", 
    "            y_var = Variable(y.type(gpu_dtype).long())\n", 
    "\n", 
    "            scores = model(x_var)\n", 
    "            \n", 
    "            loss = loss_fn(scores, y_var)\n", 
    "            if (t + 1) % print_every == 0:\n", 
    "                print('t = %d, loss = %.4f' % (t + 1, loss.data[0]))\n", 
    "\n", 
    "            optimizer.zero_grad()\n", 
    "            loss.backward()\n", 
    "            optimizer.step()\n", 
    "\n", 
    "def check_accuracy(model, loader):\n", 
    "    if loader.dataset.train:\n", 
    "        print('Checking accuracy on validation set')\n", 
    "    else:\n", 
    "        print('Checking accuracy on test set')   \n", 
    "    num_correct = 0\n", 
    "    num_samples = 0\n", 
    "    model.eval() # Put the model in test mode (the opposite of model.train(), essentially)\n", 
    "    for x, y in loader:\n", 
    "        x_var = Variable(x.type(gpu_dtype), volatile=True)\n", 
    "\n", 
    "        scores = model(x_var)\n", 
    "        _, preds = scores.data.cpu().max(1)\n", 
    "        num_correct += (preds == y).sum()\n", 
    "        num_samples += preds.size(0)\n", 
    "    acc = float(num_correct) / num_samples\n", 
    "    print('Got %d / %d correct (%.2f)' % (num_correct, num_samples, 100 * acc))"
   ], 
   "outputs": [], 
   "metadata": {
    "collapsed": true
   }
  }, 
  {
   "source": [
    "### Check the accuracy of the model.\n", 
    "\n", 
    "Let's see the train and check_accuracy code in action -- feel free to use these methods when evaluating the models you develop below.\n", 
    "\n", 
    "You should get a training loss of around 1.2-1.4, and a validation accuracy of around 50-60%. As mentioned above, if you re-run the cells, you'll be training more epochs, so your performance will improve past these numbers.\n", 
    "\n", 
    "But don't worry about getting these numbers better -- this was just practice before you tackle designing your own model."
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "execution_count": null, 
   "cell_type": "code", 
   "source": [
    "torch.cuda.random.manual_seed(12345)\n", 
    "fixed_model_gpu.apply(reset)\n", 
    "train(fixed_model_gpu, loss_fn, optimizer, num_epochs=1)\n", 
    "check_accuracy(fixed_model_gpu, loader_val)"
   ], 
   "outputs": [], 
   "metadata": {
    "collapsed": false
   }
  }, 
  {
   "source": [
    "### Don't forget the validation set!\n", 
    "\n", 
    "And note that you can use the check_accuracy function to evaluate on either the test set or the validation set, by passing either **loader_test** or **loader_val** as the second argument to check_accuracy. You should not touch the test set until you have finished your architecture and hyperparameter tuning, and only run the test set once at the end to report a final value. "
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "source": [
    "## Train a _great_ model on CIFAR-10!\n", 
    "\n", 
    "Now it's your job to experiment with architectures, hyperparameters, loss functions, and optimizers to train a model that achieves **>=70%** accuracy on the CIFAR-10 **validation** set. You can use the check_accuracy and train functions from above."
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "source": [
    "### Things you should try:\n", 
    "- **Filter size**: Above we used 7x7; this makes pretty pictures but smaller filters may be more efficient\n", 
    "- **Number of filters**: Above we used 32 filters. Do more or fewer do better?\n", 
    "- **Pooling vs Strided Convolution**: Do you use max pooling or just stride convolutions?\n", 
    "- **Batch normalization**: Try adding spatial batch normalization after convolution layers and vanilla batch normalization after affine layers. Do your networks train faster?\n", 
    "- **Network architecture**: The network above has two layers of trainable parameters. Can you do better with a deep network? Good architectures to try include:\n", 
    "    - [conv-relu-pool]xN -> [affine]xM -> [softmax or SVM]\n", 
    "    - [conv-relu-conv-relu-pool]xN -> [affine]xM -> [softmax or SVM]\n", 
    "    - [batchnorm-relu-conv]xN -> [affine]xM -> [softmax or SVM]\n", 
    "- **Global Average Pooling**: Instead of flattening and then having multiple affine layers, perform convolutions until your image gets small (7x7 or so) and then perform an average pooling operation to get to a 1x1 image picture (1, 1 , Filter#), which is then reshaped into a (Filter#) vector. This is used in [Google's Inception Network](https://arxiv.org/abs/1512.00567) (See Table 1 for their architecture).\n", 
    "- **Regularization**: Add l2 weight regularization, or perhaps use Dropout.\n", 
    "\n", 
    "### Tips for training\n", 
    "For each network architecture that you try, you should tune the learning rate and regularization strength. When doing this there are a couple important things to keep in mind:\n", 
    "\n", 
    "- If the parameters are working well, you should see improvement within a few hundred iterations\n", 
    "- Remember the coarse-to-fine approach for hyperparameter tuning: start by testing a large range of hyperparameters for just a few training iterations to find the combinations of parameters that are working at all.\n", 
    "- Once you have found some sets of parameters that seem to work, search more finely around these parameters. You may need to train for more epochs.\n", 
    "- You should use the validation set for hyperparameter search, and save your test set for evaluating your architecture on the best parameters as selected by the validation set.\n", 
    "\n", 
    "### Going above and beyond\n", 
    "If you are feeling adventurous there are many other features you can implement to try and improve your performance. You are **not required** to implement any of these; however they would be good things to try for extra credit.\n", 
    "\n", 
    "- Alternative update steps: For the assignment we implemented SGD+momentum, RMSprop, and Adam; you could try alternatives like AdaGrad or AdaDelta.\n", 
    "- Alternative activation functions such as leaky ReLU, parametric ReLU, ELU, or MaxOut.\n", 
    "- Model ensembles\n", 
    "- Data augmentation\n", 
    "- New Architectures\n", 
    "  - [ResNets](https://arxiv.org/abs/1512.03385) where the input from the previous layer is added to the output.\n", 
    "  - [DenseNets](https://arxiv.org/abs/1608.06993) where inputs into previous layers are concatenated together.\n", 
    "  - [This blog has an in-depth overview](https://chatbotslife.com/resnets-highwaynets-and-densenets-oh-my-9bb15918ee32)\n", 
    "\n", 
    "If you do decide to implement something extra, clearly describe it in the \"Extra Credit Description\" cell below.\n", 
    "\n", 
    "### What we expect\n", 
    "At the very least, you should be able to train a ConvNet that gets at least 70% accuracy on the validation set. This is just a lower bound - if you are careful it should be possible to get accuracies much higher than that! Extra credit points will be awarded for particularly high-scoring models or unique approaches.\n", 
    "\n", 
    "You should use the space below to experiment and train your network. \n", 
    "\n", 
    "Have fun and happy training!"
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "execution_count": null, 
   "cell_type": "code", 
   "source": [
    "# Train your model here, and make sure the output of this cell is the accuracy of your best model on the \n", 
    "# train, val, and test sets. Here's some code to get you started. The output of this cell should be the training\n", 
    "# and validation accuracy on your best model (measured by validation accuracy).\n", 
    "\n", 
    "model = None\n", 
    "loss_fn = None\n", 
    "optimizer = None\n", 
    "\n", 
    "train(model, loss_fn, optimizer, num_epochs=1)\n", 
    "check_accuracy(model, loader_val)"
   ], 
   "outputs": [], 
   "metadata": {
    "collapsed": true
   }
  }, 
  {
   "source": [
    "### Describe what you did \n", 
    "\n", 
    "In the cell below you should write an explanation of what you did, any additional features that you implemented, and any visualizations or graphs that you make in the process of training and evaluating your network."
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "source": [
    "Tell us here!"
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "source": [
    "## Test set -- run this only once\n", 
    "\n", 
    "Now that we've gotten a result we're happy with, we test our final model on the test set (which you should store in best_model).  This would be the score we would achieve on a competition. Think about how this compares to your validation set accuracy."
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }, 
  {
   "execution_count": null, 
   "cell_type": "code", 
   "source": [
    "best_model = None\n", 
    "check_accuracy(best_model, loader_test)"
   ], 
   "outputs": [], 
   "metadata": {
    "collapsed": true
   }
  }, 
  {
   "source": [
    "## Going further with PyTorch\n", 
    "\n", 
    "The next assignment will make heavy use of PyTorch. You might also find it useful for your projects. \n", 
    "\n", 
    "Here's a nice tutorial by Justin Johnson that shows off some of PyTorch's features, like dynamic graphs and custom NN modules: http://pytorch.org/tutorials/beginner/pytorch_with_examples.html\n", 
    "\n", 
    "If you're interested in reinforcement learning for your final project, this is a good (more advanced) DQN tutorial in PyTorch: http://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html"
   ], 
   "cell_type": "markdown", 
   "metadata": {}
  }
 ], 
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3", 
   "name": "python3", 
   "language": "python"
  }, 
  "language_info": {
   "mimetype": "text/x-python", 
   "nbconvert_exporter": "python", 
   "name": "python", 
   "file_extension": ".py", 
   "version": "3.5.0", 
   "pygments_lexer": "ipython3", 
   "codemirror_mode": {
    "version": 3, 
    "name": "ipython"
   }
  }
 }
}