2B_Word_Embeddings_and_RNNs_Assignment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyTorch Assignment: Natural Language Processing (NLP)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[Duke Community Standard](http://integrity.duke.edu/standard.html): By typing your name below, you are certifying that you have adhered to the Duke Community Standard in completing this assignment.**\n",
    "\n",
    "Name: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text Classification\n",
    "\n",
    "In Notebook 2A, we built a sentiment analysis model for movie reviews.\n",
    "That particular sentiment analysis was a two-class classification problem, where the two classes were whether the review was positive or negative.\n",
    "Of course, natural language comes in all sorts of different forms; sometimes we want to perform other types of classification.\n",
    "\n",
    "For example, the AG News dataset contains text from 127600 online news articles, from 4 different categories: World, Sports, Business, and Science/Technology.\n",
    "AG News is typically used for topic classification: given an unseen news article, we're interested in predicting the topic.\n",
    "For this assignment, you'll be training several models on the AG News dataset.\n",
    "Unlike the quick example we trained in Notebook 2A, however, we're going to *learn* the word embeddings.\n",
    "Since you may be unfamiliar with AG News, we're going to walk through how to load the data, to get you started.\n",
    "\n",
    "\n",
    "### Loading AG News with Torchtext\n",
    "\n",
    "The AG News dataset is one of many included Torchtext.\n",
    "It can be found grouped together with many of the other text classification datasets.\n",
    "While we can download the source text online, Torchtext makes it retrievable with a quick API call&ast;:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "datasets\\ag_news_csv.tar.gz: 11.8MB [00:00, 18.8MB/s]\n",
      "120000lines [00:21, 5627.20lines/s]\n",
      "120000lines [00:31, 3771.66lines/s]\n",
      "7600lines [00:01, 4211.05lines/s]\n"
     ]
    }
   ],
   "source": [
    "import torchtext\n",
    "\n",
    "agnews_train, agnews_test = torchtext.datasets.text_classification.DATASETS[\"AG_NEWS\"](root=\"./datasets\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"1\">&ast;At the time this notebook was created, Torchtext contains a small bug in its csv reader.\n",
    "You may need to change one line in the source code, as suggested [here](https://discuss.pytorch.org/t/one-error-about-the-utils-pys-code/53885) to successfully load the AG News dataset.\n",
    "</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect the first data example to see how the data is formatted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2, tensor([    572,     564,       2,    2326,   49106,     150,      88,       3,\n",
      "           1143,      14,      32,      15,      32,      16,  443749,       4,\n",
      "            572,     499,      17,      10,  741769,       7,  468770,       4,\n",
      "             52,    7019,    1050,     442,       2,   14341,     673,  141447,\n",
      "         326092,   55044,    7887,     411,    9870,  628642,      43,      44,\n",
      "            144,     145,  299709,  443750,   51274,     703,   14312,      23,\n",
      "        1111134,  741770,  411508,  468771,    3779,   86384,  135944,  371666,\n",
      "           4052]))\n"
     ]
    }
   ],
   "source": [
    "print(agnews_train[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that Torchtext has each example as a tuple, with the first element being the label (0, 1, 2, or 3), and the second element the text data.\n",
    "Notice that the text is already \"tokenized\": the words of the news article have been represented as word IDs, with each number corresponding to a unique word.\n",
    "\n",
    "In previous notebooks, we've used `DataLoader`s to handle shuffling and batching.\n",
    "However, if we directly try to feed these dataset objects into a `DataLoader`, we will face an error when we try to draw our first batch.\n",
    "Can you figure out why?\n",
    "Here's a hint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Length of the first text example: 57\n",
      "Length of the second text example: 83\n"
     ]
    }
   ],
   "source": [
    "print(\"Length of the first text example: {}\".format(len(agnews_train[0][1])))\n",
    "print(\"Length of the second text example: {}\".format(len(agnews_train[1][1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because each example is a news snippet, they can vary in length.\n",
    "This is natural, as humans don't stick to consistent sentence length while writing.\n",
    "This creates a bit of a problem while batching, as default tensors expect the size of each dimension to be consistent.\n",
    "\n",
    "How do we fix this?\n",
    "The common solution is to perform padding and/or truncation, picking a maximum sequence length $L$.\n",
    "Inputs longer than the maximum length are truncated (i.e. $x_{t>L}$ are discarded), and shorter sequences have zeros padded to the end until they are all of length of $L$.\n",
    "We'll focus on padding here, for simplicity.\n",
    "\n",
    "We can perform this padding manually, but Pytorch has this functionality implemented.\n",
    "As an example, let's pad the first two sequences to the same length:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First sequence padded: tensor([    572,     564,       2,    2326,   49106,     150,      88,       3,\n",
      "           1143,      14,      32,      15,      32,      16,  443749,       4,\n",
      "            572,     499,      17,      10,  741769,       7,  468770,       4,\n",
      "             52,    7019,    1050,     442,       2,   14341,     673,  141447,\n",
      "         326092,   55044,    7887,     411,    9870,  628642,      43,      44,\n",
      "            144,     145,  299709,  443750,   51274,     703,   14312,      23,\n",
      "        1111134,  741770,  411508,  468771,    3779,   86384,  135944,  371666,\n",
      "           4052,       0,       0,       0,       0,       0,       0,       0,\n",
      "              0,       0,       0,       0,       0,       0,       0,       0,\n",
      "              0,       0,       0,       0,       0,       0,       0,       0,\n",
      "              0,       0,       0])\n",
      "First sequence length: 83\n",
      "Second sequence padded: tensor([  55003,    1474,    1150,    1832,    7559,      14,      32,      15,\n",
      "             32,      16,    1262,    1072,     436,   55003,     131,       4,\n",
      "         142576,      33,       6,    8062,      12,     756,  475640,       9,\n",
      "         991346,    3186,       8,       3,     698,     329,       4,      33,\n",
      "           6764, 1040465,   13979,      11,     278,     483,       7,       3,\n",
      "            172,       2,  659973,  193730, 1237754,  684719,  556644,      43,\n",
      "             44,     144,     145,   77775,   56578,   32382,  782124,   79225,\n",
      "           2908,  140697,  540900,    2031,   31960,   45339,   21562,  936430,\n",
      "        1282186,  578442,  991347,   69671,      26,    9260,  717285,    5378,\n",
      "            597,   27622, 1070413, 1040466,   38669,   27790,  175394,     711,\n",
      "             29,    1404,    1818])\n",
      "Second sequence length: 83\n"
     ]
    }
   ],
   "source": [
    "from torch.nn.utils.rnn import pad_sequence\n",
    "\n",
    "padded_exs = pad_sequence([agnews_train[0][1], agnews_train[1][1]])\n",
    "print(\"First sequence padded: {}\".format(padded_exs[:,0]))\n",
    "print(\"First sequence length: {}\".format(len(padded_exs[:,0])))\n",
    "print(\"Second sequence padded: {}\".format(padded_exs[:,1]))\n",
    "print(\"Second sequence length: {}\".format(len(padded_exs[:,1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although originally of unequal lengths, both sequences are now the same length, with the shorter one padded with zeros.\n",
    "\n",
    "We'd like the `DataLoader` to perform this padding operation as part of its batching process, as this will allow us to effectively combine varying-length sequences in the same input tensor.\n",
    "Fortunately, `Dataloader`s let us override the default batching behavior with the `collate_fn` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "\n",
    "def collator(batch):\n",
    "    labels = torch.tensor([example[0] for example in batch])\n",
    "    sentences = [example[1] for example in batch]\n",
    "    data = pad_sequence(sentences)\n",
    "    \n",
    "    return [data, labels]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our collator padding our sequences, we can create our `DataLoader`s.\n",
    "One last thing we need to do is choose a batch size for our `DataLoader`.\n",
    "This may be something you have to play around with.\n",
    "Too big and you may exceed your system's memory; too small and training may take longer (especially on CPU).\n",
    "Batch size also tends to influence training dynamics and model generalization.\n",
    "Fiddle around and see what works best."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 128\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(agnews_train, batch_size=BATCH_SIZE, shuffle=True, collate_fn=collator)\n",
    "test_loader = torch.utils.data.DataLoader(agnews_test, batch_size=BATCH_SIZE, shuffle=False, collate_fn=collator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple Word Embedding Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's try out the Simple Word Embedding Model (SWEM) that we built in Notebook 2A on the AG News dataset.\n",
    "Unlike before though, instead of loading pre-trained embeddings, let's learn the embeddings from scratch.\n",
    "Before we begin, it will be helpful to define a few more hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "VOCAB_SIZE = len(agnews_train.get_vocab())\n",
    "EMBED_DIM = 100\n",
    "HIDDEN_DIM = 64\n",
    "NUM_OUTPUTS = len(agnews_train.get_labels())\n",
    "NUM_EPOCHS = 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, we're going to organize our model as a `nn.Module`.\n",
    "Instead of assuming the input is already an embedding, we're going to make learning the embedding as part of our model.\n",
    "We do this by using `nn.Embedding` to perform an embedding look-up at the beginning of our forward pass.\n",
    "Once we've done the look up, we'll have a minibatch of embedded sequences of dimension $L \\times$ `BATCH_SIZE` $\\times$ `EMBED_DIM`.\n",
    "For SWEM, remember, we take the mean&ast; across the length dimension to get an average embedding for the sequence.\n",
    "\n",
    "<font size=\"1\"> \n",
    "&ast;Note: Technically we should only take the mean across the embeddings at the positions corresponding to \"real\" words in our input, and not for the zero paddings we artificially added.\n",
    "This can be done by generating a binary mask while doing the padding to track the \"real\" words in the input.\n",
    "Ultimately though, this refinement doesn't have much impact on the results for this particular task, so we omit it for simplicity.\n",
    "</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class SWEM(nn.Module):\n",
    "    def __init__(self, vocab_size, embedding_size, hidden_dim, num_outputs):\n",
    "        super().__init__()\n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_size)\n",
    "        \n",
    "        self.fc1 = nn.Linear(embedding_size, hidden_dim)\n",
    "        self.fc2 = nn.Linear(hidden_dim, num_outputs)\n",
    "\n",
    "    def forward(self, x):\n",
    "        embed = self.embedding(x)\n",
    "        embed_mean = torch.mean(embed, dim=0)\n",
    "        \n",
    "        h = self.fc1(embed_mean)\n",
    "        h = F.relu(h)\n",
    "        h = self.fc2(h)\n",
    "        return h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the model defined, we can instantiate, train, and evaluate.\n",
    "Try doing so below!\n",
    "Because of the way we organized our model as an `nn.Module` and our data pipeline with a `DataLoader`, you should be able to use much of the same code as we have in other examples.\n",
    "\n",
    "<font size=\"1\"> \n",
    "Note: Depending on you system, training may take up to a few hours, depending on how many training epochs you set.\n",
    "To see results sooner, you can train for less iterations, but perhaps at the cost of final accuracy.\n",
    "On the other hand, using a GPU (and GPU-enabled PyTorch) should enable full training in a couple minutes.\n",
    "</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "### YOUR CODE HERE ###\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RNNs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SWEM takes a mean over the time dimension, which means we're losing any information about the order of the data sequence.\n",
    "How detrimental is this for document topic classification?\n",
    "Modify the SWEM model to use an RNN instead.\n",
    "Once you get an RNN working, try a GRU and LSTM as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### YOUR CODE HERE ###\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Short Answer:\n",
    "\n",
    "1\\. How do the RNN, GRU, and LSTM compare to SWEM for AG News topic classification?\n",
    "Are you surprised?\n",
    "What about classification might make SWEM so effective for topic classification?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`[Your answer here]`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2\\. How many learnable parameters do each of the models you've trained have?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`[Your answer here]`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}