Assignment xixiaoyao#3 , w/ans to written parts

Assignments/assignment1/Herais/README.txt

@@ -0,0 +1,28 @@
+Welcome to CS224N!
+
+We'll be using Python throughout the course. If you've got a good Python setup already, great! But make sure that it is at least Python version 3.5. If not, the easiest thing to do is to make sure you have at least 3GB free on your computer and then to head over to (https://www.anaconda.com/download/) and install the Python 3 version of Anaconda. It will work on any operating system.
+
+After you have installed conda, close any open terminals you might have. Then open a new terminal and run the following command:
+
+# 1. Create an environment with dependencies specified in env.yml:
+
+    conda env create -f env.yml
+
+# 2. Activate the new environment:
+
+    conda activate cs224n
+
+# 3. Inside the new environment, instatll IPython kernel so we can use this environment in jupyter notebook: 
+
+    python -m ipykernel install --user --name cs224n
+
+
+# 4. Homework 1 (only) is a Jupyter Notebook. With the above done you should be able to get underway by typing:
+
+    jupyter notebook exploring_word_vectors.ipynb
+
+# 5. To make sure we are using the right environment, go to the toolbar of exploring_word_vectors.ipynb, click on Kernel -> Change kernel, you should see and select cs224n in the drop-down menu.
+
+# To deactivate an active environment, use
+
+    conda deactivate

Assignments/assignment3/Herais/collect_submission.sh

@@ -0,0 +1,2 @@
+rm -f assignment3.zip
+zip -r assignment3.zip *.py ./data ./utils

Assignments/assignment3/Herais/hw3_parser_model.ipynb

@@ -0,0 +1 @@
+{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"hw3_parser_model.ipynb","provenance":[],"collapsed_sections":[],"toc_visible":true,"mount_file_id":"1ED5MVnsUxTK8Cu4sAgtO1-nLcz-HQ49_","authorship_tag":"ABX9TyMvBCZF8m6Imaz07OYy4vnS"},"kernelspec":{"name":"python3","display_name":"Python 3"}},"cells":[{"cell_type":"code","metadata":{"id":"RUEHv7P_2eRO","colab_type":"code","colab":{}},"source":["#!/usr/bin/env python3\n","# -*- coding: utf-8 -*-\n","\"\"\"\n","CS224N 2019-20: Homework 3\n","parser_model.py: Feed-Forward Neural Network for Dependency Parsing\n","Sahil Chopra <[email protected]>\n","Haoshen Hong <[email protected]>\n","\"\"\"\n","import argparse\n","import numpy as np\n","\n","import torch\n","import torch.nn as nn\n","import torch.nn.functional as F"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"4u0IK78EwG3a","colab_type":"code","colab":{}},"source":["class ParserModel(nn.Module):\n","    \"\"\" Feedforward neural network with an embedding layer and two hidden layers.\n","    The ParserModel will predict which transition should be applied to a\n","    given partial parse configuration.\n","\n","    PyTorch Notes:\n","        - Note that \"ParserModel\" is a subclass of the \"nn.Module\" class. In PyTorch all neural networks\n","            are a subclass of this \"nn.Module\".\n","        - The \"__init__\" method is where you define all the layers and parameters\n","            (embedding layers, linear layers, dropout layers, etc.).\n","        - \"__init__\" gets automatically called when you create a new instance of your class, e.g.\n","            when you write \"m = ParserModel()\".\n","        - Other methods of ParserModel can access variables that have \"self.\" prefix. Thus,\n","            you should add the \"self.\" prefix layers, values, etc. that you want to utilize\n","            in other ParserModel methods.\n","        - For further documentation on \"nn.Module\" please see https://pytorch.org/docs/stable/nn.html.\n","    \"\"\"\n","    def __init__(self, embeddings, n_features=36,\n","        hidden_size=200, n_classes=3, dropout_prob=0.5):\n","        \"\"\" Initialize the parser model.\n","\n","        @param embeddings (ndarray): word embeddings (num_words, embedding_size)\n","        @param n_features (int): number of input features\n","        @param hidden_size (int): number of hidden units\n","        @param n_classes (int): number of output classes\n","        @param dropout_prob (float): dropout probability\n","        \"\"\"\n","        super(ParserModel, self).__init__()\n","        self.n_features = n_features\n","        self.n_classes = n_classes\n","        self.dropout_prob = dropout_prob\n","        self.embed_size = embeddings.shape[1]\n","        self.hidden_size = hidden_size\n","        self.embeddings = nn.Parameter(torch.tensor(embeddings))\n","\n","        ### YOUR CODE HERE (~10 Lines)\n","        ### TODO:\n","        ###     1) Declare `self.embed_to_hidden_weight` and `self.embed_to_hidden_bias` as `nn.Parameter`.\n","        ###        Initialize weight with the `nn.init.xavier_uniform_` function and bias with `nn.init.uniform_`\n","        ###        with default parameters.\n","        ###     2) Construct `self.dropout` layer.\n","        ###     3) Declare `self.hidden_to_logits_weight` and `self.hidden_to_logits_bias` as `nn.Parameter`.\n","        ###        Initialize weight with the `nn.init.xavier_uniform_` function and bias with `nn.init.uniform_`\n","        ###        with default parameters.\n","        ###\n","        ### Note: Trainable variables are declared as `nn.Parameter` which is a commonly used API\n","        ###       to include a tensor into a computational graph to support updating w.r.t its gradient.\n","        ###       Here, we use Xavier Uniform Initialization for our Weight initialization.\n","        ###       It has been shown empirically, that this provides better initial weights\n","        ###       for training networks than random uniform initialization.\n","        ###       For more details checkout this great blogpost:\n","        ###             http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization\n","        ###\n","        ### Please see the following docs for support:\n","        ###     nn.Parameter: https://pytorch.org/docs/stable/nn.html#parameters\n","        ###     Initialization: https://pytorch.org/docs/stable/nn.init.html\n","        ###     Dropout: https://pytorch.org/docs/stable/nn.html#dropout-layers\n","        \n","        self.embed_to_hidden_weight = nn.Parameter(nn.init.xavier_uniform_(torch.empty(self.embed_size * self.n_features, self.hidden_size)))\n","        self.embed_to_hidden_bias = nn.Parameter(nn.init.uniform_(torch.empty(1, self.hidden_size)))\n","        self.dropout = nn.Dropout(self.dropout_prob)\n","        self.hidden_to_logits_weight = nn.Parameter(nn.init.xavier_uniform_(torch.empty(self.hidden_size, self.n_classes)))\n","        self.hidden_to_logits_bias = nn.Parameter(nn.init.uniform_(torch.empty(1, self.n_classes)))\n","        ### END YOUR CODE\n","\n","    def embedding_lookup(self, w):\n","        \"\"\" Utilize `w` to select embeddings from embedding matrix `self.embeddings`\n","            @param w (Tensor): input tensor of word indices (batch_size, n_features)\n","\n","            @return x (Tensor): tensor of embeddings for words represented in w\n","                                (batch_size, n_features * embed_size)\n","        \"\"\"\n","\n","        ### YOUR CODE HERE (~1-3 Lines)\n","        ### TODO:\n","        ###     1) For each index `i` in `w`, select `i`th vector from self.embeddings\n","        ###     2) Reshape the tensor using `view` function if necessary\n","        ###\n","        ### Note: All embedding vectors are stacked and stored as a matrix. The model receives\n","        ###       a list of indices representing a sequence of words, then it calls this lookup\n","        ###       function to map indices to sequence of embeddings.\n","        ###\n","        ###       This problem aims to test your understanding of embedding lookup,\n","        ###       so DO NOT use any high level API like nn.Embedding\n","        ###       (we are asking you to implement that!). Pay attention to tensor shapes\n","        ###       and reshape if necessary. Make sure you know each tensor's shape before you run the code!\n","        ###\n","        ### Pytorch has some useful APIs for you, and you can use either one\n","        ### in this problem (except nn.Embedding). These docs might be helpful:\n","        ###     Index select: https://pytorch.org/docs/stable/torch.html#torch.index_select\n","        ###     Gather: https://pytorch.org/docs/stable/torch.html#torch.gather\n","        ###     View: https://pytorch.org/docs/stable/tensors.html#torch.Tensor.view\n","        x = self.embeddings[w].view(w.shape[0],-1)\n","      \n","        ### END YOUR CODE\n","        return x\n","\n","\n","    def forward(self, w):\n","        \"\"\" Run the model forward.\n","\n","            Note that we will not apply the softmax function here because it is included in the loss function nn.CrossEntropyLoss\n","\n","            PyTorch Notes:\n","                - Every nn.Module object (PyTorch model) has a `forward` function.\n","                - When you apply your nn.Module to an input tensor `w` this function is applied to the tensor.\n","                    For example, if you created an instance of your ParserModel and applied it to some `w` as follows,\n","                    the `forward` function would called on `w` and the result would be stored in the `output` variable:\n","                        model = ParserModel()\n","                        output = model(w) # this calls the forward function\n","                - For more details checkout: https://pytorch.org/docs/stable/nn.html#torch.nn.Module.forward\n","\n","        @param w (Tensor): input tensor of tokens (batch_size, n_features)\n","\n","        @return logits (Tensor): tensor of predictions (output after applying the layers of the network)\n","                                 without applying softmax (batch_size, n_classes)\n","        \"\"\"\n","        ### YOUR CODE HERE (~3-5 lines)\n","        ### TODO:\n","        ###     Complete the forward computation as described in write-up. In addition, include a dropout layer\n","        ###     as decleared in `__init__` after ReLU function.\n","        ###\n","        ### Note: We do not apply the softmax to the logits here, because\n","        ### the loss function (torch.nn.CrossEntropyLoss) applies it more efficiently.\n","        ###\n","        ### Please see the following docs for support:\n","        ###     Matrix product: https://pytorch.org/docs/stable/torch.html#torch.matmul\n","        ###     ReLU: https://pytorch.org/docs/stable/nn.html?highlight=relu#torch.nn.functional.relu\n","        x = self.embedding_lookup(w)\n","        h = nn.functional.relu(torch.matmul(x, self.embed_to_hidden_weight) + self.embed_to_hidden_bias)\n","        h = self.dropout(h)\n","        logits = torch.matmul(h, self.hidden_to_logits_weight) + self.hidden_to_logits_bias\n","\n","        ### END YOUR CODE\n","        return logits\n"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"eiaSnQbZzqpj","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":70},"outputId":"743a27cd-255c-40d6-e1f0-4e11d0f9a466","executionInfo":{"status":"ok","timestamp":1588044269175,"user_tz":420,"elapsed":3055,"user":{"displayName":"Xu Vee","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjNTzTx2PmIBbypp7-eWIRwMo4-I9DLyTz70KE9_g=s64","userId":"12955926615284697296"}}},"source":["%cd /content/drive/My Drive/cs224n/assignment3\n","!python parser_model.py -e -f"],"execution_count":73,"outputs":[{"output_type":"stream","text":["/content/drive/My Drive/cs224n/assignment3\n","Embedding_lookup sanity check passes!\n","Forward sanity check passes!\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"2zosYD-mo8ri","colab_type":"code","colab":{}},"source":["#### MY WORKOUTS ####\n","embeddings = np.zeros((100, 30), dtype=np.float32)\n","embeddings = nn.Parameter(torch.tensor(embeddings))\n","w = torch.randint(0, 100, (4, 36), dtype=torch.long)\n","pm = ParserModel(embeddings)\n","\n","x =  pm.embedding_lookup(w)\n","W = pm.embed_to_hidden_weight\n","h = nn.functional.relu(torch.matmul(x,W) + pm.embed_to_hidden_bias)\n","pm.dropout(h)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"yeOlRVpY2ouh","colab_type":"code","colab":{}},"source":["if __name__ == \"__main__\":\n","\n","    parser = argparse.ArgumentParser(description='Simple sanity check for parser_model.py')\n","    parser.add_argument('-e', '--embedding', action='store_true', help='sanity check for embeding_lookup function')\n","    parser.add_argument('-f', '--forward', action='store_true', help='sanity check for forward function')\n","    args = parser.parse_args()\n","\n","    embeddings = np.zeros((100, 30), dtype=np.float32)\n","    model = ParserModel(embeddings)\n","\n","    def check_embedding():\n","        inds = torch.randint(0, 100, (4, 36), dtype=torch.long)\n","        selected = model.embedding_lookup(inds)\n","        assert np.all(selected.data.numpy() == 0), \"The result of embedding lookup: \" \\\n","                                      + repr(selected) + \" contains non-zero elements.\"\n","\n","    def check_forward():\n","        inputs =torch.randint(0, 100, (4, 36), dtype=torch.long)\n","        out = model(inputs)\n","        expected_out_shape = (4, 3)\n","        assert out.shape == expected_out_shape, \"The result shape of forward is: \" + repr(out.shape) + \\\n","                                                \" which doesn't match expected \" + repr(expected_out_shape)\n","\n","    if args.embedding:\n","        check_embedding()\n","        print(\"Embedding_lookup sanity check passes!\")\n","\n","    if args.forward:\n","        check_forward()\n","        print(\"Forward sanity check passes!\")"],"execution_count":0,"outputs":[]}]}

Assignments/assignment3/Herais/local_env.yml

@@ -0,0 +1,11 @@
+name: a3
+channels:
+  - pytorch
+  - defaults
+dependencies:
+  - python=3.7
+  - numpy
+  - tqdm
+  - docopt
+  - pytorch
+  - torchvision