TFIDF_WithoutLibrary.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "h9464I-uxLiw"
   },
   "source": [
    "# Assignment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IvLvmewlxLix"
   },
   "source": [
    "<font face='georgia'>\n",
    "    \n",
    "   <h4><strong>What does tf-idf mean?</strong></h4>\n",
    "\n",
    "   <p>    \n",
    "Tf-idf stands for <em>term frequency-inverse document frequency</em>, and the tf-idf weight is a weight often used in information retrieval and text mining. This weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus. The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus. Variations of the tf-idf weighting scheme are often used by search engines as a central tool in scoring and ranking a document's relevance given a user query.\n",
    "</p>\n",
    "    \n",
    "   <p>\n",
    "One of the simplest ranking functions is computed by summing the tf-idf for each query term; many more sophisticated ranking functions are variants of this simple model.\n",
    "</p>\n",
    "    \n",
    "   <p>\n",
    "Tf-idf can be successfully used for stop-words filtering in various subject fields including text summarization and classification.\n",
    "</p>\n",
    "    \n",
    "</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4XLaGFUMxLiy"
   },
   "source": [
    "<font face='georgia'>\n",
    "    <h4><strong>How to Compute:</strong></h4>\n",
    "\n",
    "Typically, the tf-idf weight is composed by two terms: the first computes the normalized Term Frequency (TF), aka. the number of times a word appears in a document, divided by the total number of words in that document; the second term is the Inverse Document Frequency (IDF), computed as the logarithm of the number of the documents in the corpus divided by the number of documents where the specific term appears.\n",
    "\n",
    " <ul>\n",
    "    <li>\n",
    "<strong>TF:</strong> Term Frequency, which measures how frequently a term occurs in a document. Since every document is different in length, it is possible that a term would appear much more times in long documents than shorter ones. Thus, the term frequency is often divided by the document length (aka. the total number of terms in the document) as a way of normalization: <br>\n",
    "\n",
    "$TF(t) = \\frac{\\text{Number of times term t appears in a document}}{\\text{Total number of terms in the document}}.$\n",
    "</li>\n",
    "<li>\n",
    "<strong>IDF:</strong> Inverse Document Frequency, which measures how important a term is. While computing TF, all terms are considered equally important. However it is known that certain terms, such as \"is\", \"of\", and \"that\", may appear a lot of times but have little importance. Thus we need to weigh down the frequent terms while scale up the rare ones, by computing the following: <br>\n",
    "\n",
    "$IDF(t) = \\log_{e}\\frac{\\text{Total  number of documents}} {\\text{Number of documents with term t in it}}.$\n",
    "for numerical stabiltiy we will be changing this formula little bit\n",
    "$IDF(t) = \\log_{e}\\frac{\\text{Total  number of documents}} {\\text{Number of documents with term t in it}+1}.$\n",
    "</li>\n",
    "</ul>\n",
    "\n",
    "<br>\n",
    "<h4><strong>Example</strong></h4>\n",
    "<p>\n",
    "\n",
    "Consider a document containing 100 words wherein the word cat appears 3 times. The term frequency (i.e., tf) for cat is then (3 / 100) = 0.03. Now, assume we have 10 million documents and the word cat appears in one thousand of these. Then, the inverse document frequency (i.e., idf) is calculated as log(10,000,000 / 1,000) = 4. Thus, the Tf-idf weight is the product of these quantities: 0.03 * 4 = 0.12.\n",
    "</p>\n",
    "</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dg2ooa4DxLiz"
   },
   "source": [
    "## Task-1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TAcTjhO8xLiz"
   },
   "source": [
    "<font face='georgia'>\n",
    "    <h4><strong>1. Build a TFIDF Vectorizer & compare its results with Sklearn:</strong></h4>\n",
    "\n",
    "<ul>\n",
    "    <li> As a part of this task you will be implementing TFIDF vectorizer on a collection of text documents.</li>\n",
    "    <br>\n",
    "    <li> You should compare the results of your own implementation of TFIDF vectorizer with that of sklearns implemenation TFIDF vectorizer.</li>\n",
    "    <br>\n",
    "    <li> Sklearn does few more tweaks in the implementation of its version of TFIDF vectorizer, so to replicate the exact results you would need to add following things to your custom implementation of tfidf vectorizer:\n",
    "       <ol>\n",
    "        <li> Sklearn has its vocabulary generated from idf sroted in alphabetical order</li>\n",
    "        <li> Sklearn formula of idf is different from the standard textbook formula. Here the constant <strong>\"1\"</strong> is added to the numerator and denominator of the idf as if an extra document was seen containing every term in the collection exactly once, which prevents zero divisions.\n",
    "            \n",
    " $IDF(t) = 1+\\log_{e}\\frac{1\\text{ }+\\text{ Total  number of documents in collection}} {1+\\text{Number of documents with term t in it}}.$\n",
    "        </li>\n",
    "        <li> Sklearn applies L2-normalization on its output matrix.</li>\n",
    "        <li> The final output of sklearn tfidf vectorizer is a sparse matrix.</li>\n",
    "    </ol>\n",
    "    <br>\n",
    "    <li>Steps to approach this task:\n",
    "    <ol>\n",
    "        <li> You would have to write both fit and transform methods for your custom implementation of tfidf vectorizer.</li>\n",
    "        <li> Print out the alphabetically sorted voacb after you fit your data and check if its the same as that of the feature names from sklearn tfidf vectorizer. </li>\n",
    "        <li> Print out the idf values from your implementation and check if its the same as that of sklearns tfidf vectorizer idf values. </li>\n",
    "        <li> Once you get your voacb and idf values to be same as that of sklearns implementation of tfidf vectorizer, proceed to the below steps. </li>\n",
    "        <li> Make sure the output of your implementation is a sparse matrix. Before generating the final output, you need to normalize your sparse matrix using L2 normalization. You can refer to this link https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.normalize.html </li>\n",
    "        <li> After completing the above steps, print the output of your custom implementation and compare it with sklearns implementation of tfidf vectorizer.</li>\n",
    "        <li> To check the output of a single document in your collection of documents,  you can convert the sparse matrix related only to that document into dense matrix and print it.</li>\n",
    "        </ol>\n",
    "    </li>\n",
    "    <br>\n",
    "   </ul>\n",
    "\n",
    "  <p> <font color=\"#e60000\"><strong>Note-1: </strong></font> All the necessary outputs of sklearns tfidf vectorizer have been provided as reference in this notebook, you can compare your outputs as mentioned in the above steps, with these outputs.<br>\n",
    "   <font color=\"#e60000\"><strong>Note-2: </strong></font> The output of your custom implementation and that of sklearns implementation would match only with the collection of document strings provided to you as reference in this notebook. It would not match for strings that contain capital letters or punctuations, etc, because sklearn version of tfidf vectorizer deals with such strings in a different way. To know further details about how sklearn tfidf vectorizer works with such string, you can always refer to its official documentation.<br>\n",
    "   <font color=\"#e60000\"><strong>Note-3: </strong></font> During this task, it would be helpful for you to debug the code you write with print statements wherever necessary. But when you are finally submitting the assignment, make sure your code is readable and try not to print things which are not part of this task.\n",
    "    </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OnV82tg1xLi0"
   },
   "source": [
    "### Corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bUsYm9wjxLi1"
   },
   "outputs": [],
   "source": [
    "## SkLearn# Collection of string documents\n",
    "\n",
    "corpus = [\n",
    "     'this is the first document',\n",
    "     'this document is the second document',\n",
    "     'and this is the third one',\n",
    "     'is this the first document',\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eLwmFZfKxLi4"
   },
   "source": [
    "### SkLearn Implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Np4dfQOkxLi4"
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "vectorizer = TfidfVectorizer()\n",
    "vectorizer.fit(corpus)\n",
    "skl_output = vectorizer.transform(corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-7Om8YpYxLi6",
    "outputId": "0a3bd0f5-4424-4400-944f-4482a80bd799"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this']\n"
     ]
    }
   ],
   "source": [
    "# sklearn feature names, they are sorted in alphabetic order by default.\n",
    "\n",
    "print(vectorizer.get_feature_names())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dTKplK96xLi-",
    "outputId": "53722fa2-6756-4aa0-f179-37b578bb6890"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1.91629073 1.22314355 1.51082562 1.         1.91629073 1.91629073\n",
      " 1.         1.91629073 1.        ]\n"
     ]
    }
   ],
   "source": [
    "# Here we will print the sklearn tfidf vectorizer idf values after applying the fit method\n",
    "# After using the fit function on the corpus the vocab has 9 words in it, and each has its idf value.\n",
    "\n",
    "print(vectorizer.idf_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-CTiWHygxLjA",
    "outputId": "8d5a9cde-2c29-4afe-f7b4-1547e88dba4f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4, 9)"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# shape of sklearn tfidf vectorizer output after applying transform method.\n",
    "\n",
    "skl_output.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bDKEpbA-xLjD",
    "outputId": "87dafd65-5313-443f-8c6e-1b05cc8c2543"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 8)\t0.38408524091481483\n",
      "  (0, 6)\t0.38408524091481483\n",
      "  (0, 3)\t0.38408524091481483\n",
      "  (0, 2)\t0.5802858236844359\n",
      "  (0, 1)\t0.46979138557992045\n"
     ]
    }
   ],
   "source": [
    "# sklearn tfidf values for first line of the above corpus.\n",
    "# Here the output is a sparse matrix\n",
    "\n",
    "print(skl_output[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3QWo34hexLjF",
    "outputId": "cdc04e08-989f-4bdc-dd7f-f1c82a9f90be"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.         0.46979139 0.58028582 0.38408524 0.         0.\n",
      "  0.38408524 0.         0.38408524]]\n"
     ]
    }
   ],
   "source": [
    "# sklearn tfidf values for first line of the above corpus.\n",
    "# To understand the output better, here we are converting the sparse output matrix to dense matrix and printing it.\n",
    "# Notice that this output is normalized using L2 normalization. sklearn does this by default.\n",
    "\n",
    "print(skl_output[0].toarray())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qfIwx5LzxLjI"
   },
   "source": [
    "### Your custom implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HjuCcJwXxLjJ"
   },
   "outputs": [],
   "source": [
    "# Write your code here.\n",
    "# Make sure its well documented and readble with appropriate comments.\n",
    "# Compare your results with the above sklearn tfidf vectorizer\n",
    "# You are not supposed to use any other library apart from the ones given below\n",
    "\n",
    "from collections import Counter\n",
    "from tqdm import tqdm\n",
    "from scipy.sparse import csr_matrix\n",
    "import math\n",
    "import operator\n",
    "from sklearn.preprocessing import normalize\n",
    "import numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'and': 0, 'document': 1, 'first': 2, 'is': 3, 'one': 4, 'second': 5, 'the': 6, 'third': 7, 'this': 8}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "#building vocabulory for task1\n",
    "def fit(corpus):\n",
    "    unique_word=set()\n",
    "    if isinstance(corpus,list):\n",
    "        for row in corpus:\n",
    "            for word in row.split(\" \"):\n",
    "                unique_word.add(word)\n",
    "        unique_word=sorted(unique_word)    \n",
    "        vocab={j:i for i,j in enumerate(tqdm(unique_word))} #creating a vocabulory dictionary with word as key and index as value\n",
    "    else:\n",
    "        print(\"Please paas a list to fit function\")\n",
    "    \n",
    "    return vocab \n",
    "\n",
    "q=fit(corpus)\n",
    "print(q)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:00<00:00, 9007.10it/s]\n",
      "100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "idf of terms  \n",
      " {'and': 1.916290731874155, 'document': 1.2231435513142097, 'first': 1.5108256237659907, 'is': 1.0, 'one': 1.916290731874155, 'second': 1.916290731874155, 'the': 1.0, 'third': 1.916290731874155, 'this': 1.0}\n",
      "====================================================================================================\n",
      "Shape of sparse matrix \n",
      " (4, 9)\n",
      "====================================================================================================\n",
      "Normalized tfidf  sparse matrix for 0th row \n",
      "   (0, 1)\t0.46979138557992045\n",
      "  (0, 2)\t0.5802858236844359\n",
      "  (0, 3)\t0.3840852409148149\n",
      "  (0, 6)\t0.3840852409148149\n",
      "  (0, 8)\t0.3840852409148149\n",
      "====================================================================================================\n",
      "Normalized tfidf  dense matrix for 0th row \n",
      " [[0.         0.46979139 0.58028582 0.38408524 0.         0.\n",
      "  0.38408524 0.         0.38408524]]\n",
      "====================================================================================================\n",
      "Normalized dense matrix [[0.         0.46979139 0.58028582 0.38408524 0.         0.\n",
      "  0.38408524 0.         0.38408524]\n",
      " [0.         0.6876236  0.         0.28108867 0.         0.53864762\n",
      "  0.28108867 0.         0.28108867]\n",
      " [0.51184851 0.         0.         0.26710379 0.51184851 0.\n",
      "  0.26710379 0.51184851 0.26710379]\n",
      " [0.         0.46979139 0.58028582 0.38408524 0.         0.\n",
      "  0.38408524 0.         0.38408524]]\n",
      "====================================================================================================\n",
      "Normalized sparse matrix   (0, 1)\t0.46979138557992045\n",
      "  (0, 2)\t0.5802858236844359\n",
      "  (0, 3)\t0.3840852409148149\n",
      "  (0, 6)\t0.3840852409148149\n",
      "  (0, 8)\t0.3840852409148149\n",
      "  (1, 1)\t0.6876235979836938\n",
      "  (1, 3)\t0.2810886740337529\n",
      "  (1, 5)\t0.5386476208856763\n",
      "  (1, 6)\t0.2810886740337529\n",
      "  (1, 8)\t0.2810886740337529\n",
      "  (2, 0)\t0.511848512707169\n",
      "  (2, 3)\t0.267103787642168\n",
      "  (2, 4)\t0.511848512707169\n",
      "  (2, 6)\t0.267103787642168\n",
      "  (2, 7)\t0.511848512707169\n",
      "  (2, 8)\t0.267103787642168\n",
      "  (3, 1)\t0.46979138557992045\n",
      "  (3, 2)\t0.5802858236844359\n",
      "  (3, 3)\t0.3840852409148149\n",
      "  (3, 6)\t0.3840852409148149\n",
      "  (3, 8)\t0.3840852409148149\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "def transform(corpus,vocab):\n",
    "    rows = []\n",
    "    columns = []\n",
    "    values = []\n",
    "    term_idf=dict()\n",
    "    for key,j in vocab.items():\n",
    "        count=0\n",
    "        for row in corpus:\n",
    "            if key in row.split(\" \"):\n",
    "                count+=1\n",
    "        term_idf[key]=(1+math.log((1+len(corpus))/(1+count))) #calculating idf for terms in corpus\n",
    "    \n",
    "    print(\"idf of terms  \\n\",term_idf)\n",
    "    print(\"=\"*100)\n",
    "    for row_id,row in enumerate(tqdm(corpus)):\n",
    "        word_freq = dict(Counter(row.split()))\n",
    "        for word, freq in word_freq.items(): \n",
    "            tf=freq/len(row)                    #calculating tf for terms in document\n",
    "            col_index=vocab.get(word,-1)\n",
    "            if col_index!=-1:\n",
    "                rows.append(row_id)\n",
    "                columns.append(col_index)\n",
    "                values.append(tf*term_idf[word])\n",
    "                \n",
    "    return csr_matrix((values, (rows,columns)), shape=(len(corpus),len(vocab)))\n",
    "\n",
    "\n",
    "vocab=fit(corpus)\n",
    "tf_idf_sparse=transform(corpus,vocab)\n",
    "print(\"Shape of sparse matrix \\n\",tf_idf_sparse.shape)\n",
    "print(\"=\"*100)\n",
    "normalized_tfidfmatrix=normalize(tf_idf_sparse,norm='l2') #normalizing the sparse matrix\n",
    "print(\"Normalized tfidf  sparse matrix for 0th row \\n\",normalized_tfidfmatrix[0])\n",
    "print(\"=\"*100)\n",
    "print(\"Normalized tfidf  dense matrix for 0th row \\n\",normalized_tfidfmatrix[0].toarray())\n",
    "print(\"=\"*100)\n",
    "print(\"Normalized dense matrix\",normalized_tfidfmatrix.toarray())\n",
    "print(\"=\"*100)\n",
    "print(\"Normalized sparse matrix\",normalized_tfidfmatrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MMxBmVZExLjK"
   },
   "source": [
    "## Task-2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "51j_OtqAxLjL"
   },
   "source": [
    "<font face='georgia'>\n",
    "    <h4><strong>2. Implement max features functionality:</strong></h4>\n",
    "\n",
    "<ul>\n",
    "    <li> As a part of this task you have to modify your fit and transform functions so that your vocab will contain only 50 terms with top idf scores.</li>\n",
    "    <br>\n",
    "    <li>This task is similar to your previous task, just that here your vocabulary is limited to only top 50 features names based on their idf values. Basically your output will have exactly 50 columns and the number of rows will depend on the number of documents you have in your corpus.</li>\n",
    "    <br>\n",
    "    <li>Here you will be give a pickle file, with file name <strong>cleaned_strings</strong>. You would have to load the corpus from this file and use it as input to your tfidf vectorizer.</li>\n",
    "    <br>\n",
    "    <li>Steps to approach this task:\n",
    "    <ol>\n",
    "        <li> You would have to write both fit and transform methods for your custom implementation of tfidf vectorizer, just like in the previous task. Additionally, here you have to limit the number of features generated to 50 as described above.</li>\n",
    "        <li> Now sort your vocab based in descending order of idf values and print out the words in the sorted voacb after you fit your data. Here you should be getting only 50 terms in your vocab. And make sure to print idf values for each term in your vocab. </li>\n",
    "        <li> Make sure the output of your implementation is a sparse matrix. Before generating the final output, you need to normalize your sparse matrix using L2 normalization. You can refer to this link https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.normalize.html </li>\n",
    "        <li> Now check the output of a single document in your collection of documents,  you can convert the sparse matrix related only to that document into dense matrix and print it. And this dense matrix should contain 1 row and 50 columns. </li>\n",
    "        </ol>\n",
    "    </li>\n",
    "    <br>\n",
    "   </ul>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NHxPLlwNxLjL",
    "outputId": "9abd8e08-0e24-4975-9a13-4d3636d60323"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of documents in corpus2 =  746\n"
     ]
    }
   ],
   "source": [
    "# Below is the code to load the cleaned_strings pickle file provided\n",
    "# Here corpus is of list type\n",
    "\n",
    "import pickle\n",
    "with open('cleaned_strings', 'rb') as f:\n",
    "    corpus2 = pickle.load(f)\n",
    "    \n",
    "# printing the length of the corpus loaded\n",
    "print(\"Number of documents in corpus2 = \",len(corpus2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1_DJnnR3xLjR"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'aailiyah': 0,\n",
       " 'abandoned': 1,\n",
       " 'ability': 2,\n",
       " 'abroad': 3,\n",
       " 'absolutely': 4,\n",
       " 'abstruse': 5,\n",
       " 'abysmal': 6,\n",
       " 'academy': 7,\n",
       " 'accents': 8,\n",
       " 'accessible': 9,\n",
       " 'acclaimed': 10,\n",
       " 'accolades': 11,\n",
       " 'accurate': 12,\n",
       " 'accurately': 13,\n",
       " 'accused': 14,\n",
       " 'achievement': 15,\n",
       " 'achille': 16,\n",
       " 'ackerman': 17,\n",
       " 'act': 18,\n",
       " 'acted': 19,\n",
       " 'acting': 20,\n",
       " 'action': 21,\n",
       " 'actions': 22,\n",
       " 'actor': 23,\n",
       " 'actors': 24,\n",
       " 'actress': 25,\n",
       " 'actresses': 26,\n",
       " 'actually': 27,\n",
       " 'adams': 28,\n",
       " 'adaptation': 29,\n",
       " 'add': 30,\n",
       " 'added': 31,\n",
       " 'addition': 32,\n",
       " 'admins': 33,\n",
       " 'admiration': 34,\n",
       " 'admitted': 35,\n",
       " 'adorable': 36,\n",
       " 'adrift': 37,\n",
       " 'adventure': 38,\n",
       " 'advise': 39,\n",
       " 'aerial': 40,\n",
       " 'aesthetically': 41,\n",
       " 'affected': 42,\n",
       " 'affleck': 43,\n",
       " 'afraid': 44,\n",
       " 'africa': 45,\n",
       " 'afternoon': 46,\n",
       " 'age': 47,\n",
       " 'aged': 48,\n",
       " 'ages': 49,\n",
       " 'ago': 50,\n",
       " 'agree': 51,\n",
       " 'agreed': 52,\n",
       " 'aimless': 53,\n",
       " 'air': 54,\n",
       " 'aired': 55,\n",
       " 'akasha': 56,\n",
       " 'akin': 57,\n",
       " 'alert': 58,\n",
       " 'alexander': 59,\n",
       " 'alike': 60,\n",
       " 'allison': 61,\n",
       " 'allow': 62,\n",
       " 'allowing': 63,\n",
       " 'almost': 64,\n",
       " 'along': 65,\n",
       " 'alongside': 66,\n",
       " 'already': 67,\n",
       " 'also': 68,\n",
       " 'although': 69,\n",
       " 'always': 70,\n",
       " 'amateurish': 71,\n",
       " 'amaze': 72,\n",
       " 'amazed': 73,\n",
       " 'amazing': 74,\n",
       " 'amazingly': 75,\n",
       " 'america': 76,\n",
       " 'american': 77,\n",
       " 'americans': 78,\n",
       " 'among': 79,\n",
       " 'amount': 80,\n",
       " 'amusing': 81,\n",
       " 'amust': 82,\n",
       " 'anatomist': 83,\n",
       " 'angel': 84,\n",
       " 'angela': 85,\n",
       " 'angeles': 86,\n",
       " 'angelina': 87,\n",
       " 'angle': 88,\n",
       " 'angles': 89,\n",
       " 'angry': 90,\n",
       " 'anguish': 91,\n",
       " 'angus': 92,\n",
       " 'animals': 93,\n",
       " 'animated': 94,\n",
       " 'animation': 95,\n",
       " 'anita': 96,\n",
       " 'ann': 97,\n",
       " 'anne': 98,\n",
       " 'anniversary': 99,\n",
       " 'annoying': 100,\n",
       " 'another': 101,\n",
       " 'anthony': 102,\n",
       " 'antithesis': 103,\n",
       " 'anyone': 104,\n",
       " 'anything': 105,\n",
       " 'anyway': 106,\n",
       " 'apart': 107,\n",
       " 'appalling': 108,\n",
       " 'appealing': 109,\n",
       " 'appearance': 110,\n",
       " 'appears': 111,\n",
       " 'applauded': 112,\n",
       " 'applause': 113,\n",
       " 'appreciate': 114,\n",
       " 'appropriate': 115,\n",
       " 'apt': 116,\n",
       " 'argued': 117,\n",
       " 'armageddon': 118,\n",
       " 'armand': 119,\n",
       " 'around': 120,\n",
       " 'array': 121,\n",
       " 'art': 122,\n",
       " 'articulated': 123,\n",
       " 'artiness': 124,\n",
       " 'artist': 125,\n",
       " 'artistic': 126,\n",
       " 'artless': 127,\n",
       " 'arts': 128,\n",
       " 'aside': 129,\n",
       " 'ask': 130,\n",
       " 'asleep': 131,\n",
       " 'aspect': 132,\n",
       " 'aspects': 133,\n",
       " 'ass': 134,\n",
       " 'assante': 135,\n",
       " 'assaulted': 136,\n",
       " 'assistant': 137,\n",
       " 'astonishingly': 138,\n",
       " 'astronaut': 139,\n",
       " 'atmosphere': 140,\n",
       " 'atrocious': 141,\n",
       " 'atrocity': 142,\n",
       " 'attempt': 143,\n",
       " 'attempted': 144,\n",
       " 'attempting': 145,\n",
       " 'attempts': 146,\n",
       " 'attention': 147,\n",
       " 'attractive': 148,\n",
       " 'audience': 149,\n",
       " 'audio': 150,\n",
       " 'aurv': 151,\n",
       " 'austen': 152,\n",
       " 'austere': 153,\n",
       " 'author': 154,\n",
       " 'average': 155,\n",
       " 'aversion': 156,\n",
       " 'avoid': 157,\n",
       " 'avoided': 158,\n",
       " 'award': 159,\n",
       " 'awarded': 160,\n",
       " 'awards': 161,\n",
       " 'away': 162,\n",
       " 'awesome': 163,\n",
       " 'awful': 164,\n",
       " 'awkwardly': 165,\n",
       " 'aye': 166,\n",
       " 'b': 167,\n",
       " 'baaaaaad': 168,\n",
       " 'babbling': 169,\n",
       " 'babie': 170,\n",
       " 'baby': 171,\n",
       " 'babysitting': 172,\n",
       " 'back': 173,\n",
       " 'backdrop': 174,\n",
       " 'backed': 175,\n",
       " 'bad': 176,\n",
       " 'badly': 177,\n",
       " 'bag': 178,\n",
       " 'bailey': 179,\n",
       " 'bakery': 180,\n",
       " 'balance': 181,\n",
       " 'balanced': 182,\n",
       " 'ball': 183,\n",
       " 'ballet': 184,\n",
       " 'balls': 185,\n",
       " 'band': 186,\n",
       " 'barcelona': 187,\n",
       " 'barely': 188,\n",
       " 'barking': 189,\n",
       " 'barney': 190,\n",
       " 'barren': 191,\n",
       " 'based': 192,\n",
       " 'basic': 193,\n",
       " 'basically': 194,\n",
       " 'bat': 195,\n",
       " 'bates': 196,\n",
       " 'baxendale': 197,\n",
       " 'bear': 198,\n",
       " 'beautiful': 199,\n",
       " 'beautifully': 200,\n",
       " 'bec': 201,\n",
       " 'became': 202,\n",
       " 'bechard': 203,\n",
       " 'become': 204,\n",
       " 'becomes': 205,\n",
       " 'began': 206,\n",
       " 'begin': 207,\n",
       " 'beginning': 208,\n",
       " 'behind': 209,\n",
       " 'behold': 210,\n",
       " 'bela': 211,\n",
       " 'believable': 212,\n",
       " 'believe': 213,\n",
       " 'believed': 214,\n",
       " 'bell': 215,\n",
       " 'bellucci': 216,\n",
       " 'belly': 217,\n",
       " 'belmondo': 218,\n",
       " 'ben': 219,\n",
       " 'bendingly': 220,\n",
       " 'bennett': 221,\n",
       " 'bergen': 222,\n",
       " 'bertolucci': 223,\n",
       " 'best': 224,\n",
       " 'better': 225,\n",
       " 'betty': 226,\n",
       " 'beware': 227,\n",
       " 'beyond': 228,\n",
       " 'bible': 229,\n",
       " 'big': 230,\n",
       " 'biggest': 231,\n",
       " 'billy': 232,\n",
       " 'biographical': 233,\n",
       " 'bipolarity': 234,\n",
       " 'bit': 235,\n",
       " 'bitchy': 236,\n",
       " 'black': 237,\n",
       " 'blah': 238,\n",
       " 'blake': 239,\n",
       " 'bland': 240,\n",
       " 'blandly': 241,\n",
       " 'blare': 242,\n",
       " 'blatant': 243,\n",
       " 'blew': 244,\n",
       " 'blood': 245,\n",
       " 'blown': 246,\n",
       " 'blue': 247,\n",
       " 'blush': 248,\n",
       " 'boasts': 249,\n",
       " 'bob': 250,\n",
       " 'body': 251,\n",
       " 'bohemian': 252,\n",
       " 'boiling': 253,\n",
       " 'bold': 254,\n",
       " 'bombardments': 255,\n",
       " 'bond': 256,\n",
       " 'bonding': 257,\n",
       " 'bonus': 258,\n",
       " 'bonuses': 259,\n",
       " 'boobs': 260,\n",
       " 'boogeyman': 261,\n",
       " 'book': 262,\n",
       " 'boost': 263,\n",
       " 'bop': 264,\n",
       " 'bordered': 265,\n",
       " 'borderlines': 266,\n",
       " 'borders': 267,\n",
       " 'bore': 268,\n",
       " 'bored': 269,\n",
       " 'boring': 270,\n",
       " 'borrowed': 271,\n",
       " 'boss': 272,\n",
       " 'bother': 273,\n",
       " 'bothersome': 274,\n",
       " 'bought': 275,\n",
       " 'box': 276,\n",
       " 'boyfriend': 277,\n",
       " 'boyle': 278,\n",
       " 'brain': 279,\n",
       " 'brainsucking': 280,\n",
       " 'brat': 281,\n",
       " 'breaking': 282,\n",
       " 'breeders': 283,\n",
       " 'brevity': 284,\n",
       " 'brian': 285,\n",
       " 'brief': 286,\n",
       " 'brigand': 287,\n",
       " 'bright': 288,\n",
       " 'brilliance': 289,\n",
       " 'brilliant': 290,\n",
       " 'brilliantly': 291,\n",
       " 'bring': 292,\n",
       " 'brings': 293,\n",
       " 'broad': 294,\n",
       " 'broke': 295,\n",
       " 'brooding': 296,\n",
       " 'brother': 297,\n",
       " 'brutal': 298,\n",
       " 'buddy': 299,\n",
       " 'budget': 300,\n",
       " 'buffalo': 301,\n",
       " 'buffet': 302,\n",
       " 'build': 303,\n",
       " 'builders': 304,\n",
       " 'buildings': 305,\n",
       " 'built': 306,\n",
       " 'bullock': 307,\n",
       " 'bully': 308,\n",
       " 'bunch': 309,\n",
       " 'burton': 310,\n",
       " 'business': 311,\n",
       " 'buy': 312,\n",
       " 'cable': 313,\n",
       " 'cailles': 314,\n",
       " 'california': 315,\n",
       " 'call': 316,\n",
       " 'called': 317,\n",
       " 'calls': 318,\n",
       " 'came': 319,\n",
       " 'cameo': 320,\n",
       " 'camera': 321,\n",
       " 'camerawork': 322,\n",
       " 'camp': 323,\n",
       " 'campy': 324,\n",
       " 'canada': 325,\n",
       " 'cancan': 326,\n",
       " 'candace': 327,\n",
       " 'candle': 328,\n",
       " 'cannot': 329,\n",
       " 'cant': 330,\n",
       " 'captain': 331,\n",
       " 'captured': 332,\n",
       " 'captures': 333,\n",
       " 'car': 334,\n",
       " 'card': 335,\n",
       " 'cardboard': 336,\n",
       " 'cardellini': 337,\n",
       " 'care': 338,\n",
       " 'carol': 339,\n",
       " 'carrell': 340,\n",
       " 'carries': 341,\n",
       " 'carry': 342,\n",
       " 'cars': 343,\n",
       " 'cartoon': 344,\n",
       " 'cartoons': 345,\n",
       " 'case': 346,\n",
       " 'cases': 347,\n",
       " 'cast': 348,\n",
       " 'casted': 349,\n",
       " 'casting': 350,\n",
       " 'cat': 351,\n",
       " 'catchy': 352,\n",
       " 'caught': 353,\n",
       " 'cause': 354,\n",
       " 'ceases': 355,\n",
       " 'celebration': 356,\n",
       " 'celebrity': 357,\n",
       " 'celluloid': 358,\n",
       " 'centers': 359,\n",
       " 'central': 360,\n",
       " 'century': 361,\n",
       " 'certain': 362,\n",
       " 'certainly': 363,\n",
       " 'cg': 364,\n",
       " 'cgi': 365,\n",
       " 'chalkboard': 366,\n",
       " 'challenges': 367,\n",
       " 'chance': 368,\n",
       " 'change': 369,\n",
       " 'changes': 370,\n",
       " 'changing': 371,\n",
       " 'channel': 372,\n",
       " 'character': 373,\n",
       " 'characterisation': 374,\n",
       " 'characters': 375,\n",
       " 'charisma': 376,\n",
       " 'charismatic': 377,\n",
       " 'charles': 378,\n",
       " 'charlie': 379,\n",
       " 'charm': 380,\n",
       " 'charming': 381,\n",
       " 'chase': 382,\n",
       " 'chasing': 383,\n",
       " 'cheap': 384,\n",
       " 'cheaply': 385,\n",
       " 'check': 386,\n",
       " 'checking': 387,\n",
       " 'cheek': 388,\n",
       " 'cheekbones': 389,\n",
       " 'cheerfull': 390,\n",
       " 'cheerless': 391,\n",
       " 'cheesiness': 392,\n",
       " 'cheesy': 393,\n",
       " 'chemistry': 394,\n",
       " 'chick': 395,\n",
       " 'child': 396,\n",
       " 'childhood': 397,\n",
       " 'children': 398,\n",
       " 'childrens': 399,\n",
       " 'chills': 400,\n",
       " 'chilly': 401,\n",
       " 'chimp': 402,\n",
       " 'chodorov': 403,\n",
       " 'choice': 404,\n",
       " 'choices': 405,\n",
       " 'choked': 406,\n",
       " 'chosen': 407,\n",
       " 'chow': 408,\n",
       " 'christmas': 409,\n",
       " 'christopher': 410,\n",
       " 'church': 411,\n",
       " 'cinema': 412,\n",
       " 'cinematic': 413,\n",
       " 'cinematographers': 414,\n",
       " 'cinematography': 415,\n",
       " 'circumstances': 416,\n",
       " 'class': 417,\n",
       " 'classic': 418,\n",
       " 'classical': 419,\n",
       " 'clear': 420,\n",
       " 'clearly': 421,\n",
       " 'clever': 422,\n",
       " 'clich': 423,\n",
       " 'cliche': 424,\n",
       " 'clients': 425,\n",
       " 'cliff': 426,\n",
       " 'climax': 427,\n",
       " 'close': 428,\n",
       " 'closed': 429,\n",
       " 'clothes': 430,\n",
       " 'club': 431,\n",
       " 'co': 432,\n",
       " 'coach': 433,\n",
       " 'coal': 434,\n",
       " 'coastal': 435,\n",
       " 'coaster': 436,\n",
       " 'coherent': 437,\n",
       " 'cold': 438,\n",
       " 'cole': 439,\n",
       " 'collect': 440,\n",
       " 'collective': 441,\n",
       " 'colored': 442,\n",
       " 'colorful': 443,\n",
       " 'colours': 444,\n",
       " 'columbo': 445,\n",
       " 'come': 446,\n",
       " 'comedic': 447,\n",
       " 'comedy': 448,\n",
       " 'comes': 449,\n",
       " 'comfortable': 450,\n",
       " 'comforting': 451,\n",
       " 'comical': 452,\n",
       " 'coming': 453,\n",
       " 'commands': 454,\n",
       " 'comment': 455,\n",
       " 'commentary': 456,\n",
       " 'commented': 457,\n",
       " 'comments': 458,\n",
       " 'commercial': 459,\n",
       " 'community': 460,\n",
       " 'company': 461,\n",
       " 'compelling': 462,\n",
       " 'competent': 463,\n",
       " 'complete': 464,\n",
       " 'completed': 465,\n",
       " 'completely': 466,\n",
       " 'complex': 467,\n",
       " 'complexity': 468,\n",
       " 'composed': 469,\n",
       " 'composition': 470,\n",
       " 'comprehensible': 471,\n",
       " 'compromise': 472,\n",
       " 'computer': 473,\n",
       " 'concentrate': 474,\n",
       " 'conception': 475,\n",
       " 'conceptually': 476,\n",
       " 'concerning': 477,\n",
       " 'concerns': 478,\n",
       " 'concert': 479,\n",
       " 'conclusion': 480,\n",
       " 'condescends': 481,\n",
       " 'confidence': 482,\n",
       " 'configuration': 483,\n",
       " 'confirm': 484,\n",
       " 'conflict': 485,\n",
       " 'confuses': 486,\n",
       " 'confusing': 487,\n",
       " 'connections': 488,\n",
       " 'connery': 489,\n",
       " 'connor': 490,\n",
       " 'conrad': 491,\n",
       " 'consequences': 492,\n",
       " 'consider': 493,\n",
       " 'considerable': 494,\n",
       " 'considered': 495,\n",
       " 'considering': 496,\n",
       " 'considers': 497,\n",
       " 'consistent': 498,\n",
       " 'consolations': 499,\n",
       " 'constant': 500,\n",
       " 'constantine': 501,\n",
       " 'constructed': 502,\n",
       " 'contained': 503,\n",
       " 'containing': 504,\n",
       " 'contains': 505,\n",
       " 'content': 506,\n",
       " 'continually': 507,\n",
       " 'continuation': 508,\n",
       " 'continue': 509,\n",
       " 'continuity': 510,\n",
       " 'continuously': 511,\n",
       " 'contract': 512,\n",
       " 'contrast': 513,\n",
       " 'contributing': 514,\n",
       " 'contributory': 515,\n",
       " 'contrived': 516,\n",
       " 'control': 517,\n",
       " 'controversy': 518,\n",
       " 'convention': 519,\n",
       " 'convey': 520,\n",
       " 'convince': 521,\n",
       " 'convincing': 522,\n",
       " 'convoluted': 523,\n",
       " 'cool': 524,\n",
       " 'coppola': 525,\n",
       " 'cords': 526,\n",
       " 'core': 527,\n",
       " 'corn': 528,\n",
       " 'corny': 529,\n",
       " 'correct': 530,\n",
       " 'cost': 531,\n",
       " 'costs': 532,\n",
       " 'costumes': 533,\n",
       " 'cotton': 534,\n",
       " 'could': 535,\n",
       " 'couple': 536,\n",
       " 'course': 537,\n",
       " 'court': 538,\n",
       " 'courtroom': 539,\n",
       " 'cover': 540,\n",
       " 'cowardice': 541,\n",
       " 'cox': 542,\n",
       " 'crackles': 543,\n",
       " 'crafted': 544,\n",
       " 'crap': 545,\n",
       " 'crash': 546,\n",
       " 'crashed': 547,\n",
       " 'crayon': 548,\n",
       " 'crayons': 549,\n",
       " 'crazy': 550,\n",
       " 'create': 551,\n",
       " 'created': 552,\n",
       " 'creates': 553,\n",
       " 'creative': 554,\n",
       " 'creativity': 555,\n",
       " 'creature': 556,\n",
       " 'credible': 557,\n",
       " 'credit': 558,\n",
       " 'credits': 559,\n",
       " 'crew': 560,\n",
       " 'crime': 561,\n",
       " 'crisp': 562,\n",
       " 'critic': 563,\n",
       " 'critical': 564,\n",
       " 'crocdodile': 565,\n",
       " 'crocs': 566,\n",
       " 'cross': 567,\n",
       " 'crowd': 568,\n",
       " 'crowe': 569,\n",
       " 'cruel': 570,\n",
       " 'cruise': 571,\n",
       " 'cry': 572,\n",
       " 'cult': 573,\n",
       " 'culture': 574,\n",
       " 'curtain': 575,\n",
       " 'custer': 576,\n",
       " 'cute': 577,\n",
       " 'cutest': 578,\n",
       " 'cutie': 579,\n",
       " 'cutouts': 580,\n",
       " 'cuts': 581,\n",
       " 'cutting': 582,\n",
       " 'dads': 583,\n",
       " 'damian': 584,\n",
       " 'damn': 585,\n",
       " 'dance': 586,\n",
       " 'dancing': 587,\n",
       " 'dangerous': 588,\n",
       " 'dark': 589,\n",
       " 'darren': 590,\n",
       " 'daughter': 591,\n",
       " 'daughters': 592,\n",
       " 'day': 593,\n",
       " 'days': 594,\n",
       " 'de': 595,\n",
       " 'dead': 596,\n",
       " 'deadly': 597,\n",
       " 'deadpan': 598,\n",
       " 'deal': 599,\n",
       " 'dealt': 600,\n",
       " 'death': 601,\n",
       " 'debated': 602,\n",
       " 'debbie': 603,\n",
       " 'debits': 604,\n",
       " 'debut': 605,\n",
       " 'decay': 606,\n",
       " 'decent': 607,\n",
       " 'decidely': 608,\n",
       " 'decipher': 609,\n",
       " 'decisions': 610,\n",
       " 'dedication': 611,\n",
       " 'dee': 612,\n",
       " 'deep': 613,\n",
       " 'deeply': 614,\n",
       " 'defensemen': 615,\n",
       " 'defined': 616,\n",
       " 'definitely': 617,\n",
       " 'delete': 618,\n",
       " 'delight': 619,\n",
       " 'delightful': 620,\n",
       " 'delights': 621,\n",
       " 'deliver': 622,\n",
       " 'delivered': 623,\n",
       " 'delivering': 624,\n",
       " 'delivers': 625,\n",
       " 'dependant': 626,\n",
       " 'depending': 627,\n",
       " 'depends': 628,\n",
       " 'depicted': 629,\n",
       " 'depicts': 630,\n",
       " 'depressing': 631,\n",
       " 'depth': 632,\n",
       " 'derivative': 633,\n",
       " 'describe': 634,\n",
       " 'describes': 635,\n",
       " 'desert': 636,\n",
       " 'deserved': 637,\n",
       " 'deserves': 638,\n",
       " 'deserving': 639,\n",
       " 'design': 640,\n",
       " 'designed': 641,\n",
       " 'designer': 642,\n",
       " 'desperately': 643,\n",
       " 'desperation': 644,\n",
       " 'despised': 645,\n",
       " 'despite': 646,\n",
       " 'destroy': 647,\n",
       " 'detailing': 648,\n",
       " 'details': 649,\n",
       " 'develop': 650,\n",
       " 'development': 651,\n",
       " 'developments': 652,\n",
       " 'di': 653,\n",
       " 'diabetic': 654,\n",
       " 'dialog': 655,\n",
       " 'dialogs': 656,\n",
       " 'dialogue': 657,\n",
       " 'diaper': 658,\n",
       " 'dickens': 659,\n",
       " 'difference': 660,\n",
       " 'different': 661,\n",
       " 'dignity': 662,\n",
       " 'dimensional': 663,\n",
       " 'direct': 664,\n",
       " 'directed': 665,\n",
       " 'directing': 666,\n",
       " 'direction': 667,\n",
       " 'director': 668,\n",
       " 'directorial': 669,\n",
       " 'directors': 670,\n",
       " 'disappointed': 671,\n",
       " 'disappointing': 672,\n",
       " 'disappointment': 673,\n",
       " 'disaster': 674,\n",
       " 'disbelief': 675,\n",
       " 'discomfort': 676,\n",
       " 'discovering': 677,\n",
       " 'discovery': 678,\n",
       " 'disgrace': 679,\n",
       " 'disgusting': 680,\n",
       " 'dislike': 681,\n",
       " 'disliked': 682,\n",
       " 'disney': 683,\n",
       " 'disparate': 684,\n",
       " 'distant': 685,\n",
       " 'distinction': 686,\n",
       " 'distorted': 687,\n",
       " 'distract': 688,\n",
       " 'distressed': 689,\n",
       " 'disturbing': 690,\n",
       " 'diving': 691,\n",
       " 'doctor': 692,\n",
       " 'documentaries': 693,\n",
       " 'documentary': 694,\n",
       " 'dodge': 695,\n",
       " 'dogs': 696,\n",
       " 'dollars': 697,\n",
       " 'dominated': 698,\n",
       " 'done': 699,\n",
       " 'donlevy': 700,\n",
       " 'dont': 701,\n",
       " 'doomed': 702,\n",
       " 'dose': 703,\n",
       " 'doubt': 704,\n",
       " 'downs': 705,\n",
       " 'dozen': 706,\n",
       " 'dr': 707,\n",
       " 'dracula': 708,\n",
       " 'draft': 709,\n",
       " 'drag': 710,\n",
       " 'drago': 711,\n",
       " 'drama': 712,\n",
       " 'dramatic': 713,\n",
       " 'drawings': 714,\n",
       " 'drawn': 715,\n",
       " 'dream': 716,\n",
       " 'dreams': 717,\n",
       " 'dreary': 718,\n",
       " 'dribble': 719,\n",
       " 'drift': 720,\n",
       " 'drifting': 721,\n",
       " 'drive': 722,\n",
       " 'drooling': 723,\n",
       " 'dropped': 724,\n",
       " 'dry': 725,\n",
       " 'due': 726,\n",
       " 'duet': 727,\n",
       " 'dull': 728,\n",
       " 'dumb': 729,\n",
       " 'dumbest': 730,\n",
       " 'duper': 731,\n",
       " 'duris': 732,\n",
       " 'dustin': 733,\n",
       " 'dvd': 734,\n",
       " 'dwight': 735,\n",
       " 'dysfunction': 736,\n",
       " 'e': 737,\n",
       " 'earlier': 738,\n",
       " 'early': 739,\n",
       " 'earth': 740,\n",
       " 'easily': 741,\n",
       " 'easy': 742,\n",
       " 'eating': 743,\n",
       " 'ebay': 744,\n",
       " 'ebola': 745,\n",
       " 'eccleston': 746,\n",
       " 'ed': 747,\n",
       " 'edge': 748,\n",
       " 'editing': 749,\n",
       " 'edition': 750,\n",
       " 'educational': 751,\n",
       " 'edward': 752,\n",
       " 'effect': 753,\n",
       " 'effective': 754,\n",
       " 'effects': 755,\n",
       " 'effort': 756,\n",
       " 'efforts': 757,\n",
       " 'egotism': 758,\n",
       " 'eighth': 759,\n",
       " 'eiko': 760,\n",
       " 'either': 761,\n",
       " 'elaborately': 762,\n",
       " 'elderly': 763,\n",
       " 'elegant': 764,\n",
       " 'element': 765,\n",
       " 'elias': 766,\n",
       " 'eloquently': 767,\n",
       " 'else': 768,\n",
       " 'elsewhere': 769,\n",
       " 'embarrassed': 770,\n",
       " 'embarrassing': 771,\n",
       " 'embassy': 772,\n",
       " 'emerge': 773,\n",
       " 'emilio': 774,\n",
       " 'emily': 775,\n",
       " 'emoting': 776,\n",
       " 'emotion': 777,\n",
       " 'emotionally': 778,\n",
       " 'emotions': 779,\n",
       " 'emperor': 780,\n",
       " 'empowerment': 781,\n",
       " 'emptiness': 782,\n",
       " 'empty': 783,\n",
       " 'en': 784,\n",
       " 'enchanting': 785,\n",
       " 'end': 786,\n",
       " 'endearing': 787,\n",
       " 'ended': 788,\n",
       " 'ending': 789,\n",
       " 'endlessly': 790,\n",
       " 'ends': 791,\n",
       " 'energetic': 792,\n",
       " 'energy': 793,\n",
       " 'engaging': 794,\n",
       " 'english': 795,\n",
       " 'enhanced': 796,\n",
       " 'enjoy': 797,\n",
       " 'enjoyable': 798,\n",
       " 'enjoyed': 799,\n",
       " 'enjoyment': 800,\n",
       " 'enough': 801,\n",
       " 'enter': 802,\n",
       " 'enterprise': 803,\n",
       " 'entertained': 804,\n",
       " 'entertaining': 805,\n",
       " 'entire': 806,\n",
       " 'entirely': 807,\n",
       " 'entrance': 808,\n",
       " 'episode': 809,\n",
       " 'episodes': 810,\n",
       " 'equivalent': 811,\n",
       " 'era': 812,\n",
       " 'errol': 813,\n",
       " 'errors': 814,\n",
       " 'escalating': 815,\n",
       " 'escapism': 816,\n",
       " 'especially': 817,\n",
       " 'essence': 818,\n",
       " 'establish': 819,\n",
       " 'established': 820,\n",
       " 'estate': 821,\n",
       " 'estevez': 822,\n",
       " 'etc': 823,\n",
       " 'european': 824,\n",
       " 'evaluate': 825,\n",
       " 'even': 826,\n",
       " 'events': 827,\n",
       " 'ever': 828,\n",
       " 'every': 829,\n",
       " 'everybody': 830,\n",
       " 'everyone': 831,\n",
       " 'everything': 832,\n",
       " 'everywhere': 833,\n",
       " 'evidently': 834,\n",
       " 'evil': 835,\n",
       " 'evinced': 836,\n",
       " 'evokes': 837,\n",
       " 'exactly': 838,\n",
       " 'exaggerating': 839,\n",
       " 'example': 840,\n",
       " 'excellent': 841,\n",
       " 'excellently': 842,\n",
       " 'except': 843,\n",
       " 'exceptional': 844,\n",
       " 'exceptionally': 845,\n",
       " 'excerpts': 846,\n",
       " 'excessively': 847,\n",
       " 'exchange': 848,\n",
       " 'exciting': 849,\n",
       " 'excruciatingly': 850,\n",
       " 'excuse': 851,\n",
       " 'excuses': 852,\n",
       " 'executed': 853,\n",
       " 'exemplars': 854,\n",
       " 'existent': 855,\n",
       " 'existential': 856,\n",
       " 'expansive': 857,\n",
       " 'expect': 858,\n",
       " 'expectations': 859,\n",
       " 'expected': 860,\n",
       " 'expecting': 861,\n",
       " 'experience': 862,\n",
       " 'experiences': 863,\n",
       " 'expert': 864,\n",
       " 'explain': 865,\n",
       " 'explains': 866,\n",
       " 'explanation': 867,\n",
       " 'exploit': 868,\n",
       " 'explorations': 869,\n",
       " 'explosion': 870,\n",
       " 'expression': 871,\n",
       " 'exquisite': 872,\n",
       " 'extant': 873,\n",
       " 'exteriors': 874,\n",
       " 'extraneous': 875,\n",
       " 'extraordinary': 876,\n",
       " 'extremely': 877,\n",
       " 'eye': 878,\n",
       " 'eyes': 879,\n",
       " 'f': 880,\n",
       " 'fabulous': 881,\n",
       " 'face': 882,\n",
       " 'faces': 883,\n",
       " 'facial': 884,\n",
       " 'facing': 885,\n",
       " 'fact': 886,\n",
       " 'factory': 887,\n",
       " 'failed': 888,\n",
       " 'fails': 889,\n",
       " 'fair': 890,\n",
       " 'fairly': 891,\n",
       " 'faithful': 892,\n",
       " 'fall': 893,\n",
       " 'falling': 894,\n",
       " 'falls': 895,\n",
       " 'falsely': 896,\n",
       " 'falwell': 897,\n",
       " 'fame': 898,\n",
       " 'famed': 899,\n",
       " 'family': 900,\n",
       " 'famous': 901,\n",
       " 'fan': 902,\n",
       " 'fanciful': 903,\n",
       " 'fans': 904,\n",
       " 'fantastic': 905,\n",
       " 'fantasy': 906,\n",
       " 'far': 907,\n",
       " 'farce': 908,\n",
       " 'fare': 909,\n",
       " 'fascinated': 910,\n",
       " 'fascinating': 911,\n",
       " 'fascination': 912,\n",
       " 'fashioned': 913,\n",
       " 'fast': 914,\n",
       " 'faster': 915,\n",
       " 'fat': 916,\n",
       " 'father': 917,\n",
       " 'faultless': 918,\n",
       " 'fausa': 919,\n",
       " 'faux': 920,\n",
       " 'favorite': 921,\n",
       " 'favourite': 922,\n",
       " 'fear': 923,\n",
       " 'feature': 924,\n",
       " 'features': 925,\n",
       " 'feel': 926,\n",
       " 'feeling': 927,\n",
       " 'feelings': 928,\n",
       " 'feet': 929,\n",
       " 'feisty': 930,\n",
       " 'fellowes': 931,\n",
       " 'felt': 932,\n",
       " 'female': 933,\n",
       " 'females': 934,\n",
       " 'ferry': 935,\n",
       " 'fest': 936,\n",
       " 'fi': 937,\n",
       " 'fields': 938,\n",
       " 'fifteen': 939,\n",
       " 'fifties': 940,\n",
       " 'fill': 941,\n",
       " 'film': 942,\n",
       " 'filmed': 943,\n",
       " 'filmiing': 944,\n",
       " 'filmmaker': 945,\n",
       " 'filmography': 946,\n",
       " 'films': 947,\n",
       " 'final': 948,\n",
       " 'finale': 949,\n",
       " 'finally': 950,\n",
       " 'financial': 951,\n",
       " 'find': 952,\n",
       " 'finds': 953,\n",
       " 'fine': 954,\n",
       " 'finest': 955,\n",
       " 'fingernails': 956,\n",
       " 'finished': 957,\n",
       " 'fire': 958,\n",
       " 'first': 959,\n",
       " 'fish': 960,\n",
       " 'fishnet': 961,\n",
       " 'fisted': 962,\n",
       " 'fit': 963,\n",
       " 'five': 964,\n",
       " 'flag': 965,\n",
       " 'flakes': 966,\n",
       " 'flaming': 967,\n",
       " 'flashbacks': 968,\n",
       " 'flat': 969,\n",
       " 'flaw': 970,\n",
       " 'flawed': 971,\n",
       " 'flaws': 972,\n",
       " 'fleshed': 973,\n",
       " 'flick': 974,\n",
       " 'flicks': 975,\n",
       " 'florida': 976,\n",
       " 'flowed': 977,\n",
       " 'flying': 978,\n",
       " 'flynn': 979,\n",
       " 'focus': 980,\n",
       " 'fodder': 981,\n",
       " 'follow': 982,\n",
       " 'following': 983,\n",
       " 'follows': 984,\n",
       " 'foolish': 985,\n",
       " 'footage': 986,\n",
       " 'football': 987,\n",
       " 'force': 988,\n",
       " 'forced': 989,\n",
       " 'forces': 990,\n",
       " 'ford': 991,\n",
       " 'foreign': 992,\n",
       " 'foreigner': 993,\n",
       " 'forever': 994,\n",
       " 'forget': 995,\n",
       " 'forgettable': 996,\n",
       " 'forgetting': 997,\n",
       " 'forgot': 998,\n",
       " 'forgotten': 999,\n",
       " ...}"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Building vocabulory for task2\n",
    "def fit2(corpus2):\n",
    "    distinct_word=set()\n",
    "    if isinstance(corpus2,list):\n",
    "        for row in corpus2:\n",
    "            for word in row.split(\" \"):\n",
    "                distinct_word.add(word)\n",
    "        distinct_word=sorted(distinct_word) \n",
    "        vocab2={j:i for i,j in enumerate(distinct_word)}\n",
    "    \n",
    "                \n",
    "    else:\n",
    "        print(\"Please provide a corpus of type list\")\n",
    "        \n",
    "    return vocab2\n",
    "        \n",
    "fit2(corpus2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 746/746 [00:00<00:00, 82772.10it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IDF values for top 50 terms \n",
      " {'aailiyah': 6.922918004572872, 'abandoned': 6.922918004572872, 'abroad': 6.922918004572872, 'abstruse': 6.922918004572872, 'academy': 6.922918004572872, 'accents': 6.922918004572872, 'accessible': 6.922918004572872, 'acclaimed': 6.922918004572872, 'accolades': 6.922918004572872, 'accurate': 6.922918004572872, 'accurately': 6.922918004572872, 'achille': 6.922918004572872, 'ackerman': 6.922918004572872, 'actions': 6.922918004572872, 'adams': 6.922918004572872, 'add': 6.922918004572872, 'added': 6.922918004572872, 'admins': 6.922918004572872, 'admiration': 6.922918004572872, 'admitted': 6.922918004572872, 'adrift': 6.922918004572872, 'adventure': 6.922918004572872, 'aesthetically': 6.922918004572872, 'affected': 6.922918004572872, 'affleck': 6.922918004572872, 'afternoon': 6.922918004572872, 'aged': 6.922918004572872, 'ages': 6.922918004572872, 'agree': 6.922918004572872, 'agreed': 6.922918004572872, 'aimless': 6.922918004572872, 'aired': 6.922918004572872, 'akasha': 6.922918004572872, 'akin': 6.922918004572872, 'alert': 6.922918004572872, 'alike': 6.922918004572872, 'allison': 6.922918004572872, 'allow': 6.922918004572872, 'allowing': 6.922918004572872, 'alongside': 6.922918004572872, 'amateurish': 6.922918004572872, 'amaze': 6.922918004572872, 'amazed': 6.922918004572872, 'amazingly': 6.922918004572872, 'amusing': 6.922918004572872, 'amust': 6.922918004572872, 'anatomist': 6.922918004572872, 'angel': 6.922918004572872, 'angela': 6.922918004572872, 'angelina': 6.922918004572872}\n",
      "====================================================================================================\n",
      "Shape of sparse matrix \n",
      " (746, 50)\n",
      "====================================================================================================\n",
      "Normalized tfidf  sparse matrix for 0th row \n",
      "   (0, 30)\t1.0\n",
      "====================================================================================================\n",
      "Normalized tfidf  dense matrix for 0th row \n",
      " [[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      "  0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      "  0. 0.]]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "#selecting the top 50 vocab values on the basis of their idfs and calculating the sparse matrix\n",
    "\n",
    "def transform2 (corpus2,vocab2):\n",
    "    idf_term=dict()\n",
    "    temp=[]\n",
    "    rows_2 = []\n",
    "    columns_2 = []\n",
    "    values_2 = []\n",
    "    if isinstance (corpus2,list):\n",
    "        for key,i in vocab2.items():\n",
    "            count=0\n",
    "            for row in corpus2:\n",
    "                if key in row.split(\" \"):\n",
    "                    count+=1\n",
    "            idf_term[key]=(1+(math.log((1+len(corpus2))/(1+count))))  #cal the idf for terms in corpus\n",
    "            j={k: v for k, v in sorted(idf_term.items(), key=lambda item: item[1],reverse=True)} #sorting idfs in desc order\n",
    "            top50_idf_term=dict(list(j.items())[0:50]) #slicing the top 50 idfs and the corresponding terms\n",
    "        \n",
    "        print(\"IDF values for top 50 terms \\n\",top50_idf_term)\n",
    "        print(\"=\"*100)\n",
    "        for k1,v1 in top50_idf_term.items():\n",
    "            temp.append(k1)\n",
    "            \n",
    "        top50_vocab={j:i for i,j in enumerate(temp)} #building the new vocab for top 50 terms with highest idf values\n",
    "        \n",
    "        \n",
    "    \n",
    "        for row_ind,row in enumerate(tqdm(corpus2)):\n",
    "            word_frequency=dict(Counter(row.split(\" \")))\n",
    "            for word,freq in word_frequency.items():\n",
    "                tf_term=(freq/len(row)) #calculating the tf for terms in docs\n",
    "                col_index=top50_vocab.get(word,-1)\n",
    "                if col_index!=-1:\n",
    "                    columns_2.append(col_index) \n",
    "                    rows_2.append(row_ind)\n",
    "                    values_2.append(tf_term*top50_idf_term[word])\n",
    "        \n",
    "                       \n",
    "    else:\n",
    "        print(\"Please provide the corpus of type list\")\n",
    "        \n",
    "\n",
    "    return csr_matrix((values_2, (rows_2,columns_2)), shape=(len(corpus2),len(top50_vocab))) \n",
    "\n",
    "\n",
    "        \n",
    "vocab2=fit2(corpus2)       \n",
    "tf_idf_sparse_50=transform2(corpus2,vocab2)\n",
    "\n",
    "\n",
    "print(\"Shape of sparse matrix \\n\",tf_idf_sparse_50.shape)\n",
    "print(\"=\"*100)\n",
    "\n",
    "normalized_tfidfmatrix_50=normalize(tf_idf_sparse_50,norm='l2') #normalizing the matrix\n",
    "\n",
    "\n",
    "print(\"Normalized tfidf  sparse matrix for 0th row \\n\",normalized_tfidfmatrix_50[0])\n",
    "print(\"=\"*100)\n",
    "print(\"Normalized tfidf  dense matrix for 0th row \\n\",normalized_tfidfmatrix_50[0].toarray())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Assignment_3_Instructions.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}