BagOfWords.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bag of Words\n",
    "\n",
    "This notebook will review the process of transforming a collection of categorized text documents into a \"bag of words\" and \"word vectors\" - data structures that can be used to train a machine learning algorithm to assign text to categories.     \n",
    "\n",
    "We'll be cleaning, parsing, and counting the words in a text file, so we'll want to use regular expressions and the \"count vectorizer\" library from scikit-learn.  To deal with the tab delimited file and numerical operations on what can become large arrays and matrices, we'll also import pandas and numpy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll break down a series of reviews into a list of their N most frequent terms, also called a \"bag of words\".  We'll start with the trivial case of two records and two categories, a good movie review and a bad movie review.\n",
    "\n",
    "This file contains three column headers, a record ID, a \"category\", and some unstructured text.  The category, marked 0 or 1, indicates whether a review was negative (0) or positive (1).  \n",
    "\n",
    "Note that we're loading this into a pandas dataframe.  There's only one record in the sampleGood.tsv file but we'll benefit from pandas operations when we generalize this to include a large number of files in a training set.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "good_review = pd.read_csv('data/samplePositive.tsv', header=0, delimiter=\"\\t\", quoting=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's the text for a fictitious positive movie review.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\"Wow what a good movie.  Absolutely excellent, so good.  I loved it.  Music, dancing, action, intrigue, really good.  I\\'d highly recommend this excellent film.\"'"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "good_review['text'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here's a clearly negative movie review.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "bad_review = pd.read_csv('data/sampleNegative.tsv', header=0, delimiter=\"\\t\", quoting=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\"What a terrible, terrible film.  Truly bad.  In fact, I might call it awful.  I might have to call it awful twice.  The music score was dreary, the acting was contrived, the plot was not believable or convincing.  Avoid. Terrible.\"  '"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bad_review['text'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "We now have two categories, \"good\" and \"bad\", and two reviews, one for each category. \n",
    "\n",
    "### Creating a \"Bag of Words\"\n",
    "\n",
    "With this minimal set of two reviews, one representing each category, we can now create a \"bag of words\", a list of the most frequent terms that show up in all movie reviews . \n",
    "\n",
    "To generate this \"bag of words\", we'll first generate a new list that holds all the reviews in the set (in this case, one positive review, one negative review).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "all_reviews = []\n",
    "all_reviews.append(good_review['text'][0])\n",
    "all_reviews.append(bad_review['text'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['\"Wow what a good movie.  Absolutely excellent, so good.  I loved it.  Music, dancing, action, intrigue, really good.  I\\'d highly recommend this excellent film.\"', '\"What a terrible, terrible film.  Truly bad.  In fact, I might call it awful.  I might have to call it awful twice.  The music score was dreary, the acting was contrived, the plot was not believable or convincing.  Avoid. Terrible.\"  ']\n"
     ]
    }
   ],
   "source": [
    "print(all_reviews)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to break this list into its most common terms.  \n",
    "\n",
    "Python's scikit-learn library has a method, CountVectorizer, for this task.  It accepts a list of strings (in our case, movie reviews), and returns a list of the N most common terms.  If no number is supplied, CountVectorizer will simply return all the words that appear in the reviews, which is fine for now since our data set is very small."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vectorizer = CountVectorizer(analyzer = \"word\")\n",
    "bag_of_words = vectorizer.fit(all_reviews)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "And voila, We now have a \"bag of words\" for our movie reviews!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['absolutely', 'acting', 'action', 'avoid', 'awful', 'bad', 'believable', 'call', 'contrived', 'convincing', 'dancing', 'dreary', 'excellent', 'fact', 'film', 'good', 'have', 'highly', 'in', 'intrigue', 'it', 'loved', 'might', 'movie', 'music', 'not', 'or', 'plot', 'really', 'recommend', 'score', 'so', 'terrible', 'the', 'this', 'to', 'truly', 'twice', 'was', 'what', 'wow']\n"
     ]
    }
   ],
   "source": [
    "print(vectorizer.get_feature_names())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the vectorizer provides feature names in alphabetical order.  To get the numerical position of each term, you can use the vocabulary_ property.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'wow': 40, 'what': 39, 'good': 15, 'movie': 23, 'absolutely': 0, 'excellent': 12, 'so': 31, 'loved': 21, 'it': 20, 'music': 24, 'dancing': 10, 'action': 2, 'intrigue': 19, 'really': 28, 'highly': 17, 'recommend': 29, 'this': 34, 'film': 14, 'terrible': 32, 'truly': 36, 'bad': 5, 'in': 18, 'fact': 13, 'might': 22, 'call': 7, 'awful': 4, 'have': 16, 'to': 35, 'twice': 37, 'the': 33, 'score': 30, 'was': 38, 'dreary': 11, 'acting': 1, 'contrived': 8, 'plot': 27, 'not': 25, 'believable': 6, 'or': 26, 'convincing': 9, 'avoid': 3}\n"
     ]
    }
   ],
   "source": [
    "print(vectorizer.vocabulary_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a \"Word Vector\" \n",
    "\n",
    "Now that we have a bag of words, we can calculate the number of times these words appear in each review.  The resulting data structure is often called a \"word vector\", an index of the frequency at which each word appears.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we can also supply \"stop words\".  These are frequently occurring terms that often have no meaning and can clutter up an algorithm (note - in real world applications, you may discover that things you thought were devoid of meaning actually make a difference in context!)  \n",
    "\n",
    "The scikit-learn library provides methods to remove stop words from a bag of words.  Here, we use the stop_words parameter to remove the common english stop words (\"a, all, and, also...\").  We'll also limit the number of terms to the most frequent ten words through the max_features parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['awful', 'excellent', 'film', 'good', 'music', 'really', 'recommend', 'score', 'terrible', 'truly']\n"
     ]
    }
   ],
   "source": [
    "vectorizer = CountVectorizer(analyzer = \"word\", stop_words = 'english', max_features = 10)\n",
    "bag_of_words = vectorizer.fit(all_reviews)\n",
    "print(vectorizer.get_feature_names())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a list of the most common words in all reviews, we count the frequency with which these words appear in each review.  Each review can be decomposed into a word count vector, a list of how often each word in the most frequent terms appears in a particular review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "good_review_vector = vectorizer.transform([good_review['text'][0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is an array with a word count corresponding to each term in the bag of words.  As you can see below, in our \"good\" review, the word \"awful\" (first index) doesn't appear, whereas \"excellent\" (second index) shows up twice.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 2, 0, 1, 3, 0, 1, 0, 1, 1, 1,\n",
       "        0, 1, 1, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1]])"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "good_review_vector.toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, we can take a look at the word count for a bad review"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bad_review_vector = vectorizer.transform([bad_review['text'][0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, awful shows up twice, and excellent is missing.  Note that some of the terms, such as film or music, show up once in both the good and bad review.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 1, 0, 1, 2, 1, 1, 2, 1, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0, 2, 0,\n",
       "        2, 0, 1, 1, 1, 1, 0, 0, 1, 0, 3, 3, 0, 1, 1, 1, 3, 1, 0]])"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bad_review_vector.toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It can be helpful to look at the three side-by-side.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['absolutely', 'acting', 'action', 'avoid', 'awful', 'bad', 'believable', 'call', 'contrived', 'convincing', 'dancing', 'dreary', 'excellent', 'fact', 'film', 'good', 'have', 'highly', 'in', 'intrigue', 'it', 'loved', 'might', 'movie', 'music', 'not', 'or', 'plot', 'really', 'recommend', 'score', 'so', 'terrible', 'the', 'this', 'to', 'truly', 'twice', 'was', 'what', 'wow']\n",
      "[[1 0 1 0 0 0 0 0 0 0 1 0 2 0 1 3 0 1 0 1 1 1 0 1 1 0 0 0 1 1 0 1 0 0 1 0 0\n",
      "  0 0 1 1]\n",
      " [0 1 0 1 2 1 1 2 1 1 0 1 0 1 1 0 1 0 1 0 2 0 2 0 1 1 1 1 0 0 1 0 3 3 0 1 1\n",
      "  1 3 1 0]]\n"
     ]
    }
   ],
   "source": [
    "all_review_vector = vectorizer.transform(all_reviews)\n",
    "print(vectorizer.get_feature_names())\n",
    "print(all_review_vector.toarray())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How does this relate to Machine Learning?\n",
    "\n",
    "So far, this has all been data carpentry.  We find the most common words in a set of documents, and we create a word vector to represent each individual document in the set.\n",
    "\n",
    "Now that we have this data, we can use it to train a computer to recognize positive and negative reviews based on patterns it finds in the word vectors associated with positive and negative reviews.  That is the essence of supervised machine learning.  We have a set of records assigned to a pre-defined set of categories, and we use it to train a computer to find a way to categorize records into those categories.  \n",
    "\n",
    "Although we haven't discussed creating a method to categorize reviews into \"positive\" and \"negative\", you may already be thinking of some strategies based on what you've seen here.  By creating a list, or \"bag of words\" of common terms, and decomposing each review into a word count vector, we can create a signature for each review.  We can then associate these vectors with each category and come up with some kind of rule for matching a word count with a positive or negative review.  \n",
    "\n",
    "Before continuing, you might try designing and implementing an approach yourself.  You have a positive and a negative review, and a single word vector for each.  How could you use this to program a computer to predict whether a new review is positive or negative?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Real Data\n",
    "\n",
    "We'll use a genuine, real world data set of positive and negative movie reviews, available at http://www.cs.cornell.edu/home/llee/papers/sentiment.home.html\n",
    "\n",
    "Keep in mind, even this has been pre-precessed a bit for us.  Data carpentry is a huge part of data science, and getting data into a form where you can use it plays a huge role in an application of machine learning.  Finding, cleaning, munging, formatting, and preparing data isn't just a technical task.  You almost always have to make some decisions about what to keep, what to discard, and how to prepare it, and these decisions often introduce patterns and assumptions that influence the outcome when you apply a ML technique.   \n",
    "\n",
    "Even in case of well managed and prepared data, as we have with the sentient data from Cornell, I did have to do some formatting to get the data set into a tab delimited format that can be easily imported into pandas.  The scripts to do this are located in this repository in rawdata/review_polarity/text_sentiment/createfile.py (another script to randomize the order of the reviews is available in shufflefile.py).  These files aren't especially elaborate, but it does go to show, you'll most likely have to do some processing filtering even under the very best circumstances (and most of your raw data won't be anywhere near as well presented as the data download for the sentient data here).  Also, this data set is small, only 500 positive and negative movie reviews.  Cleaning and parsing a very large data set is an entirely different challenge!\n",
    "\n",
    "With that said, let's create a bag of words and word vector for the sentiment data set.  \n",
    "\n",
    "### Bag of Words, revisited...\n",
    "\n",
    "Although our example above was limited to two reviews and two categories, we'll the same approach at a larger scale. \n",
    "\n",
    "The file trainRecords.csv is a tab delimited file with has 500 reviews, split into positive and negative reviews.  The tabs correspond to our sample files above - the first column is an identifier, the second is a category (0 for negative, 1 for positive), and the third contains the text of the review.  We'll use this data, long with the vectorizer method from scikit-learn, to build a bag of words, with a maximum of 5000 terms.  \n",
    "\n",
    "First, we'll read in all the reviews from a tab delimied file.  Each row contains an ID, a category (1 represents a positive review, 0 a negative one), and the text of the movie review. \n",
    "\n",
    "First, let's load the file into a pandas dataframe, as before.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train = pd.read_csv('data/trainReviews.tsv', header=0, delimiter=\"\\t\", quoting=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we create a list of reviews.  Note that we're using a regular expression to remove all text that is not alphanumeric, so we don't have to deal with punctuation or tags.  This keeps it simple, but keep in mind, those tags and other characters may actualy have meaning sometimes.  You may actaully lose context or other information when you do this.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_records = []\n",
    "for i in range( 0, len(train[\"text\"])):\n",
    "    text = train[\"text\"][i]\n",
    "    text = re.sub(\"[^a-zA-Z0-9]\",\" \", text)\n",
    "    train_records.append(text.lower())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use the list of text to find a bag of words for all reviews, positive and negative.  We'll limit the bag of words to 200 terms so we can inspect it more easily.  For a real application, we'd probabably want set set this value to a much higher number.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "vectorizer = CountVectorizer(analyzer = \"word\",   \\\n",
    "                             tokenizer = None,    \\\n",
    "                             preprocessor = None, \\\n",
    "                             stop_words = 'english',   \\\n",
    "                             max_features = 200)\n",
    "\n",
    "\n",
    "train_data_features = vectorizer.fit_transform(train_records)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Now that we've created a bag of wrods, we can list the 200 most frequent terms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['10', 'acting', 'action', 'actor', 'actors', 'actually', 'audience', 'away', 'bad', 'based', 'begins', 'best', 'better', 'big', 'bit', 'black', 'called', 'cast', 'character', 'characters', 'city', 'come', 'comedy', 'comes', 'comic', 'completely', 'couple', 'course', 'david', 'day', 'dead', 'death', 'despite', 'dialogue', 'did', 'didn', 'different', 'director', 'does', 'doesn', 'don', 'effects', 'end', 'ending', 'especially', 'evil', 'example', 'fact', 'family', 'far', 'father', 'feel', 'film', 'films', 'final', 'friend', 'fun', 'funny', 'game', 'gets', 'getting', 'given', 'gives', 'goes', 'going', 'good', 'got', 'great', 'group', 'guy', 'half', 'hand', 'hard', 'having', 'head', 'help', 'high', 'hollywood', 'home', 'horror', 'hour', 'human', 'humor', 'idea', 'instead', 'interesting', 'isn', 'jackie', 'james', 'job', 'john', 'just', 'kind', 'know', 'left', 'let', 'life', 'like', 'line', 'little', 'll', 'long', 'look', 'looking', 'looks', 'lost', 'lot', 'love', 'main', 'make', 'makes', 'making', 'man', 'men', 'michael', 'mind', 'minutes', 'moments', 'money', 'mother', 'movie', 'movies', 'mr', 'music', 'new', 'night', 'old', 'original', 'people', 'performance', 'performances', 'picture', 'place', 'planet', 'play', 'played', 'plays', 'plot', 'point', 'pretty', 'probably', 'quite', 'real', 'really', 'reason', 'rest', 'right', 'role', 'run', 'say', 'scene', 'scenes', 'school', 'screen', 'script', 'second', 'seen', 'sense', 'series', 'set', 'sex', 'shows', 'simply', 'small', 'soon', 'special', 'star', 'stars', 'story', 'sure', 'takes', 'thing', 'things', 'think', 'thought', 'time', 'times', 'town', 'true', 'trying', 'turns', 'unfortunately', 'use', 've', 'want', 'wants', 'watch', 'watching', 'way', 'wife', 'woman', 'won', 'work', 'works', 'world', 'written', 'wrong', 'year', 'years', 'young']\n"
     ]
    }
   ],
   "source": [
    "print(vectorizer.get_feature_names())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the most common 200 terms, we can decompose each review in the set into a word vector, counting the frequency of each term in a review. \n",
    "\n",
    "Let's take a look at a sample negative review..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "negative_review = pd.read_csv('data/negativeReview.tsv', header=0, delimiter=\"\\t\", quoting=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'after a marketing windup of striking visuals and the promise of star caliber actors   mission to mars ends up throwing a whiffleball    fiercely unoriginal   director depalma cobbles together a film by borrowing heavily from what has gone before him    there are aliens similar to those in close encounters of the third kind    the stranded astronaut theme is reminiscent of robinson crusoe on mars    the astronauts encounter space flight difficulties that smack of apollo 13    interior spacecraft visuals are redolent of 2001   a space odyssey    instead of using these components as a launching pad to create his own movie   de palma stops right there   refusing to infuse the film with anything even remotely resembling cleverness or heart    mission to mars takes it s first wobbly steps at a pre launch barbeque in which the perfunctory character introductions are done    during these surface scans of the characters   we learn that jim mcconnell   sinise   has lost his wife    it s a plot point revisted throughout the film with jackhammer subtlety    the rest of the crew exhibit a bland affability    there is no contentiousness   no friction to add the the dramatic tension of these men and women being confined to close quarters for an extended length of time    maybe depalma was going for the comraderie of the right stuff   but in that movie   the astronauts had embers of personality to warm us through the technical aspects    it s the year 2020 and this is nasa s first manned excursion to the red planet    a crew   led by luke graham   cheadle     arrives on mars and quickly discovers an anomaly   which they investigate with tragic results    graham is able to transmit a garbled distress call back to earth    in response   earth sends a rescue team comprised of mcconnell   woody blake   robbins     wife terri fisher   nielsen   and phil ohlmyer   o connell      obstacles are put in the crew s way and and they matter of  factly go about solving them    i should say   mcconnell goes about solving them    time and again   mcconnell is presented as some kind of wunderkind   which wouldn t be so bad if the rest of the crew didn t come across as so aggressivelly unremarkable       mention should be made of the misogynistic handling of fisher in a situation where the entire crew s mission and life is in mortal danger    on a team of professionals   she is portrayed as an emotion directed weak link    women serve no purpose in the movie other than to serve as a reflection of a male character s personality trait      by the time they land on mars and try to solve the mystery of what occurred   mission to mars starts laying on the cliches and stilted dialogue with a heavy brush    there is an adage in film to   show   don t tell      mission to mars does both    repeatedly    characters obsessively explain the obvious   explain their actions as they are doing them   explain to fellow astronauts facts which should be fundamental knowledge to them    the film s conclusion is momumentally derivative   anti climatic and unsatisying    as i walked out i wondered who the target audience might be for this film    the best i could come up with is pre teen age boys   but in this media saturated era   this film s components would have been old hat even for them    i have to think what attracted such talent to this film was the lure of making a good   modern day b movie    the key to such a venture is a certain depth and sincerity towards the material    i felt no such earnestness    '"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "negative_review[\"text\"][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can decompose this review into a word count vector, showing the frequency of each term in our bag of words (the most common 200 terms in the entire data set)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "bad_review_data_features = vectorizer.transform([full_bad_review[\"text\"][0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 0 0 0 1 0 1 0 1 0 0 1 0 0 0 0 0 0 2 2 0 2 0 0 0 0 0 0 0 1 0 0 0 1 0 1 0\n",
      " 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 2 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0\n",
      " 1 0 1 0 0 0 0 0 0 4 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 1 1 0 0 0 0 0 0 2 2 0\n",
      " 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 1 0 3 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 1 2 0 0 0 0 0 0 0 1 0 0]\n"
     ]
    }
   ],
   "source": [
    "print(bad_review_data_features.toarray()[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a little hard to parse without seeing the words, so let's create a loop to print the word and the frequency side by side for this review.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10 0\n",
      "acting 0\n",
      "action 0\n",
      "actor 0\n",
      "actors 1\n",
      "actually 0\n",
      "audience 1\n",
      "away 0\n",
      "bad 1\n",
      "based 0\n",
      "begins 0\n",
      "best 1\n",
      "better 0\n",
      "big 0\n",
      "bit 0\n",
      "black 0\n",
      "called 0\n",
      "cast 0\n",
      "character 2\n",
      "characters 2\n",
      "city 0\n",
      "come 2\n",
      "comedy 0\n",
      "comes 0\n",
      "comic 0\n",
      "completely 0\n",
      "couple 0\n",
      "course 0\n",
      "david 0\n",
      "day 1\n",
      "dead 0\n",
      "death 0\n",
      "despite 0\n",
      "dialogue 1\n",
      "did 0\n",
      "didn 1\n",
      "different 0\n",
      "director 1\n",
      "does 1\n",
      "doesn 0\n",
      "don 1\n",
      "effects 0\n",
      "end 0\n",
      "ending 0\n",
      "especially 0\n",
      "evil 0\n",
      "example 0\n",
      "fact 0\n",
      "family 0\n",
      "far 0\n",
      "father 0\n",
      "feel 0\n",
      "film 8\n",
      "films 0\n",
      "final 0\n",
      "friend 0\n",
      "fun 0\n",
      "funny 0\n",
      "game 0\n",
      "gets 0\n",
      "getting 0\n",
      "given 0\n",
      "gives 0\n",
      "goes 1\n",
      "going 1\n",
      "good 1\n",
      "got 0\n",
      "great 0\n",
      "group 0\n",
      "guy 0\n",
      "half 0\n",
      "hand 0\n",
      "hard 0\n",
      "having 0\n",
      "head 0\n",
      "help 0\n",
      "high 0\n",
      "hollywood 0\n",
      "home 0\n",
      "horror 0\n",
      "hour 0\n",
      "human 0\n",
      "humor 0\n",
      "idea 0\n",
      "instead 1\n",
      "interesting 0\n",
      "isn 0\n",
      "jackie 0\n",
      "james 0\n",
      "job 0\n",
      "john 0\n",
      "just 0\n",
      "kind 2\n",
      "know 0\n",
      "left 0\n",
      "let 0\n",
      "life 1\n",
      "like 0\n",
      "line 0\n",
      "little 0\n",
      "ll 0\n",
      "long 0\n",
      "look 0\n",
      "looking 0\n",
      "looks 0\n",
      "lost 1\n",
      "lot 0\n",
      "love 0\n",
      "main 0\n",
      "make 0\n",
      "makes 0\n",
      "making 1\n",
      "man 0\n",
      "men 1\n",
      "michael 0\n",
      "mind 0\n",
      "minutes 0\n",
      "moments 0\n",
      "money 0\n",
      "mother 0\n",
      "movie 4\n",
      "movies 0\n",
      "mr 0\n",
      "music 0\n",
      "new 0\n",
      "night 0\n",
      "old 1\n",
      "original 0\n",
      "people 0\n",
      "performance 0\n",
      "performances 0\n",
      "picture 0\n",
      "place 0\n",
      "planet 1\n",
      "play 0\n",
      "played 0\n",
      "plays 0\n",
      "plot 1\n",
      "point 1\n",
      "pretty 0\n",
      "probably 0\n",
      "quite 0\n",
      "real 0\n",
      "really 0\n",
      "reason 0\n",
      "rest 2\n",
      "right 2\n",
      "role 0\n",
      "run 0\n",
      "say 1\n",
      "scene 0\n",
      "scenes 0\n",
      "school 0\n",
      "screen 0\n",
      "script 0\n",
      "second 0\n",
      "seen 0\n",
      "sense 0\n",
      "series 0\n",
      "set 0\n",
      "sex 0\n",
      "shows 0\n",
      "simply 0\n",
      "small 0\n",
      "soon 0\n",
      "special 0\n",
      "star 1\n",
      "stars 0\n",
      "story 0\n",
      "sure 0\n",
      "takes 1\n",
      "thing 0\n",
      "things 0\n",
      "think 1\n",
      "thought 0\n",
      "time 3\n",
      "times 0\n",
      "town 0\n",
      "true 0\n",
      "trying 0\n",
      "turns 0\n",
      "unfortunately 0\n",
      "use 0\n",
      "ve 0\n",
      "want 0\n",
      "wants 0\n",
      "watch 0\n",
      "watching 0\n",
      "way 1\n",
      "wife 2\n",
      "woman 0\n",
      "won 0\n",
      "work 0\n",
      "works 0\n",
      "world 0\n",
      "written 0\n",
      "wrong 0\n",
      "year 1\n",
      "years 0\n",
      "young 0\n"
     ]
    }
   ],
   "source": [
    "bad_review_wordcount = bad_review_data_features.toarray()[0]\n",
    "for i in range(0, len(bad_review_wordcount)):\n",
    "    print(vectorizer.get_feature_names()[i], bad_review_wordcount[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may notice there aren't a lot of matches here, that most words have a count of zero.  This isn't uncommon, especially if you set the max_features to a low number.  In addition, many of the terms here, like \"film\", are too general to be associated with a positive or negative review.  To use this in a real training scenario, we'd want to increase the number of words in the bag considerably.  When we apply this in the next example, where we apply machine learning techniques to predict whether a review is positive or negative, we'll go up to 5,000 terms.  You can do that by changing a single parameter in some of the method calls - feel free to do that now before moving on to the next section.    \n",
    "\n",
    "Next up, we'll use the wordcount vectors we prepared here to train and run a machine learning model to classify new reviews into positive and negative.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}