07_naive/07_02_naive.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Naive Bayes Example\n",
    "\n",
    "We are going to explore using **classification of emails into spam and ham (not spam) using Naive Bayes.**\n",
    "\n",
    "Agenda:\n",
    "* Read in Files\n",
    "* Tokenize with CountVectorize\n",
    "* Examine the data\n",
    "* Generate a few relevant variables in manually calculating Naive Bayes\n",
    "* Use SKLearn's Implementation of Naive Bayes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To motivate this example, we'll be using a modified form of the [Ling-Spam](http://csmining.org/index.php/ling-spam-datasets.html) dataset. More precisely, this is a set of 960 emails that people have manually read and labeled them as spam or ham.\n",
    "\n",
    "This modified dataset from the Stanford OpenClassroom (and should be available in your repo), performs the following preprocessing steps on the original emails:\n",
    "* Stop Word Removal - remove common, nonmeaningful words such as \"and\",\"the\",\"of\"\n",
    "* Lemmatization - shorten similar words. \"Includes\", \"Included\",\" Include\" are all shortened to \"Include\"\n",
    "* Non-Word Removal - remove numbers, punctuation, tabs. (In practice, perhaps you would want punctuations! i.e. If you see \"Buy this now !!!!!!\", multiple exclamation marks are pretty useful in classifying spam/ham)\n",
    "\n",
    "Finally, I've condensed the entire set of files into a single CSV containing the following columns:\n",
    "* Set: Train/Test - I've explicitly set the emails into train or test sets\n",
    "* Label: Spam/Ham\n",
    "* Text: Words in the email"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read in Files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>Set</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Ham</td>\n",
       "      <td>Train</td>\n",
       "      <td>posting hi m work phonetics project modern iri...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Ham</td>\n",
       "      <td>Train</td>\n",
       "      <td>re help re iaupe hay alguien que sepa como se ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Ham</td>\n",
       "      <td>Train</td>\n",
       "      <td>proofread medieval e text humanist discussion ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Ham</td>\n",
       "      <td>Train</td>\n",
       "      <td>learn french quebec city de ami de l universit...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Ham</td>\n",
       "      <td>Train</td>\n",
       "      <td>basic journal face major cut library periodica...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Label    Set                                               Text\n",
       "0   Ham  Train  posting hi m work phonetics project modern iri...\n",
       "1   Ham  Train  re help re iaupe hay alguien que sepa como se ...\n",
       "2   Ham  Train  proofread medieval e text humanist discussion ...\n",
       "3   Ham  Train  learn french quebec city de ami de l universit...\n",
       "4   Ham  Train  basic journal face major cut library periodica..."
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path_to_repo=\"/Users/brianchung/Desktop/ga-ds/\"\n",
    "data = pd.read_csv(path_to_repo + '/07_naive/07_emailspam.csv')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Set</th>\n",
       "      <th>Label</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Test</th>\n",
       "      <th>Ham</th>\n",
       "      <td>130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Spam</th>\n",
       "      <td>130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Train</th>\n",
       "      <th>Ham</th>\n",
       "      <td>350</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Spam</th>\n",
       "      <td>350</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Text\n",
       "Set   Label      \n",
       "Test  Ham     130\n",
       "      Spam    130\n",
       "Train Ham     350\n",
       "      Spam    350"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's see how the training/testing sample are set up\n",
    "# Technically, cheating by looking at the \"test\" samples as well.\n",
    "# In general, want to discipline ourselves to not looking at test set until the end\n",
    "data.groupby(['Set','Label']).count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(700, 3) (260, 3)\n"
     ]
    }
   ],
   "source": [
    "# Let's go ahead and for now, split up our data into a train_data, and test_data\n",
    "train_data = data[ data.Set == 'Train']\n",
    "test_data  = data[ data.Set == 'Test' ]\n",
    "\n",
    "print train_data.shape, test_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tokenize\n",
    "\n",
    "To get probabilities for $P(w|c)$, we would need to know how often each word is repeated across the various emails. We COULD go in, split the text, and create a dict of {word : #count}, but its easier to let sklearn do it for us.\n",
    "\n",
    "\n",
    "## CountVectorizer\n",
    "\n",
    "`sklearn` comes with many built-in feature extraction and manipulation tools. For dealing with text data, there is the  `sklearn.feature_extraction.text` module, which contains the **`CountVectorizer`**\n",
    "\n",
    "CountVectorizer is a class that transforms text (either in the form of a list of strings, dataframe columns, Series), and outputs a matrix of document x tokens (where tokens represent a word or a phrase).\n",
    "\n",
    "For instance, if we had a 2 element array of [\"apple is an apple\", \"why is this blue\"], the output matrix would look like this:\n",
    "\n",
    "is|an|apple|why|this|blue\n",
    "--|--|-----|---|----|----\n",
    " 1| 1|    2|  0|   0|   0\n",
    " 1| 0|    0|  1|   1|   1\n",
    "\n",
    ",where the first row represents the tokenization of the first sentence \"apple is an apple\"\n",
    ", and the second row represents the tokenziation of the second sentence \"why is this blue\".\n",
    "\n",
    "\n",
    "The `CountVectorizer` (and most feature extraction methods in sklearn) follows a very simple interface:\n",
    "- `fit` takes a dataset and learns the features it's trying to extract. In this case that means that the algorithm learns the vocabulary of all samples\n",
    "- `transform` takes a dataset and produces the matrix as described above, based on the vocabulary (or feature elements) it learned.\n",
    "- `fit_transform` combines the two steps at once.\n",
    "\n",
    "For example, you may want to fit a vocabulary to a training set, transform the training set to train a model and then continually transform any new incoming examples you want to classify. You will generally only perform the fit step once but the transform step many times for any new datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "vect = CountVectorizer()\n",
    "\n",
    "# Fit learns the vocabularies within the training data\n",
    "vect.fit(train_data.Text)\n",
    "\n",
    "# When we pass in a set of documents to transform, countvectorizer returns a matrix of frequencies\n",
    "# with only words that have been fitted into the vocabulary\n",
    "train_csr = vect.transform(train_data.Text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'scipy.sparse.csr.csr_matrix'> \n",
      "\n",
      "(700, 19073) \n",
      "\n"
     ]
    }
   ],
   "source": [
    "#It returns a sparse matrix, which is more efficient when only a few entries of the matrix are non-zero\n",
    "print type(train_csr), '\\n'\n",
    "print train_csr.shape, '\\n'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<type 'numpy.ndarray'> \n",
      "\n",
      "<class 'numpy.matrixlib.defmatrix.matrix'> \n",
      "\n",
      "(700, 19073) \n",
      "\n",
      "(700, 3)\n"
     ]
    }
   ],
   "source": [
    "# Can also convert to a dense (full) matrix\n",
    "print train_csr.toarray()),'\\n'\n",
    "\n",
    "print train_csr.toarray().shape,'\\n'\n",
    "print train_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'aa',\n",
       " u'aaa',\n",
       " u'aaai',\n",
       " u'aaarghh',\n",
       " u'aaccess',\n",
       " u'aachen',\n",
       " u'aad',\n",
       " u'aafter',\n",
       " u'aalborg',\n",
       " u'aaron',\n",
       " u'ab',\n",
       " u'aback',\n",
       " u'abandon',\n",
       " u'abank',\n",
       " u'abba',\n",
       " u'abbadingo',\n",
       " u'abbas',\n",
       " u'abbildung',\n",
       " u'abbreviation',\n",
       " u'abc',\n",
       " u'abduction',\n",
       " u'abdulfanova',\n",
       " u'abe',\n",
       " u'abeer',\n",
       " u'abeille',\n",
       " u'aberdeen',\n",
       " u'aberystwyth',\n",
       " u'abhaengigkeit',\n",
       " u'abijggccndoqggwgwayikkpzgaq',\n",
       " u'abilities',\n",
       " u'ability',\n",
       " u'abillity',\n",
       " u'abitualus',\n",
       " u'able',\n",
       " u'ablead',\n",
       " u'ableist',\n",
       " u'abney',\n",
       " u'aboard',\n",
       " u'abord',\n",
       " u'aboriginal',\n",
       " u'aborigine',\n",
       " u'aborigines',\n",
       " u'abou',\n",
       " u'about',\n",
       " u'above',\n",
       " u'abovemention',\n",
       " u'abraham',\n",
       " u'abramov',\n",
       " u'abreviatura',\n",
       " u'abrg',\n",
       " u'abroad',\n",
       " u'abrufbar',\n",
       " u'abrufen',\n",
       " u'absence',\n",
       " u'absent',\n",
       " u'absolute',\n",
       " u'absolutely',\n",
       " u'absolutly',\n",
       " u'absorb',\n",
       " u'absorption',\n",
       " u'abstact',\n",
       " u'abstr',\n",
       " u'abstract',\n",
       " u'abstraction',\n",
       " u'abstracts',\n",
       " u'abt',\n",
       " u'abteilung',\n",
       " u'abu',\n",
       " u'abus',\n",
       " u'abuse',\n",
       " u'aby',\n",
       " u'ac',\n",
       " u'academe',\n",
       " u'academia',\n",
       " u'academic',\n",
       " u'academium',\n",
       " u'academy',\n",
       " u'accelerate',\n",
       " u'accent',\n",
       " u'accept',\n",
       " u'acceptable',\n",
       " u'acceptance',\n",
       " u'accepted',\n",
       " u'access',\n",
       " u'accessible',\n",
       " u'accident',\n",
       " u'acclaim',\n",
       " u'accommodate',\n",
       " u'accommodation',\n",
       " u'accomodate',\n",
       " u'accomodation',\n",
       " u'accompagnee',\n",
       " u'accompany',\n",
       " u'accompanying',\n",
       " u'accomplish',\n",
       " u'accomplishment',\n",
       " u'accor',\n",
       " u'accord',\n",
       " u'accordance',\n",
       " u'accordingly',\n",
       " u'account',\n",
       " u'accountant',\n",
       " u'accounts',\n",
       " u'accpaper',\n",
       " u'accredit',\n",
       " u'accreditation',\n",
       " u'accrue',\n",
       " u'acct',\n",
       " u'accueil',\n",
       " u'accueillera',\n",
       " u'acculturation',\n",
       " u'accumulate',\n",
       " u'accumulation',\n",
       " u'accuracy',\n",
       " u'accurate',\n",
       " u'accurately',\n",
       " u'accusative',\n",
       " u'accuse',\n",
       " u'accutrak',\n",
       " u'ace',\n",
       " u'acedemic',\n",
       " u'acehiseygceqegsggmxehseeohcjrcekvkedkhejxkeksefopcgsxceop',\n",
       " u'aceto',\n",
       " u'acgom',\n",
       " u'achard',\n",
       " u'ache',\n",
       " u'acheck',\n",
       " u'achievable',\n",
       " u'achieve',\n",
       " u'achievement',\n",
       " u'achievements',\n",
       " u'achievment',\n",
       " u'acid',\n",
       " u'acido',\n",
       " u'ackerman',\n",
       " u'acknowledge',\n",
       " u'acknowledgement',\n",
       " u'acknowledgment',\n",
       " u'acl',\n",
       " u'aclaim',\n",
       " u'aclic',\n",
       " u'acll',\n",
       " u'acm',\n",
       " u'acn',\n",
       " u'acollection',\n",
       " u'acould',\n",
       " u'acount',\n",
       " u'acoustic',\n",
       " u'acoustical',\n",
       " u'acqua',\n",
       " u'acquaintance',\n",
       " u'acquire',\n",
       " u'acquirer',\n",
       " u'acquise',\n",
       " u'acquisition',\n",
       " u'acquisitional',\n",
       " u'acreage',\n",
       " u'acrolect',\n",
       " u'acronym',\n",
       " u'across',\n",
       " u'acsu',\n",
       " u'act',\n",
       " u'acte',\n",
       " u'acting',\n",
       " u'action',\n",
       " u'actional',\n",
       " u'actionality',\n",
       " u'actiontype',\n",
       " u'activate',\n",
       " u'activation',\n",
       " u'active',\n",
       " u'actively',\n",
       " u'activestative',\n",
       " u'activit',\n",
       " u'activite',\n",
       " u'activity',\n",
       " u'actor',\n",
       " u'actual',\n",
       " u'actually',\n",
       " u'actualmente',\n",
       " u'acura',\n",
       " u'acusative',\n",
       " u'ad',\n",
       " u'ada',\n",
       " u'adam',\n",
       " u'adamantio',\n",
       " u'adamare',\n",
       " u'adamp',\n",
       " u'adams',\n",
       " u'adamson',\n",
       " u'adapt',\n",
       " u'adaptable',\n",
       " u'adaptation',\n",
       " u'adapter',\n",
       " u'adapting',\n",
       " u'adaptive',\n",
       " u'add',\n",
       " u'addasite',\n",
       " u'added',\n",
       " u'addendum',\n",
       " u'addict',\n",
       " u'adding',\n",
       " u'addison',\n",
       " u'addition',\n",
       " u'additional',\n",
       " u'additionally',\n",
       " u'addittional',\n",
       " u'addon',\n",
       " u'addr',\n",
       " u'address',\n",
       " u'addressed',\n",
       " u'addressee',\n",
       " u'addresses',\n",
       " u'adds',\n",
       " u'adduce',\n",
       " u'addweb',\n",
       " u'adele',\n",
       " u'adequate',\n",
       " u'adequation',\n",
       " u'adhering',\n",
       " u'adiego',\n",
       " u'adj',\n",
       " u'adjacency',\n",
       " u'adjacent',\n",
       " u'adjacently',\n",
       " u'adjectival',\n",
       " u'adjective',\n",
       " u'adjoin',\n",
       " u'adjudication',\n",
       " u'adjunct',\n",
       " u'adjunction',\n",
       " u'adjust',\n",
       " u'adjustable',\n",
       " u'adlu',\n",
       " u'adm',\n",
       " u'administer',\n",
       " u'administration',\n",
       " u'administrative',\n",
       " u'administrator',\n",
       " u'admiraal',\n",
       " u'admirable',\n",
       " u'admiral',\n",
       " u'admiralty',\n",
       " u'admiration',\n",
       " u'admire',\n",
       " u'admirer',\n",
       " u'admission',\n",
       " u'admit',\n",
       " u'admits',\n",
       " u'admyte',\n",
       " u'adnominal',\n",
       " u'adobe',\n",
       " u'adolfo',\n",
       " u'adopt',\n",
       " u'adoption',\n",
       " u'adposition',\n",
       " u'adresa',\n",
       " u'adress',\n",
       " u'adresse',\n",
       " u'adressee',\n",
       " u'adressenuebersichten',\n",
       " u'adresser',\n",
       " u'adriaensen',\n",
       " u'adrian',\n",
       " u'adrive',\n",
       " u'ads',\n",
       " u'adsheet',\n",
       " u'adtek',\n",
       " u'adtekphotomask',\n",
       " u'adult',\n",
       " u'adulterous',\n",
       " u'adultlearner',\n",
       " u'adults',\n",
       " u'adv',\n",
       " u'advance',\n",
       " u'advanced',\n",
       " u'advancement',\n",
       " u'advances',\n",
       " u'advantage',\n",
       " u'advent',\n",
       " u'adventure',\n",
       " u'adverb',\n",
       " u'adverbal',\n",
       " u'adverbial',\n",
       " u'adverse',\n",
       " u'advert',\n",
       " u'advertise',\n",
       " u'advertisement',\n",
       " u'advertiser',\n",
       " u'advertisers',\n",
       " u'advertises',\n",
       " u'advertising',\n",
       " u'advertisment',\n",
       " u'advertse',\n",
       " u'advice',\n",
       " u'advise',\n",
       " u'advisory',\n",
       " u'advocate',\n",
       " u'ae',\n",
       " u'aeasy',\n",
       " u'aedward',\n",
       " u'aee',\n",
       " u'aeegerd',\n",
       " u'aegis',\n",
       " u'aeroad',\n",
       " u'aerosmith',\n",
       " u'aespecially',\n",
       " u'aeternitati',\n",
       " u'aever',\n",
       " u'af',\n",
       " u'afecta',\n",
       " u'affair',\n",
       " u'affect',\n",
       " u'affectedness',\n",
       " u'affective',\n",
       " u'affidavit',\n",
       " u'affiliate',\n",
       " u'affiliation',\n",
       " u'affirmative',\n",
       " u'affix',\n",
       " u'affixed',\n",
       " u'affodable',\n",
       " u'afford',\n",
       " u'affordable',\n",
       " u'affricate',\n",
       " u'afit',\n",
       " u'afoot',\n",
       " u'afor',\n",
       " u'aformulum',\n",
       " u'afraid',\n",
       " u'aframerus',\n",
       " u'afresh',\n",
       " u'africa',\n",
       " u'african',\n",
       " u'afrika',\n",
       " u'afrikaans',\n",
       " u'afrikanistentag',\n",
       " u'afrikanistischer',\n",
       " u'afrikatale',\n",
       " u'afro',\n",
       " u'afrom',\n",
       " u'afrus',\n",
       " u'after',\n",
       " u'afternoon',\n",
       " u'afterward',\n",
       " u'ag',\n",
       " u'again',\n",
       " u'against',\n",
       " u'agbayanus',\n",
       " u'age',\n",
       " u'ageist',\n",
       " u'agencies',\n",
       " u'agency',\n",
       " u'agenda',\n",
       " u'agent',\n",
       " u'agentive',\n",
       " u'agentivity',\n",
       " u'agfox',\n",
       " u'aggravation',\n",
       " u'aggressive',\n",
       " u'agirre',\n",
       " u'agit',\n",
       " u'ago',\n",
       " u'agogique',\n",
       " u'agp',\n",
       " u'agr',\n",
       " u'agranat',\n",
       " u'agree',\n",
       " u'agreeing',\n",
       " u'agreement',\n",
       " u'agressive',\n",
       " u'agribusiness',\n",
       " u'agro',\n",
       " u'agrupado',\n",
       " u'aguaranteed',\n",
       " u'aguero',\n",
       " u'agus',\n",
       " u'agustus',\n",
       " u'agykd',\n",
       " u'ah',\n",
       " u'ahead',\n",
       " u'ahiawatha',\n",
       " u'ahlqvist',\n",
       " u'ahlsen',\n",
       " u'ahn',\n",
       " u'ahold',\n",
       " u'ahom',\n",
       " u'ai',\n",
       " u'aid',\n",
       " u'aida',\n",
       " u'aide',\n",
       " u'aideen',\n",
       " u'aif',\n",
       " u'aijob',\n",
       " u'ailment',\n",
       " u'aim',\n",
       " u'aime',\n",
       " u'aims',\n",
       " u'aimun',\n",
       " u'ain',\n",
       " u'ainsus',\n",
       " u'ainternet',\n",
       " u'ainu',\n",
       " u'aiqhiouewo',\n",
       " u'air',\n",
       " u'airborne',\n",
       " u'airfare',\n",
       " u'airline',\n",
       " u'airmail',\n",
       " u'airport',\n",
       " u'airpressure',\n",
       " u'airtight',\n",
       " u'airwave',\n",
       " u'aisb',\n",
       " u'aise',\n",
       " u'aissen',\n",
       " u'ait',\n",
       " u'aitchison',\n",
       " u'aix',\n",
       " u'aj',\n",
       " u'ajayus',\n",
       " u'ajh',\n",
       " u'ak',\n",
       " u'akademie',\n",
       " u'akan',\n",
       " u'akhadov',\n",
       " u'akiko',\n",
       " u'akin',\n",
       " u'akio',\n",
       " u'akira',\n",
       " u'akito',\n",
       " u'akitoshus',\n",
       " u'akkadian',\n",
       " u'akron',\n",
       " u'aktionsart',\n",
       " u'aktionsbeschreibungen',\n",
       " u'aktuelle',\n",
       " u'akus',\n",
       " u'akustik',\n",
       " u'akustischer',\n",
       " u'al',\n",
       " u'ala',\n",
       " u'alabama',\n",
       " u'aladin',\n",
       " u'alake',\n",
       " u'alamo',\n",
       " u'alan',\n",
       " u'alani',\n",
       " u'alank',\n",
       " u'alarm',\n",
       " u'alasa',\n",
       " u'alaska',\n",
       " u'alati',\n",
       " u'alba',\n",
       " u'albanian',\n",
       " u'albanium',\n",
       " u'albert',\n",
       " u'alberta',\n",
       " u'alberto',\n",
       " u'albertson',\n",
       " u'albrand',\n",
       " u'album',\n",
       " u'albuquerque',\n",
       " u'alcancia',\n",
       " u'alcazar',\n",
       " u'alcohol',\n",
       " u'alderete',\n",
       " u'alderson',\n",
       " u'alec',\n",
       " u'alefirenko',\n",
       " u'alegrium',\n",
       " u'alekseeva',\n",
       " u'alemana',\n",
       " u'alert',\n",
       " u'alessandra',\n",
       " u'alessio',\n",
       " u'aleut',\n",
       " u'alex',\n",
       " u'alexander',\n",
       " u'alexandr',\n",
       " u'alexandra',\n",
       " u'alexandre',\n",
       " u'alexandrova',\n",
       " u'alexeus',\n",
       " u'alexiadou',\n",
       " u'alexis',\n",
       " u'aleya',\n",
       " u'alf',\n",
       " u'alfa',\n",
       " u'alfon',\n",
       " u'alfonsine',\n",
       " u'alfonso',\n",
       " u'alfred',\n",
       " u'algebra',\n",
       " u'algebraic',\n",
       " u'algo',\n",
       " u'algonkianist',\n",
       " u'algorithm',\n",
       " u'algorithman',\n",
       " u'algorithmic',\n",
       " u'algortihmic',\n",
       " u'alguien',\n",
       " u'alhambra',\n",
       " u'alia',\n",
       " u'alias',\n",
       " u'alicante',\n",
       " u'alie',\n",
       " u'alien',\n",
       " u'aliens',\n",
       " u'align',\n",
       " u'alignment',\n",
       " u'alike',\n",
       " u'alimenter',\n",
       " u'alimony',\n",
       " u'aliou',\n",
       " u'aliseda',\n",
       " u'alisova',\n",
       " u'alisp',\n",
       " u'alist',\n",
       " u'alister',\n",
       " u'alium',\n",
       " u'alius',\n",
       " u'alive',\n",
       " u'alkaline',\n",
       " u'alkuper',\n",
       " u'all',\n",
       " u'allan',\n",
       " u'allative',\n",
       " u'alle',\n",
       " u'allee',\n",
       " u'alleen',\n",
       " u'allege',\n",
       " u'allemagne',\n",
       " u'allen',\n",
       " u'allesklar',\n",
       " u'alleviate',\n",
       " u'alley',\n",
       " u'allgemeine',\n",
       " u'allgender',\n",
       " u'alliance',\n",
       " u'alligaten',\n",
       " u'allimportant',\n",
       " u'allocate',\n",
       " u'allot',\n",
       " u'allote',\n",
       " u'allow',\n",
       " u'allowance',\n",
       " u'allserv',\n",
       " u'allstate',\n",
       " u'allusive',\n",
       " u'allwood',\n",
       " u'ally',\n",
       " u'almost',\n",
       " u'alnijazov',\n",
       " u'alon',\n",
       " u'alone',\n",
       " u'along',\n",
       " u'alot',\n",
       " u'aloud',\n",
       " u'alouse',\n",
       " u'alp',\n",
       " u'alpatov',\n",
       " u'alphabet',\n",
       " u'alphabetic',\n",
       " u'alphanumeric',\n",
       " u'alphonse',\n",
       " u'already',\n",
       " u'alsig',\n",
       " u'alsina',\n",
       " u'alster',\n",
       " u'alt',\n",
       " u'alta',\n",
       " u'altabayeva',\n",
       " u'altaic',\n",
       " u'altavista',\n",
       " u'altenberg',\n",
       " u'alter',\n",
       " u'alteration',\n",
       " u'altercation',\n",
       " u'alternate',\n",
       " u'alternately',\n",
       " u'alternates',\n",
       " u'alternation',\n",
       " u'alternative',\n",
       " u'alternatively',\n",
       " u'although',\n",
       " u'altitude',\n",
       " u'altmann',\n",
       " u'alto',\n",
       " u'altus',\n",
       " u'aludra',\n",
       " u'alumno',\n",
       " u'alumnus',\n",
       " u'alus',\n",
       " u'alvarez',\n",
       " u'alveolar',\n",
       " u'alvin',\n",
       " u'alwan',\n",
       " u'alway',\n",
       " u'always',\n",
       " u'alwaysfriend',\n",
       " u'am',\n",
       " u'ama',\n",
       " u'amanda',\n",
       " u'amarket',\n",
       " u'amass',\n",
       " u'amateur',\n",
       " u'amateurporn',\n",
       " u'amaze',\n",
       " u'amazing',\n",
       " u'amazingly',\n",
       " u'amazon',\n",
       " u'ambiguitaetserhaltenden',\n",
       " u'ambiguity',\n",
       " u'ambiguous',\n",
       " u'ambitious',\n",
       " u'ambra',\n",
       " u'ambraresource',\n",
       " u'amcap',\n",
       " u'amd',\n",
       " u'ame',\n",
       " u'amea',\n",
       " u'amendment',\n",
       " u'amer',\n",
       " u'amerchant',\n",
       " u'america',\n",
       " u'americal',\n",
       " u'american',\n",
       " u'americana',\n",
       " u'americanet',\n",
       " u'americanism',\n",
       " u'americannet',\n",
       " u'amerikanskoj',\n",
       " u'amerind',\n",
       " u'amerindian',\n",
       " u'amerindianist',\n",
       " u'amex',\n",
       " u'amherst',\n",
       " u'ami',\n",
       " u'amid',\n",
       " u'amidst',\n",
       " u'amil',\n",
       " u'amino',\n",
       " u'aminolyze',\n",
       " u'aminova',\n",
       " u'amir',\n",
       " u'amman',\n",
       " u'ammerlaan',\n",
       " u'ammianus',\n",
       " u'ammond',\n",
       " u'ammondt',\n",
       " u'amnagement',\n",
       " u'amnet',\n",
       " u'among',\n",
       " u'amongst',\n",
       " u'amonth',\n",
       " u'amorce',\n",
       " u'amorgo',\n",
       " u'amorn',\n",
       " u'amount',\n",
       " u'amounts',\n",
       " u'amp',\n",
       " u'amphitheatr',\n",
       " u'amphus',\n",
       " u'ample',\n",
       " u'amr',\n",
       " u'ams',\n",
       " u'amsilus',\n",
       " u'amsterdam',\n",
       " u'amta',\n",
       " u'amu',\n",
       " u'amur',\n",
       " u'amuse',\n",
       " u'amusement',\n",
       " u'amwgodfniifpaxxosvgnhlnsmckvfyhfpcwiq',\n",
       " u'amx',\n",
       " u'amy',\n",
       " u'ana',\n",
       " u'anagnostopoulou',\n",
       " u'anal',\n",
       " u'analog',\n",
       " u'analogous',\n",
       " u'analogy',\n",
       " u'analyisis',\n",
       " u'analyse',\n",
       " u'analysis',\n",
       " u'analyst',\n",
       " u'analysts',\n",
       " u'analytic',\n",
       " u'analytical',\n",
       " u'analyze',\n",
       " u'analyzer',\n",
       " u'ananiadou',\n",
       " u'anaphora',\n",
       " u'anaphore',\n",
       " u'anaphoric',\n",
       " u'anastasium',\n",
       " u'anatolius',\n",
       " u'anatomical',\n",
       " u'anaysis',\n",
       " u'ancestor',\n",
       " u'ancient',\n",
       " u'ancr',\n",
       " u'ancrage',\n",
       " u'and',\n",
       " u'andalucia',\n",
       " u'andalucium',\n",
       " u'ander',\n",
       " u'anderson',\n",
       " u'andr',\n",
       " u'andramonova',\n",
       " u'andre',\n",
       " u'andrea',\n",
       " u'andree',\n",
       " u'andreeva',\n",
       " u'andreevskaya',\n",
       " u'andrew',\n",
       " u'andrewsd',\n",
       " u'andrey',\n",
       " u'andriaensen',\n",
       " u'androutsopoulo',\n",
       " u'anecdotal',\n",
       " u'angel',\n",
       " u'angele',\n",
       " u'angelfire',\n",
       " u'angelika',\n",
       " u'angelova',\n",
       " u'angels',\n",
       " u'angelum',\n",
       " u'angerufen',\n",
       " u'angewandte',\n",
       " u'angiportum',\n",
       " u'anglai',\n",
       " u'angle',\n",
       " u'anglicisation',\n",
       " u'anglicize',\n",
       " u'anglistik',\n",
       " u'anglocentrism',\n",
       " u'anglophone',\n",
       " u'angry',\n",
       " u'angulus',\n",
       " u'anhand',\n",
       " u'anik',\n",
       " u'aniko',\n",
       " u'animacy',\n",
       " u'animal',\n",
       " u'animan',\n",
       " u'animate',\n",
       " u'animation',\n",
       " u'anita',\n",
       " u'ank',\n",
       " u'anke',\n",
       " u'anlum',\n",
       " u'anmeldung',\n",
       " u'ann',\n",
       " u'anna',\n",
       " u'annabel',\n",
       " u'anne',\n",
       " u'annee',\n",
       " u'anneke',\n",
       " u'annemarie',\n",
       " u'annesp',\n",
       " u'annette',\n",
       " u'annick',\n",
       " u'annie',\n",
       " u'anniversarry',\n",
       " u'anniversary',\n",
       " u'annonce',\n",
       " u'annoncer',\n",
       " u'annonymous',\n",
       " u'annotate',\n",
       " u'annotation',\n",
       " u'annotator',\n",
       " u'annotext',\n",
       " u'announce',\n",
       " u'announcement',\n",
       " u'announcer',\n",
       " u'announcing',\n",
       " u'annoy',\n",
       " u'annoyance',\n",
       " u'annoying',\n",
       " u'annual',\n",
       " u'annually',\n",
       " u'annum',\n",
       " u'anoint',\n",
       " u'anon',\n",
       " u'anonymen',\n",
       " u'anonymity',\n",
       " u'anonymous',\n",
       " u'anonymously',\n",
       " u'anonyomus',\n",
       " u'anoschenkov',\n",
       " u'another',\n",
       " u'anow',\n",
       " u'anreise',\n",
       " u'anrufen',\n",
       " u'ansatze',\n",
       " u'answer',\n",
       " u'answering',\n",
       " u'answers',\n",
       " u'antecedent',\n",
       " u'antecedentgovern',\n",
       " u'anteposition',\n",
       " u'anthony',\n",
       " u'anthro',\n",
       " u'anthropo',\n",
       " u'anthropological',\n",
       " u'anthropologist',\n",
       " u'anthropology',\n",
       " u'anthropomorphic',\n",
       " u'anti',\n",
       " u'anticipate',\n",
       " u'anticipation',\n",
       " u'anticlockwise',\n",
       " u'anticommerce',\n",
       " u'anticommunist',\n",
       " u'antics',\n",
       " u'antigua',\n",
       " u'antiguo',\n",
       " u'antiinternet',\n",
       " u'antillean',\n",
       " u'antioquium',\n",
       " u'antiquity',\n",
       " u'antispam',\n",
       " u'antisymmetric',\n",
       " u'antisymmetry',\n",
       " u'antje',\n",
       " u'antoniadi',\n",
       " u'antoniadis',\n",
       " u'antonio',\n",
       " u'antonium',\n",
       " u'antus',\n",
       " u'antwerp',\n",
       " u'antwerpen',\n",
       " u'antwortextraktion',\n",
       " u'antworthy',\n",
       " u'anu',\n",
       " u'anually',\n",
       " u'anuchsvdxbffdeemihcpwryweowvsalnenqtalyqhwvvqm',\n",
       " u'anuj',\n",
       " u'anus',\n",
       " u'anwendersicht',\n",
       " u'anxiety',\n",
       " u'anxious',\n",
       " u'anybody',\n",
       " u'anymore',\n",
       " u'anyone',\n",
       " u'anyplace',\n",
       " u'anythe',\n",
       " u'anything',\n",
       " u'anytime',\n",
       " u'anyway',\n",
       " u'anywhere',\n",
       " u'anzwer',\n",
       " u'ao',\n",
       " u'aok',\n",
       " u'aokus',\n",
       " u'aol',\n",
       " u'aorder',\n",
       " u'aoun',\n",
       " u'ap',\n",
       " u'apart',\n",
       " u'apartment',\n",
       " u'aphasia',\n",
       " u'aphasic',\n",
       " u'aphasium',\n",
       " u'api',\n",
       " u'apic',\n",
       " u'apiece',\n",
       " u'apollo',\n",
       " u'apologise',\n",
       " u'apologize',\n",
       " u'apology',\n",
       " u'apostroph',\n",
       " u'apostrophe',\n",
       " u'app',\n",
       " u'apparatus',\n",
       " u'apparent',\n",
       " u'apparently',\n",
       " u'appartment',\n",
       " u'appeal',\n",
       " u'appear',\n",
       " u'appearance',\n",
       " u'appel',\n",
       " u'appellativen',\n",
       " u'appelt',\n",
       " u'append',\n",
       " u'appendix',\n",
       " u'appetite',\n",
       " u'apple',\n",
       " u'appleiss',\n",
       " u'applepie',\n",
       " u'appliance',\n",
       " u'appliances',\n",
       " u'applicability',\n",
       " u'applicable',\n",
       " u'applicant',\n",
       " u'applicants',\n",
       " u'application',\n",
       " u'applications',\n",
       " u'applicationspecific',\n",
       " u'applicative',\n",
       " u'applied',\n",
       " u'appliqus',\n",
       " u'apply',\n",
       " u'appoint',\n",
       " u'appointee',\n",
       " u'appointment',\n",
       " u'appology',\n",
       " u'appraisal',\n",
       " u'appraise',\n",
       " u'appreciable',\n",
       " u'appreciate',\n",
       " u'appreciation',\n",
       " u'apprendre',\n",
       " u'apprenticeship',\n",
       " u'apprentissage',\n",
       " u'approach',\n",
       " u'approaches',\n",
       " u'approche',\n",
       " u'appropriate',\n",
       " u'appropriately',\n",
       " u'approval',\n",
       " u'approve',\n",
       " u'approved',\n",
       " u'approx',\n",
       " u'approximate',\n",
       " u'approximately',\n",
       " u'approximative',\n",
       " u'apr',\n",
       " u'apre',\n",
       " u'apresmidus',\n",
       " u'apretiate',\n",
       " u'aprettywoman',\n",
       " u'april',\n",
       " u'apt',\n",
       " u'aptitude',\n",
       " u'aqhyaldaxuahbafwjqwxrvzus',\n",
       " u'aqua',\n",
       " u'aquisition',\n",
       " u'ar',\n",
       " u'arabe',\n",
       " u'arabem',\n",
       " u'arabic',\n",
       " u'arabium',\n",
       " u'aramaic',\n",
       " u'aravind',\n",
       " u'arbeitsgruppe',\n",
       " u'arbitrage',\n",
       " u'arbitrary',\n",
       " u'arbor',\n",
       " u'arcade',\n",
       " u'archaeological',\n",
       " u'archaeology',\n",
       " u'archaism',\n",
       " u'archangelus',\n",
       " u'archeologie',\n",
       " u'archery',\n",
       " u'archetype',\n",
       " u'architect',\n",
       " u'architectural',\n",
       " u'architecture',\n",
       " u'archival',\n",
       " u'archive',\n",
       " u'archivo',\n",
       " u'arcle',\n",
       " u'arcling',\n",
       " u'ardus',\n",
       " u'are',\n",
       " u'area',\n",
       " u'areal',\n",
       " u'aren',\n",
       " u'arena',\n",
       " u'argentina',\n",
       " u'argo',\n",
       " u'argonautica',\n",
       " u'arguable',\n",
       " u'argue',\n",
       " u'argument',\n",
       " u'argumentation',\n",
       " u'arian',\n",
       " u'arianna',\n",
       " u'ariel',\n",
       " u'arinshtein',\n",
       " u'arise',\n",
       " u'aristar',\n",
       " u'aristote',\n",
       " u'aristotle',\n",
       " u'arizona',\n",
       " u'ark',\n",
       " u'arkesinus',\n",
       " u'arkg',\n",
       " u'arkhipov',\n",
       " u'arlanda',\n",
       " u'arlene',\n",
       " u'arlington',\n",
       " u'arm',\n",
       " u'armand',\n",
       " u'armey',\n",
       " u'armeyeva',\n",
       " ...]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Well, it has the same number of documents, but what are the words?\n",
    "vect.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aa</th>\n",
       "      <th>aaa</th>\n",
       "      <th>aaai</th>\n",
       "      <th>aaarghh</th>\n",
       "      <th>aaccess</th>\n",
       "      <th>aachen</th>\n",
       "      <th>aad</th>\n",
       "      <th>aafter</th>\n",
       "      <th>aalborg</th>\n",
       "      <th>aaron</th>\n",
       "      <th>...</th>\n",
       "      <th>zus</th>\n",
       "      <th>zusammengest</th>\n",
       "      <th>zusammenstellung</th>\n",
       "      <th>zvegintseva</th>\n",
       "      <th>zwart</th>\n",
       "      <th>zwischen</th>\n",
       "      <th>zxgahqabjh</th>\n",
       "      <th>zybatov</th>\n",
       "      <th>zybatow</th>\n",
       "      <th>zz</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 19073 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   aa  aaa  aaai  aaarghh  aaccess  aachen  aad  aafter  aalborg  aaron ...  \\\n",
       "0   0    0     0        0        0       0    0       0        0      0 ...   \n",
       "1   0    0     0        0        0       0    0       0        0      0 ...   \n",
       "2   0    0     0        0        0       0    0       0        0      0 ...   \n",
       "3   0    0     0        0        0       0    0       0        0      0 ...   \n",
       "4   0    0     0        0        0       0    0       0        0      0 ...   \n",
       "\n",
       "   zus  zusammengest  zusammenstellung  zvegintseva  zwart  zwischen  \\\n",
       "0    0             0                 0            0      0         0   \n",
       "1    0             0                 0            0      0         0   \n",
       "2    0             0                 0            0      0         0   \n",
       "3    0             0                 0            0      0         0   \n",
       "4    0             0                 0            0      0         0   \n",
       "\n",
       "   zxgahqabjh  zybatov  zybatow  zz  \n",
       "0           0        0        0   0  \n",
       "1           0        0        0   0  \n",
       "2           0        0        0   0  \n",
       "3           0        0        0   0  \n",
       "4           0        0        0   0  \n",
       "\n",
       "[5 rows x 19073 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Can convert to a pandas dataframe too (generally not needed for SKLearn methods)\n",
    "train_df = pd.DataFrame( train_csr.toarray(), columns=vect.get_feature_names() )\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "aa\n",
      "5\n"
     ]
    }
   ],
   "source": [
    "# How many times does 'aa' occur throughout all the training documents?\n",
    "print train_df.columns[0] \n",
    "print train_df.iloc[:,0].sum() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     29\n",
       "1     44\n",
       "2    322\n",
       "3     90\n",
       "4     48\n",
       "dtype: int64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# How many tokens are there per row? \n",
    "# I.E. How many words are there per email? \n",
    "# Tokens are individual words in this case \n",
    "train_df.sum(axis=1).head() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Max: 1701\n",
      "Word: email\n"
     ]
    }
   ],
   "source": [
    "# Whats the most frequent token?\n",
    "token_count = train_df.sum(axis=0)\n",
    "\n",
    "print \"Max:\",  np.max(token_count)\n",
    "print \"Word:\", np.argmax(token_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "aa         3\n",
       "aaa        9\n",
       "aaai       0\n",
       "aaarghh    1\n",
       "aaccess    1\n",
       "dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# What are the total word frequencies for spam emails?\n",
    "train_labels = train_data.Label\n",
    "train_df[ train_labels == 'Spam'].sum(axis=0).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise\n",
    "\n",
    "If you have not yet already, review and run the code blocks up to this point.\n",
    "\n",
    "You will need the following defined (at least):\n",
    "* train_data\n",
    "* train_labels\n",
    "* train_df \n",
    "\n",
    "**Count the total number of emails in train_data. Save this to \"doc_total\"**\n",
    "\n",
    "**Count the total number of spam emails in train_data. Save this to \"doc_spam\"**\n",
    "\n",
    "**Count the total number of ham emails in train_data. Save this to \"doc_ham\"**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code here\n",
    "doc_spam = np.sum(train_labels == \"Ham\")\n",
    "doc_ham  = np.sum(train_labels == \"Spam\")\n",
    "doc_total = len(train_labels)\n",
    "\n",
    "\n",
    "\n",
    "assert(doc_spam + doc_ham == doc_total)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**How many total unique words are there in the training set (Vocabulary)? Hint: This is the number of columns in train_df. Save this to \"V_train\"**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Your code here\n",
    "V_train = train_df.shape[1]\n",
    "\n",
    "assert(V_train==19073)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What is the total count of words in spam emails? Save this as \"word_total_spam\". **\n",
    "\n",
    "**What is the total count of words in ham emails? Save this as \"word_total_ham\". **\n",
    "\n",
    "i.e. if there are two spam emails:\n",
    "\n",
    "    A: \"hello world hello\"\n",
    "    B: \"is spam\"\n",
    "then word_total_spam = 3 + 2 = 5\n",
    "\n",
    "Hint: Hint: Subset train_df on the labels, and use 'sum' to get total frequencies similar. You may need to call xx.sum().sum() to get the sum of every element where xx is the dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Your code here\n",
    "word_total_spam = train_df[ train_labels=='Spam' ].sum().sum()\n",
    "word_total_ham  = train_df[ train_labels=='Ham'  ].sum().sum()\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "assert(word_total_spam==105771 and word_total_ham==86102)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Generate the total counts per word per class.**\n",
    "\n",
    "Hint: Subset train_df on the labels, and use 'sum' to get total frequencies similar to above (Use the axis= argument as well in sum).\n",
    "\n",
    "Save the resulting series as **word_count_ham**, and **word_count_spam**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Your code here\n",
    "word_count_spam = train_df[ train_labels=='Spam' ].sum(axis=0) \n",
    "word_count_ham  = train_df[ train_labels=='Ham'  ].sum(axis=0) \n",
    "\n",
    "\n",
    "\n",
    "assert( len(word_count_spam)==19073 and len(word_count_ham)==19073 and \n",
    "       np.sum(word_count_spam)==105771 and np.sum(word_count_ham)==86102 and\n",
    "      type(word_count_spam) is pd.Series and type(word_count_ham) is pd.Series )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** What is the probability of seeing the word \"web\" given class == \"Spam\" ? What is the probability of seeing the word \"web\" given class == \"Ham\" ?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.00376284614876 0.00106850015098\n"
     ]
    }
   ],
   "source": [
    "#Your code here\n",
    "#Your code here\n",
    "\n",
    "\n",
    "p_web_spam = word_count_spam['web'] / np.float(np.sum(word_count_spam))\n",
    "\n",
    "p_web_ham  = word_count_ham['web'] / np.float(np.sum(word_count_ham))\n",
    "\n",
    "print p_web_spam, p_web_ham\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** What is the probability of seeing the word \"Brrahhhhh\" given class == \"Spam\"? What about \"Ham\"?**\n",
    "\n",
    "Hint: Laplace Smoothing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "Now you're ready to classify documents! Why? Let's think about it\n",
    "\n",
    "* You can calculate class priors P(C==ham) and P(C==spam). This is just doc_ham/doc_total, and doc_spam/doc_total.\n",
    "* You can also iterate through the class labels, and generate P(w|C==ham) and p(w|C==spam). This is just the count of the word `w` divided by the total number of words in the class.\n",
    "* In essence, if there were a `NaiveBayes` model in SKLearn, it's as if you had calculate all the components of `fit`\n",
    "\n",
    "```\n",
    "Algorithm:\n",
    "    Calculate class Priors p(spam), p(ham)\n",
    "    For each row in the data:\n",
    "        Keep a running total probability for spam and ham (logP_spam, logP_ham)\n",
    "        For each word in row:\n",
    "            Calculate P(w|spam), then add this to the running total probability for spam\n",
    "            [P(w|spam)= data frequency of w * log( (word_count_spam[w] + 1) / (word_total_spam + V_train))] \n",
    "            Calculate P(w|ham), then add this to the running total probability for ham\n",
    "            [P(w|ham) = data frequency of w *  log((word_count_ham[w] + 1) / (word_total_ham + V_train))] \n",
    "        Compare logP_spam and logP_ham. Classify this row as whichever is higher\n",
    "            \n",
    "```\n",
    "\n",
    "\n",
    "I've included my version of Naive Bayes using a Multinomial model, but you will have the opportunity to write your own later as well!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive Bayes Prediction on the Training Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.998571428571\n"
     ]
    }
   ],
   "source": [
    "# Recreate a count vectorizer here, and fit it on the training data\n",
    "vect = CountVectorizer()\n",
    "vect.fit(train_data.Text)\n",
    "\n",
    "# Transform the training data into a sparse matrix, and then to a pandas dataframe for ease\n",
    "train_csr = vect.transform(train_data.Text)\n",
    "train_df = pd.DataFrame( train_csr.toarray(), columns=vect.get_feature_names() )\n",
    "\n",
    "\n",
    "# Save a list of the truth and one for my predictions\n",
    "all_true_labels = train_data.Label\n",
    "all_pred_labels = [\"\"]*len(train_data.Label)\n",
    "\n",
    "\n",
    "# Calculate our class priors\n",
    "p_c_spam = np.float(doc_spam) / np.float(doc_total)\n",
    "p_c_ham  = np.float(doc_ham) / np.float(doc_total)\n",
    "\n",
    "for row_idx in range(train_data.shape[0]):\n",
    "\n",
    "    \n",
    "    # Save a running log probability \n",
    "    logP_spam = np.float64(0.0)\n",
    "    logP_ham  = np.float64(0.0)\n",
    "    \n",
    "    # Grab all the tokens for that row in the form of a Series w/index = word, value = frequency of word\n",
    "    words = train_df.iloc[row_idx,:]\n",
    "    words = words[ words > 0]\n",
    "    \n",
    "    for word in words.index:\n",
    "        cnt = words[word]\n",
    "\n",
    "        #Calculate the quantity p(w|spam)\n",
    "        p_word_spam = 0.0\n",
    "        if word in word_count_spam.index:\n",
    "            p_word_spam = (word_count_spam[word] + 1.0) / ( np.float(word_total_spam) + V_train )\n",
    "        else:\n",
    "            # If this word is not in our dictionary, give it the laplace smoothed prior!\n",
    "            p_word_spam = 1.0 / ( np.float(word_total_spam) + V_train )\n",
    "\n",
    "\n",
    "        #Calculate the quantity p(w|spam)\n",
    "        p_word_ham = 0.0\n",
    "        if word in word_count_ham.index:\n",
    "            p_word_ham = (word_count_ham[word] + 1.0) / ( np.float(word_total_ham) + V_train )\n",
    "        else:\n",
    "            # If this word is not in our dictionary, give it the laplace smoothed prior!\n",
    "            p_word_ham = 1.0 / ( np.float(word_total_ham) + V_train )\n",
    "        \n",
    "        # Add the correct count * log( p(w|c) ) to both categories\n",
    "        logP_spam = logP_spam + cnt * np.log(p_word_spam)\n",
    "        logP_ham = logP_ham +   cnt * np.log(p_word_ham)\n",
    "\n",
    "    logP_spam = logP_spam + np.log(p_c_spam)\n",
    "    logP_ham = logP_ham + np.log(p_c_ham)\n",
    "\n",
    "    # Check which value is higher. Remember we didn't compute any normalization, since we don't care\n",
    "    if logP_spam > logP_ham: \n",
    "        all_pred_labels[row_idx] = \"Spam\"\n",
    "    else:\n",
    "        all_pred_labels[row_idx] = \"Ham\"\n",
    "\n",
    "\n",
    "print \"Accuracy:\", np.sum(all_pred_labels == all_true_labels) / np.float(len(all_true_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive Bayes Prediction on the Test Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.976923076923\n"
     ]
    }
   ],
   "source": [
    "# Test set Using the Probabilities we've calculated with the training set\n",
    "# !!Notice that for p(w|c) we are still using the probabilities off the training set!\n",
    "\n",
    "\n",
    "# Similar to before, but we don't call fit again! We use the same vocabulary as in the training set\n",
    "test_csr = vect.transform(test_data.Text)\n",
    "test_df = pd.DataFrame( test_csr.toarray(), columns=vect.get_feature_names() )\n",
    "\n",
    "\n",
    "all_true_labels = test_data.Label\n",
    "all_pred_labels = [\"\"]*len(test_data.Label)\n",
    "\n",
    "p_c_spam = np.float(doc_spam) / np.float(doc_total)\n",
    "p_c_ham  = np.float(doc_ham) / np.float(doc_total)\n",
    "\n",
    "for row_idx in range(test_data.shape[0]):\n",
    "    \n",
    "    # Save a running log probability \n",
    "    logP_spam = np.float(0.0)\n",
    "    logP_ham  = np.float(0.0)\n",
    "    \n",
    "    words = test_df.iloc[row_idx,:]\n",
    "    words = words[ words > 0 ]\n",
    "\n",
    "    for word in words.index:\n",
    "        cnt = np.float(words[word])\n",
    "        \n",
    "        p_word_spam = 0.0\n",
    "        if word in word_count_spam.index:\n",
    "            p_word_spam = (word_count_spam[word] + 1.0) / ( np.float(word_total_spam) + V_train )\n",
    "        else:\n",
    "            # Give it the laplace smoothed prior!\n",
    "            p_word_spam = 1.0 / ( np.float(word_total_spam) + V_train )\n",
    "\n",
    "        p_word_ham = 0.0\n",
    "        if word in word_count_ham.index:\n",
    "            p_word_ham = (word_count_ham[word] + 1.0) / ( np.float(word_total_ham) + V_train )\n",
    "        else:\n",
    "            # Give it the laplace smoothed prior!\n",
    "            p_word_ham = 1.0 / ( np.float(word_total_ham) + V_train )\n",
    "        \n",
    "        logP_spam = logP_spam + cnt * np.log(p_word_spam)\n",
    "        logP_ham  = logP_ham  + cnt * np.log(p_word_ham)\n",
    "        \n",
    "    logP_spam = logP_spam + np.log(p_c_spam)\n",
    "    logP_ham = logP_ham + np.log(p_c_ham)\n",
    "    \n",
    "    if logP_spam > logP_ham: \n",
    "        all_pred_labels[row_idx] = \"Spam\"\n",
    "    else:\n",
    "        all_pred_labels[row_idx] = \"Ham\"\n",
    "\n",
    "\n",
    "print \"Accuracy:\", np.sum(all_pred_labels == all_true_labels)/ np.float(len(all_true_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That was quite a lot of work (and probably poor coding) to create a multinomial Naive Bayes solution. You'll have a chance later on the improve upon this and write your own version."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SKLearn `naive_bayes`\n",
    "\n",
    "For now, let's also see how SKLearn's implementation of naive bayes operates.\n",
    "\n",
    "There are many variants of Naive Bayes in the module, and they all mostly differ in how they encode P(x|c) (or the sampling model)\n",
    "* MultinomialNB() : What we've learned so far, where multiple counts of words increase the probability of a class\n",
    "* BernoulliNB(): Where the presence or absence (1 and 0s, not counts) will increase or decrease the probability of a class\n",
    "* GaussianNB(): Where the probability of a class is based on how far this feature is from the center of the class. This would be good for continuous normally distributed features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "\n",
    "vect = CountVectorizer()\n",
    "vect.fit(train_data.Text)\n",
    "\n",
    "mm = MultinomialNB()\n",
    "\n",
    "\n",
    "train_csr = vect.transform(train_data.Text)\n",
    "y_train = train_data.Label\n",
    "\n",
    "mm.fit(train_csr,y_train)\n",
    "print \"Training:\", mm.score(train_csr,y_train)\n",
    "\n",
    "test_csr = vect.transform(test_data.Text)\n",
    "y_test = test_data.Label\n",
    "print \"Testing:\", mm.score(test_csr,y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import BernoulliNB\n",
    "\n",
    "vect = CountVectorizer()\n",
    "vect.fit(train_data.Text)\n",
    "\n",
    "mm = BernoulliNB()\n",
    "\n",
    "\n",
    "train_csr = vect.transform(train_data.Text)\n",
    "y_train = train_data.Label\n",
    "\n",
    "mm.fit(train_csr,y_train)\n",
    "print \"Training:\", mm.score(train_csr,y_train)\n",
    "\n",
    "test_csr = vect.transform(test_data.Text)\n",
    "y_test = test_data.Label\n",
    "print \"Testing:\", mm.score(test_csr,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ngrams\n",
    "\n",
    "With CountVectorizer, we don't need to only have single words as tokens. We can have phrases of any length as tokens.\n",
    "\n",
    "For instance, with the default way, the sentence \"DONT BUY\" would be tokenized into \"DONT\", \"BUY\". However, we would think the total phrase of length 2 (aka 2-gram, or bi-gram), is important.\n",
    "\n",
    "Using the ngram_range feature in CountVectorizer, we can specify the range of n-grams we want. For instance, ngram_range=(1,5), would generate n-grams of length 1 (single words), length 2 (2 word phrases), length 3 (3 word phrases), and so on to 5.\n",
    "\n",
    "Be careful! Setting ngram_range too high is not likely to help, and it increases the computing cost greatly!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Similar to above, but I've used the ngram_range argument in creating the CountVectorizer\n",
    "ngram2vect = CountVectorizer(ngram_range=(1,2))\n",
    "ngram2vect.fit(train_data.Text)\n",
    "ngram2_fn = ngram2vect.get_feature_names()\n",
    "ngram2_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some helpful tools in diagnosing your models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "vect = CountVectorizer()\n",
    "vect.fit(train_data.Text)\n",
    "\n",
    "train_csr = vect.transform(train_data.Text)\n",
    "y_train = train_data.Label\n",
    "\n",
    "mm = MultinomialNB()\n",
    "mm.fit(train_csr,y_train)\n",
    "\n",
    "print mm.score(train_csr,y_train)\n",
    "print mm.class_count_\n",
    "print mm.class_log_prior_\n",
    "print mm.class_prior\n",
    "print mm.classes_\n",
    "print mm.predict_log_proba(train_csr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.cross_validation import cross_val_score\n",
    "cross_val_score(MultinomialNB(), train_csr, y_train, cv=5, scoring=\"accuracy\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}