03-NLTK.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NLTK\n",
    "\n",
    "NLTK is a large collection of NLP tools.We won't have time to cover everything, so we'll focus on the most common tools:\n",
    "\n",
    "[Existing corpora](#existing)<br>\n",
    "\n",
    "[Tokenization](#tokenization)<br>\n",
    "\n",
    "[Sentence segmentation](#sent-seg)<br>\n",
    "\n",
    "[Collocations](#collocations)<br>\n",
    "\n",
    "[Sentiment analysis](#sentiment)<br>\n",
    "\n",
    "[Stemming](#stemming)<br>\n",
    "\n",
    "[What we didn't cover](#didnt)<br>\n",
    "\n",
    "### Time\n",
    "- Teaching: 30 minutes\n",
    "- Exercises: 30 minutes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import os\n",
    "import nltk\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "sns.set()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Existing corpora <a id='existing'></a>\n",
    "\n",
    "When you downloaded data from nltk using `nltk.download('all')`, you downloaded a whole bunch of great corpora (collections of text documents) and lexical resources (structured information about words). This gives us data to work with already! If you ever want to learn/practice an NLP method, know that just by importing nltk you have access to some data. Here are some corpora and resources that are particularly useful and that we'll use throughout this workshop:\n",
    "\n",
    "- ABC\n",
    "- Brown\n",
    "- CMU pronunciation dictionary\n",
    "- Genesis\n",
    "- Project Gutenberg selections\n",
    "- Inaugural addresses\n",
    "- Movie reviews\n",
    "- Names\n",
    "- State of the Union addresses\n",
    "- Stopwords\n",
    "- Twitter samples\n",
    "- Universal Declaration of Human Rights\n",
    "- WordNet\n",
    "\n",
    "Full list of data in NLTK [here](http://www.nltk.org/nltk_data/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import (abc, brown, cmudict, genesis, gutenberg,\n",
    "                         inaugural, movie_reviews, names, state_union, \n",
    "                         stopwords, swadesh, twitter_samples, udhr2, \n",
    "                         wordnet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Corpora in NLTK are special objects in NLTK that give you the exact data you want only when you ask for it. For example, `brown` is not a string or a list of words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "brown"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Words, raw, sents, fileids\n",
    "\n",
    "But if I wanted the Brown corpus as a list of words, I could ask for it like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "brown.words()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, if I wanted the text of the ABC corpus as a string, I could get it like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "abc.raw()[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you wanted the sentences of a corpus, you can ask for them like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movie_reviews.sents()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These corpora are often made up of multiple files. You can see the file names by using the `fileids` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "names.fileids()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To restrict the words, raw or sents to just the words/raw/sents in a particular file, you can list the file name as an optional argument to the `words`/`raw`/`sents` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "male_names = names.words('male.txt')\n",
    "male_names[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Unique properties\n",
    "\n",
    "Some corpora have unique aspects to them. For example, the CMU pronunciation dictionary lists (some standard) pronunciation of English words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pronunciation = cmudict.dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pronunciation['hello']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Male vs. female names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "male_names = names.words('male.txt')\n",
    "female_names = names.words('female.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def last_letter(name):\n",
    "    \"\"\"Returns the last letter of `name`.\"\"\"\n",
    "    return name.strip()[-1]\n",
    "\n",
    "def count_letters(names):\n",
    "    \"\"\"Returns the distribution of the last letters in `names`.\"\"\"\n",
    "    return pd.Series([last_letter(n) for n in names]).value_counts(normalize=True)\n",
    "\n",
    "def letter_distribution():\n",
    "    male_value_counts = count_letters(male_names)\n",
    "    female_value_counts = count_letters(female_names)\n",
    "    return pd.DataFrame.from_dict({'male': male_value_counts, 'female': female_value_counts})\n",
    "\n",
    "df = letter_distribution()\n",
    "df.plot(kind='bar', figsize=(16, 8))\n",
    "plt.legend(prop={'size': 20})\n",
    "plt.xticks(rotation=0, size=20);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Challenge\n",
    "\n",
    "- Count the lengths of the sentences (i.e. the number of words per sentence) in the `inaugural` corpus. Find the minimum, average and maximum sentence length.\n",
    "- Visualize the distribution of lengths.\n",
    "- Count the number of times the following words appear in the corpus: \"america\", \"citizen\", \"united\", \"senate\" and \"freedom\".\n",
    "- If you are surprised by anything in the answer to the last question, think about capitalization issues. Make all words lowercase and then perform your counts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your answer goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your answer goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for word in [\"america\", \"citizen\", \"united\", \"senate\", \"freedom\"]:\n",
    "    # your answer goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your answer goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization <a id='tokenization'></a>\n",
    "\n",
    "More often than not, you'll want to analyze some text that doesn't come from NLTK. Perhaps you've scraped a few websites and stored the text in a text file. One of the first steps in processing your text data is tokenization. **Tokenization refers to breaking a running string of text into individual words.**\n",
    "\n",
    "I've download the text contents of the Wikipedia page on [Python][1], and saved it in the `data` directory. We can read it in as follows:\n",
    "\n",
    "[1]: https://en.wikipedia.org/wiki/Python_(programming_language)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_DIR = 'data'\n",
    "python_wiki_fname = os.path.join(DATA_DIR, 'python_wikipedia.txt')\n",
    "with open(python_wiki_fname) as f:\n",
    "    text = f.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, `text` is a string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can tokenize this string by using nltk's `word_tokenize` function, which returns a list of strings. Each string is either a word or a punctuation symbol."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = nltk.word_tokenize(text)\n",
    "tokens[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This uses NLTK's recommended tokenizer. There are plenty of [other tokenizers in NLTK](https://github.com/nltk/nltk/tree/develop/nltk/tokenize), but unless you have good reason to do otherwise it's best to stick to the recommended tokenizer.\n",
    "\n",
    "### Challenge\n",
    "\n",
    "I've also downloaded the Wikipedia page for [Berkeley, California][2], and saved the contents as a file called 'berkeley_wikipedia.txt'. Borrowing from the code above, read this file in and tokenize the text. Then find the 10 most frequenct \"words\". After that, if you don't like counting punctuation symbols as \"words\", then remove all punctuation symbols then find the 10 most frequenct words.\n",
    "\n",
    "[2]: https://en.wikipedia.org/wiki/Berkeley,_California"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your answer goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from string import punctuation\n",
    "# your answer goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentence segmentation <a id='sent-seg'></a>\n",
    "\n",
    "Sentence segmentation refers to finding the beginnings and ends of sentences. It's also sometimes called sentence tokenization. Again, there are lots of ways in NLTK to do this, but they have conviently chosen a default method for us. The `nltk.sent_tokenize` function takes in a string and returns a list of strings, where each string is a sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sents = nltk.sent_tokenize(text)\n",
    "sents[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collocations <a id='collocations'></a>\n",
    "\n",
    "Collocations are words that frequently appear together. They can help us identify key phrases in a text. Collocations can be bigrams (two words), tri-grams (three) or 4-grams. In NLTK, we can use the `BigramCollocationFinder` to find all the bigram collocations in a text. First, we feed in the tokenized text. Here, we'll use the 'learned' portion of the Brown corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = brown.words(categories='learned')\n",
    "collocations = nltk.BigramCollocationFinder.from_words(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we decide which words to filter out. I don't want words less than three characters or stopwords."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ignored_words = stopwords.words('english')\n",
    "word_filter = lambda w: len(w) < 3 or w.lower() in ignored_words\n",
    "collocations.apply_freq_filter(3)\n",
    "collocations.apply_word_filter(word_filter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we decide what method NLTK should use to decide what makes a collocation special. We'll use the likelihood ratio, which is a good standard choice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scorer = nltk.collocations.BigramAssocMeasures.likelihood_ratio\n",
    "collocations.nbest(scorer, 15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This was kinda messy. We can wrap all this up into a nicer function that just takes in the tokens and spits out the collocations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_collocations(tokens):\n",
    "    collocations = nltk.BigramCollocationFinder.from_words(tokens)\n",
    "    ignored_words = stopwords.words('english')\n",
    "    word_filter = lambda w: len(w) < 3 or w.lower() in ignored_words\n",
    "    collocations.apply_freq_filter(3)\n",
    "    collocations.apply_word_filter(word_filter)\n",
    "    scorer = nltk.collocations.BigramAssocMeasures.likelihood_ratio\n",
    "    return collocations.nbest(scorer, 15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now run `my_collocations` on some new text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_collocations(state_union.words())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "emma = gutenberg.words('austen-emma.txt')\n",
    "my_collocations(emma)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "my_collocations(genesis.words('english-kjv.txt'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentiment analysis <a id='sentiment'></a>\n",
    "\n",
    "NLTK has support for sentiment analysis. [Sentiment analysis](https://en.wikipedia.org/wiki/Sentiment_analysis) is the task of extracting [affective states](https://en.wikipedia.org/wiki/Affect_(psychology)) from text. The VADER (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media. There was a [Python package](https://github.com/cjhutto/vaderSentiment) developed for it outside of NLTK, which was then incorporated into NLTK. Loading it through NLTK is often buggy, but we can install the original package if it fails through NLTK. It ends up working the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.sentiment import SentimentIntensityAnalyzer\n",
    "try:\n",
    "    sentiment = SentimentIntensityAnalyzer()\n",
    "except LookupError:\n",
    "    print('Sentiment analysis in NLTK is not working at the moment :(')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the `SentimentIntensityAnalyzer` isn't loading properly from `nltk`, then you'll have to install the original package using the line below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U vaderSentiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then import it like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whether you used NLTK's `SentimentIntensityAnalyzer` or gor it from `vaderSentiment`, the rest of the code is identical."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentiment = SentimentIntensityAnalyzer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Analyzing a sentence for its sentiment returns a dictionary with four items. The `compound` key holds the overall score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence = \"I hate this sentence so much. I just want it to end. It sucks!\"\n",
    "sentiment.polarity_scores(sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentences = [\"VADER is smart, handsome, and funny.\",      # positive sentence example\n",
    "            \"VADER is not smart, handsome, nor funny.\",   # negation sentence example\n",
    "            \"VADER is smart, handsome, and funny!\",       # punctuation emphasis handled correctly (sentiment intensity adjusted)\n",
    "            \"VADER is very smart, handsome, and funny.\",  # booster words handled correctly (sentiment intensity adjusted)\n",
    "            \"VADER is VERY SMART, handsome, and FUNNY.\",  # emphasis for ALLCAPS handled\n",
    "            \"VADER is VERY SMART, handsome, and FUNNY!!!\",# combination of signals - VADER appropriately adjusts intensity\n",
    "            \"VADER is VERY SMART, uber handsome, and FRIGGIN FUNNY!!!\",# booster words & punctuation make this close to ceiling for score\n",
    "            \"The book was good.\",                                     # positive sentence\n",
    "            \"The book was kind of good.\",                 # qualified positive sentence is handled correctly (intensity adjusted)\n",
    "            \"The plot was good, but the characters are uncompelling and the dialog is not great.\", # mixed negation sentence\n",
    "            \"At least it isn't a horrible book.\",         # negated negative sentence with contraction\n",
    "            \"Make sure you :) or :D today!\",              # emoticons handled\n",
    "            \"Today SUX!\",                                 # negative slang with capitalization emphasis\n",
    "            \"Today only kinda sux! But I'll get by, lol\"  # mixed sentiment example with slang and constrastive conjunction \"but\"\n",
    "             ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = []\n",
    "for sent in sentences:\n",
    "    score = sentiment.polarity_scores(sent)\n",
    "    scores.append(score)\n",
    "df = pd.DataFrame(scores)\n",
    "df['sentence'] = sentences\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> _The compound score is computed by summing the valence scores of each word in the lexicon, adjusted according to the rules, and then normalized to be between -1 (most extreme negative) and +1 (most extreme positive). This is the most useful metric if you want a single unidimensional measure of sentiment for a given sentence. Calling it a 'normalized, weighted composite score' is accurate._\n",
    "\n",
    "> _It is also useful for researchers who would like to set standardized thresholds for classifying sentences as either positive, neutral, or negative._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['positive_sentiment'] = df['compound'] >= 0.5\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Challenge\n",
    "\n",
    "I've read in a bunch of tweets from Trump, and stored them as a list of strings in `tweet_text`. Use the code from above to find the positive sentiment tweets and save them to a list called `positive_tweets`. Do the same for negative tweets, storing them in a variable called `negative_tweets`. What's the proportion of positive to negative tweets?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets_fname = os.path.join(DATA_DIR, 'trump-tweets.csv')\n",
    "tweets = pd.read_csv(tweets_fname)\n",
    "tweet_text = list(tweets['Tweet_Text'].values)\n",
    "tweet_text[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your answer goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your answer goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stemming <a id='stemming'></a>\n",
    "\n",
    "Stemming and lemmatization both refer to removing morphological affixes on words. For example, if we stem the word \"grows\", we get \"grow\". If we stem the word \"running\", we get \"run\". We do this because often we care more about the core content of the word (i.e. that it has something to do with growth or running, rather than the fact that it's a third person present tense verb, or progressive participle).\n",
    "\n",
    "NLTK provides many algorithms for stemming. For English, a great baseline is the [Porter algorithm](https://tartarus.org/martin/PorterStemmer/), which is in spirit isn't that far from a bunch of regular expressions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem import PorterStemmer\n",
    "stemmer = PorterStemmer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stemmer.stem('grows')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stemmer.stem('running')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stemmer.stem('leaves')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NLTK has a variety of other stemming algorithms, and lemmatizers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem import SnowballStemmer, WordNetLemmatizer\n",
    "snowball = SnowballStemmer('english')\n",
    "lemmatizer = WordNetLemmatizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(snowball.stem('running'))\n",
    "print(snowball.stem('eats'))\n",
    "print(snowball.stem('embarassed'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But watch out for errors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Thanks to Chris Hench for these examples\n",
    "print(snowball.stem('cylinder'))\n",
    "print(snowball.stem('cylindrical'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And collisions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Thanks to Chris Hench for these examples\n",
    "print(snowball.stem('vacation'))\n",
    "print(snowball.stem('vacate'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(lemmatizer.lemmatize('vacation'))\n",
    "print(lemmatizer.lemmatize('vacate'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But why would you want to stem words in the first place? Well, stemming improves performance!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Thanks again to Chris Hench for inspiration of this example\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", category=DeprecationWarning)\n",
    "warnings.filterwarnings(\"ignore\", category=FutureWarning)\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Don't worry about following along with this code, although it's great if you do!\n",
    "def read_data():\n",
    "    airline_fname = 'airline_tweets.csv'\n",
    "    airline_fname = os.path.join(DATA_DIR, airline_fname)\n",
    "    df = pd.read_csv(airline_fname)\n",
    "    twitter_handle_pattern = r'@(\\w+)'\n",
    "    hashtag_pattern = r'(?:^|\\s)[＃#]{1}(\\w+)'\n",
    "    url_pattern = r'https?:\\/\\/.*.com'\n",
    "    df['clean_text'] = (df['text']\n",
    "                        .str.replace(hashtag_pattern, 'HASHTAG')\n",
    "                        .str.replace(twitter_handle_pattern, 'USER')\n",
    "                        .str.replace(url_pattern, 'URL')\n",
    "                              )\n",
    "    text = list(df['clean_text'].str.lower())\n",
    "    sentiment = list(df['airline_sentiment'])\n",
    "    return text, sentiment\n",
    "\n",
    "def prepare_stems(sents):\n",
    "    snowball = SnowballStemmer('english')\n",
    "    tokenized_sents = [nltk.word_tokenize(s) for s in sents]\n",
    "    stemmed_sents = [[snowball.stem(s) for s in tokenized_sent] for tokenized_sent in tokenized_sents]\n",
    "    return [' '.join(sent) for sent in stemmed_sents]\n",
    "\n",
    "def prepare_no_stems(sents):\n",
    "    tokenized_sents = [nltk.word_tokenize(s) for s in sents]\n",
    "    return [' '.join(sent) for sent in tokenized_sents]\n",
    "\n",
    "def fit_model(X_train, y_train):\n",
    "    model = RandomForestClassifier(n_estimators=10, criterion='gini')                \n",
    "    model.fit(X_train, y_train)\n",
    "    return model\n",
    "\n",
    "def test_model(model, X_test, y_test):\n",
    "    print('Accuracy: ', model.score(X_test, y_test))\n",
    "\n",
    "def classify(sents, target):\n",
    "    vectorizer = TfidfVectorizer(max_features=5000, binary=True)\n",
    "    X = vectorizer.fit_transform(sents)\n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, target, test_size=0.25, random_state=42)\n",
    "    model = fit_model(X_train, y_train)\n",
    "    test_model(model, X_test, y_test)\n",
    "\n",
    "text, sentiment = read_data()\n",
    "stemmed_text = prepare_stems(text)\n",
    "unstemmed_text = prepare_no_stems(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classify(stemmed_text, sentiment)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classify(unstemmed_text, sentiment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What we didn't cover <a id='didnt'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distance\n",
    "\n",
    "NLTK has some functionality for calculating the distance between two strings. String distance is a measure of how different two strings are. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nltk.edit_distance('hello', 'helo')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nltk.edit_distance('hello', 'hi')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are lots of different ways to measure edit distance. This method uses Levenshtein distance, which is the number of insertions, deletions and substitutions required to turn one string into another. Edit distance is useful if you're looking for spelling mistakes.\n",
    "\n",
    "The [fuzzywuzzy library](https://github.com/seatgeek/fuzzywuzzy) does a great job of edit distance too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U fuzzywuzzy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'this is a test' == 'this is a test!'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fuzzywuzzy import fuzz\n",
    "fuzz.ratio('this is a test', 'this is a test!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Translation\n",
    "\n",
    "NLTK offers [some tools](https://github.com/nltk/nltk/tree/develop/nltk/translate) for machine translation. This is great for learning traditional translation models, but is out-dated. If you actually need to translate some text, currently I'd highly using the [Google Translate API](https://cloud.google.com/translate/docs/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text classification\n",
    "\n",
    "NLTK has [support for text classification](https://github.com/nltk/nltk/tree/develop/nltk/classify) using machine learning. However, I'd recommend using [scikit-learn](http://scikit-learn.org/stable/), [TensorFlow](https://www.tensorflow.org/) or [Keras](https://keras.io/) for this now."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chatbots\n",
    "\n",
    "These are mainly just for fun. But check out the [source code](https://github.com/nltk/nltk/tree/develop/nltk/chat) if you're ever interested in building a simple chatbot yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# doesn't work so well in a Jupyter notebook because it requires interaction,\n",
    "# but try it in a terminal or IDE!\n",
    "#nltk.chat.chatbots()"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}