Project+2+Sentiment+Analysis+with+TFLearn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment analysis with TFLearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/new/workspace/anaconda3/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import tflearn\n",
    "from tflearn.data_utils import to_categorical"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "                                                   0\n",
      "0  bromwell high is a cartoon comedy . it ran at ...\n",
      "1  story of a man who has unnatural feelings for ...\n",
      "2  homelessness  or houselessness as george carli...\n",
      "3  airport    starts as a brand new luxury    pla...\n",
      "4  brilliant over  acting by lesley ann warren . ...\n"
     ]
    }
   ],
   "source": [
    "reviews = pd.read_csv('reviews.txt', header=None)\n",
    "labels = pd.read_csv('labels.txt', header=None)\n",
    "\n",
    "print(type(reviews))\n",
    "print(reviews.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Counting word frequency"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total words in data set:  74074\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "total_counts = Counter()\n",
    "\n",
    "for idx, row in reviews.iterrows():\n",
    "    review = row[0]\n",
    "    for word in review.split(' '):\n",
    "        total_counts[word] += 1\n",
    "\n",
    "print(\"Total words in data set: \", len(total_counts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's keep the first 10000 most frequent words. As Andrew noted, most of the words in the vocabulary are rarely used so they will have little effect on our predictions. Below, we'll sort `vocab` by the count value and keep the 10000 most frequent words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['', 'the', '.', 'and', 'a', 'of', 'to', 'is', 'br', 'it', 'in', 'i', 'this', 'that', 's', 'was', 'as', 'for', 'with', 'movie', 'but', 'film', 'you', 'on', 't', 'not', 'he', 'are', 'his', 'have', 'be', 'one', 'all', 'at', 'they', 'by', 'an', 'who', 'so', 'from', 'like', 'there', 'her', 'or', 'just', 'about', 'out', 'if', 'has', 'what', 'some', 'good', 'can', 'more', 'she', 'when', 'very', 'up', 'time', 'no']\n"
     ]
    }
   ],
   "source": [
    "vocab = sorted(total_counts, key=total_counts.get, reverse=True)[:10000]\n",
    "print(vocab[:60])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What's the last word in our vocabulary? We can use this to judge if 10000 is too few. If the last word is pretty common, we probably need to keep more words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fulfilled :  30\n"
     ]
    }
   ],
   "source": [
    "print(vocab[-1], ': ', total_counts[vocab[-1]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last word in our vocabulary shows up in 30 reviews out of 25000. I think it's fair to say this is a tiny proportion of reviews. We are probably fine with this number of words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "word2idx = {word: index for index, word in enumerate(vocab)}## create the word-to-index dictionary here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text to vector function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def text_to_vector(text):\n",
    "    vector = np.zeros(len(vocab))\n",
    "    for word in text.split(' '):\n",
    "        index = word2idx.get(word, None)\n",
    "        if index:\n",
    "            vector[index] += 1\n",
    "    return vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you do this right, the following code should return\n",
    "\n",
    "```\n",
    "text_to_vector('The tea is for a party to celebrate '\n",
    "               'the movie so she has no time for a cake')[:65]\n",
    "                   \n",
    "array([0, 1, 0, 0, 2, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 1, 0, 0, 0,\n",
    "       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,\n",
    "       0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0])\n",
    "```       "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.,  1.,  0.,  0.,  2.,  0.,  1.,  1.,  0.,  0.,  0.,  0.,  0.,\n",
       "        0.,  0.,  0.,  0.,  2.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,\n",
       "        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,\n",
       "        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,\n",
       "        0.,  0.,  1.,  0.,  0.,  0.,  1.,  1.,  0.,  0.,  0.,  0.,  0.])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_to_vector('The tea is for a party to celebrate '\n",
    "               'the movie so she has no time for a cake')[:65]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, run through our entire review data set and convert each review to a word vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "word_vectors = np.zeros((len(reviews), len(vocab)), dtype=np.int_)\n",
    "for ii, (_, text) in enumerate(reviews.iterrows()):\n",
    "    word_vectors[ii] = text_to_vector(text[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0,  9, 27,  1,  4,  4,  6,  4,  0,  2,  2,  5,  0,  4,  1,  0,  2,\n",
       "         0,  0,  0,  0,  0,  0],\n",
       "       [ 0,  4,  8,  1,  7,  3,  1,  2,  0,  4,  0,  0,  0,  1,  2,  0,  0,\n",
       "         1,  3,  0,  0,  0,  1],\n",
       "       [ 0, 24, 12,  4, 17,  5, 20,  2,  8,  8,  2,  1,  1,  2,  8,  0,  5,\n",
       "         5,  4,  0,  2,  1,  4],\n",
       "       [ 0, 53, 23,  0, 22, 23, 13, 14,  8, 10,  8, 12,  9,  4, 11,  2, 11,\n",
       "         5, 11,  0,  5,  3,  0],\n",
       "       [ 0, 10, 11,  4,  6,  2,  2,  5,  0,  1,  2,  3,  1,  0,  0,  0,  3,\n",
       "         1,  0,  1,  0,  0,  0]])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Printing out the first 5 word vectors\n",
    "word_vectors[:5, :23]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train, Validation, Test sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Y = (labels=='positive').astype(np.int_)\n",
    "records = len(labels)\n",
    "shuffle = np.arange(records)\n",
    "np.random.shuffle(shuffle)\n",
    "test_fraction = 0.9\n",
    "\n",
    "train_split, test_split = shuffle[:int(records*test_fraction)], shuffle[int(records*test_fraction):]\n",
    "trainX, trainY = word_vectors[train_split,:], to_categorical(Y.values[train_split], 2)\n",
    "testX, testY = word_vectors[test_split,:], to_categorical(Y.values[test_split], 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.,  0.],\n",
       "       [ 0.,  1.],\n",
       "       [ 1.,  0.],\n",
       "       ..., \n",
       "       [ 0.,  1.],\n",
       "       [ 1.,  0.],\n",
       "       [ 1.,  0.]])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainY"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the network\n",
    "\n",
    "\n",
    "### Input layer\n",
    "\n",
    "\n",
    "### Adding layers\n",
    "\n",
    "\n",
    "### Output layer\n",
    "\n",
    "\n",
    "### Training\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Network building\n",
    "def build_model(learning_rate=0.1):\n",
    "    # This resets all parameters and variables, leave this here\n",
    "    tf.reset_default_graph()\n",
    "    \n",
    "    #### Your code ####\n",
    "    net = tflearn.input_data([None, len(vocab)])\n",
    "    net = tflearn.fully_connected(net, 5, activation='ReLU')\n",
    "    net = tflearn.fully_connected(net, 2, activation='softmax')\n",
    "    net = tflearn.regression(net, optimizer='sgd', learning_rate=learning_rate, loss='categorical_crossentropy')\n",
    "    model = tflearn.DNN(net)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intializing the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = build_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Step: 17489  | total loss: \u001b[1m\u001b[32m0.16024\u001b[0m\u001b[0m | time: 4.163s\n",
      "| SGD | epoch: 110 | loss: 0.16024 - acc: 0.9410 -- iter: 20224/20250\n",
      "Training Step: 17490  | total loss: \u001b[1m\u001b[32m0.15608\u001b[0m\u001b[0m | time: 5.199s\n",
      "| SGD | epoch: 110 | loss: 0.15608 - acc: 0.9430 | val_loss: 0.34777 - val_acc: 0.8769 -- iter: 20250/20250\n",
      "--\n"
     ]
    }
   ],
   "source": [
    "# Training\n",
    "model.fit(trainX, trainY, validation_set=0.1, show_metric=True, batch_size=128, n_epoch=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing\n",
    "\n",
    "After you're satisified with your hyperparameters, you can run the network on the test set to measure it's performance. Remember, *only do this after finalizing the hyperparameters*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy:  0.8728\n"
     ]
    }
   ],
   "source": [
    "predictions = (np.array(model.predict(testX))[:,0] >= 0.5).astype(np.int_)\n",
    "test_accuracy = np.mean(predictions == testY[:,0], axis=0)\n",
    "print(\"Test accuracy: \", test_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Try out your own text!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Helper function that uses your model to predict sentiment\n",
    "def test_sentence(sentence):\n",
    "    positive_prob = model.predict([text_to_vector(sentence.lower())])[0][1]\n",
    "    print('Sentence: {}'.format(sentence))\n",
    "    print('P(positive) = {:.3f} :'.format(positive_prob), \n",
    "          'Positive' if positive_prob > 0.5 else 'Negative')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentence: Moonlight is by far the best movie of 2016.\n",
      "P(positive) = 0.834 : Positive\n",
      "Sentence: It's amazing anyone could be talented enough to make something this spectacularly awful\n",
      "P(positive) = 0.070 : Negative\n"
     ]
    }
   ],
   "source": [
    "sentence = \"Moonlight is by far the best movie of 2016.\"\n",
    "test_sentence(sentence)\n",
    "\n",
    "sentence = \"It's amazing anyone could be talented enough to make something this spectacularly awful\"\n",
    "test_sentence(sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}