14_dimension/14_03_lsa.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SVD and Latent Semantic Analysis (LSA)\n",
    "\n",
    "Singular Value Decomposition (SVD) applied to text documents is often referred to as _Latent Semantic Analysis_ (LSA).\n",
    "\n",
    "Let's $A$ be our $N \\times n$ feature matrix with $N$ samples and $n$ features, and let's write the SVD as follows, trunacted to some dimension $k < n$:\n",
    "\n",
    "\\begin{align}\n",
    "A = U \\Sigma V^{T}\n",
    "\\end{align}\n",
    "\n",
    "Each of these matrices represents some grouping the data:\n",
    "- $U$ ($N \\times k$) represents how each datapoint relates to some latent concept,\n",
    "- $V$ ($n \\times k$) represents how each concept relates to each original feature ($V^T$ would be $k \\times n$) \n",
    "- $\\Sigma$ ($k \\times k$) represents the strength of each of those concepts in the dataset.\n",
    "\n",
    "This reduces the number of dimensions from $n$ to $k$, since instead of modelling our data for each feature, we now model our data for each concept. Our intuition is that datapoints with similar relevant features will get similar concepts.\n",
    "\n",
    "A very concrete example of this is text classification, in which case this method is often called _Latent Semantic Analysis_ (LSA).  As always with Natural Language Processing (NLP), we transform our $N$ documents into a $N \\times n$ feature matrix, where each column represents a certain word or n-gram (e.g., using `CountVectorizer`).  \n",
    "\n",
    "This notebook demonstrates SVD and LSA with a simple example.\n",
    "\n",
    "\n",
    "This example is over text, but really, you could even use this for **recommendation systems!**. If you had a matrix of users x items, where each entry is the # that they purchase, you could apply the same technique."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import code and data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.datasets import fetch_20newsgroups\n",
    "from sklearn.decomposition import TruncatedSVD\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import Normalizer\n",
    "import seaborn\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "categories = ['alt.atheism', 'talk.religion.misc', 'comp.graphics', 'sci.space',]\n",
    "dataset = fetch_20newsgroups(subset='all', categories=categories, shuffle=True)\n",
    "\n",
    "documents = dataset.data\n",
    "y = dataset.target\n",
    "\n",
    "true_k = np.unique(y).shape[0]\n",
    "\n",
    "print \"There are %d true labels\" % (true_k)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Feature extraction\n",
    "Extract features from the training dataset using a sparse vectorizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Remove english stop words. Remove words that \n",
    "vectorizer = CountVectorizer(stop_words='english', max_features=10000, min_df=.01, max_df=0.5)\n",
    "X = vectorizer.fit_transform(documents)\n",
    "n_samples, n_features = X.shape\n",
    "print \"Dimensions feature matrix:\", X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dimension reduction\n",
    "We could reduce the number of dimensions by performing a SVD. We hope that we could describe most of the data in a much lower-dimensional space, in which the dimensions represent topics (or concepts, or _latent semantics_) rather than just words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_components = 150"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "svd = TruncatedSVD(n_components)\n",
    "lsa = make_pipeline(svd, Normalizer(copy=False))\n",
    "\n",
    "X_red = lsa.fit_transform(X)\n",
    "\n",
    "explained_variance = svd.explained_variance_ratio_.sum()\n",
    "\n",
    "print \"Explained variance of the SVD step: %d%%\" % (explained_variance * 100)\n",
    "print \"Dimensions feature matrix (old):\", X.shape\n",
    "print \"Dimensions feature matrix (new):\", X_red.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "f = plt.plot(svd.explained_variance_ratio_.cumsum())  # these are the singular values \n",
    "f = plt.title(\"Explained variance by number of components, total: %d\" % svd.explained_variance_ratio_.shape[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, we can describe almost 90% of our variance by just 150 dimensions, and even better, 50% of our variance by only a handful of components.\n",
    "\n",
    "Each component is made out of a certain word combination with different weights. Let's have a look at themost important components."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "words = np.array(vectorizer.get_feature_names())\n",
    "top_words = 10\n",
    "n_rows, n_cols = 2, 4\n",
    "top_components = n_rows * n_cols\n",
    "f, axes = plt.subplots(n_rows, n_cols, figsize=(20, 7))\n",
    "for no, component in enumerate(svd.components_[:top_components]):\n",
    "    ax = axes[no / n_cols][no % n_cols]\n",
    "    s = pd.Series(svd.components_[no], index=words).sort(inplace=False)[-top_words:]\n",
    "    s.plot(kind='barh', ax=ax, title=\"Component %d\\nexplained variance %4.1f\" % \n",
    "           (no, 100 * svd.explained_variance_ratio_[no]))\n",
    "f = plt.tight_layout()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Very clear results! \n",
    "\n",
    "Let's look at our documents and see how an article is represented in these dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "f, axes = plt.subplots(1, true_k, figsize=(20, 4))\n",
    "for label in xrange(true_k):\n",
    "    X_red_class = X_red[y == label]\n",
    "    no = np.abs(X_red_class[:, :top_components]).max(axis=1).argmax()  # pick most outspoken sample\n",
    "    f = pd.Series(X_red_class[no,:top_components]).plot(\n",
    "        title=\"Most outspoken article in %s\" % dataset.target_names[label],\n",
    "        kind='bar', rot=0, ylim=(0, 1), ax=axes[label])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed, the article has stronger representation in the component that resembles its category.\n",
    "\n",
    "This is just data magic!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}