Topic Modling using LDA(21-JAN-2015)(indeed).ipynb

{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Topic Modeling Using LDA\n",
      "Clustering of Documents into N-groups using Latent Dirichlet allocation with 1000 iteration.\n",
      "All Nessary modules are imported bellow."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "subslide"
      }
     },
     "source": [
      "##### LDA Formula\n",
      "![Facebook app page](https://github.com/ahmadfaizalbh/Meetup-Resources/raw/master/ldaformula.png)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from BeautifulSoup import BeautifulSoup\n",
      "from urllib import urlopen, urlencode\n",
      "from os.path import isfile, join\n",
      "from os import listdir\n",
      "from optparse import *\n",
      "import MySQLdb as mdb\n",
      "import numpy as np\n",
      "import datetime\n",
      "import logging\n",
      "import pandas\n",
      "import time\n",
      "import json\n",
      "import lda\n",
      "import re\n",
      "import csv\n",
      "import sys"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "If the system being used under proxy then set proxy with appropriate 'IP:Port' string."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "proxies=None"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Function to obtain list of titles written in a file.\n",
      "where titles are seperated by line."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def Obtain_Title_list(Source_file_name):\n",
      "    f=open(Source_file_name,\"r\")\n",
      "    title_list=f.read().splitlines()\n",
      "    f.close()\n",
      "    return [t for t in title_list if t.strip()]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 3
    },
    {
     "cell_type": "raw",
     "metadata": {},
     "source": [
      "CSV writter function.\n",
      "TAkes a file name and 2 dimentional list and writes them into CSV file."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def write_to_csv(file_Name,S_S):\n",
      "    csv_file=open(file_Name,\"w\")\n",
      "    w=csv.writer(csv_file,dialect='excel')\n",
      "    w.writerows(S_S)\n",
      "    csv_file.close()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Indeed Job search Using INDEED API\n",
      "This function searches specified term in Indeed Using Indeed API and returns the result as list of dictionaries with snippet, title, URL, Published date, etc."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def getJob(**kwargs):\n",
      "    '''getJob(q=job title,start=start from,format=json|xml,...)'''\n",
      "    args = dict(publisher=1288266145526105, v=2,format=\"json\", highlight=0, filter=0, limit=10)\n",
      "    args.update(**kwargs)\n",
      "    url = 'http://api.indeed.com/ads/apisearch?' + urlencode(args)\n",
      "    jobs= json.load(urlopen(url,proxies=proxies))\n",
      "    return jobs['results']"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "function returns content of page/URL specified"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def getpage(url):\n",
      "    return urlopen(url,proxies=proxies).read()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 6
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This function takes a indeed page as HTML and Extracts only Summary part using BeautifulSoup module and returns Extracted summary."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def getSummary(content):\n",
      "    Soup=BeautifulSoup(content)\n",
      "    summary=Soup.find(name=\"span\",attrs={\"class\":\"summary\"})\n",
      "    return summary.getText(separator='. ')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 7
    },
    {
     "cell_type": "raw",
     "metadata": {},
     "source": [
      "Global Stop word list is defined.\n",
      "A global list is declared. this stop word list is updated by a stopword file specified."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "Stop_words={}\n",
      "less_than_3_char_skills=[\"bi\",\"c\",\"c#\",\"e#\",\"es\",\"f#\",\"j#\",\"ml\",\"r\",\"vb\",\"x#\"]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 8
    },
    {
     "cell_type": "raw",
     "metadata": {},
     "source": [
      "Function that returns unique set of terms with its frequency of occurence for each description."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def uniqeSet_with_count(r,l,vocab):\n",
      "\tdoc={}\n",
      "\tfor i in l:\n",
      "\t\tW=r[2].sub(\"\",(i.strip(\"-\\t\\n\\r \") if i else \"\"))\n",
      "\t \tif len(W)>=3 or W in less_than_3_char_skills:\n",
      "\t\t\tif W in doc:\n",
      "\t\t\t\tdoc[W]+=1\n",
      "\t\t\telse:\n",
      "\t\t\t\tif not W in vocab:\n",
      "\t\t\t\t\tvocab.append(W)\n",
      "\t\t\t\tdoc[W]=1\n",
      "\treturn doc"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Function that detects whether a word is stopword or not."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def is_StopWord(x):\n",
      "    l=len(x)\n",
      "    if l in Stop_words and x in Stop_words[l]:\n",
      "        return True\n",
      "    elif \"-\" in x:\n",
      "        return all(is_StopWord(i) for i in x.split(\"-\"))\n",
      "    elif l>5:\n",
      "        return any((is_StopWord(x[:i]) and is_StopWord(x[i:])) for i in range(2,l-1) if (i-1) in Stop_words and (l-i) in Stop_words )\n",
      "    else:\n",
      "        return False"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 10
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Function that updates vocabulary by elemination stopwords. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def update_vocab(V):\n",
      "   return tuple(W for W in  V if not is_StopWord(W.strip(\"-0123456789\")))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 11
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Splits the string into vocabulary."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def split(r,x):\n",
      "    return [i for i in r[0].split(r[1].sub(r\"\\1-\\2\",x)) if i]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 12
    },
    {
     "cell_type": "raw",
     "metadata": {},
     "source": [
      "main function"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def main(input_file=\"topic_list.txt\",Num_topics=10,Stop_word=\"Stop_words.txt\"):\n",
      "    global Stop_words\n",
      "    #r=re.compile(r'[^a-zA-Z#\\+\\-]+')\n",
      "    r=[]\n",
      "    r.append(re.compile(r'[^a-zA-Z#\\+\\-0-9]+|(?<![a-zA-Z])[0-9]+(?![a-zA-Z])'))\n",
      "    r.append(re.compile(r\"([a-zA-Z])\\s*\\-\\s*([a-zA-Z])\"))\n",
      "    r.append(re.compile(r\"^[0-9\\-#\\+]\"))\n",
      "    Stop_word_file=open(Stop_word,\"r\")\n",
      "    for i in sorted(Stop_word_file.read().split(),key=lambda l:len(l)):\n",
      "        length=len(i)\n",
      "        if length in Stop_words:\n",
      "            Stop_words[length].append(i)\n",
      "        else:\n",
      "            Stop_words[length]=[i]\n",
      "    Stop_word_file.close()\n",
      "    vocab=[] #vocab Uniqe words\n",
      "    Titles=Obtain_Title_list(input_file)\n",
      "    #Jobs=get_JobDetails()\n",
      "    Jobs=[]\n",
      "    for title in Titles:\n",
      "        title = re.compile(r'^\\s*|\\s*$').sub('',title)\n",
      "        queted_title = 'title:(\"' + title + '\")' if '\")' not in title else title\n",
      "        Result=getJob(q=queted_title.encode('ascii', 'ignore'),start=0, limit=5)\n",
      "        print title,len(Result)\n",
      "        Jobs+=Result\n",
      "    for i in Jobs:\n",
      "        i['summary']=getSummary(getpage(i[\"url\"]))\n",
      "    documents= [uniqeSet_with_count(r,split(r,J[\"jobtitle\"].lower())+split(r,J[\"summary\"].lower()),vocab) for J in Jobs]#set of all document with word and frequency of occurence\n",
      "    s={i:[k for j in i.split(\"-\") for k in (r[2].sub(\"\",j),) if k in vocab] for i in vocab if \"-\" in i}\n",
      "    s={i:s[i] for i in s if s[i]}\n",
      "    for d in documents:\n",
      "        for w in list(d):\n",
      "            if \"-\" in w:\n",
      "                for nw in [j for i in w.split(\"-\") for j in (r[2].sub(\"\",i),) if j in vocab]:\n",
      "                    if nw in d:\n",
      "                        d[nw]+=1\n",
      "                    else:\n",
      "                        d[nw]=1  \n",
      "        for i in s:\n",
      "            if all([(j in d) for j in s[i]]):\n",
      "                if i in d:\n",
      "                    d[i]+=1\n",
      "                else:\n",
      "                    d[i]=1                 \n",
      "    vocab=update_vocab(vocab)\n",
      "    n_documents= len(documents)#number of documents\n",
      "    vocab_size=len(vocab)#number of uniqe Words\n",
      "    model = lda.LDA(n_topics=Num_topics, n_iter=1000, random_state=1)\n",
      "    X=np.ndarray((n_documents,vocab_size), \n",
      "            buffer=np.array([(documents[i][vocab[j]] if vocab[j] in documents[i] else 0) for i in range(n_documents) for j in range(vocab_size)]),\n",
      "            offset=0,dtype=int)\n",
      "    model.fit(X)\n",
      "    topic_word = model.topic_word_\n",
      "    n_top_words = 25\n",
      "    topic_top_words=[[\"Topic ID\"]+[\"Vocab {}\".format(i+1) for i in range(n_top_words)]]\n",
      "    for i, topic_dist in enumerate(topic_word):\n",
      "        topic_words = np.array(vocab)[np.argsort(topic_dist)][:-(n_top_words+1):-1]\n",
      "        topic_top_words.append([\"Topic {:2}\".format(i+1)]+[t.encode('utf-8',\"ignore\") for t in topic_words])\n",
      "    write_to_csv(\"Topics_top_%s_words(indeed).csv\" % n_top_words,topic_top_words)\n",
      "    doc_topic = model.doc_topic_\n",
      "    Topic_grouping=[]\n",
      "    for i in range(n_documents):\n",
      "        Topic_grouping.append(\"Topic {:2}\".format(doc_topic[i].argmax()+1))\n",
      "    heading=[[\"Topic ID\",\"JOB Title\",\"URL\",\"Job Description\"]]\n",
      "    Topic_grouping=sorted([[Topic_grouping[i],Jobs[i][\"jobtitle\"].encode('ascii',\"ignore\"),Jobs[i][\"url\"],Jobs[i]['summary'].encode('ascii',\"ignore\")] for i in range(n_documents)],key=lambda l:l[0])\n",
      "    write_to_csv(\"Topics_Job_Description(indeed).csv\",heading+Topic_grouping)\n",
      "    return topic_word, doc_topic,Jobs,vocab"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 27
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "if __name__ == '__main__':\n",
      "   topics,docs,Jobs,vocab= main(input_file=\"Topics\",Num_topics=5,Stop_word=\"Stop_words.txt\")    "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Ruby on rails 5\n",
        "python Developer"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " 5\n",
        "Network Admin"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " 5\n",
        "Graphic Design"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " 5\n",
        "Struts"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " 5\n"
       ]
      }
     ],
     "prompt_number": 28
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Displaying topic model relation using pandas\n",
      "    Both topic v/s vocab and topic v/s Documents are listed bellow"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pandas.read_csv('Topics_top_25_words(indeed).csv')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>Topic ID</th>\n",
        "      <th>Vocab 1</th>\n",
        "      <th>Vocab 2</th>\n",
        "      <th>Vocab 3</th>\n",
        "      <th>Vocab 4</th>\n",
        "      <th>Vocab 5</th>\n",
        "      <th>Vocab 6</th>\n",
        "      <th>Vocab 7</th>\n",
        "      <th>Vocab 8</th>\n",
        "      <th>Vocab 9</th>\n",
        "      <th>...</th>\n",
        "      <th>Vocab 16</th>\n",
        "      <th>Vocab 17</th>\n",
        "      <th>Vocab 18</th>\n",
        "      <th>Vocab 19</th>\n",
        "      <th>Vocab 20</th>\n",
        "      <th>Vocab 21</th>\n",
        "      <th>Vocab 22</th>\n",
        "      <th>Vocab 23</th>\n",
        "      <th>Vocab 24</th>\n",
        "      <th>Vocab 25</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> Topic  1</td>\n",
        "      <td>      design</td>\n",
        "      <td>    graphic</td>\n",
        "      <td>    assets</td>\n",
        "      <td>        visual</td>\n",
        "      <td>              adobe</td>\n",
        "      <td>     blog</td>\n",
        "      <td>     information</td>\n",
        "      <td> studio-quality</td>\n",
        "      <td> portfolio</td>\n",
        "      <td>...</td>\n",
        "      <td> multimedia</td>\n",
        "      <td>            tvm</td>\n",
        "      <td>    imagery</td>\n",
        "      <td>    co-founder</td>\n",
        "      <td>     word</td>\n",
        "      <td>    removed</td>\n",
        "      <td>     luxe</td>\n",
        "      <td> intended</td>\n",
        "      <td> executive</td>\n",
        "      <td> copywriting</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> Topic  2</td>\n",
        "      <td>        ruby</td>\n",
        "      <td>      rails</td>\n",
        "      <td>      data</td>\n",
        "      <td> ruby-on-rails</td>\n",
        "      <td>              linux</td>\n",
        "      <td>     daya</td>\n",
        "      <td> position-python</td>\n",
        "      <td>   architecture</td>\n",
        "      <td>     ninja</td>\n",
        "      <td>...</td>\n",
        "      <td>       joel</td>\n",
        "      <td> virtualization</td>\n",
        "      <td>    related</td>\n",
        "      <td>          wlan</td>\n",
        "      <td>      ipy</td>\n",
        "      <td> naperville</td>\n",
        "      <td>     sdlc</td>\n",
        "      <td> shashtri</td>\n",
        "      <td>     eigrp</td>\n",
        "      <td>         stp</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> Topic  3</td>\n",
        "      <td>       vince</td>\n",
        "      <td> javascript</td>\n",
        "      <td>    jquery</td>\n",
        "      <td>           css</td>\n",
        "      <td>               html</td>\n",
        "      <td>     apps</td>\n",
        "      <td>          online</td>\n",
        "      <td>            app</td>\n",
        "      <td>    django</td>\n",
        "      <td>...</td>\n",
        "      <td>       ajax</td>\n",
        "      <td>        reports</td>\n",
        "      <td>     oracle</td>\n",
        "      <td> multi-tasking</td>\n",
        "      <td> scrumban</td>\n",
        "      <td>  buganizer</td>\n",
        "      <td> icentral</td>\n",
        "      <td>   become</td>\n",
        "      <td> lead-java</td>\n",
        "      <td>       flask</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> Topic  4</td>\n",
        "      <td>       cisco</td>\n",
        "      <td>      admin</td>\n",
        "      <td>       lan</td>\n",
        "      <td>        issues</td>\n",
        "      <td>            related</td>\n",
        "      <td>     ccnp</td>\n",
        "      <td>         senthil</td>\n",
        "      <td>          nexus</td>\n",
        "      <td>      dhcp</td>\n",
        "      <td>...</td>\n",
        "      <td>       ospf</td>\n",
        "      <td>            vrf</td>\n",
        "      <td>       hsrp</td>\n",
        "      <td>           cpt</td>\n",
        "      <td>      qos</td>\n",
        "      <td>        jan</td>\n",
        "      <td>  narayan</td>\n",
        "      <td>   selvam</td>\n",
        "      <td>     ipsec</td>\n",
        "      <td>        waas</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> Topic  5</td>\n",
        "      <td> development</td>\n",
        "      <td>     python</td>\n",
        "      <td> developer</td>\n",
        "      <td>          java</td>\n",
        "      <td> developer-jcsim-r0</td>\n",
        "      <td> analysis</td>\n",
        "      <td>      developers</td>\n",
        "      <td>        testing</td>\n",
        "      <td>     mysql</td>\n",
        "      <td>...</td>\n",
        "      <td>  databases</td>\n",
        "      <td>           unix</td>\n",
        "      <td> postgresql</td>\n",
        "      <td>       writing</td>\n",
        "      <td>      mvc</td>\n",
        "      <td>       york</td>\n",
        "      <td> relocate</td>\n",
        "      <td>    jboss</td>\n",
        "      <td>       pdf</td>\n",
        "      <td>    resource</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "<p>5 rows \u00d7 26 columns</p>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 34,
       "text": [
        "   Topic ID      Vocab 1     Vocab 2    Vocab 3        Vocab 4  \\\n",
        "0  Topic  1       design     graphic     assets         visual   \n",
        "1  Topic  2         ruby       rails       data  ruby-on-rails   \n",
        "2  Topic  3        vince  javascript     jquery            css   \n",
        "3  Topic  4        cisco       admin        lan         issues   \n",
        "4  Topic  5  development      python  developer           java   \n",
        "\n",
        "              Vocab 5   Vocab 6          Vocab 7         Vocab 8    Vocab 9  \\\n",
        "0               adobe      blog      information  studio-quality  portfolio   \n",
        "1               linux      daya  position-python    architecture      ninja   \n",
        "2                html      apps           online             app     django   \n",
        "3             related      ccnp          senthil           nexus       dhcp   \n",
        "4  developer-jcsim-r0  analysis       developers         testing      mysql   \n",
        "\n",
        "      ...         Vocab 16        Vocab 17    Vocab 18       Vocab 19  \\\n",
        "0     ...       multimedia             tvm     imagery     co-founder   \n",
        "1     ...             joel  virtualization     related           wlan   \n",
        "2     ...             ajax         reports      oracle  multi-tasking   \n",
        "3     ...             ospf             vrf        hsrp            cpt   \n",
        "4     ...        databases            unix  postgresql        writing   \n",
        "\n",
        "   Vocab 20    Vocab 21  Vocab 22  Vocab 23   Vocab 24     Vocab 25  \n",
        "0      word     removed      luxe  intended  executive  copywriting  \n",
        "1       ipy  naperville      sdlc  shashtri      eigrp          stp  \n",
        "2  scrumban   buganizer  icentral    become  lead-java        flask  \n",
        "3       qos         jan   narayan    selvam      ipsec         waas  \n",
        "4       mvc        york  relocate     jboss        pdf     resource  \n",
        "\n",
        "[5 rows x 26 columns]"
       ]
      }
     ],
     "prompt_number": 34
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pandas.read_csv('Topics_Job_Description(indeed).csv')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>Topic ID</th>\n",
        "      <th>JOB Title</th>\n",
        "      <th>URL</th>\n",
        "      <th>Job Description</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0 </th>\n",
        "      <td> Topic  1</td>\n",
        "      <td>                            Graphic Design Manager</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=a387f490377ce...</td>\n",
        "      <td> Graphic Design &amp; Creative Manager. . General D...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1 </th>\n",
        "      <td> Topic  1</td>\n",
        "      <td>          Marketing and Graphic Design Coordinator</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=4005fb3086f8a...</td>\n",
        "      <td>                                               NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2 </th>\n",
        "      <td> Topic  1</td>\n",
        "      <td>                 Graphic Design &amp; Video Specialist</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=3deb7e940beaf...</td>\n",
        "      <td> We are looking for a talented multimedia speci...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3 </th>\n",
        "      <td> Topic  1</td>\n",
        "      <td>                          Graphic Design Team Lead</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=aabf479572024...</td>\n",
        "      <td> Responsible for providing leadership and direc...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4 </th>\n",
        "      <td> Topic  1</td>\n",
        "      <td>  Senior Product Engineer - Monroe Shocks &amp; Struts</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=22462ef7bb1ae...</td>\n",
        "      <td> This position is for a self-motivated engineer...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5 </th>\n",
        "      <td> Topic  2</td>\n",
        "      <td>               Backend Ruby on Rails Ninja Warrior</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=9ee46e1b813e7...</td>\n",
        "      <td>                                               NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>6 </th>\n",
        "      <td> Topic  2</td>\n",
        "      <td>               Backend Ruby on Rails Ninja Warrior</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=32d69875e29dc...</td>\n",
        "      <td> We're looking for a Ninja Warrior, Problem Sol...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>7 </th>\n",
        "      <td> Topic  2</td>\n",
        "      <td>                Ruby on Rails Developers - startup</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=6d18f4a4d7d5c...</td>\n",
        "      <td> A well-funded startup in Boston who is looking...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>8 </th>\n",
        "      <td> Topic  2</td>\n",
        "      <td>                                     Ruby On Rails</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=4ebeb6090916d...</td>\n",
        "      <td> Dear Candidate ,. I hope you are well!. Are yo...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>9 </th>\n",
        "      <td> Topic  3</td>\n",
        "      <td>                               Sr Python Developer</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=a9c0ed6041ce7...</td>\n",
        "      <td> Sr Python Developer.. Responsibilities. Full s...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>10</th>\n",
        "      <td> Topic  3</td>\n",
        "      <td> Art Direction/ Graphic Design Intern NY - Spri...</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=b48e2177c9e7a...</td>\n",
        "      <td> Vince is a leading luxury, contemporary brand ...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>11</th>\n",
        "      <td> Topic  3</td>\n",
        "      <td> Technical Lead-Java,J2EE,Spring MVC,Struts 2,O...</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=fa6ba5120acec...</td>\n",
        "      <td>                                               NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>12</th>\n",
        "      <td> Topic  4</td>\n",
        "      <td>       Gunjan Hada_Network Admin in Alpharetta, GA</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=59ea7e2001fa5...</td>\n",
        "      <td> 7+ years of experience in the area of networki...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>13</th>\n",
        "      <td> Topic  4</td>\n",
        "      <td>                        Position for Network Admin</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=5dc3c2da7a686...</td>\n",
        "      <td> Experience in Network Support, Design and Impl...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>14</th>\n",
        "      <td> Topic  4</td>\n",
        "      <td> Narayan Kunte_Telecommunication Network Admin_...</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=0b205ef495bdb...</td>\n",
        "      <td> Experience Level: 8+ Years, Location: Must be ...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>15</th>\n",
        "      <td> Topic  4</td>\n",
        "      <td>                                     Network Admin</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=2e9df38e906a5...</td>\n",
        "      <td>                                               NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>16</th>\n",
        "      <td> Topic  4</td>\n",
        "      <td>                            Network Admin (#17005)</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=fe262706d662c...</td>\n",
        "      <td> Hi,. . Hope you are doing great.. . I have the...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>17</th>\n",
        "      <td> Topic  5</td>\n",
        "      <td>                Ruby on Rails Full Time Internship</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=fc06d98e7dd0e...</td>\n",
        "      <td> Job Description. We are looking for interns to...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>18</th>\n",
        "      <td> Topic  5</td>\n",
        "      <td>                                  Python Developer</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=d1e221a931457...</td>\n",
        "      <td>                                               NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>19</th>\n",
        "      <td> Topic  5</td>\n",
        "      <td>                                  Python Developer</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=566c8ddac15cb...</td>\n",
        "      <td> BuzzFeed is seeking an experienced Python Deve...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>20</th>\n",
        "      <td> Topic  5</td>\n",
        "      <td>                           Senior Python Developer</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=f3e49bcc5ded8...</td>\n",
        "      <td> Strong knowledge and experience with Python an...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>21</th>\n",
        "      <td> Topic  5</td>\n",
        "      <td> Full Time Position - Python Developer for New ...</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=c83a92ce80d32...</td>\n",
        "      <td> JOB DESCRIPTION:. . . Position : Python Develo...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>22</th>\n",
        "      <td> Topic  5</td>\n",
        "      <td>        Java/J2EE developer with Struts experience</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=ae51428b6dcc7...</td>\n",
        "      <td> Java/J2EE developer with Struts experience\\r\\n...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>23</th>\n",
        "      <td> Topic  5</td>\n",
        "      <td>                Java Struts Developer - JCSIM-R001</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=15fd6e883d405...</td>\n",
        "      <td>                                               NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>24</th>\n",
        "      <td> Topic  5</td>\n",
        "      <td>                Java Websphere Struts II Developer</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=3d8e8dc415f57...</td>\n",
        "      <td>                                               NaN</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 35,
       "text": [
        "    Topic ID                                          JOB Title  \\\n",
        "0   Topic  1                             Graphic Design Manager   \n",
        "1   Topic  1           Marketing and Graphic Design Coordinator   \n",
        "2   Topic  1                  Graphic Design & Video Specialist   \n",
        "3   Topic  1                           Graphic Design Team Lead   \n",
        "4   Topic  1   Senior Product Engineer - Monroe Shocks & Struts   \n",
        "5   Topic  2                Backend Ruby on Rails Ninja Warrior   \n",
        "6   Topic  2                Backend Ruby on Rails Ninja Warrior   \n",
        "7   Topic  2                 Ruby on Rails Developers - startup   \n",
        "8   Topic  2                                      Ruby On Rails   \n",
        "9   Topic  3                                Sr Python Developer   \n",
        "10  Topic  3  Art Direction/ Graphic Design Intern NY - Spri...   \n",
        "11  Topic  3  Technical Lead-Java,J2EE,Spring MVC,Struts 2,O...   \n",
        "12  Topic  4        Gunjan Hada_Network Admin in Alpharetta, GA   \n",
        "13  Topic  4                         Position for Network Admin   \n",
        "14  Topic  4  Narayan Kunte_Telecommunication Network Admin_...   \n",
        "15  Topic  4                                      Network Admin   \n",
        "16  Topic  4                             Network Admin (#17005)   \n",
        "17  Topic  5                 Ruby on Rails Full Time Internship   \n",
        "18  Topic  5                                   Python Developer   \n",
        "19  Topic  5                                   Python Developer   \n",
        "20  Topic  5                            Senior Python Developer   \n",
        "21  Topic  5  Full Time Position - Python Developer for New ...   \n",
        "22  Topic  5         Java/J2EE developer with Struts experience   \n",
        "23  Topic  5                 Java Struts Developer - JCSIM-R001   \n",
        "24  Topic  5                 Java Websphere Struts II Developer   \n",
        "\n",
        "                                                  URL  \\\n",
        "0   http://www.indeed.com/viewjob?jk=a387f490377ce...   \n",
        "1   http://www.indeed.com/viewjob?jk=4005fb3086f8a...   \n",
        "2   http://www.indeed.com/viewjob?jk=3deb7e940beaf...   \n",
        "3   http://www.indeed.com/viewjob?jk=aabf479572024...   \n",
        "4   http://www.indeed.com/viewjob?jk=22462ef7bb1ae...   \n",
        "5   http://www.indeed.com/viewjob?jk=9ee46e1b813e7...   \n",
        "6   http://www.indeed.com/viewjob?jk=32d69875e29dc...   \n",
        "7   http://www.indeed.com/viewjob?jk=6d18f4a4d7d5c...   \n",
        "8   http://www.indeed.com/viewjob?jk=4ebeb6090916d...   \n",
        "9   http://www.indeed.com/viewjob?jk=a9c0ed6041ce7...   \n",
        "10  http://www.indeed.com/viewjob?jk=b48e2177c9e7a...   \n",
        "11  http://www.indeed.com/viewjob?jk=fa6ba5120acec...   \n",
        "12  http://www.indeed.com/viewjob?jk=59ea7e2001fa5...   \n",
        "13  http://www.indeed.com/viewjob?jk=5dc3c2da7a686...   \n",
        "14  http://www.indeed.com/viewjob?jk=0b205ef495bdb...   \n",
        "15  http://www.indeed.com/viewjob?jk=2e9df38e906a5...   \n",
        "16  http://www.indeed.com/viewjob?jk=fe262706d662c...   \n",
        "17  http://www.indeed.com/viewjob?jk=fc06d98e7dd0e...   \n",
        "18  http://www.indeed.com/viewjob?jk=d1e221a931457...   \n",
        "19  http://www.indeed.com/viewjob?jk=566c8ddac15cb...   \n",
        "20  http://www.indeed.com/viewjob?jk=f3e49bcc5ded8...   \n",
        "21  http://www.indeed.com/viewjob?jk=c83a92ce80d32...   \n",
        "22  http://www.indeed.com/viewjob?jk=ae51428b6dcc7...   \n",
        "23  http://www.indeed.com/viewjob?jk=15fd6e883d405...   \n",
        "24  http://www.indeed.com/viewjob?jk=3d8e8dc415f57...   \n",
        "\n",
        "                                      Job Description  \n",
        "0   Graphic Design & Creative Manager. . General D...  \n",
        "1                                                 NaN  \n",
        "2   We are looking for a talented multimedia speci...  \n",
        "3   Responsible for providing leadership and direc...  \n",
        "4   This position is for a self-motivated engineer...  \n",
        "5                                                 NaN  \n",
        "6   We're looking for a Ninja Warrior, Problem Sol...  \n",
        "7   A well-funded startup in Boston who is looking...  \n",
        "8   Dear Candidate ,. I hope you are well!. Are yo...  \n",
        "9   Sr Python Developer.. Responsibilities. Full s...  \n",
        "10  Vince is a leading luxury, contemporary brand ...  \n",
        "11                                                NaN  \n",
        "12  7+ years of experience in the area of networki...  \n",
        "13  Experience in Network Support, Design and Impl...  \n",
        "14  Experience Level: 8+ Years, Location: Must be ...  \n",
        "15                                                NaN  \n",
        "16  Hi,. . Hope you are doing great.. . I have the...  \n",
        "17  Job Description. We are looking for interns to...  \n",
        "18                                                NaN  \n",
        "19  BuzzFeed is seeking an experienced Python Deve...  \n",
        "20  Strong knowledge and experience with Python an...  \n",
        "21  JOB DESCRIPTION:. . . Position : Python Develo...  \n",
        "22  Java/J2EE developer with Struts experience\\r\\n...  \n",
        "23                                                NaN  \n",
        "24                                                NaN  "
       ]
      }
     ],
     "prompt_number": 35
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "write_to_csv(\"Job_Description(indeed).csv\",[[\"JOB Title\",\"URL\",\"Job Description\"]]+[[i[\"jobtitle\"].encode('ascii',\"ignore\"),i[\"url\"],i['summary'].encode('ascii',\"ignore\")] for i in Jobs])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 37
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "write_to_csv(\"vocab-probability(indeed).csv\",[[\"Vocab\",\"Topic 1\",\"Topic 2\",\"Topic 3\",\"Topic 4\",\"Topic 5\",]]+[[vocab[i]]+[j[i] for j in topics] for i in range(len(vocab))])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 38
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "write_to_csv(\"Doc-probability(indeed).csv\",[[\"JOB Title\",\"Topic 1\",\"Topic 2\",\"Topic 3\",\"Topic 4\",\"Topic 5\",]]+[[Jobs[i][\"jobtitle\"].encode('ascii',\"ignore\")]+[j for j in docs[i]] for i in range(len(Jobs))])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 39
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pandas.read_csv('Job_Description(indeed).csv')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>JOB Title</th>\n",
        "      <th>URL</th>\n",
        "      <th>Job Description</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0 </th>\n",
        "      <td>               Backend Ruby on Rails Ninja Warrior</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=9ee46e1b813e7...</td>\n",
        "      <td>                                               NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1 </th>\n",
        "      <td>               Backend Ruby on Rails Ninja Warrior</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=32d69875e29dc...</td>\n",
        "      <td> We're looking for a Ninja Warrior, Problem Sol...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2 </th>\n",
        "      <td>                Ruby on Rails Developers - startup</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=6d18f4a4d7d5c...</td>\n",
        "      <td> A well-funded startup in Boston who is looking...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3 </th>\n",
        "      <td>                                     Ruby On Rails</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=4ebeb6090916d...</td>\n",
        "      <td> Dear Candidate ,. I hope you are well!. Are yo...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4 </th>\n",
        "      <td>                Ruby on Rails Full Time Internship</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=fc06d98e7dd0e...</td>\n",
        "      <td> Job Description. We are looking for interns to...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5 </th>\n",
        "      <td>                                  Python Developer</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=d1e221a931457...</td>\n",
        "      <td>                                               NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>6 </th>\n",
        "      <td>                                  Python Developer</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=566c8ddac15cb...</td>\n",
        "      <td> BuzzFeed is seeking an experienced Python Deve...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>7 </th>\n",
        "      <td>                               Sr Python Developer</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=a9c0ed6041ce7...</td>\n",
        "      <td> Sr Python Developer.. Responsibilities. Full s...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>8 </th>\n",
        "      <td>                           Senior Python Developer</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=f3e49bcc5ded8...</td>\n",
        "      <td> Strong knowledge and experience with Python an...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>9 </th>\n",
        "      <td> Full Time Position - Python Developer for New ...</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=c83a92ce80d32...</td>\n",
        "      <td> JOB DESCRIPTION:. . . Position : Python Develo...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>10</th>\n",
        "      <td>       Gunjan Hada_Network Admin in Alpharetta, GA</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=59ea7e2001fa5...</td>\n",
        "      <td> 7+ years of experience in the area of networki...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>11</th>\n",
        "      <td>                        Position for Network Admin</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=5dc3c2da7a686...</td>\n",
        "      <td> Experience in Network Support, Design and Impl...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>12</th>\n",
        "      <td> Narayan Kunte_Telecommunication Network Admin_...</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=0b205ef495bdb...</td>\n",
        "      <td> Experience Level: 8+ Years, Location: Must be ...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>13</th>\n",
        "      <td>                                     Network Admin</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=2e9df38e906a5...</td>\n",
        "      <td>                                               NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>14</th>\n",
        "      <td>                            Network Admin (#17005)</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=fe262706d662c...</td>\n",
        "      <td> Hi,. . Hope you are doing great.. . I have the...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>15</th>\n",
        "      <td>                            Graphic Design Manager</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=a387f490377ce...</td>\n",
        "      <td> Graphic Design &amp; Creative Manager. . General D...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>16</th>\n",
        "      <td>          Marketing and Graphic Design Coordinator</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=4005fb3086f8a...</td>\n",
        "      <td>                                               NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>17</th>\n",
        "      <td> Art Direction/ Graphic Design Intern NY - Spri...</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=b48e2177c9e7a...</td>\n",
        "      <td> Vince is a leading luxury, contemporary brand ...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>18</th>\n",
        "      <td>                 Graphic Design &amp; Video Specialist</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=3deb7e940beaf...</td>\n",
        "      <td> We are looking for a talented multimedia speci...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>19</th>\n",
        "      <td>                          Graphic Design Team Lead</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=aabf479572024...</td>\n",
        "      <td> Responsible for providing leadership and direc...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>20</th>\n",
        "      <td>        Java/J2EE developer with Struts experience</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=ae51428b6dcc7...</td>\n",
        "      <td> Java/J2EE developer with Struts experience\\n-\\...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>21</th>\n",
        "      <td>                Java Struts Developer - JCSIM-R001</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=15fd6e883d405...</td>\n",
        "      <td>                                               NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>22</th>\n",
        "      <td>                Java Websphere Struts II Developer</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=3d8e8dc415f57...</td>\n",
        "      <td>                                               NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>23</th>\n",
        "      <td> Technical Lead-Java,J2EE,Spring MVC,Struts 2,O...</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=fa6ba5120acec...</td>\n",
        "      <td>                                               NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>24</th>\n",
        "      <td>  Senior Product Engineer - Monroe Shocks &amp; Struts</td>\n",
        "      <td> http://www.indeed.com/viewjob?jk=22462ef7bb1ae...</td>\n",
        "      <td> This position is for a self-motivated engineer...</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 41,
       "text": [
        "                                            JOB Title  \\\n",
        "0                 Backend Ruby on Rails Ninja Warrior   \n",
        "1                 Backend Ruby on Rails Ninja Warrior   \n",
        "2                  Ruby on Rails Developers - startup   \n",
        "3                                       Ruby On Rails   \n",
        "4                  Ruby on Rails Full Time Internship   \n",
        "5                                    Python Developer   \n",
        "6                                    Python Developer   \n",
        "7                                 Sr Python Developer   \n",
        "8                             Senior Python Developer   \n",
        "9   Full Time Position - Python Developer for New ...   \n",
        "10        Gunjan Hada_Network Admin in Alpharetta, GA   \n",
        "11                         Position for Network Admin   \n",
        "12  Narayan Kunte_Telecommunication Network Admin_...   \n",
        "13                                      Network Admin   \n",
        "14                             Network Admin (#17005)   \n",
        "15                             Graphic Design Manager   \n",
        "16           Marketing and Graphic Design Coordinator   \n",
        "17  Art Direction/ Graphic Design Intern NY - Spri...   \n",
        "18                  Graphic Design & Video Specialist   \n",
        "19                           Graphic Design Team Lead   \n",
        "20         Java/J2EE developer with Struts experience   \n",
        "21                 Java Struts Developer - JCSIM-R001   \n",
        "22                 Java Websphere Struts II Developer   \n",
        "23  Technical Lead-Java,J2EE,Spring MVC,Struts 2,O...   \n",
        "24   Senior Product Engineer - Monroe Shocks & Struts   \n",
        "\n",
        "                                                  URL  \\\n",
        "0   http://www.indeed.com/viewjob?jk=9ee46e1b813e7...   \n",
        "1   http://www.indeed.com/viewjob?jk=32d69875e29dc...   \n",
        "2   http://www.indeed.com/viewjob?jk=6d18f4a4d7d5c...   \n",
        "3   http://www.indeed.com/viewjob?jk=4ebeb6090916d...   \n",
        "4   http://www.indeed.com/viewjob?jk=fc06d98e7dd0e...   \n",
        "5   http://www.indeed.com/viewjob?jk=d1e221a931457...   \n",
        "6   http://www.indeed.com/viewjob?jk=566c8ddac15cb...   \n",
        "7   http://www.indeed.com/viewjob?jk=a9c0ed6041ce7...   \n",
        "8   http://www.indeed.com/viewjob?jk=f3e49bcc5ded8...   \n",
        "9   http://www.indeed.com/viewjob?jk=c83a92ce80d32...   \n",
        "10  http://www.indeed.com/viewjob?jk=59ea7e2001fa5...   \n",
        "11  http://www.indeed.com/viewjob?jk=5dc3c2da7a686...   \n",
        "12  http://www.indeed.com/viewjob?jk=0b205ef495bdb...   \n",
        "13  http://www.indeed.com/viewjob?jk=2e9df38e906a5...   \n",
        "14  http://www.indeed.com/viewjob?jk=fe262706d662c...   \n",
        "15  http://www.indeed.com/viewjob?jk=a387f490377ce...   \n",
        "16  http://www.indeed.com/viewjob?jk=4005fb3086f8a...   \n",
        "17  http://www.indeed.com/viewjob?jk=b48e2177c9e7a...   \n",
        "18  http://www.indeed.com/viewjob?jk=3deb7e940beaf...   \n",
        "19  http://www.indeed.com/viewjob?jk=aabf479572024...   \n",
        "20  http://www.indeed.com/viewjob?jk=ae51428b6dcc7...   \n",
        "21  http://www.indeed.com/viewjob?jk=15fd6e883d405...   \n",
        "22  http://www.indeed.com/viewjob?jk=3d8e8dc415f57...   \n",
        "23  http://www.indeed.com/viewjob?jk=fa6ba5120acec...   \n",
        "24  http://www.indeed.com/viewjob?jk=22462ef7bb1ae...   \n",
        "\n",
        "                                      Job Description  \n",
        "0                                                 NaN  \n",
        "1   We're looking for a Ninja Warrior, Problem Sol...  \n",
        "2   A well-funded startup in Boston who is looking...  \n",
        "3   Dear Candidate ,. I hope you are well!. Are yo...  \n",
        "4   Job Description. We are looking for interns to...  \n",
        "5                                                 NaN  \n",
        "6   BuzzFeed is seeking an experienced Python Deve...  \n",
        "7   Sr Python Developer.. Responsibilities. Full s...  \n",
        "8   Strong knowledge and experience with Python an...  \n",
        "9   JOB DESCRIPTION:. . . Position : Python Develo...  \n",
        "10  7+ years of experience in the area of networki...  \n",
        "11  Experience in Network Support, Design and Impl...  \n",
        "12  Experience Level: 8+ Years, Location: Must be ...  \n",
        "13                                                NaN  \n",
        "14  Hi,. . Hope you are doing great.. . I have the...  \n",
        "15  Graphic Design & Creative Manager. . General D...  \n",
        "16                                                NaN  \n",
        "17  Vince is a leading luxury, contemporary brand ...  \n",
        "18  We are looking for a talented multimedia speci...  \n",
        "19  Responsible for providing leadership and direc...  \n",
        "20  Java/J2EE developer with Struts experience\\n-\\...  \n",
        "21                                                NaN  \n",
        "22                                                NaN  \n",
        "23                                                NaN  \n",
        "24  This position is for a self-motivated engineer...  "
       ]
      }
     ],
     "prompt_number": 41
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pandas.read_csv('vocab-probability(indeed).csv')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>Vocab</th>\n",
        "      <th>Topic 1</th>\n",
        "      <th>Topic 2</th>\n",
        "      <th>Topic 3</th>\n",
        "      <th>Topic 4</th>\n",
        "      <th>Topic 5</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0  </th>\n",
        "      <td>               ruby</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.127200</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1  </th>\n",
        "      <td>              rails</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.107646</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2  </th>\n",
        "      <td>              ninja</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.029429</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3  </th>\n",
        "      <td>      ruby-on-rails</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.068537</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4  </th>\n",
        "      <td>               apps</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.034192</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5  </th>\n",
        "      <td>             become</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.017138</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>6  </th>\n",
        "      <td>            vibrant</td>\n",
        "      <td> 0.008467</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>7  </th>\n",
        "      <td>        development</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.188651</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>8  </th>\n",
        "      <td>       architecture</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.029429</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.025353</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>9  </th>\n",
        "      <td>               data</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.097869</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>10 </th>\n",
        "      <td>                oop</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.008612</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>11 </th>\n",
        "      <td>             design</td>\n",
        "      <td> 0.368964</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>12 </th>\n",
        "      <td>            mongodb</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.016696</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>13 </th>\n",
        "      <td>              linux</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.039206</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>14 </th>\n",
        "      <td>              nginx</td>\n",
        "      <td> 0.008467</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>15 </th>\n",
        "      <td>             jquery</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.051245</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>16 </th>\n",
        "      <td>                mvc</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.011149</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>17 </th>\n",
        "      <td>          angularjs</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.005602</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>18 </th>\n",
        "      <td>            emberjs</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.009875</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>19 </th>\n",
        "      <td>             worked</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.009875</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>20 </th>\n",
        "      <td>         developers</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.038884</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>21 </th>\n",
        "      <td>              would</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.017138</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>22 </th>\n",
        "      <td>         nderstands</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.008612</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>23 </th>\n",
        "      <td>             config</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.009875</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>24 </th>\n",
        "      <td>         stealhmode</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.009875</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>25 </th>\n",
        "      <td>            removed</td>\n",
        "      <td> 0.008467</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>26 </th>\n",
        "      <td>               joel</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.019652</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>27 </th>\n",
        "      <td>           database</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.022243</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>28 </th>\n",
        "      <td>             before</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.009875</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>29 </th>\n",
        "      <td>            writing</td>\n",
        "      <td> 0.016851</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.011149</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>...</th>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>198</th>\n",
        "      <td>       infographics</td>\n",
        "      <td> 0.008467</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>199</th>\n",
        "      <td>     studio-quality</td>\n",
        "      <td> 0.025235</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>200</th>\n",
        "      <td>          resources</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.012740</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>201</th>\n",
        "      <td>                b2b</td>\n",
        "      <td> 0.008467</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>202</th>\n",
        "      <td>         executives</td>\n",
        "      <td> 0.008467</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>203</th>\n",
        "      <td>             christ</td>\n",
        "      <td> 0.008467</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>204</th>\n",
        "      <td>         increasing</td>\n",
        "      <td> 0.008467</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>205</th>\n",
        "      <td>               j2ee</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.033337</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>206</th>\n",
        "      <td>          residency</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.012740</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>207</th>\n",
        "      <td>           relocate</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.005602</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>208</th>\n",
        "      <td>           resource</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.005602</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>209</th>\n",
        "      <td>                ejb</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.008612</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>210</th>\n",
        "      <td>                ide</td>\n",
        "      <td> 0.008467</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>211</th>\n",
        "      <td>              jboss</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.005602</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>212</th>\n",
        "      <td>                cms</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.005602</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>213</th>\n",
        "      <td>                tdd</td>\n",
        "      <td> 0.008467</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>214</th>\n",
        "      <td>           exstream</td>\n",
        "      <td> 0.008467</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>215</th>\n",
        "      <td>               ssrs</td>\n",
        "      <td> 0.008467</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>216</th>\n",
        "      <td>            clarety</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.005602</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>217</th>\n",
        "      <td>             studio</td>\n",
        "      <td> 0.016851</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>218</th>\n",
        "      <td> states-texas-plano</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.005602</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>219</th>\n",
        "      <td> states-ohio-dublin</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.005602</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>220</th>\n",
        "      <td>                jan</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.012740</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>221</th>\n",
        "      <td> developer-jcsim-r0</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.055525</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>222</th>\n",
        "      <td>          lead-java</td>\n",
        "      <td> 0.000084</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.017138</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>223</th>\n",
        "      <td>              sigma</td>\n",
        "      <td> 0.008467</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>224</th>\n",
        "      <td>            release</td>\n",
        "      <td> 0.008467</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>225</th>\n",
        "      <td>               fmea</td>\n",
        "      <td> 0.008467</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>226</th>\n",
        "      <td>                tvm</td>\n",
        "      <td> 0.008467</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>227</th>\n",
        "      <td>               word</td>\n",
        "      <td> 0.008467</td>\n",
        "      <td> 0.000098</td>\n",
        "      <td> 0.000085</td>\n",
        "      <td> 0.000126</td>\n",
        "      <td> 0.000055</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "<p>228 rows \u00d7 6 columns</p>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 42,
       "text": [
        "                  Vocab   Topic 1   Topic 2   Topic 3   Topic 4   Topic 5\n",
        "0                  ruby  0.000084  0.127200  0.000085  0.000126  0.000055\n",
        "1                 rails  0.000084  0.107646  0.000085  0.000126  0.000055\n",
        "2                 ninja  0.000084  0.029429  0.000085  0.000126  0.000055\n",
        "3         ruby-on-rails  0.000084  0.068537  0.000085  0.000126  0.000055\n",
        "4                  apps  0.000084  0.000098  0.034192  0.000126  0.000055\n",
        "5                become  0.000084  0.000098  0.017138  0.000126  0.000055\n",
        "6               vibrant  0.008467  0.000098  0.000085  0.000126  0.000055\n",
        "7           development  0.000084  0.000098  0.000085  0.000126  0.188651\n",
        "8          architecture  0.000084  0.029429  0.000085  0.025353  0.000055\n",
        "9                  data  0.000084  0.097869  0.000085  0.000126  0.000055\n",
        "10                  oop  0.000084  0.000098  0.008612  0.000126  0.000055\n",
        "11               design  0.368964  0.000098  0.000085  0.000126  0.000055\n",
        "12              mongodb  0.000084  0.000098  0.000085  0.000126  0.016696\n",
        "13                linux  0.000084  0.039206  0.000085  0.000126  0.000055\n",
        "14                nginx  0.008467  0.000098  0.000085  0.000126  0.000055\n",
        "15               jquery  0.000084  0.000098  0.051245  0.000126  0.000055\n",
        "16                  mvc  0.000084  0.000098  0.000085  0.000126  0.011149\n",
        "17            angularjs  0.000084  0.000098  0.000085  0.000126  0.005602\n",
        "18              emberjs  0.000084  0.009875  0.000085  0.000126  0.000055\n",
        "19               worked  0.000084  0.009875  0.000085  0.000126  0.000055\n",
        "20           developers  0.000084  0.000098  0.000085  0.000126  0.038884\n",
        "21                would  0.000084  0.000098  0.017138  0.000126  0.000055\n",
        "22           nderstands  0.000084  0.000098  0.008612  0.000126  0.000055\n",
        "23               config  0.000084  0.009875  0.000085  0.000126  0.000055\n",
        "24           stealhmode  0.000084  0.009875  0.000085  0.000126  0.000055\n",
        "25              removed  0.008467  0.000098  0.000085  0.000126  0.000055\n",
        "26                 joel  0.000084  0.019652  0.000085  0.000126  0.000055\n",
        "27             database  0.000084  0.000098  0.000085  0.000126  0.022243\n",
        "28               before  0.000084  0.009875  0.000085  0.000126  0.000055\n",
        "29              writing  0.016851  0.000098  0.000085  0.000126  0.011149\n",
        "..                  ...       ...       ...       ...       ...       ...\n",
        "198        infographics  0.008467  0.000098  0.000085  0.000126  0.000055\n",
        "199      studio-quality  0.025235  0.000098  0.000085  0.000126  0.000055\n",
        "200           resources  0.000084  0.000098  0.000085  0.012740  0.000055\n",
        "201                 b2b  0.008467  0.000098  0.000085  0.000126  0.000055\n",
        "202          executives  0.008467  0.000098  0.000085  0.000126  0.000055\n",
        "203              christ  0.008467  0.000098  0.000085  0.000126  0.000055\n",
        "204          increasing  0.008467  0.000098  0.000085  0.000126  0.000055\n",
        "205                j2ee  0.000084  0.000098  0.000085  0.000126  0.033337\n",
        "206           residency  0.000084  0.000098  0.000085  0.012740  0.000055\n",
        "207            relocate  0.000084  0.000098  0.000085  0.000126  0.005602\n",
        "208            resource  0.000084  0.000098  0.000085  0.000126  0.005602\n",
        "209                 ejb  0.000084  0.000098  0.008612  0.000126  0.000055\n",
        "210                 ide  0.008467  0.000098  0.000085  0.000126  0.000055\n",
        "211               jboss  0.000084  0.000098  0.000085  0.000126  0.005602\n",
        "212                 cms  0.000084  0.000098  0.000085  0.000126  0.005602\n",
        "213                 tdd  0.008467  0.000098  0.000085  0.000126  0.000055\n",
        "214            exstream  0.008467  0.000098  0.000085  0.000126  0.000055\n",
        "215                ssrs  0.008467  0.000098  0.000085  0.000126  0.000055\n",
        "216             clarety  0.000084  0.000098  0.000085  0.000126  0.005602\n",
        "217              studio  0.016851  0.000098  0.000085  0.000126  0.000055\n",
        "218  states-texas-plano  0.000084  0.000098  0.000085  0.000126  0.005602\n",
        "219  states-ohio-dublin  0.000084  0.000098  0.000085  0.000126  0.005602\n",
        "220                 jan  0.000084  0.000098  0.000085  0.012740  0.000055\n",
        "221  developer-jcsim-r0  0.000084  0.000098  0.000085  0.000126  0.055525\n",
        "222           lead-java  0.000084  0.000098  0.017138  0.000126  0.000055\n",
        "223               sigma  0.008467  0.000098  0.000085  0.000126  0.000055\n",
        "224             release  0.008467  0.000098  0.000085  0.000126  0.000055\n",
        "225                fmea  0.008467  0.000098  0.000085  0.000126  0.000055\n",
        "226                 tvm  0.008467  0.000098  0.000085  0.000126  0.000055\n",
        "227                word  0.008467  0.000098  0.000085  0.000126  0.000055\n",
        "\n",
        "[228 rows x 6 columns]"
       ]
      }
     ],
     "prompt_number": 42
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pandas.read_csv('Doc-probability(indeed).csv')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>JOB Title</th>\n",
        "      <th>Topic 1</th>\n",
        "      <th>Topic 2</th>\n",
        "      <th>Topic 3</th>\n",
        "      <th>Topic 4</th>\n",
        "      <th>Topic 5</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0 </th>\n",
        "      <td>               Backend Ruby on Rails Ninja Warrior</td>\n",
        "      <td> 0.022222</td>\n",
        "      <td> 0.911111</td>\n",
        "      <td> 0.022222</td>\n",
        "      <td> 0.022222</td>\n",
        "      <td> 0.022222</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1 </th>\n",
        "      <td>               Backend Ruby on Rails Ninja Warrior</td>\n",
        "      <td> 0.112088</td>\n",
        "      <td> 0.485714</td>\n",
        "      <td> 0.221978</td>\n",
        "      <td> 0.002198</td>\n",
        "      <td> 0.178022</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2 </th>\n",
        "      <td>                Ruby on Rails Developers - startup</td>\n",
        "      <td> 0.002817</td>\n",
        "      <td> 0.594366</td>\n",
        "      <td> 0.143662</td>\n",
        "      <td> 0.002817</td>\n",
        "      <td> 0.256338</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3 </th>\n",
        "      <td>                                     Ruby On Rails</td>\n",
        "      <td> 0.002985</td>\n",
        "      <td> 0.749254</td>\n",
        "      <td> 0.002985</td>\n",
        "      <td> 0.002985</td>\n",
        "      <td> 0.241791</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4 </th>\n",
        "      <td>                Ruby on Rails Full Time Internship</td>\n",
        "      <td> 0.150617</td>\n",
        "      <td> 0.224691</td>\n",
        "      <td> 0.298765</td>\n",
        "      <td> 0.002469</td>\n",
        "      <td> 0.323457</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5 </th>\n",
        "      <td>                                  Python Developer</td>\n",
        "      <td> 0.028571</td>\n",
        "      <td> 0.028571</td>\n",
        "      <td> 0.028571</td>\n",
        "      <td> 0.028571</td>\n",
        "      <td> 0.885714</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>6 </th>\n",
        "      <td>                                  Python Developer</td>\n",
        "      <td> 0.003774</td>\n",
        "      <td> 0.041509</td>\n",
        "      <td> 0.343396</td>\n",
        "      <td> 0.003774</td>\n",
        "      <td> 0.607547</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>7 </th>\n",
        "      <td>                               Sr Python Developer</td>\n",
        "      <td> 0.024176</td>\n",
        "      <td> 0.002198</td>\n",
        "      <td> 0.595604</td>\n",
        "      <td> 0.002198</td>\n",
        "      <td> 0.375824</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>8 </th>\n",
        "      <td>                           Senior Python Developer</td>\n",
        "      <td> 0.003774</td>\n",
        "      <td> 0.003774</td>\n",
        "      <td> 0.267925</td>\n",
        "      <td> 0.003774</td>\n",
        "      <td> 0.720755</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>9 </th>\n",
        "      <td> Full Time Position - Python Developer for New ...</td>\n",
        "      <td> 0.060870</td>\n",
        "      <td> 0.234783</td>\n",
        "      <td> 0.002899</td>\n",
        "      <td> 0.002899</td>\n",
        "      <td> 0.698551</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>10</th>\n",
        "      <td>       Gunjan Hada_Network Admin in Alpharetta, GA</td>\n",
        "      <td> 0.003636</td>\n",
        "      <td> 0.003636</td>\n",
        "      <td> 0.003636</td>\n",
        "      <td> 0.985455</td>\n",
        "      <td> 0.003636</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>11</th>\n",
        "      <td>                        Position for Network Admin</td>\n",
        "      <td> 0.283721</td>\n",
        "      <td> 0.237209</td>\n",
        "      <td> 0.004651</td>\n",
        "      <td> 0.469767</td>\n",
        "      <td> 0.004651</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>12</th>\n",
        "      <td> Narayan Kunte_Telecommunication Network Admin_...</td>\n",
        "      <td> 0.004255</td>\n",
        "      <td> 0.004255</td>\n",
        "      <td> 0.004255</td>\n",
        "      <td> 0.982979</td>\n",
        "      <td> 0.004255</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>13</th>\n",
        "      <td>                                     Network Admin</td>\n",
        "      <td> 0.066667</td>\n",
        "      <td> 0.066667</td>\n",
        "      <td> 0.066667</td>\n",
        "      <td> 0.733333</td>\n",
        "      <td> 0.066667</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>14</th>\n",
        "      <td>                            Network Admin (#17005)</td>\n",
        "      <td> 0.004878</td>\n",
        "      <td> 0.248780</td>\n",
        "      <td> 0.004878</td>\n",
        "      <td> 0.590244</td>\n",
        "      <td> 0.151220</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>15</th>\n",
        "      <td>                            Graphic Design Manager</td>\n",
        "      <td> 0.895385</td>\n",
        "      <td> 0.003077</td>\n",
        "      <td> 0.095385</td>\n",
        "      <td> 0.003077</td>\n",
        "      <td> 0.003077</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>16</th>\n",
        "      <td>          Marketing and Graphic Design Coordinator</td>\n",
        "      <td> 0.840000</td>\n",
        "      <td> 0.040000</td>\n",
        "      <td> 0.040000</td>\n",
        "      <td> 0.040000</td>\n",
        "      <td> 0.040000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>17</th>\n",
        "      <td> Art Direction/ Graphic Design Intern NY - Spri...</td>\n",
        "      <td> 0.120000</td>\n",
        "      <td> 0.002353</td>\n",
        "      <td> 0.825882</td>\n",
        "      <td> 0.002353</td>\n",
        "      <td> 0.049412</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>18</th>\n",
        "      <td>                 Graphic Design &amp; Video Specialist</td>\n",
        "      <td> 0.955556</td>\n",
        "      <td> 0.003175</td>\n",
        "      <td> 0.003175</td>\n",
        "      <td> 0.034921</td>\n",
        "      <td> 0.003175</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>19</th>\n",
        "      <td>                          Graphic Design Team Lead</td>\n",
        "      <td> 0.888000</td>\n",
        "      <td> 0.008000</td>\n",
        "      <td> 0.008000</td>\n",
        "      <td> 0.008000</td>\n",
        "      <td> 0.088000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>20</th>\n",
        "      <td>        Java/J2EE developer with Struts experience</td>\n",
        "      <td> 0.207477</td>\n",
        "      <td> 0.001869</td>\n",
        "      <td> 0.020561</td>\n",
        "      <td> 0.039252</td>\n",
        "      <td> 0.730841</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>21</th>\n",
        "      <td>                Java Struts Developer - JCSIM-R001</td>\n",
        "      <td> 0.022222</td>\n",
        "      <td> 0.022222</td>\n",
        "      <td> 0.022222</td>\n",
        "      <td> 0.022222</td>\n",
        "      <td> 0.911111</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>22</th>\n",
        "      <td>                Java Websphere Struts II Developer</td>\n",
        "      <td> 0.028571</td>\n",
        "      <td> 0.028571</td>\n",
        "      <td> 0.028571</td>\n",
        "      <td> 0.028571</td>\n",
        "      <td> 0.885714</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>23</th>\n",
        "      <td> Technical Lead-Java,J2EE,Spring MVC,Struts 2,O...</td>\n",
        "      <td> 0.009524</td>\n",
        "      <td> 0.009524</td>\n",
        "      <td> 0.580952</td>\n",
        "      <td> 0.009524</td>\n",
        "      <td> 0.390476</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>24</th>\n",
        "      <td>  Senior Product Engineer - Monroe Shocks &amp; Struts</td>\n",
        "      <td> 0.587097</td>\n",
        "      <td> 0.006452</td>\n",
        "      <td> 0.006452</td>\n",
        "      <td> 0.070968</td>\n",
        "      <td> 0.329032</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 43,
       "text": [
        "                                            JOB Title   Topic 1   Topic 2  \\\n",
        "0                 Backend Ruby on Rails Ninja Warrior  0.022222  0.911111   \n",
        "1                 Backend Ruby on Rails Ninja Warrior  0.112088  0.485714   \n",
        "2                  Ruby on Rails Developers - startup  0.002817  0.594366   \n",
        "3                                       Ruby On Rails  0.002985  0.749254   \n",
        "4                  Ruby on Rails Full Time Internship  0.150617  0.224691   \n",
        "5                                    Python Developer  0.028571  0.028571   \n",
        "6                                    Python Developer  0.003774  0.041509   \n",
        "7                                 Sr Python Developer  0.024176  0.002198   \n",
        "8                             Senior Python Developer  0.003774  0.003774   \n",
        "9   Full Time Position - Python Developer for New ...  0.060870  0.234783   \n",
        "10        Gunjan Hada_Network Admin in Alpharetta, GA  0.003636  0.003636   \n",
        "11                         Position for Network Admin  0.283721  0.237209   \n",
        "12  Narayan Kunte_Telecommunication Network Admin_...  0.004255  0.004255   \n",
        "13                                      Network Admin  0.066667  0.066667   \n",
        "14                             Network Admin (#17005)  0.004878  0.248780   \n",
        "15                             Graphic Design Manager  0.895385  0.003077   \n",
        "16           Marketing and Graphic Design Coordinator  0.840000  0.040000   \n",
        "17  Art Direction/ Graphic Design Intern NY - Spri...  0.120000  0.002353   \n",
        "18                  Graphic Design & Video Specialist  0.955556  0.003175   \n",
        "19                           Graphic Design Team Lead  0.888000  0.008000   \n",
        "20         Java/J2EE developer with Struts experience  0.207477  0.001869   \n",
        "21                 Java Struts Developer - JCSIM-R001  0.022222  0.022222   \n",
        "22                 Java Websphere Struts II Developer  0.028571  0.028571   \n",
        "23  Technical Lead-Java,J2EE,Spring MVC,Struts 2,O...  0.009524  0.009524   \n",
        "24   Senior Product Engineer - Monroe Shocks & Struts  0.587097  0.006452   \n",
        "\n",
        "     Topic 3   Topic 4   Topic 5  \n",
        "0   0.022222  0.022222  0.022222  \n",
        "1   0.221978  0.002198  0.178022  \n",
        "2   0.143662  0.002817  0.256338  \n",
        "3   0.002985  0.002985  0.241791  \n",
        "4   0.298765  0.002469  0.323457  \n",
        "5   0.028571  0.028571  0.885714  \n",
        "6   0.343396  0.003774  0.607547  \n",
        "7   0.595604  0.002198  0.375824  \n",
        "8   0.267925  0.003774  0.720755  \n",
        "9   0.002899  0.002899  0.698551  \n",
        "10  0.003636  0.985455  0.003636  \n",
        "11  0.004651  0.469767  0.004651  \n",
        "12  0.004255  0.982979  0.004255  \n",
        "13  0.066667  0.733333  0.066667  \n",
        "14  0.004878  0.590244  0.151220  \n",
        "15  0.095385  0.003077  0.003077  \n",
        "16  0.040000  0.040000  0.040000  \n",
        "17  0.825882  0.002353  0.049412  \n",
        "18  0.003175  0.034921  0.003175  \n",
        "19  0.008000  0.008000  0.088000  \n",
        "20  0.020561  0.039252  0.730841  \n",
        "21  0.022222  0.022222  0.911111  \n",
        "22  0.028571  0.028571  0.885714  \n",
        "23  0.580952  0.009524  0.390476  \n",
        "24  0.006452  0.070968  0.329032  "
       ]
      }
     ],
     "prompt_number": 43
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}