02-match-more-languages.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start the previous notebook with this code. Let's give it another spin."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "import pandas as pd\n",
    "from spacy import displacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load(\"en_core_web_sm\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def has_go_token(doc):\n",
    "    for t in doc:\n",
    "        if t.lower_ in ['go', 'golang', 'python', 'ruby', 'objective-c']:\n",
    "            if t.pos_ != 'VERB':\n",
    "                return True\n",
    "    return False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = nlp(\"i am an iOS dev and I like to code in objective-c\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[i, am, an, iOS, dev, and, I, like, to, code, in, objective, -, c]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[t for t in doc]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our current approach won't work for languages with more than one token. So instead I'll need to use matchers instead. There's a bunch of them applied below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy.matcher import Matcher"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "obj_c_pattern1 = [{'LOWER': 'objective'},\n",
    "                  {'IS_PUNCT': True, 'OP': '?'},\n",
    "                  {'LOWER': 'c'}]\n",
    "\n",
    "obj_c_pattern2 = [{'LOWER': 'objectivec'}]\n",
    "\n",
    "golang_pattern1 = [{'LOWER': 'golang'}] \n",
    "golang_pattern2 = [{'LOWER': 'go', \n",
    "                    'POS': {'NOT_IN': ['VERB']}}]\n",
    "\n",
    "python_pattern = [{'LOWER': 'python'}]\n",
    "ruby_pattern   = [{'LOWER': 'ruby'}]\n",
    "js_pattern     = [{'LOWER': {'IN': ['js', 'javascript']}}]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "matcher = Matcher(nlp.vocab, validate=True)\n",
    "matcher.add(\"OBJ_C_LANG\", None, obj_c_pattern1, obj_c_pattern2)\n",
    "matcher.add(\"PYTHON_LANG\", None, python_pattern)\n",
    "matcher.add(\"GO_LANG\", None, golang_pattern1, golang_pattern2)\n",
    "matcher.add(\"JS_LANG\", None, js_pattern)\n",
    "matcher.add(\"RUBY_LANG\", None, ruby_pattern)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "python\n",
      "golang\n",
      "objective-c\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(\"I am an iOS dev who codes in both python, go/golang as well as objective-c\")\n",
    "for match_id, start, end in matcher(doc):\n",
    "    print(doc[start: end])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "js\n",
      "ruby\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(\"I've done some js and ruby and go programming\")\n",
    "for match_id, start, end in matcher(doc):\n",
    "    print(doc[start: end])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmarking\n",
    "\n",
    "Our current approach works, but it would be good to confirm this with data. I'll do a soft benchmark; I'll check for the occurence of a string, like \"objective\", and I'll see which instances my matcher does not pick up. If there's stuff that I am missing I should get a pretty clear picture of it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = (pd.read_csv(\"Questions.csv\", nrows=1_000_000, \n",
    "                  encoding=\"ISO-8859-1\", usecols=['Title', 'Id']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "titles = (_ for _ in df['Title'] if \"python\" in _.lower())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mod_python/MySQL error on INSERT with a lot of data: \"OperationalError: (2006, 'MySQL server has gone away')\"\n",
      "Running subversion under apache and mod_python\n",
      "What's the best way to embed IronPython inside my C# App?\n",
      "How to set the PYTHONPATH in Emacs?\n",
      "wxPython wxDC object from win32gui.GetDC\n",
      "Need skeleton code to call Excel VBA from PythonWin\n",
      "Questions for python->scheme conversion\n",
      "wxPython and sharing objects between windows\n",
      "Django on IronPython\n",
      "IronPython Webframework\n",
      "A SuggestBox for wxPython?\n",
      "Intercepting Method Access on the Host Program of IronPython\n",
      "Is there anything like IPython / IRB for Perl?\n"
     ]
    }
   ],
   "source": [
    "for i in range(200):\n",
    "    doc = nlp(next(titles))\n",
    "    if len(matcher(doc)) == 0:\n",
    "        print(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I've used this benchmark to find some mistakes but this is another milestone for now. The next step; prepare the dataset for machine learning."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}