lectures/06_Intro_to_NLP.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Introduction to Python and Natural Language Technologies\n",
    "## Lecture 06, NLP Introduction\n",
    "\n",
    "March 16, 2020\n",
    "\n",
    "Ádám Kovács\n",
    "\n",
    "This lecture aims to give an introduction to the main concepts of NLP and word representations.\n",
    "\n",
    "## Preparation\n",
    "\n",
    "[Download GLOVE](http://sandbox.hlt.bme.hu/~adaamko/glove.6B.100d.txt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "!pip install spacy\n",
    "\n",
    "!pip install textacy\n",
    "\n",
    "!pip install flair\n",
    "\n",
    "!pip install gensim\n",
    "\n",
    "!pip install -U scikit-learn\n",
    "\n",
    "!python -m spacy download en"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## NLP Tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Why do we need NLP?\n",
    "\n",
    "    - Make the computer understand text\n",
    "    - Extract useful information from it\n",
    "    - A collection that helps us processing huge amount of texts\n",
    "    - We have two directions:\n",
    "        - Analysis: Convert text to a structural representation\n",
    "        - Generation: Generate text from formal representation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Tasks most people would think of\n",
    "\n",
    "- Spellchecking\n",
    "\n",
    "- Machine translation\n",
    "\n",
    "- Chatbots"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "these are not an exhaustive list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### \"Real\" tasks?\n",
    "\n",
    "- Basic NLP tasks (but very important):\n",
    "    - tokenization\n",
    "    - lemmatization\n",
    "    - POS tagging\n",
    "    - syntactic parsing\n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- More semantic tasks:\n",
    "    - summarization\n",
    "    - question answering\n",
    "    - information extraction (e.g. NER tagging)\n",
    "    - relation extraction \n",
    "    - chatbots\n",
    "    - machine translation\n",
    "    - ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Spacy\n",
    "\n",
    "- For demonstrating NLP tasks, we are going to use the library [spacy](https://spacy.io/) a lot.\n",
    "- It is an open-source NLP library for Python\n",
    "- It features a lot of out-of-the-box models for NLP\n",
    "- NER, POS tagging, dependency parsing, vectorization..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import spacy\n",
    "from spacy import displacy\n",
    "\n",
    "#loading the english model\n",
    "nlp = spacy.load('en')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Basic preprocessing tasks, text normalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<h3 id=\"Tokenization\">Tokenization</h3>\n",
    "<ul>\n",
    "<li>Splitting text into words, sentences, documents, etc..</li>\n",
    "<li>One of the goals of tokenizing text into words is to create a <strong>vocabulary</strong></li>\n",
    "</ul>\n",
    "<p><em>Muffins cost <strong>$3.88</strong> in New York. Please buy me two as I <strong>can't</strong> go. <strong>They'll</strong> taste good. I'm going to <strong>Finland's</strong> capital to hear about <strong>state-of-the-art</strong> solutions in NLP.</em></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- $3.88 - split on the period?\n",
    "- can't - can not?\n",
    "- They'll - they will?\n",
    "- Finland's - Finland?\n",
    "- state-of-the-art?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "sens = \"Muffins cost $3.88 in New York. Please buy me two as I can't go.\" \\\n",
    "\" They'll taste good. I'm going to Finland's capital to hear about state-of-the-art solutions in NLP.\"\n",
    "\n",
    "print(sens.split())\n",
    "\n",
    "print(len(sens.split()))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "sens = \"Muffins cost $3.88 in New York. Please buy me two as I can't go.\" \\\n",
    "\" They'll taste good. I'm going to Finland's capital to hear about state-of-the-art solutions in NLP.\"\n",
    "\n",
    "doc = nlp(sens)\n",
    "\n",
    "tokens = [token.text for token in doc]\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "for sen in doc.sents:\n",
    "    print(sen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "for token in doc:\n",
    "    print(token.text, token.is_alpha, token.is_stop)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Lemmatization, stemming\n",
    "\n",
    "- The goal of lemmatization is to find the dictionary form of the words\n",
    "- Called the \"lemma\" of a word\n",
    "- _dogs_ -> _dog_ , _went_ -> _go_\n",
    "- Ambiguity plays a role: _saw_ -> _see_?\n",
    "- Needs POS tag to disambiguate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "doc = nlp(\"I saw two dogs yesterday.\")\n",
    "\n",
    "lemmata = [token.lemma_ for token in doc]\n",
    "print(lemmata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### POS tagging\n",
    "\n",
    "- Words can be groupped into grammatical categories.\n",
    "- These are called the Part Of Speech tags of the words.\n",
    "- Words belonging to the same group are interchangable\n",
    "- Ambiguity: _guard_ ?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "doc = nlp(\"The white dog went to play football yesterday.\")\n",
    "\n",
    "[token.pos_ for token in doc]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<h3 id=\"Morphological-analysis\">Morphological analysis</h3>\n",
    "<ul>\n",
    "<li>Splitting words into morphemes</li>\n",
    "<li>Morphemes are the smallest meaningful units in a language (part of the words)</li>\n",
    "<li>friend<span style=\"color: #e03e2d;\">s</span>, wait<span style=\"color: #e03e2d;\">ing</span>, friend<span style=\"color: #e03e2d;\">li</span><span style=\"color: #3598db;\">er</span></li>\n",
    "<li>Tagging them with morphological tags</li>\n",
    "<li>Ambiguity: <em>v&aacute;rnak</em></li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "doc = nlp(\"Yesterday I went to buy two dogs\")\n",
    "nlp.vocab.morphology.tag_map[doc[-1].tag_]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Advanced tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Syntactic parsing\n",
    "\n",
    "\n",
    "-  *Colorless green ideas sleep furiously.* \n",
    "\n",
    "- *Furiously sleep ideas green colorless.*\n",
    "\n",
    "Chomsky (1956)\n",
    "\n",
    "\n",
    "Two types.\n",
    "- Phrase structure grammar\n",
    "- __Dependency grammar__\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Universal Dependency Parsing\n",
    "- Started and standardized in the [UD](http://universaldependencies.org/) project.\n",
    "- The types are Language-independent\n",
    "- The annotations are trying to be consistent accross 70+ languages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "doc = nlp(\"Colorless green ideas sleep furiously\")\n",
    "displacy.render(doc, style='dep', jupyter=True, options={'distance': 100})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Named entity recognition\n",
    "\n",
    "- Identify the present entities in the text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "sens = \"Muffins cost $3.88 in New York. Please buy me two as I can't go.\" \\\n",
    "\" They'll taste good. I'm going to Finland's capital to hear about state-of-the-art solutions in NLP.\"\n",
    "\n",
    "doc = nlp(sens)\n",
    "for ent in doc.ents:\n",
    "    print(ent)\n",
    "\n",
    "    \n",
    "displacy.render(doc, style='ent', jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Language modelling\n",
    "\n",
    "- One of the most important task in NLP\n",
    "- The goal is to compute the \"probability\" of a sentence\n",
    "- Can be used in:\n",
    "    - Machine Translation\n",
    "    - Text generation\n",
    "    - Correcting spelling\n",
    "    - Word vectors?\n",
    "- P(the quick brown __fox__) > P(the quick brown __stick__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "#!pip install transformers\n",
    "from transformers import pipeline\n",
    "\n",
    "text_generator = pipeline(\"text-generation\")\n",
    "print(text_generator(\"The quick brown \", max_length=10, do_sample=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Semantic tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Summarization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "summarizer = pipeline(\"summarization\")\n",
    "summarizer(\"Deep learning is used almost exclusively in a Linux environment.\\\n",
    "You need to be comfortable using the command line if you are serious about deep learning and NLP.\\\n",
    "    Most NLP and deep learning libraries have better support for Linux and MacOS than Windows. \\\n",
    "    Most papers nowadays release the source code for their experiments with Linux support only.\",\n",
    "           min_length=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Sentiment Analysis\n",
    "- In the simplest case, decide whether a text is negative or positive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "sentiment = pipeline(\"sentiment-analysis\")\n",
    "sentiment(['This class is really cool! I would recommend this to anyone!'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Question Answering\n",
    "\n",
    "- Given a context and a question choose the right answer\n",
    "- Can be extractive or abstractive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "question_answerer = pipeline('question-answering')\n",
    "question_answerer({\n",
    "    'question': 'Who went to the store ?',\n",
    "    'context': 'Adam went to the store yesterday.'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <center>Lexical Inference, Natural Language Inference</center>\n",
    "\n",
    "\n",
    "\n",
    "<div class=\"frame\">\n",
    "\n",
    "| **entailment**                                                |     |     |\n",
    "|:--------------------------------------------------------------|:----|:----|\n",
    "| A young family enjoys feeling ocean waves lap at their feet.  |     |     |\n",
    "| A family is at the beach                                      |     |     |\n",
    "| **contradiction**                                             |     |     |\n",
    "| There is no man wearing a black helmet and pushing a bicycle  |     |     |\n",
    "| One man is wearing a black helmet and pushing a bicycle       |     |     |\n",
    "| **neutral**                                                   |     |     |\n",
    "| An old man with a package poses in front of an advertisement. |     |     |\n",
    "| A man poses in front of an ad for beer.                       |     |     |\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Machine Comprehension\n",
    "\n",
    "- https://demo.allennlp.org/reading-comprehension"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Machine translation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "translation = pipeline(\"translation_en_to_de\")\n",
    "text = \"I like to study Data Science and Machine Learning\"\n",
    "translated_text = translation(text, max_length=40)[0]['translation_text']\n",
    "print(translated_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Chatbots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "#!pip install chatbotAI\n",
    "from chatbot import demo\n",
    "demo()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Demos\n",
    "    - http://e-magyar.hu/hu/parser\n",
    "    - https://demo.allennlp.org/\n",
    "    - https://talktotransformer.com/\n",
    "    - [GPT-3](https://github.com/elyase/awesome-gpt3) (*has 175B parameters*)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Representations\n",
    "\n",
    "To be able to run machine learning algorithms the computer needs numerical representations. For natural text input this means we need a mapping that converts strings to a numerical represenatation. **one-hot encoding** is the easiest approach where we map each word to an integer id.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "sentence = \"yesterday the lazy dog went to the store to buy food\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "mapping = dict()\n",
    "max_id = 0\n",
    "\n",
    "for word in sentence.split():\n",
    "    if word not in mapping:\n",
    "        mapping[word] = max_id\n",
    "        max_id = max_id + 1\n",
    "        \n",
    "print(mapping)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Load matplotlib and pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Data analysis\n",
    "\n",
    "- we use nlp frameworks for the basic tasks\n",
    "- for the preprocessing tasks (lemmatization, tokenization) we use [spaCy](https://spacy.io/)\n",
    "- for keyword extraction and various text analyzation tasks we use [textacy](https://github.com/chartbeat-labs/textacy)\n",
    "- textacy builds on spaCy output\n",
    "- both are open source python libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<p><strong>AG_NEWS</strong> classes:</p>\n",
    "<ul>\n",
    "<li>\n",
    "<p>World - <em>Venezuela Prepares for Chavez Recall Vote</em></p>\n",
    "</li>\n",
    "<li>\n",
    "<p>Sports - <em>Johnson Back to His Best as D-Backs End Streak</em></p>\n",
    "</li>\n",
    "<li>\n",
    "<p>Business - <em>Intel to delay product aimed for high-definition TVs</em></p>\n",
    "</li>\n",
    "<li>\n",
    "<p>Sci/Tech - <em>China's Red Flag Linux to focus on enterprise</em></p>\n",
    "</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "NGRAMS = 2\n",
    "from torchtext import data\n",
    "from torchtext.datasets import text_classification\n",
    "import os\n",
    "if not os.path.isdir('./data'):\n",
    "    os.mkdir('./data')\n",
    "text_classification.DATASETS['AG_NEWS'](\n",
    "    root='./data', ngrams=NGRAMS, vocab=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "train_data = pd.read_csv(\"./data/ag_news_csv/train.csv\",quotechar='\"', names=['label', 'title', 'description'])\n",
    "test_data = pd.read_csv(\"./data/ag_news_csv/test.csv\",quotechar='\"', names=['label', 'title', 'description'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "train_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "train_data.label.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "text_sports = train_data[train_data.label == 2]\n",
    "\n",
    "text = \" \".join(text_sports.title.tolist())\n",
    "doc_text = nlp(text[:200000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import textacy\n",
    "from textacy.extract import ngrams\n",
    "from collections import Counter\n",
    "\n",
    "Counter([ng.text.lower() for n in [2,4] for ng in ngrams(doc_text, n)]).most_common(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from textacy.ke import textrank\n",
    "\n",
    "textrank(\n",
    "    doc_text,\n",
    "    normalize = \"lemma\",\n",
    "    window_size=2, edge_weighting=\"binary\", position_bias=False\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "textrank(\n",
    "    doc_text,\n",
    "    window_size=10, edge_weighting=\"count\", position_bias=False\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import math\n",
    "from collections import Counter \n",
    "words = [tok for tok in doc_text if tok.is_alpha and not tok.is_stop]\n",
    "word_probs = {tok.text.lower(): tok.prob for tok in words}\n",
    "\n",
    "freqs = Counter(tok.text for tok in words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "#!pip install wordcloud\n",
    "from wordcloud import WordCloud\n",
    "print(len(freqs))\n",
    "wordcloud = WordCloud(background_color=\"white\", max_words=30, scale=1.5).generate_from_frequencies(freqs)\n",
    "image = wordcloud.to_image()\n",
    "image.save(\"./wordcloud.png\")\n",
    "\n",
    "from IPython.display import Image \n",
    "Image(filename='./wordcloud.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Building a classficiation pipeline\n",
    "\n",
    "The AG's news topic classification dataset is constructed by Xiang Zhang (xiang.zhang@nyu.edu) from the dataset above. It is used as a text classification benchmark in the following paper: Xiang Zhang, Junbo Zhao, Yann LeCun. Character-level Convolutional Networks for Text Classification. Advances in Neural Information Processing Systems 28 (NIPS 2015).\n",
    "\n",
    "The AG's news topic classification dataset is constructed by choosing 4 largest classes from the original corpus. Each class contains 30,000 training samples and 1,900 testing samples. The total number of training samples is 120,000 and testing 7,600.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "!pip install torchtext==0.4\n",
    "!pip install torch\n",
    "!pip install gensim\n",
    "!pip install scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "NGRAMS = 2\n",
    "from torchtext import data\n",
    "from torchtext.datasets import text_classification\n",
    "import os\n",
    "if not os.path.isdir('./data'):\n",
    "    os.mkdir('./data')\n",
    "text_classification.DATASETS['AG_NEWS'](\n",
    "    root='./data', ngrams=NGRAMS, vocab=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "#Import the needed libraries\n",
    "from tqdm import tqdm\n",
    "from sklearn.model_selection import train_test_split as split\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Now we use pandas to read in the dataset into a DataFrame. We are also going to just take a fraction of the dataset to be more efficient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "#1-World, 2-Sports, 3-Business, 4-Sci/Tech\n",
    "train_data = pd.read_csv(\"./data/ag_news_csv/train.csv\",quotechar='\"', names=['label', 'title', 'description'])\n",
    "test_data = pd.read_csv(\"./data/ag_news_csv/test.csv\",quotechar='\"', names=['label', 'title', 'description'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "train_data = train_data.groupby('label').apply(lambda x: x.sample(frac=0.2, random_state=1234)).sample(frac=1.0)\n",
    "\n",
    "train_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We need a way of converting raw data to features!\n",
    "![features](https://developers.google.com/machine-learning/crash-course/images/RawDataToFeatureVector.svg)\n",
    "\n",
    "_(image from [link](https://developers.google.com/machine-learning))_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The easiest way of converting raw data to features is called the [Bag of Words](https://en.wikipedia.org/wiki/Bag-of-words_model) model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "word_to_ix = defaultdict(int)\n",
    "for sent in train_data.title:\n",
    "    for word in sent.split():\n",
    "        if word not in word_to_ix:\n",
    "            word_to_ix[word] = len(word_to_ix)\n",
    "len(word_to_ix)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We are going to use Python's machine learning library, called scikit-learn to build a classical ML pipeline\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "corpora = ['hello my name is adam','i am the instructor for this class']\n",
    "# instantiate the vectorizer object\n",
    "vectorizer = CountVectorizer()\n",
    "# convert th documents into a matrix\n",
    "wm = vectorizer.fit_transform(corpora)\n",
    "#retrieve the terms found in the corpora\n",
    "tokens = vectorizer.get_feature_names()\n",
    "df_vect = pd.DataFrame(data = wm.toarray(),index = ['Doc1','Doc2'],columns = tokens)\n",
    "df_vect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "vectorizer = CountVectorizer(max_features=10000, stop_words=\"english\")\n",
    "\n",
    "X = vectorizer.fit(train_data.title)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "X.vocabulary_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "c = X.transform([\"Hello my name is adam\"]).toarray()\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Displaying the most frequent terms from CountVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "import matplotlib\n",
    "matplotlib.style.use('ggplot')\n",
    "matplotlib.pyplot.rcParams['figure.figsize'] = (16, 10)\n",
    "matplotlib.pyplot.rcParams['font.family'] = 'sans-serif'\n",
    "matplotlib.pyplot.rcParams['font.size'] = 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "vectorize = CountVectorizer(stop_words=\"english\")\n",
    "\n",
    "c = vectorize.fit(train_data.title)\n",
    "C = c.transform(train_data.title)\n",
    "\n",
    "summ = np.sum(C,axis=0)\n",
    "total = np.squeeze(np.asarray(summ))\n",
    "term_freq_df = pd.DataFrame([total],columns=c.get_feature_names()).transpose()\n",
    "term_freq_df.columns = [\"frequency\"]\n",
    "\n",
    "term_freq_df.sort_values(by=\"frequency\", ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "y_pos = np.arange(500)\n",
    "plt.figure(figsize=(10,8))\n",
    "s = 1\n",
    "expected_zipf = [term_freq_df.sort_values(by='frequency', ascending=False)['frequency'][0]/(i+1)**s for i in y_pos]\n",
    "plt.bar(y_pos, term_freq_df.sort_values(by='frequency', ascending=False)['frequency'][:500], align='center', alpha=0.5)\n",
    "plt.plot(y_pos, expected_zipf, color='r', linestyle='--',linewidth=2,alpha=0.5)\n",
    "plt.ylabel('Frequency')\n",
    "plt.title('Top 500 tokens')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We first build a feature extraction method that takes raw texts as input and runs builds features on the whole dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import gensim\n",
    "from tqdm import tqdm\n",
    "from sklearn.model_selection import train_test_split as split\n",
    "import numpy as np\n",
    "\n",
    "def vectorize_to_bow(tr_data, tst_data):\n",
    "    \n",
    "    tr_vectors = X.transform(tr_data)\n",
    "    \n",
    "    tst_vectors = X.transform(tst_data)\n",
    "    return tr_vectors, tst_vectors\n",
    "\n",
    "def get_features_and_labels(data, labels):\n",
    "    tr_data,tst_data,tr_labels,tst_labels = split(data,labels, test_size=0.3, random_state=1234)\n",
    "    \n",
    "    tst_vecs = []\n",
    "    tr_vecs = []\n",
    "    tr_vecs, tst_vecs = vectorize_to_bow(tr_data, tst_data)    \n",
    "    return tr_vecs, tr_labels, tst_vecs, tst_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "tr_vecs, tr_labels, tst_vecs, tst_labels = get_features_and_labels(train_data.title, train_data.label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "tr_vecs.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2 id=\"Machine-Learning\">Machine Learning</h2>\n",
    "<ul>\n",
    "<li>We have a datasets with labels</li>\n",
    "<li>We can train a machine learning algorithm using the labels as \"gold\" data - <span style=\"color: #e03e2d;\">Supervised learning</span></li>\n",
    "<li>The algorithm will predict unseen data points using the trained model</li>\n",
    "<li>We will use <a href=\"https://scikit-learn.org/stable/\" target=\"_blank\" rel=\"noopener\">sklearn</a> for the models</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Logistic Regression\n",
    "- One of the simplest method for classification tasks\n",
    "\n",
    "![lr](https://www.equiskill.com/wp-content/uploads/2018/07/WhatsApp-Image-2020-02-11-at-8.30.11-PM.jpeg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "#Import a bunch of stuff from sklearn\n",
    "from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "lr  = LogisticRegression(n_jobs=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "lr.fit(tr_vecs, tr_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "#!pip install eli5\n",
    "import eli5\n",
    "eli5.show_weights(lr, feature_names=X.get_feature_names())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "print(type(tst_vecs))\n",
    "\n",
    "lr_pred = lr.predict(tst_vecs)\n",
    "print(\"Logistic Regression Test accuracy : {}\".format(accuracy_score(tst_labels, lr_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Bag of words are the simplest method for featurizing your data. If we want a more sophisticated method, we could use [TF-IDf](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- __TF__: The term frequency of a word in a document. \n",
    "- __IDF__: The inverse document frequency of the word across a set of documents. This means, how common or rare a word is in the entire document set. The closer it is to 0, the more common a word is.\n",
    "- The higher the score, the more relevant that word is in that particular document"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "![tfidf](https://miro.medium.com/max/700/1*qQgnyPLDIkUmeZKN2_ZWbQ.png)\n",
    "\n",
    "_(image from [link](https://miro.medium.com))_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "vectorizer = TfidfVectorizer(max_features=1000, use_idf=True)\n",
    "vectors = vectorizer.fit(train_data.title)\n",
    "\n",
    "tfidf_vectorizer_vectors = vectors.transform(train_data.title)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "first_vector_tfidfvectorizer=tfidf_vectorizer_vectors[6] \n",
    " \n",
    "# place tf-idf values in a pandas data frame \n",
    "df = pd.DataFrame(first_vector_tfidfvectorizer.T.todense(), index=vectors.get_feature_names(), columns=[\"tfidf\"]) \n",
    "df = df.sort_values(by=[\"tfidf\"],ascending=False)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Sklearn allows us to build [pipelines](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) with defining each step of the pipeline, like:\n",
    "- Vectorizers\n",
    "- Classifiers\n",
    "- Voting strategies\n",
    "- Optionally merge feature extraction from multiple sources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Problems\n",
    "- When representing words with id's we assign them to the words in the order of the encounter. \n",
    "- This means that we may assign different vectors to the words each time we run the algorithm.\n",
    "- Doesn't include any concept of similarity, e.g: `similarity(embedding(cat, dog)) > similarity(embedding(cat, computer))`\n",
    "- The representation is very sparse and could have very high dimension, which would also slow the computations. The size is given by the vocabulary of our corpus, that can be over 100000 dimension."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Word embeddings\n",
    "\n",
    "- map each word to a small dimensional (around 100-300) continuous vectors.\n",
    "- this means that similar words should have similar vectors.\n",
    "    - what do we mean by word similarity ?\n",
    "    \n",
    "    \n",
    "### Cosine similarity\n",
    "\n",
    "- Now that we have word vectors, we need a way to quantify the similarity between individual words, according to these vectors. One such metric is cosine-similarity. We will be using this to find words that are \"close\" and \"far\" from one another."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](https://cmry.github.io/sources/eucos.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<h2 id=\"Creating-word-embeddings\">Creating word embeddings</h2>\n",
    "<p>\"a word is characterized by the company it keeps\" -- popularized by <em>John Rupert Firth</em></p>\n",
    "<ul>\n",
    "<li>A popular theory is that words are as similar as their context is</li>\n",
    "<li>Word embeddings are also created with neural networks that predicts the word's context from the word itself</li>\n",
    "</ul>\n",
    "<p>To create word embeddings, a neural network is trained to perform the tasks. But then it is not used actually for the task it was trained it on. The goal is actually to learn the weights of the hidden layer. Then, these weights will be our vectors called \"word embeddings\".</p>\n",
    "<p><strong>Neural Network?</strong></p>\n",
    "<p>Instead of computing the actual angle, we can leave the similarity in terms of <span class=\"MathJax_Preview\" style=\"color: inherit;\"><span id=\"MJXp-Span-73\" class=\"MJXp-math\"><span id=\"MJXp-Span-74\" class=\"MJXp-mi MJXp-italic\">s</span><span id=\"MJXp-Span-75\" class=\"MJXp-mi MJXp-italic\">i</span><span id=\"MJXp-Span-76\" class=\"MJXp-mi MJXp-italic\">m</span><span id=\"MJXp-Span-77\" class=\"MJXp-mi MJXp-italic\">i</span><span id=\"MJXp-Span-78\" class=\"MJXp-mi MJXp-italic\">l</span><span id=\"MJXp-Span-79\" class=\"MJXp-mi MJXp-italic\">a</span><span id=\"MJXp-Span-80\" class=\"MJXp-mi MJXp-italic\">r</span><span id=\"MJXp-Span-81\" class=\"MJXp-mi MJXp-italic\">i</span><span id=\"MJXp-Span-82\" class=\"MJXp-mi MJXp-italic\">t</span><span id=\"MJXp-Span-83\" class=\"MJXp-mi MJXp-italic\">y</span><span id=\"MJXp-Span-84\" class=\"MJXp-mo\" style=\"margin-left: 0.333em; margin-right: 0.333em;\">=</span><span id=\"MJXp-Span-85\" class=\"MJXp-mi MJXp-italic\">c</span><span id=\"MJXp-Span-86\" class=\"MJXp-mi MJXp-italic\">o</span><span id=\"MJXp-Span-87\" class=\"MJXp-mi MJXp-italic\">s</span><span id=\"MJXp-Span-88\" class=\"MJXp-mo\" style=\"margin-left: 0em; margin-right: 0em;\">(</span><span id=\"MJXp-Span-89\" class=\"MJXp-mi\">&Theta;</span><span id=\"MJXp-Span-90\" class=\"MJXp-mo\" style=\"margin-left: 0em; margin-right: 0em;\">)</span></span></span>. Formally the <a href=\"https://en.wikipedia.org/wiki/Cosine_similarity\" target=\"_blank\" rel=\"noopener\">Cosine Similarity</a> <span class=\"MathJax_Preview\" style=\"color: inherit;\"><span id=\"MJXp-Span-91\" class=\"MJXp-math\"><span id=\"MJXp-Span-92\" class=\"MJXp-mi MJXp-italic\">s</span></span></span> between two vectors <span class=\"MathJax_Preview\" style=\"color: inherit;\"><span id=\"MJXp-Span-93\" class=\"MJXp-math\"><span id=\"MJXp-Span-94\" class=\"MJXp-mi MJXp-italic\">p</span></span></span> and <span class=\"MathJax_Preview\" style=\"color: inherit;\"><span id=\"MJXp-Span-95\" class=\"MJXp-math\"><span id=\"MJXp-Span-96\" class=\"MJXp-mi MJXp-italic\">q</span></span></span> is defined as:</p>\n",
    "<p><span class=\"MathJax_Preview\" style=\"color: inherit;\"><span id=\"MJXp-Span-97\" class=\"MJXp-math MJXp-display\"><span id=\"MJXp-Span-98\" class=\"MJXp-mi MJXp-italic\">s</span><span id=\"MJXp-Span-99\" class=\"MJXp-mo\" style=\"margin-left: 0.333em; margin-right: 0.333em;\">=</span><span id=\"MJXp-Span-100\" class=\"MJXp-mfrac\" style=\"vertical-align: 0.25em;\"><span class=\"MJXp-box\"><span id=\"MJXp-Span-101\" class=\"MJXp-mi MJXp-italic\">p</span><span id=\"MJXp-Span-102\" class=\"MJXp-mo\" style=\"margin-left: 0.267em; margin-right: 0.267em;\">&sdot;</span><span id=\"MJXp-Span-103\" class=\"MJXp-mi MJXp-italic\">q</span></span><span class=\"MJXp-box\" style=\"margin-top: -0.9em;\"><span class=\"MJXp-denom\"><span class=\"MJXp-box\"><span id=\"MJXp-Span-104\" class=\"MJXp-mrow\"><span id=\"MJXp-Span-105\" class=\"MJXp-mo\" style=\"margin-left: 0.167em; margin-right: 0.167em;\">|</span></span><span id=\"MJXp-Span-106\" class=\"MJXp-mrow\"><span id=\"MJXp-Span-107\" class=\"MJXp-mo\" style=\"margin-left: 0.167em; margin-right: 0.167em;\">|</span></span><span id=\"MJXp-Span-108\" class=\"MJXp-mi MJXp-italic\">p</span><span id=\"MJXp-Span-109\" class=\"MJXp-mrow\"><span id=\"MJXp-Span-110\" class=\"MJXp-mo\" style=\"margin-left: 0.167em; margin-right: 0.167em;\">|</span></span><span id=\"MJXp-Span-111\" class=\"MJXp-mrow\"><span id=\"MJXp-Span-112\" class=\"MJXp-mo\" style=\"margin-left: 0.167em; margin-right: 0.167em;\">|</span></span><span id=\"MJXp-Span-113\" class=\"MJXp-mrow\"><span id=\"MJXp-Span-114\" class=\"MJXp-mo\" style=\"margin-left: 0.167em; margin-right: 0.167em;\">|</span></span><span id=\"MJXp-Span-115\" class=\"MJXp-mrow\"><span id=\"MJXp-Span-116\" class=\"MJXp-mo\" style=\"margin-left: 0.167em; margin-right: 0.167em;\">|</span></span><span id=\"MJXp-Span-117\" class=\"MJXp-mi MJXp-italic\">q</span><span id=\"MJXp-Span-118\" class=\"MJXp-mrow\"><span id=\"MJXp-Span-119\" class=\"MJXp-mo\" style=\"margin-left: 0.167em; margin-right: 0.167em;\">|</span></span><span id=\"MJXp-Span-120\" class=\"MJXp-mrow\"><span id=\"MJXp-Span-121\" class=\"MJXp-mo\" style=\"margin-left: 0.167em; margin-right: 0.167em;\">|</span></span></span></span></span></span><span id=\"MJXp-Span-122\" class=\"MJXp-mo\" style=\"margin-left: 0em; margin-right: 0.222em;\">,</span><span id=\"MJXp-Span-123\" class=\"MJXp-mrow\"><span id=\"MJXp-Span-124\" class=\"MJXp-mtext\">&nbsp;where&nbsp;</span></span><span id=\"MJXp-Span-125\" class=\"MJXp-mi MJXp-italic\">s</span><span id=\"MJXp-Span-126\" class=\"MJXp-mo\" style=\"margin-left: 0.333em; margin-right: 0.333em;\">&isin;</span><span id=\"MJXp-Span-127\" class=\"MJXp-mo\" style=\"margin-left: 0em; margin-right: 0em;\">[</span><span id=\"MJXp-Span-128\" class=\"MJXp-mo\" style=\"margin-left: 0.267em; margin-right: 0.267em;\">&minus;</span><span id=\"MJXp-Span-129\" class=\"MJXp-mn\">1</span><span id=\"MJXp-Span-130\" class=\"MJXp-mo\" style=\"margin-left: 0em; margin-right: 0.222em;\">,</span><span id=\"MJXp-Span-131\" class=\"MJXp-mn\">1</span><span id=\"MJXp-Span-132\" class=\"MJXp-mo\" style=\"margin-left: 0em; margin-right: 0em;\">]</span></span></span></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Creating word embeddings 2\n",
    "\n",
    "Word embeddings are learned with neural networks. The target can be:\n",
    "\n",
    "- Tries to predict the word given the context - The Continous Bag Of Words model (CBOW)\n",
    "- Tries to predict the context given a words - The SkipGram model\n",
    "\n",
    "The training examples are generated from big text corpora. For example from the sentence “The quick brown fox jumps over the lazy dog.” we can generate the following inputs:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![training examples](http://mccormickml.com/assets/word2vec/training_data.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To do this, we first build a vocabulary of words from our training documents–let’s say we have a vocabulary of 10,000 unique words. The vocabulary of big corporas can be much more then 10,000 unique words, to handle them we usually substitute rare words with a special token (this is usually the _UNK_ token).\n",
    "\n",
    "First we build the vocabulary of our documents, then for representing words, we will use one-hot vectors. The output of the network will be a single vector that contains the probabilities for the \"nearby\" words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Famous static word embeddings for English\n",
    "\n",
    "- [Word2vec](https://arxiv.org/pdf/1301.3781.pdf)\n",
    "- [GLOVE](https://nlp.stanford.edu/projects/glove/)\n",
    "\n",
    "### Contextual embeddings?\n",
    "\n",
    "- [Elmo](https://allennlp.org/elmo)\n",
    "- [BERT](https://arxiv.org/abs/1810.04805)\n",
    "- [Flair](https://www.aclweb.org/anthology/N19-4010/)\n",
    "\n",
    "For static embeddings, we will use a GLOVE embedding of 100 dimensional vectors trained on 6B tokens.\n",
    "\n",
    "[Download GLOVE](http://sandbox.hlt.bme.hu/~adaamko/glove.6B.100d.txt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Using word embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "As we discussed, more recently prediction-based word vectors have demonstrated better performance, such as word2vec and GloVe (which also utilizes the benefit of counts). Here, we shall explore the embeddings produced by GloVe. If you want to know more about embeddings, try reading [GloVe's original paper](https://nlp.stanford.edu/pubs/glove.pdf)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "embedding_file = \"glove.6B.100d.txt\"\n",
    "\n",
    "embedding = gensim.models.KeyedVectors.load_word2vec_format(embedding_file, binary=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "dog_vector = embedding[\"dog\"]\n",
    "dog_vector.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "embedding.most_similar(\"president\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "embedding.most_similar(positive=['woman', 'king'], negative=['man'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "embedding.similarity(\"woman\", \"computer\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.manifold import TSNE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def tsne_plot(model, size=500):\n",
    "    \"Creates and TSNE model and plots it\"\n",
    "    labels = []\n",
    "    tokens = []\n",
    "    \n",
    "    for i, word in enumerate(model.wv.vocab):\n",
    "        if len(tokens) > size:\n",
    "            break\n",
    "        tokens.append(model[word])\n",
    "        labels.append(word)\n",
    "    \n",
    "    tsne_model = TSNE(perplexity=40, n_components=2, init='pca', n_iter=2500, random_state=23)\n",
    "    new_values = tsne_model.fit_transform(tokens)\n",
    "\n",
    "    x = []\n",
    "    y = []\n",
    "    for value in new_values:\n",
    "        x.append(value[0])\n",
    "        y.append(value[1])\n",
    "        \n",
    "    for i in range(len(x)):\n",
    "        plt.scatter(x[i],y[i])\n",
    "        plt.annotate(labels[i],\n",
    "                     xy=(x[i], y[i]),\n",
    "                     xytext=(5, 2),\n",
    "                     textcoords='offset points',\n",
    "                     ha='right',\n",
    "                     va='bottom')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "tsne_plot(embedding, 100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Analogies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def analogy(word1, word2, word3, n=5):\n",
    "    \n",
    "    #get vectors for each word\n",
    "    word1_vector = embedding[word1]\n",
    "    word2_vector = embedding[word2]\n",
    "    word3_vector = embedding[word3]\n",
    "    \n",
    "    #calculate analogy vector\n",
    "    analogy_vector = embedding.most_similar(positive=[word3, word2], negative=[word1])\n",
    "    \n",
    "    print(word1 + \" is to \" + word2 + \" as \" + word3 + \" is to...\")\n",
    "    \n",
    "    return analogy_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "analogy('man', 'king', 'woman')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Spacy also has pretrained embeddings!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "doc = nlp(\"man woman\")\n",
    "cosine_similarity(doc[0].vector.reshape(1, -1), doc[1].vector.reshape(1, -1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "__Or with using the built in similarity function:__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "nlp(\"My name is adam\").similarity(nlp(\"My name is andrea\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Contextual embeddings\n",
    "\n",
    "In GloVe and Word2vec representations, words have a static representation. But words can have different meaning in different contexts, e.g. the word \"stick\":\n",
    "\n",
    "1. Find some dry sticks and we'll make a campfire.\n",
    "2. Let's stick with glove embeddings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![elmo](http://jalammar.github.io/images/elmo-embedding-robin-williams.png)\n",
    "\n",
    "_(Peters et. al., 2018 in the ELMo paper)_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# The sentence objects holds a sentence that we may want to embed or tag\n",
    "from flair.data import Sentence\n",
    "from flair.embeddings import FlairEmbeddings\n",
    "\n",
    "# init embedding\n",
    "flair_embedding_forward = FlairEmbeddings('news-forward')\n",
    "\n",
    "# create a sentence\n",
    "sentence1 = Sentence(\"Find some dry sticks and we'll make a campfire.\")\n",
    "sentence2 = Sentence(\"Let's stick with glove embeddings.\")\n",
    "\n",
    "# embed words in sentence\n",
    "flair_embedding_forward.embed(sentence2)\n",
    "for token in sentence2:\n",
    "    print(token)\n",
    "    print(token.embedding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "In Flair, a pretrained NER tagger is also available for use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}