NLP_Preprocessing.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "NLP_Preprocessing.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EML680rzvhd2"
      },
      "source": [
        "# **Preprocessing NLP - Tutoriel pour nettoyer rapidement un texte** - [voir l'article](https://inside-machinelearning.com/preprocessing-nlp-tutoriel-pour-nettoyer-rapidement-un-texte/)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4cE99te-vw55"
      },
      "source": [
        "Je vous propose aujourd'hui **un tutoriel de Preprocessing NLP** pour voir en détail **comment nettoyer des données textes !**\n",
        " \n",
        "On va voir **plusieurs approches** qui seront adaptable autant aux **textes en anglais** qu'aux **textes en français**.\n",
        " \n",
        "Ensuite, on verra comment **encoder ces données** en format compréhensible, **interprétable** par nos modèles de **Machine Learning et Deep Learning.**\n",
        " \n",
        "**C'est parti !**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e8tq0Yk-8liP"
      },
      "source": [
        "# **Preprocessing**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QrVnaM4L0mTm"
      },
      "source": [
        "## **Charger les données**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YlClXkP9xZd-"
      },
      "source": [
        "**Premièrement**, comme à notre habitude, on va **charger nos données**.\n",
        " \n",
        "Ici on prend un **fichier csv** tiré de [cette compétition Kaggle](https://www.kaggle.com/c/nlp-getting-started) contenant **plusieurs milliers de phrases en anglais.**\n",
        " \n",
        "**Parfait pour nous ;)**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4oOZTiIEx-8r"
      },
      "source": [
        "On charge les **phrases** depuis ce [répertoire Github](https://github.com/tkeldenich/NLP_Preprocessing)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n6pnywgf5Hj6"
      },
      "source": [
        "!git clone https://github.com/tkeldenich/NLP_Preprocessing.git &> /dev/null"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MGAjurnoyFyg"
      },
      "source": [
        "Puis on **insère ces données** dans un **Dataframe Pandas**."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "Eh3cHaty53XG",
        "outputId": "01a087bb-a3d5-4f6a-b22a-54057dba35bc"
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "train_data = pd.read_csv('/content/NLP_Preprocessing/train.csv')\n",
        "\n",
        "train_data.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Forest fire near La Ronge Sask. Canada</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>All residents asked to 'shelter in place' are ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>13,000 people receive #wildfires evacuation or...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Just got sent this photo from Ruby #Alaska as ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>#RockyFire Update =&gt; California Hwy. 20 closed...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                text\n",
              "0             Forest fire near La Ronge Sask. Canada\n",
              "1  All residents asked to 'shelter in place' are ...\n",
              "2  13,000 people receive #wildfires evacuation or...\n",
              "3  Just got sent this photo from Ruby #Alaska as ...\n",
              "4  #RockyFire Update => California Hwy. 20 closed..."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mwsTsoFN-hcI"
      },
      "source": [
        "## **Nettoyer les données**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WVoHZ6_lfhqj"
      },
      "source": [
        "Une fois que les **données sont chargées** il faut les **nettoyer**, faire ce qu'on appelle un **preprocessing**.\n",
        " \n",
        "**Dans la plupart des cas pour du NLP**, le preprocessing consiste à **enlever les caractères** qui ne sont pas des lettres comme \"#\", \"-\", \"!\", les **chiffres** ou bien encore **les mots qui n'ont pas de sens** ou qui ne font pas partie de la langue analysée.\n",
        " \n",
        "**Garder en tête** cependant que pour certains **type de problèmes** il peut être intéressant de **préserver certains types de caractères.**\n",
        " \n",
        "Par exemple : pour analyser **si un email est un spam ou non**, on peut imaginer que **les '!' sont un bon indicateur** et donc **ne pas les enlever lors du nettoyage.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2U5jbOlGSXKA"
      },
      "source": [
        "Nous allons ici **coder deux fonctions :**\n",
        "- une pour **nettoyer nos phrases en langue anglaise**\n",
        "- une pour **nettoyer nos phrases en langue française**\n",
        " \n",
        "Ces deux fonctions ont, par ailleurs, **la même architecture.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IA9CcTw-Cnlp"
      },
      "source": [
        "### **Texte en anglais**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tucQs48Cjype"
      },
      "source": [
        "Premièrement on **importe** toutes les **librairies nécessaires :**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7dOIHAznmhIn",
        "outputId": "b96360b0-a3d0-4623-a57d-33b09c3bd9f5"
      },
      "source": [
        "import nltk\n",
        "import string\n",
        "from nltk.stem import WordNetLemmatizer \n",
        "\n",
        "nltk.download('stopwords')\n",
        "nltk.download('punkt')\n",
        "nltk.download('words')\n",
        "nltk.download('wordnet')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n",
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n",
            "[nltk_data] Downloading package words to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/words.zip.\n",
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/wordnet.zip.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XP6KWEwWmv_b"
      },
      "source": [
        "Ensuite on **initialise** :\n",
        "- les **stopwords**, ce sont les mots qui apparaissent très fréquemment mais qui n'apporte pas de sens à la phrase (comme \"de\", \"le\", \"une\")\n",
        "- les mots (**words**) qui proviennent d'un dictionnaire **anglais**\n",
        "- un **lemmatizer**, cette objet nous permet de préserver la racine des mots de tel sorte que deux mots ayant une même souche seront considérés comme un seul et même mot (exemple : 'voisine' et 'voisinage' seront tous deux changer en 'voisin')"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tAw5L6O_mjpK"
      },
      "source": [
        "stopwords = nltk.corpus.stopwords.words('english')\n",
        "words = set(nltk.corpus.words.words())\n",
        "lemmatizer = WordNetLemmatizer()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tp8R_SVCmofm"
      },
      "source": [
        "Puis on construit notre **fonction de preprocessing** qui va successivement :\n",
        "- enlever la **ponctuation**\n",
        "- enlever les **chiffres**\n",
        "- transformer les phrases en **liste de tokens** (en liste de mots)\n",
        "- enlever les **stopwords** (mots n'apportant pas de sens)\n",
        "- **lemmatizer**\n",
        "- enlever les **majuscules**\n",
        "- **reformer les phrases** avec les mots restants"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IekfBPBg8c6L"
      },
      "source": [
        "def Preprocess_listofSentence(listofSentence):\n",
        "    preprocess_list = []\n",
        "    for sentence in listofSentence :\n",
        "        \n",
        "        sentence_w_punct = \"\".join([i.lower() for i in sentence if i not in string.punctuation])\n",
        "        \n",
        "        sentence_w_num = ''.join(i for i in sentence_w_punct if not i.isdigit())\n",
        "\n",
        "        tokenize_sentence = nltk.tokenize.word_tokenize(sentence_w_num)\n",
        "        \n",
        "        words_w_stopwords = [i for i in tokenize_sentence if i not in stopwords]\n",
        "        \n",
        "        words_lemmatize = (lemmatizer.lemmatize(w) for w in words_w_stopwords)\n",
        "\n",
        "        sentence_clean = ' '.join(w for w in words_lemmatize if w.lower() in words or not w.isalpha())\n",
        "        \n",
        "        preprocess_list.append(sentence_clean)\n",
        "    \n",
        "    return preprocess_list "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eYojRBPCnYti"
      },
      "source": [
        "Puis on l'**utilise :**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uP3geUUC8pCo"
      },
      "source": [
        "preprocess_list = Preprocess_listofSentence(train_data['text'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v7aAX1iTneW0"
      },
      "source": [
        "Ensuite on peut **afficher** un exemple de **phrase nettoyée :**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "B9Gev3yr-oy9",
        "outputId": "a0cf105d-9d21-434b-8e26-5b1eabbd690e"
      },
      "source": [
        "print('Phrase de base : '+train_data['text'][2])\n",
        "print('Phrase nettoyée : '+preprocess_list[2])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Phrase de base : 13,000 people receive #wildfires evacuation orders in California \n",
            "Phrase nettoyée : people receive wildfire evacuation order\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "poLi6m_VCtKZ"
      },
      "source": [
        "### **Texte en français**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b3yf6PNpj8ID"
      },
      "source": [
        "Ici on va d'abord installer **la librairie** [FrenchLefffLemmatizer](https://github.com/ClaudeCoulombe/FrenchLefffLemmatizer) qui permet d'effectuer une **lemmatization en français.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xgDU1JV4tCLw"
      },
      "source": [
        " !pip install git+https://github.com/ClaudeCoulombe/FrenchLefffLemmatizer.git &> /dev/null"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tMQXjLJnok2l"
      },
      "source": [
        "On **importe** ensuite les librairies."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qD-vu7LKoljx",
        "outputId": "b3dac47d-87c5-4927-b40d-dded04f3dc41"
      },
      "source": [
        "import nltk\n",
        "import string\n",
        "from french_lefff_lemmatizer.french_lefff_lemmatizer import FrenchLefffLemmatizer\n",
        " \n",
        "nltk.download('wordnet')\n",
        "nltk.download('punkt')\n",
        "nltk.download('stopwords')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Package wordnet is already up-to-date!\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mhYzTzIxpHCh"
      },
      "source": [
        "Ensuite on **initialise** :\n",
        "- les **stopwords**, ce sont les mots qui apparaissent très fréquemment mais qui n'apporte pas de sens à la phrase (comme \"de\", \"le\", \"une\")\n",
        "- les **mots** qui proviennent d'un dictionnaire **français**\n",
        "- un **lemmatizer**, cette objet nous permet de préserver la racine des mots de tel sorte que deux mots ayant une même souche seront considérés comme un même mot (exemple : 'voisine' et 'voisinage' seront tous deux changer en 'voisin')"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iVNXfml_ovCQ"
      },
      "source": [
        "french_stopwords = nltk.corpus.stopwords.words('french')\n",
        "mots = set(line.strip() for line in open('/content/NLP_Preprocessing/dictionnaire.txt'))\n",
        "lemmatizer = FrenchLefffLemmatizer()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E_x-Hw3JpY7e"
      },
      "source": [
        "Puis on construit notre fonction de preprocessing qui va successivement :\n",
        "- enlever la **ponctuation**\n",
        "- enlever les **chiffres**\n",
        "- transformer les phrases en **liste de tokens** (en liste de mots)\n",
        "- enlever les **stopwords** (mots n'apportant pas de sens)\n",
        "- **lemmatizer**\n",
        "- garder seulement **les mots présent dans le dictionnaire**\n",
        "- enlever les **majuscules**\n",
        "- **reformer les phrases** avec les mots restant"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-_yAX-J3rOxO"
      },
      "source": [
        "def French_Preprocess_listofSentence(listofSentence):\n",
        "  preprocess_list = []\n",
        "  for sentence in listofSentence :\n",
        "        \n",
        "    sentence_w_punct = \"\".join([i.lower() for i in sentence if i not in string.punctuation])\n",
        "        \n",
        "    sentence_w_num = ''.join(i for i in sentence_w_punct if not i.isdigit())\n",
        " \n",
        "    tokenize_sentence = nltk.tokenize.word_tokenize(sentence_w_num)\n",
        "        \n",
        "    words_w_stopwords = [i for i in tokenize_sentence if i not in french_stopwords]\n",
        "        \n",
        "    words_lemmatize = (lemmatizer.lemmatize(w) for w in words_w_stopwords)\n",
        " \n",
        "    sentence_clean = ' '.join(w for w in words_lemmatize if w.lower() in mots or not w.isalpha())\n",
        "        \n",
        "    preprocess_list.append(sentence_clean)\n",
        "    \n",
        "  return preprocess_list"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ra1Y7e2UprV5"
      },
      "source": [
        "On crée des données pour **tester notre fonction :**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BuBkpm2ssoKY"
      },
      "source": [
        "lst = ['C\\'est un test pour lemmatizer',\n",
        "       'plusieurs phrases pour un nettoyage',\n",
        "       'eh voilà la troisième !']\n",
        "french_text = pd.DataFrame(lst, columns =['text'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GS3Au-SApyAc"
      },
      "source": [
        "Ensuite on l'**utilise :**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hHOgJJ7CrO4T"
      },
      "source": [
        "french_preprocess_list = French_Preprocess_listofSentence(french_text['text'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wTVo7lRcqJSQ"
      },
      "source": [
        "Et on **regarde le résultat :**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PgdWpgHCrPJm",
        "outputId": "201ebf78-511c-4391-fc00-69c8871b6f71"
      },
      "source": [
        "print('Phrase de base : '+lst[1])\n",
        "print('Phrase nettoyée : '+french_preprocess_list[1])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Phrase de base : plusieurs phrases pour un nettoyage\n",
            "Phrase nettoyée : plusieurs phrase nettoyage\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yQs34jL3_ROy"
      },
      "source": [
        "# **Les différents encodages**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QRRNaFlZvoeA"
      },
      "source": [
        "Une fois qu'on a **extrait les informations utiles** de nos phrases, on peut passer à **la phase d'encodage.**\n",
        " \n",
        "L'encodage est une **étape essentielle** pour pouvoir faire du **Machine Learning.**\n",
        " \n",
        "En effet, il permet de **transformer les données texte en chiffres** que la machine peut interprérer, **que la machine peut comprendre**.\n",
        " \n",
        "Il existe **différents types d'encodage** et nous allons dès maintenant aborder **les plus connus !**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "60trRt8HAOpM"
      },
      "source": [
        "## **One-Hot Encoding**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A-AcWETHwcCk"
      },
      "source": [
        "Le **One-Hot Encoding** est à la fois **la méthode la plus connu**, la **plus simple à réaliser**, et celle qui m'a permis d'avoir **la meilleure précision** dans la plupart de **mes travaux personnels en NLP.**\n",
        " \n",
        "Le One-Hot consite à **créer un dictionnaire** avec **tous les mots qui apparaissent** dans nos phrases nettoyées.\n",
        " \n",
        "Ce dictionnaire est en fait **un tableau** où **chaque colonne représente un mot** et **chaque ligne représente une phrase**.\n",
        " \n",
        "Si **tel mot apparaît dans tel phrase**, on met une **valeur de 1** dans l'élément du tableau, **sinon** on met une **valeur de 0.**\n",
        " \n",
        "On aura donc un **tableau composé uniquement de 0 et de 1**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IQWScdmX0sf2"
      },
      "source": [
        "Pour réaliser le **One-Hot Encoding** en **Python**, on initialise le **dictionnaire** avec la fonction *CountVectorizer()* de la librairie *Sklearn*.\n",
        " \n",
        "Puis on utilise la **fonction** *fit_transform()* sur **nos données preprocessées.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eeRWaRr-_IRQ"
      },
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        " \n",
        "vectorizer = CountVectorizer()\n",
        " \n",
        "X = vectorizer.fit_transform(preprocess_list)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gBDpcTwH_afh",
        "outputId": "d87ea6ab-5966-4e0a-ae14-a21fb519f64f"
      },
      "source": [
        "X.toarray()[0]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0, 0, 0, ..., 0, 0, 0])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KfLu-9ymzQy9"
      },
      "source": [
        "En fait la **classe** *vectorizer* garde beaucoup d'autres **informations sur le dictionnaire.**\n",
        " \n",
        "En outre, si nous voulons **encoder de nouvelles phrases** pour utiliser notre **modèle de Machine Learning entraîné** il faudra utiliser la **fonction** *fit()* de la **classe** *vectorizer*.\n",
        " \n",
        "On peut ainsi **adapter ces nouvelles phrases à notre dictionnaire.** Cela implique néanmoins que si ces nouvelles phrases contiennent **un mot qui n'est pas dans le dictionnaire**, il ne sera **pas pris en compte.**\n",
        " \n",
        "On peut voir **les mots composants ce dictionnaire** avec la **fonction**  *get_feature_names()* de *vectorizer*."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hAlU4FCF_cJQ",
        "outputId": "fba9acbd-2fb1-42e5-dd30-e7361a372916"
      },
      "source": [
        "vectorizer.get_feature_names()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['aa',\n",
              " 'aal',\n",
              " 'aba',\n",
              " 'abandon',\n",
              " 'abandoned',\n",
              " 'ability',\n",
              " 'abject',\n",
              " 'ablaze',\n",
              " 'able',\n",
              " 'aboard',\n",
              " 'abomination',\n",
              " 'abortion',\n",
              " 'abouts',\n",
              " 'absence',\n",
              " 'absolute',\n",
              " 'absolutely',\n",
              " 'abstract',\n",
              " 'absurd',\n",
              " 'absurdly',\n",
              " 'abuse',\n",
              " 'accept',\n",
              " 'access',\n",
              " 'accident',\n",
              " 'accidentally',\n",
              " 'accidently',\n",
              " 'accidents',\n",
              " 'according',\n",
              " 'accordingly',\n",
              " 'account',\n",
              " 'accountable',\n",
              " 'accuracy',\n",
              " 'accused',\n",
              " 'accustomed',\n",
              " 'ace',\n",
              " 'achieve',\n",
              " 'achievement',\n",
              " 'aching',\n",
              " 'acid',\n",
              " 'acids',\n",
              " 'acne',\n",
              " 'acoustic',\n",
              " 'acquiesce',\n",
              " 'acquire',\n",
              " 'acquired',\n",
              " 'acquisition',\n",
              " 'acre',\n",
              " 'acronym',\n",
              " 'across',\n",
              " 'acrylic',\n",
              " 'act',\n",
              " 'actin',\n",
              " 'acting',\n",
              " 'action',\n",
              " 'activate',\n",
              " 'active',\n",
              " 'actively',\n",
              " 'activist',\n",
              " 'activity',\n",
              " 'actor',\n",
              " 'actress',\n",
              " 'actual',\n",
              " 'actually',\n",
              " 'acute',\n",
              " 'ad',\n",
              " 'adaptation',\n",
              " 'add',\n",
              " 'added',\n",
              " 'addict',\n",
              " 'addiction',\n",
              " 'addition',\n",
              " 'address',\n",
              " 'adjust',\n",
              " 'adjustable',\n",
              " 'adjuster',\n",
              " 'administration',\n",
              " 'administrative',\n",
              " 'admit',\n",
              " 'adopt',\n",
              " 'adoption',\n",
              " 'adoptive',\n",
              " 'adorable',\n",
              " 'adult',\n",
              " 'advance',\n",
              " 'advanced',\n",
              " 'advantage',\n",
              " 'adventure',\n",
              " 'adverse',\n",
              " 'advertise',\n",
              " 'advice',\n",
              " 'advised',\n",
              " 'advisory',\n",
              " 'aeroplane',\n",
              " 'aesthetic',\n",
              " 'affair',\n",
              " 'affect',\n",
              " 'affected',\n",
              " 'affecting',\n",
              " 'affiliate',\n",
              " 'affiliation',\n",
              " 'affliction',\n",
              " 'afloat',\n",
              " 'afraid',\n",
              " 'afterhours',\n",
              " 'afterlife',\n",
              " 'aftermath',\n",
              " 'afternoon',\n",
              " 'aftershock',\n",
              " 'afterwards',\n",
              " 'agalloch',\n",
              " 'age',\n",
              " 'agency',\n",
              " 'agent',\n",
              " 'aggression',\n",
              " 'aggressive',\n",
              " 'aggressively',\n",
              " 'agnus',\n",
              " 'ago',\n",
              " 'agony',\n",
              " 'agree',\n",
              " 'agreed',\n",
              " 'agreement',\n",
              " 'ah',\n",
              " 'ahead',\n",
              " 'ai',\n",
              " 'aid',\n",
              " 'aim',\n",
              " 'aimlessly',\n",
              " 'aint',\n",
              " 'air',\n",
              " 'aircraft',\n",
              " 'airhead',\n",
              " 'airing',\n",
              " 'airlift',\n",
              " 'airplane',\n",
              " 'airport',\n",
              " 'aisle',\n",
              " 'ak',\n",
              " 'aka',\n",
              " 'al',\n",
              " 'ala',\n",
              " 'alameda',\n",
              " 'alarm',\n",
              " 'alarmed',\n",
              " 'alarming',\n",
              " 'alarmingly',\n",
              " 'alaska',\n",
              " 'alba',\n",
              " 'albeit',\n",
              " 'album',\n",
              " 'alchemist',\n",
              " 'alcohol',\n",
              " 'alec',\n",
              " 'alert',\n",
              " 'algae',\n",
              " 'alien',\n",
              " 'align',\n",
              " 'alive',\n",
              " 'allay',\n",
              " 'allegation',\n",
              " 'allegedly',\n",
              " 'allegiance',\n",
              " 'allergic',\n",
              " 'alley',\n",
              " 'alliance',\n",
              " 'allied',\n",
              " 'alloosh',\n",
              " 'allotment',\n",
              " 'allow',\n",
              " 'alloy',\n",
              " 'ally',\n",
              " 'almighty',\n",
              " 'almost',\n",
              " 'alone',\n",
              " 'along',\n",
              " 'alp',\n",
              " 'alpha',\n",
              " 'already',\n",
              " 'alright',\n",
              " 'alrighty',\n",
              " 'also',\n",
              " 'alt',\n",
              " 'alternate',\n",
              " 'alternative',\n",
              " 'although',\n",
              " 'aluminum',\n",
              " 'always',\n",
              " 'ama',\n",
              " 'amateur',\n",
              " 'amazed',\n",
              " 'amazing',\n",
              " 'amazon',\n",
              " 'amber',\n",
              " 'ambition',\n",
              " 'ambulance',\n",
              " 'amen',\n",
              " 'amends',\n",
              " 'america',\n",
              " 'amid',\n",
              " 'amino',\n",
              " 'amman',\n",
              " 'among',\n",
              " 'amongst',\n",
              " 'amplifier',\n",
              " 'an',\n",
              " 'ana',\n",
              " 'analysis',\n",
              " 'anarchy',\n",
              " 'anatomy',\n",
              " 'anchor',\n",
              " 'anchorage',\n",
              " 'ancient',\n",
              " 'and',\n",
              " 'android',\n",
              " 'anew',\n",
              " 'angel',\n",
              " 'angelriveralib',\n",
              " 'anger',\n",
              " 'angioplasty',\n",
              " 'angry',\n",
              " 'ani',\n",
              " 'animal',\n",
              " 'animation',\n",
              " 'anime',\n",
              " 'ankle',\n",
              " 'anna',\n",
              " 'annihilate',\n",
              " 'annihilation',\n",
              " 'anniversary',\n",
              " 'announce',\n",
              " 'announcement',\n",
              " 'annoying',\n",
              " 'annual',\n",
              " 'anonymous',\n",
              " 'another',\n",
              " 'answer',\n",
              " 'ant',\n",
              " 'ante',\n",
              " 'anthelmintic',\n",
              " 'anthology',\n",
              " 'anthrax',\n",
              " 'anti',\n",
              " 'antichrist',\n",
              " 'antifeminist',\n",
              " 'anxiety',\n",
              " 'anxious',\n",
              " 'anybody',\n",
              " 'anyone',\n",
              " 'anything',\n",
              " 'anyway',\n",
              " 'anyways',\n",
              " 'anywhere',\n",
              " 'apart',\n",
              " 'apartment',\n",
              " 'apiece',\n",
              " 'apocalypse',\n",
              " 'apocalyptic',\n",
              " 'apologize',\n",
              " 'apology',\n",
              " 'appalling',\n",
              " 'apparent',\n",
              " 'apparently',\n",
              " 'appeal',\n",
              " 'appease',\n",
              " 'apperception',\n",
              " 'appetite',\n",
              " 'applaud',\n",
              " 'apple',\n",
              " 'application',\n",
              " 'applied',\n",
              " 'apply',\n",
              " 'appointment',\n",
              " 'appraisal',\n",
              " 'appreciate',\n",
              " 'approach',\n",
              " 'approaching',\n",
              " 'appropriate',\n",
              " 'appropriation',\n",
              " 'approval',\n",
              " 'apropos',\n",
              " 'apt',\n",
              " 'aquarium',\n",
              " 'ar',\n",
              " 'ara',\n",
              " 'arcade',\n",
              " 'archetype',\n",
              " 'architect',\n",
              " 'architecture',\n",
              " 'are',\n",
              " 'area',\n",
              " 'areal',\n",
              " 'aren',\n",
              " 'arena',\n",
              " 'arent',\n",
              " 'argue',\n",
              " 'argument',\n",
              " 'ari',\n",
              " 'ariz',\n",
              " 'arm',\n",
              " 'armed',\n",
              " 'armory',\n",
              " 'army',\n",
              " 'around',\n",
              " 'arrest',\n",
              " 'arrival',\n",
              " 'arrive',\n",
              " 'arrogant',\n",
              " 'arse',\n",
              " 'arsenal',\n",
              " 'arson',\n",
              " 'arsonist',\n",
              " 'art',\n",
              " 'article',\n",
              " 'artificial',\n",
              " 'artillery',\n",
              " 'artist',\n",
              " 'ary',\n",
              " 'as',\n",
              " 'ascend',\n",
              " 'ash',\n",
              " 'aside',\n",
              " 'ask',\n",
              " 'asleep',\n",
              " 'aspect',\n",
              " 'asphalt',\n",
              " 'aspiring',\n",
              " 'ass',\n",
              " 'assailant',\n",
              " 'assassins',\n",
              " 'assault',\n",
              " 'assembly',\n",
              " 'assertative',\n",
              " 'assessment',\n",
              " 'asset',\n",
              " 'assistance',\n",
              " 'assistant',\n",
              " 'associated',\n",
              " 'association',\n",
              " 'assume',\n",
              " 'assured',\n",
              " 'astonishing',\n",
              " 'astounding',\n",
              " 'astrakhan',\n",
              " 'astrologian',\n",
              " 'astrology',\n",
              " 'asylum',\n",
              " 'at',\n",
              " 'ate',\n",
              " 'atheistic',\n",
              " 'athlete',\n",
              " 'athletics',\n",
              " 'atlantic',\n",
              " 'atlas',\n",
              " 'atmosphere',\n",
              " 'atmospheric',\n",
              " 'atom',\n",
              " 'atomic',\n",
              " 'attached',\n",
              " 'attack',\n",
              " 'attempt',\n",
              " 'attend',\n",
              " 'attendance',\n",
              " 'attention',\n",
              " 'attic',\n",
              " 'attitude',\n",
              " 'attraction',\n",
              " 'attractive',\n",
              " 'auburn',\n",
              " 'auction',\n",
              " 'audience',\n",
              " 'audio',\n",
              " 'audit',\n",
              " 'august',\n",
              " 'aunt',\n",
              " 'aurora',\n",
              " 'australia',\n",
              " 'authentic',\n",
              " 'author',\n",
              " 'authority',\n",
              " 'autism',\n",
              " 'autistic',\n",
              " 'auto',\n",
              " 'autobiography',\n",
              " 'automatic',\n",
              " 'autumn',\n",
              " 'ava',\n",
              " 'available',\n",
              " 'avalanche',\n",
              " 'ave',\n",
              " 'avenger',\n",
              " 'avenue',\n",
              " 'average',\n",
              " 'avert',\n",
              " 'averted',\n",
              " 'aviation',\n",
              " 'avoid',\n",
              " 'aw',\n",
              " 'await',\n",
              " 'awake',\n",
              " 'awakening',\n",
              " 'award',\n",
              " 'aware',\n",
              " 'awareness',\n",
              " 'awash',\n",
              " 'away',\n",
              " 'awesome',\n",
              " 'awful',\n",
              " 'awkward',\n",
              " 'awn',\n",
              " 'ay',\n",
              " 'ba',\n",
              " 'babe',\n",
              " 'baby',\n",
              " 'back',\n",
              " 'background',\n",
              " 'backing',\n",
              " 'backlash',\n",
              " 'backup',\n",
              " 'bacteria',\n",
              " 'bad',\n",
              " 'badge',\n",
              " 'badges',\n",
              " 'badly',\n",
              " 'badu',\n",
              " 'bae',\n",
              " 'baffle',\n",
              " 'baffling',\n",
              " 'bag',\n",
              " 'baggage',\n",
              " 'bagged',\n",
              " 'bagging',\n",
              " 'bago',\n",
              " 'bags',\n",
              " 'bah',\n",
              " 'bail',\n",
              " 'bait',\n",
              " 'bake',\n",
              " 'baked',\n",
              " 'baking',\n",
              " 'bal',\n",
              " 'balance',\n",
              " 'balcony',\n",
              " 'bali',\n",
              " 'ball',\n",
              " 'ban',\n",
              " 'banana',\n",
              " 'band',\n",
              " 'bang',\n",
              " 'bank',\n",
              " 'banking',\n",
              " 'bannister',\n",
              " 'banquet',\n",
              " 'bar',\n",
              " 'barbaric',\n",
              " 'barber',\n",
              " 'barcelona',\n",
              " 'bard',\n",
              " 'bardo',\n",
              " 'bare',\n",
              " 'barely',\n",
              " 'bargain',\n",
              " 'bark',\n",
              " 'barking',\n",
              " 'barn',\n",
              " 'barra',\n",
              " 'barrack',\n",
              " 'barrier',\n",
              " 'barring',\n",
              " 'barry',\n",
              " 'bartender',\n",
              " 'basalt',\n",
              " 'base',\n",
              " 'baseball',\n",
              " 'based',\n",
              " 'baseman',\n",
              " 'basement',\n",
              " 'bash',\n",
              " 'basic',\n",
              " 'basically',\n",
              " 'basis',\n",
              " 'bask',\n",
              " 'basket',\n",
              " 'bass',\n",
              " 'bastard',\n",
              " 'bat',\n",
              " 'bath',\n",
              " 'bathe',\n",
              " 'bathroom',\n",
              " 'batter',\n",
              " 'battered',\n",
              " 'battery',\n",
              " 'batting',\n",
              " 'battle',\n",
              " 'battlefield',\n",
              " 'battleship',\n",
              " 'bay',\n",
              " 'bayonet',\n",
              " 'be',\n",
              " 'beach',\n",
              " 'beached',\n",
              " 'beacon',\n",
              " 'beam',\n",
              " 'bean',\n",
              " 'bear',\n",
              " 'beard',\n",
              " 'beast',\n",
              " 'beastly',\n",
              " 'beat',\n",
              " 'beaten',\n",
              " 'beating',\n",
              " 'beautiful',\n",
              " 'beautifully',\n",
              " 'beauty',\n",
              " 'beckoning',\n",
              " 'become',\n",
              " 'becomes',\n",
              " 'becoming',\n",
              " 'bed',\n",
              " 'bedding',\n",
              " 'bedroom',\n",
              " 'bee',\n",
              " 'beef',\n",
              " 'beer',\n",
              " 'beet',\n",
              " 'beetroot',\n",
              " 'begging',\n",
              " 'begin',\n",
              " 'beginner',\n",
              " 'beginning',\n",
              " 'begun',\n",
              " 'behalf',\n",
              " 'behavior',\n",
              " 'behaviour',\n",
              " 'behead',\n",
              " 'behind',\n",
              " 'behold',\n",
              " 'beige',\n",
              " 'belie',\n",
              " 'belief',\n",
              " 'believe',\n",
              " 'believing',\n",
              " 'bell',\n",
              " 'belle',\n",
              " 'belligerent',\n",
              " 'belly',\n",
              " 'belt',\n",
              " 'belter',\n",
              " 'beluga',\n",
              " 'ben',\n",
              " 'bend',\n",
              " 'bene',\n",
              " 'beneath',\n",
              " 'benedict',\n",
              " 'benediction',\n",
              " 'benefit',\n",
              " 'benjamin',\n",
              " 'berlin',\n",
              " 'berry',\n",
              " 'beside',\n",
              " 'besides',\n",
              " 'besieged',\n",
              " 'best',\n",
              " 'bet',\n",
              " 'beth',\n",
              " 'better',\n",
              " 'betting',\n",
              " 'beware',\n",
              " 'beyond',\n",
              " 'bias',\n",
              " 'bicentennial',\n",
              " 'bicycle',\n",
              " 'bicyclist',\n",
              " 'bid',\n",
              " 'big',\n",
              " 'bigamist',\n",
              " 'bigger',\n",
              " 'biggest',\n",
              " 'bike',\n",
              " 'bilic',\n",
              " 'bilingual',\n",
              " 'bill',\n",
              " 'billboard',\n",
              " 'billing',\n",
              " 'billion',\n",
              " 'billionaire',\n",
              " 'billy',\n",
              " 'bin',\n",
              " 'bind',\n",
              " 'biological',\n",
              " 'bioter',\n",
              " 'bird',\n",
              " 'birth',\n",
              " 'birthday',\n",
              " 'bistro',\n",
              " 'bit',\n",
              " 'bitch',\n",
              " 'bite',\n",
              " 'biting',\n",
              " 'bitten',\n",
              " 'bitter',\n",
              " 'biz',\n",
              " 'black',\n",
              " 'blackberry',\n",
              " 'blacken',\n",
              " 'blackmail',\n",
              " 'blah',\n",
              " 'blake',\n",
              " 'blame',\n",
              " 'blamed',\n",
              " 'blaming',\n",
              " 'bland',\n",
              " 'blanket',\n",
              " 'blast',\n",
              " 'blasting',\n",
              " 'blaze',\n",
              " 'blazing',\n",
              " 'bleacher',\n",
              " 'bleed',\n",
              " 'bleeding',\n",
              " 'blending',\n",
              " 'bless',\n",
              " 'blessed',\n",
              " 'blessing',\n",
              " 'blight',\n",
              " 'blind',\n",
              " 'blink',\n",
              " 'blinked',\n",
              " 'blinker',\n",
              " 'bliss',\n",
              " 'blitz',\n",
              " 'blizzard',\n",
              " 'block',\n",
              " 'blockage',\n",
              " 'blocked',\n",
              " 'blocking',\n",
              " 'bloke',\n",
              " 'blonde',\n",
              " 'blood',\n",
              " 'bloody',\n",
              " 'bloom',\n",
              " 'blossom',\n",
              " 'blow',\n",
              " 'blower',\n",
              " 'blowing',\n",
              " 'blown',\n",
              " 'blowout',\n",
              " 'blubber',\n",
              " 'blue',\n",
              " 'bluebell',\n",
              " 'blueprint',\n",
              " 'bluff',\n",
              " 'blunt',\n",
              " 'bn',\n",
              " 'bo',\n",
              " 'board',\n",
              " 'boarding',\n",
              " 'boasting',\n",
              " 'boat',\n",
              " 'bob',\n",
              " 'bobble',\n",
              " 'bobcat',\n",
              " 'bod',\n",
              " 'bodied',\n",
              " 'body',\n",
              " 'bolster',\n",
              " 'bolt',\n",
              " 'bomb',\n",
              " 'bombed',\n",
              " 'bomber',\n",
              " 'bon',\n",
              " 'bone',\n",
              " 'bong',\n",
              " 'bonsai',\n",
              " 'bonus',\n",
              " 'boob',\n",
              " 'book',\n",
              " 'bookmobile',\n",
              " 'boom',\n",
              " 'boost',\n",
              " 'boot',\n",
              " 'booth',\n",
              " 'booty',\n",
              " 'booze',\n",
              " 'border',\n",
              " 'borderland',\n",
              " 'bore',\n",
              " 'boring',\n",
              " 'born',\n",
              " 'borrower',\n",
              " 'boston',\n",
              " 'bot',\n",
              " 'bother',\n",
              " 'bottle',\n",
              " 'bottling',\n",
              " 'bottom',\n",
              " 'bought',\n",
              " 'boulder',\n",
              " 'bounce',\n",
              " 'bound',\n",
              " 'boundary',\n",
              " 'bounty',\n",
              " 'bout',\n",
              " 'bovine',\n",
              " 'bow',\n",
              " 'bowe',\n",
              " 'bowery',\n",
              " 'bowknot',\n",
              " 'bowl',\n",
              " 'bowling',\n",
              " 'box',\n",
              " 'boxer',\n",
              " 'boxing',\n",
              " 'boy',\n",
              " 'brace',\n",
              " 'bracelet',\n",
              " 'bracing',\n",
              " 'bracket',\n",
              " 'brain',\n",
              " 'brainless',\n",
              " 'brake',\n",
              " 'branch',\n",
              " 'brand',\n",
              " 'brant',\n",
              " 'brass',\n",
              " 'brasswork',\n",
              " 'brave',\n",
              " 'bravery',\n",
              " 'brazil',\n",
              " 'breach',\n",
              " 'bread',\n",
              " 'break',\n",
              " 'breakdown',\n",
              " 'breakfast',\n",
              " 'breaking',\n",
              " 'breast',\n",
              " 'breath',\n",
              " 'breathe',\n",
              " 'breathing',\n",
              " 'bred',\n",
              " 'breed',\n",
              " 'breeder',\n",
              " 'brewer',\n",
              " 'brewing',\n",
              " 'brick',\n",
              " 'bridal',\n",
              " 'bride',\n",
              " 'bridge',\n",
              " 'bridgework',\n",
              " 'brief',\n",
              " 'briefing',\n",
              " 'brig',\n",
              " 'brigade',\n",
              " 'bright',\n",
              " 'brightening',\n",
              " 'brightly',\n",
              " 'brilliant',\n",
              " 'bring',\n",
              " 'brisk',\n",
              " 'brit',\n",
              " 'british',\n",
              " 'broad',\n",
              " 'broadcast',\n",
              " 'broadly',\n",
              " 'broadway',\n",
              " 'brochure',\n",
              " 'brock',\n",
              " 'broke',\n",
              " 'broken',\n",
              " 'brother',\n",
              " 'brought',\n",
              " 'brown',\n",
              " 'browner',\n",
              " 'brownie',\n",
              " 'browser',\n",
              " 'browsing',\n",
              " 'bruise',\n",
              " 'brunette',\n",
              " 'brunt',\n",
              " 'brush',\n",
              " 'brut',\n",
              " 'brutal',\n",
              " 'brutality',\n",
              " 'brutally',\n",
              " 'bu',\n",
              " 'bubble',\n",
              " 'buck',\n",
              " 'buckle',\n",
              " 'bud',\n",
              " 'buddy',\n",
              " 'budget',\n",
              " 'buff',\n",
              " 'buffalo',\n",
              " 'buffer',\n",
              " 'buffet',\n",
              " 'bug',\n",
              " 'build',\n",
              " 'builder',\n",
              " 'building',\n",
              " 'buildings',\n",
              " 'buildup',\n",
              " 'built',\n",
              " 'bull',\n",
              " 'bullet',\n",
              " 'bulletin',\n",
              " 'bulletproof',\n",
              " 'bullets',\n",
              " 'bully',\n",
              " 'bummer',\n",
              " 'bump',\n",
              " 'bumper',\n",
              " 'bunch',\n",
              " 'bundle',\n",
              " 'bundy',\n",
              " 'bunting',\n",
              " 'burglar',\n",
              " 'burglary',\n",
              " 'buried',\n",
              " 'burn',\n",
              " 'burned',\n",
              " 'burner',\n",
              " 'burning',\n",
              " 'burnt',\n",
              " 'burp',\n",
              " 'burst',\n",
              " 'burton',\n",
              " 'bury',\n",
              " 'bus',\n",
              " 'bush',\n",
              " 'business',\n",
              " 'businessman',\n",
              " 'bust',\n",
              " 'busted',\n",
              " 'busy',\n",
              " 'butch',\n",
              " 'butt',\n",
              " 'butter',\n",
              " 'button',\n",
              " 'buy',\n",
              " 'by',\n",
              " 'bye',\n",
              " 'bypass',\n",
              " 'ca',\n",
              " 'cab',\n",
              " 'cabbage',\n",
              " 'cabin',\n",
              " 'cable',\n",
              " 'cabling',\n",
              " 'cachet',\n",
              " 'cafì',\n",
              " 'cage',\n",
              " 'cain',\n",
              " 'cake',\n",
              " 'cal',\n",
              " 'calais',\n",
              " 'calamity',\n",
              " 'call',\n",
              " 'calling',\n",
              " 'calm',\n",
              " 'calmly',\n",
              " 'calorie',\n",
              " 'calumet',\n",
              " 'cam',\n",
              " 'came',\n",
              " 'cameo',\n",
              " 'camera',\n",
              " 'camilla',\n",
              " 'camp',\n",
              " 'campaign',\n",
              " 'camper',\n",
              " 'campfire',\n",
              " 'campground',\n",
              " 'campus',\n",
              " 'can',\n",
              " 'canada',\n",
              " 'canal',\n",
              " 'cancel',\n",
              " 'cancer',\n",
              " 'candidate',\n",
              " 'candle',\n",
              " 'candy',\n",
              " 'cannibalism',\n",
              " 'cannon',\n",
              " 'canoe',\n",
              " 'cant',\n",
              " 'cantar',\n",
              " 'canvas',\n",
              " 'canyon',\n",
              " 'cap',\n",
              " 'capable',\n",
              " 'capacity',\n",
              " 'cape',\n",
              " 'capital',\n",
              " 'capitalism',\n",
              " 'capitalist',\n",
              " 'capitulation',\n",
              " 'capsize',\n",
              " 'captain',\n",
              " 'caption',\n",
              " 'captive',\n",
              " 'captivity',\n",
              " 'captor',\n",
              " 'capture',\n",
              " 'car',\n",
              " 'caravan',\n",
              " 'carcinoma',\n",
              " 'card',\n",
              " 'cardboard',\n",
              " 'care',\n",
              " 'careen',\n",
              " 'career',\n",
              " 'careful',\n",
              " 'carefully',\n",
              " 'careless',\n",
              " 'carful',\n",
              " 'cargo',\n",
              " 'carl',\n",
              " 'carnage',\n",
              " 'caroline',\n",
              " 'carpet',\n",
              " 'carr',\n",
              " 'carriage',\n",
              " 'carried',\n",
              " 'carry',\n",
              " 'carrying',\n",
              " 'cart',\n",
              " 'cartel',\n",
              " 'cartoon',\n",
              " 'cartridge',\n",
              " 'cascade',\n",
              " 'case',\n",
              " 'cash',\n",
              " 'casing',\n",
              " 'casino',\n",
              " 'cast',\n",
              " 'casting',\n",
              " 'castle',\n",
              " 'casual',\n",
              " 'casually',\n",
              " 'casualties',\n",
              " 'casualty',\n",
              " 'cat',\n",
              " 'cataclysmic',\n",
              " 'catalogue',\n",
              " 'catastrophe',\n",
              " 'catastrophic',\n",
              " 'catch',\n",
              " 'catcher',\n",
              " 'catching',\n",
              " 'catechize',\n",
              " 'catfish',\n",
              " 'caught',\n",
              " 'causation',\n",
              " 'cause',\n",
              " 'causing',\n",
              " 'caution',\n",
              " 'cautious',\n",
              " 'cave',\n",
              " 'ce',\n",
              " 'cee',\n",
              " 'celebration',\n",
              " 'celebrity',\n",
              " 'celestial',\n",
              " 'cell',\n",
              " 'cement',\n",
              " 'censor',\n",
              " 'censorship',\n",
              " 'census',\n",
              " 'center',\n",
              " 'centipede',\n",
              " 'central',\n",
              " 'cereal',\n",
              " 'cern',\n",
              " 'cerography',\n",
              " 'certain',\n",
              " 'certainly',\n",
              " 'certainty',\n",
              " 'certificate',\n",
              " 'certified',\n",
              " 'cervix',\n",
              " 'cest',\n",
              " 'chain',\n",
              " 'chair',\n",
              " 'chairman',\n",
              " 'challenge',\n",
              " 'champ',\n",
              " 'champagne',\n",
              " 'champaign',\n",
              " 'champion',\n",
              " 'championship',\n",
              " 'chance',\n",
              " 'change',\n",
              " 'channel',\n",
              " 'chaos',\n",
              " 'chapter',\n",
              " 'character',\n",
              " 'charcoal',\n",
              " 'charge',\n",
              " 'charger',\n",
              " 'charging',\n",
              " 'charity',\n",
              " 'charming',\n",
              " ...]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xq_b_u0LxafM"
      },
      "source": [
        "Le seul **inconvénient du One-Hot Encoding** c'est que l'on **perd la hiérarchie**, l'ordre des mots.\n",
        " \n",
        "Cela nous fait donc **perdre** le contexte, **le sens de la phrase** et en théorie cela devrait appauvrir les résultats de notre modèle.\n",
        " \n",
        "En pratique, cela est bien différent, on peut avoir des **résultats avec 80-85% de précision** ce qui est déjà **très intéressant pour du NLP !**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cSSjaVLrAVUv"
      },
      "source": [
        "## **Word embeddings**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Np-iNAosEkwy"
      },
      "source": [
        "### **Encodage Hiérarchique**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mRDKp56x27gZ"
      },
      "source": [
        "Ici on utilise un **autre type** d'encodage: **l'encodage hiérarchique.**\n",
        " \n",
        "Contrairement au One-Hot Encoding, vous vous en doutez, **on garde la hiérarchie**, l'ordre des mots et donc **le sens de la phrase.**\n",
        " \n",
        "On a un **autre type de dictionnaire** ici. En fait, **chaque mot est représenté par un chiffre.**\n",
        " \n",
        "Chaque **phrase** sera donc **une suite de chiffres.**\n",
        " \n",
        "Un **exemple** sera plus parlant :\n",
        "- \"je joue au jeu vidéo\" sera [1, 2, 3, 4, 5]\n",
        "- \"je regarde une vidéo\" sera [1, 6, 7, 5]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H1UpGjsa54L_"
      },
      "source": [
        "Pour **cet encodage** on importe la **bibliothèque** *zeugma*. Si on est sur **Google Colab** on utilise la **commande suivante** :"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tle8AG7AAcqr"
      },
      "source": [
        "!pip install zeugma &> /dev/null"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fmx-YxEj6b0H"
      },
      "source": [
        "Sinon on exécute cette **même commande**, sans le \"!\" du début, dans le **terminal**.\n",
        " \n",
        "Ensuite, le fonctionnement est à peu près **le même que pour le one-hot encoding** : on utilise la fonction *TextsToSequences()*  pour **créer notre dictionnaire.**\n",
        " \n",
        "Puis on utilise la **fonction** *fit_transform()* sur **nos phrases preprocessées.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9bhKujH4_eYx",
        "outputId": "4ebbb24f-da2c-4cf8-d06e-c1a94126981d"
      },
      "source": [
        "from zeugma import TextsToSequences\n",
        " \n",
        "sequencer = TextsToSequences()\n",
        "embedded_sequ = sequencer.fit_transform(preprocess_list)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/zeugma/keras_transformers.py:33: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
            "  return np.array(self.texts_to_sequences(texts))\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dRImqSHp65aT"
      },
      "source": [
        "Un exemple de phrase avec **l'encodage hiérarchique :**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZaMBloUgAbGn",
        "outputId": "881bbfd8-f062-48ed-add3-86c15b057ff9"
      },
      "source": [
        "embedded_sequ[0]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[95, 1, 135, 434, 874]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cUt5Guw-6_qi"
      },
      "source": [
        "Une dernière chose à faire : **normaliser nos données.**\n",
        " \n",
        "Ehh oui, pour utiliser du **Machine Learning** il faut que nos données soit en **format tenseur.**\n",
        " \n",
        "Cela implique qu'il faut que **toutes les phrases encodées aient la même taille.**\n",
        " \n",
        "Pour que **nos phrases aient la même taille**, on a ici deux choix:\n",
        "- **ajouter du 'vide'** aux phrases les plus courtes\n",
        "- **tronquer les phrases** les plus longues\n",
        " \n",
        "Pour ces deux choix, **une même fonction existe :** *sequence.pad_sequences()*\n",
        " \n",
        "Elle possède **deux paramètres :**\n",
        "- **sentences** la liste de phrases à remplir/tronquer\n",
        "- **maxlen** la longueur finale que chaque phrase aura\n",
        " \n",
        "En fait cette fonction **tronque les phrases** ayant une **longueur supérieur** à maxlen et **remplit de 0 les phrases** ayant une **longueur inférieur** à maxlen."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C6ajqJ_SAzVI"
      },
      "source": [
        "from keras.preprocessing import sequence\n",
        "max_len = 40\n",
        " \n",
        "pad_sequ = sequence.pad_sequences(embedded_sequ, maxlen=max_len)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YALTtmwZZo1l"
      },
      "source": [
        "On peut **afficher** une phrase encodée pour voir le **résultat** :"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4aWCBrAQA4pS",
        "outputId": "3504b657-ca76-4f89-cf90-c0e7b18242ae"
      },
      "source": [
        "print(pad_sequ[0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
            "   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0  95\n",
            "   1 135 434 874]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QITOeQGREggD"
      },
      "source": [
        "### **Couche Embedding**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HB0lNL75ZwP9"
      },
      "source": [
        "En fait, l'**encodage hiérarchique** est un préalable pour utiliser la **couche** *Embedding* de **Keras**.\n",
        " \n",
        "Cette couche permet de **donner des coordonnées** à tous les **mots de notre dictionnaire** tout en réalisant un **apprentissage**.\n",
        " \n",
        "L'idée est que **plus les mots ont un sens proche**, **plus les mots ont des coordonnées proches.**\n",
        " \n",
        "**La couche** va donc **s'améliorer**, comme toutes les autres couches, **au cours de l'apprentissage.** Ainsi à la fin de l'apprentissage, la couche aura donné **des coordonnées précises** pour chacun des mots."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oPhf-1KAbl_C"
      },
      "source": [
        "La couche *Embedding* a **trois paramètres:**\n",
        "- **input_dim**, le nombre de mots dans notre dictionnaire + 1\n",
        "- **output_dim**, la dimension du tenseur de sortie\n",
        "- **input_length**, la longueur des vecteurs (longueur des phrases normalisé)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Mwtn5Z8zHKdW"
      },
      "source": [
        "longueur_dict = max(list(map(lambda x: max(x), pad_sequ)))+1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qPDePo4tA5cp",
        "outputId": "57e03471-5fd5-40aa-97fc-89abe6bdf7fa"
      },
      "source": [
        "from keras.models import Sequential\n",
        "from keras.layers import Embedding, Flatten, Dense\n",
        "\n",
        "model = Sequential()\n",
        "model.add(Embedding(longueur_dict, 8, input_length = max_len))\n",
        "model.add(Flatten())\n",
        "model.add(Dense(1, activation='sigmoid'))\n",
        "\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding (Embedding)        (None, 40, 8)             56520     \n",
            "_________________________________________________________________\n",
            "flatten (Flatten)            (None, 320)               0         \n",
            "_________________________________________________________________\n",
            "dense (Dense)                (None, 1)                 321       \n",
            "=================================================================\n",
            "Total params: 56,841\n",
            "Trainable params: 56,841\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4aEL_3V9xgT1"
      },
      "source": [
        "On pourra ensuite **lancer l'entraînement** avec la fonction *fit()* puis **utiliser notre modèle !**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xelXbIpqAK2r"
      },
      "source": [
        "# **Pour aller plus loin...**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8eHHiNUxHplL"
      },
      "source": [
        "## **Word2Vec Embedding**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_TTRbaWpELhk"
      },
      "source": [
        "### **Entraînement**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xvhZk133uHW_"
      },
      "source": [
        "Il existe **différentes librairies** pour faire de l'**embedding**.\n",
        " \n",
        "**Keras** est particulièrement utile pour cette tâche car elle permet d'**entraîner l'embedding en même temps que le modèle de Deep Learning**.\n",
        " \n",
        "La **librairie Gensim** est au moins aussi intéressante que Keras car elle nous permet de **visualiser cet embedding**.\n",
        " \n",
        "C'est-à-dire qu'on va pouvoir **analyser l'embedding** en regardant **quel mot est similaire à quel autre** par exemple."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LBYbVM4TGHf6"
      },
      "source": [
        "Pour cet **embedding**, il faut que **nos données** soient **sous forme de tokens** (chaque mot séparé) et non sous forme de phrases."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VpZy9Bq3OSrn"
      },
      "source": [
        "tokenize_sentences = []\n",
        "\n",
        "for i in range(len(preprocess_list)):\n",
        "  tokenize_sentences.append(nltk.tokenize.word_tokenize(preprocess_list[i]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "otLyXqmiNZ1a"
      },
      "source": [
        "Ensuite, on utilise **la fonction** *Word2Vec* de la **librairie** *Gensim*.\n",
        " \n",
        "Cette fonction possède **cinq paramètres principaux** :\n",
        "- **size :** La dimension du vecteur créé, idéalement inférieur au nombre de mots du vocabulaire\n",
        " \n",
        "- **fenêtre :** La distance maximale entre un mot cible et les mots autour du mot cible. La fenêtre par défaut est de 5.\n",
        " \n",
        "- **min_count :** Le nombre minimum de mots à prendre en compte lors de l'apprentissage du modèle ; les mots dont l'occurrence est inférieure à ce nombre seront ignorés. La valeur par défaut de min_count est 5.\n",
        " \n",
        "- **worker :** Le nombre de lots créés pour l'entraînement, par défaut il y en a 3.\n",
        " \n",
        "Premièrement on **initialise le Word2Vec**, puis on l'**entraîne sur nos données** !"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Qiajm5jE8NTk",
        "outputId": "4b8b069f-90c8-4848-8634-07e3693e0d88"
      },
      "source": [
        "from gensim.test.utils import common_texts\n",
        "from gensim.models import Word2Vec\n",
        "\n",
        "model_W2V = Word2Vec(sentences=tokenize_sentences, size=100, window=5, min_count=1, workers=4)\n",
        "model_W2V.train(tokenize_sentences, total_examples=len(tokenize_sentences), epochs=50)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(2479234, 2527700)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iIRE47TgEERQ"
      },
      "source": [
        "### **Visualisation**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gBl8Vh-4YPyy"
      },
      "source": [
        "Le **modèle a appris** la similarité des mots **en fonction du contexte de nos phrases**.\n",
        "\n",
        "Le **titre de notre jeu de données** est *'Disaster Tweet'* - *'Les tweets parlant de catastrophes(naturelles ou non)'*\n",
        " \n",
        " On peut par exemple **regarder quel mot se rapproche de 'fire'** grâce à la fonction *similar_by_word()*."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 37
        },
        "id": "vrvibRntYYX9",
        "outputId": "bd4bf7a3-1199-4c3c-a319-c209d65954ea"
      },
      "source": [
        "tokenize_sentences[0][1]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'fire'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O5R0_PaZ9J3H",
        "outputId": "2b820995-22b7-4dc5-9b24-f22e32ae0a39"
      },
      "source": [
        "model_W2V.similar_by_word(tokenize_sentences[0][1])[:5]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:1: DeprecationWarning: Call to deprecated `similar_by_word` (Method will be removed in 4.0.0, use self.wv.similar_by_word() instead).\n",
            "  \"\"\"Entry point for launching an IPython kernel.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('township', 0.7251554727554321),\n",
              " ('spreading', 0.7012003660202026),\n",
              " ('alarm', 0.6951850652694702),\n",
              " ('warden', 0.6813154220581055),\n",
              " ('acre', 0.6741729974746704)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qAjRvw8iZNX6"
      },
      "source": [
        "Les **cinq premiers** sont *'decomposition'*, *'township'*, *'racer'*, *'beast'* et *'apartment'*.\n",
        " \n",
        "Cela veut dire que **la plupart du temps**, *'fire'* à été **utilisé au côté de ces mots**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pif17_KUZ3me"
      },
      "source": [
        "Pour mieux **visualiser cette similarité** on peut utiliser **la fonction suivante** :"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Agg7pGfbeZv"
      },
      "source": [
        "from sklearn.manifold import TSNE\n",
        "from matplotlib import pyplot as plt\n",
        "\n",
        "def display_closestwords_tsnescatterplot_perso(model, word):\n",
        "    \n",
        "    arr = np.empty((0,100), dtype='f')\n",
        "    word_labels = [word]\n",
        "\n",
        "    numb_sim_words = 5\n",
        "\n",
        "    # get close words\n",
        "    close_words = model.similar_by_word(word)[:numb_sim_words]\n",
        "    \n",
        "    # add the vector for each of the closest words to the array\n",
        "    arr = np.append(arr, np.array([model[word]]), axis=0)\n",
        "    for wrd_score in close_words:\n",
        "        wrd_vector = model[wrd_score[0]]\n",
        "        word_labels.append(wrd_score[0])\n",
        "        arr = np.append(arr, np.array([wrd_vector]), axis=0)\n",
        "        \n",
        "    # find tsne coords for 2 dimensions\n",
        "    tsne = TSNE(n_components=2, random_state=0)\n",
        "    np.set_printoptions(suppress=True)\n",
        "    Y = tsne.fit_transform(arr)\n",
        "\n",
        "    x_coords = Y[:, 0]\n",
        "    y_coords = Y[:, 1]\n",
        "\n",
        "    # color for words\n",
        "    color = ['red']\n",
        "    for i in range(numb_sim_words):\n",
        "      color.append('blue')\n",
        "     \n",
        "    # display scatter plot\n",
        "    plt.scatter(x_coords, y_coords, c = color)\n",
        "\n",
        "    for label, x, y in zip(word_labels, x_coords, y_coords):\n",
        "        plt.annotate(label, xy=(x, y), xytext=(1, 5), textcoords='offset points')\n",
        "    plt.xlim(min(x_coords)-100, max(x_coords)+100)\n",
        "    plt.ylim(min(y_coords)-100, max(y_coords)+100)\n",
        "    plt.show()\n",
        "    print(\"Word most similar to : \"+word)\n",
        "    print([sim_word[0] for sim_word in close_words])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MNLbvGguZ_zb"
      },
      "source": [
        "Et ensuite l'utiliser **en précisant le modèle d'Embedding** et **le mot à analyser** :"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 472
        },
        "id": "zv9oBC5j-WSA",
        "outputId": "c5af0408-250f-47e9-9490-8baf54c6b17d"
      },
      "source": [
        "display_closestwords_tsnescatterplot_perso(model_W2V, tokenize_sentences[0][0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:12: DeprecationWarning: Call to deprecated `similar_by_word` (Method will be removed in 4.0.0, use self.wv.similar_by_word() instead).\n",
            "  if sys.path[0] == '':\n",
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:15: DeprecationWarning: Call to deprecated `__getitem__` (Method will be removed in 4.0.0, use self.wv.__getitem__() instead).\n",
            "  from ipykernel import kernelapp as app\n",
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:17: DeprecationWarning: Call to deprecated `__getitem__` (Method will be removed in 4.0.0, use self.wv.__getitem__() instead).\n",
            "/usr/local/lib/python3.7/dist-packages/matplotlib/backends/backend_agg.py:214: RuntimeWarning: Glyph 137 missing from current font.\n",
            "  font.set_text(s, 0.0, flags=flags)\n",
            "/usr/local/lib/python3.7/dist-packages/matplotlib/backends/backend_agg.py:183: RuntimeWarning: Glyph 137 missing from current font.\n",
            "  font.set_text(s, 0, flags=flags)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de3wU9b3/8dcnEO5oqyCi8DPRRiKQC7kJhgByMSAqiCAgqNGjVvB2tFI5p1Kjp57TIlZFEYSqaCsaRUUUL4AQgVYKSQwXMdiAAaEpBpBruAW+vz92s00kQCAkSzLv5+OxD3Znvjv7mTHue2fmO/M15xwiIuJdIcEuQEREgktBICLicQoCERGPUxCIiHicgkBExOPqB7uAymrRooULCwsLdhkiIrVGdnb2VudcyxO1qzVBEBYWRlZWVrDLEBGpNcxsQ2Xa6dCQiIjHKQhERDxOQSAi4nEKAhERj1MQiIh4nIJARMTjFAQiIh6nIBAR8TgFgYiIxykIREQ8TkEgIuJxCgIREY9TEIiIeJyCQETE4xQEIiIepyAQEfE4BYGIiMcpCEREPE5BICLicQoCERGPUxCIiHicgkBExOMUBCIiHqcgEBHxOAWBiIjHKQhERDxOQSAi4nEKAhERj1MQiIh4nIJARMTjFAQiIh6nIBAR8bj61f0BZlYA7AYOAyXOuQQzOwfIAMKAAuBG59yP1V2LiIgcrab2CK50zsU65xL8r8cCnzvnIoDP/a9FRCQIgnVoaADwmv/5a8DAINUhIuJ5NREEDphrZtlmdpd/WivnXKH/+b+AVhW90czuMrMsM8sqKiqqgVJFRLyn2s8RAF2dc5vN7DxgnpnllZ3pnHNm5ip6o3NuKjAVICEhocI2IiJSNdW+R+Cc2+z/9wfgfSAJ2GJmrQH8//5Q3XWIiEjFqjUIzKypmTUvfQ5cBawGZgO3+pvdCnxQnXWIiMixVfehoVbA+2ZW+lkznHOfmtly4G0z+w9gA3BjNdchIiLHUK1B4JxbD8RUMH0b0Ks6P1tERCpHVxaLiHicgkBExOMUBCIiHqcgEBHxOAWBiIjHKQhERDxOQSAi4nEKAhERj1MQiIh4nIJARMTjFAQiIh6nIBAR8TgFgYiIxykIREQ8TkEgIuJxCgIREY9TEIiIeJyCQETE4xQEIiIepyAQEfE4BYGIiMcpCEREPE5BICLicQoCERGPUxCIiHicgkBExOMUBCIiHqcgEBHxOAWBiIjHKQhERDxOQSAiVfLnP/+ZdevWBbsMqQIFgcgZ5o477mDNmjXBLqNSVqxYwbx58/jtb38b7FKkCuoHuwARKe9Pf/pTsEsop3PnzrRo0SLw+rvvvmPOnDmEhYWxceNGJk+ezNKlSykqKqJly5ZBrFROlfYIRIJo79699O/fn5iYGDp27EhGRgY9evQgKysLgGbNmjFmzBg6dOhA7969WbZsGT169ODiiy9m9uzZAEyfPp0BAwbQo0cPIiIiePzxx4+5bIDPP/+cTp06ERUVxe23386BAwcACAsL47HHHiMuLo6oqCjy8vIAXxB89NFHgcedd94JQHp6OmvXrqVp06b06tWLK6+8koKCgprcfHKaBC0IzKyvma01s3wzGxusOkSC6dNPP+WCCy5gxYoVrF69mr59+5abv3fvXnr27MnXX39N8+bNefTRR5k3bx7vv/9+ucMxy5Yt491332XlypW88847ZGVlVbjs/fv3k5aWRkZGBqtWraKkpITJkycHltOiRQtycnIYNWoUEyZMqLHtIMEVlCAws3rAJKAf0B4Ybmbtg1GLSDBFRUUxb948HnnkERYvXszZZ59dbn6DBg0C4RAVFUX37t0JDQ0lKiqq3K/vPn36cO6559K4cWMGDRrEkiVLKlz22rVrCQ8P59JLLwXg1ltvZdGiRYHlDBo0CID4+Hj9uveQYO0RJAH5zrn1zrmDwFvAgCDVIhI0l156KTk5OURFRfHoo4/yxBNPlJsfGhqKmQEQEhJCw4YNA89LSkoC7UrblH19omVXpHT59erVK7f8itSvX58jR44EXu/fv/+Ey5czU7CC4ELg+zKvN/mnlWNmd5lZlpllFRUV1VhxImVlZmZyzTXXVMuy//nPf9KkSRNGjhzJmDFjyMnJOaXlzJs3j+3bt7Nv3z5mzZpFcnJyhctu164dBQUF5OfnA76un927dz+lzwwLCwvUm5OTw3fffXdKy5HgO6N7DTnnpgJTARISElyQyxGPOHz4MPXq1auRz1q1ahVjxowhJCSE0NBQJk+ezMMPP3zSy0lKSuKGG25g06ZNjBw5koSEBD777LOjlt2oUSNeffVVhgwZQklJCYmJidx9992nVPsNN9zA66+/TocOHbj88ssDh5uk9glWEGwG2pZ53cY/TaRKnnrqKRo2bMj999/Pgw8+yIoVK1iwYAELFizg5Zdf5qyzzmL58uXs27ePwYMHB3rYhIWFMXToUObNm8evf/1rfvazn/Gf//mfNGnShK5duwaWv3fvXu677z5Wr17NoUOHSE9PZ8CAAUyfPp3Zs2dTXFzMunXruP766xk/fvxR9a1YAdOnw+7dMGgQ9O2bysqVqeXaZGZmBp7v2bMn8Dw9Pb1cu7Lz2rRpw6xZs8rNT01NJTW1/LIBevXqxVdffXXU9LLnBBISEsrVUZHGjRszd+7c47aR2iFYQbAciDCzcHwBMAy4KUi1SB2SkpLC008/zf33309WVhYHDhzg0KFDLF68mG7dujFkyBDOOeccDh8+TK9evVi5ciXR0dEAnHvuueTk5LB//34iIiJYsGABv/jFLxg6dGhg+U8++SQ9e/bklVdeYceOHSQlJdG7d28AcnNz+eqrr2jYsCHt2rXjvvvuo23bf//eeeEFeOQR2L8fjhyBt96Cq66CmTMh5AzuyL1x40YGDhwYeL1z505uvPHGIFYkp1tQgsA5V2Jm9wKfAfWAV5xzXwejFqlb4uPjyc7OZteuXTRs2JC4uDiysrJYvHgxEydO5O2332bq1KmUlJRQWFjImjVrAkFQ+oWfl5dHeHg4ERERAIwcOZKpU6cCMHfuXGbPnh3oWrl//342btwI+H5pl/b6ad++PRs2bAgEQVERjBnjC4FSe/fC3LnwySfQv/+pr3NaWhppaWmnvoATeO+996pt2XJmCNo5Aufcx8DHwfp8qZtCQ0MJDw9n+vTpXHHFFURHR7Nw4ULy8/Np3LgxEyZMYPny5fz85z8nLS2tXE+Xpk2bnnD5zjneffdd2rVrV2763//+90CPGzi61838+RAaWj4IwBcGM2dWLQhEquoM3iEVOTUpKSlMmDCBbt26kZKSwpQpU+jUqRO7du2iadOmnH322WzZsoVPPvmkwvdHRkZSUFAQuJHam2++GZiXmprK888/j3O+vgsVHWuvSOPGFU8PCYFK5I9ItVIQSJ2TkpJCYWEhXbp0oVWrVjRq1IiUlBRiYmLo1KkTkZGR3HTTTSQnJ1f4/kaNGjF16lT69+9PXFwc5513XmDeuHHjOHToENHR0XTo0IFx48ZVqqbUVPhJV3//Z8Ftt53SaoqcNlb6y+ZMl5CQ4ErvvyJSasMGePZZyMqCmBh46CG4+OJgV1WxL76Aa6/1PXcOSkrgd7+DX/0quHVJ3WVm2c65hBO2UxBIbbVqFSQn+467HzoE9ev7fmEvXAgJJ/zTD47iYvj0U9+5gT594Pzzg12R1GWVDYIz+oIykeN54AFff/xSJSWwZw+MHg3LlgWvruNp0sR3/YDImUTnCKTWWrKk4ulZWb5++iJSOQoCqbWaN694euPGFZ+YFZGKKQik1rr77qO7ZTZuDHfcoSAQORkKAqm10tPhuut8J4jPPtv3b2oq/OEPwa5MpHbRyWKptUJDfffr+f57yMuDiAgICwt2VSK1j4JAar22bX0PL8vMzKRBgwZcccUVJ9VuypQpNGnShFtuuaUmypQzlIJApJrVxPgGmZmZNGvWrFJBULbdqY5FIHWLzhGIVEFBQQGRkZGMGDGCyy67jMGDB1NcXExYWBiPPPIIcXFxvPPOO8ydO5cuXboQFxfHkCFD2LNnD59++ilDhgwJLKvsSGgVtQffuAmPPfYYcXFxREVFkZeXR0FBAVOmTOGZZ54hNjaWxYsX8+GHH3L55ZfTqVMnevfuzZYtWypsl56ezoQJE8jLyyMpKancekVFRQGQnZ1N9+7diY+PJzU1lcLCwhrcwlITFAQiVbR27VpGjx7NN998w1lnncWLL74I/Ht8g969e/O73/2O+fPnk5OTQ0JCAn/84x/p3bs3f//739m7dy8AGRkZDBs2jK1bt1bYvlSLFi3Iyclh1KhRTJgwgbCwMO6++24efPBBcnNzSUlJoWvXrixdupSvvvqKYcOGMXDgQGbOnHlUu1KRkZEcPHgwMNxkRkYGQ4cO5dChQ9x3333MnDmT7Oxsbr/9dn7zm9/U4NaFq6++mh07drBjx47AtgXfMJ+DBw+u0VrqKh0aEqmitm3bBm5gN3LkSCZOnAj8e3yDpUuXsmbNmkCbgwcP0qVLF+rXr0/fvn358MMPGTx4MHPmzGH8+PF88cUXFbYvNch/aXJ8fPwxxwrYtGkTQ4cOpbCwkIMHD3L48OETrseNN95IRkYGY8eOJSMjg4yMDNauXcvq1avp06cP4DvM1bp161PZTKfs4499d6svKCjgxRdfZPTo0QBccMEFzJw5s0Zrqau0RyBSRfaTixZKX5eOb+Cco0+fPuTm5pKbm8uaNWt4+eWXARg2bBhvv/02CxYsICEhgebNmx+3PRAY9+CnYx6Udd9993Heeedx4MABGjRowK5duwDYvn0706ZNIz4+npSUFLZu3QrAli1byMzM5PHHHycyMpLi4mIiIiJ47bXXKCkpoaSkhLS0NFatWsXUqVOJjIwkLS2NSy+9lBEjRjB//nySk5OJiIhgmf/+Hunp6dx888106dKFiIgIpk2bFtgeY8aMoWPHjkRFRZGRkQFAYWEh3bp1IzY2lo4dO7J48WLAdzhs69atjB07lnXr1hEbG8uYMWMoKCigY8eOgG+AoNtuu42oqCg6derEwoULAZg+fTqDBg2ib9++RERE8Otf//qU/zvXZQoCkSrauHEjX375JQAzZswoN8YxQOfOnfnrX/9Kfn4+4Bv3+NtvvwWge/fu5OTkMG3aNIYNG3bC9sfSvHlzdpe58VJhYSFLly4lNzeXmJiYwLxPPvmEPn36kJ2dzYQJE5gzZw4A999/P1dffTUdO3YkISGBm266iezsbD7++GNat27Nc889x7Rp01i2bBnffvst+fn5/OpXvyIvL4+8vDxmzJjBkiVLmDBhAv/7v/8bqGPlypUsWLCAL7/8kieeeIJ//vOfvPfee+Tm5rJixQrmz5/PmDFjKCwsZMaMGaSmpgbmxcbGllvH3//+91xyySXk5uby1FNPlZs3adIkzIxVq1bx5ptvcuuttwYGHcrNzSUjI4NVq1aRkZHB999/f6L/pJ6jIBCponbt2jFp0iQuu+wyfvzxR0aNGlVufsuWLZk+fTrDhw8nOjqaLl26kJeXB/h+1V9zzTV88skngRPFx2t/LNdeey3vv/9+4CRw165d2bZtGykpKbRu3ZoWLVoEhtV8+eWXady4MSNGjAgExIIFCxg1ahRDhw7ljTfe4JZbbmHJkiXccMMNvPvuuzz++ONs3bqVAQMGkJ2dTXh4OFFRUYSEhNChQwd69eqFmREVFUVBQUGgrgEDBtC4cWNatGjBlVdeybJly1iyZAnDhw+nXr16tGrViu7du7N8+XISExN59dVXSU9PZ9WqVTQ/1j1EKrBkyRJGjhwJ+M53XHTRRYHwLB1CtFGjRoEhRKU8nSMQOUnbt/suYLvoIt/r+vXr85e//KVcm7JfhgA9e/Zk+fLlFS7vhRde4IUXXqhU+7LLTUhIIDMzE4BLL72UlStXBuZlZ2dz4YUX8sQTTwC+Y/tHjhzhnHPOqbDXzx/8l2M//PDDPPzww+XmxcbGsmjRIsaNG0fLli257rrreOONNwLzQ0JCAoerQkJCyh2uOtZhs4p069aNRYsWMWfOHNLS0njooYdOy/UNxxtCVHy0RyBSSc75Br654AK4+mr4xS/grrt808803bp1Y9asWezbt4/du3fz4Ycf0qRJE8LDw3nnnXcA37H6FStWAL5fzZMnTwZ8obFz505SUlKYNWsWxcXF7N27l/fff79cT6PK+OCDD9i/fz/btm0jMzOTxMREUlJSyMjI4PDhwxQVFbFo0SKSkpLYsGEDrVq14s477+SOO+4gJyen3LJ+evirrJSUlEA4ffvtt2zcuPGocaXl2BQEIpU0aRK89BIcOAA7d/oGxFm8OIzOnVcHu7SjxMXFMXToUGJiYujXrx+JiYkAvPHGG7z88svExMTQoUMHPvjgAwCee+45Fi5cSFRUFPHx8axZs4a4uDjS0tJISkri8ssv54477qBTp04nVUd0dDRXXnklnTt3Zty4cVxwwQVcf/31REdHExMTQ8+ePRk/fjznn38+mZmZgeFEMzIyeOCBB8ot69xzzyU5OZmOHTsyZsyYcvNGjx7NkSNHiIqKYujQoUyfPr3cnoAcn0YoE6mkiy8Gfzf7cho1gh07IJjfOwUFMH06/OtfvhvvXXutb8S2YEpPT6dZs2ZHHWqSmlPZEcq0RyC12sSJE7nssssYMWLEaV92QUEBM2bMCLzevr3idocP+4agDJY5c6BDB/i///PtsdxyC3Tv7ttzEakM7RFIrRYZGcn8+fNp06bNCduWlJRQ/yR+JmdmZjJhwgQ++ugjAAYMgA8/PPqcQHg4rFsXnDEQDh2C887z7ZGU1aQJPPWUb9hO8S7tEUidd/fdd7N+/Xr69evH008/zcCBA4mOjqZz586BHjSlFzUlJydz8803U1RUxA033EBiYiKJiYn89a9/BeCLL74gNjaW2NhYOnXqxO7duxk7diyLFy8mNjaWZ555hj/8wTcqWmio7/NDQnxfuFOmBG8gnOxs3x7JTxUXQ5mOPSLH55yrFY/4+Hgn8lMXXXSRKyoqcvfee69LT093zjn3+eefu5iYGOecc4899piLi4tzxcXFzjnnhg8f7hYvXuycc27Dhg0uMjLSOefcNddc45YsWeKcc2737t3u0KFDbuHCha5///7lPq+gwLnRo53r1Mm54cOd++qrGlnNY8rJca5ZM+d8+ynlH716Bbc2CT4gy1Xi+1XXEUidsGTJEt59913A1wd/27ZtgdsqXHfddTT2j2k5f/581qxZE3jfrl272LNnD8nJyTz00EOMGDGCQYMGHfNQ00UX+XoPnSliY+Gcc8B/c9KApk3hl78MTk1S+ygIpM4rvecPwJEjR1i6dCmNGjUq12bs2LH079+fjz/+mOTkZD777LOaLvOUmPnOW/TsCQcP+g4THTkCI0aAbswplaVzBFInlL2gKDMzkxYtWnDWWWcd1e6qq67i+eefD7zOzc0FYN26dURFRfHII4+QmJhIXl7ecS9gOpNER8PmzfD66/Dcc7Biha/3ULDOW0jtoz0CqRPS09O5/fbbiY6OpkmTJrz22msVtps4cSL33HMP0dHRlJSU0K1bN6ZMmcKzzz7LwoULA/fO6devHyEhIdSrV4+YmBjS0tJ48MEHa3itKq9hQxg4MNhVSG2l7qNS++zbBw0aQDUP/yhS26n7qNQ98+ZBRISvD+dZZ/lu/HPoULCrEqn1dGhIaoesLN+xj9JLeIuLfR34f/wRXn01uLWJ1HLVtkdgZulmttnMcv2Pq8vM+y8zyzeztWaWWl01SB3y5JO+Q0Jl7dsHb70F27YFpyaROqK6Dw0945yL9T8+BjCz9sAwoAPQF3jRzHSwV47vm28qvt9zgwawcWPN11NJZYdTrMoyyt7z6GRcccUVVfps8YZgnCMYALzlnDvgnPsOyAeSglCH1CYJCRWfHD54EC65pObrqUGnEgSlg6/87W9/q46SpI6p7iC418xWmtkrZvZz/7QLgbKDhm7yTzuKmd1lZllmllVUVFTNpcoZ7dFHffd7LqtJE7j3Xt+J4zNYSUkJI0aM4LLLLmPw4MEUFxfzxBNPkJiYSMeOHbnrrrso7b2Xn59P7969iYmJIS4ujnXr1h11z6PDhw8zZswYEhMTiY6O5qWXXgJ810+kpKRw3XXX0b59ewCaNWsGwJ49e+jVqxdxcXFERUUFxiEQAap2ryFgPrC6gscAoBVQD1/YPAm84n/PC8DIMst4GRh8os/SvYbEZWc716OHc02aONemjXPPPuvckSPBruq4vvvuOwcE7mN02223uaeeespt27Yt0GbkyJFu9uzZzjnnkpKS3Hvvveecc27fvn1u7969R93z6KWXXnL/8z//45xzbv/+/S4+Pt6tX7/eLVy40DVp0sStX78+0LZp06bOOecOHTrkdu7c6ZxzrqioyF1yySXuyBm+7aTqqIl7DTnnelemnZlNAz7yv9wMtC0zu41/msjxxcXBwoXBruKktW3bluTkZABGjhzJxIkTCQ8PZ/z48RQXF7N9+3Y6dOhAjx492Lx5M9dffz3AUbfBKDV37lxWrlzJzJkzAdi5cyf/+Mc/aNCgAUlJSYSHhx/1Hucc//3f/82iRYsICQlh8+bNbNmyhfPPP7+a1lpqk2rrPmpmrZ1zpaNkX49vTwFgNjDDzP4IXABEAMuqqw6RYKtoAPfRo0eTlZVF27ZtSU9PZ//+/ZVennOO559/ntTU8h3uMjMzy91Xqaw33niDoqIisrOzCQ0NJSws7KQ+U+q26jxHMN7MVpnZSuBK4EEA59zXwNvAGuBT4B7nXAV3VBepGzZu3MiXX34JwIwZM+jatSsALVq0YM+ePYFf9s2bN6dNmzbMmjULgAMHDlBcXHzUPY9SU1OZPHkyh/wX03377bfs3bv3uDXs3LmT8847j9DQUBYuXMiGDRtO+3pK7VVtewTOuZuPM+9JfOcNROq8du3aMWnSJG6//Xbat2/PqFGj+PHHH+nYsSPnn39+YGB5gD//+c/88pe/5Le//S2hoaG88847REdHl7vn0QMPPEBBQQFxcXE452jZsmUgPI5lxIgRXHvttURFRZGQkEBkZGR1r7bUIrrXkMhptnEjFBZC+/a+u2GIBIvuNSRSw3bsgD59oF07uOoqaNXKN6C8yJlOQSBymowcCYsWwf79sGuX7w4YTz4J770X7MpEjk9BIHIaFBXB/Pm+C53L2rsXxo8PTk0ilaUgEDkNtm+H0NCK5/3wQ83WInKyFAQip8HFF1ccBPXr+84biJzJFAQip0FoKEyc6Lv9UakGDeDss2HcuODVJVIZCgKR02TkSJg7FwYMgJgY3/3wVq2CNm2CXZnI8WmEMpHTKDnZ9xCpTbRHICLicQoCERGPUxCIiHicgkBExOMUBCIiHqcgEBHxOAWBiIjHKQhERDxOQSAi4nEKAhERj1MQiIh4nIJARMTjFAQiIh6nIBAR8TgFgYiIxykIREQ8TkEgIuJxCgIREY9TEIiIeJyCQETE4xQEIiIepyAQEfE4BYGIiMcpCEREPE5BICLicVUKAjMbYmZfm9kRM0v4ybz/MrN8M1trZqllpvf1T8s3s7FV+XwREam6qu4RrAYGAYvKTjSz9sAwoAPQF3jRzOqZWT1gEtAPaA8M97cVEZEgqV+VNzvnvgEws5/OGgC85Zw7AHxnZvlAkn9evnNuvf99b/nbrqlKHSIicuqq6xzBhcD3ZV5v8k871vQKmdldZpZlZllFRUXVUqiIiNedcI/AzOYD51cw6zfOuQ9Of0n/5pybCkwFSEhIcNX5WSIiXnXCIHDO9T6F5W4G2pZ53cY/jeNMFxGRIKiuQ0OzgWFm1tDMwoEIYBmwHIgws3Aza4DvhPLsaqpBREQqoUoni83seuB5oCUwx8xynXOpzrmvzextfCeBS4B7nHOH/e+5F/gMqAe84pz7ukprICIiVWLO1Y5D7wkJCS4rKyvYZYiI1Bpmlu2cSzhRO11ZLCLicQoCERGPUxCIiHicgkBExOMUBCIiHqcgEBHxOAWBiIjHKQhERDxOQSAi4nEKAhERj1MQiIh4nIJARMTjFAQiIh6nIBAR8TgFgYiIxykIREQ8TkEgIuJxCgIREY9TEIiIeJyCQETE4xQEIiIepyAQEfE4BYGIiMcpCEREPE5BICLicQoCERGPUxCIiHicgkBExOMUBCIiHqcgEBHxOAWBiIjHKQhERDxOQSAi4nEKAhERj6tSEJjZEDP72syOmFlCmelhZrbPzHL9jyll5sWb2SozyzeziWZmValBRESqpqp7BKuBQcCiCuatc87F+h93l5k+GbgTiPA/+laxBhERqYIqBYFz7hvn3NrKtjez1sBZzrmlzjkHvA4MrEoNIiJSNdV5jiDczL4ysy/MLMU/7UJgU5k2m/zTKmRmd5lZlpllFRUVVWOpIiLeVf9EDcxsPnB+BbN+45z74BhvKwT+n3Num5nFA7PMrMPJFuecmwpMBUhISHAn+34RETmxEwaBc673yS7UOXcAOOB/nm1m64BLgc1AmzJN2/iniYhIkFTLoSEza2lm9fzPL8Z3Uni9c64Q2GVmnf29hW4BjrVXISIiNaCq3UevN7NNQBdgjpl95p/VDVhpZrnATOBu59x2/7zRwJ+AfGAd8ElVahARkaoxX+edM19CQoLLysoKdhkiIrWGmWU75xJO1E5XFouIeJyCQETE4xQEIiIepyAQEfG4WnOy2MyKgA3BriOIWgBbg13EGUDbwUfbwUfbwedY2+Ei51zLE7251gSB15lZVmXO/td12g4+2g4+2g4+Vd0OOjQkIuJxCgIREY9TENQeU4NdwBlC28FH28FH28GnSttB5whERDxOewQiIh6nIBAR8TgFwRnMzNLNbLOZ5fofV5eZ919mlm9ma80sNZh11gQz6+tf13wzGxvsemqSmRWY2Sr/30CWf9o5ZjbPzP7h//fnwa6zOpjZK2b2g5mtLjOtwnU3n4n+v5GVZhYXvMpPr2Nsh9P2/aAgOPM945yL9T8+BjCz9sAwoAPQF3ixdPyHusi/bpOAfkB7YLh/G3jJlf6/gdK+4mOBz4WD8n0AAAI7SURBVJ1zEcDn/td10XR8f+NlHWvd++Eb+yQCuAuYXEM11oTpHL0d4DR9PygIaqcBwFvOuQPOue/wje2QFOSaqlMSkO+cW++cOwi8hW8beNkA4DX/89eAgUGspdo45xYB238y+VjrPgB43fksBX5mZq1rptLqdYztcCwn/f2gIDjz3evfzX2lzO7/hcD3Zdps8k+rq7y2vj/lgLlmlm1md/mntfKP+AfwL6BVcEoLimOtuxf/Tk7L94OCIMjMbL6Zra7gMQDfru0lQCxQCDwd1GIlWLo65+LwHfq4x8y6lZ3pfH3APdkP3Mvrzmn8fjjh4PVSvZxzvSvTzsymAR/5X24G2paZ3cY/ra7y2vqW45zb7P/3BzN7H99u/hYza+2cK/Qf/vghqEXWrGOtu6f+TpxzW0qfV/X7QXsEZ7CfHN+8HijtMTAbGGZmDc0sHN/JsWU1XV8NWg5EmFm4mTXAdyJsdpBrqhFm1tTMmpc+B67C93cwG7jV3+xW4IPgVBgUx1r32cAt/t5DnYGdZQ4h1Tmn8/tBewRntvFmFotv17cA+CWAc+5rM3sbWAOUAPc45w4Hrcpq5pwrMbN7gc+AesArzrmvg1xWTWkFvG9m4Pv/dYZz7lMzWw68bWb/ge/27DcGscZqY2ZvAj2AFma2CXgM+D0Vr/vHwNX4To4WA7fVeMHV5Bjbocfp+n7QLSZERDxOh4ZERDxOQSAi4nEKAhERj1MQiIh4nIJARMTjFAQiIh6nIBAR8bj/D8ALKpn7nWijAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Word most similar to : forest\n",
            "['decomposition', 'bacteria', 'warden', 'preventative', 'simpson\\x89û']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KH7SbIAdFbdg"
      },
      "source": [
        "**L'inconvénient de Word2vec** c'est qu'il apprend le sens d'un mot **uniquement en fonction des mots qui l'entourent**, là où **Keras**  apprend **le sens des mots en fonction de l'objectif** (y_train) fixé lors de l'apprentissage.\n",
        " \n",
        "En fait, *Word2Vec* a une **approche non-supervisée** et *Keras* une **approche supervisée**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B2wxDaC_-LWH"
      },
      "source": [
        "## **Word2Vec Pré-entraîné par Google**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cvcAVT_TaoHj"
      },
      "source": [
        "**Une autre approche** consiste à prendre un **Word2Vec déjà entraîné**.\n",
        "\n",
        "**Google** et **Facebook** propose le leur qui, vous l'imaginait bien, a été **entraîné sur des millions (milliards ?) de données !**\n",
        "\n",
        "Idéal pour avoir une **représentation générale du vocabulaire d'une langue.**\n",
        "\n",
        "**À savoir cependant**, les Word2Vec entraînés par Facebook, Google, ou autre **ne peuvent pas être adaptés**, **entraînés sur nos phrases**. On peut seulement **les utiliser en gardant le contexte général** sur lequel ils ont été entraîné."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TPtu27Wt-GcF"
      },
      "source": [
        "On télécharge ici le **Word2Vec entraîné par Google** (1.5GB) (disponible sur [ce lien](https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit) si vous voulez le télécharger en local) :"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NBaOPjTFwDYv",
        "outputId": "04d1ffc9-b6c5-42ed-8a52-0192954d02c2"
      },
      "source": [
        "!wget -q --show-progress --load-cookies /tmp/cookies.txt \"https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=0B7XkCwpI5KDYNlNUTTlSS21pQmM' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=0B7XkCwpI5KDYNlNUTTlSS21pQmM\" -O word2vec_pretrained.bin.gz && rm -rf /tmp/cookies.txt\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "word2vec_pretrained     [               <=>  ]   1.53G   112MB/s    in 13s     \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g3LAYCSPcCjH"
      },
      "source": [
        "Ensuite on utilise la **bibliothèque** *sh* pour le **dézipper**."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ptm-yEec2daT"
      },
      "source": [
        "!pip install sh &> /dev/null"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o2IyTJ0Jct7A"
      },
      "source": [
        "On le **dézip :**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Y2_e1xru2ZRF",
        "outputId": "45822042-d0c8-4a0c-89d2-5ce2c77b7652"
      },
      "source": [
        "from sh import gunzip\n",
        "\n",
        "gunzip('/content/word2vec_pretrained.bin.gz')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              ""
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "erVNHKMsc1Ez"
      },
      "source": [
        "Puis on utilise la **fonction** *load_word2vec_format()* de la class *KeyedVectors* pour **charger le Word2vec de Google.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6_sIv9XnxIc_"
      },
      "source": [
        "from gensim.models import KeyedVectors\n",
        "\n",
        "model = KeyedVectors.load_word2vec_format('/content/word2vec_pretrained.bin', binary=True)  "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PnVFSjYndPMY"
      },
      "source": [
        "On peut ensuite **l'utiliser** avec un **modèle de Deep Learning**..\n",
        " \n",
        "Je vous propose ici d'**explorer la représentation du modèle**, les liens qu'il fait entre les mots ...\n",
        " \n",
        "On peut par exemple **regarder les mots similaires** à *'koala'* on obtient *'koalas'*, *'wombat'*, *'quoll'*, *'orang_utan'*, *'Koala'*."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7kK6X8oxdHpW",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "83c1b4ff-f64b-45e6-be23-db09e0ff1787"
      },
      "source": [
        "model.similar_by_word('koala')[:5]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('koalas', 0.7420616745948792),\n",
              " ('wombat', 0.6323026418685913),\n",
              " ('quoll', 0.6266179084777832),\n",
              " ('orang_utan', 0.6036992073059082),\n",
              " ('Koala', 0.5906292200088501)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RJvi43bidpmH"
      },
      "source": [
        "Ou **comparer la similarité** entre **deux mots** :"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "poXQ1IdJeTco",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "74d3f5e5-7b06-4be2-b3c9-c0aa26b6f9de"
      },
      "source": [
        "model.similarity('hotdog', 'hamburger')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.65520716"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CS7n5bUDefiQ"
      },
      "source": [
        "Ou encore utiliser **notre fonction** de tout à l'heure pour une **visualisation plus concrète**."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-QRzgmSr-nML"
      },
      "source": [
        "from sklearn.manifold import TSNE\n",
        "from matplotlib import pyplot as plt\n",
        "import numpy as np\n",
        "\n",
        "def display_closestwords_tsnescatterplot(model, word):\n",
        "    \n",
        "    arr = np.empty((0,300), dtype='f')\n",
        "    word_labels = [word]\n",
        "\n",
        "    numb_sim_words = 5\n",
        "\n",
        "    # get close words\n",
        "    close_words = model.similar_by_word(word)[:numb_sim_words]\n",
        "    \n",
        "    # add the vector for each of the closest words to the array\n",
        "    arr = np.append(arr, np.array([model[word]]), axis=0)\n",
        "    for wrd_score in close_words:\n",
        "        wrd_vector = model[wrd_score[0]]\n",
        "        word_labels.append(wrd_score[0])\n",
        "        arr = np.append(arr, np.array([wrd_vector]), axis=0)\n",
        "        \n",
        "    # find tsne coords for 2 dimensions\n",
        "    tsne = TSNE(n_components=2, random_state=0)\n",
        "    np.set_printoptions(suppress=True)\n",
        "    Y = tsne.fit_transform(arr)\n",
        "\n",
        "    x_coords = Y[:, 0]\n",
        "    y_coords = Y[:, 1]\n",
        "\n",
        "    # color for words\n",
        "    color = ['red']\n",
        "    for i in range(numb_sim_words):\n",
        "      color.append('blue')\n",
        "     \n",
        "    # display scatter plot\n",
        "    plt.scatter(x_coords, y_coords, c = color)\n",
        "\n",
        "    for label, x, y in zip(word_labels, x_coords, y_coords):\n",
        "        plt.annotate(label, xy=(x, y), xytext=(1, 5), textcoords='offset points')\n",
        "    plt.xlim(min(x_coords)-100, max(x_coords)+100)\n",
        "    plt.ylim(min(y_coords)-100, max(y_coords)+100)\n",
        "    plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dlonk415b1Z3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "outputId": "c8004519-9d5d-40df-96be-01d0531143de"
      },
      "source": [
        "display_closestwords_tsnescatterplot(model, 'challenge')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAeF0lEQVR4nO3de3RV5bnv8e9DAIGgUAqCFSuBHYUAYQVCEPECyE2kIIhKxQpWxSIMrA5txXu30mr1eFeOekRE8YigaER3xQuKSPeWRKkmiAUxKBQx3HISDLfkOX+sxWpCEgjmspLM32eMNciat/XMOcL65Z2X9zV3R0REgqtRrAsQEZHYUhCIiAScgkBEJOAUBCIiAacgEBEJuMaxLqCy2rZt6506dYp1GSIi9UZmZuY2d293pOXqTRB06tSJjIyMWJchIlJvmNnGyiynU0MiIgGnIBARCTgFgYhIwCkIREQCTkEgIhJwCgIRkYBTEIiIBJyCQEQk4BQEIiIBpyAQEQk4BYGISMApCEREAk5BICIScAoCEZGAUxCIiAScgkBEJOAUBCIiAacgEBEJOAWBiEjAKQhERAJOQSAiEnAKAhGRgFMQiIgEnIJARCTgFAQiIgGnIBARCTgFgYhIwCkIREQCTkEgIhJwCgIRkYBTEIiIBJyCQEQk4BrX9AeYWQ6QDxQBB9w91czaAAuATkAOcJG776zpWkREpKzaahEMcveQu6dG3t8EvOfuicB7kfciIhIDsTo1NAZ4LvLzc8D5MapDRCTwaiMIHFhqZplmNiUyrb27b4n8/D3QvrwVzWyKmWWYWUZubm4tlCoiEjw1fo0AOMPdN5vZ8cA7Zra25Ex3dzPz8lZ096eApwBSU1PLXUZERKqmxlsE7r458u8PwGIgDdhqZicARP79oabrEBGR8tVoEJhZvJkde/BnYBiQBaQDkyKLTQJer8k6RESkYjV9aqg9sNjMDn7Wi+7+NzNbBbxsZlcAG4GLargOERGpQI0GgbtvAHqVM307cE5NfraIiFSOniwWEQk4BYGISMApCEREAk5BICIScAoCEZGAUxCIiAScgkBEJOAUBCIiAacgEBEJOAWBiEjAKQhERAJOQSAiEnAKAhGRgFMQiIgEnIJARCTgFAQiIgGnIBARCTgFgYhIwCkIREQCTkEgIhJwCgIRkYBTEIiIBJyCQEQk4BQEIiIBpyAQEQk4BYGISMApCEREAk5BICIScAoCEZGAUxCIiAScgkBEJOAUBCIxlJOTQ48ePaptewMHDiQjIwOATp06sW3btmrbtjRcCgKReurAgQOxLkEaCAWBSIwVFRVx1VVX0b17d4YNG0ZhYWGpv+y3bdtGp06dAJg7dy6jR49m8ODBnHPOORQWFjJhwgS6devG2LFjKSwsLPczXnjhBdLS0giFQlx99dUUFRXV1u5JPRCzIDCzEWb2lZmtN7ObYlWHSKytW7eOadOmkZ2dTevWrXnllVcOu/ynn37KokWL+PDDD5k9ezYtWrTgyy+/5E9/+hOZmZlllv/yyy9ZsGABH3/8MatXryYuLo758+fX1O5IPdQ4Fh9qZnHA48BQYBOwyszS3X1NLOoRiaWEhARCoRAAffr0IScn57DLDx06lDZt2gCwfPlyZsyYAUBycjLJycllln/vvffIzMykb9++ABQWFnL88cdX4x5IfReTIADSgPXuvgHAzF4CxgAKAgmcY445JvpzXFwchYWFNG7cmOLiYgD27NlTavn4+Pij2r67M2nSJP7yl79UvVhpkGJ1auhE4LsS7zdFppViZlPMLMPMMnJzc2utOJFY69SpU/Q0z6JFiypc7qyzzuLFF18EICsri88//7zMMueccw6LFi3ihx9+AGDHjh1s3LixBqqW+qpOXyx296fcPdXdU9u1axfrcqQeuvPOO7n//vurdZsPPfQQP/74Y/T9yJEj2bVrV7Vt//vvv6dv377Mnj2blJSUw94COnXqVAoKCujWrRu33347PXr04OKLLy61TFJSEnfffTfDhg0jOTmZoUOHsmXLlmqrV+o/c/fa/1Cz/sCd7j488n4mgLtX2HZNTU31g3dRiFTWnXfeScuWLbnhhhuqbZudOnUiIyODtm3bHvW6//gHzJ0L+fkwbhyMGAGNDvlzbO7cuWRkZPDYY48d9fZzcnIYNWoUWVlZR72uNDxmlunuqUdaLlYtglVAopklmFlTYAKQHqNapIGZNWsWp5xyCmeccQZfffUVAE8//TR9+/alV69eXHDBBdG/6CdPnlzq1EvLli0B+OCDDxg4cCDjx4+na9euTJw4EXfnkUce4V//+heDBg1i0KBBwL8f3MrJyaFbt25lbgUFWLVqFSeemExKSoiHHrqRZ57pwUUXwfjxELkUAMC+ffu4/fbbWbBgAaFQiAULFvDJJ5/Qv39/UlJSOP3006P7lJ2dHb0lNDk5mXXr1pU6Dhs2bCAlJYVVq1bVzIGWhsPdY/ICRgL/BL4GbjnS8n369HGRI8nIyPAePXr47t27PS8vz7t06eL33Xefb9u2LbrMLbfc4o888oi7u0+aNMkXLlwYnRcfH+/u7suWLfPjjjvOv/vuOy8qKvLTTjvNP/roI3d3P/nkkz03Nze6zsH333zzjcfFxflnn33m7u4XXnihP//88+7u3rVrd2/adKWDO/zRobuDe3y8+5Ilpffh2Wef9WnTpkXf5+Xl+f79+93d/Z133vFx48a5u/v06dP9hRdecHf3vXv3+o8//ujffPONd+/e3deuXeuhUMhXr15d9YMq9RaQ4ZX4Po7VXUO4+1vAW7H6fGmYPvroI8aOHUuLFi0AGD16NBC+kHrrrbeya9cuCgoKGD58+BG3lZaWRseOHQEIhULk5ORwxhlnHHad8m4F3bVrF9u25XPMMf3Ztw/gEmAJALt3w6JFcN55FW8zLy+PSZMmsW7dOsyM/fv3A9C/f39mzZrFpk2bGDduHImJiQDk5uYyZswYXn31VZKSko64nyJ1+mKxSHWZPHkyjz32GF988QV33HFH9JbMkrdpFhcXsy/8TQ2Uva2zMl06VLSOWfnLN2oER7ob9LbbbmPQoEFkZWXxxhtvRGu/5JJLSE9Pp3nz5owcOZL3338fgFatWvHLX/6SFStWHLFeEVAQSANz1lln8dprr1FYWEh+fj5vvPEGAPn5+Zxwwgns37+/1FO1JW/TTE9Pj/61fTjHHnss+fn5la6pdevWtGt3LEVF/xOZ8lJ0XrNmcPnlh99+Xl4eJ54Yvrt67ty50ekbNmygc+fOzJgxgzFjxkRvHW3atCmLFy9m3rx50VtLRQ5HQSD12saNcN11cOaZMH06tG7dm4svvphevXpx7rnnRp+mveuuu+jXrx8DBgyga9eu0fWvuuoqPvzwQ3r16sXf//73Sj2sNWXKFEaMGBG9WFwZc+Y8Q4cOV9GoUYgmTXbTqFErmjWD//xP6NOn9LKDBg1izZo10YvFf/jDH5g5cyYpKSmlWiUvv/wyPXr0IBQKkZWVxWWXXRadFx8fz5IlS3jwwQdJT9d9GHJ4Mbl99KfQ7aNyqC++gAEDYM8e2L8fGjcO/4W9bBmkHvGGudpVUFBAy5Yt+fFHmDLlHnJzt/Dccw/ToUOsK5OGrK7fPipSZddeG74f/+DZnAMHoKAArrkmtnWV58033yQUCpGW1oOdOz/ihRduVQhInaEWgdRbTZv+OwRKMguHwqEPatU1b7/9Nn/84x9LTUtISGDx4sUxqkgamsq2CGJ2+6hIVR17LOzYUXZ68+YV36VTlwwfPrxSt7GK1LQ6/jeTSMV+97vwl35JzZvDlVfWjyAQqSsUBFJv3XknjB4dvkDcqlX43+HD4d57Y12ZSP2iU0NSbzVpAi+9BN99B2vXQmIiREZ0FJGjoBaB1HsnnQRDh9adEMjJyaFHjx6VXv61115jzZqfPibTwY7yRH4qBYFIjFU1CESqSkEgUkUPPPAAPXr0oEePHjz00EMAHDhwgIkTJ9KtWzfGjx8f7fb6pptuIikpieTkZG644QZWrlxJeno6N954I6FQiK+//rrCLrO3bt3K2LFj6dWrF7169WLlypVlarnvvvvo27cvycnJ3HHHHbV3EKR+q0wXpXXhpW6opS462O11QUGB5+fne1JSkn/66acO+IoVK9zd/fLLL492hX3KKad4cXGxu7vv3LnT3ct2hV1Rl9kXXXSRP/jgg+7ufuDAAd+1a5e7/7vr7LffftuvuuoqLy4u9qKiIj/vvPP8ww8/jG7r0M+pjIPbPti9tdQvVLIbarUIRKpgxYoVjB07lvj4eFq2bMm4ceP46KOPOOmkkxgwYAAAl156KStWrKBVq1Y0a9aMK664gldffTXaVfahsrKyOPPMM+nZsyfz588nOzsbgPfff5+pU6cC4Z5NW7VqVWq9pUuXsnTpUlJSUujduzdr164tM1iNSHkUBCI1wA55kMHMaNy4MZ988gnjx49nyZIljBgxotx1K+oy+0jcnZkzZ7J69WpWr17N7bffzsMPP0yvXr34zW9+A8Dy5cs5/fTT6dy5c3RktoKCAs455xx69+5Nz549ef311w/7OUVFRdx4443RU1BPPvkkUPGobgBvvfUWXbt2pU+fPsyYMYNRo0YBsHv3bn7729+SlpZGSkpK9LOPNPqaVLPKNBvqwkunhqQuyszM9J49e/ru3bu9oKDAu3fvHj01tHLlSnd3v+KKK/z+++/3/Px837p1q7u779q1y9u0aePu4ZHG5syZE93mz3/+c9+6davv27fPhwwZ4pMmTXJ394svvviIp4bS0tI8Pz/fs7KyPCEhwdesWePu7tu3b/dJkyb5+PHjvaioyLOzs71Lly7u7r5//37Py8tzd/fc3Fzv0qVL9PRVeaeGnnzySb/rrrvc3X3Pnj3ep08f37BhQ4WjuhUWFnrHjh19w4YN7u4+YcIEP++889zdfebMmdFR3Hbu3OmJiYleUFBQ7uhrcvTQqSGRmte7d28mT55MWloa/fr148orr+RnP/sZp556Ko8//jjdunVj586dTJ06lfz8fEaNGkVycjJnnHEGDzzwAAATJkzgvvvuIyUlha+//rrCLrMffvhhli1bRs+ePenTp0+ZO42GDRvGJZdcQv/+/Rk6dCj79u2jadOmALRp0waA888/n0aNGpGUlMTWrVuB8B+DN998M8nJyQwZMoTNmzdH55Vn6dKlzJs3j1AoRL9+/di+fXv0L/aDo7o1atQoOqrb2rVr6dy5MwkJCQD8+te/LrWte+65h1AoxMCBA9mzZw/ffvst/fv3589//jP33nsvGzdupPmhj5BLtdIDZSJHaceO8ANsJ58MJ54I119/Pddff32pZdauXVtmvRYtWvDJJ5+UmT5gwIBSX+pTp06NXgsoqX379uWetikoKIj+fO2113Lttdfy6KOP8v3339OlS5dSy5YcQc0jp23mz59Pbm4umZmZNGnShE6dOh32dJS78+ijj5bpJ+mDDz446lHd3J1XXnmFU089tdT0bt260a9fP958801GjhzJk08+yeDBgw+7Lfnp1CIQqSR3uP56+MUvYORI+I//gLFjobAw1pWVNXjwYBYuXMj27dsB2FFe73wReXl5HH/88TRp0oRly5axcePGw257+PDhzJ49Ozqa2z//+U92795d4fKnnnoqGzZsICcnB4AFCxaU2tajjz4aDaXPPvsMqHj0NakZahGIVNLjj8OTT8LeveEXwN/+Fh4Z7ZlnYlvbobp3784tt9zC2WefTVxcHCkpKRUuO3HiRH71q1/Rs2dPUlNTS52OKs+VV15JTk4OvXv3xt1p164dr732WoXLN2/enCeeeIIRI0YQHx8fHTUOwuMx//73vyc5OZni4mISEhJYsmQJL7/8Ms8//zxNmjShQ4cO3HzzzUd/EKTSNB6BSCV17gzffFN2erNmsGsXlDgrUutycmDuXPj++3DHe7/6VXjEtrri4Aht7s60adNITEzkuuuui3VZDZ5GKJNAmjx5cvS2yMo62FfPkfoIqujsSlERRB7+jYk334Tu3eEvfwm3WC67DM4++9+tlrrg6aefJhQK0b17d/Ly8rj66qtjXZKUoCAQqaSzzy5/nIOOHaF169qvB8IjtF16aTiI9u0LTysogNWr69bpquuuu47Vq1ezZs0a5s+fX+HDdBIbCgKp1+bNm0dycnKtPDR1773QvPkHmA0ExgNdiYubyOzZjtnRPzRVHTIzwy2SQ/34I8yfX20fIw1cHTqLKHJ0srOzufvuu1m5ciVt27Zlx44dXH/99WzZsoUVK1awdu1aRo8ezfjx42nWrBmLFy/muOOOY9u2bZx22mmMHj26zBPABz3zzDO0atWKVatWsXfvXgYMGMCwYcN45hmYNOkzEhOz6dnzF2RnDyA+/mP27Enl6quvZvny5SQkJJS6V37WrFkMHjyYOXPmsGvXLtLS0hgyZAjx8fFVPgbHHBO+m6k8uvVeKktBIPXW+++/z4UXXkjbtm2Byj00tXz5cho1ahR9aKpDhw7lbnvp0qV8/vnn0RZFXl4e69ato0OHppx1VhrvvNMRgKlTww9NtWzZssxDU0899VR0W+np6dx///0A0YemunXrVuVjEApBmzbh00ElxceDTsNLZSkIpMGpLw9NVQczeOMNGDw4fI2gqAiKi2HiRBg/vto/ThooXSOQequ+PzRVXZKTYfNmmDcPHn4Y/vGP8N1DFZz1EilDLQKpt+r7Q1PV6Zhj4Pzzq3WTEiB6oEzqn8JCaNoU4uJiXUkpemhK6ho9UCYNzzvvQGIiHHssHHdcuOOfyKmbukAPTUl9pRaB1A8ZGeEnuko+wtu8OVx8MTz7bOzqEqnDYt4iMLM7zWyzma2OvEaWmDfTzNab2VdmNvxw2xEBYNasst18FhbCSy9B5GKxiPw0NX1q6EF3D0VebwGYWRIwAegOjACeMLO6dbJX6p4vvyz/yammTeHbb2u/niqoyf6Qfqr09HTuueeeat+u1A+xuEYwBnjJ3fe6+zfAeiAtBnVIfZKaWv7F4X374JDBV+TojR49mptuuinWZUiM1HQQTDezz81sjpn9LDLtROC7Estsikwrw8ymmFmGmWXk5ubWcKlSp916a7i/55JatAgPBnDccbGpqZJqsz8k+GmDyM+dO5fp06cD4RbLjBkzytRXXFzMNddcQ9euXRk6dCgjR4486paN1FGVGdi4ohfwLpBVzmsM0B6IIxw2s4A5kXUeAy4tsY1ngPFH+iwNXi+emek+cKB7ixbuHTu6P/SQe2SQ9boqKyvLExMTPTc3193r7iDyzz77rE+bNs3dvcL6Fi5c6Oeee64XFRX5li1bvHXr1r5w4cIaP4by01HJweur9ECZuw+pzHJm9jRw8AmazcBJJWZ3jEwTObzevWHZslhXcVRi0R9S06ZNo4PIA9FB5A/XH9KhyqtvxYoVXHjhhTRq1IgOHTowaNCgajpKEms19mSxmZ3g7lsib8cSbikApAMvmtkDwC+ARKDsiN4iDVhd6w+pMvVJw1WT1wj+amZfmNnnwCDgOgB3zwZeBtYAfwOmuXs5PaqL1H/1pT+kyhgwYACvvPIKxcXFbN26lQ8++OCo1pe6q8ZaBO7+m8PMm0X4uoFIg1Zf+kOqjAsuuID33nuPpKQkTjrpJHr37k2rVq2OahtSN+nJYpFq9u23sGULJCWFe8OoS6raH9LB9bdv305aWhoff/xxhdcwJPZi/mSxSNDs2gVDh8Kpp8KwYdC+fXhA+bqkqv0hjRo1ilAoxJlnnsltt92mEGgg1CIQqSajRoX7xTs4iDyERwqbNw/GjYtdXRJcahGI1KLcXHj33dIhALB7N/z1r7GpSaSyFAQi1WDHDmjSpPx5P/xQu7WIHC0FgUg16Ny5/CBo3Dh83UCkLlMQiFSDJk3gkUfC3R8d1LQptGoFt90Wu7pEKkNBIFJNLr0Uli6FMWOgV69wf3hffAGRnh5E6iwNXi9SjQYMCL9E6hO1CEREAk5BICIScAoCEZGAUxCIiAScgkBEJOAUBCIiAacgEBEJOAWBiEjAKQhERAJOQSAiEnAKAhGRgFMQiIgEnIJARCTgFAQiIgGnIBARCTgFgYhIwCkIREQCTkEgIhJwCgIRkYBTEIiIBJyCQEQk4BQEIiIBpyAQEQk4BYGISMApCEREAq5KQWBmF5pZtpkVm1nqIfNmmtl6M/vKzIaXmD4iMm29md1Ulc8XEZGqq2qLIAsYBywvOdHMkoAJQHdgBPCEmcWZWRzwOHAukAT8OrKsiIjESOOqrOzuXwKY2aGzxgAvufte4BszWw+kReatd/cNkfVeiiy7pip1iIjIT1dT1whOBL4r8X5TZFpF08tlZlPMLMPMMnJzc2ukUBGRoDtii8DM3gU6lDPrFnd/vfpL+jd3fwp4CiA1NdVr8rNERILqiEHg7kN+wnY3AyeVeN8xMo3DTBcRkRioqVND6cAEMzvGzBKAROATYBWQaGYJZtaU8AXl9BqqQUREKqFKF4vNbCzwKNAOeNPMVrv7cHfPNrOXCV8EPgBMc/eiyDrTgbeBOGCOu2dXaQ9ERKRKzL1+nHpPTU31jIyMWJchIlJvmFmmu6ceaTk9WSwiEnAKAhGRgFMQiIgEnIJARCTgFAQiIgGnIBARCTgFgYhIwCkIREQCTkEgIhJwCgIRkYBTEIiIBJyCQEQk4BQEIiIBpyAQEQk4BYGISMApCEREAk5BICIScAoCEZGAUxCIiAScgkBEJOAUBCIiAacgEBEJOAWBiEjAKQhERAJOQSAiEnAKAhGRgFMQiIgEnIJARCTgFAQiIgGnIBARCTgFgYhIwCkIREQCTkEgIhJwCgIRkYCrUhCY2YVmlm1mxWaWWmJ6JzMrNLPVkdf/LjGvj5l9YWbrzewRM7Oq1CAiIlVT1RZBFjAOWF7OvK/dPRR5/a7E9NnAVUBi5DWiijWIiEgVVCkI3P1Ld/+qssub2QnAce7+3+7uwDzg/KrUICIiVVOT1wgSzOwzM/vQzM6MTDsR2FRimU2RaeUysylmlmFmGbm5uTVYqohIcDU+0gJm9i7QoZxZt7j76xWstgX4pbtvN7M+wGtm1v1oi3P3p4CnAFJTU/1o1xcRkSM7YhC4+5Cj3ai77wX2Rn7ONLOvgVOAzUDHEot2jEwTEZEYqZFTQ2bWzsziIj93JnxReIO7bwH+n5mdFrlb6DKgolaFiIjUgqrePjrWzDYB/YE3zeztyKyzgM/NbDWwCPidu++IzLsG+D/AeuBr4L+qUoOIiFSNhW/eqftSU1M9IyMj1mWIiNQbZpbp7qlHWk5PFouIBJyCQEQk4BQEIiIBpyAQEQm4enOx2MxygY2xriOG2gLbYl1EHaDjEKbjEKbjEFbRcTjZ3dsdaeV6EwRBZ2YZlbn639DpOITpOITpOIRV9Tjo1JCISMApCEREAk5BUH88FesC6ggdhzAdhzAdh7AqHQddIxARCTi1CEREAk5BICIScAqCOszM7jSzzWa2OvIaWWLeTDNbb2ZfmdnwWNZZG8xsRGRf15vZTbGupzaZWY6ZfRH5HciITGtjZu+Y2brIvz+LdZ01wczmmNkPZpZVYlq5+25hj0R+Rz43s96xq7x6VXAcqu37QUFQ9z3o7qHI6y0AM0sCJgDdgRHAEwfHf2iIIvv2OHAukAT8OnIMgmRQ5Hfg4L3iNwHvuXsi8F7kfUM0l/DveEkV7fu5hMc+SQSmALNrqcbaMJeyxwGq6ftBQVA/jQFecve97v4N4bEd0mJcU01KA9a7+wZ33we8RPgYBNkY4LnIz88B58ewlhrj7suBHYdMrmjfxwDzPOy/gdZmdkLtVFqzKjgOFTnq7wcFQd03PdLMnVOi+X8i8F2JZTZFpjVUQdvfQzmw1MwyzWxKZFr7yIh/AN8D7WNTWkxUtO9B/D2plu8HBUGMmdm7ZpZVzmsM4aZtFyAEbAH+V0yLlVg5w917Ez71Mc3Mzio508P3gAfyPvAg7zvV+P1wxMHrpWa5+5DKLGdmTwNLIm83AyeVmN0xMq2hCtr+luLumyP//mBmiwk387ea2QnuviVy+uOHmBZZuyra90D9nrj71oM/V/X7QS2COuyQ85tjgYN3DKQDE8zsGDNLIHxx7JParq8WrQISzSzBzJoSvhCWHuOaaoWZxZvZsQd/BoYR/j1IByZFFpsEvB6bCmOion1PBy6L3D10GpBX4hRSg1Od3w9qEdRtfzWzEOGmbw5wNYC7Z5vZy8Aa4AAwzd2LYlZlDXP3A2Y2HXgbiAPmuHt2jMuqLe2BxWYG4f+vL7r738xsFfCymV1BuHv2i2JYY40xs/8LDATamtkm4A7gHsrf97eAkYQvjv4IXF7rBdeQCo7DwOr6flAXEyIiAadTQyIiAacgEBEJOAWBiEjAKQhERAJOQSAiEnAKAhGRgFMQiIgE3P8H4k/DLmzFYlgAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RCqkZa7Revfi"
      },
      "source": [
        "Avec ces **connaissances**, vous devriez **avoir les outils en main** pour **entraîner vos propres modèles** de NLP ou **améliorer vos modèles** déjà entraînés.\n",
        " \n",
        "Si vous voulez en savoir plus sur **les modèles NLP de Machine Learning** n'hésitez pas à voir [nos autres articles sur le sujet !](https://inside-machinelearning.com/un-modele-classification-binaire-en-nlp/)"
      ]
    }
  ]
}