06_biasandethics.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyMYVG/kGS9AtF3LUFrMGXQx",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/nicolashernandez/teaching_nlp/blob/main/06_biasandethics.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C2NDYD5Es0Ic"
      },
      "source": [
        "--\n",
        "# Bias in data-driven models \n",
        "\n",
        "For the following questions, explain your approach and give codes that support your observations.\n",
        "\n",
        "You may need to turn the execution mode to GPU."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Static word models and word similarity\n",
        "\n",
        "[**Word2Vec** (Google)](https://github.com/tmikolov/word2vec), [**GloVe** (Stanford)](https://nlp.stanford.edu/projects/glove/), [**FastText** (Facebook)](https://github.com/facebookresearch/fastText)... are methods to build semantic word representations from corpora. Some of them uses global word co-occurrence information, others are more sensitive to morphological variations. All these methods are appealing because the word vectors are dense and there are little dimension comparing to the vocabulary size. But the major drawback of these approaches is that representations are non contextual. They remain the same for a word whatever the context is.\n",
        "\n",
        "[**gensim**](https://radimrehurek.com/gensim/) is a library which allows to play with pre-trained models for word or document similarity tasks or to build your own models from your data. "
      ],
      "metadata": {
        "id": "vEukLqdiHbYg"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### QUESTION\n",
        "\n",
        "Have a look at the [gensim-data repository](https://github.com/RaRe-Technologies/gensim-data) and check if it exists models built from twitter. If so give a name. The associated number at the end of a model name correspond to the number of dimensions used for describing a word."
      ],
      "metadata": {
        "id": "xXP-d_qZ2EUe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import gensim.downloader as api\n",
        "\n",
        "api.info()  # show info about available models/datasets"
      ],
      "metadata": {
        "id": "IBe-tmQh0-G6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Load some models"
      ],
      "metadata": {
        "id": "qyi94kxa7Lhm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "wiki_model_50 = api.load(\"glove-wiki-gigaword-50\")\n",
        "wiki_model_200 = api.load(\"glove-wiki-gigaword-200\")\n",
        "#twitter_model = api.load(\"TODO\")"
      ],
      "metadata": {
        "id": "cFxNbswC7Lxb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Get the similar words \n",
        "\n",
        "For each question below, play the game and take the time to make suggestions for answers before running the code that will allow you to look up the model's knowledge and find out what it would answer."
      ],
      "metadata": {
        "id": "iiT2u2uR5nRW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "If I tell you 'king', what do you think of? Make a few suggestions of synonyms or semantically close substitutable words. The `most_similar` method will display the 10 closest words to a given word, from the most similar to the least similar, with for each a similarity score with the given word (thus decreasing scores).\n",
        "\n",
        "Compare the knowledge of distinct models in terms of size and data genre."
      ],
      "metadata": {
        "id": "SBpBH1NE6ErX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "wiki_model_50.most_similar(\"king\")"
      ],
      "metadata": {
        "id": "p4cUkVfe5n04"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "compare with a larger model"
      ],
      "metadata": {
        "id": "BNpAInk57oQf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "wiki_model_200.most_similar(\"king\")"
      ],
      "metadata": {
        "id": "pCA_7HhT7oYk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "compare with a model from a distinct genre"
      ],
      "metadata": {
        "id": "paebkhwv7v1l"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "twitter_model.most_similar(\"king\")"
      ],
      "metadata": {
        "id": "OQlu5Y0t6tSI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "If I ask you to give me words related to 'palace' and 'paris', what do you think? For information, the method accepts a list of words as parameters."
      ],
      "metadata": {
        "id": "yPXSloqg6Rse"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# obtenir les mots similaires relatifs à une liste\n",
        "wiki_model_200.most_similar(['palace', 'paris'])"
      ],
      "metadata": {
        "id": "DCAgSCkn6R0-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "If I add the king and woman vectors and remove the man vector what do I get? Answer before running the code below."
      ],
      "metadata": {
        "id": "fGGgxJrX6JJd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Si j'ajoute les vecteurs de roi et de femme et que je retire le vecteur homme qu'est ce que j'obtiens ?\n",
        "wiki_model_200.most_similar(positive = ['king', 'woman'], negative = ['man'])\n"
      ],
      "metadata": {
        "id": "iOzP0Bjk6Q4w"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### QUESTION\n",
        "\n",
        "* Same question but if I add the vectors of 'paris' and 'japan' and remove the vector of 'france'. Make a proposal and write the code to check."
      ],
      "metadata": {
        "id": "oMF5BJF08MxN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# TODO"
      ],
      "metadata": {
        "id": "aRThGWud8M-8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### QUESTION \n",
        "* Play with the operation `most_similar` on static embedding vectors. Take the word \"human\", remove \"male\" and add \"job\"...\n",
        "* Do you see any situations in operations that expose sexist, racist, religious or other biases? You may compare the various genre models. Give an example of each bias you find."
      ],
      "metadata": {
        "id": "UpbnaBQH3pw9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# TODO"
      ],
      "metadata": {
        "id": "4b2mn7-l8n8v"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3D Visualization of word embeddings with _tensorflow projector_\n",
        "\n",
        "1. Open http://projector.tensorflow.org/\n",
        "2. Select available tensors\n",
        "3. and PROFIT!\n",
        "\n"
      ],
      "metadata": {
        "id": "Zq0FjAc_fSYc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### QUESTION\n",
        "\n",
        "* Do you observe areas that are denser than others? What does this mean? \n",
        "* Test the 3D labels, click on a point/word (set the neighborhood to the minimum value) to observe the illumination of an area, search for a word, view 'isolate 6 points'."
      ],
      "metadata": {
        "id": "5ZkFb50RgFFJ"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "irvZq8LXc6Mj"
      },
      "source": [
        "## Building word2vec and fasttext models with gensim\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iqjuUMhvdXhq"
      },
      "source": [
        "Both Word2Vec and FastText take a normalized corpus segmented into sentences and tokenized into words. \n",
        "\n",
        "We could very well use spaCy or nltk to do this, but this kind of pre-processing takes \"a little time\". We will directly use a corpus from the nltk database available with the segmentation into sentences and the tokenization into words.\n",
        "\n",
        "The code below uses the selection of the gutenberg corpus segmented into sentences and tokens by nltk."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QZ4817rGdX-w",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fccfcf87-dee0-4823-ac0c-57cf13ca244f"
      },
      "source": [
        "import re\n",
        "import nltk\n",
        "nltk.download('gutenberg')\n",
        "nltk.download('punkt')\n",
        "\n",
        "nltk_gutenberg_corpus = list()\n",
        "word_counter = 0\n",
        "\n",
        "for fileid in nltk.corpus.gutenberg.fileids():\n",
        "  segmented_and_tokenized_doc = nltk.corpus.gutenberg.sents(fileid)\n",
        "  for sent in segmented_and_tokenized_doc:\n",
        "    words = [re.sub(r'[^a-zA-Z\\s]', '', word, re.I|re.A).lower() for word in sent]\n",
        "    # 98552 2621785\n",
        "    words = [word for word in words if len(word) > 3]\n",
        "    # 95804 1154977\n",
        "    if len(words)>0: \n",
        "      nltk_gutenberg_corpus.append(words)\n",
        "      word_counter += len(words)\n",
        "      \n",
        "print ('sentences_len:', len(nltk_gutenberg_corpus), 'words_len:', word_counter)"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package gutenberg to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/gutenberg.zip.\n",
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "sentences_len: 95801 words_len: 1154977\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UI2LwobJfVqO"
      },
      "source": [
        "The most common hyper-parameters of\n",
        "[Word2Vec](https://radimrehurek.com/gensim/models/word2vec.html#gensim.models.word2vec.Word2Vec) :\n",
        "\n",
        "* `corpus`: List of tokenized sentences \n",
        "* `size` : Dimensionality of the word vectors (default: 100)\n",
        "* `window` : Maximum distance between the current and predicted word within a sentence\n",
        "* `sg` : Training algorithm: 1 for skip-gram; otherwise CBOW\n",
        "* `iter` :  Number of iterations (epochs) over the corpus\n",
        "* `workers` Use these many worker threads to train the model (=faster training with multicore machines)\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dUUD2EMae1y9"
      },
      "source": [
        "# Set values for various parameters\n",
        "lr = 0.05   # Learning rate\n",
        "dim = 100   # Word vector dimensionality  \n",
        "ws = 5      # Context window size    \n",
        "epoch = 5\n",
        "minCount = 5 # Minimum word count \n",
        "neg = 5\n",
        "loss = 'ns'\n",
        "t = 1e-4\n",
        "#sample = 1e-3   # Downsample setting for frequent words\n",
        "sg=1 \n",
        "\n",
        "params = {\n",
        "    'alpha': lr,\n",
        "    'size': dim,\n",
        "    'window': ws,\n",
        "    'iter': epoch,\n",
        "    'min_count': minCount,\n",
        "    'sample': t,\n",
        "    'sg': 1,\n",
        "    'hs': 0,\n",
        "    'negative': neg\n",
        "}"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v1R9Ee81fRw7"
      },
      "source": [
        "Construction of the models. \n",
        "Observe the speed given the number of words."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fvJPnIjme657",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fa5f0ade-bed9-478c-990f-870cc44e73b4"
      },
      "source": [
        "from gensim.models import Word2Vec, KeyedVectors\n",
        "\n",
        "# \n",
        "%time w2v_model = Word2Vec(nltk_gutenberg_corpus, **params) \n",
        "\n",
        "# save the model\n",
        "!mkdir -p models\n",
        "w2v_model_path = 'models/w2v_nltk-gutenberg_100_5_5_sg.gensim-bin'\n",
        "w2v_model.save(w2v_model_path)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 28.2 s, sys: 134 ms, total: 28.3 s\n",
            "Wall time: 17.9 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P_SvnXRufQk5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ee9a4221-a146-464f-ca99-62caba69fbe3"
      },
      "source": [
        "from gensim.models.fasttext import FastText\n",
        "\n",
        "#\n",
        "%time ft_model = FastText(nltk_gutenberg_corpus, **params)\n",
        "\n",
        "# save the model\n",
        "!mkdir -p models\n",
        "ft_model_path = 'models/ft_nltk-gutenberg_100_5_5_sg.gensim-bin'\n",
        "ft_model.save(ft_model_path)"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 55.6 s, sys: 352 ms, total: 56 s\n",
            "Wall time: 32.4 s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Have a look at the models"
      ],
      "metadata": {
        "id": "ELdEUNnD0zaG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "w2v_model.most_similar(\"love\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MMXKy57l021P",
        "outputId": "22926f7c-1fbb-4f97-f540-e8d26fd01f47"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:1: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).\n",
            "  \"\"\"Entry point for launching an IPython kernel.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('dearly', 0.7445521354675293),\n",
              " ('unfeigned', 0.7432754039764404),\n",
              " ('forbearing', 0.7375692129135132),\n",
              " ('uprightness', 0.7363731861114502),\n",
              " ('integrity', 0.7290647029876709),\n",
              " ('brotherly', 0.7261554002761841),\n",
              " ('forego', 0.7226293087005615),\n",
              " ('solace', 0.7222168445587158),\n",
              " ('bountifully', 0.7199023365974426),\n",
              " ('meekness', 0.7172915935516357)]"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "ft_model.most_similar(\"love\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MbdoKvJ303fV",
        "outputId": "de0b76c0-3be5-4622-8bb7-4f5b59cdce9c"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:1: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).\n",
            "  \"\"\"Entry point for launching an IPython kernel.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('lover', 0.7898315191268921),\n",
              " ('loves', 0.7840328216552734),\n",
              " ('glove', 0.7417938709259033),\n",
              " ('loved', 0.7322890162467957),\n",
              " ('lovers', 0.7271362543106079),\n",
              " ('lovest', 0.6935369968414307),\n",
              " ('beloved', 0.6896727085113525),\n",
              " ('loveit', 0.6596423387527466),\n",
              " ('swerve', 0.6556262969970703),\n",
              " ('loveliest', 0.6489120721817017)]"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F6BI2xNr3Cch"
      },
      "source": [
        "If you want to visualize the model built and saved in gensim-w2v format via projector tensorflow, you can execute the following command and go to the previous section \"visualize a w2v model in 3D via projector tensorflow\"."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yqJSbyzWpefJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dd86e85d-42f5-4788-d94c-622daffc05f6"
      },
      "source": [
        "import gensim\n",
        "from gensim.scripts.word2vec2tensor import word2vec2tensor\n",
        "\n",
        "def convert_gensim_w2v_to_w2v (gensim_w2v_in_path, w2v_out_path):\n",
        "  \"\"\"\n",
        "  convert a model from gensim_w2v format to w2v (orginal) format\n",
        "  \"\"\"\n",
        "  w2v_model = KeyedVectors.load(gensim_w2v_in_path)\n",
        "  vectors = w2v_model.wv\n",
        "  # save memory\n",
        "  # del model\n",
        "\n",
        "  # The trained word vectors can also be stored/loaded from a format compatible\n",
        "  # with the original word2vec implementation via Word2Vec.wv.save_word2vec_format \n",
        "  # and gensim.models.keyedvectors.KeyedVectors.load_word2vec_format().\n",
        "  vectors.save_word2vec_format(w2v_out_path, binary = True)\n",
        "\n",
        "def convert_w2v_to_tsv (w2v_in_path, tsv_out_path):\n",
        "  \"\"\"\n",
        "  convert a model from w2v original format to tsv format\n",
        "  \"\"\"\n",
        "  # When running word2vec2tensor with a file resulting from \n",
        "  # save_word2vec_format, we obtain the following error:\n",
        "  # UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbd in position 0: invalid start byte\n",
        "  # To solve the issue, I have to load with load_word2vec_format the saved file \n",
        "  # and save it again with save_word2vec_format\n",
        "  w2v_model = gensim.models.KeyedVectors.load_word2vec_format(w2v_in_path, binary=True, unicode_errors='ignore')   \n",
        "  w2v_model.wv.save_word2vec_format(w2v_in_path+\".tmp\", binary = True)\n",
        "  word2vec2tensor(w2v_in_path+\".tmp\", tsv_out_path,  binary = True)\n",
        "\n",
        "def convert_gensim_w2v_to_tsv (gensim_w2v_in_path, tsv_out_path):\n",
        "  \"\"\"\n",
        "  convert a model from gensim w2v format to tsv format\n",
        "  \"\"\"\n",
        "  convert_gensim_w2v_to_w2v (gensim_w2v_in_path, gensim_w2v_in_path+\".tmp\")\n",
        "  convert_w2v_to_tsv (gensim_w2v_in_path+\".tmp\", tsv_out_path)\n",
        "\n",
        "w2v_model_path = 'models/w2v_nltk-gutenberg_100_5_5_sg.gensim-bin'\n",
        "tensor_filename = 'models/tensor_nltk-gutenberg_100_5_5_sg.tsv'\n",
        "\n",
        "convert_gensim_w2v_to_tsv(w2v_model_path, tensor_filename)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:28: DeprecationWarning: Call to deprecated `wv` (Attribute will be removed in 4.0.0, use self instead).\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Models comparison and evaluation\n",
        "\n",
        "[`gensim` implements model comparison](https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/Word2Vec_FastText_Comparison.ipynb) according to the **analogical reasoning** task as described in [section 4.1 of the 2013 paper by Mikolov et al.](https://arxiv.org/pdf/1301.3781v3.pdf).\n",
        "\n",
        "```\n",
        ":capital-common-countries\n",
        "Athens Greece Baghdad Iraq\n",
        "Athens Greece Bangkok Thailand\n",
        "...\n",
        ":capital-world\n",
        "Algiers Algeria Baghdad Iraq\n",
        "Ankara Turkey Dublin Ireland\n",
        "...\n",
        ": city-in-state\n",
        "Chicago Illinois Houston Texas\n",
        "Chicago Illinois Philadelphia Pennsylvania\n",
        "...\n",
        ": gram1-adjective-to-adverb\n",
        "amazing amazingly apparent apparently\n",
        "amazing amazingly calm calmly\n",
        "...\n",
        "```\n",
        "\n",
        "Other intrinsic evaluations are possible such as the [computation of a correlation coefficient between a similarity rate computed on the basis of human judgment and a cosine similarity score between Word2Vec representations](https://nlp-ensae.github.io/materials/course2/).\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "o46xofzTgVGi"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below we implement the analogical reasoning task of Mikolov et al."
      ],
      "metadata": {
        "id": "trfpdyhTgt6B"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r7WhKE0_fglD"
      },
      "source": [
        "# download the file questions-words.txt to be used for comparing word embeddings\n",
        "!wget https://raw.githubusercontent.com/tmikolov/word2vec/master/questions-words.txt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AC2EoBqyiwbr"
      },
      "source": [
        "# un oeil sur les n premières lignes du fichier\n",
        "!head questions-words.txt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZZcqh-0Ni8Ms"
      },
      "source": [
        "Definition of the method for computing the resolution performance of the analogy task"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qbiByLHzi8y1"
      },
      "source": [
        "def print_accuracy(model, questions_file):\n",
        "    print('Evaluating...\\n')\n",
        "    acc = model.accuracy(questions_file)\n",
        "    #acc = model.wv.evaluate_word_analogies(questions_file)\n",
        "\n",
        "    sem_correct = sum((len(acc[i]['correct']) for i in range(5)))\n",
        "    sem_total = sum((len(acc[i]['correct']) + len(acc[i]['incorrect'])) for i in range(5))\n",
        "    sem_acc = 100*float(sem_correct)/sem_total\n",
        "    print('\\nSemantic: {:d}/{:d}, Accuracy: {:.2f}%'.format(sem_correct, sem_total, sem_acc))\n",
        "    \n",
        "    syn_correct = sum((len(acc[i]['correct']) for i in range(5, len(acc)-1)))\n",
        "    syn_total = sum((len(acc[i]['correct']) + len(acc[i]['incorrect'])) for i in range(5,len(acc)-1))\n",
        "    syn_acc = 100*float(syn_correct)/syn_total\n",
        "    print('Morphologic: {:d}/{:d}, Accuracy: {:.2f}%\\n'.format(syn_correct, syn_total, syn_acc))\n",
        "    return (sem_acc, syn_acc)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fULOXjn8jOK_"
      },
      "source": [
        "Run the evaluation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sw8fHge1jIw7"
      },
      "source": [
        "#\n",
        "word_analogies_file = 'questions-words.txt'\n",
        "\n",
        "print('\\nLoading Word2Vec embeddings')\n",
        "w2v_model = KeyedVectors.load(w2v_model_path)\n",
        "print('Accuracy for Word2Vec:')\n",
        "print_accuracy(w2v_model, word_analogies_file)\n",
        "\n",
        "print('\\nLoading FastText embeddings')\n",
        "ft_model = KeyedVectors.load(ft_model_path)\n",
        "print('Accuracy for FastText (with n-grams):')\n",
        "print_accuracy(ft_model, word_analogies_file)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N11hzoM9koW2"
      },
      "source": [
        "#### QUESTION\n",
        "* Which of the two models gives the best results on morphological analysis? On semantic analysis? Is this consistent with what you know about the models? \n",
        "* Rerun the construction of the models and then compare them. Do you get the same performance scores? Why?\n",
        "* The training data are classic novels from the Gutenberg collection. If the data had been taken from Wikipedia, what results would have changed? If you want to test, below I give you a snippet of code that retrieves a normalized version of the wikipedia and runs the w2v and ft model building. It will take a few minutes... \n",
        "* In your opinion, from a model comparison perspective, is it important to build them on the same data?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0DJ8-vccGwrP"
      },
      "source": [
        "## Text generation\n",
        "\n",
        "[huggingface](https://huggingface.co/models) plays the role of the \"github\" for pre-trained and fine-tuned language models.\n",
        "\n",
        "The code below allows you to use the gpt2 model and to test the generation of text in English. \n",
        "\n",
        "For your information, [BLOOM](https://huggingface.co/bigscience/bloom) which stands for BigScience Large Open-science Open-access Multilingual Language Model is one of the most recent auto-regressive model which has been created. More than 50 Gb to load... We won't use it here."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install transformers"
      ],
      "metadata": {
        "id": "NQ6HCpZFz9qN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import transformers\n",
        "\n",
        "from transformers import pipeline, set_seed\n",
        "\n",
        "generator = pipeline('text-generation', model='gpt2')\n",
        "\n",
        "set_seed(42)\n",
        "\n",
        "print(generator(\"The White man worked as a\", max_length=10, num_return_sequences=5))\n",
        "print(generator(\"The Black man worked as a\", max_length=10, num_return_sequences=5))"
      ],
      "metadata": {
        "id": "2ZPo88Tuz-iS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LogIfuPQIQ1f"
      },
      "source": [
        "### QUESTION\n",
        "\n",
        "* Among the hugging face community resources, look for the text generation resource that uses the `gpt-fr-cased-base` template for French. Give the link to the page and implement the code provided on the page. \n",
        "* Give the beginnings of sentences to start the generation. Do you find situations that reveal sexist, racist, religious or other biases? Give examples of each.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YyEqlMYvGy-2"
      },
      "source": [
        "#TODO"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Question/answering\n",
        "\n"
      ],
      "metadata": {
        "id": "EPY1zqnN1fpq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "* Same question as previously but on question/answering models. Search a huggingface model and imagine some questions to detect bias"
      ],
      "metadata": {
        "id": "bRWkRCiw12ot"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# TODO"
      ],
      "metadata": {
        "id": "UAfENjxF2COS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Translation (Google)\n",
        "\n",
        "\n",
        "> *She is a doctor. He is a nurse.*\n"
      ],
      "metadata": {
        "id": "8vq37kPRHS5r"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MaZ-kRIWui7t"
      },
      "source": [
        "\n",
        "### QUESTIONS\n",
        "\n",
        "* Open [Google Translate in your browser](https://translate.google.fr/?hl=fr&sl=en&tl=fr&text=She%20is%20a%20doctor.%20He%20is%20a%20nurse.&op=translate)\n",
        "* Translate from English (source language) to French (target language). Click twice on \"Switch languages\" (to translate once to French and then to translate back from French to English). Do you notice anything?\n",
        "* Do the same thing using Hungarian as the target language. Do you observe anything?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "scPvHf9wumB1"
      },
      "source": [
        "**TODO**\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Automatic detection of bias\n",
        "\n",
        "huggingface holds the following model *d4data/bias-detection-model* for detecting bias in news. This model is part of the [Research topic \"Bias and Fairness in AI\" conducted by Deepak John Reji, Shaina Raza](https://github.com/dreji18/Fairness-in-AI).\n",
        "\n"
      ],
      "metadata": {
        "id": "I5o0ghH4N9E7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### QUESTION\n",
        "* Give the URL of the model. Test the bias prediction model (starting from the example code) and qualitatively assess the limitations. Does it detect any bias? Give false positive/negative examples."
      ],
      "metadata": {
        "id": "4zeIYGF3P6QL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**TODO**"
      ],
      "metadata": {
        "id": "IpMjXJOeQT5Q"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Translation with sequence to sequence T5\n",
        "\n",
        "The following code allows to [use the *t5-small* prompt-based seq-to-seq model available on hugging face](https://huggingface.co/t5-small).\n"
      ],
      "metadata": {
        "id": "PegyjFkiNRLG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install transformers\n",
        "\n",
        "\n",
        "from transformers import T5TokenizerFast, T5ForConditionalGeneration\n",
        "\n",
        "tokenizer = T5TokenizerFast.from_pretrained('t5-small')\n",
        "\n",
        "model = T5ForConditionalGeneration.from_pretrained('t5-small', return_dict=True)\n",
        "\n",
        "input = \"My name is Azeem and I live in India\"\n",
        "\n",
        "# You can also use \"translate English to French\" and \"translate English to Romanian\"\n",
        "input_ids = tokenizer(\"translate English to Romanian: \"+input, return_tensors=\"pt\").input_ids  # Batch size 1\n",
        "\n",
        "outputs = model.generate(input_ids)\n",
        "\n",
        "decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)\n",
        "\n",
        "print(decoded)"
      ],
      "metadata": {
        "id": "DOX4lW9L4-VS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### QUESTION\n",
        "* Test the translation model to translate from/to your favorite languages and qualitatively assess the limitations. \n",
        "* Test translating from English to Russian and from Russian to English... e.g. \"The mind is strong but the flesh is weak\".\n",
        "* Develop an application that predicts bias in your favorite through translation from English. You may need to find an adequate translation model. How good the bias detection model is through the translation ? "
      ],
      "metadata": {
        "id": "jZAjav6dOtjA"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "**TODO**"
      ],
      "metadata": {
        "id": "z_Rsu2IWQuhP"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#ELIZA: a very basic Rogerian psychotherapist chatbot\n",
        "\n",
        "> [ELIZA](https://en.wikipedia.org/wiki/ELIZA)  was made to respond like a Rogerian psychotherapist. In this instance, the therapist \"reflects\" on questions by turning the questions back at the patient. Created to demonstrate the superficiality of communication between humans and machines, Eliza simulated conversation by using a \"pattern matching\" and substitution methodology that gave users an illusion of understanding on the part of the program, but had no built in framework for contextualizing events. An [example of ELIZA conversation here](https://upload.wikimedia.org/wikipedia/commons/7/79/ELIZA_conversation.png) and a [ELIZA demo there](http://psych.fullerton.edu/mbirnbaum/psych101/eliza.htm). \n",
        "\n",
        "Write your own psychotherapist chatbot. Based on the available models on huggingface or other NLP technology (such as [spaCy](https://spacy.io/)), extend the simple following chatbot by adding new abilities such as:\n",
        "- evaluate your sentiment and make feedback about it \n",
        "- generate question taking noun phrases of your utterances as input\n",
        "- recognize named entities and generate questions about them\n",
        "- classify your message in a topic category and generate questions about it\n",
        "- whatever you want... even make two agents discussing toguether..."
      ],
      "metadata": {
        "id": "OZ1yl1wCEhYA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print ('Good morning, my name is Eliza. Is something troubling you ?')\n",
        "message = input()\n",
        "while message != 'stop':\n",
        "  print('Why do you say', message)\n",
        "  message = input()"
      ],
      "metadata": {
        "id": "Yh_zmogMEkYF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Calculate the CO2 impact of your GPU usage in this course\n",
        "\n",
        "To do this use the [Machine Learning has a carbon footprint] application (https://mlco2.github.io/impact).\n",
        "\n",
        "Start by identifying your GPU, then approximate the time spent on GPU and calculate... observe the equivalences.\n",
        "\n"
      ],
      "metadata": {
        "id": "uyewU5oVHxqE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "**TODO**"
      ],
      "metadata": {
        "id": "2oxFY_gRQzAp"
      }
    }
  ]
}