abstractive_summarization_on_FB.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-D5C8B7pZPt5"
      },
      "source": [
        "# Fine Tuning Transformer for Summary Generation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "99AmMJ1hZPt8"
      },
      "source": [
        "\n",
        "### Introduction\n",
        "\n",
        "In this tutorial we will be fine tuning a transformer model for **Summarization Task**. \n",
        "In this task a summary of a given article/document is generated when passed through a network. There are 2 types of summary generation mechanisms:\n",
        "\n",
        "1. ***Extractive Summary:*** the network calculates the most important sentences from the article and gets them together to provide the most meaningful information from the article.\n",
        "2. ***Abstractive Summary***: The network creates new sentences to encapsulate maximum gist of the article and generates that as output. The sentences in the summary may or may not be contained in the article. \n",
        "\n",
        "In this tutorial we will be generating ***Abstractive Summary***. \n",
        "\n",
        "#### Flow of the notebook\n",
        "\n",
        "* As with all the tutorials previously, this notebook also follows a easy to follow steps. Making the process of fine tuning and training a Transformers model a straight forward task.\n",
        "* However, unlike the other notebooks, in the tutorial, most of the sections have been created into functions, and they are called from the `main()` in the end of the notebook. \n",
        "* This is done to leverage the [Weights and Biases Service](https://www.wandb.com/) WandB in short.\n",
        "* It is a experiment tracking, parameter optimization and artifact management service. That can be very easily integrated to any of the Deep learning or Machine learning frameworks. \n",
        "\n",
        "The notebook will be divided into separate sections to provide a organized walk through for the process used. This process can be modified for individual use cases. The sections are:\n",
        "\n",
        "1. [Preparing Environment and Importing Libraries](#section01)\n",
        "2. [Preparing the Dataset for data processing: Class](#section02)\n",
        "3. [Fine Tuning the Model: Function](#section03)\n",
        "4. [Validating the Model Performance: Function](#section04)\n",
        "5. [Main Function](#section05)\n",
        "    * [Initializing WandB](#section501)\n",
        "    * [Importing and Pre-Processing the domain data](#section502)\n",
        "    * [Creation of Dataset and Dataloader](#section503)\n",
        "    * [Neural Network and Optimizer](#section504)\n",
        "    * [Training Model and Logging to WandB](#section505)\n",
        "    * [Validation and generation of Summary](#section506)\n",
        "6. [Examples of the Summary Generated from the model](#section06)\n",
        "\n",
        "\n",
        "#### Technical Details\n",
        "\n",
        "This script leverages on multiple tools designed by other teams. Details of the tools used below. Please ensure that these elements are present in your setup to successfully implement this script.\n",
        "\n",
        "- **Data**:\n",
        "\t- We are using the News Summary dataset available at [Kaggle](https://www.kaggle.com/sunnysai12345/news-summary)\n",
        "\t- This dataset is the collection created from Newspapers published in India, extracting, details that are listed below.  We are referring only to the first csv file from the data dump: `news_summary.csv`\n",
        "\t- There are`4514` rows of data.  Where each row has the following data-point:\n",
        "\t\t- **author** : Author of the article\n",
        "\t\t- **date** : Date the article was published\n",
        "\t\t- **headline**: Headline for the published article\n",
        "\t\t- **read_more** : URL for the article to follow online\n",
        "\t\t- **text**: This is the summary of the article\n",
        "\t\t- **ctext**: This is the complete article\n",
        "\n",
        "\n",
        "- **Language Model Used**: \n",
        "    - This notebook uses one of the most recent and novel transformers model ***T5***. [Research Paper](https://arxiv.org/abs/1910.10683)    \n",
        "    - ***T5*** in many ways is one of its kind transformers architecture that not only gives state of the art results in many NLP tasks, but also has a very radical approach to NLP tasks.\n",
        "    - **Text-2-Text** - According to the graphic taken from the T5 paper. All NLP tasks are converted to a **text-to-text** problem. Tasks such as translation, classification, summarization and question answering, all of them are treated as a text-to-text conversion problem, rather than seen as separate unique problem statements.\n",
        "    - **Unified approach for NLP Deep Learning** - Since the task is reflected purely in the text input and output, you can use the same model, objective, training procedure, and decoding process to ANY task. Above framework can be used for any task - show Q&A, summarization, etc. \n",
        "   - We will be taking inputs from the T5 paper to prepare our dataset prior to fine tuning and training.    \n",
        "   - [Documentation for python](https://huggingface.co/transformers/model_doc/t5.html)\n",
        "\n",
        "![**Each NLP problem as a “text-to-text” problem** - input: text, output: text](https://miro.medium.com/max/4006/1*D0J1gNQf8vrrUpKeyD8wPA.png) \n",
        "\t \n",
        "\n",
        "\n",
        "- Hardware Requirements: \n",
        "\t- Python 3.6 and above\n",
        "\t- Pytorch, Transformers and\n",
        "\t- All the stock Python ML Library\n",
        "\t- GPU enabled setup \n",
        "   \n",
        "\n",
        "- **Script Objective**:\n",
        "\t- The objective of this script is to fine tune ***T5 *** to be able to generate summary, that a close to or better than the actual summary  while ensuring the important information from the article is not lost.\n",
        "\n",
        "---\n",
        "NOTE: \n",
        "We are using the Weights and Biases Tool-set in  this tutorial. The different components will be explained as we go through the article."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z-rfbVRoZPt_"
      },
      "source": [
        "<a id='section01'></a>\n",
        "### Preparing Environment and Importing Libraries\n",
        "\n",
        "At this step we will be installing the necessary libraries followed by importing the libraries and modules needed to run our script. \n",
        "We will be installing:\n",
        "* transformers\n",
        "* wandb\n",
        "\n",
        "Libraries imported are:\n",
        "* Pandas\n",
        "* Pytorch\n",
        "* Pytorch Utils for Dataset and Dataloader\n",
        "* Transformers\n",
        "* T5 Model and Tokenizer\n",
        "* wandb\n",
        "\n",
        "Followed by that we will preapre the device for CUDA execeution. This configuration is needed if you want to leverage on onboard GPU. First we will check the GPU avaiable to us, using the nvidia command followed by defining our device.\n",
        "\n",
        "Finally, we will be logging into the [wandb](https://www.wandb.com/) serice using the login command"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "%cd /content/drive/MyDrive/code/CS224N"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hNhtiv-QaBia",
        "outputId": "73bc2538-e490-4c57-895c-9272495fa1c6"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n",
            "/content/drive/MyDrive/code/CS224N\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "! ls data"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Vg2zmcL4cNKy",
        "outputId": "2fa7e97d-1e75-46bf-e059-fb7135f46046"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "mydata.csv  SCBAll.csv\t       SCBAll_trained2.csv  SCBAll_trained.csv\n",
            "mydata.pkl  SCBAll_pruned.csv  SCBAll_trained3.csv  SCB_all_v2.csv\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install sentencepiece\n",
        "!pip install bert-score"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L5BfAMRuZu_5",
        "outputId": "7f506d23-171b-43b4-a725-a36484dd5884"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting sentencepiece\n",
            "  Downloading sentencepiece-0.1.96-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)\n",
            "\u001b[?25l\r\u001b[K     |▎                               | 10 kB 26.4 MB/s eta 0:00:01\r\u001b[K     |▌                               | 20 kB 21.4 MB/s eta 0:00:01\r\u001b[K     |▉                               | 30 kB 16.3 MB/s eta 0:00:01\r\u001b[K     |█                               | 40 kB 14.3 MB/s eta 0:00:01\r\u001b[K     |█▍                              | 51 kB 7.4 MB/s eta 0:00:01\r\u001b[K     |█▋                              | 61 kB 8.6 MB/s eta 0:00:01\r\u001b[K     |██                              | 71 kB 9.7 MB/s eta 0:00:01\r\u001b[K     |██▏                             | 81 kB 8.9 MB/s eta 0:00:01\r\u001b[K     |██▍                             | 92 kB 9.7 MB/s eta 0:00:01\r\u001b[K     |██▊                             | 102 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███                             | 112 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███▎                            | 122 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███▌                            | 133 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███▉                            | 143 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |████                            | 153 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |████▎                           | 163 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |████▋                           | 174 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |████▉                           | 184 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████▏                          | 194 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████▍                          | 204 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████▊                          | 215 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████                          | 225 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████▏                         | 235 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████▌                         | 245 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████▊                         | 256 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████                         | 266 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████▎                        | 276 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████▋                        | 286 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████▉                        | 296 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████                        | 307 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████▍                       | 317 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████▋                       | 327 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████████                       | 337 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████████▏                      | 348 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████████▌                      | 358 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████████▊                      | 368 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████████                      | 378 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████████▎                     | 389 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████████▌                     | 399 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████████▉                     | 409 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████                     | 419 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████▍                    | 430 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████▋                    | 440 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████▉                    | 450 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████████▏                   | 460 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████████▍                   | 471 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████████▊                   | 481 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████████████                   | 491 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████████████▎                  | 501 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████████████▌                  | 512 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████████████▊                  | 522 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████████████                  | 532 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████████████▎                 | 542 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████████████▋                 | 552 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████████████▉                 | 563 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████████▏                | 573 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████████▍                | 583 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████████▋                | 593 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████████████                | 604 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████████████▏               | 614 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████████████▌               | 624 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████████████▊               | 634 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████████████████               | 645 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████████████████▎              | 655 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████████████████▌              | 665 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████████████████▉              | 675 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████████████████              | 686 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████████████████▍             | 696 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████████████████▋             | 706 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████████████             | 716 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████████████▏            | 727 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████████████▍            | 737 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████████████▊            | 747 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████████████████            | 757 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████████████████▎           | 768 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████████████████▌           | 778 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████████████████▉           | 788 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████████████████████           | 798 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████████████████████▎          | 808 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████████████████████▋          | 819 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████████████████████▉          | 829 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████████████████████▏         | 839 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████████████████████▍         | 849 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████████████████████▊         | 860 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████████████████         | 870 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████████████████▏        | 880 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████████████████▌        | 890 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████████████████▊        | 901 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████████████████████        | 911 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████████████████████▎       | 921 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████████████████████▋       | 931 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████████████████████▉       | 942 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████████████████████████       | 952 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████████████████████████▍      | 962 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████████████████████████▋      | 972 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████████████████████████      | 983 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▏     | 993 kB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▌     | 1.0 MB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▊     | 1.0 MB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████████████████████     | 1.0 MB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▎    | 1.0 MB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▌    | 1.0 MB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▉    | 1.1 MB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████████████████████████    | 1.1 MB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▍   | 1.1 MB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▋   | 1.1 MB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▉   | 1.1 MB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▏  | 1.1 MB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▍  | 1.1 MB 8.1 MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▊  | 1.1 MB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████  | 1.1 MB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▎ | 1.1 MB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▌ | 1.2 MB 8.1 MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▊ | 1.2 MB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████ | 1.2 MB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▎| 1.2 MB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▋| 1.2 MB 8.1 MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████▉| 1.2 MB 8.1 MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 1.2 MB 8.1 MB/s \n",
            "\u001b[?25hInstalling collected packages: sentencepiece\n",
            "Successfully installed sentencepiece-0.1.96\n",
            "Collecting bert-score\n",
            "  Downloading bert_score-0.3.11-py3-none-any.whl (60 kB)\n",
            "\u001b[K     |████████████████████████████████| 60 kB 5.0 MB/s \n",
            "\u001b[?25hRequirement already satisfied: torch>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from bert-score) (1.10.0+cu111)\n",
            "Requirement already satisfied: tqdm>=4.31.1 in /usr/local/lib/python3.7/dist-packages (from bert-score) (4.63.0)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from bert-score) (3.2.2)\n",
            "Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.7/dist-packages (from bert-score) (21.3)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from bert-score) (2.23.0)\n",
            "Requirement already satisfied: pandas>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from bert-score) (1.3.5)\n",
            "Collecting transformers>=3.0.0numpy\n",
            "  Downloading transformers-4.17.0-py3-none-any.whl (3.8 MB)\n",
            "\u001b[K     |████████████████████████████████| 3.8 MB 15.7 MB/s \n",
            "\u001b[?25hRequirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.9->bert-score) (3.0.7)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=1.0.1->bert-score) (2018.9)\n",
            "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=1.0.1->bert-score) (2.8.2)\n",
            "Requirement already satisfied: numpy>=1.17.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=1.0.1->bert-score) (1.21.5)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas>=1.0.1->bert-score) (1.15.0)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch>=1.0.0->bert-score) (3.10.0.2)\n",
            "Collecting sacremoses\n",
            "  Downloading sacremoses-0.0.47-py2.py3-none-any.whl (895 kB)\n",
            "\u001b[K     |████████████████████████████████| 895 kB 57.4 MB/s \n",
            "\u001b[?25hRequirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from transformers>=3.0.0numpy->bert-score) (4.11.2)\n",
            "Collecting huggingface-hub<1.0,>=0.1.0\n",
            "  Downloading huggingface_hub-0.4.0-py3-none-any.whl (67 kB)\n",
            "\u001b[K     |████████████████████████████████| 67 kB 7.5 MB/s \n",
            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers>=3.0.0numpy->bert-score) (3.6.0)\n",
            "Collecting tokenizers!=0.11.3,>=0.11.1\n",
            "  Downloading tokenizers-0.11.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.5 MB)\n",
            "\u001b[K     |████████████████████████████████| 6.5 MB 55.9 MB/s \n",
            "\u001b[?25hCollecting pyyaml>=5.1\n",
            "  Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)\n",
            "\u001b[K     |████████████████████████████████| 596 kB 77.5 MB/s \n",
            "\u001b[?25hRequirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers>=3.0.0numpy->bert-score) (2019.12.20)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers>=3.0.0numpy->bert-score) (3.7.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->bert-score) (0.11.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->bert-score) (1.3.2)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->bert-score) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->bert-score) (2021.10.8)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->bert-score) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->bert-score) (2.10)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers>=3.0.0numpy->bert-score) (1.1.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers>=3.0.0numpy->bert-score) (7.1.2)\n",
            "Installing collected packages: pyyaml, tokenizers, sacremoses, huggingface-hub, transformers, bert-score\n",
            "  Attempting uninstall: pyyaml\n",
            "    Found existing installation: PyYAML 3.13\n",
            "    Uninstalling PyYAML-3.13:\n",
            "      Successfully uninstalled PyYAML-3.13\n",
            "Successfully installed bert-score-0.3.11 huggingface-hub-0.4.0 pyyaml-6.0 sacremoses-0.0.47 tokenizers-0.11.6 transformers-4.17.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "WD_vnyLXZQzD",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "996f3dad-e3c5-4a5e-f5ab-1e0fd6dac045"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[K     |████████████████████████████████| 1.7 MB 7.1 MB/s \n",
            "\u001b[K     |████████████████████████████████| 144 kB 78.3 MB/s \n",
            "\u001b[K     |████████████████████████████████| 181 kB 69.9 MB/s \n",
            "\u001b[K     |████████████████████████████████| 63 kB 1.4 MB/s \n",
            "\u001b[?25h  Building wheel for pathtools (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
          ]
        }
      ],
      "source": [
        "!pip install transformers -q\n",
        "!pip install wandb -q\n",
        "\n",
        "# Code for TPU packages install\n",
        "# !curl -q https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py\n",
        "# !python pytorch-xla-env-setup.py --apt-packages libomp5 libopenblas-dev"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "pzM1_ykHaFur"
      },
      "outputs": [],
      "source": [
        "# Importing stock libraries\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import torch\n",
        "import torch.nn.functional as F\n",
        "from torch.utils.data import Dataset, DataLoader, RandomSampler, SequentialSampler\n",
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Importing the T5 modules from huggingface/transformers\n",
        "# from transformers import T5Tokenizer, T5ForConditionalGeneration\n",
        "# from transformers import T5ForConditionalGeneration\n",
        "# from transformers import AutoTokenizer as T5Tokenizer\n",
        "# from transformers import AutoModelWithLMHead as T5ForConditionalGeneration\n",
        "from transformers import AutoModelWithLMHead, AutoTokenizer\n",
        "# AutoModelWithLMHead\n",
        "from bert_score import score\n",
        "\n",
        "# tokenizer = AutoTokenizer.from_pretrained(\"t5-base\")\n",
        "\n",
        "# model = AutoModelWithLMHead.from_pretrained(\"t5-base\")\n",
        "\n",
        "# WandB – Import the wandb library\n",
        "import wandb"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KvPxXdKJguYB",
        "outputId": "74eb78f4-c6c8-48d7-ca64-62f852c64ad4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mon Mar 14 02:40:47 2022       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   33C    P0    26W / 250W |      0MiB / 16280MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ]
        }
      ],
      "source": [
        "# Checking out the GPU we have access to. This is output is from the google colab version. \n",
        "!nvidia-smi"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "NLxxwd1scQNv"
      },
      "outputs": [],
      "source": [
        "# # Setting up the device for GPU usage\n",
        "from torch import cuda\n",
        "device = 'cuda' if cuda.is_available() else 'cpu'\n",
        "\n",
        "# Preparing for TPU usage\n",
        "# import torch_xla\n",
        "# import torch_xla.core.xla_model as xm\n",
        "# device = xm.xla_device()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L-ePh9dEKXMw",
        "outputId": "dfd3f1fe-a0a4-4e9e-fff9-6d9b0429cbc3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[34m\u001b[1mwandb\u001b[0m: You can find your API key in your browser here: https://wandb.ai/authorize\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: Paste an API key from your profile and hit enter, or press ctrl+c to quit: \n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: Appending key for api.wandb.ai to your netrc file: /root/.netrc\n"
          ]
        }
      ],
      "source": [
        "# Login to wandb to log the model run and all the parameters\n",
        "!wandb login"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D6tqtNUNZPuH"
      },
      "source": [
        "<a id='section02'></a>\n",
        "### Preparing the Dataset for data processing: Class\n",
        "\n",
        "We will start with creation of Dataset class - This defines how the text is pre-processed before sending it to the neural network. This dataset will be used the the Dataloader method that will feed  the data in batches to the neural network for suitable training and processing. \n",
        "The Dataloader and Dataset will be used inside the `main()`.\n",
        "Dataset and Dataloader are constructs of the PyTorch library for defining and controlling the data pre-processing and its passage to neural network. For further reading into Dataset and Dataloader read the [docs at PyTorch](https://pytorch.org/docs/stable/data.html)\n",
        "\n",
        "#### *CustomDataset* Dataset Class\n",
        "- This class is defined to accept the Dataframe as input and generate tokenized output that is used by the **T5** model for training. \n",
        "- We are using the **T5** tokenizer to tokenize the data in the `text` and `ctext` column of the dataframe. \n",
        "- The tokenizer uses the ` batch_encode_plus` method to perform tokenization and generate the necessary outputs, namely: `source_id`, `source_mask` from the actual text and `target_id` and `target_mask` from the summary text.\n",
        "- To read further into the tokenizer, [refer to this document](https://huggingface.co/transformers/model_doc/t5.html#t5tokenizer)\n",
        "- The *CustomDataset* class is used to create 2 datasets, for training and for validation.\n",
        "- *Training Dataset* is used to fine tune the model: **80% of the original data**\n",
        "- *Validation Dataset* is used to evaluate the performance of the model. The model has not seen this data during training. \n",
        "\n",
        "#### Dataloader: Called inside the `main()`\n",
        "- Dataloader is used to for creating training and validation dataloader that load data to the neural network in a defined manner. This is needed because all the data from the dataset cannot be loaded to the memory at once, hence the amount of data loaded to the memory and then passed to the neural network needs to be controlled.\n",
        "- This control is achieved using the parameters such as `batch_size` and `max_len`.\n",
        "- Training and Validation dataloaders are used in the training and validation part of the flow respectively"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "id": "932p8NhxeNw4"
      },
      "outputs": [],
      "source": [
        "# Creating a custom dataset for reading the dataframe and loading it into the dataloader to pass it to the neural network at a later stage for finetuning the model and to prepare it for predictions\n",
        "\n",
        "class CustomDataset(Dataset): ### seq2seq from ctext to text\n",
        "\n",
        "    def __init__(self, dataframe, tokenizer, source_len, summ_len):\n",
        "        self.tokenizer = tokenizer\n",
        "        self.data = dataframe\n",
        "        self.source_len = source_len\n",
        "        self.summ_len = summ_len\n",
        "        self.text = self.data.text\n",
        "        self.ctext = self.data.ctext\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.text)\n",
        "\n",
        "    def __getitem__(self, index):\n",
        "        ctext = str(self.ctext[index])\n",
        "        ctext = ' '.join(ctext.split())\n",
        "\n",
        "        text = str(self.text[index])\n",
        "        text = ' '.join(text.split())\n",
        "\n",
        "        source = self.tokenizer.batch_encode_plus([ctext], max_length= self.source_len, pad_to_max_length=True,return_tensors='pt')\n",
        "        target = self.tokenizer.batch_encode_plus([text], max_length= self.summ_len, pad_to_max_length=True,return_tensors='pt')\n",
        "\n",
        "        source_ids = source['input_ids'].squeeze()\n",
        "        source_mask = source['attention_mask'].squeeze()\n",
        "        target_ids = target['input_ids'].squeeze()\n",
        "        target_mask = target['attention_mask'].squeeze()\n",
        "\n",
        "        return {\n",
        "            'source_ids': source_ids.to(dtype=torch.long), \n",
        "            'source_mask': source_mask.to(dtype=torch.long), \n",
        "            'target_ids': target_ids.to(dtype=torch.long),\n",
        "            'target_ids_y': target_ids.to(dtype=torch.long)\n",
        "        }"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NzWPG4heZPuJ"
      },
      "source": [
        "<a id='section03'></a>\n",
        "### Fine Tuning the Model: Function\n",
        "\n",
        "Here we define a training function that trains the model on the training dataset created above, specified number of times (EPOCH), An epoch defines how many times the complete data will be passed through the network. \n",
        "\n",
        "This function is called in the `main()`\n",
        "\n",
        "Following events happen in this function to fine tune the neural network:\n",
        "- The epoch, tokenizer, model, device details, testing_ dataloader and optimizer are passed to the `train ()` when its called from the `main()`\n",
        "- The dataloader passes data to the model based on the batch size.\n",
        "- `language_model_labels` are calculated from the `target_ids` also, `source_id` and `attention_mask` are extracted.\n",
        "- The model outputs first element gives the loss for the forward pass. \n",
        "- Loss value is used to optimize the weights of the neurons in the network.\n",
        "- After every 10 steps the loss value is logged in the wandb service. This log is then used to generate graphs for analysis. Such as [these](https://app.wandb.ai/abhimishra-91/transformers_tutorials_summarization?workspace=user-abhimishra-91)\n",
        "- After every 500 steps the loss value is printed in the console."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "id": "SaPAR7TWmxoM"
      },
      "outputs": [],
      "source": [
        "# Creating the training function. This will be called in the main function. It is run depending on the epoch value.\n",
        "# The model is put into train mode and then we wnumerate over the training loader and passed to the defined network \n",
        "def train(epoch, tokenizer, model, device, loader, optimizer):\n",
        "    model.train()\n",
        "    for _,data in enumerate(loader, 0):\n",
        "        y = data['target_ids'].to(device, dtype = torch.long)\n",
        "        y_ids = y[:, :-1].contiguous()\n",
        "        # lm_labels = y[:, 1:].clone().detach()\n",
        "        # lm_labels[y[:, 1:] == tokenizer.pad_token_id] = -100\n",
        "        lm_labels = y[:, :].clone().detach()\n",
        "        lm_labels[lm_labels == tokenizer.pad_token_id] = -100\n",
        "        ids = data['source_ids'].to(device, dtype = torch.long)\n",
        "        mask = data['source_mask'].to(device, dtype = torch.long)\n",
        "\n",
        "        # outputs = model(input_ids = ids, attention_mask = mask, decoder_input_ids=y_ids, lm_labels=lm_labels)\n",
        "        # outputs = model(input_ids = ids, attention_mask = mask, decoder_input_ids=y_ids, labels=lm_labels)\n",
        "        outputs = model(input_ids = ids, attention_mask = mask, labels=lm_labels)\n",
        "        loss = outputs[0]\n",
        "        \n",
        "        if _%10 == 0:\n",
        "            wandb.log({\"Training Loss\": loss.item()})\n",
        "\n",
        "        if _%500==0:\n",
        "            print(f'Epoch: {epoch}, Loss:  {loss.item()}')\n",
        "        \n",
        "        optimizer.zero_grad()\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "        # xm.optimizer_step(optimizer)\n",
        "        # xm.mark_step()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ioF6Q5EOZPuK"
      },
      "source": [
        "<a id='section04'></a>\n",
        "### Validating the Model Performance: Function\n",
        "\n",
        "During the validation stage we pass the unseen data(Testing Dataset), trained model, tokenizer and device details to the function to perform the validation run. This step generates new summary for dataset that it has not seen during the training session. \n",
        "\n",
        "This function is called in the `main()`\n",
        "\n",
        "This unseen data is the 20% of `news_summary.csv` which was seperated during the Dataset creation stage. \n",
        "During the validation stage the weights of the model are not updated. We use the generate method for generating new text for the summary. \n",
        "\n",
        "It depends on the `Beam-Search coding` method developed for sequence generation for models with LM head. \n",
        "\n",
        "The generated text and originally summary are decoded from tokens to text and returned to the `main()`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "id": "j9TNdHlQ0CLz"
      },
      "outputs": [],
      "source": [
        "def validate(epoch, tokenizer, model, device, loader):\n",
        "    model.eval()\n",
        "    predictions = []\n",
        "    actuals = []\n",
        "    with torch.no_grad():\n",
        "        for _, data in enumerate(loader, 0):\n",
        "            y = data['target_ids'].to(device, dtype = torch.long)\n",
        "            ids = data['source_ids'].to(device, dtype = torch.long)\n",
        "            mask = data['source_mask'].to(device, dtype = torch.long)\n",
        "\n",
        "            generated_ids = model.generate(\n",
        "                input_ids = ids,\n",
        "                attention_mask = mask, \n",
        "                max_length=150, \n",
        "                num_beams=2,\n",
        "                repetition_penalty=2.5, \n",
        "                length_penalty=1.0, \n",
        "                early_stopping=True\n",
        "                )\n",
        "            preds = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=True) for g in generated_ids]\n",
        "            target = [tokenizer.decode(t, skip_special_tokens=True, clean_up_tokenization_spaces=True)for t in y]\n",
        "            if _%100==0:\n",
        "                print(f'Completed {_}')\n",
        "\n",
        "            predictions.extend(preds)\n",
        "            actuals.extend(target)\n",
        "    return predictions, actuals"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4f0zNAk0ZPuL"
      },
      "source": [
        "<a id='section05'></a>\n",
        "### Main Function\n",
        "\n",
        "The `main()` as the name suggests is the central location to execute all the functions/flows created above in the notebook. The following steps are executed in the `main()`:\n",
        "\n",
        "\n",
        "<a id='section501'></a>\n",
        "#### Initializing WandB \n",
        "\n",
        "* The `main()` begins with initializing WandB run under a specific project. This command initiates a new run for each execution of this command. \n",
        "\n",
        "* Before we proceed any further i will give a brief overview of the **[WandB Service](https://www.wandb.com/)**\n",
        "\n",
        "* This service has been created to track ML experiments, Optimize the experiments and save artifacts. It is designed to seamlessly integrate with all the Machine Learning and Deep Learning Frameworks. Each script can be organized into *Project* and each execution of the script will be registered as a *run* in the respective project.\n",
        "\n",
        "* The service can be configured to log several default metrics, such a network weights, hardware usage, gradients and weights of the network. \n",
        "\n",
        "* It can also be used to log user defined metrics, such a loss in the `train()`.\n",
        "\n",
        "* This particular tutorial is logged in the project: **[transformers_tutorials_summarization](https://app.wandb.ai/abhimishra-91/transformers_tutorials_summarization?workspace=user-abhimishra-91)**\n",
        "\n",
        "**One of the charts from the project**\n",
        "![](meta/wandb.png)\n",
        "\n",
        "* Visit the project page to see the details of different runs and what information is logged by the service. \n",
        "\n",
        "* Following the initialization of the WandB service we define configuration parameters that will be used across the tutorial such as `batch_size`, `epoch`, `learning_rate` etc.\n",
        "\n",
        "* These parameters are also passed to the WandB config. The config construct with all the parameters can be optimized using the Sweep service from WandB. Currently, that is outof scope of this tutorial. \n",
        "\n",
        "* Next we defining seed values so that the experiment and results can be reproduced.\n",
        "\n",
        "\n",
        "<a id='section502'></a>\n",
        "#### Importing and Pre-Processing the domain data\n",
        "\n",
        "We will be working with the data and preparing it for fine tuning purposes. \n",
        "*Assuming that the `news_summary.csv` is already downloaded in your `data` folder*\n",
        "\n",
        "* The file is imported as a dataframe and give it the headers as per the documentation.\n",
        "* Cleaning the file to remove the unwanted columns.\n",
        "* A new string is added to the main article column `summarize: ` prior to the actual article. This is done because **T5** had similar formatting for the summarization dataset. \n",
        "* The final Dataframe will be something like this:\n",
        "\n",
        "|text|ctext|\n",
        "|--|--|\n",
        "|summary-1|summarize: article 1|\n",
        "|summary-2|summarize: article 2|\n",
        "|summary-3|summarize: article 3|\n",
        "\n",
        "* Top 5 rows of the dataframe are printed on the console.\n",
        "\n",
        "<a id='section503'></a>\n",
        "#### Creation of Dataset and Dataloader\n",
        "\n",
        "* The updated dataframe is divided into 80-20 ratio for test and validation. \n",
        "* Both the data-frames are passed to the `CustomerDataset` class for tokenization of the new articles and their summaries.\n",
        "* The tokenization is done using the length parameters passed to the class.\n",
        "* Train and Validation parameters are defined and passed to the `pytorch Dataloader contstruct` to create `train` and `validation` data loaders.\n",
        "* These dataloaders will be passed to `train()` and `validate()` respectively for training and validation action.\n",
        "* The shape of datasets is printed in the console.\n",
        "\n",
        "\n",
        "<a id='section504'></a>\n",
        "#### Neural Network and Optimizer\n",
        "\n",
        "* In this stage we define the model and optimizer that will be used for training and to update the weights of the network. \n",
        "* We are using the `t5-base` transformer model for our project. You can read about the `T5 model` and its features above. \n",
        "* We use the `T5ForConditionalGeneration.from_pretrained(\"t5-base\")` commad to define our model. The `T5ForConditionalGeneration` adds a Language Model head to our `T5 model`. The Language Model head allows us to generate text based on the training of `T5 model`.\n",
        "* We are using the `Adam` optimizer for our project. This has been a standard for all our tutorials and is something that can be changed updated to see how different optimizer perform with different learning rates. \n",
        "* There is also a scope for doing more with Optimizer such a decay, momentum to dynamically update the Learning rate and other parameters. All those concepts have been kept out of scope for these tutorials. \n",
        "\n",
        "\n",
        "<a id='section505'></a>\n",
        "#### Training Model and Logging to WandB\n",
        "\n",
        "* Now we log all the metrics in WandB project that we have initialized above.\n",
        "* Followed by that we call the `train()` with all the necessary parameters.\n",
        "* Loss at every 500th step is printed on the console.\n",
        "* Loss at every 10th step is logged as Loss in the WandB service.\n",
        "\n",
        "\n",
        "<a id='section506'></a>\n",
        "#### Validation and generation of Summary\n",
        "\n",
        "* After the training is completed, the validation step is initiated.\n",
        "* As defined in the validation function, the model weights are not updated. We use the fine tuned model to generate new summaries based on the article text.\n",
        "* An output is printed on the console giving a count of how many steps are complete after every 100th step. \n",
        "* The original summary and generated summary are converted into a list and returned to the main function. \n",
        "* Both the lists are used to create the final dataframe with 2 columns **Generated Summary** and **Actual Summary**\n",
        "* The dataframe is saved as a csv file in the local drive.\n",
        "* A qualitative analysis can be done with the Dataframe. "
      ]
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "tiH0Ii2iajpg"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZtNs9ytpCow2"
      },
      "outputs": [],
      "source": [
        "# def main():\n",
        "#     # WandB – Initialize a new run\n",
        "#     wandb.init(project=\"transformers_tutorials_summarization\")\n",
        "\n",
        "#     # WandB – Config is a variable that holds and saves hyperparameters and inputs\n",
        "#     # Defining some key variables that will be used later on in the training  \n",
        "#     config = wandb.config          # Initialize config\n",
        "#     config.TRAIN_BATCH_SIZE = 2    # input batch size for training (default: 64)\n",
        "#     config.VALID_BATCH_SIZE = 2    # input batch size for testing (default: 1000)\n",
        "#     config.TRAIN_EPOCHS = 2        # number of epochs to train (default: 10)\n",
        "#     config.VAL_EPOCHS = 1 \n",
        "#     config.LEARNING_RATE = 1e-4    # learning rate (default: 0.01)\n",
        "#     config.SEED = 42               # random seed (default: 42)\n",
        "#     config.MAX_LEN = 512\n",
        "#     config.SUMMARY_LEN = 150 \n",
        "\n",
        "#     # Set random seeds and deterministic pytorch for reproducibility\n",
        "#     torch.manual_seed(config.SEED) # pytorch random seed\n",
        "#     np.random.seed(config.SEED) # numpy random seed\n",
        "#     torch.backends.cudnn.deterministic = True\n",
        "\n",
        "#     # tokenzier for encoding the text\n",
        "#     tokenizer = T5Tokenizer.from_pretrained(\"t5-base\")\n",
        "    \n",
        "\n",
        "#     # Importing and Pre-Processing the domain data\n",
        "#     # Selecting the needed columns only. \n",
        "#     # Adding the summarzie text in front of the text. This is to format the dataset similar to how T5 model was trained for summarization task. \n",
        "#     df = pd.read_csv('./data/news_summary.csv',encoding='latin-1')\n",
        "#     df = df[['text','ctext']]\n",
        "#     df.ctext = 'summarize: ' + df.ctext\n",
        "#     print(df.head())\n",
        "\n",
        "    \n",
        "#     # Creation of Dataset and Dataloader\n",
        "#     # Defining the train size. So 80% of the data will be used for training and the rest will be used for validation. \n",
        "#     train_size = 0.8\n",
        "#     train_dataset=df.sample(frac=train_size,random_state = config.SEED)\n",
        "#     val_dataset=df.drop(train_dataset.index).reset_index(drop=True)\n",
        "#     train_dataset = train_dataset.reset_index(drop=True)\n",
        "\n",
        "#     print(\"FULL Dataset: {}\".format(df.shape))\n",
        "#     print(\"TRAIN Dataset: {}\".format(train_dataset.shape))\n",
        "#     print(\"TEST Dataset: {}\".format(val_dataset.shape))\n",
        "\n",
        "\n",
        "#     # Creating the Training and Validation dataset for further creation of Dataloader\n",
        "#     training_set = CustomDataset(train_dataset, tokenizer, config.MAX_LEN, config.SUMMARY_LEN)\n",
        "#     val_set = CustomDataset(val_dataset, tokenizer, config.MAX_LEN, config.SUMMARY_LEN)\n",
        "\n",
        "#     # Defining the parameters for creation of dataloaders\n",
        "#     train_params = {\n",
        "#         'batch_size': config.TRAIN_BATCH_SIZE,\n",
        "#         'shuffle': True,\n",
        "#         'num_workers': 0\n",
        "#         }\n",
        "\n",
        "#     val_params = {\n",
        "#         'batch_size': config.VALID_BATCH_SIZE,\n",
        "#         'shuffle': False,\n",
        "#         'num_workers': 0\n",
        "#         }\n",
        "\n",
        "#     # Creation of Dataloaders for testing and validation. This will be used down for training and validation stage for the model.\n",
        "#     training_loader = DataLoader(training_set, **train_params)\n",
        "#     val_loader = DataLoader(val_set, **val_params)\n",
        "\n",
        "\n",
        "    \n",
        "#     # Defining the model. We are using t5-base model and added a Language model layer on top for generation of Summary. \n",
        "#     # Further this model is sent to device (GPU/TPU) for using the hardware.\n",
        "#     model = T5ForConditionalGeneration.from_pretrained(\"t5-base\")\n",
        "#     model = model.to(device)\n",
        "\n",
        "#     # Defining the optimizer that will be used to tune the weights of the network in the training session. \n",
        "#     optimizer = torch.optim.Adam(params =  model.parameters(), lr=config.LEARNING_RATE)\n",
        "\n",
        "#     # Log metrics with wandb\n",
        "#     wandb.watch(model, log=\"all\")\n",
        "#     # Training loop\n",
        "#     print('Initiating Fine-Tuning for the model on our dataset')\n",
        "\n",
        "#     for epoch in range(config.TRAIN_EPOCHS):\n",
        "#         train(epoch, tokenizer, model, device, training_loader, optimizer)\n",
        "\n",
        "\n",
        "#     # Validation loop and saving the resulting file with predictions and acutals in a dataframe.\n",
        "#     # Saving the dataframe as predictions.csv\n",
        "#     print('Now generating summaries on our fine tuned model for the validation dataset and saving it in a dataframe')\n",
        "#     for epoch in range(config.VAL_EPOCHS):\n",
        "#         predictions, actuals = validate(epoch, tokenizer, model, device, val_loader)\n",
        "#         final_df = pd.DataFrame({'Generated Text':predictions,'Actual Text':actuals})\n",
        "#         final_df.to_csv('./models/predictions.csv')\n",
        "#         print('Output Files generated for review')\n",
        "\n",
        "# if __name__ == '__main__':\n",
        "#     main()"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# tmp = pd.read_csv('./predictions_10epochs.csv')\n",
        "# tmp.to_csv('./predictions_10epochs_0.csv')"
      ],
      "metadata": {
        "id": "6iPKrMR0jeLf"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# from bert_score import score"
      ],
      "metadata": {
        "id": "59opPGAmjvZM"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df_val = pd.read_csv('df_val.csv')\n",
        "df_test = pd.read_csv('df_test.csv')"
      ],
      "metadata": {
        "id": "bIstiro0CsMA"
      },
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df_val.columns\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3Yqm45dDjf0H",
        "outputId": "094bda8a-4442-4bdd-82bf-8d8ecbe1e089"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Index(['Unnamed: 0', 'Actual Text', 'Generated Text', 'Generated Text 1',\n",
              "       'Generated Text 2', 'Generated Text 3', 'Generated Text 4',\n",
              "       'Generated Text 5', 'Generated Text 6', 'Generated Text 7',\n",
              "       'Generated Text 8', 'Generated Text 9', 'Generated Text 10',\n",
              "       'Generated Text 11', 'Generated Text 12', 'Generated Text 13',\n",
              "       'Generated Text 14', 'Generated Text 15', 'Generated Text 16',\n",
              "       'Generated Text 17', 'Generated Text 18', 'Generated Text 19',\n",
              "       'Generated Text 20'],\n",
              "      dtype='object')"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pd.concat([val_dataset[['title']],df_val[['Actual Text','Generated Text','Generated Text 20']]], axis=1)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1216
        },
        "id": "4jXYV4pwf1MG",
        "outputId": "78d9c929-57bd-45f3-c2e4-81feca030873"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                 title  \\\n",
              "0    ‘Toy Story 4’ Boycotted by Christian Group Ove...   \n",
              "1            Mary Tyle Moore's Cause of Death Revealed   \n",
              "2    Adding This 1 Ridiculous Word Will Improve You...   \n",
              "3    Gamer figures out how much the full Battlefiel...   \n",
              "4    Cam Newton Busts Out Gunslinger Look With Epic...   \n",
              "..                                                 ...   \n",
              "121  Will wearing water jugs on your head combat co...   \n",
              "122  Here's Why Flight Attendants Ask You to Raise ...   \n",
              "123  You won't be able to buy some corded blinds st...   \n",
              "124  Australia Will Suddenly Move 1.8 Meters North ...   \n",
              "125  Warning: If You See This Plant- Whatever You D...   \n",
              "\n",
              "                                           Actual Text  \\\n",
              "0                                   Scene in comments.   \n",
              "1                              Cardiopulmonary arrest.   \n",
              "2                                              Cheers.   \n",
              "3          $120. $70 for the game, $50 for all the DLC   \n",
              "4                             Picture in the comments.   \n",
              "..                                                 ...   \n",
              "121                      The CDC doesn't recommend it.   \n",
              "122  So the crew can assess the surroundings in an ...   \n",
              "123  Industry standards are eliminating/reducing si...   \n",
              "124  Australians aren't actually going anywhere...t...   \n",
              "125  It's an invasive species called giant hogweed....   \n",
              "\n",
              "                                        Generated Text  \\\n",
              "0                     Toy Story 4 was a hit with kids.   \n",
              "1                   She died of cardiopulmonary arrest   \n",
              "2                       No answer available in context   \n",
              "3                       No answer available in context   \n",
              "4                       No answer available in context   \n",
              "..                                                 ...   \n",
              "121                     No answer available in context   \n",
              "122                     No answer available in context   \n",
              "123                     No answer available in context   \n",
              "124  No, the country will jut northwards by 1.8 met...   \n",
              "125  It is a dangerous plant that can cause severe ...   \n",
              "\n",
              "                                     Generated Text 20  \n",
              "0                                    One Million Moms.  \n",
              "1                              Cardiopulmonary arrest.  \n",
              "2                                              Cheers.  \n",
              "3                                                $120.  \n",
              "4    Patriots quarterback Cam Newton wore a gunslin...  \n",
              "..                                                 ...  \n",
              "121                                               Yes.  \n",
              "122                                  To ensure safety.  \n",
              "123                    To prevent child strangulation.  \n",
              "124  The Australian Plate is moving about 7 centime...  \n",
              "125     It contains toxic sap that can cause blisters.  \n",
              "\n",
              "[126 rows x 4 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-d8ee2ab3-450f-4d55-9ea9-7efd79df751d\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>title</th>\n",
              "      <th>Actual Text</th>\n",
              "      <th>Generated Text</th>\n",
              "      <th>Generated Text 20</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>‘Toy Story 4’ Boycotted by Christian Group Ove...</td>\n",
              "      <td>Scene in comments.</td>\n",
              "      <td>Toy Story 4 was a hit with kids.</td>\n",
              "      <td>One Million Moms.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Mary Tyle Moore's Cause of Death Revealed</td>\n",
              "      <td>Cardiopulmonary arrest.</td>\n",
              "      <td>She died of cardiopulmonary arrest</td>\n",
              "      <td>Cardiopulmonary arrest.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Adding This 1 Ridiculous Word Will Improve You...</td>\n",
              "      <td>Cheers.</td>\n",
              "      <td>No answer available in context</td>\n",
              "      <td>Cheers.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Gamer figures out how much the full Battlefiel...</td>\n",
              "      <td>$120. $70 for the game, $50 for all the DLC</td>\n",
              "      <td>No answer available in context</td>\n",
              "      <td>$120.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Cam Newton Busts Out Gunslinger Look With Epic...</td>\n",
              "      <td>Picture in the comments.</td>\n",
              "      <td>No answer available in context</td>\n",
              "      <td>Patriots quarterback Cam Newton wore a gunslin...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>121</th>\n",
              "      <td>Will wearing water jugs on your head combat co...</td>\n",
              "      <td>The CDC doesn't recommend it.</td>\n",
              "      <td>No answer available in context</td>\n",
              "      <td>Yes.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>122</th>\n",
              "      <td>Here's Why Flight Attendants Ask You to Raise ...</td>\n",
              "      <td>So the crew can assess the surroundings in an ...</td>\n",
              "      <td>No answer available in context</td>\n",
              "      <td>To ensure safety.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>123</th>\n",
              "      <td>You won't be able to buy some corded blinds st...</td>\n",
              "      <td>Industry standards are eliminating/reducing si...</td>\n",
              "      <td>No answer available in context</td>\n",
              "      <td>To prevent child strangulation.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>124</th>\n",
              "      <td>Australia Will Suddenly Move 1.8 Meters North ...</td>\n",
              "      <td>Australians aren't actually going anywhere...t...</td>\n",
              "      <td>No, the country will jut northwards by 1.8 met...</td>\n",
              "      <td>The Australian Plate is moving about 7 centime...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>125</th>\n",
              "      <td>Warning: If You See This Plant- Whatever You D...</td>\n",
              "      <td>It's an invasive species called giant hogweed....</td>\n",
              "      <td>It is a dangerous plant that can cause severe ...</td>\n",
              "      <td>It contains toxic sap that can cause blisters.</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>126 rows × 4 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d8ee2ab3-450f-4d55-9ea9-7efd79df751d')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-d8ee2ab3-450f-4d55-9ea9-7efd79df751d button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-d8ee2ab3-450f-4d55-9ea9-7efd79df751d');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pd.concat([test_dataset[['title']],df_test[['Actual Text','Generated Text','Generated Text 20']]], axis=1)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1167
        },
        "id": "Mo_qkVV4gsnh",
        "outputId": "80143e63-86c8-4eb6-efbf-df6122c7a57a"
      },
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                 title  \\\n",
              "0    Fans Leaving The Show, Because Of This… AMC Pa...   \n",
              "1    It took more than Ian McShane and a prayer to ...   \n",
              "2    This Couple Had a Pizza Instead of Cake At The...   \n",
              "3    Oprah Winfrey's 'greatest extravagance' is fro...   \n",
              "4    Rogue One Writer Can't Believe What Disney Let...   \n",
              "..                                                 ...   \n",
              "120  This Is The Rarest Hair And Eye Color Combination   \n",
              "121  This is why Emma Watson avoids taking selfies ...   \n",
              "122      Stop Picky Eating with this One Simple Phrase   \n",
              "123  Ross Lynch Will Play An Iconic Character On Ne...   \n",
              "124              Is French the language of the future?   \n",
              "\n",
              "                                           Actual Text  \\\n",
              "0    The Walking Dead has become more about gore th...   \n",
              "1    Basically, HBO handed back the rights because ...   \n",
              "2          Pizza. It's literally right in the picture.   \n",
              "3               English Muffins from The Model Bakery.   \n",
              "4                   He killed all the main characters.   \n",
              "..                                                 ...   \n",
              "120         Red hair and blue eyes (0.017% of people.)   \n",
              "121  She doesn't like her location being revealed o...   \n",
              "122                        \"You don't have to eat it.\"   \n",
              "123                                        Ross Lynch.   \n",
              "124  French, due to the large growth of French-spea...   \n",
              "\n",
              "                                        Generated Text  \\\n",
              "0    They are blaming the showrunner for their mani...   \n",
              "1                       No answer available in context   \n",
              "2    They had a pizza instead of cake at their wedd...   \n",
              "3    The muffins are made by two women in Napa Valley.   \n",
              "4           They are a part of the Star Wars universe.   \n",
              "..                                                 ...   \n",
              "120                                 No, it’s not blue.   \n",
              "121                     No answer available in context   \n",
              "122                               They have to try it.   \n",
              "123                     No answer available in context   \n",
              "124                      No, French is not the future.   \n",
              "\n",
              "                                     Generated Text 20  \n",
              "0    They feel the show is manipulating its audienc...  \n",
              "1    \"This is the first time I'm saying this in an ...  \n",
              "2                       They served it in a pizza box.  \n",
              "3                                     English muffins.  \n",
              "4                         \"They can't kill everybody.\"  \n",
              "..                                                 ...  \n",
              "120                            Red hair and blue eyes.  \n",
              "121                           Because they're tracked.  \n",
              "122                                        They won't.  \n",
              "123                                         Ross Lynch  \n",
              "124                                            French.  \n",
              "\n",
              "[125 rows x 4 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-a9f56f5b-fb3b-48b9-81e3-97cb9dfee38c\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>title</th>\n",
              "      <th>Actual Text</th>\n",
              "      <th>Generated Text</th>\n",
              "      <th>Generated Text 20</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Fans Leaving The Show, Because Of This… AMC Pa...</td>\n",
              "      <td>The Walking Dead has become more about gore th...</td>\n",
              "      <td>They are blaming the showrunner for their mani...</td>\n",
              "      <td>They feel the show is manipulating its audienc...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>It took more than Ian McShane and a prayer to ...</td>\n",
              "      <td>Basically, HBO handed back the rights because ...</td>\n",
              "      <td>No answer available in context</td>\n",
              "      <td>\"This is the first time I'm saying this in an ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>This Couple Had a Pizza Instead of Cake At The...</td>\n",
              "      <td>Pizza. It's literally right in the picture.</td>\n",
              "      <td>They had a pizza instead of cake at their wedd...</td>\n",
              "      <td>They served it in a pizza box.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Oprah Winfrey's 'greatest extravagance' is fro...</td>\n",
              "      <td>English Muffins from The Model Bakery.</td>\n",
              "      <td>The muffins are made by two women in Napa Valley.</td>\n",
              "      <td>English muffins.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Rogue One Writer Can't Believe What Disney Let...</td>\n",
              "      <td>He killed all the main characters.</td>\n",
              "      <td>They are a part of the Star Wars universe.</td>\n",
              "      <td>\"They can't kill everybody.\"</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>120</th>\n",
              "      <td>This Is The Rarest Hair And Eye Color Combination</td>\n",
              "      <td>Red hair and blue eyes (0.017% of people.)</td>\n",
              "      <td>No, it’s not blue.</td>\n",
              "      <td>Red hair and blue eyes.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>121</th>\n",
              "      <td>This is why Emma Watson avoids taking selfies ...</td>\n",
              "      <td>She doesn't like her location being revealed o...</td>\n",
              "      <td>No answer available in context</td>\n",
              "      <td>Because they're tracked.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>122</th>\n",
              "      <td>Stop Picky Eating with this One Simple Phrase</td>\n",
              "      <td>\"You don't have to eat it.\"</td>\n",
              "      <td>They have to try it.</td>\n",
              "      <td>They won't.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>123</th>\n",
              "      <td>Ross Lynch Will Play An Iconic Character On Ne...</td>\n",
              "      <td>Ross Lynch.</td>\n",
              "      <td>No answer available in context</td>\n",
              "      <td>Ross Lynch</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>124</th>\n",
              "      <td>Is French the language of the future?</td>\n",
              "      <td>French, due to the large growth of French-spea...</td>\n",
              "      <td>No, French is not the future.</td>\n",
              "      <td>French.</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>125 rows × 4 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a9f56f5b-fb3b-48b9-81e3-97cb9dfee38c')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-a9f56f5b-fb3b-48b9-81e3-97cb9dfee38c button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-a9f56f5b-fb3b-48b9-81e3-97cb9dfee38c');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# P, R, F1 = score([str(i) for i in tmp['Actual Text'].tolist()], [str(i) for i in tmp['Generated Text'].tolist()], lang='en')\n",
        "# f1scores = F1.tolist()\n",
        "# epochs = [0]*len(F1)\n",
        "# for epoch in range(1,11):\n",
        "#   P, R, F1 = score([str(i) for i in tmp['Actual Text'].tolist()], [str(i) for i in tmp[f'Generated Text {epoch}'].tolist()], lang='en')\n",
        "#   f1scores += F1.tolist()\n",
        "#   epochs += [epoch]*len(F1)"
      ],
      "metadata": {
        "id": "js_TFsphC4vf"
      },
      "execution_count": 37,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2zDWRQ7ljx8y",
        "outputId": "972f1615-98bf-4495-a2ad-38593fcfeb3b"
      },
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "2O2NLLwfA6Iw"
      },
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "3XwDfDHvBZmm"
      },
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# f1scores"
      ],
      "metadata": {
        "id": "RUoXRKenB0EM"
      },
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# plt.plot(epochs, f1scores, yerr = epochs)\n",
        "\n",
        "# sns.lineplot(x=epochs, y=f1scores, ci='sd')\n",
        "# plt.xlabel('Epochs')\n",
        "# plt.ylabel('F1 score')\n",
        "# plt.savefig('validation_score.png')\n",
        "# sns.lineplot(x=[1,2], y=[2,4],ci=[1,3])"
      ],
      "metadata": {
        "id": "HEQmdBsABOgd"
      },
      "execution_count": 68,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# df = pd.read_csv('./data/SCBAll_pruned.csv')\n",
        "df = pd.read_csv('./data/SCB_all_v2.csv')\n",
        "# df = df[['text','ctext']]\n",
        "# df.ctext = 'summarize: ' + df.ctext\n",
        "# print(df.head())\n"
      ],
      "metadata": {
        "id": "8YlLAVAYk4AR"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "## for pruning; no need to run if you load pruned data\n",
        "df = df[['article', 'title', 'summary']]\n",
        "df = df.dropna()\n",
        "for i in range(len(df)):\n",
        "  if '#' in df.loc[df.index[i],'summary']:\n",
        "    print(i)\n",
        "    df.loc[df.index[i],'summary'] = ' '.join([i  if '#' not in i else ''  for i in df.loc[df.index[i],'summary'].split(' ')])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "S1d0UIASJoa4",
        "outputId": "83962790-41e8-442f-b31b-083609c8028b"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "7\n",
            "15\n",
            "45\n",
            "46\n",
            "47\n",
            "48\n",
            "53\n",
            "69\n",
            "71\n",
            "74\n",
            "81\n",
            "159\n",
            "163\n",
            "164\n",
            "165\n",
            "172\n",
            "256\n",
            "364\n",
            "419\n",
            "420\n",
            "421\n",
            "422\n",
            "423\n",
            "425\n",
            "426\n",
            "427\n",
            "428\n",
            "429\n",
            "430\n",
            "431\n",
            "432\n",
            "433\n",
            "434\n",
            "435\n",
            "436\n",
            "437\n",
            "438\n",
            "439\n",
            "440\n",
            "441\n",
            "446\n",
            "602\n",
            "616\n",
            "642\n",
            "1251\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# df.to_csv('SCB_pruned2.csv')"
      ],
      "metadata": {
        "id": "vwzQbyOGJ7E4"
      },
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df['ctext'] = \"context: \"+df.article +\" <question for context: \"+ df.title +\" </s>\"\n",
        "df['text'] = df.summary"
      ],
      "metadata": {
        "id": "up166KQTlG44"
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "AoC3r2fFOVx1"
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# df_test = df.iloc[-len(df)//10]\n",
        "# df = df.iloc[:-len(df)//10]"
      ],
      "metadata": {
        "id": "BScPo9hQN84A"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "len(df)\n",
        "# df.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JCF_czIdk8Y4",
        "outputId": "77861d62-d011-4d59-e20b-d5fb0d1d962d"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1257"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "F3NRl9jhKD3A"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "# WandB – Initialize a new run\n",
        "wandb.init(project=\"transformers_tutorials_summarization\")\n",
        "\n",
        "# WandB – Config is a variable that holds and saves hyperparameters and inputs\n",
        "# Defining some key variables that will be used later on in the training  \n",
        "config = wandb.config          # Initialize config\n",
        "config.TRAIN_BATCH_SIZE = 2    # input batch size for training (default: 64)\n",
        "config.VALID_BATCH_SIZE = 2    # input batch size for testing (default: 1000)\n",
        "config.TRAIN_EPOCHS = 20        # number of epochs to train (default: 10)\n",
        "config.VAL_EPOCHS = 1 \n",
        "config.LEARNING_RATE = 1e-5    # learning rate (default: 0.01)\n",
        "config.SEED = 42               # random seed (default: 42)\n",
        "config.MAX_LEN = 512\n",
        "config.SUMMARY_LEN = 150 \n",
        "\n",
        "# Set random seeds and deterministic pytorch for reproducibility\n",
        "torch.manual_seed(config.SEED) # pytorch random seed\n",
        "np.random.seed(config.SEED) # numpy random seed\n",
        "torch.backends.cudnn.deterministic = True\n",
        "\n",
        "# tokenzier for encoding the text\n",
        "# tokenizer = T5Tokenizer.from_pretrained(\"t5-base\")\n",
        "tokenizer = AutoTokenizer.from_pretrained(\"tuner007/t5_abs_qa\")\n",
        "\n",
        "\n",
        "\n",
        "# Importing and Pre-Processing the domain data\n",
        "# Selecting the needed columns only. \n",
        "# Adding the summarzie text in front of the text. This is to format the dataset similar to how T5 model was trained for summarization task. \n",
        "# df = pd.read_csv('./data/news_summary.csv',encoding='latin-1')\n",
        "# df = df[['text','ctext']]\n",
        "# df.ctext = 'summarize: ' + df.ctext\n",
        "# print(df.head())\n",
        "\n",
        "\n",
        "# Creation of Dataset and Dataloader\n",
        "# Defining the train size. So 90% of the rest of data (10% already taken for test, so 0.81) will be used for training and the rest will be used for validation. \n",
        "train_size = 0.8\n",
        "val_size = 0.1 ### the rest goes to test\n",
        "train_dataset=df.sample(frac=train_size,random_state = config.SEED)\n",
        "# val_dataset=df.drop(train_dataset.index).reset_index(drop=True)\n",
        "val_test_dataset=df.drop(train_dataset.index).reset_index(drop=True)\n",
        "train_dataset = train_dataset.reset_index(drop=True)\n",
        "val_dataset=val_test_dataset.sample(frac=val_size / (1-train_size),random_state = config.SEED)\n",
        "test_dataset=val_test_dataset.drop(val_dataset.index).reset_index(drop=True)\n",
        "val_dataset = val_dataset.reset_index(drop=True)\n",
        "\n",
        "print(\"FULL Dataset: {}\".format(df.shape))\n",
        "print(\"TRAIN Dataset: {}\".format(train_dataset.shape))\n",
        "print(\"TEST Dataset: {}\".format(val_dataset.shape))\n",
        "\n",
        "\n",
        "# Creating the Training and Validation dataset for further creation of Dataloader\n",
        "training_set = CustomDataset(train_dataset, tokenizer, config.MAX_LEN, config.SUMMARY_LEN)\n",
        "val_set = CustomDataset(val_dataset, tokenizer, config.MAX_LEN, config.SUMMARY_LEN)\n",
        "test_set = CustomDataset(test_dataset, tokenizer, config.MAX_LEN, config.SUMMARY_LEN)\n",
        "\n",
        "# Defining the parameters for creation of dataloaders\n",
        "train_params = {\n",
        "    'batch_size': config.TRAIN_BATCH_SIZE,\n",
        "    'shuffle': True,\n",
        "    'num_workers': 0\n",
        "    }\n",
        "\n",
        "val_params = {\n",
        "    'batch_size': config.VALID_BATCH_SIZE,\n",
        "    'shuffle': False,\n",
        "    'num_workers': 0\n",
        "    } ### also used for test set\n",
        "\n",
        "# Creation of Dataloaders for testing and validation. This will be used down for training and validation stage for the model.\n",
        "training_loader = DataLoader(training_set, **train_params)\n",
        "val_loader = DataLoader(val_set, **val_params)\n",
        "test_loader = DataLoader(test_set, **val_params)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295,
          "referenced_widgets": [
            "61b93cde55f54af28c9e85adc0570d43",
            "8127d3d4d61c4407b608bf8c28b873d2",
            "a8b09d03fbe441fdbf641703d0ac0ccb",
            "542d51ab91324ee8b0baf156c52a2842",
            "61c857a1f25943ad92d1d137fff02d80",
            "37661820d4ee4f23a77382cf409e2edb",
            "1ef42899a12d47ffbee2d6358d608264",
            "fe722548b8f84d4f90baa96a4f692e13"
          ]
        },
        "id": "syEVpjvNdIDc",
        "outputId": "48eb7bfd-c673-41a0-ce52-8b4d5cf8d03a"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Finishing last run (ID:3atmqded) before initializing another..."
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "VBox(children=(Label(value='0.001 MB of 0.001 MB uploaded (0.000 MB deduped)\\r'), FloatProgress(value=1.0, max…"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "61b93cde55f54af28c9e85adc0570d43"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Synced <strong style=\"color:#cdcd00\">pecan-square-25</strong>: <a href=\"https://wandb.ai/lbc45123/transformers_tutorials_summarization/runs/3atmqded\" target=\"_blank\">https://wandb.ai/lbc45123/transformers_tutorials_summarization/runs/3atmqded</a><br/>Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Find logs at: <code>./wandb/run-20220314_033809-3atmqded/logs</code>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Successfully finished last run (ID:3atmqded). Initializing new run:<br/>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Tracking run with wandb version 0.12.11"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Run data is saved locally in <code>/content/drive/MyDrive/code/CS224N/wandb/run-20220314_033848-ukh3icfz</code>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Syncing run <strong><a href=\"https://wandb.ai/lbc45123/transformers_tutorials_summarization/runs/ukh3icfz\" target=\"_blank\">blueberry-mousse-26</a></strong> to <a href=\"https://wandb.ai/lbc45123/transformers_tutorials_summarization\" target=\"_blank\">Weights & Biases</a> (<a href=\"https://wandb.me/run\" target=\"_blank\">docs</a>)<br/>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "The `xla_device` argument has been deprecated in v4.4.0 of Transformers. It is ignored and you can safely remove it from your `config.json` file.\n",
            "The `xla_device` argument has been deprecated in v4.4.0 of Transformers. It is ignored and you can safely remove it from your `config.json` file.\n",
            "The `xla_device` argument has been deprecated in v4.4.0 of Transformers. It is ignored and you can safely remove it from your `config.json` file.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "FULL Dataset: (1257, 5)\n",
            "TRAIN Dataset: (1006, 5)\n",
            "TEST Dataset: (126, 5)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "len(train_dataset), len(val_dataset), len(test_dataset)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YV58xAXtCMw5",
        "outputId": "ff756cc2-349e-419f-d757-6759a6113c56"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(1006, 126, 125)"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        " # Defining the model. We are using t5-base model and added a Language model layer on top for generation of Summary. \n",
        "# Further this model is sent to device (GPU/TPU) for using the hardware.\n",
        "# model = T5ForConditionalGeneration.from_pretrained(\"t5-base\")\n",
        "model = AutoModelWithLMHead.from_pretrained(\"tuner007/t5_abs_qa\")\n",
        "model = model.to(device)\n",
        "# model.load_state_dict(torch.load('model_20epochs_state_dict'))\n",
        "\n",
        "# Defining the optimizer that will be used to tune the weights of the network in the training session. \n",
        "optimizer = torch.optim.Adam(params =  model.parameters(), lr=config.LEARNING_RATE)\n",
        "\n",
        "# Log metrics with wandb\n",
        "wandb.watch(model, log=\"all\")\n",
        "# Training loop\n",
        "print('Initiating Fine-Tuning for the model on our dataset')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 372,
          "referenced_widgets": [
            "73797baf2c1840a48647a8a9edfd1fa7",
            "f8ad0ad7b2b9473c80463091a286460d",
            "290a7b5c3d814e23a86913bfec387fa8",
            "48d5643c543241cfb7fe133c6ed316b6",
            "97940b58e151452c86c401ab83584a6b",
            "635e4c298c314881a9259730600a17a9",
            "926c592dc4bb42648a385eae789054f4",
            "8ca58cead9fa44919e6855465780cadb",
            "7505defd1fd64753829a1223c107c530",
            "fa5c0d651d9e425ca4ad157dde500981",
            "b960d07fe7314a27beeab0c2b71158e5",
            "73c2bab7716d4caf937baa176fb2d9f5",
            "3d341817cf034fa9a0aba0eff55482b3",
            "b8bf1f0c8b914fb88bc359edb2b7357a",
            "f88c520b80d248149a7c3c248477d7fa",
            "35688b55437149d29db0763d0a8a92c1",
            "ef8843d3328d4ee6957596953a712f62",
            "ac5ca22dc3e249669f2692f02cacd78b",
            "820911dfa8764d93a17b6f72a6b8b0ee",
            "049c40fdbfc64ae59f6b6add33f06886",
            "57b6c5e71d504157adfc93322fd754f5",
            "36bc524c1916404aa5221610f4c36572"
          ]
        },
        "id": "SYkfkoiKc1YV",
        "outputId": "722cb4df-5be2-4c35-c96f-731668c71e4e"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/transformers/models/auto/modeling_auto.py:882: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.\n",
            "  FutureWarning,\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/1.20k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "73797baf2c1840a48647a8a9edfd1fa7"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "The `xla_device` argument has been deprecated in v4.4.0 of Transformers. It is ignored and you can safely remove it from your `config.json` file.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/1.11G [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "73c2bab7716d4caf937baa176fb2d9f5"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-11-5e75742f231f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;31m# Defining the optimizer that will be used to tune the weights of the network in the training session.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m \u001b[0moptimizer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptim\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mAdam\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparams\u001b[0m \u001b[0;34m=\u001b[0m  \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparameters\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlr\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mconfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mLEARNING_RATE\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      9\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0;31m# Log metrics with wandb\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mNameError\u001b[0m: name 'config' is not defined"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# df\n",
        "predictions, actuals = validate(0, tokenizer, model, device, val_loader)\n",
        "df_val = pd.DataFrame({'Actual Text':actuals, 'Generated Text':predictions})"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kdDWs7QHKtyW",
        "outputId": "2b0b8df5-3f93-4c8f-fa0c-5ee0dad9a19f"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.\n",
            "/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py:2277: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).\n",
            "  FutureWarning,\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Completed 0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# df\n",
        "predictions, actuals = validate(0, tokenizer, model, device, test_loader)\n",
        "df_test = pd.DataFrame({'Actual Text':actuals, 'Generated Text':predictions})"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4k7BfLYZSmfh",
        "outputId": "c18481ff-dfec-4060-bb4d-42ddeadf6511"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py:2277: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).\n",
            "  FutureWarning,\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Completed 0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "NmnUR5guogVQ"
      },
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for epoch in range(config.TRAIN_EPOCHS):\n",
        "# for epoch in range(config.TRAIN_EPOCHS, config.TRAIN_EPOCHS*2):\n",
        "    train(epoch, tokenizer, model, device, training_loader, optimizer)\n",
        "    predictions, actuals = validate(epoch, tokenizer, model, device, val_loader)\n",
        "    df_val[f'Generated Text {epoch+1}'] = predictions\n",
        "    predictions, actuals = validate(epoch, tokenizer, model, device, test_loader)\n",
        "    df_test[f'Generated Text {epoch+1}'] = predictions\n",
        "    # final_df[f'Actual Text {epoch+1}'] = actuals"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kiiaU1WSdK66",
        "outputId": "62d931d7-da4a-41e8-9c51-2c33661cd5da"
      },
      "execution_count": 29,
      "outputs": [
        {
          "metadata": {
            "tags": null
          },
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py:2277: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).\n",
            "  FutureWarning,\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch: 0, Loss:  3.235375165939331\n",
            "Epoch: 0, Loss:  0.9674309492111206\n",
            "Completed 0\n",
            "Completed 0\n",
            "Epoch: 1, Loss:  1.8763514757156372\n",
            "Epoch: 1, Loss:  1.52723228931427\n",
            "Completed 0\n",
            "Completed 0\n",
            "Epoch: 2, Loss:  1.0854079723358154\n",
            "Epoch: 2, Loss:  4.263283729553223\n",
            "Completed 0\n",
            "Completed 0\n",
            "Epoch: 3, Loss:  0.5704552531242371\n",
            "Epoch: 3, Loss:  3.7501766681671143\n",
            "Completed 0\n",
            "Completed 0\n",
            "Epoch: 4, Loss:  2.8043479919433594\n",
            "Epoch: 4, Loss:  2.338731527328491\n",
            "Completed 0\n",
            "Completed 0\n",
            "Epoch: 5, Loss:  2.7318851947784424\n",
            "Epoch: 5, Loss:  0.5311393141746521\n",
            "Completed 0\n",
            "Completed 0\n",
            "Epoch: 6, Loss:  1.9248708486557007\n",
            "Epoch: 6, Loss:  2.9369451999664307\n",
            "Completed 0\n",
            "Completed 0\n",
            "Epoch: 7, Loss:  1.7517833709716797\n",
            "Epoch: 7, Loss:  0.16751804947853088\n",
            "Completed 0\n",
            "Completed 0\n",
            "Epoch: 8, Loss:  2.0895678997039795\n",
            "Epoch: 8, Loss:  1.1681727170944214\n",
            "Completed 0\n",
            "Completed 0\n",
            "Epoch: 9, Loss:  1.821096420288086\n",
            "Epoch: 9, Loss:  1.6945159435272217\n",
            "Completed 0\n",
            "Completed 0\n",
            "Epoch: 10, Loss:  2.5235583782196045\n",
            "Epoch: 10, Loss:  1.3311009407043457\n",
            "Completed 0\n",
            "Completed 0\n",
            "Epoch: 11, Loss:  2.144425630569458\n",
            "Epoch: 11, Loss:  1.2477140426635742\n",
            "Completed 0\n",
            "Completed 0\n",
            "Epoch: 12, Loss:  1.484473466873169\n",
            "Epoch: 12, Loss:  1.052532434463501\n",
            "Completed 0\n",
            "Completed 0\n",
            "Epoch: 13, Loss:  0.5246647000312805\n",
            "Epoch: 13, Loss:  1.1218798160552979\n",
            "Completed 0\n",
            "Completed 0\n",
            "Epoch: 14, Loss:  0.48514416813850403\n",
            "Epoch: 14, Loss:  0.45976853370666504\n",
            "Completed 0\n",
            "Completed 0\n",
            "Epoch: 15, Loss:  0.8130148649215698\n",
            "Epoch: 15, Loss:  0.1019233986735344\n",
            "Completed 0\n",
            "Completed 0\n",
            "Epoch: 16, Loss:  0.04325366020202637\n",
            "Epoch: 16, Loss:  0.44090011715888977\n",
            "Completed 0\n",
            "Completed 0\n",
            "Epoch: 17, Loss:  2.0975959300994873\n",
            "Epoch: 17, Loss:  2.3675730228424072\n",
            "Completed 0\n",
            "Completed 0\n",
            "Epoch: 18, Loss:  1.1528775691986084\n",
            "Epoch: 18, Loss:  1.027637004852295\n",
            "Completed 0\n",
            "Completed 0\n",
            "Epoch: 19, Loss:  3.0244219303131104\n",
            "Epoch: 19, Loss:  1.1618705987930298\n",
            "Completed 0\n",
            "Completed 0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_val"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 922
        },
        "id": "glAp4x_no597",
        "outputId": "a8e57b2a-6b75-4867-bd4a-d676a7165cbc"
      },
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                           Actual Text  \\\n",
              "0                                   Scene in comments.   \n",
              "1                              Cardiopulmonary arrest.   \n",
              "2                                              Cheers.   \n",
              "3          $120. $70 for the game, $50 for all the DLC   \n",
              "4                             Picture in the comments.   \n",
              "..                                                 ...   \n",
              "121                      The CDC doesn't recommend it.   \n",
              "122  So the crew can assess the surroundings in an ...   \n",
              "123  Industry standards are eliminating/reducing si...   \n",
              "124  Australians aren't actually going anywhere...t...   \n",
              "125  It's an invasive species called giant hogweed....   \n",
              "\n",
              "                                        Generated Text  \\\n",
              "0                     Toy Story 4 was a hit with kids.   \n",
              "1                   She died of cardiopulmonary arrest   \n",
              "2                       No answer available in context   \n",
              "3                       No answer available in context   \n",
              "4                       No answer available in context   \n",
              "..                                                 ...   \n",
              "121                     No answer available in context   \n",
              "122                     No answer available in context   \n",
              "123                     No answer available in context   \n",
              "124  No, the country will jut northwards by 1.8 met...   \n",
              "125  It is a dangerous plant that can cause severe ...   \n",
              "\n",
              "                                      Generated Text 1  \\\n",
              "0                               Toy Story 4 was a hit.   \n",
              "1                  She died of cardiopulmonary arrest.   \n",
              "2                                              It did.   \n",
              "3                                                $120.   \n",
              "4                 Cam Newton wore a gunslinger outfit.   \n",
              "..                                                 ...   \n",
              "121                                                No.   \n",
              "122  They want to protect passengers and crew from ...   \n",
              "123           Corded blinds are dangerous to children.   \n",
              "124  The Australian Plate is moving about 7 centime...   \n",
              "125                           It is a poisonous plant.   \n",
              "\n",
              "                                      Generated Text 2  \\\n",
              "0               Toy Story 4 was released on June 21st.   \n",
              "1                  She died of cardiopulmonary arrest.   \n",
              "2                           \"Cheers\" as your sign-off.   \n",
              "3                                                $120.   \n",
              "4                         He wore a gunslinger outfit.   \n",
              "..                                                 ...   \n",
              "121                                               Yes.   \n",
              "122          They want to protect passengers and crew.   \n",
              "123                                     Corded blinds.   \n",
              "124  The Australian Plate is moving about 7 centime...   \n",
              "125                           It is a poisonous plant.   \n",
              "\n",
              "                                      Generated Text 3  \\\n",
              "0                                    One Million Moms.   \n",
              "1                              Cardiopulmonary arrest.   \n",
              "2                                             \"Cheers\"   \n",
              "3                                                $120.   \n",
              "4                         He wore a gunslinger outfit.   \n",
              "..                                                 ...   \n",
              "121                                               Yes.   \n",
              "122                        They want to ensure safety.   \n",
              "123                                     Corded blinds.   \n",
              "124  The Australian Plate is moving about 7 centime...   \n",
              "125                                 It's a poison oak.   \n",
              "\n",
              "                                      Generated Text 4  \\\n",
              "0                                    One Million Moms.   \n",
              "1                              Cardiopulmonary arrest.   \n",
              "2                           \"Cheers\" as your sign-off.   \n",
              "3                                                $120.   \n",
              "4                         He wore a gunslinger outfit.   \n",
              "..                                                 ...   \n",
              "121                                               Yes.   \n",
              "122                        They want to ensure safety.   \n",
              "123                                     Corded blinds.   \n",
              "124  The Australian Plate is moving about 7 centime...   \n",
              "125                                 It's a poison oak.   \n",
              "\n",
              "                                      Generated Text 5  \\\n",
              "0                                    One Million Moms.   \n",
              "1                              Cardiopulmonary arrest.   \n",
              "2                                            \"Cheers.\"   \n",
              "3                                                $120.   \n",
              "4                         He wore a gunslinger outfit.   \n",
              "..                                                 ...   \n",
              "121                                               Yes.   \n",
              "122                        They want to ensure safety.   \n",
              "123                                     Corded blinds.   \n",
              "124  The Australian Plate is moving about 7 centime...   \n",
              "125                                 It's a poison oak.   \n",
              "\n",
              "                                      Generated Text 6  \\\n",
              "0                                    One Million Moms.   \n",
              "1                              Cardiopulmonary arrest.   \n",
              "2                                            \"Cheers.\"   \n",
              "3                                                $120.   \n",
              "4                         He wore a gunslinger outfit.   \n",
              "..                                                 ...   \n",
              "121                                               Yes.   \n",
              "122                                   It's for safety.   \n",
              "123                                     Corded blinds.   \n",
              "124  The Australian Plate is moving about 7 centime...   \n",
              "125                                 It's a poison oak.   \n",
              "\n",
              "                                      Generated Text 7  \\\n",
              "0                                    One Million Moms.   \n",
              "1                              Cardiopulmonary arrest.   \n",
              "2                                            \"Cheers.\"   \n",
              "3                                                $120.   \n",
              "4                         He wore a gunslinger outfit.   \n",
              "..                                                 ...   \n",
              "121                                               Yes.   \n",
              "122                                  To ensure safety.   \n",
              "123                                     Corded blinds.   \n",
              "124  The Australian Plate is moving about 7 centime...   \n",
              "125                                 It's a poison oak.   \n",
              "\n",
              "                                      Generated Text 8  ...  \\\n",
              "0                                    One Million Moms.  ...   \n",
              "1                              Cardiopulmonary arrest.  ...   \n",
              "2                                            \"Cheers.\"  ...   \n",
              "3                                                $120.  ...   \n",
              "4                         He wore a gunslinger outfit.  ...   \n",
              "..                                                 ...  ...   \n",
              "121                                               Yes.  ...   \n",
              "122                                  To ensure safety.  ...   \n",
              "123                                     Corded blinds.  ...   \n",
              "124  The Australian Plate is moving about 7 centime...  ...   \n",
              "125                                 It's a poison oak.  ...   \n",
              "\n",
              "                                     Generated Text 11  \\\n",
              "0                                    One Million Moms.   \n",
              "1                              Cardiopulmonary arrest.   \n",
              "2                                            \"Cheers.\"   \n",
              "3                                                $120.   \n",
              "4                         He wore a gunslinger outfit.   \n",
              "..                                                 ...   \n",
              "121                                               Yes.   \n",
              "122                                  To ensure safety.   \n",
              "123                    To prevent child strangulation.   \n",
              "124  The Australian Plate is moving about 7 centime...   \n",
              "125                                 It's a poison oak.   \n",
              "\n",
              "                                     Generated Text 12  \\\n",
              "0                                    One Million Moms.   \n",
              "1                              Cardiopulmonary arrest.   \n",
              "2                                            \"Cheers.\"   \n",
              "3                                                $120.   \n",
              "4                         He wore a gunslinger outfit.   \n",
              "..                                                 ...   \n",
              "121                                               Yes.   \n",
              "122                                  To ensure safety.   \n",
              "123                    To prevent child strangulation.   \n",
              "124  The Australian Plate is moving about 7 centime...   \n",
              "125                             Scratching against it.   \n",
              "\n",
              "                   Generated Text 13  \\\n",
              "0                  One Million Moms.   \n",
              "1            Cardiopulmonary arrest.   \n",
              "2                          \"Cheers.\"   \n",
              "3                              $120.   \n",
              "4       He wore a gunslinger outfit.   \n",
              "..                               ...   \n",
              "121                             Yes.   \n",
              "122                To ensure safety.   \n",
              "123  To prevent child strangulation.   \n",
              "124            1.8 meters (5.9 feet)   \n",
              "125               It's a poison oak.   \n",
              "\n",
              "                                     Generated Text 14  \\\n",
              "0                                    One Million Moms.   \n",
              "1                              Cardiopulmonary arrest.   \n",
              "2                                              Cheers.   \n",
              "3                                                $120.   \n",
              "4                         He wore a gunslinger outfit.   \n",
              "..                                                 ...   \n",
              "121                                               Yes.   \n",
              "122                                  To ensure safety.   \n",
              "123                    To prevent child strangulation.   \n",
              "124  The Australian Plate is moving about 7 centime...   \n",
              "125          Scratching against it can cause blisters.   \n",
              "\n",
              "                                     Generated Text 15  \\\n",
              "0                                    One Million Moms.   \n",
              "1                              Cardiopulmonary arrest.   \n",
              "2                                              Cheers.   \n",
              "3                                                $120.   \n",
              "4                         He wore a gunslinger outfit.   \n",
              "..                                                 ...   \n",
              "121                                               Yes.   \n",
              "122                                  To ensure safety.   \n",
              "123                    To prevent child strangulation.   \n",
              "124  The Australian Plate is moving about 7 centime...   \n",
              "125     It contains toxic sap that can cause blisters.   \n",
              "\n",
              "                                     Generated Text 16  \\\n",
              "0                                    One Million Moms.   \n",
              "1                              Cardiopulmonary arrest.   \n",
              "2                                              Cheers.   \n",
              "3                                                $120.   \n",
              "4                         He wore a gunslinger outfit.   \n",
              "..                                                 ...   \n",
              "121                                               Yes.   \n",
              "122                                  To ensure safety.   \n",
              "123                    To prevent child strangulation.   \n",
              "124  The Australian Plate is moving about 7 centime...   \n",
              "125     It contains toxic sap that can cause blisters.   \n",
              "\n",
              "                                     Generated Text 17  \\\n",
              "0                                    One Million Moms.   \n",
              "1                              Cardiopulmonary arrest.   \n",
              "2                                              Cheers.   \n",
              "3                                                $120.   \n",
              "4    Patriots quarterback Cam Newton wore a gunslin...   \n",
              "..                                                 ...   \n",
              "121                                               Yes.   \n",
              "122                                  To ensure safety.   \n",
              "123                    To prevent child strangulation.   \n",
              "124  The Australian Plate is moving about 7 centime...   \n",
              "125                      Scratches from the toxic sap.   \n",
              "\n",
              "                                     Generated Text 18  \\\n",
              "0                                    One Million Moms.   \n",
              "1                              Cardiopulmonary arrest.   \n",
              "2                                              Cheers.   \n",
              "3                                                $120.   \n",
              "4    Patriots quarterback Cam Newton wore a gunslin...   \n",
              "..                                                 ...   \n",
              "121                                               Yes.   \n",
              "122                                  To ensure safety.   \n",
              "123                    To prevent child strangulation.   \n",
              "124                              1.8 meters (5.9 feet)   \n",
              "125     It contains toxic sap that can cause blisters.   \n",
              "\n",
              "                                     Generated Text 19  \\\n",
              "0                                    One Million Moms.   \n",
              "1                              Cardiopulmonary arrest.   \n",
              "2                                              Cheers.   \n",
              "3                                                $120.   \n",
              "4                         He wore a gunslinger outfit.   \n",
              "..                                                 ...   \n",
              "121                                               Yes.   \n",
              "122                                  To ensure safety.   \n",
              "123                    To prevent child strangulation.   \n",
              "124  1.8 meters (5.9 feet) northwards on New Year's...   \n",
              "125     It contains toxic sap that can cause blisters.   \n",
              "\n",
              "                                     Generated Text 20  \n",
              "0                                    One Million Moms.  \n",
              "1                              Cardiopulmonary arrest.  \n",
              "2                                              Cheers.  \n",
              "3                                                $120.  \n",
              "4    Patriots quarterback Cam Newton wore a gunslin...  \n",
              "..                                                 ...  \n",
              "121                                               Yes.  \n",
              "122                                  To ensure safety.  \n",
              "123                    To prevent child strangulation.  \n",
              "124  The Australian Plate is moving about 7 centime...  \n",
              "125     It contains toxic sap that can cause blisters.  \n",
              "\n",
              "[126 rows x 22 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-2e1bc774-f9bd-43dd-81a2-aaeb8ff89a3f\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Actual Text</th>\n",
              "      <th>Generated Text</th>\n",
              "      <th>Generated Text 1</th>\n",
              "      <th>Generated Text 2</th>\n",
              "      <th>Generated Text 3</th>\n",
              "      <th>Generated Text 4</th>\n",
              "      <th>Generated Text 5</th>\n",
              "      <th>Generated Text 6</th>\n",
              "      <th>Generated Text 7</th>\n",
              "      <th>Generated Text 8</th>\n",
              "      <th>...</th>\n",
              "      <th>Generated Text 11</th>\n",
              "      <th>Generated Text 12</th>\n",
              "      <th>Generated Text 13</th>\n",
              "      <th>Generated Text 14</th>\n",
              "      <th>Generated Text 15</th>\n",
              "      <th>Generated Text 16</th>\n",
              "      <th>Generated Text 17</th>\n",
              "      <th>Generated Text 18</th>\n",
              "      <th>Generated Text 19</th>\n",
              "      <th>Generated Text 20</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Scene in comments.</td>\n",
              "      <td>Toy Story 4 was a hit with kids.</td>\n",
              "      <td>Toy Story 4 was a hit.</td>\n",
              "      <td>Toy Story 4 was released on June 21st.</td>\n",
              "      <td>One Million Moms.</td>\n",
              "      <td>One Million Moms.</td>\n",
              "      <td>One Million Moms.</td>\n",
              "      <td>One Million Moms.</td>\n",
              "      <td>One Million Moms.</td>\n",
              "      <td>One Million Moms.</td>\n",
              "      <td>...</td>\n",
              "      <td>One Million Moms.</td>\n",
              "      <td>One Million Moms.</td>\n",
              "      <td>One Million Moms.</td>\n",
              "      <td>One Million Moms.</td>\n",
              "      <td>One Million Moms.</td>\n",
              "      <td>One Million Moms.</td>\n",
              "      <td>One Million Moms.</td>\n",
              "      <td>One Million Moms.</td>\n",
              "      <td>One Million Moms.</td>\n",
              "      <td>One Million Moms.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Cardiopulmonary arrest.</td>\n",
              "      <td>She died of cardiopulmonary arrest</td>\n",
              "      <td>She died of cardiopulmonary arrest.</td>\n",
              "      <td>She died of cardiopulmonary arrest.</td>\n",
              "      <td>Cardiopulmonary arrest.</td>\n",
              "      <td>Cardiopulmonary arrest.</td>\n",
              "      <td>Cardiopulmonary arrest.</td>\n",
              "      <td>Cardiopulmonary arrest.</td>\n",
              "      <td>Cardiopulmonary arrest.</td>\n",
              "      <td>Cardiopulmonary arrest.</td>\n",
              "      <td>...</td>\n",
              "      <td>Cardiopulmonary arrest.</td>\n",
              "      <td>Cardiopulmonary arrest.</td>\n",
              "      <td>Cardiopulmonary arrest.</td>\n",
              "      <td>Cardiopulmonary arrest.</td>\n",
              "      <td>Cardiopulmonary arrest.</td>\n",
              "      <td>Cardiopulmonary arrest.</td>\n",
              "      <td>Cardiopulmonary arrest.</td>\n",
              "      <td>Cardiopulmonary arrest.</td>\n",
              "      <td>Cardiopulmonary arrest.</td>\n",
              "      <td>Cardiopulmonary arrest.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Cheers.</td>\n",
              "      <td>No answer available in context</td>\n",
              "      <td>It did.</td>\n",
              "      <td>\"Cheers\" as your sign-off.</td>\n",
              "      <td>\"Cheers\"</td>\n",
              "      <td>\"Cheers\" as your sign-off.</td>\n",
              "      <td>\"Cheers.\"</td>\n",
              "      <td>\"Cheers.\"</td>\n",
              "      <td>\"Cheers.\"</td>\n",
              "      <td>\"Cheers.\"</td>\n",
              "      <td>...</td>\n",
              "      <td>\"Cheers.\"</td>\n",
              "      <td>\"Cheers.\"</td>\n",
              "      <td>\"Cheers.\"</td>\n",
              "      <td>Cheers.</td>\n",
              "      <td>Cheers.</td>\n",
              "      <td>Cheers.</td>\n",
              "      <td>Cheers.</td>\n",
              "      <td>Cheers.</td>\n",
              "      <td>Cheers.</td>\n",
              "      <td>Cheers.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>$120. $70 for the game, $50 for all the DLC</td>\n",
              "      <td>No answer available in context</td>\n",
              "      <td>$120.</td>\n",
              "      <td>$120.</td>\n",
              "      <td>$120.</td>\n",
              "      <td>$120.</td>\n",
              "      <td>$120.</td>\n",
              "      <td>$120.</td>\n",
              "      <td>$120.</td>\n",
              "      <td>$120.</td>\n",
              "      <td>...</td>\n",
              "      <td>$120.</td>\n",
              "      <td>$120.</td>\n",
              "      <td>$120.</td>\n",
              "      <td>$120.</td>\n",
              "      <td>$120.</td>\n",
              "      <td>$120.</td>\n",
              "      <td>$120.</td>\n",
              "      <td>$120.</td>\n",
              "      <td>$120.</td>\n",
              "      <td>$120.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Picture in the comments.</td>\n",
              "      <td>No answer available in context</td>\n",
              "      <td>Cam Newton wore a gunslinger outfit.</td>\n",
              "      <td>He wore a gunslinger outfit.</td>\n",
              "      <td>He wore a gunslinger outfit.</td>\n",
              "      <td>He wore a gunslinger outfit.</td>\n",
              "      <td>He wore a gunslinger outfit.</td>\n",
              "      <td>He wore a gunslinger outfit.</td>\n",
              "      <td>He wore a gunslinger outfit.</td>\n",
              "      <td>He wore a gunslinger outfit.</td>\n",
              "      <td>...</td>\n",
              "      <td>He wore a gunslinger outfit.</td>\n",
              "      <td>He wore a gunslinger outfit.</td>\n",
              "      <td>He wore a gunslinger outfit.</td>\n",
              "      <td>He wore a gunslinger outfit.</td>\n",
              "      <td>He wore a gunslinger outfit.</td>\n",
              "      <td>He wore a gunslinger outfit.</td>\n",
              "      <td>Patriots quarterback Cam Newton wore a gunslin...</td>\n",
              "      <td>Patriots quarterback Cam Newton wore a gunslin...</td>\n",
              "      <td>He wore a gunslinger outfit.</td>\n",
              "      <td>Patriots quarterback Cam Newton wore a gunslin...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>121</th>\n",
              "      <td>The CDC doesn't recommend it.</td>\n",
              "      <td>No answer available in context</td>\n",
              "      <td>No.</td>\n",
              "      <td>Yes.</td>\n",
              "      <td>Yes.</td>\n",
              "      <td>Yes.</td>\n",
              "      <td>Yes.</td>\n",
              "      <td>Yes.</td>\n",
              "      <td>Yes.</td>\n",
              "      <td>Yes.</td>\n",
              "      <td>...</td>\n",
              "      <td>Yes.</td>\n",
              "      <td>Yes.</td>\n",
              "      <td>Yes.</td>\n",
              "      <td>Yes.</td>\n",
              "      <td>Yes.</td>\n",
              "      <td>Yes.</td>\n",
              "      <td>Yes.</td>\n",
              "      <td>Yes.</td>\n",
              "      <td>Yes.</td>\n",
              "      <td>Yes.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>122</th>\n",
              "      <td>So the crew can assess the surroundings in an ...</td>\n",
              "      <td>No answer available in context</td>\n",
              "      <td>They want to protect passengers and crew from ...</td>\n",
              "      <td>They want to protect passengers and crew.</td>\n",
              "      <td>They want to ensure safety.</td>\n",
              "      <td>They want to ensure safety.</td>\n",
              "      <td>They want to ensure safety.</td>\n",
              "      <td>It's for safety.</td>\n",
              "      <td>To ensure safety.</td>\n",
              "      <td>To ensure safety.</td>\n",
              "      <td>...</td>\n",
              "      <td>To ensure safety.</td>\n",
              "      <td>To ensure safety.</td>\n",
              "      <td>To ensure safety.</td>\n",
              "      <td>To ensure safety.</td>\n",
              "      <td>To ensure safety.</td>\n",
              "      <td>To ensure safety.</td>\n",
              "      <td>To ensure safety.</td>\n",
              "      <td>To ensure safety.</td>\n",
              "      <td>To ensure safety.</td>\n",
              "      <td>To ensure safety.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>123</th>\n",
              "      <td>Industry standards are eliminating/reducing si...</td>\n",
              "      <td>No answer available in context</td>\n",
              "      <td>Corded blinds are dangerous to children.</td>\n",
              "      <td>Corded blinds.</td>\n",
              "      <td>Corded blinds.</td>\n",
              "      <td>Corded blinds.</td>\n",
              "      <td>Corded blinds.</td>\n",
              "      <td>Corded blinds.</td>\n",
              "      <td>Corded blinds.</td>\n",
              "      <td>Corded blinds.</td>\n",
              "      <td>...</td>\n",
              "      <td>To prevent child strangulation.</td>\n",
              "      <td>To prevent child strangulation.</td>\n",
              "      <td>To prevent child strangulation.</td>\n",
              "      <td>To prevent child strangulation.</td>\n",
              "      <td>To prevent child strangulation.</td>\n",
              "      <td>To prevent child strangulation.</td>\n",
              "      <td>To prevent child strangulation.</td>\n",
              "      <td>To prevent child strangulation.</td>\n",
              "      <td>To prevent child strangulation.</td>\n",
              "      <td>To prevent child strangulation.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>124</th>\n",
              "      <td>Australians aren't actually going anywhere...t...</td>\n",
              "      <td>No, the country will jut northwards by 1.8 met...</td>\n",
              "      <td>The Australian Plate is moving about 7 centime...</td>\n",
              "      <td>The Australian Plate is moving about 7 centime...</td>\n",
              "      <td>The Australian Plate is moving about 7 centime...</td>\n",
              "      <td>The Australian Plate is moving about 7 centime...</td>\n",
              "      <td>The Australian Plate is moving about 7 centime...</td>\n",
              "      <td>The Australian Plate is moving about 7 centime...</td>\n",
              "      <td>The Australian Plate is moving about 7 centime...</td>\n",
              "      <td>The Australian Plate is moving about 7 centime...</td>\n",
              "      <td>...</td>\n",
              "      <td>The Australian Plate is moving about 7 centime...</td>\n",
              "      <td>The Australian Plate is moving about 7 centime...</td>\n",
              "      <td>1.8 meters (5.9 feet)</td>\n",
              "      <td>The Australian Plate is moving about 7 centime...</td>\n",
              "      <td>The Australian Plate is moving about 7 centime...</td>\n",
              "      <td>The Australian Plate is moving about 7 centime...</td>\n",
              "      <td>The Australian Plate is moving about 7 centime...</td>\n",
              "      <td>1.8 meters (5.9 feet)</td>\n",
              "      <td>1.8 meters (5.9 feet) northwards on New Year's...</td>\n",
              "      <td>The Australian Plate is moving about 7 centime...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>125</th>\n",
              "      <td>It's an invasive species called giant hogweed....</td>\n",
              "      <td>It is a dangerous plant that can cause severe ...</td>\n",
              "      <td>It is a poisonous plant.</td>\n",
              "      <td>It is a poisonous plant.</td>\n",
              "      <td>It's a poison oak.</td>\n",
              "      <td>It's a poison oak.</td>\n",
              "      <td>It's a poison oak.</td>\n",
              "      <td>It's a poison oak.</td>\n",
              "      <td>It's a poison oak.</td>\n",
              "      <td>It's a poison oak.</td>\n",
              "      <td>...</td>\n",
              "      <td>It's a poison oak.</td>\n",
              "      <td>Scratching against it.</td>\n",
              "      <td>It's a poison oak.</td>\n",
              "      <td>Scratching against it can cause blisters.</td>\n",
              "      <td>It contains toxic sap that can cause blisters.</td>\n",
              "      <td>It contains toxic sap that can cause blisters.</td>\n",
              "      <td>Scratches from the toxic sap.</td>\n",
              "      <td>It contains toxic sap that can cause blisters.</td>\n",
              "      <td>It contains toxic sap that can cause blisters.</td>\n",
              "      <td>It contains toxic sap that can cause blisters.</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>126 rows × 22 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2e1bc774-f9bd-43dd-81a2-aaeb8ff89a3f')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-2e1bc774-f9bd-43dd-81a2-aaeb8ff89a3f button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-2e1bc774-f9bd-43dd-81a2-aaeb8ff89a3f');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df_val.to_csv('./df_val.csv')\n",
        "df_test.to_csv('./df_test.csv')"
      ],
      "metadata": {
        "id": "WKbFO3aFrF5c"
      },
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# wandb.finish()"
      ],
      "metadata": {
        "id": "-xfVNTarDDto"
      },
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "len(df_val)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TEKolI1tqK-9",
        "outputId": "ddf4c3ed-0f62-45ec-8e3c-bd7ed00449ec"
      },
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "126"
            ]
          },
          "metadata": {},
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "torch.save(model.state_dict(),'model_20epochs_state_dict')"
      ],
      "metadata": {
        "id": "t3T5AbYRrpTb"
      },
      "execution_count": 53,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# torch.load_state_dict('model_20epochs_state_dict')\n",
        "model.load_state_dict(torch.load('model_20epochs_state_dict'))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GYkbZmm3S4W6",
        "outputId": "9f157d6a-3c2d-42b4-c61b-8743fb3b7498"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<All keys matched successfully>"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "SUUXcqrFS8nA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tmp = df_val\n",
        "P, R, F1 = score([str(i) for i in tmp['Actual Text'].tolist()], [str(i) for i in tmp['Generated Text'].tolist()], lang='en')\n",
        "f1scores_val = F1.tolist()\n",
        "epochs_val = [0]*len(F1)\n",
        "avgs_val = [np.average(f1scores)]\n",
        "for epoch in range(1,21):\n",
        "  P, R, F1 = score([str(i) for i in tmp['Actual Text'].tolist()], [str(i) for i in tmp[f'Generated Text {epoch}'].tolist()], lang='en')\n",
        "  f1scores_val += F1.tolist()\n",
        "  epochs_val += [epoch]*len(F1)\n",
        "  avgs_val.append(np.average(F1.tolist()))\n",
        "# sns.lineplot(x=epochs, y=f1scores, ci='sd')\n",
        "sns.lineplot(x=epochs_val, y=f1scores_val, ci=95)\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('F1_Bert')\n",
        "plt.savefig('validation_score.png')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "Gu3MWkl7D_eW",
        "outputId": "73b6d3e9-e48f-45e2-df21-ceea203d06bd"
      },
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEGCAYAAACQO2mwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5hkV3ng/+9bOXQOM9MTeoI0QjNKSIyFQASBEJZkGy322kY2XpKN7TWOrG28ZvnxYP+cdtePjQ3eH8YEY5tgr4MeW0ISIBAgAQoohwmaPNMznbsr3/D+/ri3e6pDzXRPV1Wn9/M8/fStW7eqTldXnfeec957jqgqxhhjzHwiy10AY4wxK5cFCWOMMTVZkDDGGFOTBQljjDE1WZAwxhhTU2y5C1BPPT09umPHjuUuhjHGrCqPPfbYkKr2znffmgoSO3bs4NFHH13uYhhjzKoiIkdr3WfdTcYYY2qyIGGMMaYmCxLGGGNqsiBhjDGmJgsSxhhjarIgYYwxpiYLEsYYY2qyIGGMMaamNXUxnTGmfooVj1NjRQqOh0DwE24EvwQRCfaF9zN1TPi7O5ukI5NoarlLjsfp8SItyTgtyRipeASZKpRZNAsSxpgZ8mWX46MFTo+ViEWFRHRmh0P1OmU6a0PDDdVg15GhArt6s2zvyhKJNL6iHs1XePbUOJ4PPnkiCLFohO5sgu6WpAWNi2BBwhgDwGTJ4fhIkTMTReLRKN3ZxJIrU19jHB0uMF50uXxTK6l4tE6lnfU6vnJ8pMChoTxtqRjJ2LnX8XxlNO8wMFFCoO5BQ1VxfcXzFcfzScQiM15/tbMgYcw6N1FyODqUZyhXIRGN0J1N1u1MOyJCdzbJRNHhkSMjXLm5nc5sfbufSo7HiwOTjOQrdGcTRGaVPRoRWlIxWsLqrjpoAMRrBA3X83H9IAC4no/jKY7rU3K94MfxqTgeFc/H16C7TQlaUW3pOH1tKTqycTKJ1V3Nru7SG7OCVVyfYsUjX3YZLVSIRYX+rizpxPKfZaoqE0WXw0M5RgoV0rFYXVoOtbSl45Rdj8ePjda1+2msUOGZU+MIQk9LckGPmR00XM+fETRiEcFXDSt+QdHpABBBiEbO/SRjUTKJ2Jz3reR47D87CUAqHqWvLUVnS4KWRKwp3W71ZEHCmDpwPJ+i41Eou4wVHEYLFcquDwRn08lYBNdXTo2V6O/KsLUrvSxdEqrKWMHh8HCesYJDJh6ltyXVlNdOxqL0tETq0v3k+8qJ0QIHz+ZoTcWX1I0Vi0ZoiUZmtDREmNMiWYxUPDpdJsfzOTpS4PBwnlhE2NCWorclSWsqRiy68hNMGx4kRORW4M+BKPBJVf2jWfdvBz4F9AIjwNtV9UR43zuAD4aH/r6qfrbR5TXmQtwwIBQdj7GCw1i+Qj7MAIKgMkzForQk43Me66tycqzIqbEiO3qy9LWnmlJR+L4yWqhweDDPZNkhk4jRu8Az73qq7n569MgIV1xE91PZ9dh/ZpKhXJmubJJonc/M6/188WiEzjDDy/OVwYkyp8aK09lfG9tStKVjK3YcQ7Q6VaHeTy4SBfYDtwAngEeAO1X1uapj/hH4d1X9rIi8EXiXqv6MiHQBjwL7CFp6jwGvUNXRWq+3b98+tfUk1rahyRIHz+ZJJyK0pxO0pmKkE0Gl3IhmfDnsey5VXCZKLuNFh1zZnb4/EY2QikeJL7Ki93xlvOgQiwqX9mbpbU01pPwV12esUOHwUJ6C49GSiDVs8Hixyq7HeNFZVPfTeMHhmVPjqCrt6eam1tabr0qxEpxsQNAl151NEI8K8ViEqAixSIRIhHNdXBL8rne3oIg8pqr75ruv0S2J64GDqvpSWJAvAHcAz1Udsxf4jXD7AeBfw+0fBO5X1ZHwsfcDtwKfb3CZzQpUcX1eGspxaqxIazJO2VGOFwt4qmH+vpBNxGjPxGhPJ0jFI6Tj0QWfpTueT8kJAkKu7DBRdJksObj+uf7oRDRCIhahK7P0vvtoROjKJnA8n+cGJsgMF9i9sZXOTHzJz11yPCaKDqfHi4wWHACyiRg92ea3HM5nMd1PqsqJ0SIHzk7Smlxa99JKEREhm4yRTQbVcMnxODFaxPcVP0wlnvrsTX0ipk7p49EI8UgQTBLRCPFYhI50nA1t9e86bHSQ2AIcr7p9AnjlrGOeBH6UoEvqrUCriHTXeOyW2S8gIu8F3gvQ399ft4KblWM0X+G50xN4vtJTlXlTPQCsqlQ8nzPjZU6MFqf3Z+JROrIJ2lIx0okgc8XzlZLjky+7TBQdJkoOJdef/iLGIsEXryUZr3vXw2zxaISebIqS4/Hk8VHaMwku6W2hPT23q+p8ChWXsbzD6YkSk6UgMKTj0boEtEZaSPdTxfU5cGaSs5NlujL1715aKarHMS7E8xVfFddTKq5LOe+TK7urMkgsxH8D/lJE3gk8CJwEvIU+WFU/AXwCgu6mRhTQLA/X8zkylOfYaCE4e0zV/gKJBJkms/t1Hc9neLIS9AFz7qxMOTegXGv8oJmmKohCxeWxoyNsaE2ys6dl+ixzNlUlV3YZzVc4PV6a7rLIJmJ0r7AWw0JMZT99//gYu3oy9IfdT+NFJ7w4ThecvbQeRCNCFGEqpkRkwVXmojU6SJwEtlXd3hrum6aqpwhaEohIC/BjqjomIieBm2Y99uuNLKxZOcaLDs+fnqDsejNaD4sVj0aIV2WurHSZRIx0PMp4weV7h0fY2plmW1eGVDyK7yuTJZehXJmBiRKO6xONCJlEjO7s6vj7zicZi9KdjXB4qMBY0aU7m+Dg2RwtyRitydXfvbRaNfqT9QiwW0R2EgSHtwE/VX2AiPQAI6rqA79DkOkEcC/wByLSGd5+c3i/WcM8Xzk2kufwYJ6WZJyuzPo7exQR2tJxfFUGJkqcHCvS25pkNF/B9ZVYJEI2ESWWWt7WTyNEJLjeYaLkMJqv0JlJrNnupdWioUFCVV0ReR9BhR8FPqWqz4rIR4BHVfUugtbCH4qIEnQ3/VL42BER+T2CQAPwkalBbLM25couL5yaIFdx6W5JLilPfS2IiNCRTuD5wYVvzRgjWSna1mAAXK0amgLbbJYCuzr5fnDtwKHBHOl4dNVPY2BMs5Ucj2hUuK6/88IHz2M5U2CNOa9CxeXFgUnGCo51LRizAlmQMMtCVRkYL7H/bI5EJGKZK8asUBYkTNPlyy6HBnMM5Sp0puOrYv4aY9YrCxKmaQoVl2MjwWI2yVhkWeYOMsYsjgUJ03DFisexkTynx0vEIpGGTkltjKkvCxKmYUqOx/GRAidGi8SjsuKniDDGzGVBYp2aWo1MROhtSdCajpOOR+tSiU9NVHZitDA9kd16v+bBmNXKgsQ6ky+7HB3OMzBRJh2PEhEYzpUBSMQi9LYm6coGyzgmYosbUC67HidHixwbKRAVoTNjwcGY1c6CxDpRcoJxgZOjJRLRCD1V4wJTF6+5VbOoKtCajIULosRpScZqXsNQcX1OjRU5Ohy0TDrSdr2DMWuFBYk1rux6YQVeIHaBrp9YNEJb+lzroeR4HB7K46sSkeCx1V1TjqcMjBc5MpxHFdotOBiz5liQWKMcz2dgvMSRsJK/mLP76vntfVXy5WAGUhTisUiwWLyvFhyMWcMsSKwxnq+cnShxaCiH611ccJhPRIIpqau7piRcStEYs3ZZkFgjfF8ZypU5NJij7Pq0peLEU427ktmukjZmfbAgscqpKqMFh0NnJ8mVPVpTsWVfZc0Ys3ZYkFjFXM/nudMTDOfKtCTjNkmeMSFVtQs368SCxCqlqhw4m2M0X6Gnpf6LnxuzGh0ZyvMn971IyfH4wb0buWXvJrqyieUu1qpmQWKVOjZcYGC8RLd9AcwyKzkeI/kKo4VK1W+HsuvxI1dvZnNHuinl+OaBQf78qwfIJKJs787yd989xj987xg37Ormtiv7uHpru13ceREsSKxCU9lL3dmkNalNw/iqnB4rMZIvM1xV+U8Fg6mAUKh4cx4biwgi8JXnz/Czr9nFm/dubNhn1fOVv334CP/8/ZPs2dTKB27bQ1c2wamxIvc+O8D9z5/hoUPD9LWnuPWKTdy8ZyPtaRu3WyhbvnSVGS86PH50lPZ0nLhlGJkGODtZ4qvPn+Urz5/h7GR5xn2JaITObJyubJKuTJzObIKubIKuTCLYzgS3W1MxhvMV/uwr+3nyxDiv3NnF+95wKR2Z+rZ8J4oO//O+F3ni+Bi3XbmJn3vtrjnfi4rr89ChIe55ZoDnTk8QiwivubSHW6/cxN6+troEL1+VwckykyWXnT3ZpqeGN3L5UgsSq0ix4vHY0RGSsXMXuRlTD47n893DI9z/3ADfPzaGAtdsbee1u3vZ1JaiKxsEgWxicZNA+qrc9eQp/vbhI2QTMX7l5t38wI6uupT50GCOP7j7eUbyFf7rTZdwy95NF3zM0eE8X35mgK+9eJZCxaO/K8NtV27iDS/bQDZ54Y6VkhPMYDA1geWJcPvkaJGK5wPQlUnw2t09vO6yXnZvaGlKa9+CxAKt5SDheD5PHBvF9aFlAR9ms3KoKiXHJ51YeYH9yFCe+58/wwMvnmWy5NLTkuDmPRt5056NbGqrX0LE0eE8/+u+FzkyXOC2Kzfx7ht3LulE54EXz/KXXztIWzrG79y2h8s2ti7q8SXH48EDg9zz9AAHB3MkYxFef1kvt13ZxyW9WcYKzowgcGI0mPK+umUlwMa2FFs70+FPhkQswkOHhnj0yCiur/S1p3jd7l5ed1kv/V2Zi/57F/L3WJBYgLUaJHxfefb0OGN5p+7NdVMfXngx48B4idPjJQYmiuHvEgPjJQoVj90bWrj9qj5eu7uHZGz5Aka+7PLggUHuf+4MB87miEWEV+7q5s17NnLNto6GdZU4ns/nvnOUf/3+STZ3pPmNWy5bdOXuej6ffugIdz15iis3t/Hbt16+5O/EgTOT3PPsAA/uH6Ts+qTiEUqOP31/Kh5ha0eGLVXBYGtHms0d6ZozJefKLg8fGuLBA0M8dWIMX2FnTzYIGLt72LDEAKyqnJ0sc3goz5HhPAfP5ri2v4MP3Lbnop7PgsQqd/DMJMdHi+v+OojxosM/PXacwckymzvSbG5P09eRYnNHmo50vOHN+pLjBUFgosSZ8PfAeBAMzk6W8fxz36VYRNjYlmJjW4q+9hStqRjfPjjE8dEiLckYb9qzgduu7Gta5o+q8uypCe5/7gzfOjRExfXZ3pXhlr0buellG5o6kPv0iTH+9CsHGMmXufP6fn78FdsWFJjGChX++Msv8MypCd5yzWbe9eoddb3yP1d2+fqLZzk5WgwDQoatneklr6Q4mq/wzYNDPLh/kBfPTAKwZ1Mrr7+slxsv7blgkCs5HkeG8xwZKnB4OM+RMDBMJQxMtWhuvXITH37LFRdVRgsSq9iJ0QIHzkyu60ymiuvzb0+e5B8fPUHZ9djYlppTKWcSUfraU2zpSNMXBpDNHSk2t6dpO08FqKoUp1I48xVGCk74uzLnd748M4snm4iyqT3FpvY0fW2pcDtFX1uK7pbknIpPVXnm5Dj/8cwA33lpGM9Xrt3Wwe1X9fEDO7rqfgavqpweL/Htg0Pc//wZTo+XyCSivG53L7fs3di0/vL55Mou/+cbh/jG/kEu39TKb9xyGX3ttQPm/jOT/OE9zzNRcnnfGy7lDS/b0MTS1s/ARIlv7h/kG/sHOTpSICJwzdYOXndZLzfs6iZfdqdbB4eHgoBwerzE1Cc9HY+yoyfLzp4sO7oz7OzJsr0riwjW3bQQay1IDOfKPHl8jK7s3ApnPfBVeXD/IH/7naMMTpa5fkcX77xxB9s6M8FEhpMlTo2VODlW5PRYkVPjRU6NlTg7WaIqftCSjIXBI0VrMsZoYWYaZ9n157x2PBosmtSVTUz/7som2BQGg772FC3J2EVXsiP5Cvc9N8CXnxlgOF+hpyXJrVds5M17N9F5kde+VFyfg4M5Xjg9wfMDE7xwepKxogPAFZvbePPejbz6kp4VlfTwjf2D/NXXD+Ir/Oxrd3LLnrmpsl957gwf/8ZBOjMJfvf2PezqbVmm0tbX0eE839g/yIMHBjkzUZ5zf197ih3dQUDY2ZNlR0+Wja3znyzamMQCraUgkSu7PHZkhJZkfNErxK0Fz54a52++dZgDZ3Ps6s3ynht3cvXWjgU91vF8zkyUODUWBI0geBQ5NV4iV3LprErdnC8QdGUSZJP1Wcr1Qjxf+d7hYe5+ZoAnjo8RjQivviS4+OvKzedPzxzOlXlhYJLnT0/wwsAkhwZzuGF07GtPsWdTG5f3tXLN1o6mdWtdjMHJMn/2lf08dXKcG3Z18b437KY9HcfxfD75rcPc/fRpXr6tg99888vO2ypcrVSV/WdyPHZ0hM5sYrp1sJhEBwsSC7RWgkTJ8Xj82CjRcHru9eTUWJHPPHSEh18apjub4L+8ajs3vWzDurhS9uRokXueOc1XXjhDvuyxrSvDD125iTdcvoFkLMrhoTwvDEzw/OlJXhiYmM60iUeF3Rta2dPXyuWb2rh8U+uqS3DwVbnriVN89uEjtKRivOfGndz9zADPn57gx67bys/csH1dtqYXyoLEAq2FIOF6Pk+dGKPoBNN9rxcTRYcvPHKMu58ZIBGN8GOv2Mod12xeUV0jzVJyPL51YIj/eOY0B8/mSMUj4f4wDz+bYM+mVi7va2PPpjZ29WbXzIWVh4fy/On9QapsKh7hV964m9fu7l3uYq14FiQWaLUHCVXlhYFJzk6U6Mquj0wmx/P596dO8cVHj1OseNyydxM/fX3/RffLrzX7z0zylefPEBVhT1/QSuit0S+9VlRcn3ufHeDqre1s784ud3FWhUYGifXVl7HCHR3Oc3q8SO86mNVVVfnWwSE++/ARzkyUecX2Tt716h1WKcxy2cbWRV9LsNolYhF+5JrNy10ME7IgsUKcGS9xaDC/5q+FUFWeOz3BZx46wgsDk+zozvCRt1zBtRd5BmSMaSwLEivAeNHh+YEJujKJNTtAe3y0wINhfvjp8RJdmQS/8sZLeePlG21A0pgVzILEMlINpnJ4/vQE2URsza0bPZQrB4HhwCAvDeYR4Oqt7fz4K7bymkt7V+RcRsaYmRoeJETkVuDPgSjwSVX9o1n39wOfBTrCYz6gqneLSAL4/4B9gA/8qqp+vdHlbZZc2eXQ2RzD+TLtqUTTr4XwfGU4V6Yzm6hrZsxkyeHbB4f5xv6zPHtqAgUu29jCz75mJ6/d3WurhBmzyjQ0SIhIFPgYcAtwAnhERO5S1eeqDvsg8CVV/SsR2QvcDewAfg5AVa8SkQ3APSLyA6o69/LYVaTi+hwbKXBsJE86HluWQeqnT4zxyW8d5qWhPBEJ5n3ZEk5YtiX82dyRprtlYd1fJcfje4dH+Mb+QR4/Fsx+uaUjzU+9sp/X7e5d0RdyGWPOr9EtieuBg6r6EoCIfAG4A6gOEgq0hdvtwKlwey/wNQBVPSsiYwStiu81uMwN4fvK2YkSBwZzqEJ3Ntn08YdTY0U+/dBhvvPSCL2tSd594w5yZY+TY8EVyU+dHKdSNUVFIhZhc3uKLZ2ZMHikpgNJOh7lieNjfGP/IN85PEzJ8enOJviRazbz+st62dWTXdNpmsasF40OEluA41W3TwCvnHXMh4H7ROSXgSzwpnD/k8BbROTzwDbgFeHvGUFCRN4LvBegv7+/zsWvj/Giw4GBSSYrDu2p+nbvLESu7PLFR47z70+dIhYVfuaG7dzx8s1zpqv2VRnJV6aDxsnRIifHihwezPHwoaEZ8yHFIoLrKy3JGK+/bAOvv6yXKza3rdmBd2PWq5UwcH0n8BlV/d8i8irgcyJyJfApYA/wKHAUeAiYs5iuqn4C+AQEF9M1rdQLUHI8Dg8F1z5kEzF6ss3tWvJ85cvPDvAP3z3KZMnlTXs28vYbttccF4iI0NOSpKclyTWz5klyPZ8zE+XpADKcL3PVlnau7e9cM1f7GmPmanSQOElw9j9la7iv2nuAWwFU9WERSQE9qnoW+PWpg0TkIWB/Y4tbH56vnBor8tJgjmhE6FmGab4fOzrK33z7MMdHCly1pZ33vGYnlyxh9sxYNMKWzjRbOm18wZj1pNFB4hFgt4jsJAgObwN+atYxx4Cbgc+IyB4gBQyKSIZg2pC8iNwCuLMGvFekkXyF/WcmKVY8OjOJpl8DcHykwN98+zCPHR2lrz3Ff799Dzfs7LLxAWPMRWlokFBVV0TeB9xLkN76KVV9VkQ+AjyqqncB7wf+WkR+nWAQ+52qqmFG070i4hMEmJ9pZFmXqlBxOTSYY3CyTGsy3vQrp8eLDp//3jHueeY06XiUd9+4gx++erN1BRljlqThYxKqejdBWmv1vg9VbT8H3DjP444AL2t0+ephsuTw2NFR4pFI01NaHc/nP546zRcePUax4nHrlX381PX9TV2O0hizdq2EgetVrxiuNdvsBVGePjHGXzxwkNPjJa7r7+DdN+60CfKMMXVlQaIOJkoO8Uhzu3W+d3iEP7zneTa0Jvl/fmQv+7Z3NfX1jTHrgwWJOsiV3aZOq/HwoSH+5N4X2dGT5SNvuYLWdbQ4kTGmuWxUsw5yJbdpA8TfPDDIH335BS7pbeH377jSAoQxpqGsJbFEFdfH9bUpqa4PvHiWP/vKfvb0tfGhH9677ta/NmYhChWXouMRFaElufZmV242q2WWqOzOuQi8Ie5/boC/+NpBrtrazv/4ob3rcu1nszqoKq6veL4Sj0aadq2Qr8pooUImEeWqLe1MlBxOjpZw/QqxSJSWZMzWLrkIFiSWqOw2flLae545zce/fohrt3Xwuz+0Z86cS8Y0WnXF7/nnthVFCC5wEgRFiQgkYlFS8QiTRQcIMv8aOa9XseKRKzvs7M3S35UlGhG6W5Js78oyWXI5O1nizEQJ11eS0SiZZNTmGVsgCxJLVCi7RBv4YbvryVP89TdfYt/2Tn7ntj1NX3fCrF+Fikuh4iECEYFkLEoyHqE1FiMRi5COR4nHIsQiQiwa/g63p5Rdj2PDBY6PFkjHYrSk6lvlTLUe0vEo+3Z20TZrjC4SEdozcdozcXb1tjBRdBiYKHF2soQqpONR0vFo3WYk8PwgSK6lGQ4sSCzRZKlxmU3//PgJPv3QEV61q5vf/MGX2dXTpilKjkeu7NKajnFtf8eS+vWTsSi7N7bS15Hm0NkcQ7kSLcl4XbpLp1oPO3qy9HdlLljGaETozCbozCa4dEML40UnnKyyggCZeOy8qyV6vuJ4/nRLyvXP9SJMtaJiEcHxFJELP99qYUFiiSbLLokGVN5ffPQ4f/edo7zm0h7ef8tlNvhmGq7i+kyWHdLxKFdvbacrm6jbGXFLMsbVW9sZLTjsPzMZrsgYv6jPta/KWKFCMh7lFTu6Lmp2gXg0Mj3jcdn1GMtXODleYjhfnj7mXDda8DsZi5CMR2lNxUjFo2QSUeLRCLGoEI9GpsdfSo7HeKHCqfD5BMgkYg0bRyy7HvmyS3umMZmOFiSWwPOVkuORrWOWkaryD987xhceOc5NL+vl126+zAbbQqrKZMnFCc/gsokYyVhkxTXtp/rvHc/H8YIzzqmKpt7dG/XgeD4TJYdELMLevjZ6WpJEGvCZExG6sgl+YEcXA+MlXhrMAYsbryg5HpNll+1dGbZ3X7j1sBDJWJSN7Wk2tqenK3iJCIlohFg0QjwqxCORBb8nqXiUVNXzjebDgJErgwSf24sNGKpK2fUpOh6+arBiWzLOjp4s3S2NWRrYgsQSlF2Pen6VVJW/ffgo//T4Cd60ZwPve8NuCxAE78tEycXxPDa1p9ncniZfdjkzWWK4EHQVJGNB5dus98ubDgI+rqf4nFvKRATSsSCbJpuMkUlGSUQjuOEU8iNT3RsNPLtc6N8wVqwQiwov29jKhrZUU96/aETY0pmmpzWx4PGKGa2H/s6GnTVPVfD1fL6+jjR9HWmKlTBgjBUZypWIiJBNxs6biOKrUnZ8Sm4QFESgNRVnZ3uWtlScTDLa8G5oCxJLUHbql9mkqvzNtw7zb0+e4tYrNvGLN12y7rMvzgUHn03tKfq7MmSTwUe2PRNnc2easusxWXIZnCwzOFnGVyUafvmW+uXxNQgEFTcIBuf2B0u7ZpNROjLJoEUTj5CIhT/R2q2bnpbk9NnlydEiw7ky0Uh9yrtQnq+MFytERLikt4VN7allGe9a6HhFyfGYKDls786yo06th+WQTkRJJ9Js7kxTqLiM5CucHgs+A5GIkE3EiEWD7qqS46EEC4G1p+Ns6UzRmo6TiUeb/vdbkFiCYGK/pVfkviqfePAl/uPp0/zw1X2897W7VlR3RLNVB4e+jhTbOs8Fh9mSsSjJlig9LUku26jkSi4jhTJnxstMFJ3p5v35uqV8VSpuGAzCriEIWgTZRJze1uR0P3QyFiURW1ruf/XZZb7sMjRZ5uR4kYlShXgkSrZB+fy+KhMlB89Xtndn2NyRXhHp1LXGK6IRYaRQIRmLsG97V8NaD8shk4iRScTY2pkhXw4CxqmxIoWKT2c2wfbu4DOfTcQa0vW3GBYkliBXWfqgta/Kxx84yL3PneGt127hXa/esW4DxGKCw3yiVemOO7qzFB2Psbwzs1sqGsXTmeMEU2fyG7NJWpMxUvGgZdCM8Y5s2CW1rSvDZCnoQhsYL+GrkooFg6NLLcPU++r6Pls60mzryqy4izHnG69wPJ9tXRl29GTXdGbf1Gdga7jq40r7/luQWILJoks8dvH/0Irr8xcPHODrLw7yE/u28fZX9q+4D0gzVFdiU91KS51yRESmz9Y2d6apuMHg7HCuTCwqtKbiJMMLvs7XPdQsM/L5e7KMFx1OhuMXEAx4Q9DVparTv1WDE43qBu3U5lRmjq+wqT3F9u6lv6+NVj1eUXH9dTU32XJ/BmtZ2Z+YFUxVyVccOtIXl1FwZqLEH93zAgcHc7z9hu385L5tF37QGtOI4FBLInYu5XGli0UjdLck6a4avxjKV4gQVKJTP8HFa0EKpogQkaAPOyKCRKa2g98rreVwIclYdEV0hRkLEhet7PqoXlz0f/zoKP/rvhfxVPngD5rTIDoAABqWSURBVO3hlTu7G1DClcsPU1ld36evPcW2BgaH1a56/MKY5WDfzItUdquTHhfGV+UfHz3O33/3GNu7M/zObXvYvI6+/FMXawlB94cFB2NWPvuGXqTFzv6aK7v86f0v8siRUW66rJdfesOlq64L4GKoKvmyR9lzScVjXLahle7WhHUlGLNKWJC4SLmSu+AlSw8P5fjDe17g7GSZX3jdLm6/qm/ZB6mmLqQCiEUi01MM1Ivj+UyWHHyFjW1JNne00p6OL/vfbYxZHAsSF2kynMbgQr72wlk+9vWDtCRj/OFbr2JPX1sTSnd+Zddjouhw6YYWWlJxRvJlhnMVxosOIhcfNFSVQsWj6HokYxF29bbQ25pcFy0mY9YqCxIXabLk0pKsnZ7neD6f/NZh7n76NFdubuO3br2czkxj5lZZjImSg6py3fZOOsLydGUTXLphaqKwIJtmOFdmvOgQEYhGgmmhawVFz1cmSw6ur3S1JLhsUysd6fiyXwRkjFk6CxIXwfF8nPMsWTqUK/NH97zAi2cmeeu1W3jHq3Ys+xxMU/Pud2QSXL6pdd6z+6m0w65sgks2tEwHjbFCEDSG88ECMrEwaHi+UnBcYhFhW1eGDW1JG4g2Zo2xb/RFKLt+zck4njoxxp/c+yIV1+cDt17OjZf2NLVs86m4PmOlCju7s+zozi74DL86aOzqbaHi+uTLLmPFCkOTZWKxCFduaKczk1j2IGiMaQwLEheh5MzNbFJV/uX7J/nsw0fY3JHmv9+2h21dmeYXbpZceD3CNVva6WlNLem5ggnsgkVbdva01KmExpiVbMFBQkS+qqo3X2jfejB7ydJCxeXPvnKAh18a5sZLuvmVm3cve7eLqjJSqNCaivHyvq41sUKWMab5LliTiUgKyAA9ItLJualh2oAtDSzbilW9ZKnj+fzWPz3F8dEC77lxJ3e8fPOyp3k6ns9YscK2zgy7elusK8gYc9EWcrr788CvAZuBxzgXJCaAv2xQuVa06iVLj40UODpS4JduupRbr9y0zCWDfNml7PlcubmdDW1L614yxpgLBglV/XMR+Uvgv6vq7zWhTCva7CVLj40UANi7eXmvf9AweymdiLJva+eiptg2xphaFnS1lKp6wI82uCyrwuwlS48NF4hFhM3ty3fW7no+Q/kyG9pSXNtvAcIYUz+LuaT2qyLyY7LcHe7LbPaSpcdGCmzpSC/bkorFisdY0WHPpjYu39S6phdnMcY032JqlJ8H/hGoiMiEiEyKyESDyrVilRxvxsD00ZE8/d3NT3X1fGU4X8ZH2bejk76O9LIPmBtj1p4F90uoamsjC7JaTJbPTexXcjzOTJR5056NTS3DRNHB8X129WTZ0pmx7CVjTMMsuCUhgbeLyP8Ib28TkesX8LhbReRFETkoIh+Y5/5+EXlARL4vIk+JyO3h/riIfFZEnhaR50XkdxbzhzVK9ZKlx8NB6/4mXTRXdj0GJ0u0pmNcv7OL/u6sBQhjTEMtprvp48CrgJ8Kb+eAj53vASISDY+5DdgL3Ckie2cd9kHgS6p6LfC28HUAfhxIqupVwCuAnxeRHYsob91NLVkar0p/hcYHCV+VkUKZsutz9dZ2rtrSvuwX6xlj1ofF1DSvVNXrROT7AKo6KiIXmtb0euCgqr4EICJfAO4Anqs6RgkuzANoB05V7c+KSAxIAxWCazOWzdSSpZGw7//YSJDZ1NfeuNXlcmWXkuOxvTvDtq6MDUwbY5pqMUHCCVsGCiAivYB//oewBThedfsE8MpZx3wYuE9EfhnIAm8K9/8TQUA5TXDF96+r6sjsFxCR9wLvBejv71/En7N4ZXduZtO2rsaMCQRXTTt0ZuJctbWdFktrNcYsg8Wcln4U+Bdgg4j8v8C3gD+oQxnuBD6jqluB24HPiUiEoBXiEVzpvRN4v4jsmv1gVf2Equ5T1X29vb11KE5tZdebsa710ZFC3buapi6KK1Rcrtzcxsu3dViAMMYsm8VkN/29iDwG3EwwNcd/UtXnL/Cwk8C2qttbw33V3gPcGr7Gw+FcUT0EYx9fVlUHOCsi3wb2AS8ttMz1liu5xMLMpkLFZXCyTP8V9QsShYpLvuKxrTPN9u7sgla+M8aYRlpULaSqLwCfBr4DDC3gIY8Au0VkZzh+8TbgrlnHHCMIPIjIHiAFDIb73xjuzwI3AC8sprz1Nlk6N2fT8ZEiUJ9Ba9fzGc6XiYqwb0cnuze2WoAwxqwIF6yJROQtInJERB4P01OfJZjY72kRecf5HquqLvA+4F7geYIspmdF5CMi8pbwsPcDPyciTwKfB96pqkqQFdUiIs8SBJtPq+pTF/l31kWufG7212MjeWDpQaJQcZkoOVy2oZXrtnfSlqq9JKoxxjTbQrqbfg94M0Hm0QPA1ar6kohsAL4KfPZ8D1bVu4G7Z+37UNX2c8CN8zwuR5AGuyI4no/j+dOD1MdGCiSiETYuYabVXNnFV2Xfji6bb8kYsyItpGbyVXU/gIgcnkpnVdWzIuI2tHQryPyZTemLzmyaKDpEo8J12zptQSBjzIq1kCARCRcbigD+rIWH1k3HeXnWkqVHhwtcvbX9op5rvFghFYty5dZ2UnELEMaYlWshQaKdmYsNPV51n849fG0qVLzpi+hyZZfhfIX+ruyin2e0UCGbjHLVlg4bnDbGrHgLWXRox0KeSESuUNVnl1yiFWqy7FRlNl3cdBwj+TIdmQR7N7fZldPGmFWhnjXV5+r4XCtOrlid2RQGiQVOEa6qDOXK9LQkucIChDFmFalnbbVmpyP1faXo+MSqMpuSsQgbWpMXfKyqMpQvs6k9xZ6+tmVbnMgYYy5GPfMu1+z4RJDZpNOL+hwLp+OIXGCRH181GLvozLCrt4WITettjFll7LR2Acru7Mym/AXHIzxfGc6V2dGd4ZINFiCMMatTPVsSlTo+14pSqkp/nSw5jBac8waJqaVFd29oob978RlQxhizUiypJSEil09tq+oNSy/OyjRZcucuNFRj0NoJ52Ha29dmAcIYs+ottbvpvrqUYoWbLM8TJOZpSQRrQFS4aks7fR2NW4jIGGOa5YLdTSLy0Vp3AR31Lc7Ko6rky+70xHvHhguk41F6W2ZmNpVdj8mSyzVbO+huuXDWkzHGrAYLGZN4F8FMreV57ruzvsVZeSqej+frjCVL+7sy05lOABXXJ1d2uba/g47MhVZ0NcaY1WMhQeIR4BlVfWj2HSLy4bqXaIWZPbHf0ZEC1+/smrEvX3G5pLfFAoQxZs1ZSJD4z0BpvjtUdWd9i7PylJ1zQWK86DBenJvZ5KvaVN/GmDVpIQPXLapaaHhJVqh8+dySpceG519oSIBU3C45McasPQup2f51akNE/m8Dy7IiBZlN58YjALZXBYlgET0hFbMpv40xa89CgkT1pcK7GlWQlSpXOjf769GRAtlElK7subEHx1OyyahdUW2MWZMWEiS0xvaa53o+ZdefnpTv2EiB/u7snMymtrStS22MWZsWMtp6jYhMELQo0uE24W1V1baGlW6ZlV1/uhmlqhwbLvDqS3tmHFPxPNpSNmhtjFmbFrLo0LrtbC+7/nTTaazgMFl25wxaK5BOWJAwxqxNlpJzHsWKS4Tag9ZTLLPJGLNWWe12HpMll3js3KA1zEx/9XwlFhGSltlkjFmjLEicR67sTmc2HRsp0JqM0ZE5N0hdcX1aUzZobYxZuyxI1OD7SqHizbhGor971pxNnk+7ZTYZY9YwCxI1VDwfVRCRMLNp7mp0ru/TYplNxpg1zIJEDWXHR8PcppF8hXzFqzFobeMRxpi1y4JEDSXHnd6eb9B6Sipmb6ExZu2yGq6GyVmD1sCM5UgdzycVj05fjW2MMWuR1XA15GYtWdqejs8YpA6m47DxCGPM2mZBoobJUlWQGC7MGY+ouD7tlv5qjFnjLEjMo+L6uJ4SjYSZTeGSpdUUJWPTcRhj1jgLEvMoux5Tl0MM5soUHY/+7vnmbLLMJmPM2tbwICEit4rIiyJyUEQ+MM/9/SLygIh8X0SeEpHbw/0/LSJPVP34IvLyRpcXZq5rfWyezCZVJSJC0jKbjDFrXENrORGJAh8DbgP2AneKyN5Zh30Q+JKqXgu8Dfg4gKr+vaq+XFVfDvwMcFhVn2hkeafMXLJ0bpAou8FFdNVXXxtjzFrU6FPh64GDqvqSqlaALwB3zDpGgak1KdqBU/M8z53hY5tiojRzydKuTGLGHE0V17c1JIwx60Kja7otwPGq2yeAV8465sPAfSLyy0AWeNM8z/OTzA0uAIjIe4H3AvT39y+xuIFcyZles3pqzqZqjm+r0Rlj1oeV0Kl+J/AZVd0K3A58TkSmyyUirwQKqvrMfA9W1U+o6j5V3dfb27vkwlQvWerXyGwCm47DGLM+NDpInAS2Vd3eGu6r9h7gSwCq+jCQAqrXCH0b8PkGlnGGinduydKzk2XKrl9jOg4LEsaYta/RQeIRYLeI7BSRBEGFf9esY44BNwOIyB6CIDEY3o4AP0ETxyNKzrklS+cbtPZ8JR6NkLDMJmPMOtDQmk5VXeB9wL3A8wRZTM+KyEdE5C3hYe8Hfk5EniRoMbxTVafq6dcBx1X1pUaWs9p8S5Zum5HZ5Nl4hDFm3Wh4io6q3g3cPWvfh6q2nwNurPHYrwM3NLJ8s1UvWXpsJE9PS4KW5Lm3qeL69LVbZpMxZn2wPpNZZi9ZOns8wlMlm7QgYYxZHyxIVFFV8pXgGgnPV46PFOcECcEym4wx64cFiSpl1wcVRIQzEyUqnj9nOg7FgoQxZv2wIFGl7PgQ5jadm7Pp3EJDrq9k4lGiEZuOwxizPliQqFJ2vXPpr9OZTemq++1Ka2PM+mJBospEySEeOTdovaE1OWPNiIrr0ZGxIGGMWT8sSFTJld3pi+RqTceRjltmkzFm/bAgUSUXLlkaZDbNHySScXvLjDHrh9V4oYrr4/rBkqWnx4u4vs4IEr4qkYgtNGSMWV+sxguVXW96e77V6CquT6stNGSMWWcsSITmW7J026zV6NpSNmhtjFlfLEiEyo43Y2K/TW2pGRfNub4/Y3U6Y4xZDyxIhBSYWkji2HCthYbs7TLGrC9W683iej4nx+bO2QQ2HYcxZv2xIDHLqfFSkNlUta616/kkYxHiUXu7jDHri9V6s8yb2eTZoLUxZn2yIDHLseE8EYGtnTZnkzHGWJCYZSqzKRk7N/7gq85Ync4YY9YLCxKzHBspzBiPAEBt0NoYsz5ZkKjiTGc2nVtDQm06DmPMOmY1X5WB8RK+zh20ziZiRGyhIWPMOmRBosrJsSIwd86mtrSNRxhj1icLElVOjBbnZDYF6a8WJIwx65MFiSonx4ps7kjPuGhOgFTCgoQxZn2yIFHl5Ojc6TgUm7PJGLN+We0XKrseZyZLM4KE5yuxiMy4ZsIYY9YTCxKhY8MFdHZmk2vTgxtj1jcLEqHDQ3lgbvpru03HYYxZxyxIhA4P5YmKsLnjXGaT6/u0WGaTMWYdsyAROjxUYFN7as504DYdhzFmPbMgETo8lGdL1fURU1I2HYcxZh2zGhAoVjxOjRXZWtXV5Hg+qXiUmC00ZIxZx6wGBA6ezaEwoyVRdn06bNDaGLPONTxIiMitIvKiiBwUkQ/Mc3+/iDwgIt8XkadE5Paq+64WkYdF5FkReVpEUo0o42CuRDoeZUtVS6Li2mp0xhjT0CAhIlHgY8BtwF7gThHZO+uwDwJfUtVrgbcBHw8fGwP+DvgFVb0CuAlwGlHON16+kXt//bX0tVfHICWTtEFrY8z61uiWxPXAQVV9SVUrwBeAO2Ydo0BbuN0OnAq33ww8papPAqjqsKp6jSpoRASR6unAxTKbjDHrXqODxBbgeNXtE+G+ah8G3i4iJ4C7gV8O918GqIjcKyKPi8hvzfcCIvJeEXlURB4dHBysS6F9VSKCLTRkjFn3VkIteCfwGVXdCtwOfE5EIkAMeA3w0+Hvt4rIzbMfrKqfUNV9qrqvt7e3LgWquD7ZVGxWy8IYY9afRgeJk8C2qttbw33V3gN8CUBVHwZSQA9Bq+NBVR1S1QJBK+O6BpcXmBq0tiutjTGm0UHiEWC3iOwUkQTBwPRds445BtwMICJ7CILEIHAvcJWIZMJB7NcDzzW4vAA4vk+bpb8aYwwNPV1WVVdE3kdQ4UeBT6nqsyLyEeBRVb0LeD/w1yLy6wSD2O9UVQVGReRPCQKNAner6n80srzVbNDaGGMaHCQAVPVugq6i6n0fqtp+DrixxmP/jiANtulStoaEMcasiIHrFcXzlXg0QsIym4wxxoLEbGXXs/EIY4wJWZCYxeZsMsaYcyxIzOKrkk1a+qsxxoAFiTkESMXtbTHGGLAgMZMCCEnLbDLGGMCCxAyO55NNRolGbDoOY4wBCxIzlD270toYY6pZkKjieDZnkzHGVLMgUSUWEdJxCxLGGDPFgkSVZCxK0jKbjDFmmtWIVZLxiC00ZIwxVaxGrNJqCw0ZY8wMFiSqtKUss8kYY6rZKG0oFYvQYtNxGGPMDFYrhnpaU8tdBGOMWXGsu8kYY0xNFiSMMcbUZEHCGGNMTRYkjDHG1GRBwhhjTE0WJIwxxtRkQcIYY0xNFiSMMcbUJKq63GWoGxEZBI5e5MN7gKE6FqdeVmq5YOWWzcq1OFauxVmL5dquqr3z3bGmgsRSiMijqrpvucsx20otF6zcslm5FsfKtTjrrVzW3WSMMaYmCxLGGGNqsiBxzieWuwA1rNRywcotm5Vrcaxci7OuymVjEsYYY2qyloQxxpiaLEgYY4ypad0FCRG5VUReFJGDIvKBee5PisgXw/u/KyI7mlCmbSLygIg8JyLPisivznPMTSIyLiJPhD8fanS5wtc9IiJPh6/56Dz3i4h8NHy/nhKR65pQppdVvQ9PiMiEiPzarGOa9n6JyKdE5KyIPFO1r0tE7heRA+HvzhqPfUd4zAEReUcTyvU/ReSF8H/1LyLSUeOx5/2/N6BcHxaRk1X/r9trPPa8398GlOuLVWU6IiJP1HhsI9+veeuHpn3GVHXd/ABR4BCwC0gATwJ7Zx3zX4H/E26/DfhiE8rVB1wXbrcC++cp103Avy/De3YE6DnP/bcD9wAC3AB8dxn+pwMEFwMty/sFvA64Dnimat+fAB8Itz8A/PE8j+sCXgp/d4bbnQ0u15uBWLj9x/OVayH/9waU68PAf1vA//q83996l2vW/f8b+NAyvF/z1g/N+oytt5bE9cBBVX1JVSvAF4A7Zh1zB/DZcPufgJtFRBpZKFU9raqPh9uTwPPAlka+Zh3dAfytBr4DdIhIXxNf/2bgkKpe7JX2S6aqDwIjs3ZXf44+C/yneR76g8D9qjqiqqPA/cCtjSyXqt6nqm548zvA1nq93lLKtUAL+f42pFxhHfATwOfr9XoLdZ76oSmfsfUWJLYAx6tun2BuZTx9TPhlGge6m1I6IOzeuhb47jx3v0pEnhSRe0TkiiYVSYH7ROQxEXnvPPcv5D1tpLdR+4u7HO/XlI2qejrcHgA2znPMcr937yZoBc7nQv/3Rnhf2A32qRpdJ8v5fr0WOKOqB2rc35T3a1b90JTP2HoLEiuaiLQA/xf4NVWdmHX34wRdKtcAfwH8a5OK9RpVvQ64DfglEXldk173gkQkAbwF+Md57l6u92sODdr9KyrXXER+F3CBv69xSLP/738FXAK8HDhN0LWzktzJ+VsRDX+/zlc/NPIztt6CxElgW9XtreG+eY8RkRjQDgw3umAiEif4APy9qv7z7PtVdUJVc+H23UBcRHoaXS5VPRn+Pgv8C0GTv9pC3tNGuQ14XFXPzL5jud6vKmemut3C32fnOWZZ3jsReSfww8BPh5XLHAv4v9eVqp5RVU9VfeCva7zecr1fMeBHgS/WOqbR71eN+qEpn7H1FiQeAXaLyM7wLPRtwF2zjrkLmMoA+M/A12p9keol7O/8G+B5Vf3TGsdsmhobEZHrCf53DQ1eIpIVkdapbYJBz2dmHXYX8F8kcAMwXtUEbrSaZ3fL8X7NUv05egfwb/Mccy/wZhHpDLtX3hzuaxgRuRX4LeAtqlqoccxC/u/1Llf1ONZba7zeQr6/jfAm4AVVPTHfnY1+v85TPzTnM9aI0fiV/EOQjbOfIEvid8N9HyH40gCkCLovDgLfA3Y1oUyvIWgqPgU8Ef7cDvwC8AvhMe8DniXI6PgO8OomlGtX+HpPhq899X5Vl0uAj4Xv59PAvib9H7MElX571b5leb8IAtVpwCHo830PwTjWV4EDwFeArvDYfcAnqx777vCzdhB4VxPKdZCgj3rqczaVybcZuPt8//cGl+tz4efnKYLKr292ucLbc76/jSxXuP8zU5+rqmOb+X7Vqh+a8hmzaTmMMcbUtN66m4wxxiyCBQljjDE1WZAwxhhTkwUJY4wxNVmQMMYYU5MFCWMWQEQ8mTnzbN1mIBWRHdUzjxqzksSWuwDGrBJFVX35chfCmGazloQxSxCuI/An4VoC3xORS8P9O0Tka+GEdV8Vkf5w/0YJ1nF4Mvx5dfhUURH563C9gPtEJB0e/yvhOgJPicgXlunPNOuYBQljFiY9q7vpJ6vuG1fVq4C/BP4s3PcXwGdV9WqCSfQ+Gu7/KPANDSYevI7gCl2A3cDHVPUKYAz4sXD/B4Brw+f5hUb9ccbUYldcG7MAIpJT1ZZ59h8B3qiqL4WTsA2oareIDBFMLeGE+0+rao+IDAJbVbVc9Rw7COb83x3e/m0grqq/LyJfBnIEs9j+q4aTFhrTLNaSMGbptMb2YpSrtj3OjRf+EMHcWNcBj4QzkhrTNBYkjFm6n6z6/XC4/RDBLKUAPw18M9z+KvCLACISFZH2Wk8qIhFgm6o+APw2wbT1c1ozxjSSnZUYszBpEXmi6vaXVXUqDbZTRJ4iaA3cGe77ZeDTIvKbwCDwrnD/rwKfEJH3ELQYfpFg5tH5RIG/CwOJAB9V1bG6/UXGLICNSRizBOGYxD5VHVrushjTCNbdZIwxpiZrSRhjjKnJWhLGGGNqsiBhjDGmJgsSxhhjarIgYYwxpiYLEsYYY2r6/wFy8eZ6rOUL/gAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "pIjA9kLsTB_K"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tmp = df_test\n",
        "P, R, F1 = score([str(i) for i in tmp['Actual Text'].tolist()], [str(i) for i in tmp['Generated Text'].tolist()], lang='en')\n",
        "f1scores_test = F1.tolist()\n",
        "epochs_test = [0]*len(F1)\n",
        "avgs_test = [np.average(f1scores)]\n",
        "for epoch in range(1,21):\n",
        "  P, R, F1 = score([str(i) for i in tmp['Actual Text'].tolist()], [str(i) for i in tmp[f'Generated Text {epoch}'].tolist()], lang='en')\n",
        "  f1scores_test += F1.tolist()\n",
        "  epochs_test += [epoch]*len(F1)\n",
        "  avgs_test.append(np.average(F1.tolist()))\n",
        "# sns.lineplot(x=epochs, y=f1scores, ci='sd')\n",
        "sns.lineplot(x=epochs_test, y=f1scores_test, ci=95)\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('F1_Bert')\n",
        "plt.savefig('test_score.png')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "4MV8jFqjD-g6",
        "outputId": "112ba536-2cd9-4905-8465-8659de68b029"
      },
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']\n",
            "- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEGCAYAAACQO2mwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXhk91ng++9bp3aVdql3tbvb3XbsuG3HtBfskIRsOA44kLDYJECYXBIWc1nChXDJk5snzAzMTJh7yRBgTCYkhEAStmDAxPEkhgTsxG4vbbu9xL0v6kV7SbWe5b1/nCN1Sa3qLnWrtNX7eR49qjrnVNVPR1Xnrd/v/S2iqhhjjDHziS13AYwxxqxcFiSMMcbUZUHCGGNMXRYkjDHG1GVBwhhjTF3x5S7AYurr69Nt27YtdzGMMWZVefLJJ4dVtX++fWsqSGzbto29e/cudzGMMWZVEZGj9fZZc5Mxxpi6LEgYY4ypy4KEMcaYuixIGGOMqcuChDHGmLosSBhjjKnLgoQxxpi6LEgYY4ypa00NpjPGLB5VZXC8DMC6jhQJx75TtiILEsaY81Q8n++cmWRoskJMhINDUwz0ZNjYmSGdcJa7eGYJWZAwxswyUXLZPziBHyj9uTQAfqAcHy1xdKTIxs40m7uz5FLNuXyoKsWqT9n1ScZjpBOO1WKWkQUJYwwQXpxPjpd45cwUbck47ZlzNQYnJnRnkwSqDE1WGRwv05dLsrWnjY5MHBG5rNcOAmWy4jFWqHJ6okzZ9SF6SgWSToyOTIL2VJxcOk7KgseSsSBhjMH1A145M8npiQo9bUmc2PwX/ZgInZkEAFMVj6eOjdGejrO9r43ubJJYncfNx/MD8mWP4akKZ/NlvEBxRGhLxWmbU0vxA6VY8ZgounhBMLM9FY+RSyfoTMfJpuKkEw7peIy4BY9FY0HCmBY3WXbZP5in6gX0t6caflwuFSeXilOq+jx7YoJ0Isa23jb621N1L9Jl1ydfcjkzWWG0UEE1rCXkUom6gQnCmkw2ef7lyvMDihWP8UIVL1CmKzTZpMPVGzpmApq5dBYkjGlRqsrpiTIvnZ4km3ToziYv6XkySYdM0qHqBbx0epKDQ1Ns7cmyvjNN0olRrPqMF6ucnqiQL7uIQCbh0J1JXnYzVdwJaw1zi16q+jx1dIyr1rezqSt92a/TyixIGNOCPD/gwNAUp8ZLdGWSi9I8k4zH6Mul8PyAw8MFDg8XSMZjVNwAEcgm4/TlGq+pXI5M0iEZj/GdM5Pkyy471+Usf3GJLEgY02IKFY8XBicoVH1621KL/i077sToaUsRqOIHSi61PE0+TkzobUtyNl9msuzy6k2d5+U6zMVZaDWmhZzNl3niyCh+QFMCRK2YyLJ/exeRMGAF8MSRUc7my8tantXIwqoxLcDzAw4NT3FiLGxeWu6L91JrS8VJxmM8d3KCK8oe2/vaLpgoX6ggUMaKVQ6PFChXfdrTCboyCXLpqMdVwlnU11tKFiSMWaOCQPECpeL5vHx6kqmKR1+Taw8rWcIJcyYnxopMll2u2dhx2aPHPT9gaLLCkZECZTegLRknl0pQcQOOl0t4QYAQjvXIJhy62pJ0pONkEnFSiRipeGzF/z8sSBjTZKpKoepTrHoIgkjYFBMTwvuxcNxYTM7tg5pjRBDAj9r4vUDxfcULAvxAqXgBFc+n6gdUvYCqG1D1A7xAowJAKu7Q27Y0SeOVLCZCb1uKfMll75FRrtvcSdcl9Ooquz5nJsocGyviB0p7KjEr95JJOmSYHYBcP2Bkssqp8dK58sSEjnSCzqjWkXBioBCoRj/h+2c6vzPz/49+AlV8BT8IyCYdrlrfceknpw4LEsY0wXRgGJ2qMDhepuR6531jlOhHo5/pbTPPMb1NABUUnTm+9ndMBCcW/sREiDsxkvHV27yxFDoyCSpe2E1257ocAz3Zhr7RFyoeJ8dLnBwvEQM6M/UHHs6VcGJRM9+5y64fKBU34ERNrSMU/ofPvS/mfoEIvzxMf9FQlELF56r1DZ6ABbAgYcwiKlQ8RgtVBsdLlFy/ZgRxermLZuZIxR162mIcGJoiX3a5an0Hyfj5uRpVJV/yODZaYHiqQsJx6MkmZy7Yl8OJyby1joXyA8WrepddnvlYkDDmMhWrUWAYK1Gs+sRiQlsyTm+bfbxWOicm9OfSjBWrPHV0lFdv7qQ9HTYbBYEyUqhwdLjIZMUjnXDoy7VesLd3sTGXoFT1GS1UGBwvUaj6CJBLJejN2UdqNerKJClVffYeGeNVG9pRhSOjBapemIxeqkGAK5G9o42ZI5hJCCqqzCQIA4VC2WNwosRk2SUWNSVZQnhtyCQdEo7w0ulJYiK0p+O0L9NAwJXEgoRpGarKmYkykxUPNwjwg7ALox8oXtQbyA903iTydKZYIfpm2XrNDq0gHnWTNec0PUiIyJ3A7wMO8ClV/d05+68APg30A6PAe1T1RLTvK8BtwL+p6vc3u6xm7SpVw5XWRgtV0nFnpqeISNgunYo7pGu6nBpjQk0NEiLiAJ8E3gKcAJ4QkQdU9YWawz4O/JmqflZE3gj8DvAT0b7/BmSBDzSznGbtmq49vHxmcmYwlTGmcc0em38LcEBVD6lqFfgC8I45x1wLfD26/UjtflX9GjDZ5DKaNars+jw/OMGLpyfpSCdmeq0YYxrX7CCxGThec/9EtK3WPuCd0e0fAtpFpLfJ5TJrmKqGE9kdHmWi6NGXq78IjjHmwlbCJ+fXgNeLyNPA64GTgN/og0Xk/SKyV0T2Dg0NNauMZpUouz4vDObZPzhBWypuK5MZc5manbg+CQzU3N8SbZuhqoNENQkRyQHvUtXxRl9AVe8H7gfYs2ePXuRws4YNTYarrAlY7yNjFkmzg8QTwC4R2U4YHO4Bfrz2ABHpA0ZVNQB+k7CnkzENq3g+h4YKnJ4o0dmC02Ab00xN/TSpqgfcBzwEvAh8SVX3i8jHROTu6LA3AC+LyHeA9cB/mn68iHwT+CvgTSJyQkS+r5nlNavP8GS4iM7wVIXetpQFCGMWWdPHSajqg8CDc7Z9pOb2XwN/Xeex39Pc0pnVquqFi+icjBbRmW9iNmPM5bMR12ZZVL0AJZz2YprqucmRVc9Nla0abp0+1vUDXj49iR8o/bnWXUTHmKVgQcI0nR8oxapHseIxWnQZLVRx/XDufBGJLv56bpEEZq+ZAOGc+dP3pqfGSKcvb3plszKpKkdGimQSDus77EvAcrMgYRZd2fUpVn3yJZfRYpXJkjdTQ0jFHbIJh7gNbDM1AlVeOj3JoweGeezQCGcnKwD05VLs3tzB7s2d7N7cZUFjGViQMJdlvlqC5wdAODdSOuHQnU3YB3sVGi1U2T84waauDFf0ZBd9QKIfKM8PTvDowRG+dXCE0WKVeEy4caCLe24eoOorz52c4Klj4zzycjgGajpoXL+5i+u2dLK+3YJGs1mQMJckX3Y5eHaKvNUS1hRVZf9gnn967hSPHRrBj9bJTjjCtt42dq7LcWV/jl3rcmy9hMDh+gH7Tozz6MERvn1ohHzZIxmPseeKbr57Ry+3bO8hmzx3WXr77o2oKsfHSjx3cuK8oNHfnmL3pk52b+7kui2dbOiw8TGLTVTXzvizPXv26N69e5e7GGveRMnlmePjpOMxMgnHvsmtAaWqz7985ywPPneKIyNF2lIOb37Vel67q4+z+QoHhqY4eHaKg0NTFKrhhAi1gWPnuhw7++cPHNNrST96cITHj4xSrPpkEg63bO/h9it7uWlrN+lE4/klVeXYaJHno6Dx3MkJ8uVw6c517Smu29zJHVf2cvO2npZ5b/qBUqh63LGz75IeLyJPquqeefdZkDALMVF0efr4WJg4XsAH26xMx8eKPPjcKb7+0lmKVZ8d/W28ffdGXrerf97/b6DK6YkyB85OzQSOA0NTFOcJHFu6M7wwmGfv0TEqXkB7Ks6tO3q4/co+bhzoWrQxLfWCxp4ruvm511/JuhaoXViQaJAFieYaL1Z55vg4uVScVNwCxGrlB8q3D4/w4HOn2HdignhMeO3OPt6+eyNXb2hf8LfvuYHjQFTjKFZ9urIJvntHL7df2cd1mzqWZKJFP1D+Yd8gf/7towC859Yr+IEbNuHE1m6twoJEgyxINM9YIQwQ7WkLEKvVWLHKV/ef5iv7TzM8VaUvl+Jt123grdeupyubXNTXClQZLVTpziaX7eJ8Nl/mj79xkCeOjLGjv4373rCTXevbl6UszWZBokEWJJpjdKrCvhMTdKQTNrL5EuRLLg+/eAYvUPpzSfpyqZmfZp9PVeXF05P807OnePTgMF6g3DjQxV27N3LLtp41/e0awr//0YMj3P+NQ4yXqrx990bec9sVs5Lja0Ezg8TaOlNm0Y1MVXjWAsQlmSy7fPmZQf5h3yAld/7Z7zszCfpzKfraw+DRPx1A2lP05ZL0tqVmXcgrnk++5JEvu0yUXPIll3w5vJ+vuT9RcsmXXSbLHn6gZJMOb7tuA3ft3siW7uxSnYJlJyLcsTPMgXzuW0f5x2dP8ejBET7w+iv57h2Ls2yNHyj7o668z5+coCOToL89xbr2FOvb0/R3hL97c6tz8kmrSawAUxWPbMIhtsK+1Q1PlnnuZJ7OTGJVvrmXS21wKLs+d+zs456bB1jfkWZkqsrwVIWhqQrDUxWGJysMRduGpyozCeBpMWGmySZfdim7wbyvKUB7Ok5HJkFHOkFnJkFHdH9TZ4Y7dvaRSVoz4cunJ/mDR17hyEiRW7f38IHXXUl/+8KXtHX9gGdPTPDowWG+VdOV97pNHZTcgKHJMiNTVWqvrgL05pL0t6dZFwWRde1p1nWEt/vbU5fclGvNTQ1ajUGiVPV5/PAoibgw0J1lXcelv1EW09BkmedOTtBlU283bLLs8vfPDPJAVHN4bRQcruhta/g5ilWPockKw7XBZLJCoBpd+BNhIKgJAp3pBG2p+JpvOlosnh/wwL5BPv/4MRwR3nPbVt6+++KJ7Yrn8/SxcR49OMzjR0YpVMKuvDdvC7vyftcVs7vyun7AyFSVM5NlhvIVzkyWOTtZYWiywpl8meGpCsGcy2/SiZFOxEgnnOgnRjpec3tmu0M6HiOVcMgkYiSdGO2ZBD91+7ZLOicWJFawQ0PhTKZtqTj5sgvAhs40m7oydCzToLSz+TLPn5ygK2sBohFzg8MdO/u4d4HBwSy9M/kyf/SvB3ny6Bg7+3P8wvfuZOe63KxjSlWfvUdHefTgCHuPjlJ2A3KpOLduP9eV91KbYf1AGSmEQeNs9FOseJRcn4obhL89n7IbUHb98Mc7d3tugNlzRTd//XO3X1JZLEisUK4f8NjBETrSiZlvMYEqU2WPqu/Tnk6wtSdLby61ZN8SLUA0bqrs8eV9J/mHfYMUq1Gz0p4BtvVZcFgtVJV/OzDMn3zzEBMll7tv2MQP3riZZ09O8O8Hhnn62DhVP6Ark+C2Hb3cfmUvuzd3Lvua6aqK62sUOHyKFR8vCPihm7Zc0vNZ4nqFGspX8DWYFQBiInRkEkAiXK/5VJ54LGyKWt+ZbuoAtjMTZfYPWoC4mPOCw5W93HPzVgsOq5CI8D27+nnN1m7+7LEj/P0zg3z5mUEA+nJJvu/V67n9yj6u2dixoprzRIRkXEjGY3SQwG8LcxLNYEFimQSBcnS0QGe6fv/06bZHzw84Olrk0HCBdR0ptnRl6cjEF3XKgdPjJfafytOTTS77t6SVaqrs8ff7TvKABYc1J5eK8/Nv2Mkbr17HvpMT3Lili13rc8RaZFqPC7EgsUzGilXKXkAudfG8Q9yJ0Z1Noqrkix5P5cfIJh229WbpyV3+kp2nxku80CIBouL5FCo+hapHseJTqHjh7er0bZ9ixWNqen/Vo1AJ948XXap+wO1RcNhuwWHNedXGDl61sWO5i7GiWJBYJkdHi2QX2HQkIuTScXLEqXg+L5yaxIlNkUk4JKKqZ9KJkYo7pOIxnJgQj8VwHMERie7LrK62g2MlXjyTP68//lpzNl/m048e4d8PDF/wuJhANhknm3RoS8VpSzqs70jTlozTkYnzxlett+BgWooFiWWQL7uMF6v05y594rFU3CGVcwhU8QPF9ZRyNRw45avOLPk5vZ5b7Spv8ZiQdGI4TozxUpXe7NoNEGXX52+eOsHfPnUSBH7wxs1s7EyTTTrkUnGyUSBoS4WBwWa1NWY2CxLLYHC8RMpZnAR0TISYIyykUjIdWPxA6WtLrcl2V1Xlm68M86ePHmZ4qsrrdvXx3tu3X9LAKWNamQWJJVZ2fU6Nl+ltW9wJ1RbiUgLLanJwaIr7v3GIF07l2dHfxq+99WpevalzuYtlzKpkQWKJnZ4o48TEmjSaYKLk8rnHjvDVF87Qno5z3/fu5M3XrF+zTWnGLAULEkvI8wOOjxWXbST1WuX5Af/43Cm+8Pgxyl7A3Tds4p5btpJL2dvbmMtln6IlNDJVwfPVvtkuoiePjvGpfzvEibESN23t5v/4nu0MtNAsp8Y0mwWJJaKqHB4p0p62U74YBsdLfOrfDvHEkTE2dab5yPdfy54ruq0Zz5hFZlesJTJedClVffpy1rvmUqgqJTcc0PbQ/tM8sG+QhBPjp2/fxg/csMmmETGmSZoeJETkTuD3AQf4lKr+7pz9VwCfBvqBUeA9qnoi2vdTwIejQ/+jqn622eVtlqMjBbI2n/8sfqDkSy7jpSrjxXARnfGSG92evW0iGu087c3XrOMnb9tG9zL2EjOmFTQ1SIiIA3wSeAtwAnhCRB5Q1RdqDvs48Geq+lkReSPwO8BPiEgP8P8AewjHgT0ZPXasmWVuhqmKx1jRbelahKpyZKTI40dGeeLwKIMTJSbL809IFo8JnZkEndkEXZkkA93Z6HaCrmyCHX05my/JmCXS7JrELcABVT0EICJfAN4B1AaJa4FfjW4/Anw5uv19wMOqOho99mHgTuAvm1zmRXdyrNiSzSGuH/D8yQkePzzK40dGOTtZAWDXuhyv3dlHVyZBZzY5c/HvzIRBoS1lo56NWSmaHSQ2A8dr7p8Abp1zzD7gnYRNUj8EtItIb53Hbp77AiLyfuD9AFu3bl20gi+WsutzaqJMd7Y1mkXyJZe9R8d4/MgoTx0do+T6JOMxbtzSxY/uGeDmbT30WBORMavGSkhc/xrwByLyXuAbwElg/lXj56Gq9wP3Q7joUDMKeDnO5ssIrMmpL6adHCvx7cMjPH5klBdP5QkUurMJvmdXH7du7+H6LV1NXQfDrB2uH1Cs+vhBmH9SwBEhnQgnrbQa5tJrdpA4CQzU3N8SbZuhqoOENQlEJAe8S1XHReQk8IY5j/2XZhZ2sXl+wLHRtTl47uXTk/z7wWEePzzKyfESANt6s/zIdw1wy/Yedq6zufhXM1WlWA1XPQOIIaQS4QzDiznOJ9BwdbWSG75OOu6wqStNT1uSdMKhVPXJl1yGC1XGSlVUw8kqw6CxuGUx82t2kHgC2CUi2wmDwz3Aj9ceICJ9wKiqBsBvEvZ0AngI+M8i0h3df2u0f9UYLVTxAl1TazSUXZ9P/dthHtp/mnhM2L25k++/fiO3bOthXcelz2prVoaK5zNV8VANV2bbtT6HE5OZzhfjxSpBAIrOfMNPxmML+kJQ9QKKVQ9fFUHozSXZ3tdGezpBZk4PwHTCobstyRV9bXh+QNH1mSp5jBarjBWr+L6ioqSccIGuVsz9NVtTg4SqeiJyH+EF3wE+rar7ReRjwF5VfYCwtvA7IqKEzU2/ED12VER+mzDQAHxsOom9GqgqR0cKtCVXQove4jhwdoqPf/VlBsdLvOumzfzongGya+jva1V+oExVPFzfJ5uMc9W6dnpyyVlNhF3ZJFu6w/d1xQubhKa7L0+UXAINW3oTsRipRLiuyXTTUKBKabpWopBJOgz0ZOnOJsml4w3XBuJOjA4nRkc6wabuzMzYmULFZ6xYZXQqLIsIxGMxcqnGn9vUJ6orrhn/ku3Zs0f37t273MUAYLxY5ZnjY/S2rf5v136g/O3TJ/j8t4/RnU3wK2++iuu3dC13scxlqG1OiseEjZ0Z1nWkyKUWvixuEIQX63CwYzi+pVANayMQ5uP6ckn629O0p+NNzU9VPJ9ixWe0UOXkeIlAlY50Ys3XMPwgXOP6jp19l/R4EXlSVffMt8++BjbJ8dEiqfjqP71nJ8v8vw9/h+cH89yxs4/73rCTnE0tsmqFy7d6KNCXS3FVVzudmcRlfeOOxSRcxS8VnxkL5PkBJddHgbbk0n2jD1dlDJuotvZmGcpXODpaYKLk0paMn9ecZS7OPu1NUKh4DE9V6G1b3YPnvvnKEJ985ACBwi+/aRdvfNU6612yCvmBMll28YKAbDLOrnmakxZb3InRvszf3hNOjE3dGTZ0phkrVjk8UmB4qkI67thYnAWwINEEpyZKxGOrt7teserxP//1EF9/+SxXr2/ng2+9io2dmeUu1qpWdn2qXhCuQ77ARO9CTOcMKl6AFwQIghODTV2X3py02sViQm8uRU9bknzZ48RYkaHJCk5MaE9dXi2qFViQWGQVz+fkWImuBgfPTVU8Kq5P7wqZsuOlU3k+/vDLDE1WuOfmAX5sz8Ca6p21VILpNn/XRwTa0wnWd6aYqnjkyy5BEDbYK5B0wsBRm+xtlOcHlL2AinduaFF7KsHm7jQd6QTZZJx0YvV+YVlMItF0L5lOilWPU+NlTo4XUQ3/P2s9b3GpLEgssqFo6olGvikGqnz4y89xcKjAlu4Mrxno4saBbnZv7lzytlM/UL74xDG+uPc4fbkUv/PO67l2Y8eSlmG18/yAQtXHC4KZZO3OdTk6MnFS8XP/z5lv+25A2fXIlz3yJY/xkouqooAgJOMxUvHYzMUrUA0f4/kzx6XiMbqzSbqzWbKpONklbP9fzbLJOFeuyzHQk+XsZJkjw0XcoEoumbhgM9z0+vCBKkEAvipBoPiqM/8TCAcAJpwYcSf8vZrHDFmQWER+oBwdKdLe4OC5xw6OcHCowJuvWcdoocpDL5zhH549RTwmXL2hndds7eY1A11c2Z9r6gf/9ESZ33v4ZV46Pcn3Xt3PB153JW22qltDKp5PqerjB0oiHmNDZ4q+XIr2dP1mDInGF6QTDp0kWB8tvx0EYfAou+Fz5ssu+ZJLvuwiCDGBjkyCjV1hL6FM0pkVfMzCJeMxtnRn2diZYWSqwrGRIsNTFWb+dSKg4bgQACcWXvSTToxUKvydiAJBKh4jFhNUoVT1mKp6lCoB+bKLHyjTT6mEXXSnHxdf4csZ25VgEY0WKlR9v6ER1n6gfP7bR9nSneG+792FExOqXsCLp/I8fXycp4+P8effOsqff+souVScG7Z08pqt3dw40MX6RRq0pqo88vJZ/vhfDxET+LW3Xs3rr+pflOdeq6b75pdcHwGySYdtfW10ZROX3d4fiwmZpEMm6dDdBpsI80B+oLh+QNIJL0Jm8TkxYV1Hmv72FPmyRxAojiM4Ijix6EdkAed/dvOx6wdUvWDmd6HiU6x6FCoeUxWfqPURgZmapBMLvxiEv2XW76VkQWKRhIPniuSSjdUivvnKEMfHSvz69109809PxmPcMNDFDQNdvJdtTJRc9kUB45nj4/z7wREANnamZwLGrnU5AlU8X/ECxfMDvOiiMvO7dp+vuEH4+/nBCR49OMKrN3Xwq2++ykZMR1Snz9e586ho+KVSoactybbeLJ3Z5vYQmhZepKzGsBSm8xaLLeHE6uY8VJWqH+D6iuuF7zc/CINJZTqw+AFVN/ztRRFFEBQlJhBomNtqBgsSiyRf9pgsew2tGeEHyl88foxtvdkLDn7pzCR43VX9vO6qflSVE+Mlnj42zjPHx/j6S2d48LlTl1XmeEz4yduu4J03bWmpduzpdmU3+mCGk8nJTJOCCGTiDplUnJ5EgrZknFTCIeGITf1gFp2IROM7mFsBmdf0lxg/+pm+3ayPsAWJRXJirEi6wfbhr790hlMTZX7rrmsaTmiJCAPdWQa6s9x9wyZcP+Dl05McHysSjwnxqG0z7sRIzLov57V/Tu+bnrCtVRSrHsWqhxMTssk4uVTYrt+WdEjGHRLx8DxdSi8jY5aKiESf56V5PQsSi6Ds+gxNVuhpoNur6wd84Ynj7FyX49btPZf8mgknxnWbO7luc+clP0erqHoBE2WXjkycm7b20JFpvbECxlwqCxKLoFQNk5iNXHgefuEMZycr/PwbdtqFqsn8QBkvVcOAuqmD/vaUnXNjFsiCxCKYqngIF7/4VL2AL+09zjUb2rlpq02Q1yyqOtPtcEdfG5u6MjYg0JhLZEFiEeRLLqnExS9CX9l/ipFClV99y1X2jbZJpioeparHpu4M23rbbEU8Yy6TBYlFkC+7F01al12fv9p7gus3d9o0201Q8Xwmyy5d2STXbe5peECjMebCLEhcJtcP+zLnUhe+KP3Tc6cYL7n85m1XLFHJWoPnB4yXXDKJGLs3d9Kbs7yDMYvJgsRlKrv+RY8pVj3+5skT3LS12+ZDWiSBKvmSC8Cu9Tk2dmZaaqyHMUvFgsRlmp6e4UIe2DfIZMXjPbduXZIyrWWBKoWKR8ULGOjJMNCTbamxHsYsNQsSlylfconH6ietp8oeX376JLdu72HX+vYlLNnaUbvUpgj059Jc0Zu1SQiNWQINf8pE5Guq+qaLbWs1+bJHMl4/SPzdMycpVH3efavlIhaidiI9gN62JDvX5+jM2Lz/xiyliwYJEUkDWaBPRLphpnWlA9jcxLKteKpKoezRUWdCsImSyz/sG+S1O/vY3te2xKVbncquT6HqRRPpJdje10ZXNnnBQGyMaZ5GahIfAH4Z2AQ8ybkgkQf+oEnlWhUqXoCvWnf+pb996gQVz+fHb7FcxIVUPJ+pigcK7ZkEr9rQTtcSzbBqjLmwiwYJVf19EfkD4P9W1d9egjKtGtPTccxnrFDlH587xeuv6megJ7uk5VoNXD9gqhLO29+WcrhqXTvdbcklX5HPGHNhDeUkVNUXkXcCFiRqFKv1p+P4qyeP4/kB99xstQgIg0Kp6uMGAQCZhMP2vjZ62pKWgDZmBVvIp/NrItyEOoMAABokSURBVPIu4G9VVS96dAuYKLvztpUPTVb45+dP8+Zr1rOpK7MMJVt+VS+g5Pr4QUCg4Qpu6ztTdGfDoGBNScasDgsJEh8AfhXwRaREtNKeqrbs6LB8cf6eTV/aexyAH9szsNRFWjbTaz0HqgRALumwqStNVzZJW8rWYjZmtWo4SKiqdfKv4foBZc8/r6nkdL7Mwy+e4c5Xb1jTy4FO5xRUw/Xccqk4Az1ZOjMJ2lJx641kzBqxkHESArwb2K6qvy0iA8BGVX38Io+7E/h9wAE+paq/O2f/VuCzQFd0zIdU9UERSQL/E9gDBMAvqeq/NPyXNVm96Ti+8PgxHBF+5Lu2LHGJlkap6lN0PRJOjCt6snRlk2RTtqSnMWvVQpqb/pDwYv1GwgT2FPBJ4OZ6DxARJzrmLcAJ4AkReUBVX6g57MPAl1T1j0TkWuBBYBvwMwCqultE1gH/LCI3q2qwgDI3Tdk9vxgnx0o88vJZfuD6TfQ2sNb1aqGqTEVTYbRn4ly3rpPubNLmSjKmBSwkSNyqqjeJyNMAqjoWfdu/kFuAA6p6CEBEvgC8A6gNEko4MA+gExiMbl8LfD16rbMiMk5Yq7hgzWWpTJbPn47jLx4/RsKJ8cNrpBbhB8pk2cULlPUdaTZ3Z+hI29KfxrSShQQJN6oZKICI9BPWLC5kM3C85v4J4NY5x3wU+KqI/CLQBrw52r4PuFtE/hIYAL4r+j0rSIjI+4H3A2zdunTdTcdLLqmadvejIwW++coQ77ppC10NrHW9klW9gHzZJe4IW3oybOzMWG8kY1rUQoLEJ4C/A9aJyH8Cfpiwqehy3Qt8RlV/T0S+G/iciFwHfBq4BtgLHAUeBc5LBKjq/cD9AHv27FmSrrnzTcfxF48fI5N0eOdNC5upxPMDRotVnJiQjjtkk86yfVMvVDxKrk8m6XDtxnZ6cinLNRjT4hbSu+nzIvIk8CbC7q8/qKovXuRhJwm//U/bEm2r9T7gzug1HovmiupT1bPAr0wfJCKPAt9ptLzNVPHCvv/T03EcHJri0YMj3HvzwIJXRBsvuexa10425XBqvMRIoYpG4woyieYHjECVybKH6wf05JJcvaGdrmzCmpSMMcACpwpX1ZdE5BhhvmC4gYc8AewSke2EweEe4MfnHHOMMPB8RkSuAdLAkIhkAVHVgoi8BfDmJLyXTdiz6Vyl5fPfPkouFecdNy6sFlGOvrVv7g4XzOnLpah6AePFKqcmSowUqwiQTcQXbbqKQJWqF66m5wUBAmzqyrCxK0PORj4bY+ZoZBbYuwmbmkYJm5c+CZwBtonIb6jqZ+s9VlU9EbkPeIiwe+unVXW/iHwM2KuqDwAfBP5ERH6F8Mr7XlXVqEfTQyISEAaYn7isv3QRFSs+0/Mclqo+e4+M8a6btix4eompissNA92zegkl4zHWdaRZ15Gm4vmMF6oMTpQZnqoQE8gmGx+t7AdKxfOjmk8Y1BwR2jNx+tsztKfj5NJxG+hmjKmrkavabwNvJex59Ahwvaoeii7iXyMc41CXqj5I2K21dttHam6/ANwxz+OOAFc3UL4lN16uziStj4wUUOBVGxc21nCq7NHTlqI7W795KhV3WN+ZYX1nhrLrM16scnI8DBgi0FYTMLxore2KFzBdy3FiQmc2yaauBLloKox0ImZNScaYhjUSJAJV/Q6AiBye7s4adUv1mlq6FSpfOjcdx6GhKQB29OUafnygStnz2T3Q2fAFO51w2NCZYUNnhlLVZ6xQZXC8xMhUBQSSToyubGJmxHM64ZCKW0AwxlyeRoJELFpsKAYEcxYearmuL54fUHZ92pLhqTs4XKA9Hacv13i314mSy0D3pecAMkmHTDLDpu6whhETsWkwjDFN0chVqpPZiw09VbOv5WaDLXuzh4YcHipwZX+u4W/snh8miwd6F2eNCRu/YIxppkYWHdrWyBOJyKtVdf9ll2iFK1XPDdXw/IAjIwXuvmFTw4+fKLtcta7dksXGmFVhMdsoPreIz7Vi1U7HcXyshBdow+tXl12fTMJhfefanR3WGLO2LGaQaIkMab58bjqO6aT1lf2NJa0nKx5XrW+3ifGMMavGYgaJNZ+f0Gh08kzPpuECqXisodXnJssu/e1JuttW97xOxpjWYl1iFqDiBfi+zpqOY1tv20VrBoEqVT9YUDdZY4xZCRYzSFQX8blWpLLrM92JKVDl8HCBHf0Xz0eMF6sMdGcXPCLbGGOW22UFCRF51fRtVb3t8ouzshUr/kxX17P5CsWqf9F8hOsHODFhoGdxurwaY8xSutyaxFcXpRSrRL7szkydfTBKWl+sZ9NEucqV/Tkb7GaMWZUameDvE/V2Ea5L3TLyNQsNHRouEBPY1ls/SJSqPrlknPUd1uXVGLM6NdJI/tOEM7VW5tl37+IWZ+Xy/ICi69MbTcdxaGiKge7sBWsIU1WXm7Z2E7Mur8aYVaqRIPEE8LyqPjp3h4h8dNFLtELNnY7j0FCBGwY66x4/UXLZ0JFe9UuZGmNaWyNB4oeB8nw7VHX74hZn5QoXGgqNFauMFqvsqJO09gPF9QO2NTgS2xhjVqpGsqk5VS02vSQr3FTZm5mO49BQAYAr6wSBiXKVbX1Zsknr8mqMWd0aCRJfnr4hIn/TxLKsaOOlak3SOurZNE9NYrrL65Zu6/JqjFn9GgkStVnXHc0qyEo2PR3HdPfXQ0MF1rWn5l0PYqIUzvI6fawxxqxmjVzJtM7tllHxAoJAZ6bfODQ0Ne8gumLVoz0Tpy+XWuoiGmNMUzQSJG4QkbyITALXR7fzIjIpIvlmF3AlqLjnejYVqx6DE+XzpuNQVQpVj1397dbl1RizZjSy6FDLr45Tcs8t5X14OExaz52sb7LisbEzQ2c2saRlM8aYZrKG8waMF12S0UpyMz2bamoSfqB4gV5w9LUxxqxGFiQaMHs6jik6Mwl6ataFKLk+69pTZJItX+kyxqwxFiQuwg+Uouuf69k0XGB7X9vMbLAAVc+n25qZjDFrkAWJiyi7/kwfYNcPODZSnNXUBGGXr4wNnDPGrEEWJC6i5PoEUcff46NFvEDPS1orkElYU5MxZu1pepAQkTtF5GUROSAiH5pn/1YReUREnhaRZ0Xkrmh7QkQ+KyLPiciLIvKbzS7rfKbmDKIDZnV/df2ATNyx9SKMMWtSU69sIuIAnwTeBlwL3Csi18457MPAl1T1NcA9wB9G238ESKnqbuC7gA+IyLZmlnc+EyWX5PRCQ8NTpBMxNnVlZvZXvIAuy0cYY9aoZn/9vQU4oKqHVLUKfAF4x5xjFOiIbncCgzXb20QkDmQI19Be0sF74XQc7kwt4dBQge29bcTmJK0tSBhj1qpmB4nNwPGa+yeibbU+CrxHRE4ADwK/GG3/a6AAnAKOAR9X1dGmlnaOqh/gRdNxBKocHi7MO6mfJa2NMWvVSmhIvxf4jKpuAe4CPiciMcJaiA9sArYDHxSR8yYYFJH3i8heEdk7NDS0qAUrV4OZnk2nJ8qUXJ8dc6YHD7CktTFm7Wp2kDgJDNTc3xJtq/U+4EsAqvoYkAb6gB8HvqKqrqqeBf4d2DP3BVT1flXdo6p7+vv7F7XwtdNxHBqeHml9ribh+gHpeMyS1saYNavZV7cngF0isl1EkoSJ6QfmHHMMeBOAiFxDGCSGou1vjLa3AbcBLzW5vLPkyx5JZ3o6jimcmHBF77l1IsKktS1PaoxZu5oaJFTVA+4DHgJeJOzFtF9EPiYid0eHfRD4GRHZB/wl8F5VVcJeUTkR2U8YbP5UVZ9tZnnnypfOJa0PDhUY6M7MWifCRlobY9a6pmdcVfVBwoR07baP1Nx+AbhjnsdNEXaDXRZ+oExVPHqimsKh4Slu2tp93nGWtDbGrGXWmF7H9HQcIsJoocp40Z1/Og5LWhtj1jALEnWUXX9mGb5DQ9Ga1jXTcXh+QMqS1saYNc6ucHUUKh5ONGju0MxCQ+dqEmUvsAWGjDFrngWJOsZLtSOtp9jQkaYtdS7/UPV8ujPWs8kYs7ZZkKgjXGgo6v46XDhvTWuAbMqS1saYtc2CxDwqnj8zHUeh4nFqosyOfpse3BjTeixIzKPsnpuO4/D0SOuafIQlrY0xrcKucvMoV71zPZuGp3s2nQsSFUtaG2NahAWJeUyUPVLRdBwHhwp0ZRL0tJ1LUlc8n660Ja2NMWufBYl51E7HcThKWkvNGhIKs3o6GWPMWmVBYo4gUAoVn4QjuH7AsdHieWtaA2SSlrQ2xqx9FiTmKHs+oIgIR0eK+IHO6v5qSWtjTCuxK90cpap/XtK6dg0JS1obY1qJBYk5Zk3HMVQgk3DY0Jme2V/1A0taG2NahgWJOSbK3qzpOLb1tRGrSVoHqpa0Nsa0DAsSc0xPxxGocnikMGsQHYAqpJN22owxrcGudjWqXoDrBzgx4dR4mbIbnJe0TidiM3M6GWPMWmdBokYpWmgIziWtd1jS2hjTwixI1KidjuPgUIF4TNjak53Zb0lrY0yrsSBRI1/2SDrnktZbe7IknHOnSC1pbYxpMRYkauRLYc8mVeXQcGHWpH4AgSWtjTEtxq54kSBQpiphTWK0UGWi5M7KR3h+QDJuSWtjTGuxIBGpnY5jek3rK/tnTw/ebUlrY0yLsSARKbvBuek4hs5fQ6LqB3RmLEgYY1qLBYlIoIpEHWAPDhXY2JkmmzyXpFZVcikLEsaY1mJBYh6HhqfmXdPaktbGmFZjV705pioeZ/IVdtQ0NfmBknAsaW2MaT1NDxIicqeIvCwiB0TkQ/Ps3yoij4jI0yLyrIjcFW1/t4g8U/MTiMiNzS7v4aHpkdbngkTZ9emyfIQxpgU1NUiIiAN8EngbcC1wr4hcO+ewDwNfUtXXAPcAfwigqp9X1RtV9UbgJ4DDqvpMM8sLcHC6Z1PNanRVP6DLejYZY1pQs2sStwAHVPWQqlaBLwDvmHOMAh3R7U5gcJ7nuTd6bNMdHirQnU3Q3XZu+g0baW2MaVXNvvJtBo7X3D8B3DrnmI8CXxWRXwTagDfP8zw/xvnBBQAReT/wfoCtW7deZnHrJ61tTWtjTCtaCYnre4HPqOoW4C7gcyIyUy4RuRUoqurz8z1YVe9X1T2quqe/v/+yClL1Ao6NFi1pbYwxkWYHiZPAQM39LdG2Wu8DvgSgqo8BaaCvZv89wF82sYwzTo4XCXTumtaWtDbGtK5mB4kngF0isl1EkoQX/AfmHHMMeBOAiFxDGCSGovsx4EdZonzE0ZEiMHukddm1pLUxpnU1NUioqgfcBzwEvEjYi2m/iHxMRO6ODvsg8DMiso+wxvBeVZ2eIeN1wHFVPdTMck47Olokk3DY0Jk+9zdgSWtjTOtq+tVPVR8EHpyz7SM1t18A7qjz2H8Bbmtm+WodHSmwo7+NmMis7emE5SOMMa1pJSSuVwQ/UI6PleZNWluQMMa0KgsSkeOjRapeMGdNa0taG2NamwWJyEunJ4E5a0hY0toY0+IsSERePjOJExO2dGdntgWWtDbGtDgLEpGXT0+ypStDwpl9SiwfYYxpZRYkCOdmevn0JFf0nqtFWNLaGGMsSABwaqLMRMlla0/tmta+LVdqjGl5FiSAihfwhqv72bmupmeTG9BtSWtjTIuzIEE4DcfHf+SGWdNx2EhrY4yxIFGXYklrY4yxIDEPP1DiMUtaG2OMBYl5VDzfBtEZYwwWJOZVcQPr2WSMMViQmJeitKctaW2MMRYk6rB8hDHGWJA4jx8oTkxIxe3UGGOMXQnnqHg+ndkkMmfhIWOMaUUWJOaouIGtIWGMMRELEnMoSs5GWhtjDGBBYl6ZpCWtjTEGLEjMYklrY4yZza6GNSxpbYwxs1mQqFH1LWltjDG1LEjUSDgxS1obY0wNCxI1UomYJa2NMaaGBYka2bhjSWtjjKlhV8QandmEJa2NMaZG04OEiNwpIi+LyAER+dA8+7eKyCMi8rSIPCsid9Xsu15EHhOR/SLynIikm1XOhBOjL5dq1tMbY8yq1NQsrYg4wCeBtwAngCdE5AFVfaHmsA8DX1LVPxKRa4EHgW0iEgf+HPgJVd0nIr2A26yydmYSYD2bjDFmlmbXJG4BDqjqIVWtAl8A3jHnGAU6otudwGB0+63As6q6D0BVR1TVb3J5jTHG1Gh2kNgMHK+5fyLaVuujwHtE5ARhLeIXo+1XASoiD4nIUyLy6/O9gIi8X0T2isjeoaGhxS29Mca0uJWQuL4X+IyqbgHuAj4nIjHCprDXAu+Ofv+QiLxp7oNV9X5V3aOqe/r7+5ey3MYYs+Y1O0icBAZq7m+JttV6H/AlAFV9DEgDfYS1jm+o6rCqFglrGTc1ubzGGGNqNDtIPAHsEpHtIpIE7gEemHPMMeBNACJyDWGQGAIeAnaLSDZKYr8eeAFjjDFLpqm9m1TVE5H7CC/4DvBpVd0vIh8D9qrqA8AHgT8RkV8hTGK/V1UVGBOR/04YaBR4UFX/qZnlNcYYM5uE1+O1Yc+ePbp3797lLoYxxqwqIvKkqu6Zb99KSFwbY4xZodZUTUJEhoCjl/EUfcDwIhVnMVm5FsbKtTBWroVZi+W6QlXn7R66poLE5RKRvfWqXMvJyrUwVq6FsXItTKuVy5qbjDHG1GVBwhhjTF0WJGa7f7kLUIeVa2GsXAtj5VqYliqX5SSMMcbUZTUJY4wxdVmQMMYYU1fLBYkGVspLicgXo/3fFpFtS1CmgWh1vheiVfh+aZ5j3iAiEyLyTPTzkWaXq+a1j0QrAz4jIucNaZfQJ6Jz9qyINH0iRhG5uuZcPCMieRH55TnHLMk5E5FPi8hZEXm+ZluPiDwsIq9Ev7vrPPanomNeEZGfWoJy/TcReSn6P/2diHTVeewF/+dNKNdHReRkzf/qrjqPveDntwnl+mJNmY6IyDN1HtvM8zXv9WHJ3mOq2jI/hPNHHQR2AElgH3DtnGN+Hvjj6PY9wBeXoFwbgZui2+3Ad+Yp1xuAf1ym83YE6LvA/ruAfwYEuA349jL8X08TDgha8nMGvI5whuLna7b9V+BD0e0PAf9lnsf1AIei393R7e4ml+utQDy6/V/mK1cj//MmlOujwK818H++4Od3scs1Z//vAR9ZhvM17/Vhqd5jrVaTaGSlvHcAn41u/zXwJhGRZhZKVU+p6lPR7UngRc5fnGklewfwZxr6FtAlIhuX8PXfBBxU1csZbX/JVPUbwOiczbXvo88CPzjPQ78PeFhVR1V1DHgYuLOZ5VLVr6qqF939FuH0/UuqzvlqRCOf36aUK7oG/Cjwl4v1eo26wPVhSd5jrRYkGlkpb+aY6MM0AfQuSemAqHnrNcC359n93SKyT0T+WURevVRlIpyF96si8qSIvH+e/Y2c12a6h/of3uU6Z+tV9VR0+zSwfp5jlvu8/QfCGuB8LvY/b4b7omawT9dpOlnO8/U9wBlVfaXO/iU5X3OuD0vyHmu1ILGiiUgO+Bvgl1U1P2f3U4TNKTcA/wP48hIW7bWqehPwNuAXROR1S/jaFyThOiV3A381z+7lPGczNKz3r6i+5iLyW4AHfL7OIUv9P/8j4ErgRuAUYdPOSnIvF65FNP18Xej60Mz3WKsFiUZWyps5RsLFjjqBkWYXTEQShG+Az6vq387dr6p5VZ2Kbj8IJESkr9nlil7vZPT7LPB3hNX+Wo2c12Z5G/CUqp6Zu2M5zxlwZrrJLfp9dp5jluW8ich7ge8H3h1dXM7TwP98UanqGVX1VTUA/qTO6y3X+YoD7wS+WO+YZp+vOteHJXmPtVqQaGSlvAeA6R4APwx8vd4HabFE7Z3/C3hRVf97nWM2TOdGROQWwv/dUgSvNhFpn75NmPh8fs5hDwA/KaHbgImaanCz1f2Gt1znLFL7Pvop4O/nOeYh4K0i0h01r7w12tY0InIn8OvA3RouCzzfMY38zxe7XLU5rB+q83qNfH6b4c3AS6p6Yr6dzT5fF7g+LM17rBnZ+JX8Q9gT5zuEvSR+K9r2McIPDYTLp/4VcAB4HNixBGV6LWFV8VngmejnLuBngZ+NjrkP2E/Yo+NbwO1LdL52RK+5L3r96XNWWzYBPhmd0+eAPUtUtjbCi35nzbYlP2eEQeoU4BK2+b6PMI/1NeAV4H8DPdGxe4BP1Tz2P0TvtQPATy9BuQ4QtlFPv8+me/JtIlz9se7/vMnl+lz03nmW8OK3cW65ovvnfX6bWa5o+2em31M1xy7l+ap3fViS95hNy2GMMaauVmtuMsYYswAWJIwxxtRlQcIYY0xdFiSMMcbUZUHCGGNMXRYkjGmAiPgye9bZRZuBVES21c48asxKEl/uAhizSpRU9cblLoQxS81qEsZchmgdgf8arSXwuIjsjLZvE5GvRxPWfU1Etkbb10u4jsO+6Of26KkcEfmTaL2Ar4pIJjr+/4zWEXhWRL6wTH+maWEWJIxpTGZOc9OP1eybUNXdwB8A/1+07X8An1XV6wkn0ftEtP0TwL9qOOngTYQjdAF2AZ9U1VcD48C7ou0fAl4TPc/PNuuPM6YeG3FtTANEZEpVc/NsPwK8UVUPRZOwnVbVXhEZJpxawo22n1LVPhEZAraoaqXmObYRzvm/K7r/G0BCVf+jiHwFmCKcwfbLGk1YaMxSsZqEMZdP69xeiErNbZ9z+cK3E86LdRPwRDQjqTFLxoKEMZfvx2p+PxbdfpRwllKAdwPfjG5/Dfg5ABFxRKSz3pOKSAwYUNVHgN8gnLb+vNqMMc1k30qMaUxGRJ6puf8VVZ3uBtstIs8S1gbujbb9IvCnIvJ/AUPAT0fbfwm4X0TeR1hj+DnCmUfn4wB/HgUSAT6hquOL9hcZ0wDLSRhzGaKcxB5VHV7ushjTDNbcZIwxpi6rSRhjjKnLahLGGGPqsiBhjDGmLgsSxhhj6rIgYYwxpi4LEsYYY+r6/wEyDQv7ZRN2HQAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# sns.lineplot(x=epochs+epochs[-1:]*1000, y=f1scores+f1scores[-21:-20]*1000)"
      ],
      "metadata": {
        "id": "y-vARfvvSWxp"
      },
      "execution_count": 48,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "avgs_test\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "91MbgYZ61kKd",
        "outputId": "a8e062dd-7545-4b20-82d7-818424e83b7b"
      },
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.9007139695712498,\n",
              " 0.8888025188446045,\n",
              " 0.8971212644577027,\n",
              " 0.8959635577201843,\n",
              " 0.8960647253990174,\n",
              " 0.8990968532562256,\n",
              " 0.9003901939392089,\n",
              " 0.9007612872123718,\n",
              " 0.9032401289939881,\n",
              " 0.9034955520629883,\n",
              " 0.9056349878311157,\n",
              " 0.9051502294540406,\n",
              " 0.9054881272315979,\n",
              " 0.9072710828781128,\n",
              " 0.9064169182777405,\n",
              " 0.9077996139526368,\n",
              " 0.9067304992675781,\n",
              " 0.9040553059577942,\n",
              " 0.9054397668838501,\n",
              " 0.904916265964508,\n",
              " 0.9047256751060486]"
            ]
          },
          "metadata": {},
          "execution_count": 51
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "avgs_val\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BEc28U3e1l4b",
        "outputId": "ff0db1d3-12b4-403f-e0ae-5335715e6951"
      },
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.9007139695712498,\n",
              " 0.8820711525659712,\n",
              " 0.8848751201516106,\n",
              " 0.8883750410307021,\n",
              " 0.8904118675088125,\n",
              " 0.8898781973218161,\n",
              " 0.8911398294426146,\n",
              " 0.8931414480247195,\n",
              " 0.8926203061663915,\n",
              " 0.8929717781997862,\n",
              " 0.8934601464914897,\n",
              " 0.8919769733671158,\n",
              " 0.8927901401406243,\n",
              " 0.8941775517804282,\n",
              " 0.8945314642928895,\n",
              " 0.8920150872260805,\n",
              " 0.894870260405162,\n",
              " 0.8934100544641889,\n",
              " 0.893971167859577,\n",
              " 0.892719532762255,\n",
              " 0.893372861165849]"
            ]
          },
          "metadata": {},
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Validation loop and saving the resulting file with predictions and acutals in a dataframe.\n",
        "# Saving the dataframe as predictions.csv\n",
        "# print('Now generating summaries on our fine tuned model for the validation dataset and saving it in a dataframe')\n",
        "# for epoch in range(config.VAL_EPOCHS):\n",
        "#     predictions, actuals = validate(epoch, tokenizer, model, device, val_loader)\n",
        "#     final_df = pd.DataFrame({'Generated Text':predictions,'Actual Text':actuals})\n",
        "#     final_df.to_csv('./predictions.csv')\n",
        "#     print('Output Files generated for review')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "J2YXar6LdMja",
        "outputId": "92af08f4-30fd-46c4-c43a-44fb645cb1f5"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Now generating summaries on our fine tuned model for the validation dataset and saving it in a dataframe\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py:2277: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).\n",
            "  FutureWarning,\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Completed 0\n",
            "Output Files generated for review\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "final_df"
      ],
      "metadata": {
        "id": "ezn9MyFWKxth",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 704
        },
        "outputId": "93e0e963-73cd-42d5-92cf-010338cde657"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                        Generated Text  \\\n",
              "0    The four-legged star behind Jon Snow's faithfu...   \n",
              "1                                                  No.   \n",
              "2                                               Pants.   \n",
              "3    \"This is the first time I'm saying this in an ...   \n",
              "4                                               Pizza.   \n",
              "..                                                 ...   \n",
              "141                                                No.   \n",
              "142                  Machu Picchu (Machu Picchu, Peru)   \n",
              "143  Two-thirds plan to leave their current organiz...   \n",
              "144  The Israeli army has killed at least 206 Pales...   \n",
              "145                                                65.   \n",
              "\n",
              "                                           Actual Text  \n",
              "0                                               Ghost.  \n",
              "1    He was traded to Nashville to be closer to his...  \n",
              "2    They once agreed that if they ever won a Gramm...  \n",
              "3    Basically, HBO handed back the rights because ...  \n",
              "4          Pizza. It's literally right in the picture.  \n",
              "..                                                 ...  \n",
              "141                 To satisfy the demand for seafood.  \n",
              "142                                      Machu Picchu.  \n",
              "143  They don't feel like their leadership skills a...  \n",
              "144  Home demolitions, denial of building permits a...  \n",
              "145                                                65.  \n",
              "\n",
              "[146 rows x 2 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-ebca724e-728b-418d-ba72-1b94f1a4cb41\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Generated Text</th>\n",
              "      <th>Actual Text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>The four-legged star behind Jon Snow's faithfu...</td>\n",
              "      <td>Ghost.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>No.</td>\n",
              "      <td>He was traded to Nashville to be closer to his...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Pants.</td>\n",
              "      <td>They once agreed that if they ever won a Gramm...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>\"This is the first time I'm saying this in an ...</td>\n",
              "      <td>Basically, HBO handed back the rights because ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Pizza.</td>\n",
              "      <td>Pizza. It's literally right in the picture.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>141</th>\n",
              "      <td>No.</td>\n",
              "      <td>To satisfy the demand for seafood.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>142</th>\n",
              "      <td>Machu Picchu (Machu Picchu, Peru)</td>\n",
              "      <td>Machu Picchu.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>143</th>\n",
              "      <td>Two-thirds plan to leave their current organiz...</td>\n",
              "      <td>They don't feel like their leadership skills a...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>144</th>\n",
              "      <td>The Israeli army has killed at least 206 Pales...</td>\n",
              "      <td>Home demolitions, denial of building permits a...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>145</th>\n",
              "      <td>65.</td>\n",
              "      <td>65.</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>146 rows × 2 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ebca724e-728b-418d-ba72-1b94f1a4cb41')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-ebca724e-728b-418d-ba72-1b94f1a4cb41 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-ebca724e-728b-418d-ba72-1b94f1a4cb41');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "for i,data in enumerate(training_loader, 0):\n",
        "  if i>0:\n",
        "    break\n",
        "  y = data['target_ids'].to(device, dtype = torch.long)\n",
        "  print(y.shape)\n",
        "  print(tokenizer.decode(y[0]))\n",
        "  print(tokenizer.decode(y[1]))\n",
        "\n",
        "  y_ids = y[:, :-1].contiguous()\n",
        "  lm_labels = y[:, 1:].clone().detach()\n",
        "  # lm_labels[y[:, 1:] == tokenizer.pad_token_id] = -100\n",
        "\n",
        "  print(tokenizer.decode(y_ids[0]))\n",
        "  print(tokenizer.decode(y_ids[1]))\n",
        "  print(tokenizer.decode(lm_labels[0]))\n",
        "  print(tokenizer.decode(lm_labels[1]))\n",
        "\n",
        "  ids = data['source_ids'].to(device, dtype = torch.long)\n",
        "  mask = data['source_mask'].to(device, dtype = torch.long)\n",
        "\n",
        "  outputs = model(input_ids = ids, attention_mask = mask, decoder_input_ids=y_ids, labels=lm_labels)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hNJ53PVY5uNu",
        "outputId": "9049c2b8-9c62-4142-87c8-81aabffb66c5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([2, 150])\n",
            "The regional transport authority has allowed the bike taxis to operate in four Uttar Pradesh districts including Ghaziabad, Gautam Budh Nagar, Hapur and Bulandshahr. There are 17 conditions that have to be fulfilled in order to get a permit for registering a bike taxi, according to the officials. Startups like Rapido, Baxi, Ola and Uber provide bike taxis.</s><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad>\n",
            "A forest department report submitted to the High Court suggested that all the seven domesticated elephants of Delhi be shifted to a sanctuary outside the city. It observed that not only were the elephants kept in poor conditions, but it was also risky to house them in crowded localities. Most of the elephants don't have proper ownership certificates, report said.</s><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad>\n",
            "The regional transport authority has allowed the bike taxis to operate in four Uttar Pradesh districts including Ghaziabad, Gautam Budh Nagar, Hapur and Bulandshahr. There are 17 conditions that have to be fulfilled in order to get a permit for registering a bike taxi, according to the officials. Startups like Rapido, Baxi, Ola and Uber provide bike taxis.</s><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad>\n",
            "A forest department report submitted to the High Court suggested that all the seven domesticated elephants of Delhi be shifted to a sanctuary outside the city. It observed that not only were the elephants kept in poor conditions, but it was also risky to house them in crowded localities. Most of the elephants don't have proper ownership certificates, report said.</s><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad>\n",
            "regional transport authority has allowed the bike taxis to operate in four Uttar Pradesh districts including Ghaziabad, Gautam Budh Nagar, Hapur and Bulandshahr. There are 17 conditions that have to be fulfilled in order to get a permit for registering a bike taxi, according to the officials. Startups like Rapido, Baxi, Ola and Uber provide bike taxis.</s><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad>\n",
            "forest department report submitted to the High Court suggested that all the seven domesticated elephants of Delhi be shifted to a sanctuary outside the city. It observed that not only were the elephants kept in poor conditions, but it was also risky to house them in crowded localities. Most of the elephants don't have proper ownership certificates, report said.</s><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad>\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py:2277: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).\n",
            "  FutureWarning,\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model(input_ids = ids, attention_mask = mask, labels=y[:, :].clone().detach())[0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cj7_pc2GRu8C",
        "outputId": "8f175993-053c-4fd3-b479-67c563c75edc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor(7.4522, device='cuda:0', grad_fn=<NllLossBackward0>)"
            ]
          },
          "metadata": {},
          "execution_count": 53
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model(input_ids = ids, attention_mask = mask, decoder_input_ids=y_ids, labels=lm_labels)[0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "v31HyP-qR18v",
        "outputId": "123a0e46-10e3-4d72-eb7b-d0bb6ea692e7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor(7.7485, device='cuda:0', grad_fn=<NllLossBackward0>)"
            ]
          },
          "metadata": {},
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer.decode(model_original.generate(input_ids = ids, attention_mask = mask)[0])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 168
        },
        "id": "H7CwlMk8W_lk",
        "outputId": "8136c6bb-ca8f-4d17-9d62-eade41a5afe5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-56-ee4a1346cfab>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mtokenizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel_original\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgenerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput_ids\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mids\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mattention_mask\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmask\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m: name 'model_original' is not defined"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer.decode(model2.generate(input_ids = ids, attention_mask = mask)[0])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 168
        },
        "id": "jAR2iHIWXGex",
        "outputId": "5083d80d-7bd4-4e74-a838-b2cedf2c45ea"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-57-f0d7282c3e0f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mtokenizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgenerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput_ids\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mids\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mattention_mask\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmask\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m: name 'model2' is not defined"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer.decode(data['source_ids'][0])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 125
        },
        "id": "OjHmsMjgI3Kc",
        "outputId": "06d0989d-6370-4ece-f8d3-6a9bb44d245f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'summarize: Karachi, Apr 30 (PTI) Bollywood superstar Salman Khans much awaited film \"Tubelight\" might not hit the screens on Eid in Pakistan as some local filmmakers have started a movement to stall the release of the film. The filmmakers along with the Film Distributors Association and the Film Producers Association want to ensure better business for Pakistani films during the Eid holidays. The distributors, producers and filmmakers say they want to protect the interest of Pakistani movies as two big films, \"Yalghaar\" and \"Shor Sharaba\", along with others in the pipeline, are slated for an Eid release. \"If Tubelight also releases on Eid, it will certainly hit the business of these Pakistani films and that will not help our industry,\" actor Mustafa Qureshi said. Those trying to stall the release of the Salman KhaN- starrer are citing a law passed by the Federal Ministry of Information in 2010 which forbids the release of any Indian film on a Muslim holiday. \"If required we can even go to the court,\" producer Altaf Hussain said. Sohail Khan, the producer of \"Shor Sharaba\", said, \"My film is supposed to release on Eid-ul-Fitr and if the government doesnt stall Tubelight, I will not release it as a mark of protest.\" Hussain said the associations are planning to send an official letter to Prime Minister Nawaz Sharif as well. He confirmed a meeting was held in Lahore at which it was decided that the entire Pakistani film industry will join the planned protest to stop the release of \"Tubelight\" on Eid. \"It has been decided that all those related to the Pakistani film industry will support the Pakistani films that are supposed to release on Eid-ul-Fitr,\" Hussain added.</s><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad>'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer.decode(model.generate(input_ids = ids, attention_mask = mask)[0])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "ingh6HUqJJWV",
        "outputId": "144c15ec-e768-4182-b616-3a5f13f0c68e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'<pad> UP Regional Transport Authority has given a green signal to bike taxis in four UP'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer.decode(y_ids[0])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 89
        },
        "id": "Ltk38o82I7hH",
        "outputId": "7fff77c8-4a3e-4686-8db8-a71f0824aeb6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'The regional transport authority has allowed the bike taxis to operate in four Uttar Pradesh districts including Ghaziabad, Gautam Budh Nagar, Hapur and Bulandshahr. There are 17 conditions that have to be fulfilled in order to get a permit for registering a bike taxi, according to the officials. Startups like Rapido, Baxi, Ola and Uber provide bike taxis.</s><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad>'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "outputs[1].shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kzT13R1V7rTc",
        "outputId": "7dc0dcc1-2ae9-4cee-e79f-720489c022ff"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([2, 149, 32128])"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "lm_labels.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Mcvl-PqYCKDo",
        "outputId": "ddbbe7fc-6274-42f1-e874-4f274ca26a36"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([2, 149])"
            ]
          },
          "metadata": {},
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "final_df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1878
        },
        "id": "fwjMKlr35hhE",
        "outputId": "3aa3c058-6c7d-406b-d20c-d076ab25aa32"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                        Generated Text  \\\n",
              "0    hotels in Mumbai and other Indian cities are t...   \n",
              "1    'State Bank of Tomato' opened in Uttar Pradesh...   \n",
              "2    a 24-year-old Indian athlete has been indicted...   \n",
              "3    the remains of a German hiker who disappeared ...   \n",
              "4    GP Manish Shah, who practised in east London, ...   \n",
              "..                                                 ...   \n",
              "898  HSBC has disclosed being probed by tax authori...   \n",
              "899  NHAI has chopped over 8,000 trees to develop a...   \n",
              "900  adolescent, who was born in India and is now l...   \n",
              "901  'Rangoon' stars Kangana Ranaut, Shahid Kapoor ...   \n",
              "902  adolescent, who was born in India and is now l...   \n",
              "\n",
              "                                           Actual Text  \n",
              "0    Hotels in Maharashtra will train their staff t...  \n",
              "1    The Congress party has opened a bank called 'S...  \n",
              "2    Tanveer Hussain, a 24-year-old Indian athlete ...  \n",
              "3    The remains of a German hiker, who disappeared...  \n",
              "4    A UK-based doctor, Manish Shah, has been charg...  \n",
              "..                                                 ...  \n",
              "898  Global banking giant HSBC has disclosed being ...  \n",
              "899  The National Highways Authority of India (NHAI...  \n",
              "900  Former Infosys CFO Mohandas Pai has said that ...  \n",
              "901  The Kangana Ranaut, Shahid Kapoor and Saif Ali...  \n",
              "902  A ticket collector on Thursday allegedly bit o...  \n",
              "\n",
              "[903 rows x 2 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-c1fd1755-373d-4b59-a0b8-d4a88f13938e\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Generated Text</th>\n",
              "      <th>Actual Text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>hotels in Mumbai and other Indian cities are t...</td>\n",
              "      <td>Hotels in Maharashtra will train their staff t...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>'State Bank of Tomato' opened in Uttar Pradesh...</td>\n",
              "      <td>The Congress party has opened a bank called 'S...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>a 24-year-old Indian athlete has been indicted...</td>\n",
              "      <td>Tanveer Hussain, a 24-year-old Indian athlete ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>the remains of a German hiker who disappeared ...</td>\n",
              "      <td>The remains of a German hiker, who disappeared...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>GP Manish Shah, who practised in east London, ...</td>\n",
              "      <td>A UK-based doctor, Manish Shah, has been charg...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>898</th>\n",
              "      <td>HSBC has disclosed being probed by tax authori...</td>\n",
              "      <td>Global banking giant HSBC has disclosed being ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>899</th>\n",
              "      <td>NHAI has chopped over 8,000 trees to develop a...</td>\n",
              "      <td>The National Highways Authority of India (NHAI...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>900</th>\n",
              "      <td>adolescent, who was born in India and is now l...</td>\n",
              "      <td>Former Infosys CFO Mohandas Pai has said that ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>901</th>\n",
              "      <td>'Rangoon' stars Kangana Ranaut, Shahid Kapoor ...</td>\n",
              "      <td>The Kangana Ranaut, Shahid Kapoor and Saif Ali...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>902</th>\n",
              "      <td>adolescent, who was born in India and is now l...</td>\n",
              "      <td>A ticket collector on Thursday allegedly bit o...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>903 rows × 2 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c1fd1755-373d-4b59-a0b8-d4a88f13938e')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-c1fd1755-373d-4b59-a0b8-d4a88f13938e button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-c1fd1755-373d-4b59-a0b8-d4a88f13938e');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yQ6M61QhZPuN"
      },
      "source": [
        "<a id='section06'></a>\n",
        "### Examples of the Summary Generated from the model\n",
        "\n",
        "##### Example 1\n",
        "\n",
        "**Original Text**\n",
        "New Delhi, Apr 25 (PTI) Union minister Vijay Goel today batted for the unification of the three municipal corporations in the national capital saying a discussion over the issue was pertinent. The BJP leader, who was confident of a good show by his party in the MCD polls, the results of which will be declared tomorrow, said the civic bodies needed to be \"revamped\" in order to deliver the services to the people more effectively. The first thing needed was a discussion on the unification of the three municipal corporations and there should also be an end to the practice of sending Delhi government officials to serve in the civic bodies, said the Union Minister of State (Independent Charge) for Youth Affairs and Sports. \"Barring one, the two other civic bodies have been incurring losses. It would be more fruitful and efficient if all the three were merged,\" he said, referring to the north, south and east Delhi municipal corporations. The erstwhile Municipal Corporation of Delhi (MCD) was trifurcated into NDMC, SDMC and EDMC by the then Sheila Dikshit-led Delhi government in 2012. Goel predicted a \"thumping\" victory for the BJP in the MCD polls. He said the newly-elected BJP councillors will be trained on the functioning of the civic bodies and dealing with the bureaucracy. \n",
        "\n",
        "\n",
        "**Original Summary**\n",
        "Union Minister Vijay Goel has favoured unification of three MCDs ? North, South and East ? in order to deliver the services more effectively. \"Barring one, the two other civic bodies have been incurring losses. It would be more fruitful and efficient if all the three were merged,\" he said. MCD was trifurcated into EDMC, NDMC and SDMC in 2012.\n",
        "\n",
        "**Generated Summary**\n",
        "BJP leader Vijay Goel on Saturday batted for the unification of three municipal corporations in the national capital saying a discussion over this was pertinent. \"Barring one, two other civic bodies have been incurring losses,\" said Goels. The erstwhile Municipal Corporations of Delhi (MCD) were trifurcated into NDMC and SDMC by the then Sheilha Dikshi-led government in 2012. Notably, the MCD poll results will be declared tomorrow."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ER5tYH-MZPuN"
      },
      "source": [
        "##### Example 2\n",
        "\n",
        "**Original Text**\n",
        "After much wait, the first UDAN flight took off from Shimla today after being flagged off by Prime Minister Narendra Modi.The flight will be operated by Alliance Air, the regional arm of Air India. PM Narendra Modi handed over boarding passes to some of passengers travelling via the first UDAN flight at the Shimla airport.Tomorrow PM @narendramodi will flag off the first UDAN flight under the Regional Connectivity Scheme, on Shimla-Delhi sector.Air India yesterday opened bookings for the first launch flight from Shimla to Delhi with all inclusive fares starting at Rs2,036.THE GREAT 'UDAN'The UDAN (Ude Desh ka Aam Naagrik) scheme seeks to make flying more affordable for the common people, holding a plan to connect over 45 unserved and under-served airports.Under UDAN, 50 per cent of the seats on each flight would have a cap of Rs 2,500 per seat/hour. The government has also extended subsidy in the form of viability gap funding to the operators flying on these routes.The scheme was launched to \"make air travel accessible to citizens in regionally important cities,\" and has been described as \"a first-of-its-kind scheme globally to stimulate regional connectivity through a market-based mechanism.\" Report have it the first flight today will not be flying at full capacity on its 70-seater ATR airplane because of payload restrictions related to the short Shimla airfield.|| Read more ||Udan scheme: Now you can fly to these 43 cities, see the full list hereUDAN scheme to fly hour-long flights capped at Rs 2,500 to smaller cities \n",
        "\n",
        "\n",
        "**Original Summary**\n",
        "PM Narendra Modi on Thursday launched Ude Desh ka Aam Nagrik (UDAN) scheme for regional flight connectivity by flagging off the inaugural flight from Shimla to Delhi. Under UDAN, government will connect small towns by air with 50% plane seats' fare capped at?2,500 for a one-hour journey of 500 kilometres. UDAN will connect over 45 unserved and under-served airports.\n",
        "\n",
        "**Generated Summary**\n",
        "UDAN (Ude Desh Ka Aam Naagrik) scheme, launched to make air travel accessible in regionally important cities under the Regional Connectivity Scheme, took off from Shimla on Tuesday. The first flight will be operated by Alliance Air, which is the regional arm of India's Air India. Under the scheme, 50% seats would have?2,500 per seat/hour and 50% of the seats would have capped at this rate. It was also extended subsidy in form-based funding for operators flying these routes as well."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mcPxgKeQZPuN"
      },
      "source": [
        "##### Example 3\n",
        "\n",
        "**Original Text**\n",
        "New Delhi, Apr 25 (PTI) The Income Tax department has issued a Rs 24,646 crore tax demand notice to Sahara Groups Aamby Valley Limited (AVL) after conducting a special audit of the company. The department, as part of a special investigation and audit into the account books of AVL, found that an income of over Rs 48,000 crore for a particular assessment year was allegedly not reflected in the record books of the firm and hence it raised a fresh tax demand and penalty amount on it. A Sahara Group spokesperson confirmed the development to PTI. \"Yes, the Income Tax Department has raised Rs 48,085.79 crores to the income of the Aamby Valley Limited with a total demand of income tax of Rs 24,646.96 crores on the Aamby Valley Limited,\" the spokesperson said in a brief statement. Officials said the notice was issued by the taxman in January this year after the special audit of AVLs income for the Assessment Year 2012-13 found that the parent firm had allegedly floated a clutch of Special Purpose Vehicles whose incomes were later accounted on the account of AVL as they were merged with the former in due course of time. The AVL, in its income return filed for AY 2012-13, had reflected a loss of few crores but the special I-T audit brought up the added income, a senior official said. The Supreme Court, last week, had asked the Bombay High Courts official liquidator to sell the Rs 34,000 crore worth of properties of Aamby Valley owned by the Sahara Group and directed its chief Subrata Roy to personally appear before it on April 28.  \n",
        "\n",
        "\n",
        "**Original Summary**\n",
        "The Income Tax Department has issued a ?24,646 crore tax demand notice to Sahara Group's Aamby Valley Limited. The department's audit found that an income of over ?48,000 crore for the assessment year 2012-13 was not reflected in the record books of the firm. A week ago, the SC ordered Bombay HC to auction Sahara's Aamby Valley worth ?34,000 crore.\n",
        "\n",
        "**Generated Summary**\n",
        "the Income Tax department has issued a?24,646 crore tax demand notice to Sahara Groups Aamby Valley Limited (AVL) after conducting an audit of the company. The notice was issued in January this year after the special audit found that the parent firm had floated Special Purpose Vehicle income for the Assessment Year 2012-13 and later accounted on its account as they were merged with the former. \"Yes...the Income Tax Department raised Rs48,085.79 crores to the income,\" he added earlier said at the notice."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "GdTZvSB6x4hQ"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "Working_transformers_summarization_wandb.ipynb",
      "provenance": [],
      "machine_shape": "hm"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.7"
    },
    "varInspector": {
      "cols": {
        "lenName": 16,
        "lenType": 16,
        "lenVar": 40
      },
      "kernels_config": {
        "python": {
          "delete_cmd_postfix": "",
          "delete_cmd_prefix": "del ",
          "library": "var_list.py",
          "varRefreshCmd": "print(var_dic_list())"
        },
        "r": {
          "delete_cmd_postfix": ") ",
          "delete_cmd_prefix": "rm(",
          "library": "var_list.r",
          "varRefreshCmd": "cat(var_dic_list()) "
        }
      },
      "types_to_exclude": [
        "module",
        "function",
        "builtin_function_or_method",
        "instance",
        "_Feature"
      ],
      "window_display": false
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "61b93cde55f54af28c9e85adc0570d43": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "VBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "VBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "VBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_8127d3d4d61c4407b608bf8c28b873d2",
              "IPY_MODEL_a8b09d03fbe441fdbf641703d0ac0ccb"
            ],
            "layout": "IPY_MODEL_542d51ab91324ee8b0baf156c52a2842"
          }
        },
        "8127d3d4d61c4407b608bf8c28b873d2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "LabelModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "LabelModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "LabelView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_61c857a1f25943ad92d1d137fff02d80",
            "placeholder": "​",
            "style": "IPY_MODEL_37661820d4ee4f23a77382cf409e2edb",
            "value": "0.009 MB of 0.009 MB uploaded (0.000 MB deduped)\r"
          }
        },
        "a8b09d03fbe441fdbf641703d0ac0ccb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1ef42899a12d47ffbee2d6358d608264",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_fe722548b8f84d4f90baa96a4f692e13",
            "value": 1
          }
        },
        "542d51ab91324ee8b0baf156c52a2842": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "61c857a1f25943ad92d1d137fff02d80": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "37661820d4ee4f23a77382cf409e2edb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "1ef42899a12d47ffbee2d6358d608264": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fe722548b8f84d4f90baa96a4f692e13": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "73797baf2c1840a48647a8a9edfd1fa7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_f8ad0ad7b2b9473c80463091a286460d",
              "IPY_MODEL_290a7b5c3d814e23a86913bfec387fa8",
              "IPY_MODEL_48d5643c543241cfb7fe133c6ed316b6"
            ],
            "layout": "IPY_MODEL_97940b58e151452c86c401ab83584a6b"
          }
        },
        "f8ad0ad7b2b9473c80463091a286460d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_635e4c298c314881a9259730600a17a9",
            "placeholder": "​",
            "style": "IPY_MODEL_926c592dc4bb42648a385eae789054f4",
            "value": "Downloading: 100%"
          }
        },
        "290a7b5c3d814e23a86913bfec387fa8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8ca58cead9fa44919e6855465780cadb",
            "max": 1230,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_7505defd1fd64753829a1223c107c530",
            "value": 1230
          }
        },
        "48d5643c543241cfb7fe133c6ed316b6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_fa5c0d651d9e425ca4ad157dde500981",
            "placeholder": "​",
            "style": "IPY_MODEL_b960d07fe7314a27beeab0c2b71158e5",
            "value": " 1.20k/1.20k [00:00&lt;00:00, 50.0kB/s]"
          }
        },
        "97940b58e151452c86c401ab83584a6b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "635e4c298c314881a9259730600a17a9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "926c592dc4bb42648a385eae789054f4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8ca58cead9fa44919e6855465780cadb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7505defd1fd64753829a1223c107c530": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "fa5c0d651d9e425ca4ad157dde500981": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b960d07fe7314a27beeab0c2b71158e5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "73c2bab7716d4caf937baa176fb2d9f5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_3d341817cf034fa9a0aba0eff55482b3",
              "IPY_MODEL_b8bf1f0c8b914fb88bc359edb2b7357a",
              "IPY_MODEL_f88c520b80d248149a7c3c248477d7fa"
            ],
            "layout": "IPY_MODEL_35688b55437149d29db0763d0a8a92c1"
          }
        },
        "3d341817cf034fa9a0aba0eff55482b3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ef8843d3328d4ee6957596953a712f62",
            "placeholder": "​",
            "style": "IPY_MODEL_ac5ca22dc3e249669f2692f02cacd78b",
            "value": "Downloading: 100%"
          }
        },
        "b8bf1f0c8b914fb88bc359edb2b7357a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_820911dfa8764d93a17b6f72a6b8b0ee",
            "max": 1187797362,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_049c40fdbfc64ae59f6b6add33f06886",
            "value": 1187797362
          }
        },
        "f88c520b80d248149a7c3c248477d7fa": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_57b6c5e71d504157adfc93322fd754f5",
            "placeholder": "​",
            "style": "IPY_MODEL_36bc524c1916404aa5221610f4c36572",
            "value": " 1.11G/1.11G [00:27&lt;00:00, 38.0MB/s]"
          }
        },
        "35688b55437149d29db0763d0a8a92c1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ef8843d3328d4ee6957596953a712f62": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ac5ca22dc3e249669f2692f02cacd78b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "820911dfa8764d93a17b6f72a6b8b0ee": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "049c40fdbfc64ae59f6b6add33f06886": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "57b6c5e71d504157adfc93322fd754f5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "36bc524c1916404aa5221610f4c36572": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}