joey_demo.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Kopie von joey-demo.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/joeynmt/joeynmt/blob/main/joey_demo.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SjX7C7AHX-8F"
      },
      "source": [
        "# Joey NMT Demo\n",
        "\n",
        "In this notebook, we'll train a Transformer model for translating between simple sentences in Esperanto (*epo*) and English (*eng*). You'll have the option to choose your own languages as well.\n",
        "\n",
        "**Important:** Before you start, set runtime type to GPU.\n",
        "\n",
        "Author: Julia Kreutzer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QYRtQZFPYzES"
      },
      "source": [
        "## Installation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y4VofGiF2MMr"
      },
      "source": [
        "Install JoeyNMT via [pip](https://pypi.org/project/joeynmt/).\n",
        "\n",
        "It works on PyTorch version 1.10.1. If that's not automatically installed, run the following command to install it.\n",
        "\n",
        "\n",
        "\n",
        "```\n",
        "!pip install torch==1.10.1+cu102 -f https://download.pytorch.org/whl/torch_stable.html\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a4u741jiY2-O",
        "outputId": "06af4a3b-9461-4c9d-cebf-17398fca0971"
      },
      "source": [
        "!pip install joeynmt"
      ],
      "execution_count": null,
      "outputs": [
        {
          "metadata": {
            "tags": null
          },
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: joeynmt in /usr/local/lib/python3.7/dist-packages (1.4)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from joeynmt) (57.4.0)\n",
            "Requirement already satisfied: numpy>=1.19.5 in /usr/local/lib/python3.7/dist-packages (from joeynmt) (1.19.5)\n",
            "Requirement already satisfied: tensorboard>=1.15 in /usr/local/lib/python3.7/dist-packages (from joeynmt) (2.7.0)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from joeynmt) (0.16.0)\n",
            "Requirement already satisfied: torchtext>=0.10.0 in /usr/local/lib/python3.7/dist-packages (from joeynmt) (0.11.0)\n",
            "Requirement already satisfied: subword-nmt in /usr/local/lib/python3.7/dist-packages (from joeynmt) (0.3.8)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from joeynmt) (3.2.2)\n",
            "Requirement already satisfied: torch>=1.9.0 in /usr/local/lib/python3.7/dist-packages (from joeynmt) (1.10.0)\n",
            "Requirement already satisfied: pylint>=2.9.6 in /usr/local/lib/python3.7/dist-packages (from joeynmt) (2.12.2)\n",
            "Requirement already satisfied: sacrebleu>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from joeynmt) (2.0.0)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.7/dist-packages (from joeynmt) (7.1.2)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.7/dist-packages (from joeynmt) (6.0)\n",
            "Requirement already satisfied: seaborn in /usr/local/lib/python3.7/dist-packages (from joeynmt) (0.11.2)\n",
            "Requirement already satisfied: wrapt==1.11.1 in /usr/local/lib/python3.7/dist-packages (from joeynmt) (1.11.1)\n",
            "Requirement already satisfied: six==1.12 in /usr/local/lib/python3.7/dist-packages (from joeynmt) (1.12.0)\n",
            "Requirement already satisfied: astroid<2.10,>=2.9.0 in /usr/local/lib/python3.7/dist-packages (from pylint>=2.9.6->joeynmt) (2.9.3)\n",
            "Requirement already satisfied: typing-extensions>=3.10.0 in /usr/local/lib/python3.7/dist-packages (from pylint>=2.9.6->joeynmt) (3.10.0.2)\n",
            "Requirement already satisfied: platformdirs>=2.2.0 in /usr/local/lib/python3.7/dist-packages (from pylint>=2.9.6->joeynmt) (2.4.1)\n",
            "Requirement already satisfied: mccabe<0.7,>=0.6 in /usr/local/lib/python3.7/dist-packages (from pylint>=2.9.6->joeynmt) (0.6.1)\n",
            "Requirement already satisfied: isort<6,>=4.2.5 in /usr/local/lib/python3.7/dist-packages (from pylint>=2.9.6->joeynmt) (5.10.1)\n",
            "Requirement already satisfied: toml>=0.9.2 in /usr/local/lib/python3.7/dist-packages (from pylint>=2.9.6->joeynmt) (0.10.2)\n",
            "Requirement already satisfied: typed-ast<2.0,>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from astroid<2.10,>=2.9.0->pylint>=2.9.6->joeynmt) (1.5.1)\n",
            "Requirement already satisfied: lazy-object-proxy>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from astroid<2.10,>=2.9.0->pylint>=2.9.6->joeynmt) (1.7.1)\n",
            "Requirement already satisfied: colorama in /usr/local/lib/python3.7/dist-packages (from sacrebleu>=2.0.0->joeynmt) (0.4.4)\n",
            "Requirement already satisfied: portalocker in /usr/local/lib/python3.7/dist-packages (from sacrebleu>=2.0.0->joeynmt) (2.3.2)\n",
            "Requirement already satisfied: tabulate>=0.8.9 in /usr/local/lib/python3.7/dist-packages (from sacrebleu>=2.0.0->joeynmt) (0.8.9)\n",
            "Requirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from sacrebleu>=2.0.0->joeynmt) (2019.12.20)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=1.15->joeynmt) (3.3.6)\n",
            "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=1.15->joeynmt) (0.6.1)\n",
            "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=1.15->joeynmt) (0.12.0)\n",
            "Requirement already satisfied: grpcio>=1.24.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=1.15->joeynmt) (1.43.0)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=1.15->joeynmt) (1.8.1)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=1.15->joeynmt) (1.0.1)\n",
            "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=1.15->joeynmt) (1.35.0)\n",
            "Requirement already satisfied: protobuf>=3.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=1.15->joeynmt) (3.17.3)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=1.15->joeynmt) (0.4.6)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=1.15->joeynmt) (0.37.1)\n",
            "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=1.15->joeynmt) (2.23.0)\n",
            "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=1.15->joeynmt) (4.2.4)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=1.15->joeynmt) (4.8)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=1.15->joeynmt) (0.2.8)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=1.15->joeynmt) (1.3.0)\n",
            "Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard>=1.15->joeynmt) (4.10.0)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard>=1.15->joeynmt) (3.7.0)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard>=1.15->joeynmt) (0.4.8)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard>=1.15->joeynmt) (2021.10.8)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard>=1.15->joeynmt) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard>=1.15->joeynmt) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard>=1.15->joeynmt) (2.10)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=1.15->joeynmt) (3.1.1)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from torchtext>=0.10.0->joeynmt) (4.62.3)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->joeynmt) (1.3.2)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->joeynmt) (0.11.0)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->joeynmt) (2.8.2)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->joeynmt) (3.0.6)\n",
            "Requirement already satisfied: scipy>=1.0 in /usr/local/lib/python3.7/dist-packages (from seaborn->joeynmt) (1.4.1)\n",
            "Requirement already satisfied: pandas>=0.23 in /usr/local/lib/python3.7/dist-packages (from seaborn->joeynmt) (1.1.5)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.23->seaborn->joeynmt) (2018.9)\n",
            "Requirement already satisfied: mock in /usr/local/lib/python3.7/dist-packages (from subword-nmt->joeynmt) (4.0.3)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HrcFeSdabKHw"
      },
      "source": [
        "# Data Preparation\n",
        "\n",
        "We'll use English - Esperanto translations from the Tatoeba challenge. \n",
        "\n",
        "If you want to use a different language pair, you'll need to replace all instances of `eng` and `epo` language identifiers to your languages of choice. You can find all available languages [here](https://github.com/Helsinki-NLP/Tatoeba-Challenge/blob/master/Data.md). Note that if the training data size is larger, data preparation and training might take much longer than the example with only 400k sentence pairs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dF1998yPmtbS"
      },
      "source": [
        "## Download"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Z6cux99eZ-gW",
        "outputId": "2f242519-e997-48e0-e877-3c7ab49bfcd5"
      },
      "source": [
        "!wget https://object.pouta.csc.fi/Tatoeba-Challenge/eng-epo.tar"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2022-01-18 02:46:19--  https://object.pouta.csc.fi/Tatoeba-Challenge/eng-epo.tar\n",
            "Resolving object.pouta.csc.fi (object.pouta.csc.fi)... 86.50.254.18, 86.50.254.19\n",
            "Connecting to object.pouta.csc.fi (object.pouta.csc.fi)|86.50.254.18|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 35747840 (34M) [application/x-tar]\n",
            "Saving to: ‘eng-epo.tar’\n",
            "\n",
            "eng-epo.tar         100%[===================>]  34.09M  17.0MB/s    in 2.0s    \n",
            "\n",
            "2022-01-18 02:46:22 (17.0 MB/s) - ‘eng-epo.tar’ saved [35747840/35747840]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "R_kVvMcBbgLg",
        "outputId": "b7f20cea-d837-4304-9a5c-769211cee23f"
      },
      "source": [
        "!tar -xvf  'eng-epo.tar'"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "data/eng-epo/\n",
            "data/eng-epo/train.src.gz\n",
            "data/eng-epo/dev.trg\n",
            "data/eng-epo/train.id.gz\n",
            "data/eng-epo/test.trg\n",
            "data/eng-epo/test.id\n",
            "data/eng-epo/dev.src\n",
            "data/eng-epo/dev.id\n",
            "data/eng-epo/test.src\n",
            "data/eng-epo/train.trg.gz\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WeIGVD2rcfFD"
      },
      "source": [
        "!gunzip 'data/eng-epo/train.src.gz'\n",
        "!gunzip 'data/eng-epo/train.trg.gz'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LBVyCK9aiPcE"
      },
      "source": [
        "We'll only use a subset of dev and test data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yHm_0HlIeshK"
      },
      "source": [
        "!mv 'data/eng-epo/train.src' 'data/eng-epo/train.eng'\n",
        "!head -n 1000 'data/eng-epo/dev.src' > 'data/eng-epo/dev.eng'\n",
        "!head -n 1000 'data/eng-epo/test.src' > 'data/eng-epo/test.eng'\n",
        "!mv 'data/eng-epo/train.trg' 'data/eng-epo/train.epo'\n",
        "!head -n 1000 'data/eng-epo/dev.trg' > 'data/eng-epo/dev.epo'\n",
        "!head -n 1000 'data/eng-epo/test.trg' > 'data/eng-epo/test.epo'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FoveftbNCyv8"
      },
      "source": [
        "The data is sentence-aligned, that means that source and target file contain one sentence per line which correspond to each other."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FLNgutcOcHK2",
        "outputId": "61b9ef6c-22c3-46f9-9a83-af60107ca52c"
      },
      "source": [
        "! head data/eng-epo/dev.eng\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "I made you something.\n",
            "Twice two is four.\n",
            "That hurts! Stop it!\n",
            "It is too much for me. I need to slow down.\n",
            "I never want to see him again.\n",
            "At what hour was she born?\n",
            "The traffic jam lasted one hour.\n",
            "After dinner, we played cards till eleven.\n",
            "I doubt if he is a lawyer.\n",
            "Who's on duty today?\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BrP5Ujk0cXxA",
        "outputId": "4d101ae3-ced2-49e8-f9dd-1db0631540d1"
      },
      "source": [
        "! head data/eng-epo/dev.epo"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mi faris ion por vi.\n",
            "Du oble du faras kvar.\n",
            "Tio suferigas min! Ĉesu!\n",
            "Estas tro multe por mi. Mi devas malrapidiĝi.\n",
            "Mi volas neniam plu vidi lin.\n",
            "Je kioma horo ŝi naskiĝis?\n",
            "La trafikmalfluo daŭris unu horon.\n",
            "Post la vespermanĝo ni kartludis ĝis la dudek tria.\n",
            "Mi dubas ĉu li estas advokato.\n",
            "Kiu deĵoras hodiaŭ?\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ydKB8OS1c0r1",
        "outputId": "328dc8ba-85cc-49b3-9a3a-8fbaada1633e"
      },
      "source": [
        "! wc -l data/eng-epo/*"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "     4001 data/eng-epo/bpe.codes.4000\n",
            "     1000 data/eng-epo/dev.4000.bpe.eng\n",
            "     1000 data/eng-epo/dev.4000.bpe.epo\n",
            "     1000 data/eng-epo/dev.eng\n",
            "     1000 data/eng-epo/dev.epo\n",
            "   235834 data/eng-epo/dev.id\n",
            "   235834 data/eng-epo/dev.src\n",
            "   235834 data/eng-epo/dev.trg\n",
            "     4540 data/eng-epo/joint.4000bpe.vocab\n",
            "     1000 data/eng-epo/test.4000.bpe.eng\n",
            "     1000 data/eng-epo/test.4000.bpe.epo\n",
            "     1000 data/eng-epo/test.eng\n",
            "     1000 data/eng-epo/test.epo\n",
            "    10000 data/eng-epo/test.id\n",
            "    10000 data/eng-epo/test.src\n",
            "    10000 data/eng-epo/test.trg\n",
            "   402180 data/eng-epo/train.4000.bpe.eng\n",
            "   402180 data/eng-epo/train.4000.bpe.epo\n",
            "   402180 data/eng-epo/train.eng\n",
            "   402180 data/eng-epo/train.epo\n",
            "   804360 data/eng-epo/train.epo-eng\n",
            "     1389 data/eng-epo/train.id.gz\n",
            "  3168512 total\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WZg4k1rem1jI"
      },
      "source": [
        "## Subword model training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vJl9sQq22c5Z"
      },
      "source": [
        "We will use the `subword_nmt` library to split words into subwords (BPE) according to their frequency in the training corpus."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tGDmK7rnqc6r"
      },
      "source": [
        "import os"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2wNsekzFd1BD"
      },
      "source": [
        "src_lang = 'epo'\n",
        "trg_lang = 'eng'\n",
        "bpe_size = 4000\n",
        "datadir = '/content/data/eng-epo/'\n",
        "name = f'{src_lang}_{trg_lang}_bpe{bpe_size}'\n",
        "\n",
        "\n",
        "train_src_file = os.path.join(datadir, f'train.{src_lang}')\n",
        "train_trg_file = os.path.join(datadir, f'train.{trg_lang}')\n",
        "train_joint_file = os.path.join(datadir, f'train.{src_lang}-{trg_lang}')\n",
        "dev_src_file = os.path.join(datadir, f'dev.{src_lang}')\n",
        "dev_trg_file = os.path.join(datadir, f'dev.{trg_lang}')\n",
        "test_src_file = os.path.join(datadir, f'test.{src_lang}')\n",
        "test_trg_file = os.path.join(datadir, f'test.{trg_lang}')\n",
        "src_files = {'train': train_src_file, 'dev': dev_src_file, 'test': test_src_file}\n",
        "trg_files = {'train': train_trg_file, 'dev': dev_trg_file, 'test': test_trg_file}\n",
        "\n",
        "\n",
        "vocab_src_file = os.path.join(datadir, f'vocab.{bpe_size}.{src_lang}')\n",
        "vocab_trg_file = os.path.join(datadir, f'vocab.{bpe_size}.{trg_lang}')\n",
        "bpe_file = os.path.join(datadir, f'bpe.codes.{bpe_size}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oJPV_l_G2ny1"
      },
      "source": [
        "Train a BPE model with 4000 symbols for both languages jointly."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eVYh30Mjm3zu",
        "outputId": "0a819d01-d9f2-46cd-8340-85cba00535ab",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "! cat $train_src_file $train_trg_file > $train_joint_file\n",
        "\n",
        "! subword-nmt learn-bpe \\\n",
        "  --input $train_joint_file \\\n",
        "  -s $bpe_size \\\n",
        "  -o $bpe_file"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "100% 4000/4000 [00:26<00:00, 149.43it/s]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QEQjXFqv2u-3"
      },
      "source": [
        "This file contains the merges of character sequences that subwords are made of."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZtUq1rg1sA8H",
        "outputId": "adb0f0af-e5c2-4f31-fd61-1cc9e8bbd0df"
      },
      "source": [
        "! head $bpe_file"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "#version: 0.2\n",
            "t h\n",
            "a n\n",
            "o n\n",
            "e r\n",
            "i n\n",
            "e n\n",
            "s t\n",
            "l a</w>\n",
            "o r\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tWdUshWa2z3V"
      },
      "source": [
        "We apply the learned BPE merges to training, development and test data.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qgdj1d3wpA1c"
      },
      "source": [
        "src_bpe_files = {}\n",
        "trg_bpe_files = {}\n",
        "for split in ['train', 'dev', 'test']:\n",
        "  src_input_file = src_files[split]\n",
        "  trg_input_file = trg_files[split]\n",
        "  src_output_file = src_input_file.replace(split, f'{split}.{bpe_size}.bpe')\n",
        "  trg_output_file = trg_input_file.replace(split, f'{split}.{bpe_size}.bpe')\n",
        "  src_bpe_files[split] = src_output_file\n",
        "  trg_bpe_files[split] = trg_output_file\n",
        "\n",
        "  ! subword-nmt apply-bpe \\\n",
        "    -c $bpe_file \\\n",
        "    < $src_input_file > $src_output_file\n",
        "\n",
        "  ! subword-nmt apply-bpe \\\n",
        "    -c $bpe_file \\\n",
        "    < $trg_input_file > $trg_output_file\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2KrKE20C27z3"
      },
      "source": [
        "The subword-split data contains `@@ ` to indicate where words were split into subwords."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SZvrbM5Qx418",
        "outputId": "d04e45b6-f69d-43cb-af93-7f33301f537c"
      },
      "source": [
        "! head data/eng-epo/dev.4000.bpe.eng"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "I made you some@@ thing.\n",
            "T@@ w@@ ice two is f@@ our@@ .\n",
            "That h@@ ur@@ t@@ s! Stop it@@ !\n",
            "It is too much for me. I need to s@@ low dow@@ n.\n",
            "I never want to see him again@@ .\n",
            "A@@ t what h@@ our was she bor@@ n@@ ?\n",
            "The traf@@ fi@@ c jam last@@ ed one h@@ our@@ .\n",
            "Af@@ ter d@@ in@@ n@@ er, we played c@@ ards til@@ l el@@ ev@@ en.\n",
            "I d@@ ou@@ b@@ t if he is a law@@ y@@ er.\n",
            "Wh@@ o@@ 's on d@@ ut@@ y t@@ od@@ ay@@ ?\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "g3uCvTzuyEqo",
        "outputId": "65c9af91-79bd-4742-9640-ad8c86dd848f"
      },
      "source": [
        "! head data/eng-epo/dev.4000.bpe.epo"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mi faris ion por vi.\n",
            "D@@ u o@@ ble du faras kvar@@ .\n",
            "Tio sufer@@ igas min@@ ! Ĉes@@ u!\n",
            "Estas tro multe por mi. Mi devas mal@@ rapi@@ di@@ ĝ@@ i.\n",
            "Mi volas neniam plu vidi lin.\n",
            "J@@ e ki@@ om@@ a h@@ oro ŝi n@@ aski@@ ĝ@@ is?\n",
            "La tra@@ fik@@ mal@@ fl@@ uo daŭ@@ ris unu hor@@ on.\n",
            "Post la vesper@@ man@@ ĝo ni kart@@ lud@@ is ĝis la dudek tri@@ a.\n",
            "Mi du@@ b@@ as ĉu li estas ad@@ vok@@ at@@ o.\n",
            "Kiu de@@ ĵ@@ or@@ as hodi@@ aŭ@@ ?\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7jS8RHLZyKKf"
      },
      "source": [
        "## Prepare the vocabulary"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ndkGOp7F3LWY"
      },
      "source": [
        "From the pre-processed training data, we extract the final vocabulary for the translation model. It should contain all subwords needed for representing the source and target training data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AGq8KMqjySXm",
        "outputId": "1fc443a0-37c5-4cb8-d9ec-707ec5a160be"
      },
      "source": [
        "! wget https://raw.githubusercontent.com/joeynmt/joeynmt/main/scripts/build_vocab.py"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2022-01-18 02:48:20--  https://raw.githubusercontent.com/joeynmt/joeynmt/main/scripts/build_vocab.py\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 2034 (2.0K) [text/plain]\n",
            "Saving to: ‘build_vocab.py’\n",
            "\n",
            "\rbuild_vocab.py        0%[                    ]       0  --.-KB/s               \rbuild_vocab.py      100%[===================>]   1.99K  --.-KB/s    in 0s      \n",
            "\n",
            "2022-01-18 02:48:20 (41.8 MB/s) - ‘build_vocab.py’ saved [2034/2034]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k1_iQEQEyJyS"
      },
      "source": [
        "vocab_src_file = src_bpe_files['train']\n",
        "vocab_trg_file = trg_bpe_files['train']\n",
        "bpe_vocab_file = os.path.join(datadir, f'joint.{bpe_size}bpe.vocab')\n",
        "\n",
        "! python build_vocab.py  \\\n",
        "  $vocab_src_file $vocab_trg_file \\\n",
        "  --output_path $bpe_vocab_file"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XSuYid3JdECc"
      },
      "source": [
        "# Model configuration"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "45uGP83v3Y24"
      },
      "source": [
        "Joey NMT reads model and training hyperparameters from a configuration file. We're generating this now to configure paths in the appropriate places. \n",
        "\n",
        "The configuration below builds a small Transformer model with shared embeddings between source and target language on the base of the subword vocabularies created above."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EirEmJmkc7sx"
      },
      "source": [
        "# Create the config\n",
        "config = \"\"\"\n",
        "name: \"{name}_transformer\"\n",
        "\n",
        "data:\n",
        "    src: \"{source_language}\"\n",
        "    trg: \"{target_language}\"\n",
        "    train: \"{datadir}/train.{bpe_size}.bpe\"\n",
        "    dev:   \"{datadir}/dev.{bpe_size}.bpe\"\n",
        "    test:  \"{datadir}/test.{bpe_size}.bpe\"\n",
        "    level: \"bpe\"\n",
        "    lowercase: False                \n",
        "    max_sent_length: 30             # Extend to longer sentences.\n",
        "    src_vocab: \"{vocab_src_file}\"\n",
        "    trg_vocab: \"{vocab_trg_file}\"\n",
        "\n",
        "testing:\n",
        "    beam_size: 5\n",
        "    alpha: 1.0\n",
        "    sacrebleu:                      # sacrebleu options\n",
        "        remove_whitespace: True     # `remove_whitespace` option in sacrebleu.corpus_chrf() function (defalut: True)\n",
        "        tokenize: \"intl\"            # `tokenize` option in sacrebleu.corpus_bleu() function (options include: \"none\" (use for already tokenized test data), \"13a\" (default minimal tokenizer), \"intl\" which mostly does punctuation and unicode, etc) \n",
        "\n",
        "training:\n",
        "    #load_model: \"models/{name}_transformer/1.ckpt\" # if uncommented, load a pre-trained model from this checkpoint\n",
        "    random_seed: 42\n",
        "    optimizer: \"adam\"\n",
        "    normalization: \"tokens\"\n",
        "    adam_betas: [0.9, 0.999] \n",
        "    scheduling: \"plateau\"           # Alternative: try switching from plateau to Noam scheduling\n",
        "    patience: 5                     # For plateau: decrease learning rate by decrease_factor if validation score has not improved for this many validation rounds.\n",
        "    learning_rate_factor: 0.5       # factor for Noam scheduler (used with Transformer)\n",
        "    learning_rate_warmup: 1000      # warmup steps for Noam scheduler (used with Transformer)\n",
        "    decrease_factor: 0.7\n",
        "    loss: \"crossentropy\"\n",
        "    learning_rate: 0.0003\n",
        "    learning_rate_min: 0.00000001\n",
        "    weight_decay: 0.0\n",
        "    label_smoothing: 0.1\n",
        "    batch_size: 4096\n",
        "    batch_type: \"token\"\n",
        "    eval_batch_size: 3600\n",
        "    eval_batch_type: \"token\"\n",
        "    batch_multiplier: 1\n",
        "    early_stopping_metric: \"ppl\"\n",
        "    epochs: 30                     # Decrease for when playing around and checking of working. Around 30 is sufficient to check if its working at all\n",
        "    validation_freq: 2000          # Set to at least once per epoch.\n",
        "    logging_freq: 200\n",
        "    eval_metric: \"bleu\"\n",
        "    model_dir: \"models/{name}_transformer\"\n",
        "    overwrite: False               # Set to True if you want to overwrite possibly existing models. \n",
        "    shuffle: True\n",
        "    use_cuda: True\n",
        "    max_output_length: 100\n",
        "    print_valid_sents: [0, 1, 2, 3]\n",
        "    keep_best_ckpts: 3\n",
        "\n",
        "model:\n",
        "    initializer: \"xavier\"\n",
        "    bias_initializer: \"zeros\"\n",
        "    init_gain: 1.0\n",
        "    embed_initializer: \"xavier\"\n",
        "    embed_init_gain: 1.0\n",
        "    tied_embeddings: True       # Requires joint vocabulary.\n",
        "    tied_softmax: True\n",
        "    encoder:\n",
        "        type: \"transformer\"\n",
        "        num_layers: 6\n",
        "        num_heads: 4             # Increase to 8 for larger data.\n",
        "        embeddings:\n",
        "            embedding_dim: 256   # Increase to 512 for larger data.\n",
        "            scale: True\n",
        "            dropout: 0.2\n",
        "        # typically ff_size = 4 x hidden_size\n",
        "        hidden_size: 256         # Increase to 512 for larger data.\n",
        "        ff_size: 1024            # Increase to 2048 for larger data.\n",
        "        dropout: 0.3\n",
        "    decoder:\n",
        "        type: \"transformer\"\n",
        "        num_layers: 6\n",
        "        num_heads: 4              # Increase to 8 for larger data.\n",
        "        embeddings:\n",
        "            embedding_dim: 256    # Increase to 512 for larger data.\n",
        "            scale: True\n",
        "            dropout: 0.2\n",
        "        # typically ff_size = 4 x hidden_size\n",
        "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
        "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
        "        dropout: 0.3\n",
        "\"\"\".format(name=name, source_language=src_lang, target_language=trg_lang,\n",
        "           datadir=datadir, vocab_src_file=bpe_vocab_file, \n",
        "           vocab_trg_file=bpe_vocab_file, bpe_size=bpe_size)\n",
        "with open(\"transformer_{name}.yaml\".format(name=name),'w') as f:\n",
        "    f.write(config)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pIOosBx1fDIQ"
      },
      "source": [
        "# Training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D20-6ecg4PvC"
      },
      "source": [
        "This will take a while. The log reports the training process, look out for the prints of example translations and the BLEU evaluation scores to get an impression of the current quality. \n",
        "\n",
        "The log is also stored in the model directory within this runtime (inspect files in the menu on the left). There you can also find a summary report of all validations. We'll also use TensorBoard to visualize the training progress on the go. This requires enabling Cookies in the browser.\n",
        "\n",
        "After 12h at the latest, Colab will disconnect, so to make sure you're progress is not lost, download the checkpoints from the model directory from time to time. You'll later be able to reload them if model hyperparameters match."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xGzLfqzQEqq9"
      },
      "source": [
        "# Load the TensorBoard notebook extension. It will be empty at first.\n",
        "%load_ext tensorboard"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 838
        },
        "id": "9QYsZ513DuFi",
        "outputId": "39b5a7c3-081a-49dc-bed4-b03d2ffbc026"
      },
      "source": [
        "%tensorboard --logdir models/epo_eng_bpe4000_transformer/tensorboard "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Reusing TensorBoard on port 6006 (pid 274), started 0:18:13 ago. (Use '!kill 274' to kill it.)"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "\n",
              "        (async () => {\n",
              "            const url = new URL(await google.colab.kernel.proxyPort(6006, {'cache': true}));\n",
              "            url.searchParams.set('tensorboardColab', 'true');\n",
              "            const iframe = document.createElement('iframe');\n",
              "            iframe.src = url;\n",
              "            iframe.setAttribute('width', '100%');\n",
              "            iframe.setAttribute('height', '800');\n",
              "            iframe.setAttribute('frameborder', 0);\n",
              "            document.body.appendChild(iframe);\n",
              "        })();\n",
              "    "
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FF9do6ohedY6",
        "outputId": "7545f77f-f466-4d5b-a101-e4307ea46655"
      },
      "source": [
        "!python -m joeynmt train transformer_epo_eng_bpe4000.yaml"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2022-01-18 02:50:08,875 - INFO - root - Hello! This is Joey-NMT (version 1.4).\n",
            "2022-01-18 02:50:08,906 - INFO - joeynmt.data - Loading training data...\n",
            "2022-01-18 02:50:14,401 - INFO - joeynmt.data - Building vocabulary...\n",
            "2022-01-18 02:50:14,708 - INFO - joeynmt.data - Loading dev data...\n",
            "2022-01-18 02:50:14,733 - INFO - joeynmt.data - Loading test data...\n",
            "2022-01-18 02:50:14,757 - INFO - joeynmt.data - Data loaded.\n",
            "2022-01-18 02:50:14,757 - INFO - joeynmt.model - Building an encoder-decoder model...\n",
            "2022-01-18 02:50:14,967 - INFO - joeynmt.model - Enc-dec model built.\n",
            "2022-01-18 02:50:16,595 - INFO - joeynmt.training - Total params: 12223488\n",
            "2022-01-18 02:50:20,051 - INFO - joeynmt.helpers -                           cfg.name : epo_eng_bpe4000_transformer\n",
            "2022-01-18 02:50:20,052 - INFO - joeynmt.helpers -                       cfg.data.src : epo\n",
            "2022-01-18 02:50:20,052 - INFO - joeynmt.helpers -                       cfg.data.trg : eng\n",
            "2022-01-18 02:50:20,052 - INFO - joeynmt.helpers -                     cfg.data.train : /content/data/eng-epo//train.4000.bpe\n",
            "2022-01-18 02:50:20,052 - INFO - joeynmt.helpers -                       cfg.data.dev : /content/data/eng-epo//dev.4000.bpe\n",
            "2022-01-18 02:50:20,052 - INFO - joeynmt.helpers -                      cfg.data.test : /content/data/eng-epo//test.4000.bpe\n",
            "2022-01-18 02:50:20,052 - INFO - joeynmt.helpers -                     cfg.data.level : bpe\n",
            "2022-01-18 02:50:20,052 - INFO - joeynmt.helpers -                 cfg.data.lowercase : False\n",
            "2022-01-18 02:50:20,052 - INFO - joeynmt.helpers -           cfg.data.max_sent_length : 30\n",
            "2022-01-18 02:50:20,052 - INFO - joeynmt.helpers -                 cfg.data.src_vocab : /content/data/eng-epo/joint.4000bpe.vocab\n",
            "2022-01-18 02:50:20,052 - INFO - joeynmt.helpers -                 cfg.data.trg_vocab : /content/data/eng-epo/joint.4000bpe.vocab\n",
            "2022-01-18 02:50:20,052 - INFO - joeynmt.helpers -              cfg.testing.beam_size : 5\n",
            "2022-01-18 02:50:20,052 - INFO - joeynmt.helpers -                  cfg.testing.alpha : 1.0\n",
            "2022-01-18 02:50:20,052 - INFO - joeynmt.helpers - cfg.testing.sacrebleu.remove_whitespace : True\n",
            "2022-01-18 02:50:20,052 - INFO - joeynmt.helpers -     cfg.testing.sacrebleu.tokenize : intl\n",
            "2022-01-18 02:50:20,053 - INFO - joeynmt.helpers -           cfg.training.random_seed : 42\n",
            "2022-01-18 02:50:20,053 - INFO - joeynmt.helpers -             cfg.training.optimizer : adam\n",
            "2022-01-18 02:50:20,053 - INFO - joeynmt.helpers -         cfg.training.normalization : tokens\n",
            "2022-01-18 02:50:20,053 - INFO - joeynmt.helpers -            cfg.training.adam_betas : [0.9, 0.999]\n",
            "2022-01-18 02:50:20,053 - INFO - joeynmt.helpers -            cfg.training.scheduling : plateau\n",
            "2022-01-18 02:50:20,053 - INFO - joeynmt.helpers -              cfg.training.patience : 5\n",
            "2022-01-18 02:50:20,053 - INFO - joeynmt.helpers -  cfg.training.learning_rate_factor : 0.5\n",
            "2022-01-18 02:50:20,053 - INFO - joeynmt.helpers -  cfg.training.learning_rate_warmup : 1000\n",
            "2022-01-18 02:50:20,053 - INFO - joeynmt.helpers -       cfg.training.decrease_factor : 0.7\n",
            "2022-01-18 02:50:20,053 - INFO - joeynmt.helpers -                  cfg.training.loss : crossentropy\n",
            "2022-01-18 02:50:20,053 - INFO - joeynmt.helpers -         cfg.training.learning_rate : 0.0003\n",
            "2022-01-18 02:50:20,053 - INFO - joeynmt.helpers -     cfg.training.learning_rate_min : 1e-08\n",
            "2022-01-18 02:50:20,053 - INFO - joeynmt.helpers -          cfg.training.weight_decay : 0.0\n",
            "2022-01-18 02:50:20,053 - INFO - joeynmt.helpers -       cfg.training.label_smoothing : 0.1\n",
            "2022-01-18 02:50:20,053 - INFO - joeynmt.helpers -            cfg.training.batch_size : 4096\n",
            "2022-01-18 02:50:20,053 - INFO - joeynmt.helpers -            cfg.training.batch_type : token\n",
            "2022-01-18 02:50:20,054 - INFO - joeynmt.helpers -       cfg.training.eval_batch_size : 3600\n",
            "2022-01-18 02:50:20,054 - INFO - joeynmt.helpers -       cfg.training.eval_batch_type : token\n",
            "2022-01-18 02:50:20,054 - INFO - joeynmt.helpers -      cfg.training.batch_multiplier : 1\n",
            "2022-01-18 02:50:20,054 - INFO - joeynmt.helpers - cfg.training.early_stopping_metric : ppl\n",
            "2022-01-18 02:50:20,054 - INFO - joeynmt.helpers -                cfg.training.epochs : 30\n",
            "2022-01-18 02:50:20,054 - INFO - joeynmt.helpers -       cfg.training.validation_freq : 2000\n",
            "2022-01-18 02:50:20,054 - INFO - joeynmt.helpers -          cfg.training.logging_freq : 200\n",
            "2022-01-18 02:50:20,054 - INFO - joeynmt.helpers -           cfg.training.eval_metric : bleu\n",
            "2022-01-18 02:50:20,054 - INFO - joeynmt.helpers -             cfg.training.model_dir : models/epo_eng_bpe4000_transformer\n",
            "2022-01-18 02:50:20,054 - INFO - joeynmt.helpers -             cfg.training.overwrite : False\n",
            "2022-01-18 02:50:20,054 - INFO - joeynmt.helpers -               cfg.training.shuffle : True\n",
            "2022-01-18 02:50:20,054 - INFO - joeynmt.helpers -              cfg.training.use_cuda : True\n",
            "2022-01-18 02:50:20,054 - INFO - joeynmt.helpers -     cfg.training.max_output_length : 100\n",
            "2022-01-18 02:50:20,054 - INFO - joeynmt.helpers -     cfg.training.print_valid_sents : [0, 1, 2, 3]\n",
            "2022-01-18 02:50:20,054 - INFO - joeynmt.helpers -       cfg.training.keep_best_ckpts : 3\n",
            "2022-01-18 02:50:20,054 - INFO - joeynmt.helpers -              cfg.model.initializer : xavier\n",
            "2022-01-18 02:50:20,055 - INFO - joeynmt.helpers -         cfg.model.bias_initializer : zeros\n",
            "2022-01-18 02:50:20,055 - INFO - joeynmt.helpers -                cfg.model.init_gain : 1.0\n",
            "2022-01-18 02:50:20,055 - INFO - joeynmt.helpers -        cfg.model.embed_initializer : xavier\n",
            "2022-01-18 02:50:20,055 - INFO - joeynmt.helpers -          cfg.model.embed_init_gain : 1.0\n",
            "2022-01-18 02:50:20,055 - INFO - joeynmt.helpers -          cfg.model.tied_embeddings : True\n",
            "2022-01-18 02:50:20,055 - INFO - joeynmt.helpers -             cfg.model.tied_softmax : True\n",
            "2022-01-18 02:50:20,055 - INFO - joeynmt.helpers -             cfg.model.encoder.type : transformer\n",
            "2022-01-18 02:50:20,055 - INFO - joeynmt.helpers -       cfg.model.encoder.num_layers : 6\n",
            "2022-01-18 02:50:20,055 - INFO - joeynmt.helpers -        cfg.model.encoder.num_heads : 4\n",
            "2022-01-18 02:50:20,055 - INFO - joeynmt.helpers - cfg.model.encoder.embeddings.embedding_dim : 256\n",
            "2022-01-18 02:50:20,055 - INFO - joeynmt.helpers - cfg.model.encoder.embeddings.scale : True\n",
            "2022-01-18 02:50:20,055 - INFO - joeynmt.helpers - cfg.model.encoder.embeddings.dropout : 0.2\n",
            "2022-01-18 02:50:20,055 - INFO - joeynmt.helpers -      cfg.model.encoder.hidden_size : 256\n",
            "2022-01-18 02:50:20,055 - INFO - joeynmt.helpers -          cfg.model.encoder.ff_size : 1024\n",
            "2022-01-18 02:50:20,055 - INFO - joeynmt.helpers -          cfg.model.encoder.dropout : 0.3\n",
            "2022-01-18 02:50:20,055 - INFO - joeynmt.helpers -             cfg.model.decoder.type : transformer\n",
            "2022-01-18 02:50:20,055 - INFO - joeynmt.helpers -       cfg.model.decoder.num_layers : 6\n",
            "2022-01-18 02:50:20,056 - INFO - joeynmt.helpers -        cfg.model.decoder.num_heads : 4\n",
            "2022-01-18 02:50:20,056 - INFO - joeynmt.helpers - cfg.model.decoder.embeddings.embedding_dim : 256\n",
            "2022-01-18 02:50:20,056 - INFO - joeynmt.helpers - cfg.model.decoder.embeddings.scale : True\n",
            "2022-01-18 02:50:20,056 - INFO - joeynmt.helpers - cfg.model.decoder.embeddings.dropout : 0.2\n",
            "2022-01-18 02:50:20,056 - INFO - joeynmt.helpers -      cfg.model.decoder.hidden_size : 256\n",
            "2022-01-18 02:50:20,056 - INFO - joeynmt.helpers -          cfg.model.decoder.ff_size : 1024\n",
            "2022-01-18 02:50:20,056 - INFO - joeynmt.helpers -          cfg.model.decoder.dropout : 0.3\n",
            "2022-01-18 02:50:20,056 - INFO - joeynmt.helpers - Data set sizes: \n",
            "\ttrain 347669,\n",
            "\tvalid 1000,\n",
            "\ttest 1000\n",
            "2022-01-18 02:50:20,056 - INFO - joeynmt.helpers - First training example:\n",
            "\t[SRC] At@@ ing@@ ebla tabela ap@@ ud@@ skribo\n",
            "\t[TRG] Acces@@ sible T@@ able C@@ ap@@ tion\n",
            "2022-01-18 02:50:20,056 - INFO - joeynmt.helpers - First 10 words (src): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) la (5) the (6) of (7) de (8) and (9) kaj\n",
            "2022-01-18 02:50:20,056 - INFO - joeynmt.helpers - First 10 words (trg): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) la (5) the (6) of (7) de (8) and (9) kaj\n",
            "2022-01-18 02:50:20,056 - INFO - joeynmt.helpers - Number of Src words (types): 4544\n",
            "2022-01-18 02:50:20,057 - INFO - joeynmt.helpers - Number of Trg words (types): 4544\n",
            "2022-01-18 02:50:20,057 - INFO - joeynmt.training - Model(\n",
            "\tencoder=TransformerEncoder(num_layers=6, num_heads=4),\n",
            "\tdecoder=TransformerDecoder(num_layers=6, num_heads=4),\n",
            "\tsrc_embed=Embeddings(embedding_dim=256, vocab_size=4544),\n",
            "\ttrg_embed=Embeddings(embedding_dim=256, vocab_size=4544))\n",
            "2022-01-18 02:50:20,060 - INFO - joeynmt.training - Train stats:\n",
            "\tdevice: cuda\n",
            "\tn_gpu: 1\n",
            "\t16-bits training: False\n",
            "\tgradient accumulation: 1\n",
            "\tbatch size per device: 4096\n",
            "\ttotal batch size (w. parallel & accumulation): 4096\n",
            "2022-01-18 02:50:20,060 - INFO - joeynmt.training - EPOCH 1\n",
            "2022-01-18 02:50:49,930 - INFO - joeynmt.training - Epoch   1, Step:      200, Batch Loss:     5.399559, Tokens per Sec:     8181, Lr: 0.000300\n",
            "2022-01-18 02:51:19,828 - INFO - joeynmt.training - Epoch   1, Step:      400, Batch Loss:     5.058511, Tokens per Sec:     8216, Lr: 0.000300\n",
            "2022-01-18 02:51:49,915 - INFO - joeynmt.training - Epoch   1, Step:      600, Batch Loss:     4.696565, Tokens per Sec:     8109, Lr: 0.000300\n",
            "2022-01-18 02:52:20,396 - INFO - joeynmt.training - Epoch   1, Step:      800, Batch Loss:     4.576544, Tokens per Sec:     8094, Lr: 0.000300\n",
            "2022-01-18 02:52:51,153 - INFO - joeynmt.training - Epoch   1, Step:     1000, Batch Loss:     4.254906, Tokens per Sec:     7999, Lr: 0.000300\n",
            "2022-01-18 02:53:22,189 - INFO - joeynmt.training - Epoch   1, Step:     1200, Batch Loss:     4.274159, Tokens per Sec:     7950, Lr: 0.000300\n",
            "2022-01-18 02:53:53,291 - INFO - joeynmt.training - Epoch   1, Step:     1400, Batch Loss:     4.017766, Tokens per Sec:     7897, Lr: 0.000300\n",
            "2022-01-18 02:54:24,431 - INFO - joeynmt.training - Epoch   1, Step:     1600, Batch Loss:     4.085143, Tokens per Sec:     7868, Lr: 0.000300\n",
            "2022-01-18 02:54:55,761 - INFO - joeynmt.training - Epoch   1, Step:     1800, Batch Loss:     3.965685, Tokens per Sec:     7866, Lr: 0.000300\n",
            "2022-01-18 02:55:27,186 - INFO - joeynmt.training - Epoch   1, Step:     2000, Batch Loss:     3.912527, Tokens per Sec:     7767, Lr: 0.000300\n",
            "2022-01-18 02:55:39,884 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
            "2022-01-18 02:55:40,295 - INFO - joeynmt.training - Example #0\n",
            "2022-01-18 02:55:40,295 - INFO - joeynmt.training - \tSource:     Mi faris ion por vi.\n",
            "2022-01-18 02:55:40,295 - INFO - joeynmt.training - \tReference:  I made you something.\n",
            "2022-01-18 02:55:40,296 - INFO - joeynmt.training - \tHypothesis: I have to see you.\n",
            "2022-01-18 02:55:40,296 - INFO - joeynmt.training - Example #1\n",
            "2022-01-18 02:55:40,296 - INFO - joeynmt.training - \tSource:     Du oble du faras kvar.\n",
            "2022-01-18 02:55:40,296 - INFO - joeynmt.training - \tReference:  Twice two is four.\n",
            "2022-01-18 02:55:40,296 - INFO - joeynmt.training - \tHypothesis: - What is a man of the way.\n",
            "2022-01-18 02:55:40,296 - INFO - joeynmt.training - Example #2\n",
            "2022-01-18 02:55:40,296 - INFO - joeynmt.training - \tSource:     Tio suferigas min! Ĉesu!\n",
            "2022-01-18 02:55:40,296 - INFO - joeynmt.training - \tReference:  That hurts! Stop it!\n",
            "2022-01-18 02:55:40,296 - INFO - joeynmt.training - \tHypothesis: It's a cance.\n",
            "2022-01-18 02:55:40,296 - INFO - joeynmt.training - Example #3\n",
            "2022-01-18 02:55:40,296 - INFO - joeynmt.training - \tSource:     Estas tro multe por mi. Mi devas malrapidiĝi.\n",
            "2022-01-18 02:55:40,296 - INFO - joeynmt.training - \tReference:  It is too much for me. I need to slow down.\n",
            "2022-01-18 02:55:40,297 - INFO - joeynmt.training - \tHypothesis: It's not not a thing I have to be a bir.\n",
            "2022-01-18 02:55:40,297 - INFO - joeynmt.training - Validation result (greedy) at epoch   1, step     2000: bleu:   2.60, loss: 46577.3711, ppl:  48.5886, duration: 13.1100s\n",
            "2022-01-18 02:56:11,764 - INFO - joeynmt.training - Epoch   1, Step:     2200, Batch Loss:     3.577899, Tokens per Sec:     7807, Lr: 0.000300\n",
            "Traceback (most recent call last):\n",
            "  File \"/usr/lib/python3.7/runpy.py\", line 193, in _run_module_as_main\n",
            "    \"__main__\", mod_spec)\n",
            "  File \"/usr/lib/python3.7/runpy.py\", line 85, in _run_code\n",
            "    exec(code, run_globals)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/__main__.py\", line 48, in <module>\n",
            "    main()\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/__main__.py\", line 35, in main\n",
            "    train(cfg_file=args.config_path, skip_test=args.skip_test)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/training.py\", line 846, in train\n",
            "    trainer.train_and_validate(train_data=train_data, valid_data=dev_data)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/training.py\", line 447, in train_and_validate\n",
            "    batch_loss += self._train_step(batch)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/training.py\", line 569, in _train_step\n",
            "    norm_batch_loss.backward()\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/torch/_tensor.py\", line 307, in backward\n",
            "    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py\", line 156, in backward\n",
            "    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag\n",
            "KeyboardInterrupt\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pgP9fnZsjO7K"
      },
      "source": [
        "## Continue training after interruption"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kj92bqnf8aOe"
      },
      "source": [
        "To continue after an interruption, the configuration needs to be modified in 2 places: \n",
        "1. `load_model` to point to the checkpoint to load.\n",
        "2. `model_dir` to create a new directory.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qS7ixekMfG5F"
      },
      "source": [
        "ckpt_number = 2000\n",
        "reload_config = config.replace(\n",
        "    f'#load_model: \"models/{name}_transformer/1.ckpt\"', f'load_model: \"models/{name}_transformer/{ckpt_number}.ckpt\"').replace(\n",
        "        f'model_dir: \"models/{name}_transformer\"', f'model_dir: \"models/{name}_transformer_continued\"')\n",
        "with open(\"transformer_{name}_reload.yaml\".format(name=name),'w') as f:\n",
        "    f.write(reload_config)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "05nDex2g9URU"
      },
      "source": [
        "Joey NMT then picks up training from there."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iU63igoskRXJ",
        "outputId": "ab85539e-2047-484e-caee-2be4057f9931"
      },
      "source": [
        "!python -m joeynmt train transformer_epo_eng_bpe4000_reload.yaml"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2021-04-14 03:31:46,458 - INFO - root - Hello! This is Joey-NMT (version 1.3).\n",
            "2021-04-14 03:31:46,490 - INFO - joeynmt.data - Loading training data...\n",
            "2021-04-14 03:31:51,688 - INFO - joeynmt.data - Building vocabulary...\n",
            "2021-04-14 03:31:52,025 - INFO - joeynmt.data - Loading dev data...\n",
            "2021-04-14 03:31:52,068 - INFO - joeynmt.data - Loading test data...\n",
            "2021-04-14 03:31:52,074 - INFO - joeynmt.data - Data loaded.\n",
            "2021-04-14 03:31:52,074 - INFO - joeynmt.model - Building an encoder-decoder model...\n",
            "2021-04-14 03:31:52,298 - INFO - joeynmt.model - Enc-dec model built.\n",
            "2021-04-14 03:31:52.426688: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0\n",
            "2021-04-14 03:31:54,316 - INFO - joeynmt.training - Total params: 12223488\n",
            "2021-04-14 03:31:57,536 - INFO - joeynmt.training - Loading model from models/epo_eng_bpe4000_transformer/2000.ckpt\n",
            "2021-04-14 03:31:57,762 - INFO - joeynmt.helpers - cfg.name                           : epo_eng_bpe4000_transformer\n",
            "2021-04-14 03:31:57,762 - INFO - joeynmt.helpers - cfg.data.src                       : epo\n",
            "2021-04-14 03:31:57,762 - INFO - joeynmt.helpers - cfg.data.trg                       : eng\n",
            "2021-04-14 03:31:57,762 - INFO - joeynmt.helpers - cfg.data.train                     : /content/data/eng-epo//train.4000.bpe\n",
            "2021-04-14 03:31:57,762 - INFO - joeynmt.helpers - cfg.data.dev                       : /content/data/eng-epo//dev.4000.bpe\n",
            "2021-04-14 03:31:57,762 - INFO - joeynmt.helpers - cfg.data.test                      : /content/data/eng-epo//test.4000.bpe\n",
            "2021-04-14 03:31:57,762 - INFO - joeynmt.helpers - cfg.data.level                     : bpe\n",
            "2021-04-14 03:31:57,762 - INFO - joeynmt.helpers - cfg.data.lowercase                 : False\n",
            "2021-04-14 03:31:57,762 - INFO - joeynmt.helpers - cfg.data.max_sent_length           : 30\n",
            "2021-04-14 03:31:57,763 - INFO - joeynmt.helpers - cfg.data.src_vocab                 : /content/data/eng-epo/joint.4000bpe.vocab\n",
            "2021-04-14 03:31:57,763 - INFO - joeynmt.helpers - cfg.data.trg_vocab                 : /content/data/eng-epo/joint.4000bpe.vocab\n",
            "2021-04-14 03:31:57,763 - INFO - joeynmt.helpers - cfg.testing.beam_size              : 5\n",
            "2021-04-14 03:31:57,763 - INFO - joeynmt.helpers - cfg.testing.alpha                  : 1.0\n",
            "2021-04-14 03:31:57,763 - INFO - joeynmt.helpers - cfg.testing.sacrebleu.remove_whitespace : True\n",
            "2021-04-14 03:31:57,763 - INFO - joeynmt.helpers - cfg.testing.sacrebleu.tokenize     : intl\n",
            "2021-04-14 03:31:57,763 - INFO - joeynmt.helpers - cfg.training.load_model            : models/epo_eng_bpe4000_transformer/2000.ckpt\n",
            "2021-04-14 03:31:57,763 - INFO - joeynmt.helpers - cfg.training.random_seed           : 42\n",
            "2021-04-14 03:31:57,763 - INFO - joeynmt.helpers - cfg.training.optimizer             : adam\n",
            "2021-04-14 03:31:57,763 - INFO - joeynmt.helpers - cfg.training.normalization         : tokens\n",
            "2021-04-14 03:31:57,763 - INFO - joeynmt.helpers - cfg.training.adam_betas            : [0.9, 0.999]\n",
            "2021-04-14 03:31:57,763 - INFO - joeynmt.helpers - cfg.training.scheduling            : plateau\n",
            "2021-04-14 03:31:57,763 - INFO - joeynmt.helpers - cfg.training.patience              : 5\n",
            "2021-04-14 03:31:57,763 - INFO - joeynmt.helpers - cfg.training.learning_rate_factor  : 0.5\n",
            "2021-04-14 03:31:57,763 - INFO - joeynmt.helpers - cfg.training.learning_rate_warmup  : 1000\n",
            "2021-04-14 03:31:57,763 - INFO - joeynmt.helpers - cfg.training.decrease_factor       : 0.7\n",
            "2021-04-14 03:31:57,764 - INFO - joeynmt.helpers - cfg.training.loss                  : crossentropy\n",
            "2021-04-14 03:31:57,764 - INFO - joeynmt.helpers - cfg.training.learning_rate         : 0.0003\n",
            "2021-04-14 03:31:57,764 - INFO - joeynmt.helpers - cfg.training.learning_rate_min     : 1e-08\n",
            "2021-04-14 03:31:57,764 - INFO - joeynmt.helpers - cfg.training.weight_decay          : 0.0\n",
            "2021-04-14 03:31:57,764 - INFO - joeynmt.helpers - cfg.training.label_smoothing       : 0.1\n",
            "2021-04-14 03:31:57,764 - INFO - joeynmt.helpers - cfg.training.batch_size            : 4096\n",
            "2021-04-14 03:31:57,764 - INFO - joeynmt.helpers - cfg.training.batch_type            : token\n",
            "2021-04-14 03:31:57,764 - INFO - joeynmt.helpers - cfg.training.eval_batch_size       : 3600\n",
            "2021-04-14 03:31:57,764 - INFO - joeynmt.helpers - cfg.training.eval_batch_type       : token\n",
            "2021-04-14 03:31:57,764 - INFO - joeynmt.helpers - cfg.training.batch_multiplier      : 1\n",
            "2021-04-14 03:31:57,764 - INFO - joeynmt.helpers - cfg.training.early_stopping_metric : ppl\n",
            "2021-04-14 03:31:57,764 - INFO - joeynmt.helpers - cfg.training.epochs                : 30\n",
            "2021-04-14 03:31:57,764 - INFO - joeynmt.helpers - cfg.training.validation_freq       : 2000\n",
            "2021-04-14 03:31:57,764 - INFO - joeynmt.helpers - cfg.training.logging_freq          : 200\n",
            "2021-04-14 03:31:57,764 - INFO - joeynmt.helpers - cfg.training.eval_metric           : bleu\n",
            "2021-04-14 03:31:57,764 - INFO - joeynmt.helpers - cfg.training.model_dir             : models/epo_eng_bpe4000_transformer_continued\n",
            "2021-04-14 03:31:57,765 - INFO - joeynmt.helpers - cfg.training.overwrite             : False\n",
            "2021-04-14 03:31:57,765 - INFO - joeynmt.helpers - cfg.training.shuffle               : True\n",
            "2021-04-14 03:31:57,765 - INFO - joeynmt.helpers - cfg.training.use_cuda              : True\n",
            "2021-04-14 03:31:57,765 - INFO - joeynmt.helpers - cfg.training.max_output_length     : 100\n",
            "2021-04-14 03:31:57,765 - INFO - joeynmt.helpers - cfg.training.print_valid_sents     : [0, 1, 2, 3]\n",
            "2021-04-14 03:31:57,765 - INFO - joeynmt.helpers - cfg.training.keep_last_ckpts       : 3\n",
            "2021-04-14 03:31:57,765 - INFO - joeynmt.helpers - cfg.model.initializer              : xavier\n",
            "2021-04-14 03:31:57,765 - INFO - joeynmt.helpers - cfg.model.bias_initializer         : zeros\n",
            "2021-04-14 03:31:57,765 - INFO - joeynmt.helpers - cfg.model.init_gain                : 1.0\n",
            "2021-04-14 03:31:57,765 - INFO - joeynmt.helpers - cfg.model.embed_initializer        : xavier\n",
            "2021-04-14 03:31:57,765 - INFO - joeynmt.helpers - cfg.model.embed_init_gain          : 1.0\n",
            "2021-04-14 03:31:57,765 - INFO - joeynmt.helpers - cfg.model.tied_embeddings          : True\n",
            "2021-04-14 03:31:57,765 - INFO - joeynmt.helpers - cfg.model.tied_softmax             : True\n",
            "2021-04-14 03:31:57,765 - INFO - joeynmt.helpers - cfg.model.encoder.type             : transformer\n",
            "2021-04-14 03:31:57,765 - INFO - joeynmt.helpers - cfg.model.encoder.num_layers       : 6\n",
            "2021-04-14 03:31:57,765 - INFO - joeynmt.helpers - cfg.model.encoder.num_heads        : 4\n",
            "2021-04-14 03:31:57,766 - INFO - joeynmt.helpers - cfg.model.encoder.embeddings.embedding_dim : 256\n",
            "2021-04-14 03:31:57,766 - INFO - joeynmt.helpers - cfg.model.encoder.embeddings.scale : True\n",
            "2021-04-14 03:31:57,766 - INFO - joeynmt.helpers - cfg.model.encoder.embeddings.dropout : 0.2\n",
            "2021-04-14 03:31:57,766 - INFO - joeynmt.helpers - cfg.model.encoder.hidden_size      : 256\n",
            "2021-04-14 03:31:57,766 - INFO - joeynmt.helpers - cfg.model.encoder.ff_size          : 1024\n",
            "2021-04-14 03:31:57,766 - INFO - joeynmt.helpers - cfg.model.encoder.dropout          : 0.3\n",
            "2021-04-14 03:31:57,766 - INFO - joeynmt.helpers - cfg.model.decoder.type             : transformer\n",
            "2021-04-14 03:31:57,766 - INFO - joeynmt.helpers - cfg.model.decoder.num_layers       : 6\n",
            "2021-04-14 03:31:57,766 - INFO - joeynmt.helpers - cfg.model.decoder.num_heads        : 4\n",
            "2021-04-14 03:31:57,766 - INFO - joeynmt.helpers - cfg.model.decoder.embeddings.embedding_dim : 256\n",
            "2021-04-14 03:31:57,766 - INFO - joeynmt.helpers - cfg.model.decoder.embeddings.scale : True\n",
            "2021-04-14 03:31:57,766 - INFO - joeynmt.helpers - cfg.model.decoder.embeddings.dropout : 0.2\n",
            "2021-04-14 03:31:57,766 - INFO - joeynmt.helpers - cfg.model.decoder.hidden_size      : 256\n",
            "2021-04-14 03:31:57,766 - INFO - joeynmt.helpers - cfg.model.decoder.ff_size          : 1024\n",
            "2021-04-14 03:31:57,766 - INFO - joeynmt.helpers - cfg.model.decoder.dropout          : 0.3\n",
            "2021-04-14 03:31:57,767 - INFO - joeynmt.helpers - Data set sizes: \n",
            "\ttrain 347669,\n",
            "\tvalid 1000,\n",
            "\ttest 1000\n",
            "2021-04-14 03:31:57,767 - INFO - joeynmt.helpers - First training example:\n",
            "\t[SRC] At@@ ing@@ ebla tabela ap@@ ud@@ skribo\n",
            "\t[TRG] Acces@@ sible T@@ able C@@ ap@@ tion\n",
            "2021-04-14 03:31:57,767 - INFO - joeynmt.helpers - First 10 words (src): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) la (5) the (6) of (7) de (8) and (9) kaj\n",
            "2021-04-14 03:31:57,767 - INFO - joeynmt.helpers - First 10 words (trg): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) la (5) the (6) of (7) de (8) and (9) kaj\n",
            "2021-04-14 03:31:57,767 - INFO - joeynmt.helpers - Number of Src words (types): 4544\n",
            "2021-04-14 03:31:57,767 - INFO - joeynmt.helpers - Number of Trg words (types): 4544\n",
            "2021-04-14 03:31:57,767 - INFO - joeynmt.training - Model(\n",
            "\tencoder=TransformerEncoder(num_layers=6, num_heads=4),\n",
            "\tdecoder=TransformerDecoder(num_layers=6, num_heads=4),\n",
            "\tsrc_embed=Embeddings(embedding_dim=256, vocab_size=4544),\n",
            "\ttrg_embed=Embeddings(embedding_dim=256, vocab_size=4544))\n",
            "2021-04-14 03:31:57,771 - INFO - joeynmt.training - Train stats:\n",
            "\tdevice: cuda\n",
            "\tn_gpu: 1\n",
            "\t16-bits training: False\n",
            "\tgradient accumulation: 1\n",
            "\tbatch size per device: 4096\n",
            "\ttotal batch size (w. parallel & accumulation): 4096\n",
            "2021-04-14 03:31:57,771 - INFO - joeynmt.training - EPOCH 1\n",
            "2021-04-14 03:32:05,319 - INFO - joeynmt.training - Epoch   1: total training loss 165.52\n",
            "2021-04-14 03:32:05,319 - INFO - joeynmt.training - EPOCH 2\n",
            "2021-04-14 03:32:25,006 - INFO - joeynmt.training - Epoch   2, Step:     2200, Batch Loss:     4.343281, Tokens per Sec:    12693, Lr: 0.000300\n",
            "Traceback (most recent call last):\n",
            "  File \"/usr/lib/python3.7/runpy.py\", line 193, in _run_module_as_main\n",
            "    \"__main__\", mod_spec)\n",
            "  File \"/usr/lib/python3.7/runpy.py\", line 85, in _run_code\n",
            "    exec(code, run_globals)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/__main__.py\", line 48, in <module>\n",
            "    main()\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/__main__.py\", line 35, in main\n",
            "    train(cfg_file=args.config_path, skip_test=args.skip_test)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/training.py\", line 804, in train\n",
            "    trainer.train_and_validate(train_data=train_data, valid_data=dev_data)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/training.py\", line 427, in train_and_validate\n",
            "    batch_loss += self._train_step(batch)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/training.py\", line 536, in _train_step\n",
            "    norm_batch_loss.backward()\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/torch/tensor.py\", line 245, in backward\n",
            "    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py\", line 147, in backward\n",
            "    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag\n",
            "KeyboardInterrupt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "99Y-03KklJd3"
      },
      "source": [
        "## Let's Translate!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vhR4-RFF5_dJ"
      },
      "source": [
        "The `test` mode can be used to translate (and evaluate on) the test set specified in the configuration. We usually do this only once after we've tuned hyperparameters on the dev set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pu59HTo_lLJG",
        "outputId": "4150a0cd-d6d6-4629-933f-d2611dd39bcb"
      },
      "source": [
        "!python -m joeynmt test models/epo_eng_bpe4000_transformer/config.yaml"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2022-01-18 02:56:43,251 - INFO - root - Hello! This is Joey-NMT (version 1.4).\n",
            "2022-01-18 02:56:43,252 - INFO - joeynmt.data - Building vocabulary...\n",
            "2022-01-18 02:56:43,558 - INFO - joeynmt.data - Loading dev data...\n",
            "2022-01-18 02:56:43,564 - INFO - joeynmt.data - Loading test data...\n",
            "2022-01-18 02:56:43,571 - INFO - joeynmt.data - Data loaded.\n",
            "2022-01-18 02:56:43,605 - INFO - joeynmt.prediction - Process device: cuda, n_gpu: 1, batch_size per device: 3600\n",
            "2022-01-18 02:56:43,605 - INFO - joeynmt.prediction - Loading model from models/epo_eng_bpe4000_transformer/2000.ckpt\n",
            "2022-01-18 02:56:47,238 - INFO - joeynmt.model - Building an encoder-decoder model...\n",
            "2022-01-18 02:56:47,431 - INFO - joeynmt.model - Enc-dec model built.\n",
            "2022-01-18 02:56:47,500 - INFO - joeynmt.prediction - Decoding on dev set (/content/data/eng-epo//dev.4000.bpe.eng)...\n",
            "2022-01-18 02:57:23,457 - INFO - joeynmt.prediction -  dev bleu[intl]:   2.55 [Beam search decoding with beam size = 5 and alpha = {beam_alpha}]\n",
            "2022-01-18 02:57:23,458 - INFO - joeynmt.prediction - Decoding on test set (/content/data/eng-epo//test.4000.bpe.eng)...\n",
            "2022-01-18 02:58:00,137 - INFO - joeynmt.prediction - test bleu[intl]:   3.00 [Beam search decoding with beam size = 5 and alpha = {beam_alpha}]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8RgkT7mp6T2l"
      },
      "source": [
        "The `translate` mode is more interactive and takes prompts to translate interactively. Warning: it requires applying the same pre-processing steps to the new input as you've applied before model training (i.e. splitting into subwords)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YpBTLaoM9g58"
      },
      "source": [
        "from subword_nmt import apply_bpe\n",
        "\n",
        "with open(bpe_file, \"r\") as merge_file:\n",
        "  bpe = apply_bpe.BPE(codes=merge_file)\n",
        "\n",
        "preprocess = lambda x: bpe.process_line(x.strip())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4clB81W46oTB"
      },
      "source": [
        "my_sentence = 'Esperanto, origine la Lingvo Internacia, estas la plej disvastiĝinta internacia planlingvo.'   # From https://eo.wikipedia.org/wiki/Esperanto"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "OUkf54ip-hVw",
        "outputId": "c1b62536-0199-462f-aba5-19c845d5fe60"
      },
      "source": [
        "preprocess(my_sentence)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'E@@ sper@@ ant@@ o, or@@ ig@@ ine la L@@ ingv@@ o In@@ tern@@ aci@@ a, estas la plej dis@@ v@@ ast@@ iĝ@@ inta intern@@ ac@@ ia plan@@ lingv@@ o.'"
            ]
          },
          "metadata": {},
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-jHqeqiSctGU"
      },
      "source": [
        "Copy the above pre-processed sentence into the field when prompted below. Stop the cell to leave the interactive mode."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YtS7xyEmmD90",
        "outputId": "f8a09417-2773-4233-8c29-603535ff2cc6"
      },
      "source": [
        "!python -m joeynmt translate models/epo_eng_bpe4000_transformer/config.yaml"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2022-01-18 03:01:14,188 - INFO - root - Hello! This is Joey-NMT (version 1.4).\n",
            "2022-01-18 03:01:14,528 - INFO - joeynmt.prediction - Loading model from models/epo_eng_bpe4000_transformer/2000.ckpt\n",
            "2022-01-18 03:01:18,138 - INFO - joeynmt.model - Building an encoder-decoder model...\n",
            "2022-01-18 03:01:18,339 - INFO - joeynmt.model - Enc-dec model built.\n",
            "\n",
            "Please enter a source sentence (pre-processed): \n",
            "E@@ sper@@ ant@@ o, or@@ ig@@ ine la L@@ ingv@@ o In@@ tern@@ aci@@ a, estas la plej dis@@ v@@ ast@@ iĝ@@ inta intern@@ ac@@ ia plan@@ lingv@@ o.\n",
            "JoeyNMT: Hypotheses ranked by score\n",
            "JoeyNMT #1: Evern, the Axs of the Ax, that is the description of the actor\n",
            "\n",
            "Please enter a source sentence (pre-processed): \n",
            "E@@ sper@@ ant@@ o, or@@ ig@@ ine la L@@ ingv@@ o In@@ tern@@ aci@@ a, estas la plej dis@@ v@@ ast@@ iĝ@@ inta intern@@ ac@@ ia plan@@ lingv@@ o.\n",
            "JoeyNMT: Hypotheses ranked by score\n",
            "JoeyNMT #1: Evern, the Axs of the Ax, that is the description of the actor\n",
            "\n",
            "Please enter a source sentence (pre-processed): \n",
            "\n",
            "Bye.\n",
            "^C\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cX0Pk16Ac4fI"
      },
      "source": [
        "You can also get the n-best hypotheses (up to the size of the beam, in our example 5), not only the highest scoring one. The better your model gets, the more interesting should the alternatives be.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fT_SslRec__L",
        "outputId": "f7989659-95cc-4b7a-fc5d-f691928d24c0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "!python -m joeynmt translate models/epo_eng_bpe4000_transformer/config.yaml -n 3"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2022-01-18 03:02:55,786 - INFO - root - Hello! This is Joey-NMT (version 1.4).\n",
            "2022-01-18 03:02:56,132 - INFO - joeynmt.prediction - Loading model from models/epo_eng_bpe4000_transformer/2000.ckpt\n",
            "2022-01-18 03:02:59,781 - INFO - joeynmt.model - Building an encoder-decoder model...\n",
            "2022-01-18 03:02:59,980 - INFO - joeynmt.model - Enc-dec model built.\n",
            "\n",
            "Please enter a source sentence (pre-processed): \n",
            "E@@ sper@@ ant@@ o, or@@ ig@@ ine la L@@ ingv@@ o In@@ tern@@ aci@@ a, estas la plej dis@@ v@@ ast@@ iĝ@@ inta intern@@ ac@@ ia plan@@ lingv@@ o.\n",
            "JoeyNMT: Hypotheses ranked by score\n",
            "JoeyNMT #1: Evern, the Axs of the Ax, that is the description of the actor\n",
            "JoeyNMT #2: Evern, the Axs of the Ax, that is the description of the conver.\n",
            "JoeyNMT #3: Evern, the Axs of the Ax, that is the description of the description is\n",
            "\n",
            "Please enter a source sentence (pre-processed): \n",
            "\n",
            "Bye.Traceback (most recent call last):\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/prediction.py\", line 516, in translate\n",
            "    src_input = input(\"\\nPlease enter a source sentence \"\n",
            "KeyboardInterrupt\n",
            "\n",
            "During handling of the above exception, another exception occurred:\n",
            "\n",
            "Traceback (most recent call last):\n",
            "  File \"/usr/lib/python3.7/runpy.py\", line 193, in _run_module_as_main\n",
            "    \"__main__\", mod_spec)\n",
            "  File \"/usr/lib/python3.7/runpy.py\", line 85, in _run_code\n",
            "    exec(code, run_globals)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/__main__.py\", line 48, in <module>\n",
            "    main()\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/__main__.py\", line 42, in main\n",
            "    n_best=args.nbest)\n",
            "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/prediction.py\", line 530, in translate\n",
            "    print(\"\\nBye.\")\n",
            "KeyboardInterrupt\n"
          ]
        }
      ]
    }
  ]
}