using_kiwi.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Welcome to the OpenKiwi Tutorial\n",
    "\n",
    "In this tutorial we'll take you through a complete test drive of the OpenKiwi framework. OpenKiwi is an end-to-end framework to train, test and infer Machine Translation Quality Estimation (Kiwi) models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup \n",
    "\n",
    "Before being able to run OpenKiwi, there is a small setup required. If you have completed this step, keep reading onwards. Otherwise please refer to the [setup instructions](https://github.com/Unbabel/KiwiCutter/blob/master/setup.md)\n",
    "\n",
    "First, we will begin by loading all necessary libraries to run this notebook. Note that most of these are for demonstration purposes and to facilitate working in a notebook. They will not be necessary for using openkiwi in a normal setting (kiwi itselft should be enough)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import utils\n",
    "import yaml\n",
    "from ipywidgets import interact, fixed, Textarea\n",
    "from functools import partial\n",
    "%load_ext yamlmagic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install Kiwi\n",
    "\n",
    "Installing Kiwi to use it as a package is a fairly simple procedure. The only thing you need to do is `pip install openkiwi`! In this case it should already be installed in your machine, so all that's left is to import it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#!pip install openkiwi\n",
    "import kiwi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Downloading pre-trained models\n",
    "First, we will begin by using a pre-trained OpenKiwi model to evaluate the quality of an existing translation. The pre-trained models made available with OpenKiwi focus mainly on En-De which is has been the primary language pair for the WMT19 shared task on quality estimation.\n",
    "\n",
    "The following cell runs a method which will (conditionally, if you haven't done it yet) download and extract the zip which contains OpenKiwi's pre-trained models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download and extract pre-trained kiwi models\n",
    "\n",
    "OK_url = 'https://github.com/unbabel/KiwiCutter/releases/download/v1.0/estimator_en_de.torch.zip'\n",
    "\n",
    "utils.download_kiwi(OK_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading and Predicting\n",
    "\n",
    "The model we are going to use is a Predictor-Estimator with an RNN-based architecture. You can find more details about it [here](https://www.aclweb.org/anthology/W17-4763). This model was trained on the [WMT Quality Estimation data](http://www.statmt.org/wmt19/qe-task.html). \n",
    "\n",
    "\n",
    "Using OpenKiwi's API is fairly straightforward. We start by loading the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = kiwi.load_model('trained_models/estimator_en_de.torch/estimator_en_de.torch')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then create the sample that we would like to test and make it into a dictionary of lists. In other words, we are creating a batch of examples that Kiwi should use for inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "source = ['the part of the regular expression within the forward slashes defines the pattern .']\n",
    "target = ['der Teil des regulären Ausdrucks innerhalb der umgekehrten Schrägstrich definiert das Muster .']\n",
    "examples = {'source': source,'target': target}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then you can simply call `model.predict`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = model.predict(examples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or you could just as easily, use our cli by passing a config with the location of the models and data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!kiwi predict --config {path_to_config}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The CLI approach is normally used when you want to produce a file of predictions. On the other hand, the Kiwi as a library approach is used when using Kiwi in the context of another application.\n",
    "\n",
    "In this case, Kiwi will return the scores attributed to each token in the default output format which is a python dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll notice that this Kiwi model returns three different types of predictions, tags, gap tags and sentence scores:\n",
    "\n",
    "    - Tags: Tags are the scores attributed to each word token. This means that if you have a sentence of length `x` Kiwi will return a list with `x` scores.\n",
    "    \n",
    "    - Gap Tags: These represent the scores of the gaps between words. A gap tag should be marked as bad if there is a word missing in between two other words. This also includes the beggining and end of sentence. As such, on our sequence of length `x`, there will be `x + 1` gap tags. \n",
    "    \n",
    "    - Sentence Score: Finally, the sentence score is a prediction of the sentence's HTER (Human-targeted Translation Error Rate). Or in other words, what is the percentage of the sentence that you would need to change to create a correct translation.\n",
    "    \n",
    "    \n",
    "On the other hand, looking at a bunch of scores with no context is not terribly informative. So, below you can find a small utility to visualize the Kiwi scores in the context of a translation. \n",
    "\n",
    "You'll notice that you can move the threshold for marking a word as bad. This is useful in real-world scenarios as you can calibrate the conservativeness of the models and the severity of the errors you want to highlight."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SOURCE = Textarea(value=source[0])\n",
    "MT = Textarea(value=target[0])\n",
    "_interact = interact(utils.KiwiViz, model=fixed(model), source=SOURCE, mt=MT, threshold=(0.0, 1.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**EXERCISE**: Mess with this translation and see how changing the translations affects the Kiwi scores. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training a model from scratch\n",
    "\n",
    "OpenKiwi supports training a set of 4 different architectures:\n",
    "    - Linear Model\n",
    "    - Quetch\n",
    "    - NuQE\n",
    "    - Predictor-Estimator\n",
    "    \n",
    "This can be easily achieved either through or API or the command line. But, contrary to inference, as training posesses a host of different options, we rely on yaml config files to pass these parameters into the framework.\n",
    "\n",
    "Below, you'll find an example config file for training a NuQE model. NuQE is a simple, end-to-end neural model often used as the baseline for WMT's quality estimation shared task. You can see more details about it [here](https://www.aclweb.org/anthology/W16-2387)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%yaml yaml_config\n",
    "#### MODEL SPECIFIC OPTIONS ####\n",
    "#\n",
    "model: nuqe\n",
    "\n",
    "seed: 42\n",
    "\n",
    "output-dir: runs/nuqe\n",
    "\n",
    "window-size: 3\n",
    "max-aligned: 5\n",
    "\n",
    "# embeddings\n",
    "source-embeddings-size: 50\n",
    "source-pos-embeddings-size: 20\n",
    "target-embeddings-size: 50\n",
    "target-pos-embeddings-size: 20\n",
    "\n",
    "# network\n",
    "hidden-sizes: [400, 200, 100, 50]\n",
    "dropout: 0.0\n",
    "embeddings-dropout: 0.5\n",
    "freeze-embeddings: false\n",
    "bad-weight: 3.0\n",
    "\n",
    "# initialization\n",
    "init-support: 0.1\n",
    "init-type: uniform\n",
    "\n",
    "### Pretrained Embedding Options ###\n",
    "# pip-install the polyglot package to use these\n",
    "#embeddings-format: polyglot\n",
    "#    source: path/to/source/embeddings_pkl.tar.bz2\n",
    "#    target: path/to/target/embeddings_pkl.tar.bz2\n",
    "\n",
    "#\n",
    "# TRAINING OPTIONS\n",
    "#\n",
    "epochs: 3\n",
    "train-batch-size: 64\n",
    "valid-batch-size: 64\n",
    "\n",
    "log-interval: 100\n",
    "checkpoint-save: true\n",
    "checkpoint-keep-only-best: 1\n",
    "checkpoint-early-stop-patience: 10\n",
    "\n",
    "optimizer: adam\n",
    "learning-rate: 0.001\n",
    "\n",
    "gpu-id: -1\n",
    "\n",
    "predict-target: true\n",
    "\n",
    "#\n",
    "# DATA OPTIONS\n",
    "#\n",
    "wmt18-format: true\n",
    "train-source: WMT19/train.src\n",
    "train-target: WMT19/train.mt\n",
    "train-target-tags: WMT19/train.tags\n",
    "train-alignments: WMT19/train.src-mt.alignments\n",
    "\n",
    "valid-source: WMT19/dev.src\n",
    "valid-target: WMT19/dev.mt\n",
    "valid-target-tags: WMT19/dev.tags\n",
    "valid-alignments: WMT19/dev.src-mt.alignments\n",
    "\n",
    "# vocabulary\n",
    "source-vocab-min-frequency: 2\n",
    "target-vocab-min-frequency: 2\n",
    "keep-rare-words-with-embeddings: true\n",
    "add-embeddings-vocab: false\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We save this config to a file so it can be loaded later into kiwi."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "utils.save_config(yaml_config, 'nuqe_config.yml')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, you can use either the API or the command line to call kiwi and begin training with this configuration!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "config = 'nuqe_config.yml'\n",
    "\n",
    "#Uncomment one of the following lines, they are virtually identical\n",
    "\n",
    "run_info = kiwi.train(config)\n",
    "#!kiwi train --config nuqe_config.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating and Finetuning an existing model\n",
    "\n",
    "Finally, OpenKiwi also provides an easy way to evaluate existing models against a QE dataset. Here, we will evaluate one of our pre-trained models against the WMT19 dev set (as the test sets are unfortunately not available). \n",
    "\n",
    "Then, we will try to continue fine-tuning this model in an attempt to increase it's performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation\n",
    "\n",
    "As with training, we defined the evaluation options through a yaml config file. Here, we will use our pre-trained model to predict the tags for the WMMT19 dev set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%yaml predest_predict\n",
    "output-dir: predictions/predest\n",
    "\n",
    "#\n",
    "# GENERAL OPTIONS\n",
    "#\n",
    "# random\n",
    "seed: 42\n",
    "\n",
    "# gpu\n",
    "gpu-id: -1\n",
    "\n",
    "model: estimator\n",
    "\n",
    "# save and load\n",
    "load-model: trained_models/estimator_en_de.torch/estimator_en_de.torch\n",
    "\n",
    "#\n",
    "# DATA OPTIONS\n",
    "#\n",
    "wmt18-format: False\n",
    "test-source: WMT19/dev.src\n",
    "test-target: WMT19/dev.mt\n",
    "valid-batch-size: 64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we save the config and call the CLI. This will create predictions in the `output-dir`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "utils.save_config(predest_predict, 'predest_predict.yml')\n",
    "!kiwi predict --config predest_predict.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we define a yaml for the evaluation pipeline. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%yaml predest_evaluate\n",
    "# Example file for configuring the evaluation pipeline\n",
    "#\n",
    "# The input type for prediction files (Probabilities[probs] or tags)\n",
    "type: probs\n",
    " \n",
    "# The format of gold files (wmt17/wmt18)\n",
    "format: wmt18\n",
    "\n",
    "# Format of predictions (wmt17/wmt18). Either they predict gaps or not.\n",
    "pred-format: wmt17\n",
    "\n",
    "# File path for the reference files\n",
    "gold-target: WMT19/dev.tags\n",
    "\n",
    "# File path for the prediction files\n",
    "pred-target: predictions/predest/tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "utils.save_config(predest_evaluate, 'predest_evaluate.yml')\n",
    "!kiwi evaluate --config predest_evaluate.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These results, are on par with what we expect for a single model in the wmt19 dev set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finetuning\n",
    "\n",
    "Finally, we can also load the pre-trained model and continue finetuning it. \n",
    "Can we further increase it's performance?\n",
    "\n",
    "Here, we will use the predictor_estimator used on the previous example and continue training it on a randomly selected sub-set of the WMT19 data. This subset is located in `WMT19/small`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%yaml predest_finetune\n",
    "### Train Predictor Estimator ###\n",
    "\n",
    "model: estimator\n",
    "\n",
    "#### MODEL SPECIFIC OPTS ####\n",
    "\n",
    "## ESTIMATOR ##\n",
    "\n",
    "# If load-model points to a pretrained Estimator,\n",
    "# These settings are ignored.\n",
    "\n",
    "# LSTM Settings\n",
    "hidden-est: 125\n",
    "rnn-layers-est: 1\n",
    "dropout-est: 0.0\n",
    "# Use linear layer to reduce dimension prior to LSTM\n",
    "mlp-est: True\n",
    "\n",
    "# Multitask Learning Settings #\n",
    "\n",
    "# Continue training the predictor on the postedited text.\n",
    "# If set, will do an additional forward pass through the predictor\n",
    "# Using the SRC, PE pair and add the `Predictor` loss for the tokens in the\n",
    "# postedited text PE. Recommended if you have access to PE\n",
    "# Requires setting train-pe, valid-pe\n",
    "token-level: True\n",
    "# Predict Sentence Level Scores\n",
    "# Requires setting train-sentence-scores, valid-sentence-scores\n",
    "sentence-level: True\n",
    "# Use probabilistic Loss for sentence scores instead of squared error.\n",
    "# If set, the model will output mean and variance of a truncated Gaussian\n",
    "# distribution over the interval [0, 1], and use log-likelihood loss instead\n",
    "# of mean squared error.\n",
    "# Seems to improve performance\n",
    "sentence-ll: False\n",
    "# Predict Binary Label for each sentence, indicating hter == 0.0\n",
    "# Requires setting train-sentence-scores, valid-sentence-scores\n",
    "binary-level: False\n",
    "\n",
    "# WMT 18 Format Settings #\n",
    "\n",
    "# Predict target tags. Requires train-target-tags, valid-target-tags to be set.\n",
    "#predict-target: true\n",
    "target-bad-weight: 2.5\n",
    "# Predict source tags. Requires train-source-tags, valid-source-tags to be set.\n",
    "#predict-source: false\n",
    "source-bad-weight: 2.5\n",
    "# Predict gap tags. Requires train-target-tags, valid-target-tags to be set.\n",
    "# and wmt18-format set to true\n",
    "#predict-gaps: true\n",
    "target-bad-weight: 2.5\n",
    "\n",
    "### GENERAL OPTS ###\n",
    "\n",
    "# Do not set or set to negative number for CPU\n",
    "gpu-id: -1\n",
    "\n",
    "### TRAIN OPTS ###\n",
    "epochs: 1\n",
    "# Additionally Eval and checkpoint every n training steps\n",
    "# Explicitly disable by setting to zero (default)\n",
    "checkpoint-validation-steps: 300000\n",
    "# If False, never save the Models\n",
    "checkpoint-save: true\n",
    "# Keep Only the n best models according to the main metric (F1Mult by default)\n",
    "# USeful to avoid filling the harddrive during a long run\n",
    "checkpoint-keep-only-best: 3\n",
    "# If greater than zero, Early Stop after n evaluation cycles without improvement\n",
    "checkpoint-early-stop-patience: 0\n",
    "\n",
    "\n",
    "# Print Train Stats Every n batches\n",
    "log-interval: 100\n",
    "# LR. Currently ADAM is only optimizer supported.\n",
    "# 1e-3 * (batch_size / 32) seems to work well\n",
    "learning-rate: 2e-3\n",
    "\n",
    "train-batch-size: 64\n",
    "valid-batch-size: 64\n",
    "\n",
    "\n",
    "\n",
    "### LOADING ###\n",
    "\n",
    "# Load pretrained (sub-)model.\n",
    "# If set, the model architecture params are ignored.\n",
    "# As the vocabulary of the pretrained model will be used,\n",
    "# all vocab-params will also be ignored.\n",
    "\n",
    "# (i) load-pred-source or load-pred-target: Predictor instance\n",
    "#     -> a new Estimator is initialized with the given predictor(s).\n",
    "# (ii) load-model: Estimator instance.\n",
    "#                  As the Predictor is a submodule of the Estimator,\n",
    "#                  load-pred-{source,target} will be ignored if this is set.\n",
    "\n",
    "load-model: trained_models/estimator_en_de.torch/estimator_en_de.torch\n",
    "# load-pred-source: path_to_predictor_source_target\n",
    "# load-pred-target: runs/model.torch\n",
    "\n",
    "\n",
    "###  DATA ###\n",
    "\n",
    "# Set to True to use target_tags in WMT18 format\n",
    "wmt18-format: true\n",
    "\n",
    "train-source: WMT19/small/train.src\n",
    "train-target: WMT19/small/train.mt\n",
    "train-pe: WMT19/small/train.pe\n",
    "train-target-tags: WMT19/small/train.tags\n",
    "train-sentence-scores: WMT19/small/train.hter\n",
    "\n",
    "\n",
    "valid-source: WMT19/dev.src\n",
    "valid-target: WMT19/dev.mt\n",
    "valid-pe: WMT19/dev.pe\n",
    "valid-target-tags: WMT19/dev.tags\n",
    "valid-sentence-scores: WMT19/dev.hter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "utils.save_config(predest_finetune, 'predest_config.yml')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "kiwi.train('predest_config.yml')\n",
    "#!kiwi train --config predest_config.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How do these scores compare to your previous evaluation? Can you improve it? :)\n",
    "\n",
    "Note: Increasing the performance of these is actually a very difficult task as these models had already been trained on this dataset. The goal is simply to learn how to continue fine-tuning a model on a different dataset.\n",
    "\n",
    "\n",
    "If you're done, go back to the repo and check the `exercises` folder!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}