errors.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Error analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook contains the source code used for the error analysis described in the project's report, where wrong answers are taken from the SQuAD v1.1 dev set for each and every model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to import source files, we have to add the `src` folder to the Python `PATH`$\\dots$ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-04T14:02:17.531865Z",
     "iopub.status.busy": "2021-02-04T14:02:17.531442Z",
     "iopub.status.idle": "2021-02-04T14:02:17.537952Z",
     "shell.execute_reply": "2021-02-04T14:02:17.536695Z",
     "shell.execute_reply.started": "2021-02-04T14:02:17.531820Z"
    }
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "sys.path.insert(0, \"src\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we can import packages as usual$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-04T14:02:21.264373Z",
     "iopub.status.busy": "2021-02-04T14:02:21.263927Z",
     "iopub.status.idle": "2021-02-04T14:02:21.496072Z",
     "shell.execute_reply": "2021-02-04T14:02:21.494558Z",
     "shell.execute_reply.started": "2021-02-04T14:02:21.264327Z"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "\n",
    "import numpy as np\n",
    "from transformers.trainer_utils import set_seed\n",
    "\n",
    "import config\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PyTorch and numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the random seed to a fixed number for reproducible results$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-04T14:04:56.187818Z",
     "iopub.status.busy": "2021-02-04T14:04:56.187378Z",
     "iopub.status.idle": "2021-02-04T14:04:57.183457Z",
     "shell.execute_reply": "2021-02-04T14:04:57.182043Z",
     "shell.execute_reply.started": "2021-02-04T14:04:56.187772Z"
    }
   },
   "outputs": [],
   "source": [
    "set_seed(config.RANDOM_SEED)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrong answers loading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the errors made on the SQuAD v1.1 dev set for each and every model$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-04T14:04:59.448122Z",
     "iopub.status.busy": "2021-02-04T14:04:59.447790Z",
     "iopub.status.idle": "2021-02-04T14:04:59.701643Z",
     "shell.execute_reply": "2021-02-04T14:04:59.700414Z",
     "shell.execute_reply.started": "2021-02-04T14:04:59.448081Z"
    }
   },
   "outputs": [],
   "source": [
    "with open('results/wrong/baseline.json') as f:\n",
    "    baseline_errors = json.load(f)\n",
    "with open('results/wrong/bidaf.json') as f:\n",
    "    bidaf_errors = json.load(f)\n",
    "with open('results/wrong/bert.json') as f:\n",
    "    bert_errors = json.load(f)\n",
    "with open('results/wrong/distilbert.json') as f:\n",
    "    distilbert_errors = json.load(f)\n",
    "with open('results/wrong/electra.json') as f:\n",
    "    electra_errors = json.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observe how much errors are made with each model$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-04T14:05:02.514592Z",
     "iopub.status.busy": "2021-02-04T14:05:02.514043Z",
     "iopub.status.idle": "2021-02-04T14:05:02.564255Z",
     "shell.execute_reply": "2021-02-04T14:05:02.563180Z",
     "shell.execute_reply.started": "2021-02-04T14:05:02.514545Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Baseline model makes 7700 errors\n",
      "The BiDAF model makes 4222 errors\n",
      "The BERT model makes 2731 errors\n",
      "The DistilBERT model makes 2789 errors\n",
      "The ELECTRA model makes 2062 errors\n"
     ]
    }
   ],
   "source": [
    "print(f\"The Baseline model makes {len(baseline_errors)} errors\")\n",
    "print(f\"The BiDAF model makes {len(bidaf_errors)} errors\")\n",
    "print(f\"The BERT model makes {len(bert_errors)} errors\")\n",
    "print(f\"The DistilBERT model makes {len(distilbert_errors)} errors\")\n",
    "print(f\"The ELECTRA model makes {len(electra_errors)} errors\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrong answers analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the common errors among all the models and the common errors among the best models, i.e. BiDAF and ELECTRA$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-04T14:05:06.785505Z",
     "iopub.status.busy": "2021-02-04T14:05:06.785050Z",
     "iopub.status.idle": "2021-02-04T14:05:06.833234Z",
     "shell.execute_reply": "2021-02-04T14:05:06.832109Z",
     "shell.execute_reply.started": "2021-02-04T14:05:06.785458Z"
    }
   },
   "outputs": [],
   "source": [
    "all_common_errors = list(\n",
    "    set(electra_errors.keys())\n",
    "    & set(bidaf_errors.keys())\n",
    "    & set(baseline_errors.keys())\n",
    "    & set(bert_errors.keys())\n",
    "    & set(distilbert_errors.keys())\n",
    ")\n",
    "best_common_errors = list(set(electra_errors.keys()) & set(bidaf_errors.keys()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-04T14:05:07.581691Z",
     "iopub.status.busy": "2021-02-04T14:05:07.581370Z",
     "iopub.status.idle": "2021-02-04T14:05:07.619387Z",
     "shell.execute_reply": "2021-02-04T14:05:07.618142Z",
     "shell.execute_reply.started": "2021-02-04T14:05:07.581649Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of common errors between all the models is 940\n",
      "The number of common errors between BiDAF and ELECTRA is 1535\n"
     ]
    }
   ],
   "source": [
    "print(f\"The number of common errors between all the models is {len(all_common_errors)}\")\n",
    "print(f\"The number of common errors between BiDAF and ELECTRA is {len(best_common_errors)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a random subset of errors (among the common ones between BiDAF and ELECTRA)$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-04T14:05:11.796007Z",
     "iopub.status.busy": "2021-02-04T14:05:11.795546Z",
     "iopub.status.idle": "2021-02-04T14:05:11.845352Z",
     "shell.execute_reply": "2021-02-04T14:05:11.843934Z",
     "shell.execute_reply.started": "2021-02-04T14:05:11.795960Z"
    }
   },
   "outputs": [],
   "source": [
    "random_best_common_errors = np.random.choice(best_common_errors, 50, replace=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show the selected errors$\\dots$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-02-04T14:05:13.190808Z",
     "iopub.status.busy": "2021-02-04T14:05:13.190439Z",
     "iopub.status.idle": "2021-02-04T14:05:13.291708Z",
     "shell.execute_reply": "2021-02-04T14:05:13.290300Z",
     "shell.execute_reply.started": "2021-02-04T14:05:13.190765Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context: European Union law is applied by the courts of member states and the Court of Justice of the European Union. Where the laws of member states provide for lesser rights European Union law can be enforced by the courts of member states. In case of European Union law which should have been transposed into the laws of member states, such as Directives, the European Commission can take proceedings against the member state under the Treaty on the Functioning of the European Union. The European Court of Justice is the highest court able to interpret European Union law. Supplementary sources of European Union law include case law by the Court of Justice, international law and general principles of European Union law.\n",
      "Question: What is one supplementary source of European Union law?\n",
      "Answers: ['international law', 'international law', 'international law', 'international law']\n",
      "Predictions: [ELECTRA] general principles of european union law [BiDAF] case law by court of justice international law and general principles\n",
      "\n",
      "Context: The plain moraine plateau has only a few natural and artificial ponds and also groups of clay pits. The pattern of the Vistula terraces is asymmetrical. The left side consist mainly of two levels: the highest one contains former flooded terraces and the lowest one the flood plain terrace. The contemporary flooded terrace still has visible valleys and ground depressions with water systems coming from the Vistula old – riverbed. They consist of still quite natural streams and lakes as well as the pattern of drainage ditches. The right side of Warsaw has a different pattern of geomorphological forms. There are several levels of the plain Vistula terraces (flooded as well as former flooded once) and only small part and not so visible moraine escarpment. Aeolian sand with a number of dunes parted by peat swamps or small ponds cover the highest terrace. These are mainly forested areas (pine forest).\n",
      "Question: Where can Aeolian sand with a number of dunes be found?\n",
      "Answers: ['plain vistula terraces', 'highest terrace', 'highest terrace']\n",
      "Predictions: [ELECTRA] peat swamps [BiDAF] peat swamps or small ponds\n",
      "\n",
      "Context: Also part of the library is the aforementioned Selznick library, the Cinerama Productions/Palomar theatrical library and the Selmur Productions catalog that the network acquired some years back, and the in-house productions it continues to produce (such as America's Funniest Home Videos, General Hospital, and ABC News productions), although Disney–ABC Domestic Television (formerly known as Buena Vista Television) handles domestic television distribution, while Disney–ABC International Television (formerly known as Buena Vista International Television) handles international television distribution.\n",
      "Question: What library contains the Selmur Productions catalogue?\n",
      "Answers: ['selznick library', 'selznick', 'selznick library']\n",
      "Predictions: [ELECTRA] selznick library cinerama productionspalomar theatrical library [BiDAF] cinerama productionspalomar theatrical library\n",
      "\n",
      "Context: In late 1886 Tesla met Alfred S. Brown, a Western Union superintendent, and New York attorney Charles F. Peck. The two men were experienced in setting up companies and promoting inventions and patents for financial gain. Based on Tesla's patents and other ideas they agreed to back him financially and handle his patents. Together in April 1887 they formed the Tesla Electric Company with an agreement that profits from generated patents would go ⅓ to Tesla, ⅓ to Peck and Brown, and ⅓ to fund development. They set up a laboratory for Tesla at 89 Liberty Street in Manhattan where he worked on improving and developing new types of electric motors, generators and other devices.\n",
      "Question: Where would profits go in this new company?\n",
      "Answers: ['⅓ to tesla ⅓ to peck and brown and ⅓ to fund development', '⅓ to tesla ⅓ to peck and brown and ⅓ to fund development', '⅓ to tesla ⅓ to peck and brown and ⅓ to fund development']\n",
      "Predictions: [ELECTRA] tesla [BiDAF] ⅓ to tesla\n",
      "\n",
      "Context: Some civil disobedients feel it is incumbent upon them to accept punishment because of their belief in the validity of the social contract, which is held to bind all to obey the laws that a government meeting certain standards of legitimacy has established, or else suffer the penalties set out in the law. Other civil disobedients who favor the existence of government still don't believe in the legitimacy of their particular government, or don't believe in the legitimacy of a particular law it has enacted. And still other civil disobedients, being anarchists, don't believe in the legitimacy of any government, and therefore see no need to accept punishment for a violation of criminal law that does not infringe the rights of others.\n",
      "Question: What do some people protest against?\n",
      "Answers: ['legitimacy of particular law', 'law', 'laws', 'legitimacy of any government']\n",
      "Predictions: [ELECTRA] punishment [BiDAF] them to accept punishment because of their belief in validity of social contract\n",
      "\n",
      "Context: He came to power by uniting many of the nomadic tribes of Northeast Asia. After founding the Mongol Empire and being proclaimed \"Genghis Khan\", he started the Mongol invasions that resulted in the conquest of most of Eurasia. These included raids or invasions of the Qara Khitai, Caucasus, Khwarezmid Empire, Western Xia and Jin dynasties. These campaigns were often accompanied by wholesale massacres of the civilian populations – especially in the Khwarezmian and Xia controlled lands. By the end of his life, the Mongol Empire occupied a substantial portion of Central Asia and China.\n",
      "Question: Who did Genghis Khan unite before he began conquering the rest of Eurasia?\n",
      "Answers: ['many of nomadic tribes of northeast asia', 'nomadic tribes of northeast asia', 'many of nomadic tribes of northeast asia']\n",
      "Predictions: [ELECTRA] nomadic tribes [BiDAF] mongol empire\n",
      "\n",
      "Context: Undergraduate students are required to take a distribution of courses to satisfy the university's core curriculum known as the Common Core. In 2012-2013, the Core classes at Chicago were limited to 17 students, and are generally led by a full-time professor (as opposed to a teaching assistant). As of the 2013–2014 school year, 15 courses and demonstrated proficiency in a foreign language are required under the Core. Undergraduate courses at the University of Chicago are known for their demanding standards, heavy workload and academic difficulty; according to Uni in the USA, \"Among the academic cream of American universities – Harvard, Yale, Princeton, MIT, and the University of Chicago – it is UChicago that can most convincingly claim to provide the most rigorous, intense learning experience.\"\n",
      "Question: UChicago claims to have what kind of learning experience compared to other universities? \n",
      "Answers: ['most rigorous intense', 'most rigorous intense']\n",
      "Predictions: [ELECTRA] intense [BiDAF] intense\n",
      "\n",
      "Context: Two days later, the FBI ordered the Alien Property Custodian to seize Tesla's belongings, even though Tesla was an American citizen. Tesla's entire estate from the Hotel New Yorker and other New York City hotels was transported to the Manhattan Storage and Warehouse Company under the Office of Alien Property (OAP) seal. John G. Trump, a professor at M.I.T. and a well-known electrical engineer serving as a technical aide to the National Defense Research Committee, was called in to analyze the Tesla items in OAP custody. After a three-day investigation, Trump's report concluded that there was nothing which would constitute a hazard in unfriendly hands, stating:\n",
      "Question: What happened to Tesla's belongings?\n",
      "Answers: ['fbi ordered alien property custodian to seize teslas belongings', 'fbi ordered alien property custodian to seize teslas belongings', 'alien property custodian']\n",
      "Predictions: [ELECTRA] transported to manhattan storage and warehouse company [BiDAF] even though tesla was american citizen\n",
      "\n",
      "Context: The outcome was one of the most significant developments in a century of Anglo-French conflict. France ceded its territory east of the Mississippi to Great Britain. It ceded French Louisiana west of the Mississippi River (including New Orleans) to its ally Spain, in compensation for Spain's loss to Britain of Florida (Spain had ceded this to Britain in exchange for the return of Havana, Cuba). France's colonial presence north of the Caribbean was reduced to the islands of Saint Pierre and Miquelon, confirming Britain's position as the dominant colonial power in eastern North America.\n",
      "Question: What territory was ceded to Britain?\n",
      "Answers: ['territory east of mississippi to great britain', 'france', 'territory east of mississippi', 'east of mississippi', 'territory east of mississippi']\n",
      "Predictions: [ELECTRA] french louisiana west of mississippi river including new orleans [BiDAF] florida\n",
      "\n",
      "Context: This is true throughout most of the United States as well. However, alternative approaches for primary education do exist. One of these, sometimes referred to as a \"platoon\" system, involves placing a group of students together in one class that moves from one specialist to another for every subject. The advantage here is that students learn from teachers who specialize in one subject and who tend to be more knowledgeable in that one area than a teacher who teaches many subjects. Students still derive a strong sense of security by staying with the same group of peers for all classes.\n",
      "Question: The teacher's in a platoon style teaching are usually more _____?\n",
      "Answers: ['knowledgeable', 'knowledgeable', 'knowledgeable']\n",
      "Predictions: [ELECTRA] teacher who teaches many subjects [BiDAF] advantage here is that students learn from teachers who specialize in one subject and who tend to be more knowledgeable in that one area than teacher who teaches many subjects\n",
      "\n",
      "Context: In September 1760, and before any hostilities erupted, Governor Vaudreuil negotiated from Montreal a capitulation with General Amherst. Amherst granted Vaudreuil's request that any French residents who chose to remain in the colony would be given freedom to continue worshiping in their Roman Catholic tradition, continued ownership of their property, and the right to remain undisturbed in their homes. The British provided medical treatment for the sick and wounded French soldiers and French regular troops were returned to France aboard British ships with an agreement that they were not to serve again in the present war.\n",
      "Question: What British General negotiated at Montreal?\n",
      "Answers: ['general amherst', 'general amherst', 'general amherst', 'amherst', 'amherst']\n",
      "Predictions: [ELECTRA] governor vaudreuil negotiated from montreal capitulation with general amherst [BiDAF] governor vaudreuil\n",
      "\n",
      "Context: Many complexity classes are defined using the concept of a reduction. A reduction is a transformation of one problem into another problem. It captures the informal notion of a problem being at least as difficult as another problem. For instance, if a problem X can be solved using an algorithm for Y, X is no more difficult than Y, and we say that X reduces to Y. There are many different types of reductions, based on the method of reduction, such as Cook reductions, Karp reductions and Levin reductions, and the bound on the complexity of reductions, such as polynomial-time reductions or log-space reductions.\n",
      "Question: What are two examples of different types of reduction?\n",
      "Answers: ['karp reductions and levin reductions', 'cook reductions karp reductions']\n",
      "Predictions: [ELECTRA] cook reductions karp reductions and levin reductions [BiDAF] cook reductions karp reductions and levin reductions and bound on complexity of reductions such as polynomialtime reductions or logspace reductions\n",
      "\n",
      "Context: Crime rate has also been shown to be correlated with inequality in society. Most studies looking into the relationship have concentrated on homicides – since homicides are almost identically defined across all nations and jurisdictions. There have been over fifty studies showing tendencies for violence to be more common in societies where income differences are larger. Research has been conducted comparing developed countries with undeveloped countries, as well as studying areas within countries. Daly et al. 2001 found that among U.S States and Canadian Provinces there is a tenfold difference in homicide rates related to inequality. They estimated that about half of all variation in homicide rates can be accounted for by differences in the amount of inequality in each province or state. Fajnzylber et al. (2002) found a similar relationship worldwide. Among comments in academic literature on the relationship between homicides and inequality are:\n",
      "Question: How much of a difference in homicide rates are related to inequality?\n",
      "Answers: ['tenfold', 'tenfold', 'tenfold']\n",
      "Predictions: [ELECTRA] tenf [BiDAF] half\n",
      "\n",
      "Context: On Carolina's next possession fullback Mike Tolbert lost a fumble while being tackled by safety Darian Stewart, which linebacker Danny Trevathan recovered on the Broncos 40-yard line. However, the Panthers soon took the ball back when defensive end Kony Ealy tipped a Manning pass to himself and then intercepted it, returning the ball 19 yards to the Panthers 39-yard line with 1:55 left on the clock. The Panthers could not gain any yards with their possession and had to punt. After a Denver punt, Carolina drove to the Broncos 45-yard line. But with 11 seconds left, Newton was sacked by DeMarcus Ware as time expired in the half.\n",
      "Question: How did the drive end for the Panthers?\n",
      "Answers: ['punt', 'newton was sacked', 'sacked']\n",
      "Predictions: [ELECTRA] had to punt [BiDAF] kony ealy tipped manning pass to himself and then intercepted it\n",
      "\n",
      "Context: A Pharmacy Technician in the UK is considered a health care professional and often does not work under the direct supervision of a pharmacist (if employed in a hospital pharmacy) but instead is supervised and managed by other senior pharmacy technicians. In the UK the role of a PhT has grown and responsibility has been passed on to them to manage the pharmacy department and specialised areas in pharmacy practice allowing pharmacists the time to specialise in their expert field as medication consultants spending more time working with patients and in research. A pharmacy technician once qualified has to register as a professional on the General Pharmaceutical Council (GPhC) register. The GPhC is the governing body for pharmacy health care professionals and this is who regulates the practice of pharmacists and pharmacy technicians.\n",
      "Question: With what body must a pharmacy technician register?\n",
      "Answers: ['general pharmaceutical council gphc register', 'general pharmaceutical council gphc', 'general pharmaceutical council gphc register']\n",
      "Predictions: [ELECTRA] general pharmaceutical council [BiDAF] gphc\n",
      "\n",
      "Context: Since the 1980s, Lutheran Church denominations have repudiated Martin Luther's statements against the Jews and have rejected the use of them to incite hatred against Lutherans. Strommen et al.'s 1970 survey of 4,745 North American Lutherans aged 15–65 found that, compared to the other minority groups under consideration, Lutherans were the least prejudiced toward Jews. Nevertheless, Professor Richard (Dick) Geary, former Professor of Modern History at the University of Nottingham, England, and the author of Hitler and Nazism (Routledge 1993), wrote in the journal History Today an article on who voted for the Nazis in elections held from 1928-1933, where he claimed that from his research he found that the Nazis gained disproportionately more votes from Protestant than Catholic areas of Germany.\n",
      "Question: Who found that Protestants voted for Nazis more than Catholics?\n",
      "Answers: ['richard dick geary', 'richard dick geary', 'geary']\n",
      "Predictions: [ELECTRA] professor richard dick geary [BiDAF] strommen et als 1970 survey of 4745 north american lutherans aged 15–65 found that compared to other minority groups under consideration lutherans were least prejudiced toward jews nevertheless professor richard dick geary former professor of modern history at university of nottingham england and author of hitler and nazism routledge 1993 wrote in journal history today article on who voted for nazis in elections held from 19281933 where he claimed that from his research\n",
      "\n",
      "Context: The length of the Rhine is conventionally measured in \"Rhine-kilometers\" (Rheinkilometer), a scale introduced in 1939 which runs from the Old Rhine Bridge at Constance (0 km) to Hoek van Holland (1036.20 km). The river length is significantly shortened from the river's natural course due to number of canalisation projects completed in the 19th and 20th century.[note 7] The \"total length of the Rhine\", to the inclusion of Lake Constance and the Alpine Rhine is more difficult to measure objectively; it was cited as 1,232 kilometres (766 miles) by the Dutch Rijkswaterstaat in 2010.[note 1]\n",
      "Question: Why has the Rhine been shortened?\n",
      "Answers: ['canalisation projects', 'canalisation projects', 'from rivers natural course due to number of canalisation projects completed in 19th and 20th century', 'canalisation projects', 'canalisation projects']\n",
      "Predictions: [ELECTRA] due to number of canalisation projects completed in 19th and 20th century [BiDAF] number of canalisation projects\n",
      "\n",
      "Context: Civil disobedients have chosen a variety of different illegal acts. Bedau writes, \"There is a whole class of acts, undertaken in the name of civil disobedience, which, even if they were widely practiced, would in themselves constitute hardly more than a nuisance (e.g. trespassing at a nuclear-missile installation)...Such acts are often just a harassment and, at least to the bystander, somewhat inane...The remoteness of the connection between the disobedient act and the objectionable law lays such acts open to the charge of ineffectiveness and absurdity.\" Bedau also notes, though, that the very harmlessness of such entirely symbolic illegal protests toward public policy goals may serve a propaganda purpose. Some civil disobedients, such as the proprietors of illegal medical cannabis dispensaries and Voice in the Wilderness, which brought medicine to Iraq without the permission of the U.S. Government, directly achieve a desired social goal (such as the provision of medication to the sick) while openly breaking the law. Julia Butterfly Hill lived in Luna, a 180-foot (55 m)-tall, 600-year-old California Redwood tree for 738 days, successfully preventing it from being cut down.\n",
      "Question: What type of actions are taken sometimes in civil disobedience?\n",
      "Answers: ['illegal acts', 'symbolic illegal protests', 'symbolic illegal protests', 'variety of different illegal acts', 'illegal']\n",
      "Predictions: [ELECTRA] harassment [BiDAF] harassment\n",
      "\n",
      "Context: Disorders of the immune system can result in autoimmune diseases, inflammatory diseases and cancer. Immunodeficiency occurs when the immune system is less active than normal, resulting in recurring and life-threatening infections. In humans, immunodeficiency can either be the result of a genetic disease such as severe combined immunodeficiency, acquired conditions such as HIV/AIDS, or the use of immunosuppressive medication. In contrast, autoimmunity results from a hyperactive immune system attacking normal tissues as if they were foreign organisms. Common autoimmune diseases include Hashimoto's thyroiditis, rheumatoid arthritis, diabetes mellitus type 1, and systemic lupus erythematosus. Immunology covers the study of all aspects of the immune system.\n",
      "Question: Name one of the causes of immunodeficiency.\n",
      "Answers: ['genetic disease', 'genetic disease', 'hivaids']\n",
      "Predictions: [ELECTRA] lifethreatening infections in humans immunodeficiency can either be result of genetic disease such as severe combined immunodeficiency acquired conditions such as hivaids or use of immunosuppressive medication [BiDAF] autoimmune diseases\n",
      "\n",
      "Context: Warsaw, especially its city centre (Śródmieście), is home not only to many national institutions and government agencies, but also to many domestic and international companies. In 2006, 304,016 companies were registered in the city. Warsaw's ever-growing business community has been noticed globally, regionally, and nationally. MasterCard Emerging Market Index has noted Warsaw's economic strength and commercial center. Moreover, Warsaw was ranked as the 7th greatest emerging market. Foreign investors' financial participation in the city's development was estimated in 2002 at over 650 million euro. Warsaw produces 12% of Poland's national income, which in 2008 was 305.1% of the Polish average, per capita (or 160% of the European Union average). The GDP per capita in Warsaw amounted to PLN 94 000 in 2008 (c. EUR 23 800, USD 33 000). Total nominal GDP of the city in 2010 amounted to 191.766 billion PLN, 111696 PLN per capita, which was 301,1 % of Polish average. Warsaw leads the region of East-Central Europe in foreign investment and in 2006, GDP growth met expectations with a level of 6.1%. It also has one of the fastest growing economies, with GDP growth at 6.5 percent in 2007 and 6.1 percent in the first quarter of 2008.\n",
      "Question: What was the total nominal GDP of Warsaw in 2010?\n",
      "Answers: ['191766 billion pln', '191766 billion pln', '191766 billion pln']\n",
      "Predictions: [ELECTRA] 191766 billion [BiDAF] 191766 billion\n",
      "\n",
      "Context: Decisions in-between the four-year meetings are made by the Mission Council (usually consisting of church bishops). One of the most high profile decisions in recent years by one of the councils was a decision by the Mission Council of the South Central Jurisdiction which in March 2007 approved a 99-year lease of 36 acres (150,000 m2) at Southern Methodist University for the George W. Bush Presidential Library. The decision generated controversy in light of Bush's support of the Iraq War which the church bishops have criticized. A debate over whether the decision should or could be submitted for approval by the Southern Jurisdictional Conference at its July 2008 meeting in Dallas, Texas, remains unresolved.\n",
      "Question: Why was the 36 acres leased?\n",
      "Answers: ['for george w bush presidential library', 'george w bush presidential library', 'george w bush presidential library']\n",
      "Predictions: [ELECTRA] bushs support of iraq war [BiDAF] one of most high profile decisions in recent years by one of councils was decision by mission council of south central jurisdiction which in march 2007 approved 99year lease of 36 acres 150000 m2 at southern methodist university for george w bush presidential library\n",
      "\n",
      "Context: In 1875, Tesla enrolled at Austrian Polytechnic in Graz, Austria, on a Military Frontier scholarship. During his first year, Tesla never missed a lecture, earned the highest grades possible, passed nine exams (nearly twice as many required), started a Serbian culture club, and even received a letter of commendation from the dean of the technical faculty to his father, which stated, \"Your son is a star of first rank.\" Tesla claimed that he worked from 3 a.m. to 11 p.m., no Sundays or holidays excepted. He was \"mortified when [his] father made light of [those] hard won honors.\" After his father's death in 1879, Tesla found a package of letters from his professors to his father, warning that unless he were removed from the school, Tesla would be killed through overwork. During his second year, Tesla came into conflict with Professor Poeschl over the Gramme dynamo, when Tesla suggested that commutators weren't necessary. At the end of his second year, Tesla lost his scholarship and became addicted to gambling. During his third year, Tesla gambled away his allowance and his tuition money, later gambling back his initial losses and returning the balance to his family. Tesla said that he \"conquered [his] passion then and there,\" but later he was known to play billiards in the US. When exam time came, Tesla was unprepared and asked for an extension to study, but was denied. He never graduated from the university and did not receive grades for the last semester.\n",
      "Question: How did Tesla lose his tuition money?\n",
      "Answers: ['gambled', 'gambling', 'gambling']\n",
      "Predictions: [ELECTRA] gambled away [BiDAF] gambled away his allowance and his tuition money later gambling back his initial losses and returning balance to his family\n",
      "\n",
      "Context: Luther's translation used the variant of German spoken at the Saxon chancellery, intelligible to both northern and southern Germans. He intended his vigorous, direct language to make the Bible accessible to everyday Germans, \"for we are removing impediments and difficulties so that other people may read it without hindrance.\"\n",
      "Question: Why did he want to make the language accessible to people?\n",
      "Answers: ['read it without hindrance', 'may read it without hindrance', 'removing impediments and difficulties so that other people may read it without hindrance']\n",
      "Predictions: [ELECTRA] we are removing impediments and difficulties [BiDAF] for we are removing impediments and difficulties\n",
      "\n",
      "Context: The earliest known movie featuring some exterior scenes filmed in the city is On the Night of the Fire (1939), though by and large the action is studio-bound. Later came The Clouded Yellow (1951) and Payroll (1961), both of which feature more extensive scenes filmed in the city. The 1971 film Get Carter was shot on location in and around Newcastle and offers an opportunity to see what Newcastle looked like in the 1960s and early 1970s. The city was also backdrop to another gangster film, the 1988 film noir thriller Stormy Monday, directed by Mike Figgis and starring Tommy Lee Jones, Melanie Griffith, Sting and Sean Bean.\n",
      "Question: What famous musician starred in Stormy Monday?\n",
      "Answers: ['sting', 'sting', 'sting']\n",
      "Predictions: [ELECTRA] sean bean [BiDAF] mike figgis\n",
      "\n",
      "Context: Montpellier was among the most important of the 66 \"villes de sûreté\" that the Edict of 1598 granted to the Huguenots. The city's political institutions and the university were all handed over to the Huguenots. Tension with Paris led to a siege by the royal army in 1622. Peace terms called for the dismantling of the city's fortifications. A royal citadel was built and the university and consulate were taken over by the Catholic party. Even before the Edict of Alès (1629), Protestant rule was dead and the ville de sûreté was no more.[citation needed]\n",
      "Question: When was this proclamation issued?\n",
      "Answers: ['1629', '1629', '1629']\n",
      "Predictions: [ELECTRA] 1598 [BiDAF] 1622\n",
      "\n",
      "Context: The collection of drawings includes over 10,000 British and 2,000 old master works, including works by: Dürer, Giovanni Benedetto Castiglione, Bernardo Buontalenti, Rembrandt, Antonio Verrio, Paul Sandby, John Russell, Angelica Kauffman, John Flaxman, Hugh Douglas Hamilton, Thomas Rowlandson, William Kilburn, Thomas Girtin, Jean Auguste Dominique Ingres, David Wilkie, John Martin, Samuel Palmer, Sir Edwin Henry Landseer, Lord Frederic Leighton, Sir Samuel Luke Fildes and Aubrey Beardsley. Modern British artists represented in the collection include: Paul Nash, Percy Wyndham Lewis, Eric Gill, Stanley Spencer, John Piper, Graham Sutherland, Lucian Freud and David Hockney.\n",
      "Question: Which famous Germain Renaissance painter and printmaker is represented in the drawings collection of the V&A?\n",
      "Answers: ['dürer', 'dürer', 'dürer']\n",
      "Predictions: [ELECTRA] jean auguste dominique ingres [BiDAF] lord frederic leighton sir samuel luke fildes and aubrey beardsley modern british artists represented in collection include paul nash percy wyndham lewis eric gill stanley spencer john piper graham sutherland lucian freud and david hockney\n",
      "\n",
      "Context: During the same year, Tesla wrote a treatise, The Art of Projecting Concentrated Non-dispersive Energy through the Natural Media, concerning charged particle beam weapons. Tesla published the document in an attempt to expound on the technical description of a \"superweapon that would put an end to all war.\" This treatise is currently in the Nikola Tesla Museum archive in Belgrade. It describes an open-ended vacuum tube with a gas jet seal that allows particles to exit, a method of charging particles to millions of volts, and a method of creating and directing non-dispersive particle streams (through electrostatic repulsion). Tesla tried to interest the US War Department, the United Kingdom, the Soviet Union, and Yugoslavia in the device.\n",
      "Question: Where can the Treatise be found?\n",
      "Answers: ['nikola tesla museum archive', 'nikola tesla museum archive', 'nikola tesla museum']\n",
      "Predictions: [ELECTRA] nikola tesla museum archive in belgrade [BiDAF] nikola tesla museum archive in belgrade\n",
      "\n",
      "Context: The other category of schools are those run and partly or fully funded by private individuals, private organizations and religious groups. The ones that accept government funds are called 'aided' schools. The private 'un-aided' schools are fully funded by private parties. The standard and the quality of education is quite high. Technically, these would be categorized as private schools, but many of them have the name \"Public School\" appended to them, e.g., the Galaxy Public School in Kathmandu. Most of the middle-class families send their children to such schools, which might be in their own city or far off, like boarding schools. The medium of education is English, but as a compulsory subject, Nepali and/or the state's official language is also taught. Preschool education is mostly limited to organized neighbourhood nursery schools.\n",
      "Question: What is a term for schools that receive government assistance?\n",
      "Answers: ['aided', 'aided', 'aided']\n",
      "Predictions: [ELECTRA] aided schools [BiDAF] aided schools\n",
      "\n",
      "Context: The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated the Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by defeating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made eight appearances in the Super Bowl.\n",
      "Question: How many times have the Panthers been in the Super Bowl?\n",
      "Answers: ['2', 'second', 'second']\n",
      "Predictions: [ELECTRA] eight [BiDAF] eight\n",
      "\n",
      "Context: Fresno (/ˈfrɛznoʊ/ FREZ-noh), the county seat of Fresno County, is a city in the U.S. state of California. As of 2015, the city's population was 520,159, making it the fifth-largest city in California, the largest inland city in California and the 34th-largest in the nation. Fresno is in the center of the San Joaquin Valley and is the largest city in the Central Valley, which contains the San Joaquin Valley. It is approximately 220 miles (350 km) northwest of Los Angeles, 170 miles (270 km) south of the state capital, Sacramento, or 185 miles (300 km) south of San Francisco. The name Fresno means \"ash tree\" in Spanish, and an ash leaf is featured on the city's flag.\n",
      "Question: How far is Fresno from Los Angeles?\n",
      "Answers: ['220 miles 350 km', '220 miles 350 km', '220 miles 350 km']\n",
      "Predictions: [ELECTRA] 220 miles [BiDAF] 220 miles\n",
      "\n",
      "Context: Many complexity classes are defined using the concept of a reduction. A reduction is a transformation of one problem into another problem. It captures the informal notion of a problem being at least as difficult as another problem. For instance, if a problem X can be solved using an algorithm for Y, X is no more difficult than Y, and we say that X reduces to Y. There are many different types of reductions, based on the method of reduction, such as Cook reductions, Karp reductions and Levin reductions, and the bound on the complexity of reductions, such as polynomial-time reductions or log-space reductions.\n",
      "Question: According to reduction, if X and Y can be solved by the same algorithm then X performs what function in relationship to Y?\n",
      "Answers: ['reduces', 'reduces', 'x reduces to y']\n",
      "Predictions: [ELECTRA] x reduces [BiDAF] no more difficult\n",
      "\n",
      "Context: The weak force is due to the exchange of the heavy W and Z bosons. Its most familiar effect is beta decay (of neutrons in atomic nuclei) and the associated radioactivity. The word \"weak\" derives from the fact that the field strength is some 1013 times less than that of the strong force. Still, it is stronger than gravity over short distances. A consistent electroweak theory has also been developed, which shows that electromagnetic forces and the weak force are indistinguishable at a temperatures in excess of approximately 1015 kelvins. Such temperatures have been probed in modern particle accelerators and show the conditions of the universe in the early moments of the Big Bang.\n",
      "Question: How many times less is the strenght of the weak field compared to the strong?\n",
      "Answers: ['1013', '1013', '1013', '1013']\n",
      "Predictions: [ELECTRA] 1013 times [BiDAF] 1013 times\n",
      "\n",
      "Context: During his time at his lab, Tesla observed unusual signals from his receiver which he concluded may be communications from another planet. He mentioned them in a letter to reporter Julian Hawthorne at the Philadelphia North American on 8 December 1899 and in a December 1900 letter about possible discoveries in the new century to the Red Cross Society where he referred to messages \"from another world\" that read \"1... 2... 3...\". Reporters treated it as a sensational story and jumped to the conclusion Tesla was hearing signals from Mars. He expanded on the signals he heard in a 9 February 1901 Collier's Weekly article \"Talking With Planets\" where he said it had not been immediately apparent to him that he was hearing \"intelligently controlled signals\" and that the signals could come from Mars, Venus, or other planets. It has been hypothesized that he may have intercepted Marconi's European experiments in July 1899—Marconi may have transmitted the letter S (dot/dot/dot) in a naval demonstration, the same three impulses that Tesla hinted at hearing in Colorado—or signals from another experimenter in wireless transmission.\n",
      "Question: When did Marconi transmit similar signals to those Tesla received?\n",
      "Answers: ['july 1899', 'july 1899', 'july 1899']\n",
      "Predictions: [ELECTRA] july 1899— [BiDAF] 9 february 1901\n",
      "\n",
      "Context: Deforestation is the conversion of forested areas to non-forested areas. The main sources of deforestation in the Amazon are human settlement and development of the land. Prior to the early 1960s, access to the forest's interior was highly restricted, and the forest remained basically intact. Farms established during the 1960s were based on crop cultivation and the slash and burn method. However, the colonists were unable to manage their fields and the crops because of the loss of soil fertility and weed invasion. The soils in the Amazon are productive for just a short period of time, so farmers are constantly moving to new areas and clearing more land. These farming practices led to deforestation and caused extensive environmental damage. Deforestation is considerable, and areas cleared of forest are visible to the naked eye from outer space.\n",
      "Question: What is notable about the Amazon forest when it is seen from space?\n",
      "Answers: ['areas cleared of forest are visible to naked eye', 'areas cleared of forest', 'areas cleared of forest are visible to naked eye']\n",
      "Predictions: [ELECTRA] deforestation is considerable and areas cleared of forest [BiDAF] naked eye\n",
      "\n",
      "Context: He insisted that, since forgiveness was God's alone to grant, those who claimed that indulgences absolved buyers from all punishments and granted them salvation were in error. Christians, he said, must not slacken in following Christ on account of such false assurances.\n",
      "Question: What did Luther call these donations?\n",
      "Answers: ['false assurances', 'false assurances', 'false assurances']\n",
      "Predictions: [ELECTRA] indulgences [BiDAF] indulgences absolved buyers from all punishments and granted them salvation were in error christians\n",
      "\n",
      "Context: In accordance with his father's wishes, Luther enrolled in law school at the same university that year but dropped out almost immediately, believing that law represented uncertainty. Luther sought assurances about life and was drawn to theology and philosophy, expressing particular interest in Aristotle, William of Ockham, and Gabriel Biel. He was deeply influenced by two tutors, Bartholomaeus Arnoldi von Usingen and Jodocus Trutfetter, who taught him to be suspicious of even the greatest thinkers and to test everything himself by experience. Philosophy proved to be unsatisfying, offering assurance about the use of reason but none about loving God, which to Luther was more important. Reason could not lead men to God, he felt, and he thereafter developed a love-hate relationship with Aristotle over the latter's emphasis on reason. For Luther, reason could be used to question men and institutions, but not God. Human beings could learn about God only through divine revelation, he believed, and Scripture therefore became increasingly important to him.\n",
      "Question: What educational study did Luther start to pursue and immediately  drop?\n",
      "Answers: ['law', 'law', 'law']\n",
      "Predictions: [ELECTRA] law school [BiDAF] law school\n",
      "\n",
      "Context: The 2010 United States Census reported that Fresno had a population of 494,665. The population density was 4,404.5 people per square mile (1,700.6/km²). The racial makeup of Fresno was 245,306 (49.6%) White, 40,960 (8.3%) African American, 8,525 (1.7%) Native American, 62,528 (12.6%) Asian (3.6% Hmong, 1.7% Indian, 1.2% Filipino, 1.2% Laotian, 1.0% Thai, 0.8% Cambodian, 0.7% Chinese, 0.5% Japanese, 0.4% Vietnamese, 0.2% Korean), 849 (0.2%) Pacific Islander, 111,984 (22.6%) from other races, and 24,513 (5.0%) from two or more races. Hispanic or Latino of any race were 232,055 persons (46.9%). Among the Hispanic population, 42.7% of the total population are Mexican, 0.4% Salvadoran, and 0.4% Puerto Rican. Non-Hispanic Whites were 30.0% of the population in 2010, down from 72.6% in 1970.\n",
      "Question: What was the percentage of whit people in Fresno in 2010?\n",
      "Answers: ['496', '496', '496']\n",
      "Predictions: [ELECTRA] 300 [BiDAF] 726\n",
      "\n",
      "Context: While experimenting, Tesla inadvertently faulted a power station generator, causing a power outage. In August 1917, Tesla explained what had happened in The Electrical Experimenter: \"As an example of what has been done with several hundred kilowatts of high frequency energy liberated, it was found that the dynamos in a power house six miles away were repeatedly burned out, due to the powerful high frequency currents set up in them, and which caused heavy sparks to jump through the windings and destroy the insulation!\"\n",
      "Question: What happened to the dynamos in the power station?\n",
      "Answers: ['repeatedly burned out', 'repeatedly burned out', 'repeatedly burned out']\n",
      "Predictions: [ELECTRA] burned out [BiDAF] dynamos in power house six miles away were repeatedly burned out due to powerful high frequency currents set up in them\n",
      "\n",
      "Context: The internal cavity forms: a mouth that can usually be closed by muscles; a pharynx (\"throat\"); a wider area in the center that acts as a stomach; and a system of internal canals. These branch through the mesoglea to the most active parts of the animal: the mouth and pharynx; the roots of the tentacles, if present; all along the underside of each comb row; and four branches round the sensory complex at the far end from the mouth – two of these four branches terminate in anal pores. The inner surface of the cavity is lined with an epithelium, the gastrodermis. The mouth and pharynx have both cilia and well-developed muscles. In other parts of the canal system, the gastrodermis is different on the sides nearest to and furthest from the organ that it supplies. The nearer side is composed of tall nutritive cells that store nutrients in vacuoles (internal compartments), germ cells that produce eggs or sperm, and photocytes that produce bioluminescence. The side furthest from the organ is covered with ciliated cells that circulate water through the canals, punctuated by ciliary rosettes, pores that are surrounded by double whorls of cilia and connect to the mesoglea.\n",
      "Question: What does the internal cavity contain?\n",
      "Answers: ['mouth that can usually be closed by muscles pharynx throat wider area in center that acts as stomach and system of internal canals', 'system of internal canals', 'gastrodermis']\n",
      "Predictions: [ELECTRA] canals [BiDAF] mouth that can usually be closed by muscles\n",
      "\n",
      "Context: During the mass high school education movement from 1910–1940, there was an increase in skilled workers, which led to a decrease in the price of skilled labor. High school education during the period was designed to equip students with necessary skill sets to be able to perform at work. In fact, it differs from the present high school education, which is regarded as a stepping-stone to acquire college and advanced degrees. This decrease in wages caused a period of compression and decreased inequality between skilled and unskilled workers. Education is very important for the growth of the economy, however educational inequality in gender also influence towards the economy. Lagerlof and Galor stated that gender inequality in education can result to low economic growth, and continued gender inequality in education, thus creating a poverty trap. It is suggested that a large gap in male and female education may indicate backwardness and so may be associated with lower economic growth, which can explain why there is economic inequality between countries.\n",
      "Question: How did the education during the high school education movement differ from the subsequent high school education?\n",
      "Answers: ['designed to equip students with necessary skill sets to be able to perform at work', 'designed to equip students with necessary skill sets to be able to perform at work', 'designed to equip students with necessary skill sets to be able to perform at work']\n",
      "Predictions: [ELECTRA] steppingstone to acquire college and advanced degrees [BiDAF] as steppingstone to acquire college and advanced degrees\n",
      "\n",
      "Context: With such a small force, the invading Mongols were forced to change strategies and resort to inciting internal revolt among Kuchlug's supporters, leaving the Qara Khitai more vulnerable to Mongol conquest. As a result, Kuchlug's army was defeated west of Kashgar. Kuchlug fled again, but was soon hunted down by Jebe's army and executed. By 1218, as a result of defeat of Qara Khitai, the Mongol Empire and its control extended as far west as Lake Balkhash, which bordered the Khwarezmia (Khwarezmid Empire), a Muslim state that reached the Caspian Sea to the west and Persian Gulf and the Arabian Sea to the south.\n",
      "Question: Which empire bordered the Mongol Empire to the west in 1218?\n",
      "Answers: ['khwarezmid empire', 'khwarezmid', 'khwarezmid empire']\n",
      "Predictions: [ELECTRA] khwarezmia khwarezmid empire [BiDAF] khwarezmia\n",
      "\n",
      "Context: As opposed to broadcasts of primetime series, CBS broadcast special episodes of its late night talk shows as its lead-out programs for Super Bowl 50, beginning with a special episode of The Late Show with Stephen Colbert following the game. Following a break for late local programming, CBS also aired a special episode of The Late Late Show with James Corden.\n",
      "Question: What talk show followed immediately after Super Bowl 50 on CBS?\n",
      "Answers: ['late show with stephen colbert', 'late show with stephen colbert', 'late show with stephen colbert']\n",
      "Predictions: [ELECTRA] late night talk shows as its leadout programs for super bowl 50 beginning with special episode of late show [BiDAF] late night\n",
      "\n",
      "Context: The strong force only acts directly upon elementary particles. However, a residual of the force is observed between hadrons (the best known example being the force that acts between nucleons in atomic nuclei) as the nuclear force. Here the strong force acts indirectly, transmitted as gluons, which form part of the virtual pi and rho mesons, which classically transmit the nuclear force (see this topic for more). The failure of many searches for free quarks has shown that the elementary particles affected are not directly observable. This phenomenon is called color confinement.\n",
      "Question: How are nuclear forces transmitted?\n",
      "Answers: ['as gluons', 'as gluons', 'as gluons', 'as gluons']\n",
      "Predictions: [ELECTRA] gluons [BiDAF] gluons\n",
      "\n",
      "Context: Another green space in Newcastle is the Town Moor, lying immediately north of the city centre. It is larger than London's famous Hyde Park and Hampstead Heath put together and the freemen of the city have the right to graze cattle on it. The right incidentally extends to the pitch of St. James' Park, Newcastle United Football Club's ground, though this is not exercised, although the Freemen do collect rent for the loss of privilege. Honorary freemen include Bob Geldof, King Harald V of Norway, Bobby Robson, Alan Shearer, the late Nelson Mandela and the Royal Shakespeare Company. The Hoppings funfair, said to be the largest travelling funfair in Europe, is held here annually in June.\n",
      "Question: What an the freeman of Newcastle do with their cows on the Town Moor?\n",
      "Answers: ['graze', 'graze', 'graze cattle on it']\n",
      "Predictions: [ELECTRA] graze cattle [BiDAF] another green space\n",
      "\n",
      "Context: The first recorded travels by Europeans to China and back date from this time. The most famous traveler of the period was the Venetian Marco Polo, whose account of his trip to \"Cambaluc,\" the capital of the Great Khan, and of life there astounded the people of Europe. The account of his travels, Il milione (or, The Million, known in English as the Travels of Marco Polo), appeared about the year 1299. Some argue over the accuracy of Marco Polo's accounts due to the lack of mentioning the Great Wall of China, tea houses, which would have been a prominent sight since Europeans had yet to adopt a tea culture, as well the practice of foot binding by the women in capital of the Great Khan. Some suggest that Marco Polo acquired much of his knowledge through contact with Persian traders since many of the places he named were in Persian.\n",
      "Question: What was the English title of Polo's book?\n",
      "Answers: ['travels of marco polo', 'travels of marco polo', 'travels of marco polo']\n",
      "Predictions: [ELECTRA] il milione or million known in english as travels of marco polo [BiDAF] il milione or million known in english as travels of marco polo\n",
      "\n",
      "Context: In Marxian analysis, capitalist firms increasingly substitute capital equipment for labor inputs (workers) under competitive pressure to reduce costs and maximize profits. Over the long-term, this trend increases the organic composition of capital, meaning that less workers are required in proportion to capital inputs, increasing unemployment (the \"reserve army of labour\"). This process exerts a downward pressure on wages. The substitution of capital equipment for labor (mechanization and automation) raises the productivity of each worker, resulting in a situation of relatively stagnant wages for the working class amidst rising levels of property income for the capitalist class.\n",
      "Question: What impact does higher worker productivity and leveled pay have on higher earners?\n",
      "Answers: ['rising levels of property income', 'rising levels of property income', 'downward pressure on wages']\n",
      "Predictions: [ELECTRA] downward pressure on wages substitution of capital equipment for labor mechanization and automation raises productivity of each worker resulting in situation of relatively stagnant wages [BiDAF] downward pressure on wages substitution of capital equipment for labor mechanization and automation\n",
      "\n",
      "Context: Following the Nice Treaty, there was an attempt to reform the constitutional law of the European Union and make it more transparent; this would have also produced a single constitutional document. However, as a result of the referendum in France and the referendum in the Netherlands, the 2004 Treaty establishing a Constitution for Europe never came into force. Instead, the Lisbon Treaty was enacted. Its substance was very similar to the proposed constitutional treaty, but it was formally an amending treaty, and – though it significantly altered the existing treaties – it did not completely replace them.\n",
      "Question: What reform was attempted following the Nice Treaty?\n",
      "Answers: ['there was attempt to reform constitutional law of european union and make it more transparent', 'attempt to reform constitutional law of european union and make it more transparent', 'there was attempt to reform constitutional law of european union and make it more transparent', 'attempt to reform constitutional law of european union and make it more transparent']\n",
      "Predictions: [ELECTRA] to reform constitutional law of european union and make it more transparent [BiDAF] constitutional law of european union\n",
      "\n",
      "Context: Following their loss in the divisional round of the previous season's playoffs, the Denver Broncos underwent numerous coaching changes, including a mutual parting with head coach John Fox (who had won four divisional championships in his four years as Broncos head coach), and the hiring of Gary Kubiak as the new head coach. Under Kubiak, the Broncos planned to install a run-oriented offense with zone blocking to blend in with quarterback Peyton Manning's shotgun passing skills, but struggled with numerous changes and injuries to the offensive line, as well as Manning having his worst statistical season since his rookie year with the Indianapolis Colts in 1998, due to a plantar fasciitis injury in his heel that he had suffered since the summer, and the simple fact that Manning was getting old, as he turned 39 in the 2015 off-season. Although the team had a 7–0 start, Manning led the NFL in interceptions. In week 10, Manning suffered a partial tear of the plantar fasciitis in his left foot. He set the NFL's all-time record for career passing yards in this game, but was benched after throwing four interceptions in favor of backup quarterback Brock Osweiler, who took over as the starter for most of the remainder of the regular season. Osweiler was injured, however, leading to Manning's return during the Week 17 regular season finale, where the Broncos were losing 13–7 against the 4–11 San Diego Chargers, resulting in Manning re-claiming the starting quarterback position for the playoffs by leading the team to a key 27–20 win that enabled the team to clinch the number one overall AFC seed. Under defensive coordinator Wade Phillips, the Broncos' defense ranked number one in total yards allowed, passing yards allowed and sacks, and like the previous three seasons, the team has continued to set numerous individual, league and franchise records. With the defense carrying the team despite the issues with the offense, the Broncos finished the regular season with a 12–4 record and earned home-field advantage throughout the AFC playoffs.\n",
      "Question: Who is the head coach of the Broncos?\n",
      "Answers: ['gary kubiak', 'gary kubiak', 'kubiak']\n",
      "Predictions: [ELECTRA] john fox [BiDAF] john fox\n",
      "\n",
      "Context: The two symbols most commonly associated with pharmacy in English-speaking countries are the mortar and pestle and the ℞ (recipere) character, which is often written as \"Rx\" in typed text. The show globe was also used until the early 20th century. Pharmacy organizations often use other symbols, such as the Bowl of Hygieia which is often used in the Netherlands, conical measures, and caduceuses in their logos. Other symbols are common in different countries: the green Greek cross in France, Argentina, the United Kingdom, Belgium, Ireland, Italy, Spain, and India, the increasingly rare Gaper in the Netherlands, and a red stylized letter A in Germany and Austria (from Apotheke, the German word for pharmacy, from the same Greek root as the English word 'apothecary').\n",
      "Question: What are the two symbols that signify pharmacy in English-speaking countries?\n",
      "Answers: ['mortar and pestle and ℞ recipere character', 'mortar and pestle and ℞ recipere character', 'mortar and pestle and ℞ recipere character']\n",
      "Predictions: [ELECTRA] mortar and pestle and ℞ recipere [BiDAF] mortar and pestle and ℞\n",
      "\n",
      "Context: Decision problems are one of the central objects of study in computational complexity theory. A decision problem is a special type of computational problem whose answer is either yes or no, or alternately either 1 or 0. A decision problem can be viewed as a formal language, where the members of the language are instances whose output is yes, and the non-members are those instances whose output is no. The objective is to decide, with the aid of an algorithm, whether a given input string is a member of the formal language under consideration. If the algorithm deciding this problem returns the answer yes, the algorithm is said to accept the input string, otherwise it is said to reject the input.\n",
      "Question: What are the two simple word responses to a decision problem?\n",
      "Answers: ['yes or no', 'yes or no', 'yes or no']\n",
      "Predictions: [ELECTRA] 1 or 0 [BiDAF] yes or no or alternately either 1 or 0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for e in random_best_common_errors:\n",
    "    context = electra_errors[e][\"context\"]\n",
    "    question = electra_errors[e][\"question\"]\n",
    "    answers = electra_errors[e][\"answers\"]\n",
    "    electra_pred = electra_errors[e][\"prediction\"]\n",
    "    bidaf_pred = bidaf_errors[e][\"prediction\"]\n",
    "    print(f\"Context: {context}\")\n",
    "    print(f\"Question: {question}\")\n",
    "    print(f\"Answers: {answers}\")\n",
    "    print(f\"Predictions: [ELECTRA] {electra_pred} [BiDAF] {bidaf_pred}\")\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}