examples/ResearchPaperEvaluator/ResearchPDFEvaluator.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build Your Own Research Paper: Transform a Unstructured Research Paper to an Finetuned LLM\n",
    "Do you want to build an agent so that you can ask it anything about a research paper? In this example, we will show you how use `uniflow` and `pykoi` to extract knowledge from a Nike research paper and then finetune an LLM on these knowledge.\n",
    "\n",
    "First, we'll use `uniflow` to generate question-answers (QAs) from a pdf using OpenAI's models via `uniflow`'s `MultiFlowPipeline`.\n",
    "\n",
    "Next, we'll use `pykoi` to run supervised fine-tuning (SFT) on the QAs generated by `uniflow`.\n",
    "\n",
    "Finally, we'll use `pykoi`'s Chatbot to run the SFT model, so you can ask questions about the 10K and get answers.\n",
    "\n",
    "For this example, we're using a the paper \"An Observational Study of the Effect of Nike Vaporfly Shoes on Marathon Performance\" from Cornell.\n",
    "\n",
    ">*Note: In order to run this notebook, you need a GPU (for the `RLHF`).*\n",
    "\n",
    "### Before running the code\n",
    "\n",
    "You will need to set up a conda environment to run this notebook. You can set up the environment following the [instruction](https://github.com/CambioML/cambio-recipes/tree/main#installation).\n",
    "\n",
    "We are using `uniflow` and several of the `pykoi` modules, so you will need to install these in your environment as well:\n",
    "\n",
    "```bash\n",
    "pip3 install uniflow\n",
    "pip3 install \"pykoi[huggingface, rag, rlhf]\"\n",
    "```\n",
    "Finally, you will need to install torch:\n",
    "\n",
    "```bash\n",
    "pip3 uninstall torch\n",
    "pip3 install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu121  # cu121 means cuda 12.1\n",
    "```\n",
    "\n",
    "Next, you will need a valid [OpenAI API key](https://platform.openai.com/api-keys) to run the code. Once you have the key, set it as the environment variable `OPENAI_API_KEY` within a `.env` file in the root directory of this repository. For more details, see this [instruction](https://github.com/CambioML/cambio-recipes/tree/main#api-keys)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Generate QAs from a Research Paper using `uniflow`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Update System Path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import sys\n",
    "\n",
    "sys.path.append(\".\")\n",
    "sys.path.append(\"..\")\n",
    "sys.path.append(\"../..\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install helper packages\n",
    "If you already have these installed, feel free to skip this step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Ignoring invalid distribution -orch (/opt/conda/envs/cookbook-research/lib/python3.10/site-packages)\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: Ignoring invalid distribution -orch (/opt/conda/envs/cookbook-research/lib/python3.10/site-packages)\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!{sys.executable} -m pip install -q transformers accelerate bitsandbytes scipy #nougat-ocr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import Dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/cookbook-research/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "from uniflow.pipeline import MultiFlowsPipeline\n",
    "from uniflow.flow.config import PipelineConfig\n",
    "from uniflow.flow.config import TransformOpenAIConfig, ExtractPDFConfig\n",
    "from uniflow.op.model.model_config import  NougatModelConfig\n",
    "from uniflow.op.prompt import Context\n",
    "from uniflow.op.extract.split.constants import PARAGRAPH_SPLITTER\n",
    "\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare Input data\n",
    "\n",
    "First, let's set the current directory and input data directory, and load the raw input data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "dir_cur = os.getcwd()\n",
    "pdf_file = \"nike-paper.pdf\"\n",
    "input_file = os.path.join(f\"{dir_cur}/data/raw_input/\", pdf_file)\n",
    "\n",
    "data = [\n",
    "    {\"pdf\": input_file},\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define extract config using Nougat\n",
    "For this example, we'll run the `ExtractPDF` flow to extract the text from the 10K pdf. This uses the [Nougat](https://pypi.org/project/nougat-ocr/0.1.17/) PDF parser."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "extract_config = ExtractPDFConfig(\n",
    "    model_config=NougatModelConfig(\n",
    "        batch_size = 4 # When batch_size>1, nougat will run on CUDA, otherwise it will run on CPU\n",
    "    ),\n",
    "    splitter=PARAGRAPH_SPLITTER\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's some reference runtimes for the `batch_size` with the `PARAGRAPH_SPLITTER`:\n",
    "\n",
    "| `batch_size` | Runtime [m:s] |\n",
    "|--------------|---------------|\n",
    "| 1 | 01:05 |\n",
    "| 2 | 00:41 |\n",
    "| 4 | 00:27 |\n",
    "| 8 | 00:15 | "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare sample prompts\n",
    "Now we need to write a bit of a prompt to generate question and answer for a given paragraph. We'll generate multiple QAs for each paragraph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Modify the number of Q&A sets as desired\n",
    "number_QAs = 3 #1, 3, or 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each prompt data includes a instruction and a list of examples with \"context\", \"question\" and \"answer\". We do this by giving a sample list of `Context` examples to the `PromptTemplate` class.\n",
    "\n",
    "Modify the `few_shot_examples` to have the same number of questions as `number_QAs`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint\n",
    "\n",
    "instruction = \"\"\"\n",
    "# Instruction\n",
    "Generate {} question(s) and the corresponding answer(s) based on the context. Following \\\n",
    "the format of the examples below to include context and qas in the response.\n",
    "\n",
    "## Note: Use the below two examples just as a reference, do not include in your response.\n",
    "\"\"\".format(number_QAs)\n",
    "\n",
    "\n",
    "few_shot_examples = [\n",
    "    Context(\n",
    "        context=\"The quick brown fox jumps over the lazy black dog.\",\n",
    "        qas=[\n",
    "        {\"Question\": \"What is the color of the fox?\", \"Answer\": \"Brown\"},\n",
    "        {\"Question\": \"What is the color of the dog?\", \"Answer\": \"Black\"},\n",
    "        {\"Question\": \"What does the fox jump over?\", \"Answer\": \"Dog\"},\n",
    "        # {\"Question\": \"Was the dog lazy?\", \"Answer\": \"Yes\"},\n",
    "        # {\"Question\": \"What did the fox do?\", \"Answer\": \"Jump\"},\n",
    "\n",
    "    ]\n",
    "    ),\n",
    "    Context(\n",
    "        context=\"Snoopy can be selfish, but loves his owner, Charlie Brown.\",\n",
    "        qas=[\n",
    "        {\"Question\": \"How does Snoopy sometimes behave?\", \"Answer\": \"Selfish\"},\n",
    "        {\"Question\": \"How does Snoopy feel about his owner?\", \"Answer\": \"Loves\"},\n",
    "        {\n",
    "            \"Question\": \"What is the name of Snoopy's owner?\",\n",
    "            \"Answer\": \"Charlie Brown\",\n",
    "        },\n",
    "        # { \"Question\": \"Who can be selfish?\", \"Answer\": \"Snoopy\"},\n",
    "        # { \"Question\": \"Who is loved?\", \"Answer\": \"Charlie Brown\"},\n",
    "    ])\n",
    "]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define transform config\n",
    "Next, we set up the `TransformConfig`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "transform_config = TransformOpenAIConfig()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we customize the config for picking the answer chunk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "transform_config.prompt_template.instruction = instruction\n",
    "transform_config.prompt_template.few_shot_prompt = few_shot_examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want the response format to be JSON, we need to update two aspects of the default config:\n",
    "\n",
    "1. Change the model_name to \"gpt-4-1106-preview\", which is the only GPT-4 model that supports the JSON format.\n",
    "1. Change the response_format to a json_object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "transform_config.model_config.model_name = \"gpt-4-1106-preview\"\n",
    "transform_config.model_config.response_format = {\"type\": \"json_object\"}\n",
    "transform_config.model_config.num_call = 1\n",
    "transform_config.model_config.temperature = 0.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we update the `num_threads` and `batch_size`. You'll want to optimize this number to maximize efficiency. Note that these must be the same number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TransformOpenAIConfig(flow_name='TransformOpenAIFlow',\n",
      "                      model_config=OpenAIModelConfig(model_name='gpt-4-1106-preview',\n",
      "                                                     model_server='OpenAIModelServer',\n",
      "                                                     num_call=1,\n",
      "                                                     temperature=0.0,\n",
      "                                                     response_format={'type': 'json_object'},\n",
      "                                                     num_thread=32,\n",
      "                                                     batch_size=32),\n",
      "                      num_thread=1,\n",
      "                      prompt_template=PromptTemplate(instruction='\\n# Instruction\\nGenerate 3 question(s) and the corresponding answer(s) based on the context. Following the format of the examples below to include context and qas in the response.\\n\\n## Note: Use the below two examples just as a reference, do not include in your response.\\n', few_shot_prompt=[Context(context='The quick brown fox jumps over the lazy black dog.', qas=[{'Question': 'What is the color of the fox?', 'Answer': 'Brown'}, {'Question': 'What is the color of the dog?', 'Answer': 'Black'}, {'Question': 'What does the fox jump over?', 'Answer': 'Dog'}]), Context(context='Snoopy can be selfish, but loves his owner, Charlie Brown.', qas=[{'Question': 'How does Snoopy sometimes behave?', 'Answer': 'Selfish'}, {'Question': 'How does Snoopy feel about his owner?', 'Answer': 'Loves'}, {'Question': \"What is the name of Snoopy's owner?\", 'Answer': 'Charlie Brown'}])]))\n"
     ]
    }
   ],
   "source": [
    "num_thread_batch_size = 32\n",
    "transform_config.model_config.num_thread = num_thread_batch_size\n",
    "transform_config.model_config.batch_size = num_thread_batch_size\n",
    "pprint(transform_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's some reference for `num_thread` / `batch_size`, generating 3 QAs for each paragraph:\n",
    "\n",
    "| `num_thread` / `batch_size` | runtime [m:s] |\n",
    "| ----------------------------|---------|\n",
    "| 2 | 13:34 |\n",
    "| 4 | 09:15 |\n",
    "| 16 | 03:09 |\n",
    "| 32 | 02:21 |\n",
    "| 64 | 01:42 |\n",
    "\n",
    "Here are some references for 4 QAs/paragraph:\n",
    "| `num_thread` / `batch_size` | runtime [m:s] |\n",
    "| ----------------------------|---------|\n",
    "| 8 | 05:12 |\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run MultiFlowsPipeline\n",
    "Let's use the `PipelineConfig` to connect `extract_config` and `transform_config`, and pass that into our `MultiFlowsPipeline` to run the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/cookbook-research/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
      "  _torch_pytree._register_pytree_node(\n",
      "/opt/conda/envs/cookbook-research/lib/python3.10/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
      "  _torch_pytree._register_pytree_node(\n",
      "/opt/conda/envs/cookbook-research/lib/python3.10/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
      "  _torch_pytree._register_pytree_node(\n",
      "/opt/conda/envs/cookbook-research/lib/python3.10/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
      "  _torch_pytree._register_pytree_node(\n"
     ]
    }
   ],
   "source": [
    "p = MultiFlowsPipeline(PipelineConfig(\n",
    "    extract_config=extract_config,\n",
    "    transform_config=transform_config,\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we call the `run` method on the `MultiFlowsPipeline` object to execute the question-answer generation operation on the data shown above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1/1 [00:27<00:00, 27.01s/it]\n",
      "100%|██████████| 4/4 [02:21<00:00, 35.49s/it]\n"
     ]
    }
   ],
   "source": [
    "output = p.run(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Output\n",
    "Let's take a look of the generated output of Abstract segmentation.\n",
    "\n",
    "#### Process the output\n",
    "Let's take a look of the generated output. First, let's make sure there weren't any errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, o in enumerate(output):\n",
    "    for j, item in enumerate(o):\n",
    "        if 'error' in item:\n",
    "            print(\"Error at output[{}][{}]: {}\".format(i, j, item['error']))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's print out the raw output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'output': [{'error': 'No errors.',\n",
      "              'response': [{'context': '# An Observational Study of the Effect '\n",
      "                                       'of Nike Vaporly Shoes on Marathon '\n",
      "                                       'Performance',\n",
      "                            'qas': [{'Answer': 'The effect of Nike Vaporly '\n",
      "                                               'Shoes on Marathon Performance',\n",
      "                                     'Question': 'What is the focus of the '\n",
      "                                                 'study?'},\n",
      "                                    {'Answer': 'An Observational Study',\n",
      "                                     'Question': 'What type of study is being '\n",
      "                                                 'conducted?'},\n",
      "                                    {'Answer': 'The performance of marathon '\n",
      "                                               'runners wearing Nike Vaporly '\n",
      "                                               'Shoes',\n",
      "                                     'Question': 'What is being observed in '\n",
      "                                                 'the study?'}]},\n",
      "                           {'context': 'We are not experts in causal inference '\n",
      "                                       'techniques, so we appreciate your '\n",
      "                                       'expert opinion on the matter. We list '\n",
      "                                       'a few possible confounders in the '\n",
      "                                       'discussion, namely fitness and price. '\n",
      "                                       'Since Mike Vaporflys are expensive '\n",
      "                                       'relative to other racing shoes, a '\n",
      "                                       'runner who is in poor fitness may be '\n",
      "                                       'unwilling to pay for them to wear in a '\n",
      "                                       'marathon, but once a runner gets into '\n",
      "                                       'good shape, they might be more willing '\n",
      "                                       'to pay for the shoes and race in them. '\n",
      "                                       'Also, some runners might try the shoes '\n",
      "                                       'in training and decide that they do '\n",
      "                                       'not want to race in them because they '\n",
      "                                       'do not get a benefit from them. A '\n",
      "                                       \"runner's personal objections to the \"\n",
      "                                       'fairness of the shoes might also '\n",
      "                                       'influence self-assignment of the '\n",
      "                                       'shoes.',\n",
      "                                       ...             \n",
      "                            'qas': [{'Answer': 'Fitness and price',\n",
      "                                     'Question': 'What are the possible '\n",
      "                                                 'confounders mentioned in the '\n",
      "                                                 'discussion?'},\n",
      "                                    {'Answer': 'Because they are expensive '\n",
      "                                               'relative to other racing shoes',\n",
      "                                     'Question': 'Why might a runner in poor '\n",
      "                                                 'fitness be unwilling to pay '\n",
      "                                                 'for Mike Vaporflys?'},\n",
      "                                    {'Answer': 'Not getting a benefit from '\n",
      "                                               'them or personal objections to '\n",
      "                                               'the fairness of the shoes',\n",
      "                                     'Question': 'What might influence a '\n",
      "                                                 \"runner's decision to not \"\n",
      "                                                 'race in Mike Vaporflys '\n",
      "                                                 'despite trying them in '\n",
      "                                                 'training?'}]}]}],\n",
      "  'root': <uniflow.node.Node object at 0x7f373a914550>}]\n"
     ]
    }
   ],
   "source": [
    "\n",
    "pprint(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need to do a little postprocessing on the raw output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_03ec2 th {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_03ec2_row0_col0, #T_03ec2_row0_col1, #T_03ec2_row0_col2, #T_03ec2_row1_col0, #T_03ec2_row1_col1, #T_03ec2_row1_col2, #T_03ec2_row2_col0, #T_03ec2_row2_col1, #T_03ec2_row2_col2, #T_03ec2_row3_col0, #T_03ec2_row3_col1, #T_03ec2_row3_col2, #T_03ec2_row4_col0, #T_03ec2_row4_col1, #T_03ec2_row4_col2, #T_03ec2_row5_col0, #T_03ec2_row5_col1, #T_03ec2_row5_col2, #T_03ec2_row6_col0, #T_03ec2_row6_col1, #T_03ec2_row6_col2, #T_03ec2_row7_col0, #T_03ec2_row7_col1, #T_03ec2_row7_col2, #T_03ec2_row8_col0, #T_03ec2_row8_col1, #T_03ec2_row8_col2, #T_03ec2_row9_col0, #T_03ec2_row9_col1, #T_03ec2_row9_col2, #T_03ec2_row10_col0, #T_03ec2_row10_col1, #T_03ec2_row10_col2, #T_03ec2_row11_col0, #T_03ec2_row11_col1, #T_03ec2_row11_col2, #T_03ec2_row12_col0, #T_03ec2_row12_col1, #T_03ec2_row12_col2, #T_03ec2_row13_col0, #T_03ec2_row13_col1, #T_03ec2_row13_col2, #T_03ec2_row14_col0, #T_03ec2_row14_col1, #T_03ec2_row14_col2, #T_03ec2_row15_col0, #T_03ec2_row15_col1, #T_03ec2_row15_col2, #T_03ec2_row16_col0, #T_03ec2_row16_col1, #T_03ec2_row16_col2, #T_03ec2_row17_col0, #T_03ec2_row17_col1, #T_03ec2_row17_col2, #T_03ec2_row18_col0, #T_03ec2_row18_col1, #T_03ec2_row18_col2, #T_03ec2_row19_col0, #T_03ec2_row19_col1, #T_03ec2_row19_col2, #T_03ec2_row20_col0, #T_03ec2_row20_col1, #T_03ec2_row20_col2, #T_03ec2_row21_col0, #T_03ec2_row21_col1, #T_03ec2_row21_col2, #T_03ec2_row22_col0, #T_03ec2_row22_col1, #T_03ec2_row22_col2, #T_03ec2_row23_col0, #T_03ec2_row23_col1, #T_03ec2_row23_col2, #T_03ec2_row24_col0, #T_03ec2_row24_col1, #T_03ec2_row24_col2, #T_03ec2_row25_col0, #T_03ec2_row25_col1, #T_03ec2_row25_col2, #T_03ec2_row26_col0, #T_03ec2_row26_col1, #T_03ec2_row26_col2, #T_03ec2_row27_col0, #T_03ec2_row27_col1, #T_03ec2_row27_col2, #T_03ec2_row28_col0, #T_03ec2_row28_col1, #T_03ec2_row28_col2, #T_03ec2_row29_col0, #T_03ec2_row29_col1, #T_03ec2_row29_col2, #T_03ec2_row30_col0, #T_03ec2_row30_col1, #T_03ec2_row30_col2, #T_03ec2_row31_col0, #T_03ec2_row31_col1, #T_03ec2_row31_col2, #T_03ec2_row32_col0, #T_03ec2_row32_col1, #T_03ec2_row32_col2, #T_03ec2_row33_col0, #T_03ec2_row33_col1, #T_03ec2_row33_col2, #T_03ec2_row34_col0, #T_03ec2_row34_col1, #T_03ec2_row34_col2, #T_03ec2_row35_col0, #T_03ec2_row35_col1, #T_03ec2_row35_col2, #T_03ec2_row36_col0, #T_03ec2_row36_col1, #T_03ec2_row36_col2, #T_03ec2_row37_col0, #T_03ec2_row37_col1, #T_03ec2_row37_col2, #T_03ec2_row38_col0, #T_03ec2_row38_col1, #T_03ec2_row38_col2, #T_03ec2_row39_col0, #T_03ec2_row39_col1, #T_03ec2_row39_col2, #T_03ec2_row40_col0, #T_03ec2_row40_col1, #T_03ec2_row40_col2, #T_03ec2_row41_col0, #T_03ec2_row41_col1, #T_03ec2_row41_col2, #T_03ec2_row42_col0, #T_03ec2_row42_col1, #T_03ec2_row42_col2, #T_03ec2_row43_col0, #T_03ec2_row43_col1, #T_03ec2_row43_col2, #T_03ec2_row44_col0, #T_03ec2_row44_col1, #T_03ec2_row44_col2, #T_03ec2_row45_col0, #T_03ec2_row45_col1, #T_03ec2_row45_col2, #T_03ec2_row46_col0, #T_03ec2_row46_col1, #T_03ec2_row46_col2, #T_03ec2_row47_col0, #T_03ec2_row47_col1, #T_03ec2_row47_col2, #T_03ec2_row48_col0, #T_03ec2_row48_col1, #T_03ec2_row48_col2, #T_03ec2_row49_col0, #T_03ec2_row49_col1, #T_03ec2_row49_col2, #T_03ec2_row50_col0, #T_03ec2_row50_col1, #T_03ec2_row50_col2, #T_03ec2_row51_col0, #T_03ec2_row51_col1, #T_03ec2_row51_col2, #T_03ec2_row52_col0, #T_03ec2_row52_col1, #T_03ec2_row52_col2, #T_03ec2_row53_col0, #T_03ec2_row53_col1, #T_03ec2_row53_col2, #T_03ec2_row54_col0, #T_03ec2_row54_col1, #T_03ec2_row54_col2, #T_03ec2_row55_col0, #T_03ec2_row55_col1, #T_03ec2_row55_col2, #T_03ec2_row56_col0, #T_03ec2_row56_col1, #T_03ec2_row56_col2, #T_03ec2_row57_col0, #T_03ec2_row57_col1, #T_03ec2_row57_col2, #T_03ec2_row58_col0, #T_03ec2_row58_col1, #T_03ec2_row58_col2, #T_03ec2_row59_col0, #T_03ec2_row59_col1, #T_03ec2_row59_col2, #T_03ec2_row60_col0, #T_03ec2_row60_col1, #T_03ec2_row60_col2, #T_03ec2_row61_col0, #T_03ec2_row61_col1, #T_03ec2_row61_col2, #T_03ec2_row62_col0, #T_03ec2_row62_col1, #T_03ec2_row62_col2, #T_03ec2_row63_col0, #T_03ec2_row63_col1, #T_03ec2_row63_col2, #T_03ec2_row64_col0, #T_03ec2_row64_col1, #T_03ec2_row64_col2, #T_03ec2_row65_col0, #T_03ec2_row65_col1, #T_03ec2_row65_col2, #T_03ec2_row66_col0, #T_03ec2_row66_col1, #T_03ec2_row66_col2, #T_03ec2_row67_col0, #T_03ec2_row67_col1, #T_03ec2_row67_col2, #T_03ec2_row68_col0, #T_03ec2_row68_col1, #T_03ec2_row68_col2, #T_03ec2_row69_col0, #T_03ec2_row69_col1, #T_03ec2_row69_col2, #T_03ec2_row70_col0, #T_03ec2_row70_col1, #T_03ec2_row70_col2, #T_03ec2_row71_col0, #T_03ec2_row71_col1, #T_03ec2_row71_col2, #T_03ec2_row72_col0, #T_03ec2_row72_col1, #T_03ec2_row72_col2, #T_03ec2_row73_col0, #T_03ec2_row73_col1, #T_03ec2_row73_col2, #T_03ec2_row74_col0, #T_03ec2_row74_col1, #T_03ec2_row74_col2, #T_03ec2_row75_col0, #T_03ec2_row75_col1, #T_03ec2_row75_col2, #T_03ec2_row76_col0, #T_03ec2_row76_col1, #T_03ec2_row76_col2, #T_03ec2_row77_col0, #T_03ec2_row77_col1, #T_03ec2_row77_col2, #T_03ec2_row78_col0, #T_03ec2_row78_col1, #T_03ec2_row78_col2, #T_03ec2_row79_col0, #T_03ec2_row79_col1, #T_03ec2_row79_col2, #T_03ec2_row80_col0, #T_03ec2_row80_col1, #T_03ec2_row80_col2, #T_03ec2_row81_col0, #T_03ec2_row81_col1, #T_03ec2_row81_col2, #T_03ec2_row82_col0, #T_03ec2_row82_col1, #T_03ec2_row82_col2, #T_03ec2_row83_col0, #T_03ec2_row83_col1, #T_03ec2_row83_col2, #T_03ec2_row84_col0, #T_03ec2_row84_col1, #T_03ec2_row84_col2, #T_03ec2_row85_col0, #T_03ec2_row85_col1, #T_03ec2_row85_col2, #T_03ec2_row86_col0, #T_03ec2_row86_col1, #T_03ec2_row86_col2, #T_03ec2_row87_col0, #T_03ec2_row87_col1, #T_03ec2_row87_col2, #T_03ec2_row88_col0, #T_03ec2_row88_col1, #T_03ec2_row88_col2, #T_03ec2_row89_col0, #T_03ec2_row89_col1, #T_03ec2_row89_col2, #T_03ec2_row90_col0, #T_03ec2_row90_col1, #T_03ec2_row90_col2, #T_03ec2_row91_col0, #T_03ec2_row91_col1, #T_03ec2_row91_col2, #T_03ec2_row92_col0, #T_03ec2_row92_col1, #T_03ec2_row92_col2, #T_03ec2_row93_col0, #T_03ec2_row93_col1, #T_03ec2_row93_col2, #T_03ec2_row94_col0, #T_03ec2_row94_col1, #T_03ec2_row94_col2, #T_03ec2_row95_col0, #T_03ec2_row95_col1, #T_03ec2_row95_col2, #T_03ec2_row96_col0, #T_03ec2_row96_col1, #T_03ec2_row96_col2, #T_03ec2_row97_col0, #T_03ec2_row97_col1, #T_03ec2_row97_col2, #T_03ec2_row98_col0, #T_03ec2_row98_col1, #T_03ec2_row98_col2, #T_03ec2_row99_col0, #T_03ec2_row99_col1, #T_03ec2_row99_col2, #T_03ec2_row100_col0, #T_03ec2_row100_col1, #T_03ec2_row100_col2, #T_03ec2_row101_col0, #T_03ec2_row101_col1, #T_03ec2_row101_col2, #T_03ec2_row102_col0, #T_03ec2_row102_col1, #T_03ec2_row102_col2, #T_03ec2_row103_col0, #T_03ec2_row103_col1, #T_03ec2_row103_col2, #T_03ec2_row104_col0, #T_03ec2_row104_col1, #T_03ec2_row104_col2, #T_03ec2_row105_col0, #T_03ec2_row105_col1, #T_03ec2_row105_col2, #T_03ec2_row106_col0, #T_03ec2_row106_col1, #T_03ec2_row106_col2, #T_03ec2_row107_col0, #T_03ec2_row107_col1, #T_03ec2_row107_col2, #T_03ec2_row108_col0, #T_03ec2_row108_col1, #T_03ec2_row108_col2, #T_03ec2_row109_col0, #T_03ec2_row109_col1, #T_03ec2_row109_col2, #T_03ec2_row110_col0, #T_03ec2_row110_col1, #T_03ec2_row110_col2, #T_03ec2_row111_col0, #T_03ec2_row111_col1, #T_03ec2_row111_col2, #T_03ec2_row112_col0, #T_03ec2_row112_col1, #T_03ec2_row112_col2, #T_03ec2_row113_col0, #T_03ec2_row113_col1, #T_03ec2_row113_col2, #T_03ec2_row114_col0, #T_03ec2_row114_col1, #T_03ec2_row114_col2, #T_03ec2_row115_col0, #T_03ec2_row115_col1, #T_03ec2_row115_col2, #T_03ec2_row116_col0, #T_03ec2_row116_col1, #T_03ec2_row116_col2, #T_03ec2_row117_col0, #T_03ec2_row117_col1, #T_03ec2_row117_col2, #T_03ec2_row118_col0, #T_03ec2_row118_col1, #T_03ec2_row118_col2, #T_03ec2_row119_col0, #T_03ec2_row119_col1, #T_03ec2_row119_col2, #T_03ec2_row120_col0, #T_03ec2_row120_col1, #T_03ec2_row120_col2, #T_03ec2_row121_col0, #T_03ec2_row121_col1, #T_03ec2_row121_col2, #T_03ec2_row122_col0, #T_03ec2_row122_col1, #T_03ec2_row122_col2, #T_03ec2_row123_col0, #T_03ec2_row123_col1, #T_03ec2_row123_col2, #T_03ec2_row124_col0, #T_03ec2_row124_col1, #T_03ec2_row124_col2, #T_03ec2_row125_col0, #T_03ec2_row125_col1, #T_03ec2_row125_col2, #T_03ec2_row126_col0, #T_03ec2_row126_col1, #T_03ec2_row126_col2, #T_03ec2_row127_col0, #T_03ec2_row127_col1, #T_03ec2_row127_col2, #T_03ec2_row128_col0, #T_03ec2_row128_col1, #T_03ec2_row128_col2, #T_03ec2_row129_col0, #T_03ec2_row129_col1, #T_03ec2_row129_col2, #T_03ec2_row130_col0, #T_03ec2_row130_col1, #T_03ec2_row130_col2, #T_03ec2_row131_col0, #T_03ec2_row131_col1, #T_03ec2_row131_col2, #T_03ec2_row132_col0, #T_03ec2_row132_col1, #T_03ec2_row132_col2, #T_03ec2_row133_col0, #T_03ec2_row133_col1, #T_03ec2_row133_col2, #T_03ec2_row134_col0, #T_03ec2_row134_col1, #T_03ec2_row134_col2, #T_03ec2_row135_col0, #T_03ec2_row135_col1, #T_03ec2_row135_col2, #T_03ec2_row136_col0, #T_03ec2_row136_col1, #T_03ec2_row136_col2, #T_03ec2_row137_col0, #T_03ec2_row137_col1, #T_03ec2_row137_col2, #T_03ec2_row138_col0, #T_03ec2_row138_col1, #T_03ec2_row138_col2, #T_03ec2_row139_col0, #T_03ec2_row139_col1, #T_03ec2_row139_col2, #T_03ec2_row140_col0, #T_03ec2_row140_col1, #T_03ec2_row140_col2, #T_03ec2_row141_col0, #T_03ec2_row141_col1, #T_03ec2_row141_col2, #T_03ec2_row142_col0, #T_03ec2_row142_col1, #T_03ec2_row142_col2, #T_03ec2_row143_col0, #T_03ec2_row143_col1, #T_03ec2_row143_col2, #T_03ec2_row144_col0, #T_03ec2_row144_col1, #T_03ec2_row144_col2, #T_03ec2_row145_col0, #T_03ec2_row145_col1, #T_03ec2_row145_col2, #T_03ec2_row146_col0, #T_03ec2_row146_col1, #T_03ec2_row146_col2, #T_03ec2_row147_col0, #T_03ec2_row147_col1, #T_03ec2_row147_col2, #T_03ec2_row148_col0, #T_03ec2_row148_col1, #T_03ec2_row148_col2, #T_03ec2_row149_col0, #T_03ec2_row149_col1, #T_03ec2_row149_col2, #T_03ec2_row150_col0, #T_03ec2_row150_col1, #T_03ec2_row150_col2, #T_03ec2_row151_col0, #T_03ec2_row151_col1, #T_03ec2_row151_col2, #T_03ec2_row152_col0, #T_03ec2_row152_col1, #T_03ec2_row152_col2, #T_03ec2_row153_col0, #T_03ec2_row153_col1, #T_03ec2_row153_col2, #T_03ec2_row154_col0, #T_03ec2_row154_col1, #T_03ec2_row154_col2, #T_03ec2_row155_col0, #T_03ec2_row155_col1, #T_03ec2_row155_col2, #T_03ec2_row156_col0, #T_03ec2_row156_col1, #T_03ec2_row156_col2, #T_03ec2_row157_col0, #T_03ec2_row157_col1, #T_03ec2_row157_col2, #T_03ec2_row158_col0, #T_03ec2_row158_col1, #T_03ec2_row158_col2, #T_03ec2_row159_col0, #T_03ec2_row159_col1, #T_03ec2_row159_col2, #T_03ec2_row160_col0, #T_03ec2_row160_col1, #T_03ec2_row160_col2, #T_03ec2_row161_col0, #T_03ec2_row161_col1, #T_03ec2_row161_col2, #T_03ec2_row162_col0, #T_03ec2_row162_col1, #T_03ec2_row162_col2, #T_03ec2_row163_col0, #T_03ec2_row163_col1, #T_03ec2_row163_col2, #T_03ec2_row164_col0, #T_03ec2_row164_col1, #T_03ec2_row164_col2, #T_03ec2_row165_col0, #T_03ec2_row165_col1, #T_03ec2_row165_col2, #T_03ec2_row166_col0, #T_03ec2_row166_col1, #T_03ec2_row166_col2, #T_03ec2_row167_col0, #T_03ec2_row167_col1, #T_03ec2_row167_col2, #T_03ec2_row168_col0, #T_03ec2_row168_col1, #T_03ec2_row168_col2, #T_03ec2_row169_col0, #T_03ec2_row169_col1, #T_03ec2_row169_col2, #T_03ec2_row170_col0, #T_03ec2_row170_col1, #T_03ec2_row170_col2, #T_03ec2_row171_col0, #T_03ec2_row171_col1, #T_03ec2_row171_col2, #T_03ec2_row172_col0, #T_03ec2_row172_col1, #T_03ec2_row172_col2, #T_03ec2_row173_col0, #T_03ec2_row173_col1, #T_03ec2_row173_col2, #T_03ec2_row174_col0, #T_03ec2_row174_col1, #T_03ec2_row174_col2, #T_03ec2_row175_col0, #T_03ec2_row175_col1, #T_03ec2_row175_col2, #T_03ec2_row176_col0, #T_03ec2_row176_col1, #T_03ec2_row176_col2, #T_03ec2_row177_col0, #T_03ec2_row177_col1, #T_03ec2_row177_col2, #T_03ec2_row178_col0, #T_03ec2_row178_col1, #T_03ec2_row178_col2, #T_03ec2_row179_col0, #T_03ec2_row179_col1, #T_03ec2_row179_col2, #T_03ec2_row180_col0, #T_03ec2_row180_col1, #T_03ec2_row180_col2, #T_03ec2_row181_col0, #T_03ec2_row181_col1, #T_03ec2_row181_col2, #T_03ec2_row182_col0, #T_03ec2_row182_col1, #T_03ec2_row182_col2, #T_03ec2_row183_col0, #T_03ec2_row183_col1, #T_03ec2_row183_col2, #T_03ec2_row184_col0, #T_03ec2_row184_col1, #T_03ec2_row184_col2, #T_03ec2_row185_col0, #T_03ec2_row185_col1, #T_03ec2_row185_col2, #T_03ec2_row186_col0, #T_03ec2_row186_col1, #T_03ec2_row186_col2, #T_03ec2_row187_col0, #T_03ec2_row187_col1, #T_03ec2_row187_col2, #T_03ec2_row188_col0, #T_03ec2_row188_col1, #T_03ec2_row188_col2, #T_03ec2_row189_col0, #T_03ec2_row189_col1, #T_03ec2_row189_col2, #T_03ec2_row190_col0, #T_03ec2_row190_col1, #T_03ec2_row190_col2, #T_03ec2_row191_col0, #T_03ec2_row191_col1, #T_03ec2_row191_col2, #T_03ec2_row192_col0, #T_03ec2_row192_col1, #T_03ec2_row192_col2, #T_03ec2_row193_col0, #T_03ec2_row193_col1, #T_03ec2_row193_col2, #T_03ec2_row194_col0, #T_03ec2_row194_col1, #T_03ec2_row194_col2, #T_03ec2_row195_col0, #T_03ec2_row195_col1, #T_03ec2_row195_col2, #T_03ec2_row196_col0, #T_03ec2_row196_col1, #T_03ec2_row196_col2, #T_03ec2_row197_col0, #T_03ec2_row197_col1, #T_03ec2_row197_col2, #T_03ec2_row198_col0, #T_03ec2_row198_col1, #T_03ec2_row198_col2, #T_03ec2_row199_col0, #T_03ec2_row199_col1, #T_03ec2_row199_col2, #T_03ec2_row200_col0, #T_03ec2_row200_col1, #T_03ec2_row200_col2, #T_03ec2_row201_col0, #T_03ec2_row201_col1, #T_03ec2_row201_col2, #T_03ec2_row202_col0, #T_03ec2_row202_col1, #T_03ec2_row202_col2, #T_03ec2_row203_col0, #T_03ec2_row203_col1, #T_03ec2_row203_col2, #T_03ec2_row204_col0, #T_03ec2_row204_col1, #T_03ec2_row204_col2, #T_03ec2_row205_col0, #T_03ec2_row205_col1, #T_03ec2_row205_col2, #T_03ec2_row206_col0, #T_03ec2_row206_col1, #T_03ec2_row206_col2, #T_03ec2_row207_col0, #T_03ec2_row207_col1, #T_03ec2_row207_col2, #T_03ec2_row208_col0, #T_03ec2_row208_col1, #T_03ec2_row208_col2, #T_03ec2_row209_col0, #T_03ec2_row209_col1, #T_03ec2_row209_col2, #T_03ec2_row210_col0, #T_03ec2_row210_col1, #T_03ec2_row210_col2, #T_03ec2_row211_col0, #T_03ec2_row211_col1, #T_03ec2_row211_col2, #T_03ec2_row212_col0, #T_03ec2_row212_col1, #T_03ec2_row212_col2, #T_03ec2_row213_col0, #T_03ec2_row213_col1, #T_03ec2_row213_col2, #T_03ec2_row214_col0, #T_03ec2_row214_col1, #T_03ec2_row214_col2, #T_03ec2_row215_col0, #T_03ec2_row215_col1, #T_03ec2_row215_col2, #T_03ec2_row216_col0, #T_03ec2_row216_col1, #T_03ec2_row216_col2, #T_03ec2_row217_col0, #T_03ec2_row217_col1, #T_03ec2_row217_col2, #T_03ec2_row218_col0, #T_03ec2_row218_col1, #T_03ec2_row218_col2, #T_03ec2_row219_col0, #T_03ec2_row219_col1, #T_03ec2_row219_col2, #T_03ec2_row220_col0, #T_03ec2_row220_col1, #T_03ec2_row220_col2, #T_03ec2_row221_col0, #T_03ec2_row221_col1, #T_03ec2_row221_col2, #T_03ec2_row222_col0, #T_03ec2_row222_col1, #T_03ec2_row222_col2, #T_03ec2_row223_col0, #T_03ec2_row223_col1, #T_03ec2_row223_col2, #T_03ec2_row224_col0, #T_03ec2_row224_col1, #T_03ec2_row224_col2, #T_03ec2_row225_col0, #T_03ec2_row225_col1, #T_03ec2_row225_col2, #T_03ec2_row226_col0, #T_03ec2_row226_col1, #T_03ec2_row226_col2, #T_03ec2_row227_col0, #T_03ec2_row227_col1, #T_03ec2_row227_col2, #T_03ec2_row228_col0, #T_03ec2_row228_col1, #T_03ec2_row228_col2, #T_03ec2_row229_col0, #T_03ec2_row229_col1, #T_03ec2_row229_col2, #T_03ec2_row230_col0, #T_03ec2_row230_col1, #T_03ec2_row230_col2, #T_03ec2_row231_col0, #T_03ec2_row231_col1, #T_03ec2_row231_col2, #T_03ec2_row232_col0, #T_03ec2_row232_col1, #T_03ec2_row232_col2, #T_03ec2_row233_col0, #T_03ec2_row233_col1, #T_03ec2_row233_col2, #T_03ec2_row234_col0, #T_03ec2_row234_col1, #T_03ec2_row234_col2, #T_03ec2_row235_col0, #T_03ec2_row235_col1, #T_03ec2_row235_col2, #T_03ec2_row236_col0, #T_03ec2_row236_col1, #T_03ec2_row236_col2, #T_03ec2_row237_col0, #T_03ec2_row237_col1, #T_03ec2_row237_col2, #T_03ec2_row238_col0, #T_03ec2_row238_col1, #T_03ec2_row238_col2, #T_03ec2_row239_col0, #T_03ec2_row239_col1, #T_03ec2_row239_col2, #T_03ec2_row240_col0, #T_03ec2_row240_col1, #T_03ec2_row240_col2, #T_03ec2_row241_col0, #T_03ec2_row241_col1, #T_03ec2_row241_col2, #T_03ec2_row242_col0, #T_03ec2_row242_col1, #T_03ec2_row242_col2, #T_03ec2_row243_col0, #T_03ec2_row243_col1, #T_03ec2_row243_col2, #T_03ec2_row244_col0, #T_03ec2_row244_col1, #T_03ec2_row244_col2, #T_03ec2_row245_col0, #T_03ec2_row245_col1, #T_03ec2_row245_col2, #T_03ec2_row246_col0, #T_03ec2_row246_col1, #T_03ec2_row246_col2, #T_03ec2_row247_col0, #T_03ec2_row247_col1, #T_03ec2_row247_col2, #T_03ec2_row248_col0, #T_03ec2_row248_col1, #T_03ec2_row248_col2, #T_03ec2_row249_col0, #T_03ec2_row249_col1, #T_03ec2_row249_col2, #T_03ec2_row250_col0, #T_03ec2_row250_col1, #T_03ec2_row250_col2, #T_03ec2_row251_col0, #T_03ec2_row251_col1, #T_03ec2_row251_col2, #T_03ec2_row252_col0, #T_03ec2_row252_col1, #T_03ec2_row252_col2, #T_03ec2_row253_col0, #T_03ec2_row253_col1, #T_03ec2_row253_col2, #T_03ec2_row254_col0, #T_03ec2_row254_col1, #T_03ec2_row254_col2, #T_03ec2_row255_col0, #T_03ec2_row255_col1, #T_03ec2_row255_col2, #T_03ec2_row256_col0, #T_03ec2_row256_col1, #T_03ec2_row256_col2, #T_03ec2_row257_col0, #T_03ec2_row257_col1, #T_03ec2_row257_col2, #T_03ec2_row258_col0, #T_03ec2_row258_col1, #T_03ec2_row258_col2, #T_03ec2_row259_col0, #T_03ec2_row259_col1, #T_03ec2_row259_col2, #T_03ec2_row260_col0, #T_03ec2_row260_col1, #T_03ec2_row260_col2, #T_03ec2_row261_col0, #T_03ec2_row261_col1, #T_03ec2_row261_col2, #T_03ec2_row262_col0, #T_03ec2_row262_col1, #T_03ec2_row262_col2, #T_03ec2_row263_col0, #T_03ec2_row263_col1, #T_03ec2_row263_col2, #T_03ec2_row264_col0, #T_03ec2_row264_col1, #T_03ec2_row264_col2, #T_03ec2_row265_col0, #T_03ec2_row265_col1, #T_03ec2_row265_col2, #T_03ec2_row266_col0, #T_03ec2_row266_col1, #T_03ec2_row266_col2, #T_03ec2_row267_col0, #T_03ec2_row267_col1, #T_03ec2_row267_col2, #T_03ec2_row268_col0, #T_03ec2_row268_col1, #T_03ec2_row268_col2, #T_03ec2_row269_col0, #T_03ec2_row269_col1, #T_03ec2_row269_col2, #T_03ec2_row270_col0, #T_03ec2_row270_col1, #T_03ec2_row270_col2, #T_03ec2_row271_col0, #T_03ec2_row271_col1, #T_03ec2_row271_col2, #T_03ec2_row272_col0, #T_03ec2_row272_col1, #T_03ec2_row272_col2, #T_03ec2_row273_col0, #T_03ec2_row273_col1, #T_03ec2_row273_col2, #T_03ec2_row274_col0, #T_03ec2_row274_col1, #T_03ec2_row274_col2, #T_03ec2_row275_col0, #T_03ec2_row275_col1, #T_03ec2_row275_col2, #T_03ec2_row276_col0, #T_03ec2_row276_col1, #T_03ec2_row276_col2, #T_03ec2_row277_col0, #T_03ec2_row277_col1, #T_03ec2_row277_col2, #T_03ec2_row278_col0, #T_03ec2_row278_col1, #T_03ec2_row278_col2 {\n",
       "  text-align: left;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_03ec2\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_03ec2_level0_col0\" class=\"col_heading level0 col0\" >Context</th>\n",
       "      <th id=\"T_03ec2_level0_col1\" class=\"col_heading level0 col1\" >Question</th>\n",
       "      <th id=\"T_03ec2_level0_col2\" class=\"col_heading level0 col2\" >Answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_03ec2_row0_col0\" class=\"data row0 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row0_col1\" class=\"data row0 col1\" >What is Mount Everest known for?</td>\n",
       "      <td id=\"T_03ec2_row0_col2\" class=\"data row0 col2\" >Mount Everest is known for being Earth's highest mountain above sea level.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_03ec2_row1_col0\" class=\"data row1 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row1_col1\" class=\"data row1 col1\" >Where is Mount Everest located?</td>\n",
       "      <td id=\"T_03ec2_row1_col2\" class=\"data row1 col2\" >Mount Everest is located in the Mahalangur Himal sub-range of the Himalayas.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_03ec2_row2_col0\" class=\"data row2 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row2_col1\" class=\"data row2 col1\" >Which countries does the international border on Mount Everest's summit point separate?</td>\n",
       "      <td id=\"T_03ec2_row2_col2\" class=\"data row2 col2\" >The international border on Mount Everest's summit point separates China (Tibet Autonomous Region) and Nepal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_03ec2_row3_col0\" class=\"data row3 col0\" >We collected marathon performance data from a systematic sample of elite and sub-elite athletes over the period 2015 to 2019, then searched the internet for publicly-available photographs of these performances, identifying whether the Nike Vaporly shoes were worn or not in each performance. Controlling for athlete ability and race difficulty, we estimated the effect on marathon times of wearing the Vaporfly shoes. Assuming that the effect of Vaporfly shoes is additive, we estimate that the Vaporfly shoes improve men's times between 2.0 and 3.9 minutes, while they improve women's times between 0.8 and 3.5 minutes. Assuming that the effect of Vaporfly shoes is multiplicative, we estimate that they improve men's times between 1.4 and 2.8 percent and women's performances between 0.6 and 2.2 percent. The improvements are in comparison to the shoe the athlete was wearing before switching to Vaporfly shoes, and represents an expected improvement rather than a guaranteed improvement.</td>\n",
       "      <td id=\"T_03ec2_row3_col1\" class=\"data row3 col1\" >What was the method used to collect data on marathon performances?</td>\n",
       "      <td id=\"T_03ec2_row3_col2\" class=\"data row3 col2\" >A systematic sample of elite and sub-elite athletes' performances from 2015 to 2019 was collected, and publicly-available photographs of these performances were searched to identify whether Nike Vaporfly shoes were worn.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_03ec2_row4_col0\" class=\"data row4 col0\" >We collected marathon performance data from a systematic sample of elite and sub-elite athletes over the period 2015 to 2019, then searched the internet for publicly-available photographs of these performances, identifying whether the Nike Vaporly shoes were worn or not in each performance. Controlling for athlete ability and race difficulty, we estimated the effect on marathon times of wearing the Vaporfly shoes. Assuming that the effect of Vaporfly shoes is additive, we estimate that the Vaporfly shoes improve men's times between 2.0 and 3.9 minutes, while they improve women's times between 0.8 and 3.5 minutes. Assuming that the effect of Vaporfly shoes is multiplicative, we estimate that they improve men's times between 1.4 and 2.8 percent and women's performances between 0.6 and 2.2 percent. The improvements are in comparison to the shoe the athlete was wearing before switching to Vaporfly shoes, and represents an expected improvement rather than a guaranteed improvement.</td>\n",
       "      <td id=\"T_03ec2_row4_col1\" class=\"data row4 col1\" >What is the estimated improvement in marathon times for men wearing Vaporfly shoes?</td>\n",
       "      <td id=\"T_03ec2_row4_col2\" class=\"data row4 col2\" >The estimated improvement in marathon times for men wearing Vaporfly shoes is between 2.0 and 3.9 minutes if the effect is additive, and between 1.4 and 2.8 percent if the effect is multiplicative.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
       "      <td id=\"T_03ec2_row5_col0\" class=\"data row5 col0\" >We collected marathon performance data from a systematic sample of elite and sub-elite athletes over the period 2015 to 2019, then searched the internet for publicly-available photographs of these performances, identifying whether the Nike Vaporly shoes were worn or not in each performance. Controlling for athlete ability and race difficulty, we estimated the effect on marathon times of wearing the Vaporfly shoes. Assuming that the effect of Vaporfly shoes is additive, we estimate that the Vaporfly shoes improve men's times between 2.0 and 3.9 minutes, while they improve women's times between 0.8 and 3.5 minutes. Assuming that the effect of Vaporfly shoes is multiplicative, we estimate that they improve men's times between 1.4 and 2.8 percent and women's performances between 0.6 and 2.2 percent. The improvements are in comparison to the shoe the athlete was wearing before switching to Vaporfly shoes, and represents an expected improvement rather than a guaranteed improvement.</td>\n",
       "      <td id=\"T_03ec2_row5_col1\" class=\"data row5 col1\" >How do Vaporfly shoes affect women's marathon performances?</td>\n",
       "      <td id=\"T_03ec2_row5_col2\" class=\"data row5 col2\" >Vaporfly shoes are estimated to improve women's marathon times between 0.8 and 3.5 minutes if the effect is additive, and between 0.6 and 2.2 percent if the effect is multiplicative.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
       "      <td id=\"T_03ec2_row6_col0\" class=\"data row6 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row6_col1\" class=\"data row6 col1\" >What is Mount Everest?</td>\n",
       "      <td id=\"T_03ec2_row6_col2\" class=\"data row6 col2\" >Earth's highest mountain above sea level</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
       "      <td id=\"T_03ec2_row7_col0\" class=\"data row7 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row7_col1\" class=\"data row7 col1\" >Where is Mount Everest located?</td>\n",
       "      <td id=\"T_03ec2_row7_col2\" class=\"data row7 col2\" >In the Mahalangur Himal sub-range of the Himalayas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
       "      <td id=\"T_03ec2_row8_col0\" class=\"data row8 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row8_col1\" class=\"data row8 col1\" >What runs across the summit point of Mount Everest?</td>\n",
       "      <td id=\"T_03ec2_row8_col2\" class=\"data row8 col2\" >The international border between China (Tibet Autonomous Region) and Nepal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
       "      <td id=\"T_03ec2_row9_col0\" class=\"data row9 col0\" >There is a growing consensus that Nike Corporation's new line of marathon racing shoes, which are commonly referred to as Vaporflys, provide a significant performance advantage to athletes who wear them. While several different versions of the shoes have appeared in races, including the Vaporfly 4%, the Vaporfly Next%, the Alphafly, and several prototype shoes, each iteration of the shoes has in common a carbon fiber plate stacked inside of a highly responsive foam sole.</td>\n",
       "      <td id=\"T_03ec2_row9_col1\" class=\"data row9 col1\" >What is the common feature of all the Nike marathon racing shoes mentioned?</td>\n",
       "      <td id=\"T_03ec2_row9_col2\" class=\"data row9 col2\" >A carbon fiber plate stacked inside of a highly responsive foam sole</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
       "      <td id=\"T_03ec2_row10_col0\" class=\"data row10 col0\" >There is a growing consensus that Nike Corporation's new line of marathon racing shoes, which are commonly referred to as Vaporflys, provide a significant performance advantage to athletes who wear them. While several different versions of the shoes have appeared in races, including the Vaporfly 4%, the Vaporfly Next%, the Alphafly, and several prototype shoes, each iteration of the shoes has in common a carbon fiber plate stacked inside of a highly responsive foam sole.</td>\n",
       "      <td id=\"T_03ec2_row10_col1\" class=\"data row10 col1\" >What is the general opinion about the performance impact of Vaporflys?</td>\n",
       "      <td id=\"T_03ec2_row10_col2\" class=\"data row10 col2\" >They provide a significant performance advantage to athletes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
       "      <td id=\"T_03ec2_row11_col0\" class=\"data row11 col0\" >There is a growing consensus that Nike Corporation's new line of marathon racing shoes, which are commonly referred to as Vaporflys, provide a significant performance advantage to athletes who wear them. While several different versions of the shoes have appeared in races, including the Vaporfly 4%, the Vaporfly Next%, the Alphafly, and several prototype shoes, each iteration of the shoes has in common a carbon fiber plate stacked inside of a highly responsive foam sole.</td>\n",
       "      <td id=\"T_03ec2_row11_col1\" class=\"data row11 col1\" >What are some of the versions of Nike's marathon racing shoes?</td>\n",
       "      <td id=\"T_03ec2_row11_col2\" class=\"data row11 col2\" >Vaporfly 4%, Vaporfly Next%, the Alphafly, and several prototype shoes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
       "      <td id=\"T_03ec2_row12_col0\" class=\"data row12 col0\" >Several research studies have investigated the magnitude of the Vaporfly performance benefit. Hoogkamer et al. (2018) and Barnes and Kilding (2019) tested highly trained distance runners in laboratory studies, measuring various biomechanical and physiological variables while subjects wore Vaporflys and several other shoes in trial runs on a treadmill. Although the measured benefits varied somewhat from athlete to athlete, both studies found a roughly 4% average reduction in energy expenditures while wearing Vaporflys, in comparison to other popular racing shoes such as the Adidas Adizero Adios Boost line of racing shoes, and Nike Zoom Matumbo track spikes.</td>\n",
       "      <td id=\"T_03ec2_row12_col1\" class=\"data row12 col1\" >What was the main focus of the research studies mentioned?</td>\n",
       "      <td id=\"T_03ec2_row12_col2\" class=\"data row12 col2\" >The magnitude of the Vaporfly performance benefit.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row13\" class=\"row_heading level0 row13\" >13</th>\n",
       "      <td id=\"T_03ec2_row13_col0\" class=\"data row13 col0\" >Several research studies have investigated the magnitude of the Vaporfly performance benefit. Hoogkamer et al. (2018) and Barnes and Kilding (2019) tested highly trained distance runners in laboratory studies, measuring various biomechanical and physiological variables while subjects wore Vaporflys and several other shoes in trial runs on a treadmill. Although the measured benefits varied somewhat from athlete to athlete, both studies found a roughly 4% average reduction in energy expenditures while wearing Vaporflys, in comparison to other popular racing shoes such as the Adidas Adizero Adios Boost line of racing shoes, and Nike Zoom Matumbo track spikes.</td>\n",
       "      <td id=\"T_03ec2_row13_col1\" class=\"data row13 col1\" >What was the average reduction in energy expenditure found when wearing Vaporflys?</td>\n",
       "      <td id=\"T_03ec2_row13_col2\" class=\"data row13 col2\" >Roughly 4%.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row14\" class=\"row_heading level0 row14\" >14</th>\n",
       "      <td id=\"T_03ec2_row14_col0\" class=\"data row14 col0\" >Several research studies have investigated the magnitude of the Vaporfly performance benefit. Hoogkamer et al. (2018) and Barnes and Kilding (2019) tested highly trained distance runners in laboratory studies, measuring various biomechanical and physiological variables while subjects wore Vaporflys and several other shoes in trial runs on a treadmill. Although the measured benefits varied somewhat from athlete to athlete, both studies found a roughly 4% average reduction in energy expenditures while wearing Vaporflys, in comparison to other popular racing shoes such as the Adidas Adizero Adios Boost line of racing shoes, and Nike Zoom Matumbo track spikes.</td>\n",
       "      <td id=\"T_03ec2_row14_col1\" class=\"data row14 col1\" >Which other popular racing shoes were compared to Vaporflys in the studies?</td>\n",
       "      <td id=\"T_03ec2_row14_col2\" class=\"data row14 col2\" >Adidas Adizero Adios Boost line of racing shoes, and Nike Zoom Matumbo track spikes.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row15\" class=\"row_heading level0 row15\" >15</th>\n",
       "      <td id=\"T_03ec2_row15_col0\" class=\"data row15 col0\" >The Upshot, a division of The New York Times, collected data from actual marathon performances recorded on Strava, a popular running log and GPS tracking website. Their study included hundreds of thousands of marathon performances, and dozens of different shoes. The Upshot found that the Vaporflys imparted a 4 to 5% advantage in finishing time over an</td>\n",
       "      <td id=\"T_03ec2_row15_col1\" class=\"data row15 col1\" >What was the source of the marathon performance data?</td>\n",
       "      <td id=\"T_03ec2_row15_col2\" class=\"data row15 col2\" >Strava, a popular running log and GPS tracking website</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row16\" class=\"row_heading level0 row16\" >16</th>\n",
       "      <td id=\"T_03ec2_row16_col0\" class=\"data row16 col0\" >The Upshot, a division of The New York Times, collected data from actual marathon performances recorded on Strava, a popular running log and GPS tracking website. Their study included hundreds of thousands of marathon performances, and dozens of different shoes. The Upshot found that the Vaporflys imparted a 4 to 5% advantage in finishing time over an</td>\n",
       "      <td id=\"T_03ec2_row16_col1\" class=\"data row16 col1\" >What did The Upshot study find about the Vaporflys?</td>\n",
       "      <td id=\"T_03ec2_row16_col2\" class=\"data row16 col2\" >They imparted a 4 to 5% advantage in finishing time</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row17\" class=\"row_heading level0 row17\" >17</th>\n",
       "      <td id=\"T_03ec2_row17_col0\" class=\"data row17 col0\" >The Upshot, a division of The New York Times, collected data from actual marathon performances recorded on Strava, a popular running log and GPS tracking website. Their study included hundreds of thousands of marathon performances, and dozens of different shoes. The Upshot found that the Vaporflys imparted a 4 to 5% advantage in finishing time over an</td>\n",
       "      <td id=\"T_03ec2_row17_col1\" class=\"data row17 col1\" >How many marathon performances were included in The Upshot's study?</td>\n",
       "      <td id=\"T_03ec2_row17_col2\" class=\"data row17 col2\" >Hundreds of thousands</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row18\" class=\"row_heading level0 row18\" >18</th>\n",
       "      <td id=\"T_03ec2_row18_col0\" class=\"data row18 col0\" >A study published by Wired Magazine found that a sample of runners in the 2017 New York City Marathon were more likely to run the second half of the race faster than the first if they were wearing Vaporflys (Thompson, 2017).</td>\n",
       "      <td id=\"T_03ec2_row18_col1\" class=\"data row18 col1\" >What did the study in Wired Magazine observe about runners in the 2017 New York City Marathon?</td>\n",
       "      <td id=\"T_03ec2_row18_col2\" class=\"data row18 col2\" >Runners were more likely to run the second half of the race faster than the first if they were wearing Vaporflys.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row19\" class=\"row_heading level0 row19\" >19</th>\n",
       "      <td id=\"T_03ec2_row19_col0\" class=\"data row19 col0\" >A study published by Wired Magazine found that a sample of runners in the 2017 New York City Marathon were more likely to run the second half of the race faster than the first if they were wearing Vaporflys (Thompson, 2017).</td>\n",
       "      <td id=\"T_03ec2_row19_col1\" class=\"data row19 col1\" >What is the name of the shoe associated with runners performing better in the second half of the race?</td>\n",
       "      <td id=\"T_03ec2_row19_col2\" class=\"data row19 col2\" >Vaporflys</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row20\" class=\"row_heading level0 row20\" >20</th>\n",
       "      <td id=\"T_03ec2_row20_col0\" class=\"data row20 col0\" >A study published by Wired Magazine found that a sample of runners in the 2017 New York City Marathon were more likely to run the second half of the race faster than the first if they were wearing Vaporflys (Thompson, 2017).</td>\n",
       "      <td id=\"T_03ec2_row20_col1\" class=\"data row20 col1\" >In which year's New York City Marathon was the observation made regarding runners' performance with Vaporflys?</td>\n",
       "      <td id=\"T_03ec2_row20_col2\" class=\"data row20 col2\" >2017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row21\" class=\"row_heading level0 row21\" >21</th>\n",
       "      <td id=\"T_03ec2_row21_col0\" class=\"data row21 col0\" >Our study is most similar to the Upshot study in that we analyze data from marathon performances and compare people's performances with and without the Vaporflys. However, our study differs in a few ways. First, instead of relying on a convenient sample of athletes who upload their data to Strava, we take an exhaustive sample of athletes who met a minimum performance standard at one of 22 of the largest marathon venues in 2015 and 2016 in the US and Canada. Second, instead of relying on self-reported shoe data, we searched the internet for photos of races and visually identified the shoes that runners wore. Third, we focus only on athletes who performed at an elite level before the Vaporflys were released to the public. Thus, we are only considering accomplished runners with marathon experience who, most likely, settled on a suitable shoe before the Vaporflys were released. These runners are also those most likely to be affected by shoe regulations because many of them compete in national Olympic qualifying races subject to regulations.</td>\n",
       "      <td id=\"T_03ec2_row21_col1\" class=\"data row21 col1\" >What aspect of marathon performance is being compared in the study?</td>\n",
       "      <td id=\"T_03ec2_row21_col2\" class=\"data row21 col2\" >The study compares people's performances with and without the Vaporflys.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row22\" class=\"row_heading level0 row22\" >22</th>\n",
       "      <td id=\"T_03ec2_row22_col0\" class=\"data row22 col0\" >Our study is most similar to the Upshot study in that we analyze data from marathon performances and compare people's performances with and without the Vaporflys. However, our study differs in a few ways. First, instead of relying on a convenient sample of athletes who upload their data to Strava, we take an exhaustive sample of athletes who met a minimum performance standard at one of 22 of the largest marathon venues in 2015 and 2016 in the US and Canada. Second, instead of relying on self-reported shoe data, we searched the internet for photos of races and visually identified the shoes that runners wore. Third, we focus only on athletes who performed at an elite level before the Vaporflys were released to the public. Thus, we are only considering accomplished runners with marathon experience who, most likely, settled on a suitable shoe before the Vaporflys were released. These runners are also those most likely to be affected by shoe regulations because many of them compete in national Olympic qualifying races subject to regulations.</td>\n",
       "      <td id=\"T_03ec2_row22_col1\" class=\"data row22 col1\" >How does the study's sample selection differ from the Upshot study?</td>\n",
       "      <td id=\"T_03ec2_row22_col2\" class=\"data row22 col2\" >The study uses an exhaustive sample of athletes who met a minimum performance standard at large marathon venues, rather than a convenient sample from Strava.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row23\" class=\"row_heading level0 row23\" >23</th>\n",
       "      <td id=\"T_03ec2_row23_col0\" class=\"data row23 col0\" >Our study is most similar to the Upshot study in that we analyze data from marathon performances and compare people's performances with and without the Vaporflys. However, our study differs in a few ways. First, instead of relying on a convenient sample of athletes who upload their data to Strava, we take an exhaustive sample of athletes who met a minimum performance standard at one of 22 of the largest marathon venues in 2015 and 2016 in the US and Canada. Second, instead of relying on self-reported shoe data, we searched the internet for photos of races and visually identified the shoes that runners wore. Third, we focus only on athletes who performed at an elite level before the Vaporflys were released to the public. Thus, we are only considering accomplished runners with marathon experience who, most likely, settled on a suitable shoe before the Vaporflys were released. These runners are also those most likely to be affected by shoe regulations because many of them compete in national Olympic qualifying races subject to regulations.</td>\n",
       "      <td id=\"T_03ec2_row23_col1\" class=\"data row23 col1\" >Why are the runners in the study likely to be affected by shoe regulations?</td>\n",
       "      <td id=\"T_03ec2_row23_col2\" class=\"data row23 col2\" >Because many of them compete in national Olympic qualifying races which are subject to regulations.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row24\" class=\"row_heading level0 row24\" >24</th>\n",
       "      <td id=\"T_03ec2_row24_col0\" class=\"data row24 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row24_col1\" class=\"data row24 col1\" >What is Mount Everest known for?</td>\n",
       "      <td id=\"T_03ec2_row24_col2\" class=\"data row24 col2\" >It is Earth's highest mountain above sea level.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row25\" class=\"row_heading level0 row25\" >25</th>\n",
       "      <td id=\"T_03ec2_row25_col0\" class=\"data row25 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row25_col1\" class=\"data row25 col1\" >Where is Mount Everest located?</td>\n",
       "      <td id=\"T_03ec2_row25_col2\" class=\"data row25 col2\" >In the Mahalangur Himal sub-range of the Himalayas.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row26\" class=\"row_heading level0 row26\" >26</th>\n",
       "      <td id=\"T_03ec2_row26_col0\" class=\"data row26 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row26_col1\" class=\"data row26 col1\" >Which two countries does the international border run between at Mount Everest's summit?</td>\n",
       "      <td id=\"T_03ec2_row26_col2\" class=\"data row26 col2\" >China (Tibet Autonomous Region) and Nepal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row27\" class=\"row_heading level0 row27\" >27</th>\n",
       "      <td id=\"T_03ec2_row27_col0\" class=\"data row27 col0\" >We selected athletes who recorded a sufficiently fast marathon time--men under 2:24 and women under 2:45--at a collection of 22 distinct marathon venues in 2015 or 2016, including the 2016 U.S. Olympic Marathon Trials, which were contested in Los Angeles in February of 2016. The list of marathons is included in the Appendix. This resulted in a sample of 270 distinct women and 308 distinct men after matching names and our best effort to correct alternate spellings of names. We recorded these athletes' performances in the same 22 marathon venues over the period 2015 to 2019, and searched publicly available online photographs, manually identifying whether or not each athlete was wearing a Nike Vaporfly shoe by visual inspection. All marathon times were downloaded from the website www.marathonguide.com.</td>\n",
       "      <td id=\"T_03ec2_row27_col1\" class=\"data row27 col1\" >What were the qualifying marathon times for men and women?</td>\n",
       "      <td id=\"T_03ec2_row27_col2\" class=\"data row27 col2\" >Men under 2:24 and women under 2:45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row28\" class=\"row_heading level0 row28\" >28</th>\n",
       "      <td id=\"T_03ec2_row28_col0\" class=\"data row28 col0\" >We selected athletes who recorded a sufficiently fast marathon time--men under 2:24 and women under 2:45--at a collection of 22 distinct marathon venues in 2015 or 2016, including the 2016 U.S. Olympic Marathon Trials, which were contested in Los Angeles in February of 2016. The list of marathons is included in the Appendix. This resulted in a sample of 270 distinct women and 308 distinct men after matching names and our best effort to correct alternate spellings of names. We recorded these athletes' performances in the same 22 marathon venues over the period 2015 to 2019, and searched publicly available online photographs, manually identifying whether or not each athlete was wearing a Nike Vaporfly shoe by visual inspection. All marathon times were downloaded from the website www.marathonguide.com.</td>\n",
       "      <td id=\"T_03ec2_row28_col1\" class=\"data row28 col1\" >How many distinct women and men were included in the sample?</td>\n",
       "      <td id=\"T_03ec2_row28_col2\" class=\"data row28 col2\" >270 distinct women and 308 distinct men</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row29\" class=\"row_heading level0 row29\" >29</th>\n",
       "      <td id=\"T_03ec2_row29_col0\" class=\"data row29 col0\" >We selected athletes who recorded a sufficiently fast marathon time--men under 2:24 and women under 2:45--at a collection of 22 distinct marathon venues in 2015 or 2016, including the 2016 U.S. Olympic Marathon Trials, which were contested in Los Angeles in February of 2016. The list of marathons is included in the Appendix. This resulted in a sample of 270 distinct women and 308 distinct men after matching names and our best effort to correct alternate spellings of names. We recorded these athletes' performances in the same 22 marathon venues over the period 2015 to 2019, and searched publicly available online photographs, manually identifying whether or not each athlete was wearing a Nike Vaporfly shoe by visual inspection. All marathon times were downloaded from the website www.marathonguide.com.</td>\n",
       "      <td id=\"T_03ec2_row29_col1\" class=\"data row29 col1\" >What specific equipment were the researchers looking for on the athletes?</td>\n",
       "      <td id=\"T_03ec2_row29_col2\" class=\"data row29 col2\" >Nike Vaporfly shoes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row30\" class=\"row_heading level0 row30\" >30</th>\n",
       "      <td id=\"T_03ec2_row30_col0\" class=\"data row30 col0\" >Our criteria for inclusion in the study were meant to satisfy certain objectives. First, we wanted to study elite and sub-elite athletes, since shoe regulations are motivated by performance advantages for athletes in this group. Second, we wanted to study athletes who had achieved success in the marathon before the Nike Vaporfly shoes had been released to the public. This ensures that inclusion in the study is unrelated to whether an athlete was wearing the shoes in the race where they qualified for inclusion in the study. This is important because, if any shoe effect exists, the magnitude of the effect may differ among different athletes. If we were to use performances potentially aided by the shoes to select the athletes, that might have biased our sample towards athletes who benefit most from the shoes.</td>\n",
       "      <td id=\"T_03ec2_row30_col1\" class=\"data row30 col1\" >What types of athletes did the study aim to include?</td>\n",
       "      <td id=\"T_03ec2_row30_col2\" class=\"data row30 col2\" >Elite and sub-elite athletes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row31\" class=\"row_heading level0 row31\" >31</th>\n",
       "      <td id=\"T_03ec2_row31_col0\" class=\"data row31 col0\" >Our criteria for inclusion in the study were meant to satisfy certain objectives. First, we wanted to study elite and sub-elite athletes, since shoe regulations are motivated by performance advantages for athletes in this group. Second, we wanted to study athletes who had achieved success in the marathon before the Nike Vaporfly shoes had been released to the public. This ensures that inclusion in the study is unrelated to whether an athlete was wearing the shoes in the race where they qualified for inclusion in the study. This is important because, if any shoe effect exists, the magnitude of the effect may differ among different athletes. If we were to use performances potentially aided by the shoes to select the athletes, that might have biased our sample towards athletes who benefit most from the shoes.</td>\n",
       "      <td id=\"T_03ec2_row31_col1\" class=\"data row31 col1\" >Why was it important to include athletes who had success before the release of Nike Vaporfly shoes?</td>\n",
       "      <td id=\"T_03ec2_row31_col2\" class=\"data row31 col2\" >To ensure that inclusion in the study is unrelated to whether an athlete was wearing the shoes when they qualified for inclusion.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row32\" class=\"row_heading level0 row32\" >32</th>\n",
       "      <td id=\"T_03ec2_row32_col0\" class=\"data row32 col0\" >Our criteria for inclusion in the study were meant to satisfy certain objectives. First, we wanted to study elite and sub-elite athletes, since shoe regulations are motivated by performance advantages for athletes in this group. Second, we wanted to study athletes who had achieved success in the marathon before the Nike Vaporfly shoes had been released to the public. This ensures that inclusion in the study is unrelated to whether an athlete was wearing the shoes in the race where they qualified for inclusion in the study. This is important because, if any shoe effect exists, the magnitude of the effect may differ among different athletes. If we were to use performances potentially aided by the shoes to select the athletes, that might have biased our sample towards athletes who benefit most from the shoes.</td>\n",
       "      <td id=\"T_03ec2_row32_col1\" class=\"data row32 col1\" >What could have biased the sample if performances aided by the shoes were used to select athletes?</td>\n",
       "      <td id=\"T_03ec2_row32_col2\" class=\"data row32 col2\" >The sample might have been biased towards athletes who benefit most from the shoes.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row33\" class=\"row_heading level0 row33\" >33</th>\n",
       "      <td id=\"T_03ec2_row33_col0\" class=\"data row33 col0\" >To identify shoes worn by the runners, we used photos posted on public websites such as marathonfoto.com, marathon-photos.com, sportphoto.com, and flashframe.io. We also collected photographs from social media sites such as facebook.com and instagram.com. We assumed that Vaporfly shoes were not worn in 2015 or 2016 by any runners except for a few elite athletes.</td>\n",
       "      <td id=\"T_03ec2_row33_col1\" class=\"data row33 col1\" >What method was used to identify the shoes worn by runners?</td>\n",
       "      <td id=\"T_03ec2_row33_col2\" class=\"data row33 col2\" >Photos posted on public websites and social media sites were used.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row34\" class=\"row_heading level0 row34\" >34</th>\n",
       "      <td id=\"T_03ec2_row34_col0\" class=\"data row34 col0\" >To identify shoes worn by the runners, we used photos posted on public websites such as marathonfoto.com, marathon-photos.com, sportphoto.com, and flashframe.io. We also collected photographs from social media sites such as facebook.com and instagram.com. We assumed that Vaporfly shoes were not worn in 2015 or 2016 by any runners except for a few elite athletes.</td>\n",
       "      <td id=\"T_03ec2_row34_col1\" class=\"data row34 col1\" >Can you name some of the websites where photos were sourced from?</td>\n",
       "      <td id=\"T_03ec2_row34_col2\" class=\"data row34 col2\" >marathonfoto.com, marathon-photos.com, sportphoto.com, flashframe.io, facebook.com, and instagram.com.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row35\" class=\"row_heading level0 row35\" >35</th>\n",
       "      <td id=\"T_03ec2_row35_col0\" class=\"data row35 col0\" >To identify shoes worn by the runners, we used photos posted on public websites such as marathonfoto.com, marathon-photos.com, sportphoto.com, and flashframe.io. We also collected photographs from social media sites such as facebook.com and instagram.com. We assumed that Vaporfly shoes were not worn in 2015 or 2016 by any runners except for a few elite athletes.</td>\n",
       "      <td id=\"T_03ec2_row35_col1\" class=\"data row35 col1\" >What was the assumption made about Vaporfly shoes?</td>\n",
       "      <td id=\"T_03ec2_row35_col2\" class=\"data row35 col2\" >It was assumed that Vaporfly shoes were not worn by any runners except for a few elite athletes in 2015 or 2016.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row36\" class=\"row_heading level0 row36\" >36</th>\n",
       "      <td id=\"T_03ec2_row36_col0\" class=\"data row36 col0\" >that were reported to have worn prototypes in the 2016 US Olympic Trials Marathon. Identification of shoes via photos is a manual process that is subject to error. We have made all of our shoe identifications publicly available at https://github.com/joeguinness/vaporfly and will update this paper with new data if we are made aware of any errors in shoe identification. We identified the shoes worn in 840 of 880 (95.5%) men's performances in our dataset and in 778 of 810 (96.0%) women's performances.</td>\n",
       "      <td id=\"T_03ec2_row36_col1\" class=\"data row36 col1\" >What event were the prototypes worn in?</td>\n",
       "      <td id=\"T_03ec2_row36_col2\" class=\"data row36 col2\" >2016 US Olympic Trials Marathon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row37\" class=\"row_heading level0 row37\" >37</th>\n",
       "      <td id=\"T_03ec2_row37_col0\" class=\"data row37 col0\" >that were reported to have worn prototypes in the 2016 US Olympic Trials Marathon. Identification of shoes via photos is a manual process that is subject to error. We have made all of our shoe identifications publicly available at https://github.com/joeguinness/vaporfly and will update this paper with new data if we are made aware of any errors in shoe identification. We identified the shoes worn in 840 of 880 (95.5%) men's performances in our dataset and in 778 of 810 (96.0%) women's performances.</td>\n",
       "      <td id=\"T_03ec2_row37_col1\" class=\"data row37 col1\" >What is the accuracy percentage of shoe identification for men's performances?</td>\n",
       "      <td id=\"T_03ec2_row37_col2\" class=\"data row37 col2\" >95.5%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row38\" class=\"row_heading level0 row38\" >38</th>\n",
       "      <td id=\"T_03ec2_row38_col0\" class=\"data row38 col0\" >that were reported to have worn prototypes in the 2016 US Olympic Trials Marathon. Identification of shoes via photos is a manual process that is subject to error. We have made all of our shoe identifications publicly available at https://github.com/joeguinness/vaporfly and will update this paper with new data if we are made aware of any errors in shoe identification. We identified the shoes worn in 840 of 880 (95.5%) men's performances in our dataset and in 778 of 810 (96.0%) women's performances.</td>\n",
       "      <td id=\"T_03ec2_row38_col1\" class=\"data row38 col1\" >Where have the shoe identifications been made publicly available?</td>\n",
       "      <td id=\"T_03ec2_row38_col2\" class=\"data row38 col2\" >https://github.com/joeguinness/vaporfly</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row39\" class=\"row_heading level0 row39\" >39</th>\n",
       "      <td id=\"T_03ec2_row39_col0\" class=\"data row39 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row39_col1\" class=\"data row39 col1\" >What is Mount Everest known for?</td>\n",
       "      <td id=\"T_03ec2_row39_col2\" class=\"data row39 col2\" >Mount Everest is known for being Earth's highest mountain above sea level.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row40\" class=\"row_heading level0 row40\" >40</th>\n",
       "      <td id=\"T_03ec2_row40_col0\" class=\"data row40 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row40_col1\" class=\"data row40 col1\" >Where is Mount Everest located?</td>\n",
       "      <td id=\"T_03ec2_row40_col2\" class=\"data row40 col2\" >Mount Everest is located in the Mahalangur Himal sub-range of the Himalayas.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row41\" class=\"row_heading level0 row41\" >41</th>\n",
       "      <td id=\"T_03ec2_row41_col0\" class=\"data row41 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row41_col1\" class=\"data row41 col1\" >Which countries does the international border on Mount Everest's summit point separate?</td>\n",
       "      <td id=\"T_03ec2_row41_col2\" class=\"data row41 col2\" >The international border on Mount Everest's summit point separates China (Tibet Autonomous Region) and Nepal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row42\" class=\"row_heading level0 row42\" >42</th>\n",
       "      <td id=\"T_03ec2_row42_col0\" class=\"data row42 col0\" >In Figure 1, we plot some summaries of the data. The left plot contains the proportion of runners wearing Vaporflys in each race in our dataset, separated by sex. Aside from a few prototypes being used in 2016, adoption of the shoes began in early 2017 and rose to over 50% on average in races at the end of 2019. The right plot contains the average marathon time for each athlete in the dataset in Vaporfly vs. non-Vaporfly shoes. Most runners' average time in Vaporfly shoes is faster than their average time in non-Vaporfly shoes. Specifically, 53 of 71 men (74.5%) who switched to Vaporflys ran faster in them, and 40 of 56 women (71.4%) who switched to Vaporflys ran faster in them.</td>\n",
       "      <td id=\"T_03ec2_row42_col1\" class=\"data row42 col1\" >When did the adoption of Vaporfly shoes begin to rise significantly?</td>\n",
       "      <td id=\"T_03ec2_row42_col2\" class=\"data row42 col2\" >Early 2017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row43\" class=\"row_heading level0 row43\" >43</th>\n",
       "      <td id=\"T_03ec2_row43_col0\" class=\"data row43 col0\" >In Figure 1, we plot some summaries of the data. The left plot contains the proportion of runners wearing Vaporflys in each race in our dataset, separated by sex. Aside from a few prototypes being used in 2016, adoption of the shoes began in early 2017 and rose to over 50% on average in races at the end of 2019. The right plot contains the average marathon time for each athlete in the dataset in Vaporfly vs. non-Vaporfly shoes. Most runners' average time in Vaporfly shoes is faster than their average time in non-Vaporfly shoes. Specifically, 53 of 71 men (74.5%) who switched to Vaporflys ran faster in them, and 40 of 56 women (71.4%) who switched to Vaporflys ran faster in them.</td>\n",
       "      <td id=\"T_03ec2_row43_col1\" class=\"data row43 col1\" >What percentage of men ran faster in Vaporfly shoes according to the data?</td>\n",
       "      <td id=\"T_03ec2_row43_col2\" class=\"data row43 col2\" >74.5%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row44\" class=\"row_heading level0 row44\" >44</th>\n",
       "      <td id=\"T_03ec2_row44_col0\" class=\"data row44 col0\" >In Figure 1, we plot some summaries of the data. The left plot contains the proportion of runners wearing Vaporflys in each race in our dataset, separated by sex. Aside from a few prototypes being used in 2016, adoption of the shoes began in early 2017 and rose to over 50% on average in races at the end of 2019. The right plot contains the average marathon time for each athlete in the dataset in Vaporfly vs. non-Vaporfly shoes. Most runners' average time in Vaporfly shoes is faster than their average time in non-Vaporfly shoes. Specifically, 53 of 71 men (74.5%) who switched to Vaporflys ran faster in them, and 40 of 56 women (71.4%) who switched to Vaporflys ran faster in them.</td>\n",
       "      <td id=\"T_03ec2_row44_col1\" class=\"data row44 col1\" >How does the average marathon time in Vaporfly shoes compare to non-Vaporfly shoes for most runners?</td>\n",
       "      <td id=\"T_03ec2_row44_col2\" class=\"data row44 col2\" >Most runners' average time in Vaporfly shoes is faster than their average time in non-Vaporfly shoes.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row45\" class=\"row_heading level0 row45\" >45</th>\n",
       "      <td id=\"T_03ec2_row45_col0\" class=\"data row45 col0\" >The right plot does not tell the whole story because it might be the case that runners who switched to Vaporflys did so when they ran on faster marathon courses. Some courses, such as the Boston Marathon course, have hills or often have poor weather, while others are flat and fast. So it is important to use the data to attempt to account for the difficulty of each</td>\n",
       "      <td id=\"T_03ec2_row45_col1\" class=\"data row45 col1\" >Why might the right plot not show the full story regarding runners' performances?</td>\n",
       "      <td id=\"T_03ec2_row45_col2\" class=\"data row45 col2\" >Because runners might switch to Vaporflys when they run on faster marathon courses, which could skew the data.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row46\" class=\"row_heading level0 row46\" >46</th>\n",
       "      <td id=\"T_03ec2_row46_col0\" class=\"data row46 col0\" >The right plot does not tell the whole story because it might be the case that runners who switched to Vaporflys did so when they ran on faster marathon courses. Some courses, such as the Boston Marathon course, have hills or often have poor weather, while others are flat and fast. So it is important to use the data to attempt to account for the difficulty of each</td>\n",
       "      <td id=\"T_03ec2_row46_col1\" class=\"data row46 col1\" >What is a characteristic of the Boston Marathon course?</td>\n",
       "      <td id=\"T_03ec2_row46_col2\" class=\"data row46 col2\" >The Boston Marathon course has hills and often has poor weather.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row47\" class=\"row_heading level0 row47\" >47</th>\n",
       "      <td id=\"T_03ec2_row47_col0\" class=\"data row47 col0\" >The right plot does not tell the whole story because it might be the case that runners who switched to Vaporflys did so when they ran on faster marathon courses. Some courses, such as the Boston Marathon course, have hills or often have poor weather, while others are flat and fast. So it is important to use the data to attempt to account for the difficulty of each</td>\n",
       "      <td id=\"T_03ec2_row47_col1\" class=\"data row47 col1\" >Why is it important to consider the data when evaluating marathon courses?</td>\n",
       "      <td id=\"T_03ec2_row47_col2\" class=\"data row47 col2\" >It is important to consider the data to account for the difficulty of each marathon course.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row48\" class=\"row_heading level0 row48\" >48</th>\n",
       "      <td id=\"T_03ec2_row48_col0\" class=\"data row48 col0\" >Figure 1: (Left) Each circle represents an individual race, with the area of the circle proportional to the number of runners from the race in our dataset, and the vertical position equal to the proportion of runners wearing Vaporfly shoes in the race. (Right) Each circle represents an athlete, with the horizontal position being the athlete’s average marathon time in non-Vaporfly shoes, and the vertical position being the athlete’s average time in Vaporfly shoes.</td>\n",
       "      <td id=\"T_03ec2_row48_col1\" class=\"data row48 col1\" >What does the area of a circle represent in the left part of Figure 1?</td>\n",
       "      <td id=\"T_03ec2_row48_col2\" class=\"data row48 col2\" >The number of runners from the race in our dataset</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row49\" class=\"row_heading level0 row49\" >49</th>\n",
       "      <td id=\"T_03ec2_row49_col0\" class=\"data row49 col0\" >Figure 1: (Left) Each circle represents an individual race, with the area of the circle proportional to the number of runners from the race in our dataset, and the vertical position equal to the proportion of runners wearing Vaporfly shoes in the race. (Right) Each circle represents an athlete, with the horizontal position being the athlete’s average marathon time in non-Vaporfly shoes, and the vertical position being the athlete’s average time in Vaporfly shoes.</td>\n",
       "      <td id=\"T_03ec2_row49_col1\" class=\"data row49 col1\" >What does the vertical position of a circle indicate in the left part of Figure 1?</td>\n",
       "      <td id=\"T_03ec2_row49_col2\" class=\"data row49 col2\" >The proportion of runners wearing Vaporfly shoes in the race</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row50\" class=\"row_heading level0 row50\" >50</th>\n",
       "      <td id=\"T_03ec2_row50_col0\" class=\"data row50 col0\" >Figure 1: (Left) Each circle represents an individual race, with the area of the circle proportional to the number of runners from the race in our dataset, and the vertical position equal to the proportion of runners wearing Vaporfly shoes in the race. (Right) Each circle represents an athlete, with the horizontal position being the athlete’s average marathon time in non-Vaporfly shoes, and the vertical position being the athlete’s average time in Vaporfly shoes.</td>\n",
       "      <td id=\"T_03ec2_row50_col1\" class=\"data row50 col1\" >In the right part of Figure 1, what does the horizontal position of a circle represent?</td>\n",
       "      <td id=\"T_03ec2_row50_col2\" class=\"data row50 col2\" >The athlete’s average marathon time in non-Vaporfly shoes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row51\" class=\"row_heading level0 row51\" >51</th>\n",
       "      <td id=\"T_03ec2_row51_col0\" class=\"data row51 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row51_col1\" class=\"data row51 col1\" >What is Mount Everest known for?</td>\n",
       "      <td id=\"T_03ec2_row51_col2\" class=\"data row51 col2\" >It is Earth's highest mountain above sea level.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row52\" class=\"row_heading level0 row52\" >52</th>\n",
       "      <td id=\"T_03ec2_row52_col0\" class=\"data row52 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row52_col1\" class=\"data row52 col1\" >Where is Mount Everest located?</td>\n",
       "      <td id=\"T_03ec2_row52_col2\" class=\"data row52 col2\" >In the Mahalangur Himal sub-range of the Himalayas.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row53\" class=\"row_heading level0 row53\" >53</th>\n",
       "      <td id=\"T_03ec2_row53_col0\" class=\"data row53 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row53_col1\" class=\"data row53 col1\" >Which two countries does the international border run between at Mount Everest's summit?</td>\n",
       "      <td id=\"T_03ec2_row53_col2\" class=\"data row53 col2\" >China (Tibet Autonomous Region) and Nepal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row54\" class=\"row_heading level0 row54\" >54</th>\n",
       "      <td id=\"T_03ec2_row54_col0\" class=\"data row54 col0\" >To get a satisfactory estimate of the effect of Vaporfly shoes, we need to analyze all of the data holistically, controlling for the strength of each runner and the difficulty of each marathon course. In the next section, we describe a statistical model intended for that purpose.</td>\n",
       "      <td id=\"T_03ec2_row54_col1\" class=\"data row54 col1\" >What needs to be analyzed to estimate the effect of Vaporfly shoes?</td>\n",
       "      <td id=\"T_03ec2_row54_col2\" class=\"data row54 col2\" >All of the data holistically, including the strength of each runner and the difficulty of each marathon course.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row55\" class=\"row_heading level0 row55\" >55</th>\n",
       "      <td id=\"T_03ec2_row55_col0\" class=\"data row55 col0\" >To get a satisfactory estimate of the effect of Vaporfly shoes, we need to analyze all of the data holistically, controlling for the strength of each runner and the difficulty of each marathon course. In the next section, we describe a statistical model intended for that purpose.</td>\n",
       "      <td id=\"T_03ec2_row55_col1\" class=\"data row55 col1\" >What should be controlled for in the analysis?</td>\n",
       "      <td id=\"T_03ec2_row55_col2\" class=\"data row55 col2\" >The strength of each runner and the difficulty of each marathon course.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row56\" class=\"row_heading level0 row56\" >56</th>\n",
       "      <td id=\"T_03ec2_row56_col0\" class=\"data row56 col0\" >To get a satisfactory estimate of the effect of Vaporfly shoes, we need to analyze all of the data holistically, controlling for the strength of each runner and the difficulty of each marathon course. In the next section, we describe a statistical model intended for that purpose.</td>\n",
       "      <td id=\"T_03ec2_row56_col1\" class=\"data row56 col1\" >What will be described in the next section?</td>\n",
       "      <td id=\"T_03ec2_row56_col2\" class=\"data row56 col2\" >A statistical model intended for estimating the effect of Vaporfly shoes.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row57\" class=\"row_heading level0 row57\" >57</th>\n",
       "      <td id=\"T_03ec2_row57_col0\" class=\"data row57 col0\" >We seek to estimate the effect of Vaporfly shoes on marathon performance, controlling for runner ability and marathon course difficulty. This is achieved by fitting a statistical model with non-random effects for Vaporfly shoes and random effects for runner ability, overall course difficulty, and course difficulty specific to year. To allow for the possibility that men and women have different performance characteristics, we analyze data from the two sexes separately. We assign each performance a label between 1 and \\(n\\) (\\(n=840\\) for men, \\(n=778\\) for women). Each athlete is assigned a label between 1 and the number of athletes \\(A\\) (\\(A=308\\) men, \\(A=270\\) women). We assign each marathon course a label between 1 and the number of marathon courses \\(C\\) (\\(C=22\\)), and we assign each individual race a label between 1 and the number of races \\(R\\), (\\(R=106\\)). We summarize our notation for the data here:</td>\n",
       "      <td id=\"T_03ec2_row57_col1\" class=\"data row57 col1\" >What is the main objective of the statistical model?</td>\n",
       "      <td id=\"T_03ec2_row57_col2\" class=\"data row57 col2\" >To estimate the effect of Vaporfly shoes on marathon performance, while controlling for runner ability and marathon course difficulty.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row58\" class=\"row_heading level0 row58\" >58</th>\n",
       "      <td id=\"T_03ec2_row58_col0\" class=\"data row58 col0\" >We seek to estimate the effect of Vaporfly shoes on marathon performance, controlling for runner ability and marathon course difficulty. This is achieved by fitting a statistical model with non-random effects for Vaporfly shoes and random effects for runner ability, overall course difficulty, and course difficulty specific to year. To allow for the possibility that men and women have different performance characteristics, we analyze data from the two sexes separately. We assign each performance a label between 1 and \\(n\\) (\\(n=840\\) for men, \\(n=778\\) for women). Each athlete is assigned a label between 1 and the number of athletes \\(A\\) (\\(A=308\\) men, \\(A=270\\) women). We assign each marathon course a label between 1 and the number of marathon courses \\(C\\) (\\(C=22\\)), and we assign each individual race a label between 1 and the number of races \\(R\\), (\\(R=106\\)). We summarize our notation for the data here:</td>\n",
       "      <td id=\"T_03ec2_row58_col1\" class=\"data row58 col1\" >How are the effects of Vaporfly shoes treated in the model?</td>\n",
       "      <td id=\"T_03ec2_row58_col2\" class=\"data row58 col2\" >The effects of Vaporfly shoes are treated as non-random effects in the statistical model.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row59\" class=\"row_heading level0 row59\" >59</th>\n",
       "      <td id=\"T_03ec2_row59_col0\" class=\"data row59 col0\" >We seek to estimate the effect of Vaporfly shoes on marathon performance, controlling for runner ability and marathon course difficulty. This is achieved by fitting a statistical model with non-random effects for Vaporfly shoes and random effects for runner ability, overall course difficulty, and course difficulty specific to year. To allow for the possibility that men and women have different performance characteristics, we analyze data from the two sexes separately. We assign each performance a label between 1 and \\(n\\) (\\(n=840\\) for men, \\(n=778\\) for women). Each athlete is assigned a label between 1 and the number of athletes \\(A\\) (\\(A=308\\) men, \\(A=270\\) women). We assign each marathon course a label between 1 and the number of marathon courses \\(C\\) (\\(C=22\\)), and we assign each individual race a label between 1 and the number of races \\(R\\), (\\(R=106\\)). We summarize our notation for the data here:</td>\n",
       "      <td id=\"T_03ec2_row59_col1\" class=\"data row59 col1\" >Are the data for men and women analyzed together or separately?</td>\n",
       "      <td id=\"T_03ec2_row59_col2\" class=\"data row59 col2\" >The data for men and women are analyzed separately to account for possible differences in performance characteristics.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row60\" class=\"row_heading level0 row60\" >60</th>\n",
       "      <td id=\"T_03ec2_row60_col0\" class=\"data row60 col0\" >[y_{i} =\\text{ marathon time in minutes for performance }i] [x_{i} =\\left\\{\\begin{array}{ll}1&\\text{if Vaporfly shoes worn in performance }i\\\\ 0&\\text{if Vaporfly shoes not worn in performance }i\\end{array}\\right.] [j(i) =\\text{ label for athlete who completed performance }i] [k(i) =\\text{ label for marathon course associated with performance }i] [\\ell(i) =\\text{ label for individual marathon race associated with performance }i]</td>\n",
       "      <td id=\"T_03ec2_row60_col1\" class=\"data row60 col1\" >What does y_{i} represent in the context?</td>\n",
       "      <td id=\"T_03ec2_row60_col2\" class=\"data row60 col2\" >Marathon time in minutes for performance i</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row61\" class=\"row_heading level0 row61\" >61</th>\n",
       "      <td id=\"T_03ec2_row61_col0\" class=\"data row61 col0\" >[y_{i} =\\text{ marathon time in minutes for performance }i] [x_{i} =\\left\\{\\begin{array}{ll}1&\\text{if Vaporfly shoes worn in performance }i\\\\ 0&\\text{if Vaporfly shoes not worn in performance }i\\end{array}\\right.] [j(i) =\\text{ label for athlete who completed performance }i] [k(i) =\\text{ label for marathon course associated with performance }i] [\\ell(i) =\\text{ label for individual marathon race associated with performance }i]</td>\n",
       "      <td id=\"T_03ec2_row61_col1\" class=\"data row61 col1\" >What does x_{i} indicate about the shoes worn during performance i?</td>\n",
       "      <td id=\"T_03ec2_row61_col2\" class=\"data row61 col2\" >x_{i} indicates whether Vaporfly shoes were worn (1) or not worn (0) in performance i</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row62\" class=\"row_heading level0 row62\" >62</th>\n",
       "      <td id=\"T_03ec2_row62_col0\" class=\"data row62 col0\" >[y_{i} =\\text{ marathon time in minutes for performance }i] [x_{i} =\\left\\{\\begin{array}{ll}1&\\text{if Vaporfly shoes worn in performance }i\\\\ 0&\\text{if Vaporfly shoes not worn in performance }i\\end{array}\\right.] [j(i) =\\text{ label for athlete who completed performance }i] [k(i) =\\text{ label for marathon course associated with performance }i] [\\ell(i) =\\text{ label for individual marathon race associated with performance }i]</td>\n",
       "      <td id=\"T_03ec2_row62_col1\" class=\"data row62 col1\" >What are j(i), k(i), and \\ell(i) labels for?</td>\n",
       "      <td id=\"T_03ec2_row62_col2\" class=\"data row62 col2\" >j(i) is the label for the athlete, k(i) is the label for the marathon course, and \\ell(i) is the label for the individual marathon race associated with performance i</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row63\" class=\"row_heading level0 row63\" >63</th>\n",
       "      <td id=\"T_03ec2_row63_col0\" class=\"data row63 col0\" >A statistical model is a family of probability distributions that encodes the assumptions we make about the processes generating the data. Models generally include unknown parameters relevant to the questions posed in the study. The goal of the analysis is to use the data to make inferences about these parameters. We consider the following two models for the performances \\(y_{1},\\ldots,y_{n}\\):</td>\n",
       "      <td id=\"T_03ec2_row63_col1\" class=\"data row63 col1\" >What is a statistical model?</td>\n",
       "      <td id=\"T_03ec2_row63_col2\" class=\"data row63 col2\" >A statistical model is a family of probability distributions that represents the assumptions about the processes generating the data.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row64\" class=\"row_heading level0 row64\" >64</th>\n",
       "      <td id=\"T_03ec2_row64_col0\" class=\"data row64 col0\" >A statistical model is a family of probability distributions that encodes the assumptions we make about the processes generating the data. Models generally include unknown parameters relevant to the questions posed in the study. The goal of the analysis is to use the data to make inferences about these parameters. We consider the following two models for the performances \\(y_{1},\\ldots,y_{n}\\):</td>\n",
       "      <td id=\"T_03ec2_row64_col1\" class=\"data row64 col1\" >What do models generally include?</td>\n",
       "      <td id=\"T_03ec2_row64_col2\" class=\"data row64 col2\" >Models generally include unknown parameters relevant to the questions posed in the study.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row65\" class=\"row_heading level0 row65\" >65</th>\n",
       "      <td id=\"T_03ec2_row65_col0\" class=\"data row65 col0\" >A statistical model is a family of probability distributions that encodes the assumptions we make about the processes generating the data. Models generally include unknown parameters relevant to the questions posed in the study. The goal of the analysis is to use the data to make inferences about these parameters. We consider the following two models for the performances \\(y_{1},\\ldots,y_{n}\\):</td>\n",
       "      <td id=\"T_03ec2_row65_col1\" class=\"data row65 col1\" >What is the goal of the analysis in statistical modeling?</td>\n",
       "      <td id=\"T_03ec2_row65_col2\" class=\"data row65 col2\" >The goal of the analysis is to use the data to make inferences about the unknown parameters.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row66\" class=\"row_heading level0 row66\" >66</th>\n",
       "      <td id=\"T_03ec2_row66_col0\" class=\"data row66 col0\" >\\[\\text{Untransformed:}\\qquad\\quad Y_{i} =b_{0}+b_{1}x_{i}+U_{j(i)}+V_{k(i)}+W_{\\ell(i)}+Z_{i}\\] \\[\\text{Log Untransformed:}\\quad\\log Y_{i} =b_{0}+b_{1}x_{i}+U_{j(i)}+V_{k(i)}+W_{\\ell(i)}+Z_{i}\\]</td>\n",
       "      <td id=\"T_03ec2_row66_col1\" class=\"data row66 col1\" >What is the dependent variable in the untransformed equation?</td>\n",
       "      <td id=\"T_03ec2_row66_col2\" class=\"data row66 col2\" >Y_{i}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row67\" class=\"row_heading level0 row67\" >67</th>\n",
       "      <td id=\"T_03ec2_row67_col0\" class=\"data row67 col0\" >\\[\\text{Untransformed:}\\qquad\\quad Y_{i} =b_{0}+b_{1}x_{i}+U_{j(i)}+V_{k(i)}+W_{\\ell(i)}+Z_{i}\\] \\[\\text{Log Untransformed:}\\quad\\log Y_{i} =b_{0}+b_{1}x_{i}+U_{j(i)}+V_{k(i)}+W_{\\ell(i)}+Z_{i}\\]</td>\n",
       "      <td id=\"T_03ec2_row67_col1\" class=\"data row67 col1\" >In the log untransformed equation, what operation is performed on the dependent variable?</td>\n",
       "      <td id=\"T_03ec2_row67_col2\" class=\"data row67 col2\" >Logarithm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row68\" class=\"row_heading level0 row68\" >68</th>\n",
       "      <td id=\"T_03ec2_row68_col0\" class=\"data row68 col0\" >\\[\\text{Untransformed:}\\qquad\\quad Y_{i} =b_{0}+b_{1}x_{i}+U_{j(i)}+V_{k(i)}+W_{\\ell(i)}+Z_{i}\\] \\[\\text{Log Untransformed:}\\quad\\log Y_{i} =b_{0}+b_{1}x_{i}+U_{j(i)}+V_{k(i)}+W_{\\ell(i)}+Z_{i}\\]</td>\n",
       "      <td id=\"T_03ec2_row68_col1\" class=\"data row68 col1\" >What is the independent variable in both equations?</td>\n",
       "      <td id=\"T_03ec2_row68_col2\" class=\"data row68 col2\" >x_{i}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row69\" class=\"row_heading level0 row69\" >69</th>\n",
       "      <td id=\"T_03ec2_row69_col0\" class=\"data row69 col0\" >The primary parameter of interest is \\(b_{1}\\), which is the effect of the Vaporfly shoes. The model assumes that, all else held constant, switching to Vaporfly shoes changes the response by \\(b_{1}\\). We do not attempt to model Vaporfly effects that vary among individual runners. The interpretations of the parameters are different depending on whether we take a log transformation of the times or not. When modeling untransformed times, the effect of Vaporfly shoes is additive, meaning that we expect the time to change by adding \\(b_{1}\\), and when modeling log-transformed times, the effect is multiplicative, meaning that we expect the time to change by multiplying by \\(\\exp(b_{1})\\).</td>\n",
       "      <td id=\"T_03ec2_row69_col1\" class=\"data row69 col1\" >What parameter represents the effect of the Vaporfly shoes in the model?</td>\n",
       "      <td id=\"T_03ec2_row69_col2\" class=\"data row69 col2\" >\\(b_{1}\\)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row70\" class=\"row_heading level0 row70\" >70</th>\n",
       "      <td id=\"T_03ec2_row70_col0\" class=\"data row70 col0\" >The primary parameter of interest is \\(b_{1}\\), which is the effect of the Vaporfly shoes. The model assumes that, all else held constant, switching to Vaporfly shoes changes the response by \\(b_{1}\\). We do not attempt to model Vaporfly effects that vary among individual runners. The interpretations of the parameters are different depending on whether we take a log transformation of the times or not. When modeling untransformed times, the effect of Vaporfly shoes is additive, meaning that we expect the time to change by adding \\(b_{1}\\), and when modeling log-transformed times, the effect is multiplicative, meaning that we expect the time to change by multiplying by \\(\\exp(b_{1})\\).</td>\n",
       "      <td id=\"T_03ec2_row70_col1\" class=\"data row70 col1\" >How does the effect of Vaporfly shoes on time differ when using log-transformed times compared to untransformed times?</td>\n",
       "      <td id=\"T_03ec2_row70_col2\" class=\"data row70 col2\" >For log-transformed times, the effect is multiplicative (time changes by multiplying by \\(\\exp(b_{1})\\)), while for untransformed times, the effect is additive (time changes by adding \\(b_{1}\\)).</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row71\" class=\"row_heading level0 row71\" >71</th>\n",
       "      <td id=\"T_03ec2_row71_col0\" class=\"data row71 col0\" >The primary parameter of interest is \\(b_{1}\\), which is the effect of the Vaporfly shoes. The model assumes that, all else held constant, switching to Vaporfly shoes changes the response by \\(b_{1}\\). We do not attempt to model Vaporfly effects that vary among individual runners. The interpretations of the parameters are different depending on whether we take a log transformation of the times or not. When modeling untransformed times, the effect of Vaporfly shoes is additive, meaning that we expect the time to change by adding \\(b_{1}\\), and when modeling log-transformed times, the effect is multiplicative, meaning that we expect the time to change by multiplying by \\(\\exp(b_{1})\\).</td>\n",
       "      <td id=\"T_03ec2_row71_col1\" class=\"data row71 col1\" >Does the model account for individual variations in the effects of Vaporfly shoes?</td>\n",
       "      <td id=\"T_03ec2_row71_col2\" class=\"data row71 col2\" >No, the model does not attempt to model Vaporfly effects that vary among individual runners.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row72\" class=\"row_heading level0 row72\" >72</th>\n",
       "      <td id=\"T_03ec2_row72_col0\" class=\"data row72 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row72_col1\" class=\"data row72 col1\" >What is Mount Everest known for?</td>\n",
       "      <td id=\"T_03ec2_row72_col2\" class=\"data row72 col2\" >It is Earth's highest mountain above sea level.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row73\" class=\"row_heading level0 row73\" >73</th>\n",
       "      <td id=\"T_03ec2_row73_col0\" class=\"data row73 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row73_col1\" class=\"data row73 col1\" >Where is Mount Everest located?</td>\n",
       "      <td id=\"T_03ec2_row73_col2\" class=\"data row73 col2\" >In the Mahalangur Himal sub-range of the Himalayas.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row74\" class=\"row_heading level0 row74\" >74</th>\n",
       "      <td id=\"T_03ec2_row74_col0\" class=\"data row74 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row74_col1\" class=\"data row74 col1\" >Which two countries does the international border cross at Mount Everest's summit?</td>\n",
       "      <td id=\"T_03ec2_row74_col2\" class=\"data row74 col2\" >China (Tibet Autonomous Region) and Nepal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row75\" class=\"row_heading level0 row75\" >75</th>\n",
       "      <td id=\"T_03ec2_row75_col0\" class=\"data row75 col0\" >Aside from \\(b_{0}\\) and \\(b_{1}x_{i}\\), the rest of the terms are independent normal random effects. Each of the \\(A\\) runners has its own offset term \\(U_{j}\\) to account for the fact that runners have differing abilities; each of the \\(C\\) marathon courses has its own offset term \\(V_{k}\\) to account for the fact that different marathon courses are slower or faster than others; each of the \\(R\\) individual races has its own offset term \\(W_{\\ell}\\) to account for the fact that race conditions vary from year to year, making some years slower or faster than others at the same course; and finally each of the \\(n\\) individual performances has a term \\(Z_{i}\\) to account for any other factors that affected the performance. We also considered including time-varying runner effects, to allow each runner's fitness to improve or decline over time independently of the fitness of other runners, but we decided that this model placed too much weight on athletes who raced frequently. In the Appendix, we also include results from a combined model for men and women.</td>\n",
       "      <td id=\"T_03ec2_row75_col1\" class=\"data row75 col1\" >What does the term \\(U_{j}\\) represent in the model?</td>\n",
       "      <td id=\"T_03ec2_row75_col2\" class=\"data row75 col2\" >It represents an offset term for each runner to account for differing abilities.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row76\" class=\"row_heading level0 row76\" >76</th>\n",
       "      <td id=\"T_03ec2_row76_col0\" class=\"data row76 col0\" >Aside from \\(b_{0}\\) and \\(b_{1}x_{i}\\), the rest of the terms are independent normal random effects. Each of the \\(A\\) runners has its own offset term \\(U_{j}\\) to account for the fact that runners have differing abilities; each of the \\(C\\) marathon courses has its own offset term \\(V_{k}\\) to account for the fact that different marathon courses are slower or faster than others; each of the \\(R\\) individual races has its own offset term \\(W_{\\ell}\\) to account for the fact that race conditions vary from year to year, making some years slower or faster than others at the same course; and finally each of the \\(n\\) individual performances has a term \\(Z_{i}\\) to account for any other factors that affected the performance. We also considered including time-varying runner effects, to allow each runner's fitness to improve or decline over time independently of the fitness of other runners, but we decided that this model placed too much weight on athletes who raced frequently. In the Appendix, we also include results from a combined model for men and women.</td>\n",
       "      <td id=\"T_03ec2_row76_col1\" class=\"data row76 col1\" >Why were time-varying runner effects not included in the final model?</td>\n",
       "      <td id=\"T_03ec2_row76_col2\" class=\"data row76 col2\" >Because it placed too much weight on athletes who raced frequently.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row77\" class=\"row_heading level0 row77\" >77</th>\n",
       "      <td id=\"T_03ec2_row77_col0\" class=\"data row77 col0\" >Aside from \\(b_{0}\\) and \\(b_{1}x_{i}\\), the rest of the terms are independent normal random effects. Each of the \\(A\\) runners has its own offset term \\(U_{j}\\) to account for the fact that runners have differing abilities; each of the \\(C\\) marathon courses has its own offset term \\(V_{k}\\) to account for the fact that different marathon courses are slower or faster than others; each of the \\(R\\) individual races has its own offset term \\(W_{\\ell}\\) to account for the fact that race conditions vary from year to year, making some years slower or faster than others at the same course; and finally each of the \\(n\\) individual performances has a term \\(Z_{i}\\) to account for any other factors that affected the performance. We also considered including time-varying runner effects, to allow each runner's fitness to improve or decline over time independently of the fitness of other runners, but we decided that this model placed too much weight on athletes who raced frequently. In the Appendix, we also include results from a combined model for men and women.</td>\n",
       "      <td id=\"T_03ec2_row77_col1\" class=\"data row77 col1\" >What is the purpose of the term \\(Z_{i}\\) in the model?</td>\n",
       "      <td id=\"T_03ec2_row77_col2\" class=\"data row77 col2\" >To account for any other factors that affected the individual performances.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row78\" class=\"row_heading level0 row78\" >78</th>\n",
       "      <td id=\"T_03ec2_row78_col0\" class=\"data row78 col0\" >To fit the models, we used the lmer function, which is part of the lme4 package (Bates et al., 2015) in the R programming language (R Core Team, 2018). The lme4 package is a well-established piece of statistical software for fitting random effects models of the type we seek to estimate in this research study. Code and data for reproducing our results are available online at https://github.com/joeguinness/vaporfly.</td>\n",
       "      <td id=\"T_03ec2_row78_col1\" class=\"data row78 col1\" >What function was used to fit the models?</td>\n",
       "      <td id=\"T_03ec2_row78_col2\" class=\"data row78 col2\" >lmer function</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row79\" class=\"row_heading level0 row79\" >79</th>\n",
       "      <td id=\"T_03ec2_row79_col0\" class=\"data row79 col0\" >To fit the models, we used the lmer function, which is part of the lme4 package (Bates et al., 2015) in the R programming language (R Core Team, 2018). The lme4 package is a well-established piece of statistical software for fitting random effects models of the type we seek to estimate in this research study. Code and data for reproducing our results are available online at https://github.com/joeguinness/vaporfly.</td>\n",
       "      <td id=\"T_03ec2_row79_col1\" class=\"data row79 col1\" >Which package does the lmer function belong to?</td>\n",
       "      <td id=\"T_03ec2_row79_col2\" class=\"data row79 col2\" >lme4 package</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row80\" class=\"row_heading level0 row80\" >80</th>\n",
       "      <td id=\"T_03ec2_row80_col0\" class=\"data row80 col0\" >To fit the models, we used the lmer function, which is part of the lme4 package (Bates et al., 2015) in the R programming language (R Core Team, 2018). The lme4 package is a well-established piece of statistical software for fitting random effects models of the type we seek to estimate in this research study. Code and data for reproducing our results are available online at https://github.com/joeguinness/vaporfly.</td>\n",
       "      <td id=\"T_03ec2_row80_col1\" class=\"data row80 col1\" >Where can the code and data for this research be found?</td>\n",
       "      <td id=\"T_03ec2_row80_col2\" class=\"data row80 col2\" >Online at https://github.com/joeguinness/vaporfly</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row81\" class=\"row_heading level0 row81\" >81</th>\n",
       "      <td id=\"T_03ec2_row81_col0\" class=\"data row81 col0\" >We fit separate models for men and women, and additionally, separate models for the untransformed and log-transformed marathon times. The estimated parameters are summarized in the table below:</td>\n",
       "      <td id=\"T_03ec2_row81_col1\" class=\"data row81 col1\" >Did the study fit the same model for both men and women?</td>\n",
       "      <td id=\"T_03ec2_row81_col2\" class=\"data row81 col2\" >No, separate models were fit for men and women.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row82\" class=\"row_heading level0 row82\" >82</th>\n",
       "      <td id=\"T_03ec2_row82_col0\" class=\"data row82 col0\" >We fit separate models for men and women, and additionally, separate models for the untransformed and log-transformed marathon times. The estimated parameters are summarized in the table below:</td>\n",
       "      <td id=\"T_03ec2_row82_col1\" class=\"data row82 col1\" >What types of marathon times were models fit for?</td>\n",
       "      <td id=\"T_03ec2_row82_col2\" class=\"data row82 col2\" >Models were fit for both untransformed and log-transformed marathon times.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row83\" class=\"row_heading level0 row83\" >83</th>\n",
       "      <td id=\"T_03ec2_row83_col0\" class=\"data row83 col0\" >We fit separate models for men and women, and additionally, separate models for the untransformed and log-transformed marathon times. The estimated parameters are summarized in the table below:</td>\n",
       "      <td id=\"T_03ec2_row83_col1\" class=\"data row83 col1\" >Where are the estimated parameters from the models displayed?</td>\n",
       "      <td id=\"T_03ec2_row83_col2\" class=\"data row83 col2\" >The estimated parameters are summarized in the table below.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row84\" class=\"row_heading level0 row84\" >84</th>\n",
       "      <td id=\"T_03ec2_row84_col0\" class=\"data row84 col0\" >For both men and women and for untransformed and log-transformed times, the Vaporfly effect is negative, indicating that the evidence supports the hypothesis that Vaporflys decrease</td>\n",
       "      <td id=\"T_03ec2_row84_col1\" class=\"data row84 col1\" >What is the Vaporfly effect on running times according to the evidence?</td>\n",
       "      <td id=\"T_03ec2_row84_col2\" class=\"data row84 col2\" >Negative, indicating a decrease in times</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row85\" class=\"row_heading level0 row85\" >85</th>\n",
       "      <td id=\"T_03ec2_row85_col0\" class=\"data row85 col0\" >For both men and women and for untransformed and log-transformed times, the Vaporfly effect is negative, indicating that the evidence supports the hypothesis that Vaporflys decrease</td>\n",
       "      <td id=\"T_03ec2_row85_col1\" class=\"data row85 col1\" >Does the Vaporfly effect apply to both genders?</td>\n",
       "      <td id=\"T_03ec2_row85_col2\" class=\"data row85 col2\" >Yes, it applies to both men and women</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row86\" class=\"row_heading level0 row86\" >86</th>\n",
       "      <td id=\"T_03ec2_row86_col0\" class=\"data row86 col0\" >For both men and women and for untransformed and log-transformed times, the Vaporfly effect is negative, indicating that the evidence supports the hypothesis that Vaporflys decrease</td>\n",
       "      <td id=\"T_03ec2_row86_col1\" class=\"data row86 col1\" >What does the evidence suggest about Vaporflys?</td>\n",
       "      <td id=\"T_03ec2_row86_col2\" class=\"data row86 col2\" >The evidence supports the hypothesis that Vaporflys decrease running times</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row87\" class=\"row_heading level0 row87\" >87</th>\n",
       "      <td id=\"T_03ec2_row87_col0\" class=\"data row87 col0\" >\\begin{table}\n",
       "\\begin{tabular}{r r r r}  & \\multicolumn{1}{c}{men minutes} & \\multicolumn{1}{c}{women minutes} & \\multicolumn{1}{c}{men log minutes} & \\multicolumn{1}{c}{women log minutes} \\\\ \\cline{2-4}  & estimate (s.e.) & estimate (s.e.) & estimate (s.e.) & estimate (s.e.) \\\\ \\hline \\(b_{0}\\) & 139.69 (0.59) & 159.83 (0.81) & 4.94 (0.004) & 5.070 (0.0050) \\\\ \\(b_{1}\\) & \\(\\boldsymbol{-2.95}\\) **(0.60)** & \\(\\boldsymbol{-2.18}\\) **(0.81)** & \\(\\boldsymbol{-0.0209}\\) **(0.0041)** & \\(\\boldsymbol{-0.0135}\\) **(0.0049)** \\\\ \\(\\sigma_{1}\\) & 4.175 & 6.40 & 0.030 & 0.041 \\\\ \\(\\sigma_{2}\\) & 1.852 & 2.33 & 0.013 & 0.014 \\\\ \\(\\sigma_{3}\\) & 1.874 & 2.43 & 0.013 & 0.015 \\\\ \\(\\sigma_{4}\\) & 4.108 & 5.02 & 0.028 & 0.030 \\\\ \\end{tabular}\n",
       "\\end{table}\n",
       "Table 1: Table of parameter estimates for the statistical models.</td>\n",
       "      <td id=\"T_03ec2_row87_col1\" class=\"data row87 col1\" >What is the estimate and standard error (s.e.) for men minutes for \\(b_{0}\\)?</td>\n",
       "      <td id=\"T_03ec2_row87_col2\" class=\"data row87 col2\" >The estimate for men minutes for \\(b_{0}\\) is 139.69 with a standard error of 0.59.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row88\" class=\"row_heading level0 row88\" >88</th>\n",
       "      <td id=\"T_03ec2_row88_col0\" class=\"data row88 col0\" >\\begin{table}\n",
       "\\begin{tabular}{r r r r}  & \\multicolumn{1}{c}{men minutes} & \\multicolumn{1}{c}{women minutes} & \\multicolumn{1}{c}{men log minutes} & \\multicolumn{1}{c}{women log minutes} \\\\ \\cline{2-4}  & estimate (s.e.) & estimate (s.e.) & estimate (s.e.) & estimate (s.e.) \\\\ \\hline \\(b_{0}\\) & 139.69 (0.59) & 159.83 (0.81) & 4.94 (0.004) & 5.070 (0.0050) \\\\ \\(b_{1}\\) & \\(\\boldsymbol{-2.95}\\) **(0.60)** & \\(\\boldsymbol{-2.18}\\) **(0.81)** & \\(\\boldsymbol{-0.0209}\\) **(0.0041)** & \\(\\boldsymbol{-0.0135}\\) **(0.0049)** \\\\ \\(\\sigma_{1}\\) & 4.175 & 6.40 & 0.030 & 0.041 \\\\ \\(\\sigma_{2}\\) & 1.852 & 2.33 & 0.013 & 0.014 \\\\ \\(\\sigma_{3}\\) & 1.874 & 2.43 & 0.013 & 0.015 \\\\ \\(\\sigma_{4}\\) & 4.108 & 5.02 & 0.028 & 0.030 \\\\ \\end{tabular}\n",
       "\\end{table}\n",
       "Table 1: Table of parameter estimates for the statistical models.</td>\n",
       "      <td id=\"T_03ec2_row88_col1\" class=\"data row88 col1\" >Which parameter estimate is bolded for women log minutes for \\(b_{1}\\)?</td>\n",
       "      <td id=\"T_03ec2_row88_col2\" class=\"data row88 col2\" >The bolded parameter estimate for women log minutes for \\(b_{1}\\) is \\(-0.0135\\) with a standard error of 0.0049.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row89\" class=\"row_heading level0 row89\" >89</th>\n",
       "      <td id=\"T_03ec2_row89_col0\" class=\"data row89 col0\" >\\begin{table}\n",
       "\\begin{tabular}{r r r r}  & \\multicolumn{1}{c}{men minutes} & \\multicolumn{1}{c}{women minutes} & \\multicolumn{1}{c}{men log minutes} & \\multicolumn{1}{c}{women log minutes} \\\\ \\cline{2-4}  & estimate (s.e.) & estimate (s.e.) & estimate (s.e.) & estimate (s.e.) \\\\ \\hline \\(b_{0}\\) & 139.69 (0.59) & 159.83 (0.81) & 4.94 (0.004) & 5.070 (0.0050) \\\\ \\(b_{1}\\) & \\(\\boldsymbol{-2.95}\\) **(0.60)** & \\(\\boldsymbol{-2.18}\\) **(0.81)** & \\(\\boldsymbol{-0.0209}\\) **(0.0041)** & \\(\\boldsymbol{-0.0135}\\) **(0.0049)** \\\\ \\(\\sigma_{1}\\) & 4.175 & 6.40 & 0.030 & 0.041 \\\\ \\(\\sigma_{2}\\) & 1.852 & 2.33 & 0.013 & 0.014 \\\\ \\(\\sigma_{3}\\) & 1.874 & 2.43 & 0.013 & 0.015 \\\\ \\(\\sigma_{4}\\) & 4.108 & 5.02 & 0.028 & 0.030 \\\\ \\end{tabular}\n",
       "\\end{table}\n",
       "Table 1: Table of parameter estimates for the statistical models.</td>\n",
       "      <td id=\"T_03ec2_row89_col1\" class=\"data row89 col1\" >What are the values of \\(\\sigma_{3}\\) for men and women minutes?</td>\n",
       "      <td id=\"T_03ec2_row89_col2\" class=\"data row89 col2\" >The value of \\(\\sigma_{3}\\) for men minutes is 1.874 and for women minutes is 2.43.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row90\" class=\"row_heading level0 row90\" >90</th>\n",
       "      <td id=\"T_03ec2_row90_col0\" class=\"data row90 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row90_col1\" class=\"data row90 col1\" >What is Mount Everest known for?</td>\n",
       "      <td id=\"T_03ec2_row90_col2\" class=\"data row90 col2\" >It is Earth's highest mountain above sea level.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row91\" class=\"row_heading level0 row91\" >91</th>\n",
       "      <td id=\"T_03ec2_row91_col0\" class=\"data row91 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row91_col1\" class=\"data row91 col1\" >Where is Mount Everest located?</td>\n",
       "      <td id=\"T_03ec2_row91_col2\" class=\"data row91 col2\" >In the Mahalangur Himal sub-range of the Himalayas.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row92\" class=\"row_heading level0 row92\" >92</th>\n",
       "      <td id=\"T_03ec2_row92_col0\" class=\"data row92 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row92_col1\" class=\"data row92 col1\" >Which two countries does the international border run between at Mount Everest's summit?</td>\n",
       "      <td id=\"T_03ec2_row92_col2\" class=\"data row92 col2\" >China (Tibet Autonomous Region) and Nepal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row93\" class=\"row_heading level0 row93\" >93</th>\n",
       "      <td id=\"T_03ec2_row93_col0\" class=\"data row93 col0\" >or improve, marathon times. Our best estimates of the additive effects are \\(-2.95\\) minutes for men and \\(-2.18\\) minutes for women. Using log-transformed data, our best estimates of the multiplicative effects are \\(\\exp(-0.0209)=0.979\\) for men, and \\(\\exp(-0.0135)=0.986\\) for women, meaning that we expect men's times to decrease by \\(2.06\\%\\), and women's times to decrease by \\(1.34\\%\\), when wearing Vaporfly shoes, as compared to the shoes each athlete was wearing before switching to Vaporfly.</td>\n",
       "      <td id=\"T_03ec2_row93_col1\" class=\"data row93 col1\" >What is the estimated additive effect of wearing Vaporfly shoes on men's marathon times?</td>\n",
       "      <td id=\"T_03ec2_row93_col2\" class=\"data row93 col2\" >-2.95 minutes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row94\" class=\"row_heading level0 row94\" >94</th>\n",
       "      <td id=\"T_03ec2_row94_col0\" class=\"data row94 col0\" >or improve, marathon times. Our best estimates of the additive effects are \\(-2.95\\) minutes for men and \\(-2.18\\) minutes for women. Using log-transformed data, our best estimates of the multiplicative effects are \\(\\exp(-0.0209)=0.979\\) for men, and \\(\\exp(-0.0135)=0.986\\) for women, meaning that we expect men's times to decrease by \\(2.06\\%\\), and women's times to decrease by \\(1.34\\%\\), when wearing Vaporfly shoes, as compared to the shoes each athlete was wearing before switching to Vaporfly.</td>\n",
       "      <td id=\"T_03ec2_row94_col1\" class=\"data row94 col1\" >By what percentage are women's marathon times expected to decrease when wearing Vaporfly shoes?</td>\n",
       "      <td id=\"T_03ec2_row94_col2\" class=\"data row94 col2\" >1.34%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row95\" class=\"row_heading level0 row95\" >95</th>\n",
       "      <td id=\"T_03ec2_row95_col0\" class=\"data row95 col0\" >or improve, marathon times. Our best estimates of the additive effects are \\(-2.95\\) minutes for men and \\(-2.18\\) minutes for women. Using log-transformed data, our best estimates of the multiplicative effects are \\(\\exp(-0.0209)=0.979\\) for men, and \\(\\exp(-0.0135)=0.986\\) for women, meaning that we expect men's times to decrease by \\(2.06\\%\\), and women's times to decrease by \\(1.34\\%\\), when wearing Vaporfly shoes, as compared to the shoes each athlete was wearing before switching to Vaporfly.</td>\n",
       "      <td id=\"T_03ec2_row95_col1\" class=\"data row95 col1\" >What is the multiplicative effect of Vaporfly shoes on men's marathon times according to the log-transformed data?</td>\n",
       "      <td id=\"T_03ec2_row95_col2\" class=\"data row95 col2\" >0.979 (a decrease of 2.06%)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row96\" class=\"row_heading level0 row96\" >96</th>\n",
       "      <td id=\"T_03ec2_row96_col0\" class=\"data row96 col0\" >While our estimates suggest that the effect of Vaporfly shoes is greater for men, the estimates come with some uncertainty. In the following table, we include \\(90\\%\\) confidence intervals for each of the Vaporfly effects, constructed using a normal approximation to the sampling distribution of the estimates.</td>\n",
       "      <td id=\"T_03ec2_row96_col1\" class=\"data row96 col1\" >For which gender do Vaporfly shoes have a greater effect according to the estimates?</td>\n",
       "      <td id=\"T_03ec2_row96_col2\" class=\"data row96 col2\" >Men</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row97\" class=\"row_heading level0 row97\" >97</th>\n",
       "      <td id=\"T_03ec2_row97_col0\" class=\"data row97 col0\" >While our estimates suggest that the effect of Vaporfly shoes is greater for men, the estimates come with some uncertainty. In the following table, we include \\(90\\%\\) confidence intervals for each of the Vaporfly effects, constructed using a normal approximation to the sampling distribution of the estimates.</td>\n",
       "      <td id=\"T_03ec2_row97_col1\" class=\"data row97 col1\" >What level of confidence is used for the confidence intervals in the table?</td>\n",
       "      <td id=\"T_03ec2_row97_col2\" class=\"data row97 col2\" >90%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row98\" class=\"row_heading level0 row98\" >98</th>\n",
       "      <td id=\"T_03ec2_row98_col0\" class=\"data row98 col0\" >While our estimates suggest that the effect of Vaporfly shoes is greater for men, the estimates come with some uncertainty. In the following table, we include \\(90\\%\\) confidence intervals for each of the Vaporfly effects, constructed using a normal approximation to the sampling distribution of the estimates.</td>\n",
       "      <td id=\"T_03ec2_row98_col1\" class=\"data row98 col1\" >What method is used to construct the confidence intervals for the Vaporfly effects?</td>\n",
       "      <td id=\"T_03ec2_row98_col2\" class=\"data row98 col2\" >Normal approximation to the sampling distribution of the estimates</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row99\" class=\"row_heading level0 row99\" >99</th>\n",
       "      <td id=\"T_03ec2_row99_col0\" class=\"data row99 col0\" >None of the intervals contain zero, which indicates strong evidence for a non-zero Vaporfly effect. There is substantial overlap between the men's and women's confidence intervals, which leaves some uncertainty about which sex benefits most from Vaporfly shoes. In the Appendix, we include an analysis assessing the difference between the men's and women's Vaporfly effects; we find that we do not have sufficient evidence to conclude that the effects differ by sex.</td>\n",
       "      <td id=\"T_03ec2_row99_col1\" class=\"data row99 col1\" >What does the absence of zero in the intervals suggest?</td>\n",
       "      <td id=\"T_03ec2_row99_col2\" class=\"data row99 col2\" >It suggests strong evidence for a non-zero Vaporfly effect.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row100\" class=\"row_heading level0 row100\" >100</th>\n",
       "      <td id=\"T_03ec2_row100_col0\" class=\"data row100 col0\" >None of the intervals contain zero, which indicates strong evidence for a non-zero Vaporfly effect. There is substantial overlap between the men's and women's confidence intervals, which leaves some uncertainty about which sex benefits most from Vaporfly shoes. In the Appendix, we include an analysis assessing the difference between the men's and women's Vaporfly effects; we find that we do not have sufficient evidence to conclude that the effects differ by sex.</td>\n",
       "      <td id=\"T_03ec2_row100_col1\" class=\"data row100 col1\" >Why is there uncertainty about which sex benefits most from Vaporfly shoes?</td>\n",
       "      <td id=\"T_03ec2_row100_col2\" class=\"data row100 col2\" >Because there is substantial overlap between the men's and women's confidence intervals.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row101\" class=\"row_heading level0 row101\" >101</th>\n",
       "      <td id=\"T_03ec2_row101_col0\" class=\"data row101 col0\" >None of the intervals contain zero, which indicates strong evidence for a non-zero Vaporfly effect. There is substantial overlap between the men's and women's confidence intervals, which leaves some uncertainty about which sex benefits most from Vaporfly shoes. In the Appendix, we include an analysis assessing the difference between the men's and women's Vaporfly effects; we find that we do not have sufficient evidence to conclude that the effects differ by sex.</td>\n",
       "      <td id=\"T_03ec2_row101_col1\" class=\"data row101 col1\" >What was the conclusion about the difference in Vaporfly effects between men and women?</td>\n",
       "      <td id=\"T_03ec2_row101_col2\" class=\"data row101 col2\" >There is not sufficient evidence to conclude that the effects differ by sex.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row102\" class=\"row_heading level0 row102\" >102</th>\n",
       "      <td id=\"T_03ec2_row102_col0\" class=\"data row102 col0\" >In random effects models such as those we use here, the estimates of the fixed effects, \\(b_{0}\\) and \\(b_{1}\\), are calculated using the generalized least squares criterion. Generalized least squares attempts to triangulate all of the dependencies in the data, for example the fact that there are several performances for each runner and for each race, to arrive at a statistically optimal estimate of the effects. The estimates are linear combinations of the responses, for example</td>\n",
       "      <td id=\"T_03ec2_row102_col1\" class=\"data row102 col1\" >What method is used to calculate the estimates of the fixed effects in random effects models?</td>\n",
       "      <td id=\"T_03ec2_row102_col2\" class=\"data row102 col2\" >Generalized least squares criterion</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row103\" class=\"row_heading level0 row103\" >103</th>\n",
       "      <td id=\"T_03ec2_row103_col0\" class=\"data row103 col0\" >In random effects models such as those we use here, the estimates of the fixed effects, \\(b_{0}\\) and \\(b_{1}\\), are calculated using the generalized least squares criterion. Generalized least squares attempts to triangulate all of the dependencies in the data, for example the fact that there are several performances for each runner and for each race, to arrive at a statistically optimal estimate of the effects. The estimates are linear combinations of the responses, for example</td>\n",
       "      <td id=\"T_03ec2_row103_col1\" class=\"data row103 col1\" >What does generalized least squares do with the data dependencies?</td>\n",
       "      <td id=\"T_03ec2_row103_col2\" class=\"data row103 col2\" >It attempts to triangulate all of the dependencies in the data.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row104\" class=\"row_heading level0 row104\" >104</th>\n",
       "      <td id=\"T_03ec2_row104_col0\" class=\"data row104 col0\" >In random effects models such as those we use here, the estimates of the fixed effects, \\(b_{0}\\) and \\(b_{1}\\), are calculated using the generalized least squares criterion. Generalized least squares attempts to triangulate all of the dependencies in the data, for example the fact that there are several performances for each runner and for each race, to arrive at a statistically optimal estimate of the effects. The estimates are linear combinations of the responses, for example</td>\n",
       "      <td id=\"T_03ec2_row104_col1\" class=\"data row104 col1\" >What are the fixed effects estimates expressed as?</td>\n",
       "      <td id=\"T_03ec2_row104_col2\" class=\"data row104 col2\" >Linear combinations of the responses</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row105\" class=\"row_heading level0 row105\" >105</th>\n",
       "      <td id=\"T_03ec2_row105_col0\" class=\"data row105 col0\" >where \\(c_{1},\\ldots,c_{n}\\) are coefficients calculated using the covariance matrix of the random effects model. These coefficients can sometimes have seemingly counter-intuitive values. In the spirit of attempting to make sense of the magic of generalized least squares, and to promote its utility for this type of problem, we plot the coefficients for the estimates of the Vaporfly effects in Figure 2.</td>\n",
       "      <td id=\"T_03ec2_row105_col1\" class=\"data row105 col1\" >What are \\(c_{1},\\ldots,c_{n}\\)?</td>\n",
       "      <td id=\"T_03ec2_row105_col2\" class=\"data row105 col2\" >Coefficients calculated using the covariance matrix of the random effects model.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row106\" class=\"row_heading level0 row106\" >106</th>\n",
       "      <td id=\"T_03ec2_row106_col0\" class=\"data row106 col0\" >where \\(c_{1},\\ldots,c_{n}\\) are coefficients calculated using the covariance matrix of the random effects model. These coefficients can sometimes have seemingly counter-intuitive values. In the spirit of attempting to make sense of the magic of generalized least squares, and to promote its utility for this type of problem, we plot the coefficients for the estimates of the Vaporfly effects in Figure 2.</td>\n",
       "      <td id=\"T_03ec2_row106_col1\" class=\"data row106 col1\" >How can the values of the coefficients appear?</td>\n",
       "      <td id=\"T_03ec2_row106_col2\" class=\"data row106 col2\" >Seemingly counter-intuitive.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row107\" class=\"row_heading level0 row107\" >107</th>\n",
       "      <td id=\"T_03ec2_row107_col0\" class=\"data row107 col0\" >where \\(c_{1},\\ldots,c_{n}\\) are coefficients calculated using the covariance matrix of the random effects model. These coefficients can sometimes have seemingly counter-intuitive values. In the spirit of attempting to make sense of the magic of generalized least squares, and to promote its utility for this type of problem, we plot the coefficients for the estimates of the Vaporfly effects in Figure 2.</td>\n",
       "      <td id=\"T_03ec2_row107_col1\" class=\"data row107 col1\" >What is plotted in Figure 2?</td>\n",
       "      <td id=\"T_03ec2_row107_col2\" class=\"data row107 col2\" >The coefficients for the estimates of the Vaporfly effects.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row108\" class=\"row_heading level0 row108\" >108</th>\n",
       "      <td id=\"T_03ec2_row108_col0\" class=\"data row108 col0\" >To help make sense of why generalized least squares picks these coefficients, consider four performances</td>\n",
       "      <td id=\"T_03ec2_row108_col1\" class=\"data row108 col1\" >What method is used to pick coefficients?</td>\n",
       "      <td id=\"T_03ec2_row108_col2\" class=\"data row108 col2\" >Generalized least squares</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row109\" class=\"row_heading level0 row109\" >109</th>\n",
       "      <td id=\"T_03ec2_row109_col0\" class=\"data row109 col0\" >To help make sense of why generalized least squares picks these coefficients, consider four performances</td>\n",
       "      <td id=\"T_03ec2_row109_col1\" class=\"data row109 col1\" >How many performances are considered in the context?</td>\n",
       "      <td id=\"T_03ec2_row109_col2\" class=\"data row109 col2\" >Four</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row110\" class=\"row_heading level0 row110\" >110</th>\n",
       "      <td id=\"T_03ec2_row110_col0\" class=\"data row110 col0\" >To help make sense of why generalized least squares picks these coefficients, consider four performances</td>\n",
       "      <td id=\"T_03ec2_row110_col1\" class=\"data row110 col1\" >What is the purpose of considering the performances?</td>\n",
       "      <td id=\"T_03ec2_row110_col2\" class=\"data row110 col2\" >To make sense of why generalized least squares picks these coefficients</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row111\" class=\"row_heading level0 row111\" >111</th>\n",
       "      <td id=\"T_03ec2_row111_col0\" class=\"data row111 col0\" >\\[\\begin{array}{l}y_{1}=\\mbox{Time for Runner 1 at Boston Marathon 2016}\\\\ y_{2}=\\mbox{Time for Runner 2 at Boston Marathon 2016}\\\\ y_{3}=\\mbox{Time for Runner 1 at Chicago Marathon 2017}\\\\ y_{4}=\\mbox{Time for Runner 2 at Chicago Marathon 2017}\\end{array}\\]</td>\n",
       "      <td id=\"T_03ec2_row111_col1\" class=\"data row111 col1\" >What does y1 represent?</td>\n",
       "      <td id=\"T_03ec2_row111_col2\" class=\"data row111 col2\" >Time for Runner 1 at Boston Marathon 2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row112\" class=\"row_heading level0 row112\" >112</th>\n",
       "      <td id=\"T_03ec2_row112_col0\" class=\"data row112 col0\" >\\[\\begin{array}{l}y_{1}=\\mbox{Time for Runner 1 at Boston Marathon 2016}\\\\ y_{2}=\\mbox{Time for Runner 2 at Boston Marathon 2016}\\\\ y_{3}=\\mbox{Time for Runner 1 at Chicago Marathon 2017}\\\\ y_{4}=\\mbox{Time for Runner 2 at Chicago Marathon 2017}\\end{array}\\]</td>\n",
       "      <td id=\"T_03ec2_row112_col1\" class=\"data row112 col1\" >What event does y3 correspond to?</td>\n",
       "      <td id=\"T_03ec2_row112_col2\" class=\"data row112 col2\" >Chicago Marathon 2017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row113\" class=\"row_heading level0 row113\" >113</th>\n",
       "      <td id=\"T_03ec2_row113_col0\" class=\"data row113 col0\" >\\[\\begin{array}{l}y_{1}=\\mbox{Time for Runner 1 at Boston Marathon 2016}\\\\ y_{2}=\\mbox{Time for Runner 2 at Boston Marathon 2016}\\\\ y_{3}=\\mbox{Time for Runner 1 at Chicago Marathon 2017}\\\\ y_{4}=\\mbox{Time for Runner 2 at Chicago Marathon 2017}\\end{array}\\]</td>\n",
       "      <td id=\"T_03ec2_row113_col1\" class=\"data row113 col1\" >Who does y2 refer to?</td>\n",
       "      <td id=\"T_03ec2_row113_col2\" class=\"data row113 col2\" >Runner 2 at Boston Marathon 2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row114\" class=\"row_heading level0 row114\" >114</th>\n",
       "      <td id=\"T_03ec2_row114_col0\" class=\"data row114 col0\" >\\begin{table}\n",
       "\\begin{tabular}{c c c c} men minutes & women minutes & men log minutes & women log minutes \\\\ \\hline \\((-3.933,-1.959)\\) & \\((-3.514,-0.847)\\) & \\((-0.028,-0.014)\\) & \\((-0.022,-0.006)\\) \\\\ \\end{tabular}\n",
       "\\end{table}\n",
       "Table 2: 90% confidence intervals for Vaporfly effects in each model.</td>\n",
       "      <td id=\"T_03ec2_row114_col1\" class=\"data row114 col1\" >What is the 90% confidence interval for Vaporfly effects on men's minutes?</td>\n",
       "      <td id=\"T_03ec2_row114_col2\" class=\"data row114 col2\" >(-3.933, -1.959)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row115\" class=\"row_heading level0 row115\" >115</th>\n",
       "      <td id=\"T_03ec2_row115_col0\" class=\"data row115 col0\" >\\begin{table}\n",
       "\\begin{tabular}{c c c c} men minutes & women minutes & men log minutes & women log minutes \\\\ \\hline \\((-3.933,-1.959)\\) & \\((-3.514,-0.847)\\) & \\((-0.028,-0.014)\\) & \\((-0.022,-0.006)\\) \\\\ \\end{tabular}\n",
       "\\end{table}\n",
       "Table 2: 90% confidence intervals for Vaporfly effects in each model.</td>\n",
       "      <td id=\"T_03ec2_row115_col1\" class=\"data row115 col1\" >What is the 90% confidence interval for Vaporfly effects on women's log minutes?</td>\n",
       "      <td id=\"T_03ec2_row115_col2\" class=\"data row115 col2\" >(-0.022, -0.006)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row116\" class=\"row_heading level0 row116\" >116</th>\n",
       "      <td id=\"T_03ec2_row116_col0\" class=\"data row116 col0\" >\\begin{table}\n",
       "\\begin{tabular}{c c c c} men minutes & women minutes & men log minutes & women log minutes \\\\ \\hline \\((-3.933,-1.959)\\) & \\((-3.514,-0.847)\\) & \\((-0.028,-0.014)\\) & \\((-0.022,-0.006)\\) \\\\ \\end{tabular}\n",
       "\\end{table}\n",
       "Table 2: 90% confidence intervals for Vaporfly effects in each model.</td>\n",
       "      <td id=\"T_03ec2_row116_col1\" class=\"data row116 col1\" >What does Table 2 represent?</td>\n",
       "      <td id=\"T_03ec2_row116_col2\" class=\"data row116 col2\" >90% confidence intervals for Vaporfly effects in each model.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row117\" class=\"row_heading level0 row117\" >117</th>\n",
       "      <td id=\"T_03ec2_row117_col0\" class=\"data row117 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row117_col1\" class=\"data row117 col1\" >What is Mount Everest known for?</td>\n",
       "      <td id=\"T_03ec2_row117_col2\" class=\"data row117 col2\" >It is Earth's highest mountain above sea level.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row118\" class=\"row_heading level0 row118\" >118</th>\n",
       "      <td id=\"T_03ec2_row118_col0\" class=\"data row118 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row118_col1\" class=\"data row118 col1\" >Where is Mount Everest located?</td>\n",
       "      <td id=\"T_03ec2_row118_col2\" class=\"data row118 col2\" >In the Mahalangur Himal sub-range of the Himalayas.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row119\" class=\"row_heading level0 row119\" >119</th>\n",
       "      <td id=\"T_03ec2_row119_col0\" class=\"data row119 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row119_col1\" class=\"data row119 col1\" >Which two countries does the international border run between at Mount Everest's summit?</td>\n",
       "      <td id=\"T_03ec2_row119_col2\" class=\"data row119 col2\" >China (Tibet Autonomous Region) and Nepal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row120\" class=\"row_heading level0 row120\" >120</th>\n",
       "      <td id=\"T_03ec2_row120_col0\" class=\"data row120 col0\" >The first runner (\\(y_{1}\\) and \\(y_{3}\\)) did not wear Vaporflys, but the second runner (\\(y_{2}\\) and \\(y_{4}\\)) switched to Vaporflys at Chicago in 2017. A reasonable estimate for the Vaporfly effect from these data might be the average of the Vaporfly performances minus the average of the non-Vaporfly performances,</td>\n",
       "      <td id=\"T_03ec2_row120_col1\" class=\"data row120 col1\" >What did the second runner switch to at Chicago in 2017?</td>\n",
       "      <td id=\"T_03ec2_row120_col2\" class=\"data row120 col2\" >Vaporflys</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row121\" class=\"row_heading level0 row121\" >121</th>\n",
       "      <td id=\"T_03ec2_row121_col0\" class=\"data row121 col0\" >The first runner (\\(y_{1}\\) and \\(y_{3}\\)) did not wear Vaporflys, but the second runner (\\(y_{2}\\) and \\(y_{4}\\)) switched to Vaporflys at Chicago in 2017. A reasonable estimate for the Vaporfly effect from these data might be the average of the Vaporfly performances minus the average of the non-Vaporfly performances,</td>\n",
       "      <td id=\"T_03ec2_row121_col1\" class=\"data row121 col1\" >Did the first runner wear Vaporflys?</td>\n",
       "      <td id=\"T_03ec2_row121_col2\" class=\"data row121 col2\" >No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row122\" class=\"row_heading level0 row122\" >122</th>\n",
       "      <td id=\"T_03ec2_row122_col0\" class=\"data row122 col0\" >The first runner (\\(y_{1}\\) and \\(y_{3}\\)) did not wear Vaporflys, but the second runner (\\(y_{2}\\) and \\(y_{4}\\)) switched to Vaporflys at Chicago in 2017. A reasonable estimate for the Vaporfly effect from these data might be the average of the Vaporfly performances minus the average of the non-Vaporfly performances,</td>\n",
       "      <td id=\"T_03ec2_row122_col1\" class=\"data row122 col1\" >How might one estimate the Vaporfly effect from the data?</td>\n",
       "      <td id=\"T_03ec2_row122_col2\" class=\"data row122 col2\" >By calculating the average of the Vaporfly performances minus the average of the non-Vaporfly performances</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row123\" class=\"row_heading level0 row123\" >123</th>\n",
       "      <td id=\"T_03ec2_row123_col0\" class=\"data row123 col0\" >which places a positive coefficient (1.0) on the Vaporfly performance and negative coefficients (-0.33) on the non-Vaporfly performances. However, a better estimate would consider how much the second athlete's advantage increased after switching to the Vaporflys</td>\n",
       "      <td id=\"T_03ec2_row123_col1\" class=\"data row123 col1\" >What coefficient is placed on the Vaporfly performance?</td>\n",
       "      <td id=\"T_03ec2_row123_col2\" class=\"data row123 col2\" >Positive coefficient (1.0)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row124\" class=\"row_heading level0 row124\" >124</th>\n",
       "      <td id=\"T_03ec2_row124_col0\" class=\"data row124 col0\" >which places a positive coefficient (1.0) on the Vaporfly performance and negative coefficients (-0.33) on the non-Vaporfly performances. However, a better estimate would consider how much the second athlete's advantage increased after switching to the Vaporflys</td>\n",
       "      <td id=\"T_03ec2_row124_col1\" class=\"data row124 col1\" >What coefficients are placed on the non-Vaporfly performances?</td>\n",
       "      <td id=\"T_03ec2_row124_col2\" class=\"data row124 col2\" >Negative coefficients (-0.33)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row125\" class=\"row_heading level0 row125\" >125</th>\n",
       "      <td id=\"T_03ec2_row125_col0\" class=\"data row125 col0\" >which places a positive coefficient (1.0) on the Vaporfly performance and negative coefficients (-0.33) on the non-Vaporfly performances. However, a better estimate would consider how much the second athlete's advantage increased after switching to the Vaporflys</td>\n",
       "      <td id=\"T_03ec2_row125_col1\" class=\"data row125 col1\" >What would a better estimate consider regarding the Vaporflys?</td>\n",
       "      <td id=\"T_03ec2_row125_col2\" class=\"data row125 col2\" >How much the second athlete's advantage increased after switching to the Vaporflys</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row126\" class=\"row_heading level0 row126\" >126</th>\n",
       "      <td id=\"T_03ec2_row126_col0\" class=\"data row126 col0\" >The first difference (\\(y_{4}-y_{3}\\)) measures how much better the second athlete did in Chicago (wearing Vaporflys), and the second difference is how much better the second athlete did in Boston (not wearing Vaporflys). This estimate places a positive coefficient on the second runner's Vaporfly performance in Chicago, a positive coefficient on the first runner's Boston performance, a negative coefficient on the second runner's Boston performance, and a negative coefficient on the first runner's Chicago performance.</td>\n",
       "      <td id=\"T_03ec2_row126_col1\" class=\"data row126 col1\" >What does the first difference (\\(y_{4}-y_{3}\\)) represent?</td>\n",
       "      <td id=\"T_03ec2_row126_col2\" class=\"data row126 col2\" >It measures how much better the second athlete performed in Chicago while wearing Vaporflys.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row127\" class=\"row_heading level0 row127\" >127</th>\n",
       "      <td id=\"T_03ec2_row127_col0\" class=\"data row127 col0\" >The first difference (\\(y_{4}-y_{3}\\)) measures how much better the second athlete did in Chicago (wearing Vaporflys), and the second difference is how much better the second athlete did in Boston (not wearing Vaporflys). This estimate places a positive coefficient on the second runner's Vaporfly performance in Chicago, a positive coefficient on the first runner's Boston performance, a negative coefficient on the second runner's Boston performance, and a negative coefficient on the first runner's Chicago performance.</td>\n",
       "      <td id=\"T_03ec2_row127_col1\" class=\"data row127 col1\" >What type of coefficient is placed on the second runner's performance in Chicago?</td>\n",
       "      <td id=\"T_03ec2_row127_col2\" class=\"data row127 col2\" >A positive coefficient.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row128\" class=\"row_heading level0 row128\" >128</th>\n",
       "      <td id=\"T_03ec2_row128_col0\" class=\"data row128 col0\" >The first difference (\\(y_{4}-y_{3}\\)) measures how much better the second athlete did in Chicago (wearing Vaporflys), and the second difference is how much better the second athlete did in Boston (not wearing Vaporflys). This estimate places a positive coefficient on the second runner's Vaporfly performance in Chicago, a positive coefficient on the first runner's Boston performance, a negative coefficient on the second runner's Boston performance, and a negative coefficient on the first runner's Chicago performance.</td>\n",
       "      <td id=\"T_03ec2_row128_col1\" class=\"data row128 col1\" >Which performance has a negative coefficient for the first runner?</td>\n",
       "      <td id=\"T_03ec2_row128_col2\" class=\"data row128 col2\" >The first runner's Chicago performance.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row129\" class=\"row_heading level0 row129\" >129</th>\n",
       "      <td id=\"T_03ec2_row129_col0\" class=\"data row129 col0\" >This pattern can be observed in Figure 2; early performances--before the Vaporfly appeared on the market--have either positive or negative coefficients, whereas later performances generally have positive coefficients when the Vaporfly is worn and negative coefficients when not worn. There is some variation in the magnitude of these coefficients, which we expect is due to the differing number of performances from each runner and from each race.</td>\n",
       "      <td id=\"T_03ec2_row129_col1\" class=\"data row129 col1\" >What is the general trend of coefficients for performances after the Vaporfly was introduced?</td>\n",
       "      <td id=\"T_03ec2_row129_col2\" class=\"data row129 col2\" >Generally positive coefficients when the Vaporfly is worn.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row130\" class=\"row_heading level0 row130\" >130</th>\n",
       "      <td id=\"T_03ec2_row130_col0\" class=\"data row130 col0\" >This pattern can be observed in Figure 2; early performances--before the Vaporfly appeared on the market--have either positive or negative coefficients, whereas later performances generally have positive coefficients when the Vaporfly is worn and negative coefficients when not worn. There is some variation in the magnitude of these coefficients, which we expect is due to the differing number of performances from each runner and from each race.</td>\n",
       "      <td id=\"T_03ec2_row130_col1\" class=\"data row130 col1\" >What kind of coefficients were observed in early performances before the Vaporfly was available?</td>\n",
       "      <td id=\"T_03ec2_row130_col2\" class=\"data row130 col2\" >Either positive or negative coefficients.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row131\" class=\"row_heading level0 row131\" >131</th>\n",
       "      <td id=\"T_03ec2_row131_col0\" class=\"data row131 col0\" >This pattern can be observed in Figure 2; early performances--before the Vaporfly appeared on the market--have either positive or negative coefficients, whereas later performances generally have positive coefficients when the Vaporfly is worn and negative coefficients when not worn. There is some variation in the magnitude of these coefficients, which we expect is due to the differing number of performances from each runner and from each race.</td>\n",
       "      <td id=\"T_03ec2_row131_col1\" class=\"data row131 col1\" >What is the expected reason for the variation in the magnitude of coefficients?</td>\n",
       "      <td id=\"T_03ec2_row131_col2\" class=\"data row131 col2\" >The differing number of performances from each runner and from each race.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row132\" class=\"row_heading level0 row132\" >132</th>\n",
       "      <td id=\"T_03ec2_row132_col0\" class=\"data row132 col0\" >Figure 2: Coefficients for the generalized least squares estimate of the Vaporfly effect \\(\\widehat{b}_{1}\\) for the untransformed data. Each point represents an individual performance \\(y_{i}\\). The height of the point is the corresponding coefficient \\(c_{i}\\).</td>\n",
       "      <td id=\"T_03ec2_row132_col1\" class=\"data row132 col1\" >What does each point on Figure 2 represent?</td>\n",
       "      <td id=\"T_03ec2_row132_col2\" class=\"data row132 col2\" >An individual performance \\(y_{i}\\)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row133\" class=\"row_heading level0 row133\" >133</th>\n",
       "      <td id=\"T_03ec2_row133_col0\" class=\"data row133 col0\" >Figure 2: Coefficients for the generalized least squares estimate of the Vaporfly effect \\(\\widehat{b}_{1}\\) for the untransformed data. Each point represents an individual performance \\(y_{i}\\). The height of the point is the corresponding coefficient \\(c_{i}\\).</td>\n",
       "      <td id=\"T_03ec2_row133_col1\" class=\"data row133 col1\" >What does the height of each point on Figure 2 indicate?</td>\n",
       "      <td id=\"T_03ec2_row133_col2\" class=\"data row133 col2\" >The corresponding coefficient \\(c_{i}\\)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row134\" class=\"row_heading level0 row134\" >134</th>\n",
       "      <td id=\"T_03ec2_row134_col0\" class=\"data row134 col0\" >Figure 2: Coefficients for the generalized least squares estimate of the Vaporfly effect \\(\\widehat{b}_{1}\\) for the untransformed data. Each point represents an individual performance \\(y_{i}\\). The height of the point is the corresponding coefficient \\(c_{i}\\).</td>\n",
       "      <td id=\"T_03ec2_row134_col1\" class=\"data row134 col1\" >What is being estimated in Figure 2?</td>\n",
       "      <td id=\"T_03ec2_row134_col2\" class=\"data row134 col2\" >The Vaporfly effect \\(\\widehat{b}_{1}\\)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row135\" class=\"row_heading level0 row135\" >135</th>\n",
       "      <td id=\"T_03ec2_row135_col0\" class=\"data row135 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row135_col1\" class=\"data row135 col1\" >What is Mount Everest?</td>\n",
       "      <td id=\"T_03ec2_row135_col2\" class=\"data row135 col2\" >Earth's highest mountain above sea level</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row136\" class=\"row_heading level0 row136\" >136</th>\n",
       "      <td id=\"T_03ec2_row136_col0\" class=\"data row136 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row136_col1\" class=\"data row136 col1\" >Where is Mount Everest located?</td>\n",
       "      <td id=\"T_03ec2_row136_col2\" class=\"data row136 col2\" >In the Mahalangur Himal sub-range of the Himalayas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row137\" class=\"row_heading level0 row137\" >137</th>\n",
       "      <td id=\"T_03ec2_row137_col0\" class=\"data row137 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row137_col1\" class=\"data row137 col1\" >What runs across the summit point of Mount Everest?</td>\n",
       "      <td id=\"T_03ec2_row137_col2\" class=\"data row137 col2\" >The international border between China (Tibet Autonomous Region) and Nepal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row138\" class=\"row_heading level0 row138\" >138</th>\n",
       "      <td id=\"T_03ec2_row138_col0\" class=\"data row138 col0\" >Note that Figure 2 shows the coefficients in the generalized least squares estimate, not raw performances.</td>\n",
       "      <td id=\"T_03ec2_row138_col1\" class=\"data row138 col1\" >What does Figure 2 display?</td>\n",
       "      <td id=\"T_03ec2_row138_col2\" class=\"data row138 col2\" >The coefficients in the generalized least squares estimate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row139\" class=\"row_heading level0 row139\" >139</th>\n",
       "      <td id=\"T_03ec2_row139_col0\" class=\"data row139 col0\" >Note that Figure 2 shows the coefficients in the generalized least squares estimate, not raw performances.</td>\n",
       "      <td id=\"T_03ec2_row139_col1\" class=\"data row139 col1\" >What does Figure 2 not show?</td>\n",
       "      <td id=\"T_03ec2_row139_col2\" class=\"data row139 col2\" >Raw performances</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row140\" class=\"row_heading level0 row140\" >140</th>\n",
       "      <td id=\"T_03ec2_row140_col0\" class=\"data row140 col0\" >Note that Figure 2 shows the coefficients in the generalized least squares estimate, not raw performances.</td>\n",
       "      <td id=\"T_03ec2_row140_col1\" class=\"data row140 col1\" >What type of estimate is shown in Figure 2?</td>\n",
       "      <td id=\"T_03ec2_row140_col2\" class=\"data row140 col2\" >Generalized least squares estimate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row141\" class=\"row_heading level0 row141\" >141</th>\n",
       "      <td id=\"T_03ec2_row141_col0\" class=\"data row141 col0\" >Mount Everest is known as the highest mountain in the world with its peak at 8,848 meters above sea level. It is located in the Himalayas on the border between Nepal and the Tibet Autonomous Region of China. The first confirmed ascent was made by Sir Edmund Hillary and Tenzing Norgay on May 29, 1953.</td>\n",
       "      <td id=\"T_03ec2_row141_col1\" class=\"data row141 col1\" >What is Mount Everest known for?</td>\n",
       "      <td id=\"T_03ec2_row141_col2\" class=\"data row141 col2\" >Being the highest mountain in the world</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row142\" class=\"row_heading level0 row142\" >142</th>\n",
       "      <td id=\"T_03ec2_row142_col0\" class=\"data row142 col0\" >Mount Everest is known as the highest mountain in the world with its peak at 8,848 meters above sea level. It is located in the Himalayas on the border between Nepal and the Tibet Autonomous Region of China. The first confirmed ascent was made by Sir Edmund Hillary and Tenzing Norgay on May 29, 1953.</td>\n",
       "      <td id=\"T_03ec2_row142_col1\" class=\"data row142 col1\" >What is the height of Mount Everest's peak?</td>\n",
       "      <td id=\"T_03ec2_row142_col2\" class=\"data row142 col2\" >8,848 meters above sea level</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row143\" class=\"row_heading level0 row143\" >143</th>\n",
       "      <td id=\"T_03ec2_row143_col0\" class=\"data row143 col0\" >Mount Everest is known as the highest mountain in the world with its peak at 8,848 meters above sea level. It is located in the Himalayas on the border between Nepal and the Tibet Autonomous Region of China. The first confirmed ascent was made by Sir Edmund Hillary and Tenzing Norgay on May 29, 1953.</td>\n",
       "      <td id=\"T_03ec2_row143_col1\" class=\"data row143 col1\" >Who were the first confirmed climbers to reach the summit of Mount Everest?</td>\n",
       "      <td id=\"T_03ec2_row143_col2\" class=\"data row143 col2\" >Sir Edmund Hillary and Tenzing Norgay</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row144\" class=\"row_heading level0 row144\" >144</th>\n",
       "      <td id=\"T_03ec2_row144_col0\" class=\"data row144 col0\" >By collecting data on marathon times and identifying shoes worn in a systematic sample of elite and sub-elite marathon runners, we studied how much a runner's marathon time can be expected to improve after switching to Vaporfly shoes. For men, the improvement is most likely somewhere between 2.0 and 3.9 minutes, or between 1.4% and 2.8%. For women it is likely between 0.8 and 3.5 minutes, or between 0.6% and 2.2%. To put these numbers into perspective, elite marathon runners cover more than half a mile in 3 minutes.</td>\n",
       "      <td id=\"T_03ec2_row144_col1\" class=\"data row144 col1\" >What is the likely improvement in marathon time for men after switching to Vaporfly shoes?</td>\n",
       "      <td id=\"T_03ec2_row144_col2\" class=\"data row144 col2\" >Between 2.0 and 3.9 minutes, or between 1.4% and 2.8%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row145\" class=\"row_heading level0 row145\" >145</th>\n",
       "      <td id=\"T_03ec2_row145_col0\" class=\"data row145 col0\" >By collecting data on marathon times and identifying shoes worn in a systematic sample of elite and sub-elite marathon runners, we studied how much a runner's marathon time can be expected to improve after switching to Vaporfly shoes. For men, the improvement is most likely somewhere between 2.0 and 3.9 minutes, or between 1.4% and 2.8%. For women it is likely between 0.8 and 3.5 minutes, or between 0.6% and 2.2%. To put these numbers into perspective, elite marathon runners cover more than half a mile in 3 minutes.</td>\n",
       "      <td id=\"T_03ec2_row145_col1\" class=\"data row145 col1\" >How much can women's marathon time improve by using Vaporfly shoes?</td>\n",
       "      <td id=\"T_03ec2_row145_col2\" class=\"data row145 col2\" >Between 0.8 and 3.5 minutes, or between 0.6% and 2.2%</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row146\" class=\"row_heading level0 row146\" >146</th>\n",
       "      <td id=\"T_03ec2_row146_col0\" class=\"data row146 col0\" >By collecting data on marathon times and identifying shoes worn in a systematic sample of elite and sub-elite marathon runners, we studied how much a runner's marathon time can be expected to improve after switching to Vaporfly shoes. For men, the improvement is most likely somewhere between 2.0 and 3.9 minutes, or between 1.4% and 2.8%. For women it is likely between 0.8 and 3.5 minutes, or between 0.6% and 2.2%. To put these numbers into perspective, elite marathon runners cover more than half a mile in 3 minutes.</td>\n",
       "      <td id=\"T_03ec2_row146_col1\" class=\"data row146 col1\" >What distance can elite marathon runners cover in 3 minutes?</td>\n",
       "      <td id=\"T_03ec2_row146_col2\" class=\"data row146 col2\" >More than half a mile</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row147\" class=\"row_heading level0 row147\" >147</th>\n",
       "      <td id=\"T_03ec2_row147_col0\" class=\"data row147 col0\" >We made several assumptions that we believe to be reasonable, but nevertheless are open for debate and could be refined. We assumed that the expected Vaporfly effect (\\(b_{1}\\)) is the same for every runner of the same sex. Including prototypes, there are several different versions of Vaporflys, and it is logical to expect that newer versions improve upon older versions. Moreover, depending on biomechanical factors, some runners may benefit from Vaporflys more than others, or could have been wearing shoes that were not optimal for them before switching to Vaporflys. The models assume that, conditional on the runner and race, the marathon time follows a normal distribution. This may not be entirely appropriate because we believe that a runner is more likely to run 5 minutes slower than expected rather than 5 minutes faster; when things go wrong in a marathon, they can go really wrong.</td>\n",
       "      <td id=\"T_03ec2_row147_col1\" class=\"data row147 col1\" >What is the assumed effect of Vaporflys on runners?</td>\n",
       "      <td id=\"T_03ec2_row147_col2\" class=\"data row147 col2\" >The assumed effect of Vaporflys is that it is the same for every runner of the same sex.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row148\" class=\"row_heading level0 row148\" >148</th>\n",
       "      <td id=\"T_03ec2_row148_col0\" class=\"data row148 col0\" >We made several assumptions that we believe to be reasonable, but nevertheless are open for debate and could be refined. We assumed that the expected Vaporfly effect (\\(b_{1}\\)) is the same for every runner of the same sex. Including prototypes, there are several different versions of Vaporflys, and it is logical to expect that newer versions improve upon older versions. Moreover, depending on biomechanical factors, some runners may benefit from Vaporflys more than others, or could have been wearing shoes that were not optimal for them before switching to Vaporflys. The models assume that, conditional on the runner and race, the marathon time follows a normal distribution. This may not be entirely appropriate because we believe that a runner is more likely to run 5 minutes slower than expected rather than 5 minutes faster; when things go wrong in a marathon, they can go really wrong.</td>\n",
       "      <td id=\"T_03ec2_row148_col1\" class=\"data row148 col1\" >Why might the assumption of marathon time following a normal distribution be questionable?</td>\n",
       "      <td id=\"T_03ec2_row148_col2\" class=\"data row148 col2\" >The assumption may be questionable because it is believed that a runner is more likely to run 5 minutes slower than expected rather than 5 minutes faster, indicating that marathon performance can vary significantly when problems occur.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row149\" class=\"row_heading level0 row149\" >149</th>\n",
       "      <td id=\"T_03ec2_row149_col0\" class=\"data row149 col0\" >We made several assumptions that we believe to be reasonable, but nevertheless are open for debate and could be refined. We assumed that the expected Vaporfly effect (\\(b_{1}\\)) is the same for every runner of the same sex. Including prototypes, there are several different versions of Vaporflys, and it is logical to expect that newer versions improve upon older versions. Moreover, depending on biomechanical factors, some runners may benefit from Vaporflys more than others, or could have been wearing shoes that were not optimal for them before switching to Vaporflys. The models assume that, conditional on the runner and race, the marathon time follows a normal distribution. This may not be entirely appropriate because we believe that a runner is more likely to run 5 minutes slower than expected rather than 5 minutes faster; when things go wrong in a marathon, they can go really wrong.</td>\n",
       "      <td id=\"T_03ec2_row149_col1\" class=\"data row149 col1\" >What factors might influence how much a runner benefits from Vaporflys?</td>\n",
       "      <td id=\"T_03ec2_row149_col2\" class=\"data row149 col2\" >Biomechanical factors might influence how much a runner benefits from Vaporflys, as well as whether they had been wearing suboptimal shoes before switching to Vaporflys.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row150\" class=\"row_heading level0 row150\" >150</th>\n",
       "      <td id=\"T_03ec2_row150_col0\" class=\"data row150 col0\" >The study did not include data on runners that did not finish their races. This data is more difficult to obtain and more difficult to model. Based on the Wired study (Thompson, 2017), which found that people wearing Vaporflys generally ran better in the second half of the race, we believe that people wearing Vaporflys are less likely to drop out, so we are more likely to miss the very worst performances in non-Vaporfly shoes. Thus, we believe that including drop-outs would only strengthen our estimate of the Vaporfly effect.</td>\n",
       "      <td id=\"T_03ec2_row150_col1\" class=\"data row150 col1\" >Why were runners who did not finish their races not included in the study?</td>\n",
       "      <td id=\"T_03ec2_row150_col2\" class=\"data row150 col2\" >The data on runners that did not finish their races is more difficult to obtain and model.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row151\" class=\"row_heading level0 row151\" >151</th>\n",
       "      <td id=\"T_03ec2_row151_col0\" class=\"data row151 col0\" >The study did not include data on runners that did not finish their races. This data is more difficult to obtain and more difficult to model. Based on the Wired study (Thompson, 2017), which found that people wearing Vaporflys generally ran better in the second half of the race, we believe that people wearing Vaporflys are less likely to drop out, so we are more likely to miss the very worst performances in non-Vaporfly shoes. Thus, we believe that including drop-outs would only strengthen our estimate of the Vaporfly effect.</td>\n",
       "      <td id=\"T_03ec2_row151_col1\" class=\"data row151 col1\" >What did the Wired study (Thompson, 2017) find about runners wearing Vaporflys?</td>\n",
       "      <td id=\"T_03ec2_row151_col2\" class=\"data row151 col2\" >The study found that people wearing Vaporflys generally ran better in the second half of the race.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row152\" class=\"row_heading level0 row152\" >152</th>\n",
       "      <td id=\"T_03ec2_row152_col0\" class=\"data row152 col0\" >The study did not include data on runners that did not finish their races. This data is more difficult to obtain and more difficult to model. Based on the Wired study (Thompson, 2017), which found that people wearing Vaporflys generally ran better in the second half of the race, we believe that people wearing Vaporflys are less likely to drop out, so we are more likely to miss the very worst performances in non-Vaporfly shoes. Thus, we believe that including drop-outs would only strengthen our estimate of the Vaporfly effect.</td>\n",
       "      <td id=\"T_03ec2_row152_col1\" class=\"data row152 col1\" >What is the expected impact of including drop-out data on the Vaporfly effect estimate?</td>\n",
       "      <td id=\"T_03ec2_row152_col2\" class=\"data row152 col2\" >Including drop-outs is believed to only strengthen the estimate of the Vaporfly effect.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row153\" class=\"row_heading level0 row153\" >153</th>\n",
       "      <td id=\"T_03ec2_row153_col0\" class=\"data row153 col0\" >The shoes were identified via a manual process of searching through photographs. We believe that this manual process for getting the shoe data is better (though labor intensive) than relying on self-reported shoes. Nonetheless, it is prone to error in misidentification of the person and the shoes. For example, Nike has an orange Vaporfly with a black Swoosh logo, and also a Zoom Fly with the same color scheme. Another example, some professional athletes ran in a neon yellow and pink prototype Vaporfly, which looks very similar to the neon yellow and pink Nike Zoom Streak 6. See Johnson (2020) for a more detailed analysis. Yet a further example, we identified one athlete who attempted to conceal the identity of his shoes by coloring in the white Swoosh on a blue pair of Vaporflys.</td>\n",
       "      <td id=\"T_03ec2_row153_col1\" class=\"data row153 col1\" >What method was used to identify the shoes?</td>\n",
       "      <td id=\"T_03ec2_row153_col2\" class=\"data row153 col2\" >A manual process of searching through photographs.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row154\" class=\"row_heading level0 row154\" >154</th>\n",
       "      <td id=\"T_03ec2_row154_col0\" class=\"data row154 col0\" >The shoes were identified via a manual process of searching through photographs. We believe that this manual process for getting the shoe data is better (though labor intensive) than relying on self-reported shoes. Nonetheless, it is prone to error in misidentification of the person and the shoes. For example, Nike has an orange Vaporfly with a black Swoosh logo, and also a Zoom Fly with the same color scheme. Another example, some professional athletes ran in a neon yellow and pink prototype Vaporfly, which looks very similar to the neon yellow and pink Nike Zoom Streak 6. See Johnson (2020) for a more detailed analysis. Yet a further example, we identified one athlete who attempted to conceal the identity of his shoes by coloring in the white Swoosh on a blue pair of Vaporflys.</td>\n",
       "      <td id=\"T_03ec2_row154_col1\" class=\"data row154 col1\" >Why is the manual process of identifying shoes prone to error?</td>\n",
       "      <td id=\"T_03ec2_row154_col2\" class=\"data row154 col2\" >It is prone to error due to misidentification of the person and the shoes.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row155\" class=\"row_heading level0 row155\" >155</th>\n",
       "      <td id=\"T_03ec2_row155_col0\" class=\"data row155 col0\" >The shoes were identified via a manual process of searching through photographs. We believe that this manual process for getting the shoe data is better (though labor intensive) than relying on self-reported shoes. Nonetheless, it is prone to error in misidentification of the person and the shoes. For example, Nike has an orange Vaporfly with a black Swoosh logo, and also a Zoom Fly with the same color scheme. Another example, some professional athletes ran in a neon yellow and pink prototype Vaporfly, which looks very similar to the neon yellow and pink Nike Zoom Streak 6. See Johnson (2020) for a more detailed analysis. Yet a further example, we identified one athlete who attempted to conceal the identity of his shoes by coloring in the white Swoosh on a blue pair of Vaporflys.</td>\n",
       "      <td id=\"T_03ec2_row155_col1\" class=\"data row155 col1\" >What did one athlete do to conceal the identity of his shoes?</td>\n",
       "      <td id=\"T_03ec2_row155_col2\" class=\"data row155 col2\" >The athlete attempted to conceal the identity of his shoes by coloring in the white Swoosh on a blue pair of Vaporflys.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row156\" class=\"row_heading level0 row156\" >156</th>\n",
       "      <td id=\"T_03ec2_row156_col0\" class=\"data row156 col0\" >Some of the athletes may have competed in marathons not included in the 22 marathons that we sampled. Missing these performances shouldn't bias our results, but our results could be strengthened if we are able to track down every performance from the athletes in the study.</td>\n",
       "      <td id=\"T_03ec2_row156_col1\" class=\"data row156 col1\" >How many marathons were sampled in the study?</td>\n",
       "      <td id=\"T_03ec2_row156_col2\" class=\"data row156 col2\" >22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row157\" class=\"row_heading level0 row157\" >157</th>\n",
       "      <td id=\"T_03ec2_row157_col0\" class=\"data row157 col0\" >Some of the athletes may have competed in marathons not included in the 22 marathons that we sampled. Missing these performances shouldn't bias our results, but our results could be strengthened if we are able to track down every performance from the athletes in the study.</td>\n",
       "      <td id=\"T_03ec2_row157_col1\" class=\"data row157 col1\" >What could strengthen the results of the study?</td>\n",
       "      <td id=\"T_03ec2_row157_col2\" class=\"data row157 col2\" >Tracking down every performance from the athletes in the study.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row158\" class=\"row_heading level0 row158\" >158</th>\n",
       "      <td id=\"T_03ec2_row158_col0\" class=\"data row158 col0\" >Some of the athletes may have competed in marathons not included in the 22 marathons that we sampled. Missing these performances shouldn't bias our results, but our results could be strengthened if we are able to track down every performance from the athletes in the study.</td>\n",
       "      <td id=\"T_03ec2_row158_col1\" class=\"data row158 col1\" >Will missing performances from some athletes bias the results?</td>\n",
       "      <td id=\"T_03ec2_row158_col2\" class=\"data row158 col2\" >No, missing these performances shouldn't bias the results.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row159\" class=\"row_heading level0 row159\" >159</th>\n",
       "      <td id=\"T_03ec2_row159_col0\" class=\"data row159 col0\" >It is possible that athletes are more likely to switch to Vaporfly shoes when they know they are ready to turn in a good marathon performance. Inversely, some athletes might not be</td>\n",
       "      <td id=\"T_03ec2_row159_col1\" class=\"data row159 col1\" >Why might athletes switch to Vaporfly shoes?</td>\n",
       "      <td id=\"T_03ec2_row159_col2\" class=\"data row159 col2\" >Because they believe they are ready to turn in a good marathon performance.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row160\" class=\"row_heading level0 row160\" >160</th>\n",
       "      <td id=\"T_03ec2_row160_col0\" class=\"data row160 col0\" >It is possible that athletes are more likely to switch to Vaporfly shoes when they know they are ready to turn in a good marathon performance. Inversely, some athletes might not be</td>\n",
       "      <td id=\"T_03ec2_row160_col1\" class=\"data row160 col1\" >What is the possible relationship between athletes' performance and their choice of shoes?</td>\n",
       "      <td id=\"T_03ec2_row160_col2\" class=\"data row160 col2\" >Athletes are more likely to choose Vaporfly shoes when they anticipate a good performance.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row161\" class=\"row_heading level0 row161\" >161</th>\n",
       "      <td id=\"T_03ec2_row161_col0\" class=\"data row161 col0\" >It is possible that athletes are more likely to switch to Vaporfly shoes when they know they are ready to turn in a good marathon performance. Inversely, some athletes might not be</td>\n",
       "      <td id=\"T_03ec2_row161_col1\" class=\"data row161 col1\" >What is left incomplete in the context about some athletes?</td>\n",
       "      <td id=\"T_03ec2_row161_col2\" class=\"data row161 col2\" >The context does not finish explaining the inverse situation regarding athletes and Vaporfly shoes.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row162\" class=\"row_heading level0 row162\" >162</th>\n",
       "      <td id=\"T_03ec2_row162_col0\" class=\"data row162 col0\" >Willing to pay $250 for shoes when they are out of shape. The Upshot study investigated this possibility by controlling for training volume; they did not see substantially different results. Further, our sample consists of solely highly accomplished runners. We believe that these athletes are generally wearing the best shoes available to them whenever they run a marathon.</td>\n",
       "      <td id=\"T_03ec2_row162_col1\" class=\"data row162 col1\" >How much are individuals willing to pay for shoes even if they are out of shape?</td>\n",
       "      <td id=\"T_03ec2_row162_col2\" class=\"data row162 col2\" >$250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row163\" class=\"row_heading level0 row163\" >163</th>\n",
       "      <td id=\"T_03ec2_row163_col0\" class=\"data row163 col0\" >Willing to pay $250 for shoes when they are out of shape. The Upshot study investigated this possibility by controlling for training volume; they did not see substantially different results. Further, our sample consists of solely highly accomplished runners. We believe that these athletes are generally wearing the best shoes available to them whenever they run a marathon.</td>\n",
       "      <td id=\"T_03ec2_row163_col1\" class=\"data row163 col1\" >What did the Upshot study control for to investigate the results?</td>\n",
       "      <td id=\"T_03ec2_row163_col2\" class=\"data row163 col2\" >Training volume</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row164\" class=\"row_heading level0 row164\" >164</th>\n",
       "      <td id=\"T_03ec2_row164_col0\" class=\"data row164 col0\" >Willing to pay $250 for shoes when they are out of shape. The Upshot study investigated this possibility by controlling for training volume; they did not see substantially different results. Further, our sample consists of solely highly accomplished runners. We believe that these athletes are generally wearing the best shoes available to them whenever they run a marathon.</td>\n",
       "      <td id=\"T_03ec2_row164_col1\" class=\"data row164 col1\" >Who does the sample consist of in the study?</td>\n",
       "      <td id=\"T_03ec2_row164_col2\" class=\"data row164 col2\" >Solely highly accomplished runners</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row165\" class=\"row_heading level0 row165\" >165</th>\n",
       "      <td id=\"T_03ec2_row165_col0\" class=\"data row165 col0\" >We were able to identify shoes in nearly all, but not all marathon performances. Athletes have the ability to suppress photographs of themselves, for example by untagging themselves in Facebook photos, or simply electing not to post pictures of themselves from their races. If athletes are more likely to suppress photos of poor performances in Vaporfly shoes, our estimated effect of Vaporfly shoes could be larger than it should be.</td>\n",
       "      <td id=\"T_03ec2_row165_col1\" class=\"data row165 col1\" >What were researchers able to identify in marathon performances?</td>\n",
       "      <td id=\"T_03ec2_row165_col2\" class=\"data row165 col2\" >Shoes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row166\" class=\"row_heading level0 row166\" >166</th>\n",
       "      <td id=\"T_03ec2_row166_col0\" class=\"data row166 col0\" >We were able to identify shoes in nearly all, but not all marathon performances. Athletes have the ability to suppress photographs of themselves, for example by untagging themselves in Facebook photos, or simply electing not to post pictures of themselves from their races. If athletes are more likely to suppress photos of poor performances in Vaporfly shoes, our estimated effect of Vaporfly shoes could be larger than it should be.</td>\n",
       "      <td id=\"T_03ec2_row166_col1\" class=\"data row166 col1\" >How can athletes suppress photographs of themselves?</td>\n",
       "      <td id=\"T_03ec2_row166_col2\" class=\"data row166 col2\" >By untagging themselves in Facebook photos or choosing not to post pictures from their races.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row167\" class=\"row_heading level0 row167\" >167</th>\n",
       "      <td id=\"T_03ec2_row167_col0\" class=\"data row167 col0\" >We were able to identify shoes in nearly all, but not all marathon performances. Athletes have the ability to suppress photographs of themselves, for example by untagging themselves in Facebook photos, or simply electing not to post pictures of themselves from their races. If athletes are more likely to suppress photos of poor performances in Vaporfly shoes, our estimated effect of Vaporfly shoes could be larger than it should be.</td>\n",
       "      <td id=\"T_03ec2_row167_col1\" class=\"data row167 col1\" >What could cause the estimated effect of Vaporfly shoes to be inaccurately large?</td>\n",
       "      <td id=\"T_03ec2_row167_col2\" class=\"data row167 col2\" >If athletes suppress photos of poor performances while wearing Vaporfly shoes.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row168\" class=\"row_heading level0 row168\" >168</th>\n",
       "      <td id=\"T_03ec2_row168_col0\" class=\"data row168 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row168_col1\" class=\"data row168 col1\" >What is Mount Everest?</td>\n",
       "      <td id=\"T_03ec2_row168_col2\" class=\"data row168 col2\" >Earth's highest mountain above sea level</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row169\" class=\"row_heading level0 row169\" >169</th>\n",
       "      <td id=\"T_03ec2_row169_col0\" class=\"data row169 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row169_col1\" class=\"data row169 col1\" >Where is Mount Everest located?</td>\n",
       "      <td id=\"T_03ec2_row169_col2\" class=\"data row169 col2\" >In the Mahalangur Himal sub-range of the Himalayas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row170\" class=\"row_heading level0 row170\" >170</th>\n",
       "      <td id=\"T_03ec2_row170_col0\" class=\"data row170 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row170_col1\" class=\"data row170 col1\" >Which countries does the international border run between at Mount Everest's summit?</td>\n",
       "      <td id=\"T_03ec2_row170_col2\" class=\"data row170 col2\" >China (Tibet Autonomous Region) and Nepal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row171\" class=\"row_heading level0 row171\" >171</th>\n",
       "      <td id=\"T_03ec2_row171_col0\" class=\"data row171 col0\" >The authors thank Richard Heffron and Melissa Hardesty for discussions and providing comments on an early draft of the manuscript, and Richard Cleary for comments on the first version of the paper. We thank two peer reviewers, Harry Crane and Ted Westling, for providing detailed assessments of the paper and suggestions for improvements. Their reviews, along with our responses, are included in the Appendix. Finally, we express gratitude for those who take photos of major marathons and post them to social media, especially Karen Mitchell, Clay Shaw, and Malrie Sonier.</td>\n",
       "      <td id=\"T_03ec2_row171_col1\" class=\"data row171 col1\" >Who provided comments on the early draft of the manuscript?</td>\n",
       "      <td id=\"T_03ec2_row171_col2\" class=\"data row171 col2\" >Richard Heffron and Melissa Hardesty</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row172\" class=\"row_heading level0 row172\" >172</th>\n",
       "      <td id=\"T_03ec2_row172_col0\" class=\"data row172 col0\" >The authors thank Richard Heffron and Melissa Hardesty for discussions and providing comments on an early draft of the manuscript, and Richard Cleary for comments on the first version of the paper. We thank two peer reviewers, Harry Crane and Ted Westling, for providing detailed assessments of the paper and suggestions for improvements. Their reviews, along with our responses, are included in the Appendix. Finally, we express gratitude for those who take photos of major marathons and post them to social media, especially Karen Mitchell, Clay Shaw, and Malrie Sonier.</td>\n",
       "      <td id=\"T_03ec2_row172_col1\" class=\"data row172 col1\" >Who are the two peer reviewers mentioned?</td>\n",
       "      <td id=\"T_03ec2_row172_col2\" class=\"data row172 col2\" >Harry Crane and Ted Westling</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row173\" class=\"row_heading level0 row173\" >173</th>\n",
       "      <td id=\"T_03ec2_row173_col0\" class=\"data row173 col0\" >The authors thank Richard Heffron and Melissa Hardesty for discussions and providing comments on an early draft of the manuscript, and Richard Cleary for comments on the first version of the paper. We thank two peer reviewers, Harry Crane and Ted Westling, for providing detailed assessments of the paper and suggestions for improvements. Their reviews, along with our responses, are included in the Appendix. Finally, we express gratitude for those who take photos of major marathons and post them to social media, especially Karen Mitchell, Clay Shaw, and Malrie Sonier.</td>\n",
       "      <td id=\"T_03ec2_row173_col1\" class=\"data row173 col1\" >What did Karen Mitchell, Clay Shaw, and Malrie Sonier do to be thanked?</td>\n",
       "      <td id=\"T_03ec2_row173_col2\" class=\"data row173 col2\" >They take photos of major marathons and post them to social media</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row174\" class=\"row_heading level0 row174\" >174</th>\n",
       "      <td id=\"T_03ec2_row174_col0\" class=\"data row174 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row174_col1\" class=\"data row174 col1\" >What is Mount Everest known for?</td>\n",
       "      <td id=\"T_03ec2_row174_col2\" class=\"data row174 col2\" >It is Earth's highest mountain above sea level.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row175\" class=\"row_heading level0 row175\" >175</th>\n",
       "      <td id=\"T_03ec2_row175_col0\" class=\"data row175 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row175_col1\" class=\"data row175 col1\" >Where is Mount Everest located?</td>\n",
       "      <td id=\"T_03ec2_row175_col2\" class=\"data row175 col2\" >In the Mahalangur Himal sub-range of the Himalayas.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row176\" class=\"row_heading level0 row176\" >176</th>\n",
       "      <td id=\"T_03ec2_row176_col0\" class=\"data row176 col0\" >Mount Everest is Earth's highest mountain above sea level, located in the Mahalangur Himal sub-range of the Himalayas. The international border between China (Tibet Autonomous Region) and Nepal runs across its summit point.</td>\n",
       "      <td id=\"T_03ec2_row176_col1\" class=\"data row176 col1\" >Which two countries does the international border run between at Mount Everest's summit?</td>\n",
       "      <td id=\"T_03ec2_row176_col2\" class=\"data row176 col2\" >China (Tibet Autonomous Region) and Nepal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row177\" class=\"row_heading level0 row177\" >177</th>\n",
       "      <td id=\"T_03ec2_row177_col0\" class=\"data row177 col0\" >* Barnes and Kilding (2019) Barnes, K. R. and Kilding, A. E. (2019). A randomized crossover study investigating the running economy of highly-trained male and female distance runners in marathon racing shoes versus track spikes. _Sports Medicine_, 49(2):331-342.\n",
       "* Bates et al. (2015) Bates, D., Machler, M., Bolker, B., and Walker, S. (2015). Fitting linear mixed-effects models using lme4. _Journal of Statistical Software_, 67(1):1-48.\n",
       "* Hoogkamer et al. (2018) Hoogkamer, W., Kipp, S., Frank, J. H., Farina, E. M., Luo, G., and Kram, R. (2018). A comparison of the energetic cost of running in marathon racing shoes versus track spikes. _Sports Medicine_, 48(4):1009-1019.\n",
       "* will anything be done about it? https://tinyurl.com/umvmy7t.\n",
       "* Kealy and Katz (2018) Kealy, K. and Katz, J. (2018). Nike says its $250 running shoes will make you run much faster. what if that's actually true? https://www.nytimes.com/interactive/2018/07/18/upshot/nike-vaporfly-shoe-strava.html.\n",
       "* Kealy and Katz (2019) Kealy, K. and Katz, J. (2019). Nike's fastest shoes may give runners an even bigger advantage than we thought. https://www.nytimes.com/interactive/2019/12/13/upshot/nike-vaporfly-next-percent-shoe-estimates.html.\n",
       "* R Core Team (2018) R Core Team (2018). _R: A Language and Environment for Statistical Computing_. R Foundation for Statistical Computing, Vienna, Austria.\n",
       "* R Core Team (2018) </td>\n",
       "      <td id=\"T_03ec2_row177_col1\" class=\"data row177 col1\" >What was the focus of the study by Barnes and Kilding in 2019?</td>\n",
       "      <td id=\"T_03ec2_row177_col2\" class=\"data row177 col2\" >The study investigated the running economy of highly-trained male and female distance runners in marathon racing shoes versus track spikes.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row178\" class=\"row_heading level0 row178\" >178</th>\n",
       "      <td id=\"T_03ec2_row178_col0\" class=\"data row178 col0\" >* Barnes and Kilding (2019) Barnes, K. R. and Kilding, A. E. (2019). A randomized crossover study investigating the running economy of highly-trained male and female distance runners in marathon racing shoes versus track spikes. _Sports Medicine_, 49(2):331-342.\n",
       "* Bates et al. (2015) Bates, D., Machler, M., Bolker, B., and Walker, S. (2015). Fitting linear mixed-effects models using lme4. _Journal of Statistical Software_, 67(1):1-48.\n",
       "* Hoogkamer et al. (2018) Hoogkamer, W., Kipp, S., Frank, J. H., Farina, E. M., Luo, G., and Kram, R. (2018). A comparison of the energetic cost of running in marathon racing shoes versus track spikes. _Sports Medicine_, 48(4):1009-1019.\n",
       "* will anything be done about it? https://tinyurl.com/umvmy7t.\n",
       "* Kealy and Katz (2018) Kealy, K. and Katz, J. (2018). Nike says its $250 running shoes will make you run much faster. what if that's actually true? https://www.nytimes.com/interactive/2018/07/18/upshot/nike-vaporfly-shoe-strava.html.\n",
       "* Kealy and Katz (2019) Kealy, K. and Katz, J. (2019). Nike's fastest shoes may give runners an even bigger advantage than we thought. https://www.nytimes.com/interactive/2019/12/13/upshot/nike-vaporfly-next-percent-shoe-estimates.html.\n",
       "* R Core Team (2018) R Core Team (2018). _R: A Language and Environment for Statistical Computing_. R Foundation for Statistical Computing, Vienna, Austria.\n",
       "* R Core Team (2018) </td>\n",
       "      <td id=\"T_03ec2_row178_col1\" class=\"data row178 col1\" >What is the title of the article by Kealy and Katz in 2018 about Nike's running shoes?</td>\n",
       "      <td id=\"T_03ec2_row178_col2\" class=\"data row178 col2\" >Nike says its $250 running shoes will make you run much faster. what if that's actually true?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row179\" class=\"row_heading level0 row179\" >179</th>\n",
       "      <td id=\"T_03ec2_row179_col0\" class=\"data row179 col0\" >* Barnes and Kilding (2019) Barnes, K. R. and Kilding, A. E. (2019). A randomized crossover study investigating the running economy of highly-trained male and female distance runners in marathon racing shoes versus track spikes. _Sports Medicine_, 49(2):331-342.\n",
       "* Bates et al. (2015) Bates, D., Machler, M., Bolker, B., and Walker, S. (2015). Fitting linear mixed-effects models using lme4. _Journal of Statistical Software_, 67(1):1-48.\n",
       "* Hoogkamer et al. (2018) Hoogkamer, W., Kipp, S., Frank, J. H., Farina, E. M., Luo, G., and Kram, R. (2018). A comparison of the energetic cost of running in marathon racing shoes versus track spikes. _Sports Medicine_, 48(4):1009-1019.\n",
       "* will anything be done about it? https://tinyurl.com/umvmy7t.\n",
       "* Kealy and Katz (2018) Kealy, K. and Katz, J. (2018). Nike says its $250 running shoes will make you run much faster. what if that's actually true? https://www.nytimes.com/interactive/2018/07/18/upshot/nike-vaporfly-shoe-strava.html.\n",
       "* Kealy and Katz (2019) Kealy, K. and Katz, J. (2019). Nike's fastest shoes may give runners an even bigger advantage than we thought. https://www.nytimes.com/interactive/2019/12/13/upshot/nike-vaporfly-next-percent-shoe-estimates.html.\n",
       "* R Core Team (2018) R Core Team (2018). _R: A Language and Environment for Statistical Computing_. R Foundation for Statistical Computing, Vienna, Austria.\n",
       "* R Core Team (2018) </td>\n",
       "      <td id=\"T_03ec2_row179_col1\" class=\"data row179 col1\" >What software is associated with the R Core Team in 2018?</td>\n",
       "      <td id=\"T_03ec2_row179_col2\" class=\"data row179 col2\" >R: A Language and Environment for Statistical Computing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row180\" class=\"row_heading level0 row180\" >180</th>\n",
       "      <td id=\"T_03ec2_row180_col0\" class=\"data row180 col0\" >Thompson, N. (2017). Do nike's new marathon shoes actually make you run faster? https://www.wired.com/story/do-nike-zoom-vaporfly-make-you-run-faster/.</td>\n",
       "      <td id=\"T_03ec2_row180_col1\" class=\"data row180 col1\" >What is the main topic of the article by Thompson?</td>\n",
       "      <td id=\"T_03ec2_row180_col2\" class=\"data row180 col2\" >Whether Nike's new marathon shoes can make you run faster</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row181\" class=\"row_heading level0 row181\" >181</th>\n",
       "      <td id=\"T_03ec2_row181_col0\" class=\"data row181 col0\" >Thompson, N. (2017). Do nike's new marathon shoes actually make you run faster? https://www.wired.com/story/do-nike-zoom-vaporfly-make-you-run-faster/.</td>\n",
       "      <td id=\"T_03ec2_row181_col1\" class=\"data row181 col1\" >In what year was the article about Nike's marathon shoes published?</td>\n",
       "      <td id=\"T_03ec2_row181_col2\" class=\"data row181 col2\" >2017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row182\" class=\"row_heading level0 row182\" >182</th>\n",
       "      <td id=\"T_03ec2_row182_col0\" class=\"data row182 col0\" >Thompson, N. (2017). Do nike's new marathon shoes actually make you run faster? https://www.wired.com/story/do-nike-zoom-vaporfly-make-you-run-faster/.</td>\n",
       "      <td id=\"T_03ec2_row182_col1\" class=\"data row182 col1\" >Where can the article by Thompson be found?</td>\n",
       "      <td id=\"T_03ec2_row182_col2\" class=\"data row182 col2\" >On the website https://www.wired.com/story/do-nike-zoom-vaporfly-make-you-run-faster/</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row183\" class=\"row_heading level0 row183\" >183</th>\n",
       "      <td id=\"T_03ec2_row183_col0\" class=\"data row183 col0\" >\\begin{tabular}{l} \\hline Boston Marathon \\\\ California International Marathon \\\\ Chicago Marathon \\\\ Columbus Marathon \\\\ Eugene Marathon \\\\ Grandma's Marathon \\\\ Houston Marathon \\\\ Indianapolis Monumental Marathon \\\\ Los Angeles Marathon \\\\ Lakefront Marathon \\\\ Marine Corps Marathon \\\\ New York City Marathon \\\\ Olympic Trials Marathon \\\\ Ottawa Marathon \\\\ Philadelphia Marathon \\\\ Phoenix Marathon \\\\ Richmond Marathon \\\\ Toronto Waterfront Marathon \\\\ Twin Cities Marathon \\\\ Vancouver International Marathon \\\\ Vermont City Marathon \\\\ Wineglass Marathon \\\\ \\hline \\end{tabular}</td>\n",
       "      <td id=\"T_03ec2_row183_col1\" class=\"data row183 col1\" >What is the name of the marathon held in New York?</td>\n",
       "      <td id=\"T_03ec2_row183_col2\" class=\"data row183 col2\" >New York City Marathon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row184\" class=\"row_heading level0 row184\" >184</th>\n",
       "      <td id=\"T_03ec2_row184_col0\" class=\"data row184 col0\" >\\begin{tabular}{l} \\hline Boston Marathon \\\\ California International Marathon \\\\ Chicago Marathon \\\\ Columbus Marathon \\\\ Eugene Marathon \\\\ Grandma's Marathon \\\\ Houston Marathon \\\\ Indianapolis Monumental Marathon \\\\ Los Angeles Marathon \\\\ Lakefront Marathon \\\\ Marine Corps Marathon \\\\ New York City Marathon \\\\ Olympic Trials Marathon \\\\ Ottawa Marathon \\\\ Philadelphia Marathon \\\\ Phoenix Marathon \\\\ Richmond Marathon \\\\ Toronto Waterfront Marathon \\\\ Twin Cities Marathon \\\\ Vancouver International Marathon \\\\ Vermont City Marathon \\\\ Wineglass Marathon \\\\ \\hline \\end{tabular}</td>\n",
       "      <td id=\"T_03ec2_row184_col1\" class=\"data row184 col1\" >Which marathon is listed that takes place in Canada?</td>\n",
       "      <td id=\"T_03ec2_row184_col2\" class=\"data row184 col2\" >Ottawa Marathon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row185\" class=\"row_heading level0 row185\" >185</th>\n",
       "      <td id=\"T_03ec2_row185_col0\" class=\"data row185 col0\" >\\begin{tabular}{l} \\hline Boston Marathon \\\\ California International Marathon \\\\ Chicago Marathon \\\\ Columbus Marathon \\\\ Eugene Marathon \\\\ Grandma's Marathon \\\\ Houston Marathon \\\\ Indianapolis Monumental Marathon \\\\ Los Angeles Marathon \\\\ Lakefront Marathon \\\\ Marine Corps Marathon \\\\ New York City Marathon \\\\ Olympic Trials Marathon \\\\ Ottawa Marathon \\\\ Philadelphia Marathon \\\\ Phoenix Marathon \\\\ Richmond Marathon \\\\ Toronto Waterfront Marathon \\\\ Twin Cities Marathon \\\\ Vancouver International Marathon \\\\ Vermont City Marathon \\\\ Wineglass Marathon \\\\ \\hline \\end{tabular}</td>\n",
       "      <td id=\"T_03ec2_row185_col1\" class=\"data row185 col1\" >Name a marathon that has a connection to the military.</td>\n",
       "      <td id=\"T_03ec2_row185_col2\" class=\"data row185 col2\" >Marine Corps Marathon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row186\" class=\"row_heading level0 row186\" >186</th>\n",
       "      <td id=\"T_03ec2_row186_col0\" class=\"data row186 col0\" >This section is divided into subsections, one for each reviewer that provided comments on the paper. The reviewers' comments are italicized, and our responses and discussions are in normal text.</td>\n",
       "      <td id=\"T_03ec2_row186_col1\" class=\"data row186 col1\" >How is the section organized?</td>\n",
       "      <td id=\"T_03ec2_row186_col2\" class=\"data row186 col2\" >Into subsections, one for each reviewer.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row187\" class=\"row_heading level0 row187\" >187</th>\n",
       "      <td id=\"T_03ec2_row187_col0\" class=\"data row187 col0\" >This section is divided into subsections, one for each reviewer that provided comments on the paper. The reviewers' comments are italicized, and our responses and discussions are in normal text.</td>\n",
       "      <td id=\"T_03ec2_row187_col1\" class=\"data row187 col1\" >How are the reviewers' comments presented in the text?</td>\n",
       "      <td id=\"T_03ec2_row187_col2\" class=\"data row187 col2\" >Italicized</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row188\" class=\"row_heading level0 row188\" >188</th>\n",
       "      <td id=\"T_03ec2_row188_col0\" class=\"data row188 col0\" >This section is divided into subsections, one for each reviewer that provided comments on the paper. The reviewers' comments are italicized, and our responses and discussions are in normal text.</td>\n",
       "      <td id=\"T_03ec2_row188_col1\" class=\"data row188 col1\" >How are the responses and discussions to the reviewers' comments presented?</td>\n",
       "      <td id=\"T_03ec2_row188_col2\" class=\"data row188 col2\" >In normal text</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row189\" class=\"row_heading level0 row189\" >189</th>\n",
       "      <td id=\"T_03ec2_row189_col0\" class=\"data row189 col0\" >This is a very interesting analysis of runner performance with and without the Nike Vaporfly. I'd expect this to be of interest to the running and sports science community, and also the general public as it shows how new technologies can impact the outcome of competitive events. In a case such as this, the early adopters will gain a competitive advantage for however long the benefits of the new product remain unknown to the broader pool of competitors. I hope this analysis will reach the most relevant communities. I have a few questions and comments</td>\n",
       "      <td id=\"T_03ec2_row189_col1\" class=\"data row189 col1\" >What is being analyzed in the context?</td>\n",
       "      <td id=\"T_03ec2_row189_col2\" class=\"data row189 col2\" >Runner performance with and without the Nike Vaporfly.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row190\" class=\"row_heading level0 row190\" >190</th>\n",
       "      <td id=\"T_03ec2_row190_col0\" class=\"data row190 col0\" >This is a very interesting analysis of runner performance with and without the Nike Vaporfly. I'd expect this to be of interest to the running and sports science community, and also the general public as it shows how new technologies can impact the outcome of competitive events. In a case such as this, the early adopters will gain a competitive advantage for however long the benefits of the new product remain unknown to the broader pool of competitors. I hope this analysis will reach the most relevant communities. I have a few questions and comments</td>\n",
       "      <td id=\"T_03ec2_row190_col1\" class=\"data row190 col1\" >Who would find this analysis of interest?</td>\n",
       "      <td id=\"T_03ec2_row190_col2\" class=\"data row190 col2\" >The running and sports science community, and the general public.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row191\" class=\"row_heading level0 row191\" >191</th>\n",
       "      <td id=\"T_03ec2_row191_col0\" class=\"data row191 col0\" >This is a very interesting analysis of runner performance with and without the Nike Vaporfly. I'd expect this to be of interest to the running and sports science community, and also the general public as it shows how new technologies can impact the outcome of competitive events. In a case such as this, the early adopters will gain a competitive advantage for however long the benefits of the new product remain unknown to the broader pool of competitors. I hope this analysis will reach the most relevant communities. I have a few questions and comments</td>\n",
       "      <td id=\"T_03ec2_row191_col1\" class=\"data row191 col1\" >What advantage do early adopters of the Nike Vaporfly gain?</td>\n",
       "      <td id=\"T_03ec2_row191_col2\" class=\"data row191 col2\" >A competitive advantage for as long as the benefits of the new product remain unknown to the broader pool of competitors.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row192\" class=\"row_heading level0 row192\" >192</th>\n",
       "      <td id=\"T_03ec2_row192_col0\" class=\"data row192 col0\" >below. Some of these might be worth commenting in the paper, but for the most part they are for my own curiosity.</td>\n",
       "      <td id=\"T_03ec2_row192_col1\" class=\"data row192 col1\" >What might be worth including in the paper?</td>\n",
       "      <td id=\"T_03ec2_row192_col2\" class=\"data row192 col2\" >Some of the points mentioned below.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row193\" class=\"row_heading level0 row193\" >193</th>\n",
       "      <td id=\"T_03ec2_row193_col0\" class=\"data row193 col0\" >below. Some of these might be worth commenting in the paper, but for the most part they are for my own curiosity.</td>\n",
       "      <td id=\"T_03ec2_row193_col1\" class=\"data row193 col1\" >What is the primary purpose of the points mentioned?</td>\n",
       "      <td id=\"T_03ec2_row193_col2\" class=\"data row193 col2\" >For the author's own curiosity.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row194\" class=\"row_heading level0 row194\" >194</th>\n",
       "      <td id=\"T_03ec2_row194_col0\" class=\"data row194 col0\" >below. Some of these might be worth commenting in the paper, but for the most part they are for my own curiosity.</td>\n",
       "      <td id=\"T_03ec2_row194_col1\" class=\"data row194 col1\" >Is it necessary to comment on all the points in the paper?</td>\n",
       "      <td id=\"T_03ec2_row194_col2\" class=\"data row194 col2\" >No, only some might be worth commenting on.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row195\" class=\"row_heading level0 row195\" >195</th>\n",
       "      <td id=\"T_03ec2_row195_col0\" class=\"data row195 col0\" >1. The analysis notes a performance improvement for athletes who have switched from non-Vaporfly to Vaporfly. Are there any instances of runners going in the other direction, from Vaporfly to non-Vaporfly? If yes, do you find anything interesting in the first performance back without Vaporfly? If not, do the authors believe this is because there has not been sufficient time to observe such a switch or because the performance enhancement is noticeable enough to the athletes that they would not want to switch?</td>\n",
       "      <td id=\"T_03ec2_row195_col1\" class=\"data row195 col1\" >What improvement is noted when athletes switch to Vaporfly?</td>\n",
       "      <td id=\"T_03ec2_row195_col2\" class=\"data row195 col2\" >Performance improvement</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row196\" class=\"row_heading level0 row196\" >196</th>\n",
       "      <td id=\"T_03ec2_row196_col0\" class=\"data row196 col0\" >1. The analysis notes a performance improvement for athletes who have switched from non-Vaporfly to Vaporfly. Are there any instances of runners going in the other direction, from Vaporfly to non-Vaporfly? If yes, do you find anything interesting in the first performance back without Vaporfly? If not, do the authors believe this is because there has not been sufficient time to observe such a switch or because the performance enhancement is noticeable enough to the athletes that they would not want to switch?</td>\n",
       "      <td id=\"T_03ec2_row196_col1\" class=\"data row196 col1\" >What is being asked about runners switching from Vaporfly to non-Vaporfly?</td>\n",
       "      <td id=\"T_03ec2_row196_col2\" class=\"data row196 col2\" >If there are instances of this happening and if there is anything interesting in their first performance back without Vaporfly.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row197\" class=\"row_heading level0 row197\" >197</th>\n",
       "      <td id=\"T_03ec2_row197_col0\" class=\"data row197 col0\" >1. The analysis notes a performance improvement for athletes who have switched from non-Vaporfly to Vaporfly. Are there any instances of runners going in the other direction, from Vaporfly to non-Vaporfly? If yes, do you find anything interesting in the first performance back without Vaporfly? If not, do the authors believe this is because there has not been sufficient time to observe such a switch or because the performance enhancement is noticeable enough to the athletes that they would not want to switch?</td>\n",
       "      <td id=\"T_03ec2_row197_col1\" class=\"data row197 col1\" >What might be the reason authors believe athletes are not switching back to non-Vaporfly?</td>\n",
       "      <td id=\"T_03ec2_row197_col2\" class=\"data row197 col2\" >Either there has not been sufficient time to observe such a switch or the performance enhancement with Vaporfly is noticeable enough that athletes would not want to switch.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row198\" class=\"row_heading level0 row198\" >198</th>\n",
       "      <td id=\"T_03ec2_row198_col0\" class=\"data row198 col0\" >We identified 5 runners who switched to the Vaporfly and then switched to a different shoe. Looking closely at each of these cases, we found that one was a case of a misidentified shoe, bringing the total to 4 runners. One of them joined a racing team sponsored by another company (Brooks). One switched to what looks like a prototype of Brooks's new racing shoe. One is wearing a different Brooks shoe, and one is wearing a different Nike shoe.</td>\n",
       "      <td id=\"T_03ec2_row198_col1\" class=\"data row198 col1\" >How many runners initially switched to the Vaporfly?</td>\n",
       "      <td id=\"T_03ec2_row198_col2\" class=\"data row198 col2\" >5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row199\" class=\"row_heading level0 row199\" >199</th>\n",
       "      <td id=\"T_03ec2_row199_col0\" class=\"data row199 col0\" >We identified 5 runners who switched to the Vaporfly and then switched to a different shoe. Looking closely at each of these cases, we found that one was a case of a misidentified shoe, bringing the total to 4 runners. One of them joined a racing team sponsored by another company (Brooks). One switched to what looks like a prototype of Brooks's new racing shoe. One is wearing a different Brooks shoe, and one is wearing a different Nike shoe.</td>\n",
       "      <td id=\"T_03ec2_row199_col1\" class=\"data row199 col1\" >Why was the number of runners who switched shoes reduced from 5 to 4?</td>\n",
       "      <td id=\"T_03ec2_row199_col2\" class=\"data row199 col2\" >One was a case of a misidentified shoe.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row200\" class=\"row_heading level0 row200\" >200</th>\n",
       "      <td id=\"T_03ec2_row200_col0\" class=\"data row200 col0\" >We identified 5 runners who switched to the Vaporfly and then switched to a different shoe. Looking closely at each of these cases, we found that one was a case of a misidentified shoe, bringing the total to 4 runners. One of them joined a racing team sponsored by another company (Brooks). One switched to what looks like a prototype of Brooks's new racing shoe. One is wearing a different Brooks shoe, and one is wearing a different Nike shoe.</td>\n",
       "      <td id=\"T_03ec2_row200_col1\" class=\"data row200 col1\" >What was the reason for one of the runners switching from the Vaporfly to another brand?</td>\n",
       "      <td id=\"T_03ec2_row200_col2\" class=\"data row200 col2\" >One of them joined a racing team sponsored by another company (Brooks).</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row201\" class=\"row_heading level0 row201\" >201</th>\n",
       "      <td id=\"T_03ec2_row201_col0\" class=\"data row201 col0\" >This is too small a sample to draw any conclusions, but of these 4 runners, 3 ran faster after switching away from the Vaporfly, although 1 of the 3 who ran faster did so on a faster course than the previous marathon in Vaporflys.</td>\n",
       "      <td id=\"T_03ec2_row201_col1\" class=\"data row201 col1\" >How many runners ran faster after switching away from the Vaporfly?</td>\n",
       "      <td id=\"T_03ec2_row201_col2\" class=\"data row201 col2\" >3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row202\" class=\"row_heading level0 row202\" >202</th>\n",
       "      <td id=\"T_03ec2_row202_col0\" class=\"data row202 col0\" >This is too small a sample to draw any conclusions, but of these 4 runners, 3 ran faster after switching away from the Vaporfly, although 1 of the 3 who ran faster did so on a faster course than the previous marathon in Vaporflys.</td>\n",
       "      <td id=\"T_03ec2_row202_col1\" class=\"data row202 col1\" >Why is it suggested that the sample size is too small?</td>\n",
       "      <td id=\"T_03ec2_row202_col2\" class=\"data row202 col2\" >Because it is not large enough to draw any conclusions</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row203\" class=\"row_heading level0 row203\" >203</th>\n",
       "      <td id=\"T_03ec2_row203_col0\" class=\"data row203 col0\" >This is too small a sample to draw any conclusions, but of these 4 runners, 3 ran faster after switching away from the Vaporfly, although 1 of the 3 who ran faster did so on a faster course than the previous marathon in Vaporflys.</td>\n",
       "      <td id=\"T_03ec2_row203_col1\" class=\"data row203 col1\" >What could have contributed to one of the runners' improved speed apart from switching shoes?</td>\n",
       "      <td id=\"T_03ec2_row203_col2\" class=\"data row203 col2\" >Running on a faster course than the previous marathon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row204\" class=\"row_heading level0 row204\" >204</th>\n",
       "      <td id=\"T_03ec2_row204_col0\" class=\"data row204 col0\" >2. Your model includes a random effect for each marathon, which I suppose captures any factors associated with all runners in that race running faster or slower than usual, e.g., bad weather, wind, etc. Have you considered to include a course effect for any marathons in your dataset that were run over the same course in different years, e.g., Boston 2016 and Boston 2017. Would this possibly help tease out any additional variation in course-day variation?</td>\n",
       "      <td id=\"T_03ec2_row204_col1\" class=\"data row204 col1\" >What does the random effect in the model account for?</td>\n",
       "      <td id=\"T_03ec2_row204_col2\" class=\"data row204 col2\" >The random effect in the model accounts for factors that affect all runners' performance in a race, such as bad weather or wind.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row205\" class=\"row_heading level0 row205\" >205</th>\n",
       "      <td id=\"T_03ec2_row205_col0\" class=\"data row205 col0\" >2. Your model includes a random effect for each marathon, which I suppose captures any factors associated with all runners in that race running faster or slower than usual, e.g., bad weather, wind, etc. Have you considered to include a course effect for any marathons in your dataset that were run over the same course in different years, e.g., Boston 2016 and Boston 2017. Would this possibly help tease out any additional variation in course-day variation?</td>\n",
       "      <td id=\"T_03ec2_row205_col1\" class=\"data row205 col1\" >What is the suggested addition to the model to account for variations in the same course over different years?</td>\n",
       "      <td id=\"T_03ec2_row205_col2\" class=\"data row205 col2\" >The suggested addition is a course effect for marathons run over the same course in different years.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row206\" class=\"row_heading level0 row206\" >206</th>\n",
       "      <td id=\"T_03ec2_row206_col0\" class=\"data row206 col0\" >2. Your model includes a random effect for each marathon, which I suppose captures any factors associated with all runners in that race running faster or slower than usual, e.g., bad weather, wind, etc. Have you considered to include a course effect for any marathons in your dataset that were run over the same course in different years, e.g., Boston 2016 and Boston 2017. Would this possibly help tease out any additional variation in course-day variation?</td>\n",
       "      <td id=\"T_03ec2_row206_col1\" class=\"data row206 col1\" >What is the purpose of including a course effect for marathons in the dataset?</td>\n",
       "      <td id=\"T_03ec2_row206_col2\" class=\"data row206 col2\" >Including a course effect could help tease out any additional variation in performance specific to the course on different days.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row207\" class=\"row_heading level0 row207\" >207</th>\n",
       "      <td id=\"T_03ec2_row207_col0\" class=\"data row207 col0\" >This is a good suggestion for making the marathon effects more flexible, as you are right that certain courses are difficult every year, but the exact difficulty varies from year to year because of weather conditions and other factors.</td>\n",
       "      <td id=\"T_03ec2_row207_col1\" class=\"data row207 col1\" >What is the suggestion aimed to improve?</td>\n",
       "      <td id=\"T_03ec2_row207_col2\" class=\"data row207 col2\" >Making the marathon effects more flexible</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row208\" class=\"row_heading level0 row208\" >208</th>\n",
       "      <td id=\"T_03ec2_row208_col0\" class=\"data row208 col0\" >This is a good suggestion for making the marathon effects more flexible, as you are right that certain courses are difficult every year, but the exact difficulty varies from year to year because of weather conditions and other factors.</td>\n",
       "      <td id=\"T_03ec2_row208_col1\" class=\"data row208 col1\" >Why do the courses' difficulty vary each year?</td>\n",
       "      <td id=\"T_03ec2_row208_col2\" class=\"data row208 col2\" >Because of weather conditions and other factors</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row209\" class=\"row_heading level0 row209\" >209</th>\n",
       "      <td id=\"T_03ec2_row209_col0\" class=\"data row209 col0\" >This is a good suggestion for making the marathon effects more flexible, as you are right that certain courses are difficult every year, but the exact difficulty varies from year to year because of weather conditions and other factors.</td>\n",
       "      <td id=\"T_03ec2_row209_col1\" class=\"data row209 col1\" >Is the difficulty of certain courses consistent every year?</td>\n",
       "      <td id=\"T_03ec2_row209_col2\" class=\"data row209 col2\" >No, the exact difficulty varies from year to year</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row210\" class=\"row_heading level0 row210\" >210</th>\n",
       "      <td id=\"T_03ec2_row210_col0\" class=\"data row210 col0\" >We have updated our analysis to include an additional set of random effects that are specific to marathon courses, in addition to the original effects that were specific to course-year combinations.</td>\n",
       "      <td id=\"T_03ec2_row210_col1\" class=\"data row210 col1\" >What has been updated in the analysis?</td>\n",
       "      <td id=\"T_03ec2_row210_col2\" class=\"data row210 col2\" >An additional set of random effects specific to marathon courses.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row211\" class=\"row_heading level0 row211\" >211</th>\n",
       "      <td id=\"T_03ec2_row211_col0\" class=\"data row211 col0\" >We have updated our analysis to include an additional set of random effects that are specific to marathon courses, in addition to the original effects that were specific to course-year combinations.</td>\n",
       "      <td id=\"T_03ec2_row211_col1\" class=\"data row211 col1\" >What were the original effects in the analysis specific to?</td>\n",
       "      <td id=\"T_03ec2_row211_col2\" class=\"data row211 col2\" >Course-year combinations.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row212\" class=\"row_heading level0 row212\" >212</th>\n",
       "      <td id=\"T_03ec2_row212_col0\" class=\"data row212 col0\" >We have updated our analysis to include an additional set of random effects that are specific to marathon courses, in addition to the original effects that were specific to course-year combinations.</td>\n",
       "      <td id=\"T_03ec2_row212_col1\" class=\"data row212 col1\" >What is the purpose of the additional set of random effects?</td>\n",
       "      <td id=\"T_03ec2_row212_col2\" class=\"data row212 col2\" >To be specific to marathon courses.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row213\" class=\"row_heading level0 row213\" >213</th>\n",
       "      <td id=\"T_03ec2_row213_col0\" class=\"data row213 col0\" >Overall, the estimates of the Vaporfly effects decrease slightly in magnitude, but none of the findings of the paper change substantially. We think that the decrease in the Vaporfly effects is due to the added flexibility in the random effects terms.</td>\n",
       "      <td id=\"T_03ec2_row213_col1\" class=\"data row213 col1\" >What happens to the estimates of the Vaporfly effects?</td>\n",
       "      <td id=\"T_03ec2_row213_col2\" class=\"data row213 col2\" >They decrease slightly in magnitude.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row214\" class=\"row_heading level0 row214\" >214</th>\n",
       "      <td id=\"T_03ec2_row214_col0\" class=\"data row214 col0\" >Overall, the estimates of the Vaporfly effects decrease slightly in magnitude, but none of the findings of the paper change substantially. We think that the decrease in the Vaporfly effects is due to the added flexibility in the random effects terms.</td>\n",
       "      <td id=\"T_03ec2_row214_col1\" class=\"data row214 col1\" >Do the findings of the paper change substantially after the decrease in Vaporfly effects?</td>\n",
       "      <td id=\"T_03ec2_row214_col2\" class=\"data row214 col2\" >No, the findings of the paper do not change substantially.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row215\" class=\"row_heading level0 row215\" >215</th>\n",
       "      <td id=\"T_03ec2_row215_col0\" class=\"data row215 col0\" >Overall, the estimates of the Vaporfly effects decrease slightly in magnitude, but none of the findings of the paper change substantially. We think that the decrease in the Vaporfly effects is due to the added flexibility in the random effects terms.</td>\n",
       "      <td id=\"T_03ec2_row215_col1\" class=\"data row215 col1\" >What is the reason for the decrease in the Vaporfly effects?</td>\n",
       "      <td id=\"T_03ec2_row215_col2\" class=\"data row215 col2\" >The added flexibility in the random effects terms.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row216\" class=\"row_heading level0 row216\" >216</th>\n",
       "      <td id=\"T_03ec2_row216_col0\" class=\"data row216 col0\" >3. Presumably athletes who switch to the new shoe would want to know whether the shoe is going to help their performance specifically, rather than just a positive association over the population of all runners who wear the shoe. Is it possible to draw a causal relationship from the results of this analysis? The authors mention some potential confounding factors, such as cost of the shoe and runners knowing whether they are training well for an upcoming race. If one were to attempt such a causal inference, what additional analyses should be carried out?</td>\n",
       "      <td id=\"T_03ec2_row216_col1\" class=\"data row216 col1\" >What do athletes want to know about the new shoe?</td>\n",
       "      <td id=\"T_03ec2_row216_col2\" class=\"data row216 col2\" >Whether the shoe is going to help their performance specifically.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row217\" class=\"row_heading level0 row217\" >217</th>\n",
       "      <td id=\"T_03ec2_row217_col0\" class=\"data row217 col0\" >3. Presumably athletes who switch to the new shoe would want to know whether the shoe is going to help their performance specifically, rather than just a positive association over the population of all runners who wear the shoe. Is it possible to draw a causal relationship from the results of this analysis? The authors mention some potential confounding factors, such as cost of the shoe and runners knowing whether they are training well for an upcoming race. If one were to attempt such a causal inference, what additional analyses should be carried out?</td>\n",
       "      <td id=\"T_03ec2_row217_col1\" class=\"data row217 col1\" >What are some potential confounding factors mentioned by the authors?</td>\n",
       "      <td id=\"T_03ec2_row217_col2\" class=\"data row217 col2\" >Cost of the shoe and runners knowing whether they are training well for an upcoming race.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row218\" class=\"row_heading level0 row218\" >218</th>\n",
       "      <td id=\"T_03ec2_row218_col0\" class=\"data row218 col0\" >3. Presumably athletes who switch to the new shoe would want to know whether the shoe is going to help their performance specifically, rather than just a positive association over the population of all runners who wear the shoe. Is it possible to draw a causal relationship from the results of this analysis? The authors mention some potential confounding factors, such as cost of the shoe and runners knowing whether they are training well for an upcoming race. If one were to attempt such a causal inference, what additional analyses should be carried out?</td>\n",
       "      <td id=\"T_03ec2_row218_col1\" class=\"data row218 col1\" >What is necessary to establish a causal relationship regarding the shoe's effect on performance?</td>\n",
       "      <td id=\"T_03ec2_row218_col2\" class=\"data row218 col2\" >Additional analyses that account for the potential confounding factors should be carried out.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row219\" class=\"row_heading level0 row219\" >219</th>\n",
       "      <td id=\"T_03ec2_row219_col0\" class=\"data row219 col0\" >As we understand it, causal inference is an attempt to control for confounders in an observational study. In that sense, one can interpret the effect as causal if one believes that all the important confounding variables have been accounted for. We think we have controlled for the most important confounders (athlete and marathon) but can't rule out the possibility that other confounding factors exist, which is why we mentioned cost and fitness.</td>\n",
       "      <td id=\"T_03ec2_row219_col1\" class=\"data row219 col1\" >What is the purpose of causal inference in an observational study?</td>\n",
       "      <td id=\"T_03ec2_row219_col2\" class=\"data row219 col2\" >The purpose of causal inference in an observational study is to control for confounders.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row220\" class=\"row_heading level0 row220\" >220</th>\n",
       "      <td id=\"T_03ec2_row220_col0\" class=\"data row220 col0\" >As we understand it, causal inference is an attempt to control for confounders in an observational study. In that sense, one can interpret the effect as causal if one believes that all the important confounding variables have been accounted for. We think we have controlled for the most important confounders (athlete and marathon) but can't rule out the possibility that other confounding factors exist, which is why we mentioned cost and fitness.</td>\n",
       "      <td id=\"T_03ec2_row220_col1\" class=\"data row220 col1\" >Which confounders do the researchers believe they have controlled for?</td>\n",
       "      <td id=\"T_03ec2_row220_col2\" class=\"data row220 col2\" >The researchers believe they have controlled for the most important confounders, which are athlete and marathon.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row221\" class=\"row_heading level0 row221\" >221</th>\n",
       "      <td id=\"T_03ec2_row221_col0\" class=\"data row221 col0\" >As we understand it, causal inference is an attempt to control for confounders in an observational study. In that sense, one can interpret the effect as causal if one believes that all the important confounding variables have been accounted for. We think we have controlled for the most important confounders (athlete and marathon) but can't rule out the possibility that other confounding factors exist, which is why we mentioned cost and fitness.</td>\n",
       "      <td id=\"T_03ec2_row221_col1\" class=\"data row221 col1\" >Why can't the researchers definitively say that they've accounted for all confounding variables?</td>\n",
       "      <td id=\"T_03ec2_row221_col2\" class=\"data row221 col2\" >The researchers can't definitively say they've accounted for all confounding variables because there is a possibility that other confounding factors exist, such as cost and fitness.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row222\" class=\"row_heading level0 row222\" >222</th>\n",
       "      <td id=\"T_03ec2_row222_col0\" class=\"data row222 col0\" >Judging by the widespread adoption of the Vaporfly shoes, runners seem to think that there is a causal effect. Some runners at the 2020 US Olympic Trials marathon even wore the new Nike Alphafly shoes after receiving them a day or two before the race. Conventional running wisdom states that you should never race in a brand new shoe that you haven't tried before. In our opinion, this break from conventional wisdom suggests that runners think the effects are causal.</td>\n",
       "      <td id=\"T_03ec2_row222_col1\" class=\"data row222 col1\" >What do runners think about the Vaporfly shoes?</td>\n",
       "      <td id=\"T_03ec2_row222_col2\" class=\"data row222 col2\" >Runners seem to think that there is a causal effect.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row223\" class=\"row_heading level0 row223\" >223</th>\n",
       "      <td id=\"T_03ec2_row223_col0\" class=\"data row223 col0\" >Judging by the widespread adoption of the Vaporfly shoes, runners seem to think that there is a causal effect. Some runners at the 2020 US Olympic Trials marathon even wore the new Nike Alphafly shoes after receiving them a day or two before the race. Conventional running wisdom states that you should never race in a brand new shoe that you haven't tried before. In our opinion, this break from conventional wisdom suggests that runners think the effects are causal.</td>\n",
       "      <td id=\"T_03ec2_row223_col1\" class=\"data row223 col1\" >What unusual choice did some runners make at the 2020 US Olympic Trials marathon?</td>\n",
       "      <td id=\"T_03ec2_row223_col2\" class=\"data row223 col2\" >Some runners chose to wear the new Nike Alphafly shoes after receiving them only a day or two before the race.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row224\" class=\"row_heading level0 row224\" >224</th>\n",
       "      <td id=\"T_03ec2_row224_col0\" class=\"data row224 col0\" >Judging by the widespread adoption of the Vaporfly shoes, runners seem to think that there is a causal effect. Some runners at the 2020 US Olympic Trials marathon even wore the new Nike Alphafly shoes after receiving them a day or two before the race. Conventional running wisdom states that you should never race in a brand new shoe that you haven't tried before. In our opinion, this break from conventional wisdom suggests that runners think the effects are causal.</td>\n",
       "      <td id=\"T_03ec2_row224_col1\" class=\"data row224 col1\" >What does conventional running wisdom advise against?</td>\n",
       "      <td id=\"T_03ec2_row224_col2\" class=\"data row224 col2\" >Conventional running wisdom advises against racing in a brand new shoe that you haven't tried before.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row225\" class=\"row_heading level0 row225\" >225</th>\n",
       "      <td id=\"T_03ec2_row225_col0\" class=\"data row225 col0\" >4. I'm curious whether any of the authors have actually worn the Vaporfly. Can the authors provide any anecdotal evidence about their experience with the shoe, particularly in comparison to their experience with other (apparently inferior) shoes.</td>\n",
       "      <td id=\"T_03ec2_row225_col1\" class=\"data row225 col1\" >What is the person curious about regarding the authors?</td>\n",
       "      <td id=\"T_03ec2_row225_col2\" class=\"data row225 col2\" >Whether any of the authors have worn the Vaporfly.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row226\" class=\"row_heading level0 row226\" >226</th>\n",
       "      <td id=\"T_03ec2_row226_col0\" class=\"data row226 col0\" >4. I'm curious whether any of the authors have actually worn the Vaporfly. Can the authors provide any anecdotal evidence about their experience with the shoe, particularly in comparison to their experience with other (apparently inferior) shoes.</td>\n",
       "      <td id=\"T_03ec2_row226_col1\" class=\"data row226 col1\" >What kind of evidence is being sought from the authors?</td>\n",
       "      <td id=\"T_03ec2_row226_col2\" class=\"data row226 col2\" >Anecdotal evidence about their experience with the Vaporfly.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row227\" class=\"row_heading level0 row227\" >227</th>\n",
       "      <td id=\"T_03ec2_row227_col0\" class=\"data row227 col0\" >4. I'm curious whether any of the authors have actually worn the Vaporfly. Can the authors provide any anecdotal evidence about their experience with the shoe, particularly in comparison to their experience with other (apparently inferior) shoes.</td>\n",
       "      <td id=\"T_03ec2_row227_col1\" class=\"data row227 col1\" >What is the Vaporfly being compared to?</td>\n",
       "      <td id=\"T_03ec2_row227_col2\" class=\"data row227 col2\" >Other shoes that are apparently inferior.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row228\" class=\"row_heading level0 row228\" >228</th>\n",
       "      <td id=\"T_03ec2_row228_col0\" class=\"data row228 col0\" >We haven't worn them yet, though several runners have provided us with anecdotal evidence that the shoes improve their performances.</td>\n",
       "      <td id=\"T_03ec2_row228_col1\" class=\"data row228 col1\" >Have the speakers worn the shoes yet?</td>\n",
       "      <td id=\"T_03ec2_row228_col2\" class=\"data row228 col2\" >No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row229\" class=\"row_heading level0 row229\" >229</th>\n",
       "      <td id=\"T_03ec2_row229_col0\" class=\"data row229 col0\" >We haven't worn them yet, though several runners have provided us with anecdotal evidence that the shoes improve their performances.</td>\n",
       "      <td id=\"T_03ec2_row229_col1\" class=\"data row229 col1\" >Who has provided evidence about the shoes?</td>\n",
       "      <td id=\"T_03ec2_row229_col2\" class=\"data row229 col2\" >Several runners</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row230\" class=\"row_heading level0 row230\" >230</th>\n",
       "      <td id=\"T_03ec2_row230_col0\" class=\"data row230 col0\" >We haven't worn them yet, though several runners have provided us with anecdotal evidence that the shoes improve their performances.</td>\n",
       "      <td id=\"T_03ec2_row230_col1\" class=\"data row230 col1\" >What type of evidence suggests the shoes improve performances?</td>\n",
       "      <td id=\"T_03ec2_row230_col2\" class=\"data row230 col2\" >Anecdotal evidence</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row231\" class=\"row_heading level0 row231\" >231</th>\n",
       "      <td id=\"T_03ec2_row231_col0\" class=\"data row231 col0\" >This article presents a clear and well-written study of the association between wearing the Vaporfly running shoe and race time in the context of elite marathon runners. I have several comments.</td>\n",
       "      <td id=\"T_03ec2_row231_col1\" class=\"data row231 col1\" >What is the main focus of the study mentioned in the article?</td>\n",
       "      <td id=\"T_03ec2_row231_col2\" class=\"data row231 col2\" >The association between wearing the Vaporfly running shoe and race time in elite marathon runners.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row232\" class=\"row_heading level0 row232\" >232</th>\n",
       "      <td id=\"T_03ec2_row232_col0\" class=\"data row232 col0\" >This article presents a clear and well-written study of the association between wearing the Vaporfly running shoe and race time in the context of elite marathon runners. I have several comments.</td>\n",
       "      <td id=\"T_03ec2_row232_col1\" class=\"data row232 col1\" >What is the general opinion of the author about the article?</td>\n",
       "      <td id=\"T_03ec2_row232_col2\" class=\"data row232 col2\" >The author thinks the article is clear and well-written.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row233\" class=\"row_heading level0 row233\" >233</th>\n",
       "      <td id=\"T_03ec2_row233_col0\" class=\"data row233 col0\" >This article presents a clear and well-written study of the association between wearing the Vaporfly running shoe and race time in the context of elite marathon runners. I have several comments.</td>\n",
       "      <td id=\"T_03ec2_row233_col1\" class=\"data row233 col1\" >What does the author intend to do after reading the article?</td>\n",
       "      <td id=\"T_03ec2_row233_col2\" class=\"data row233 col2\" >The author intends to provide several comments.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row234\" class=\"row_heading level0 row234\" >234</th>\n",
       "      <td id=\"T_03ec2_row234_col0\" class=\"data row234 col0\" >1. Why have separate models for women and men? Once you control for individual runner effects, doesn't this automatically include sex differences? I would expect that putting all the runners in a single model would improve power and also allow a formal assessment of the interaction of sex and Vaporfly effect via an interaction of the two.</td>\n",
       "      <td id=\"T_03ec2_row234_col1\" class=\"data row234 col1\" >What is the expected benefit of combining women and men in a single model?</td>\n",
       "      <td id=\"T_03ec2_row234_col2\" class=\"data row234 col2\" >It would improve power and allow a formal assessment of the interaction of sex and Vaporfly effect.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row235\" class=\"row_heading level0 row235\" >235</th>\n",
       "      <td id=\"T_03ec2_row235_col0\" class=\"data row235 col0\" >1. Why have separate models for women and men? Once you control for individual runner effects, doesn't this automatically include sex differences? I would expect that putting all the runners in a single model would improve power and also allow a formal assessment of the interaction of sex and Vaporfly effect via an interaction of the two.</td>\n",
       "      <td id=\"T_03ec2_row235_col1\" class=\"data row235 col1\" >What does controlling for individual runner effects imply about sex differences?</td>\n",
       "      <td id=\"T_03ec2_row235_col2\" class=\"data row235 col2\" >It implies that sex differences are automatically included when individual runner effects are controlled for.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row236\" class=\"row_heading level0 row236\" >236</th>\n",
       "      <td id=\"T_03ec2_row236_col0\" class=\"data row236 col0\" >1. Why have separate models for women and men? Once you control for individual runner effects, doesn't this automatically include sex differences? I would expect that putting all the runners in a single model would improve power and also allow a formal assessment of the interaction of sex and Vaporfly effect via an interaction of the two.</td>\n",
       "      <td id=\"T_03ec2_row236_col1\" class=\"data row236 col1\" >What interaction is suggested to be formally assessed in a combined model?</td>\n",
       "      <td id=\"T_03ec2_row236_col2\" class=\"data row236 col2\" >The interaction of sex and Vaporfly effect.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row237\" class=\"row_heading level0 row237\" >237</th>\n",
       "      <td id=\"T_03ec2_row237_col0\" class=\"data row237 col0\" >This is a good question and suggestion. Our main reason for fitting separate models is that we wanted to avoid assuming that the variances of the random effects terms were the same for men and women. Looking at the table of parameter estimates, in the untransformed model, the women's standard deviations are all higher than the men's standard deviations, which is not surprising because the women's times are overall larger than the men's. For the log transformed model, the standard deviations are much closer, with the exception of the runner standard deviations, which are about 25% larger for the women.</td>\n",
       "      <td id=\"T_03ec2_row237_col1\" class=\"data row237 col1\" >Why were separate models fitted for men and women?</td>\n",
       "      <td id=\"T_03ec2_row237_col2\" class=\"data row237 col2\" >To avoid assuming that the variances of the random effects terms were the same for men and women.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row238\" class=\"row_heading level0 row238\" >238</th>\n",
       "      <td id=\"T_03ec2_row238_col0\" class=\"data row238 col0\" >This is a good question and suggestion. Our main reason for fitting separate models is that we wanted to avoid assuming that the variances of the random effects terms were the same for men and women. Looking at the table of parameter estimates, in the untransformed model, the women's standard deviations are all higher than the men's standard deviations, which is not surprising because the women's times are overall larger than the men's. For the log transformed model, the standard deviations are much closer, with the exception of the runner standard deviations, which are about 25% larger for the women.</td>\n",
       "      <td id=\"T_03ec2_row238_col1\" class=\"data row238 col1\" >In the untransformed model, how do the women's standard deviations compare to the men's?</td>\n",
       "      <td id=\"T_03ec2_row238_col2\" class=\"data row238 col2\" >The women's standard deviations are all higher than the men's standard deviations.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row239\" class=\"row_heading level0 row239\" >239</th>\n",
       "      <td id=\"T_03ec2_row239_col0\" class=\"data row239 col0\" >This is a good question and suggestion. Our main reason for fitting separate models is that we wanted to avoid assuming that the variances of the random effects terms were the same for men and women. Looking at the table of parameter estimates, in the untransformed model, the women's standard deviations are all higher than the men's standard deviations, which is not surprising because the women's times are overall larger than the men's. For the log transformed model, the standard deviations are much closer, with the exception of the runner standard deviations, which are about 25% larger for the women.</td>\n",
       "      <td id=\"T_03ec2_row239_col1\" class=\"data row239 col1\" >In the log transformed model, which standard deviations are about 25% larger for women?</td>\n",
       "      <td id=\"T_03ec2_row239_col2\" class=\"data row239 col2\" >The runner standard deviations.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row240\" class=\"row_heading level0 row240\" >240</th>\n",
       "      <td id=\"T_03ec2_row240_col0\" class=\"data row240 col0\" >Thanks to your suggestion, we fit the interaction model and include the results in the table below, followed by the table of confidence intervals.</td>\n",
       "      <td id=\"T_03ec2_row240_col1\" class=\"data row240 col1\" >What was done as a result of the suggestion?</td>\n",
       "      <td id=\"T_03ec2_row240_col2\" class=\"data row240 col2\" >The interaction model was fit and the results were included in the table below.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row241\" class=\"row_heading level0 row241\" >241</th>\n",
       "      <td id=\"T_03ec2_row241_col0\" class=\"data row241 col0\" >Thanks to your suggestion, we fit the interaction model and include the results in the table below, followed by the table of confidence intervals.</td>\n",
       "      <td id=\"T_03ec2_row241_col1\" class=\"data row241 col1\" >What follows the results in the document?</td>\n",
       "      <td id=\"T_03ec2_row241_col2\" class=\"data row241 col2\" >The table of confidence intervals follows the results.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row242\" class=\"row_heading level0 row242\" >242</th>\n",
       "      <td id=\"T_03ec2_row242_col0\" class=\"data row242 col0\" >Thanks to your suggestion, we fit the interaction model and include the results in the table below, followed by the table of confidence intervals.</td>\n",
       "      <td id=\"T_03ec2_row242_col1\" class=\"data row242 col1\" >What type of model was fit?</td>\n",
       "      <td id=\"T_03ec2_row242_col2\" class=\"data row242 col2\" >An interaction model was fit.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row243\" class=\"row_heading level0 row243\" >243</th>\n",
       "      <td id=\"T_03ec2_row243_col0\" class=\"data row243 col0\" >The model is parameterized so that \\(b_{1}\\) is the men's Vaporfly effect, and \\(b_{1}+b_{3}\\) is the women's effect, giving men's effects of \\(-3.35\\) and \\(-0.0229\\) for the untransformed and log-transformed models, and women's effects of \\(-2.63\\) and \\(-0.0167\\). These effects are all larger in magnitude than those from the sex-separated analyses. We do not find statistical evidence that the difference between the men's and women's effects, \\(b_{3}\\), is different from zero, as the standard error of the estimates are comparable to the estimates in both models.</td>\n",
       "      <td id=\"T_03ec2_row243_col1\" class=\"data row243 col1\" >What does \\(b_{1}\\) represent in the model?</td>\n",
       "      <td id=\"T_03ec2_row243_col2\" class=\"data row243 col2\" >The men's Vaporfly effect</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row244\" class=\"row_heading level0 row244\" >244</th>\n",
       "      <td id=\"T_03ec2_row244_col0\" class=\"data row244 col0\" >The model is parameterized so that \\(b_{1}\\) is the men's Vaporfly effect, and \\(b_{1}+b_{3}\\) is the women's effect, giving men's effects of \\(-3.35\\) and \\(-0.0229\\) for the untransformed and log-transformed models, and women's effects of \\(-2.63\\) and \\(-0.0167\\). These effects are all larger in magnitude than those from the sex-separated analyses. We do not find statistical evidence that the difference between the men's and women's effects, \\(b_{3}\\), is different from zero, as the standard error of the estimates are comparable to the estimates in both models.</td>\n",
       "      <td id=\"T_03ec2_row244_col1\" class=\"data row244 col1\" >What are the values of the men's effects in the untransformed and log-transformed models?</td>\n",
       "      <td id=\"T_03ec2_row244_col2\" class=\"data row244 col2\" >-3.35 for the untransformed model and -0.0229 for the log-transformed model</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row245\" class=\"row_heading level0 row245\" >245</th>\n",
       "      <td id=\"T_03ec2_row245_col0\" class=\"data row245 col0\" >The model is parameterized so that \\(b_{1}\\) is the men's Vaporfly effect, and \\(b_{1}+b_{3}\\) is the women's effect, giving men's effects of \\(-3.35\\) and \\(-0.0229\\) for the untransformed and log-transformed models, and women's effects of \\(-2.63\\) and \\(-0.0167\\). These effects are all larger in magnitude than those from the sex-separated analyses. We do not find statistical evidence that the difference between the men's and women's effects, \\(b_{3}\\), is different from zero, as the standard error of the estimates are comparable to the estimates in both models.</td>\n",
       "      <td id=\"T_03ec2_row245_col1\" class=\"data row245 col1\" >Is there statistical evidence that the difference between men's and women's effects is significant?</td>\n",
       "      <td id=\"T_03ec2_row245_col2\" class=\"data row245 col2\" >No, there is no statistical evidence that the difference is significant as the standard error of the estimates are comparable to the estimates in both models.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row246\" class=\"row_heading level0 row246\" >246</th>\n",
       "      <td id=\"T_03ec2_row246_col0\" class=\"data row246 col0\" >As Harry Crane pointed out, controlling for marathon effects doesn't account for variation in marathon conditions that vary year-to-year. I would expect this to substantively impact the conclusions only if marathon conditions are also associated with the choice to wear Vaporflys. I would be interested to know whether the authors expect this to be the case. Relatedly, as part of the data exploration it would also be interesting to see a plot of the proportion of runners wearing Vaporflys in each marathon over time.</td>\n",
       "      <td id=\"T_03ec2_row246_col1\" class=\"data row246 col1\" >What does Harry Crane suggest is not accounted for when controlling for marathon effects?</td>\n",
       "      <td id=\"T_03ec2_row246_col2\" class=\"data row246 col2\" >Variation in marathon conditions that vary year-to-year.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row247\" class=\"row_heading level0 row247\" >247</th>\n",
       "      <td id=\"T_03ec2_row247_col0\" class=\"data row247 col0\" >As Harry Crane pointed out, controlling for marathon effects doesn't account for variation in marathon conditions that vary year-to-year. I would expect this to substantively impact the conclusions only if marathon conditions are also associated with the choice to wear Vaporflys. I would be interested to know whether the authors expect this to be the case. Relatedly, as part of the data exploration it would also be interesting to see a plot of the proportion of runners wearing Vaporflys in each marathon over time.</td>\n",
       "      <td id=\"T_03ec2_row247_col1\" class=\"data row247 col1\" >Under what condition does the speaker expect variations in marathon conditions to impact the conclusions?</td>\n",
       "      <td id=\"T_03ec2_row247_col2\" class=\"data row247 col2\" >Only if marathon conditions are also associated with the choice to wear Vaporflys.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row248\" class=\"row_heading level0 row248\" >248</th>\n",
       "      <td id=\"T_03ec2_row248_col0\" class=\"data row248 col0\" >As Harry Crane pointed out, controlling for marathon effects doesn't account for variation in marathon conditions that vary year-to-year. I would expect this to substantively impact the conclusions only if marathon conditions are also associated with the choice to wear Vaporflys. I would be interested to know whether the authors expect this to be the case. Relatedly, as part of the data exploration it would also be interesting to see a plot of the proportion of runners wearing Vaporflys in each marathon over time.</td>\n",
       "      <td id=\"T_03ec2_row248_col1\" class=\"data row248 col1\" >What additional information does the speaker express interest in seeing?</td>\n",
       "      <td id=\"T_03ec2_row248_col2\" class=\"data row248 col2\" >A plot of the proportion of runners wearing Vaporflys in each marathon over time.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row249\" class=\"row_heading level0 row249\" >249</th>\n",
       "      <td id=\"T_03ec2_row249_col0\" class=\"data row249 col0\" >This is probably a miscommunication on our part. We do control for variation from year to year in the same marathon. In the revised version, we have added effects for each individual course, which do not vary from year to year. Hopefully, now that we have both types of effects and describe them separately, the description of the effects is more clear.</td>\n",
       "      <td id=\"T_03ec2_row249_col1\" class=\"data row249 col1\" >What do the authors control for in the marathon study?</td>\n",
       "      <td id=\"T_03ec2_row249_col2\" class=\"data row249 col2\" >Variation from year to year in the same marathon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row250\" class=\"row_heading level0 row250\" >250</th>\n",
       "      <td id=\"T_03ec2_row250_col0\" class=\"data row250 col0\" >This is probably a miscommunication on our part. We do control for variation from year to year in the same marathon. In the revised version, we have added effects for each individual course, which do not vary from year to year. Hopefully, now that we have both types of effects and describe them separately, the description of the effects is more clear.</td>\n",
       "      <td id=\"T_03ec2_row250_col1\" class=\"data row250 col1\" >What has been added to the revised version of the study?</td>\n",
       "      <td id=\"T_03ec2_row250_col2\" class=\"data row250 col2\" >Effects for each individual course, which do not vary from year to year</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row251\" class=\"row_heading level0 row251\" >251</th>\n",
       "      <td id=\"T_03ec2_row251_col0\" class=\"data row251 col0\" >This is probably a miscommunication on our part. We do control for variation from year to year in the same marathon. In the revised version, we have added effects for each individual course, which do not vary from year to year. Hopefully, now that we have both types of effects and describe them separately, the description of the effects is more clear.</td>\n",
       "      <td id=\"T_03ec2_row251_col1\" class=\"data row251 col1\" >What is the purpose of describing the two types of effects separately?</td>\n",
       "      <td id=\"T_03ec2_row251_col2\" class=\"data row251 col2\" >To make the description of the effects more clear</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row252\" class=\"row_heading level0 row252\" >252</th>\n",
       "      <td id=\"T_03ec2_row252_col0\" class=\"data row252 col0\" >Thank you for the suggestion for the plot. We have replaced our original plot with one that contains the proportion of runners wearing Vaporflys in each race as a function of race date.</td>\n",
       "      <td id=\"T_03ec2_row252_col1\" class=\"data row252 col1\" >What was replaced in the plot?</td>\n",
       "      <td id=\"T_03ec2_row252_col2\" class=\"data row252 col2\" >The original plot was replaced with one that shows the proportion of runners wearing Vaporflys in each race.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row253\" class=\"row_heading level0 row253\" >253</th>\n",
       "      <td id=\"T_03ec2_row253_col0\" class=\"data row253 col0\" >Thank you for the suggestion for the plot. We have replaced our original plot with one that contains the proportion of runners wearing Vaporflys in each race as a function of race date.</td>\n",
       "      <td id=\"T_03ec2_row253_col1\" class=\"data row253 col1\" >What does the new plot show?</td>\n",
       "      <td id=\"T_03ec2_row253_col2\" class=\"data row253 col2\" >The new plot shows the proportion of runners wearing Vaporflys in each race as a function of race date.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row254\" class=\"row_heading level0 row254\" >254</th>\n",
       "      <td id=\"T_03ec2_row254_col0\" class=\"data row254 col0\" >Thank you for the suggestion for the plot. We have replaced our original plot with one that contains the proportion of runners wearing Vaporflys in each race as a function of race date.</td>\n",
       "      <td id=\"T_03ec2_row254_col1\" class=\"data row254 col1\" >Why was the original plot replaced?</td>\n",
       "      <td id=\"T_03ec2_row254_col2\" class=\"data row254 col2\" >The original plot was replaced following a suggestion for improvement.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row255\" class=\"row_heading level0 row255\" >255</th>\n",
       "      <td id=\"T_03ec2_row255_col0\" class=\"data row255 col0\" >3. Regarding the data collection/sampling strategy for the study: were elite runners first identified, and subsequently all marathon performances for these runners recorded? Or were marathons first identified, and only the performances meeting the stated time criteria included in the study? In the second strategy, a runner could theoretically be in the data for one marathon, but excluded for another due to their time not qualifying. In this case, I'd also be curious about whether and how the truncation in the sampling strategy might impact the results.</td>\n",
       "      <td id=\"T_03ec2_row255_col1\" class=\"data row255 col1\" >What are the two possible sampling strategies mentioned for the study?</td>\n",
       "      <td id=\"T_03ec2_row255_col2\" class=\"data row255 col2\" >The first strategy is identifying elite runners and recording all their marathon performances. The second strategy is identifying marathons and including only the performances that meet the stated time criteria.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row256\" class=\"row_heading level0 row256\" >256</th>\n",
       "      <td id=\"T_03ec2_row256_col0\" class=\"data row256 col0\" >3. Regarding the data collection/sampling strategy for the study: were elite runners first identified, and subsequently all marathon performances for these runners recorded? Or were marathons first identified, and only the performances meeting the stated time criteria included in the study? In the second strategy, a runner could theoretically be in the data for one marathon, but excluded for another due to their time not qualifying. In this case, I'd also be curious about whether and how the truncation in the sampling strategy might impact the results.</td>\n",
       "      <td id=\"T_03ec2_row256_col1\" class=\"data row256 col1\" >What could happen to a runner's data in the second sampling strategy?</td>\n",
       "      <td id=\"T_03ec2_row256_col2\" class=\"data row256 col2\" >In the second sampling strategy, a runner's data could be included for one marathon if their time qualifies, but excluded for another marathon if their time does not qualify.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row257\" class=\"row_heading level0 row257\" >257</th>\n",
       "      <td id=\"T_03ec2_row257_col0\" class=\"data row257 col0\" >3. Regarding the data collection/sampling strategy for the study: were elite runners first identified, and subsequently all marathon performances for these runners recorded? Or were marathons first identified, and only the performances meeting the stated time criteria included in the study? In the second strategy, a runner could theoretically be in the data for one marathon, but excluded for another due to their time not qualifying. In this case, I'd also be curious about whether and how the truncation in the sampling strategy might impact the results.</td>\n",
       "      <td id=\"T_03ec2_row257_col1\" class=\"data row257 col1\" >What concern is raised about the impact of the sampling strategy?</td>\n",
       "      <td id=\"T_03ec2_row257_col2\" class=\"data row257 col2\" >The concern is about how the truncation in the sampling strategy might impact the results of the study.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row258\" class=\"row_heading level0 row258\" >258</th>\n",
       "      <td id=\"T_03ec2_row258_col0\" class=\"data row258 col0\" >We first selected a sample of marathons consisting of 22 of the most competitive marathons in the United States and Canada. Then from those marathons, we selected runners who</td>\n",
       "      <td id=\"T_03ec2_row258_col1\" class=\"data row258 col1\" >How many marathons were selected for the sample?</td>\n",
       "      <td id=\"T_03ec2_row258_col2\" class=\"data row258 col2\" >22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row259\" class=\"row_heading level0 row259\" >259</th>\n",
       "      <td id=\"T_03ec2_row259_col0\" class=\"data row259 col0\" >We first selected a sample of marathons consisting of 22 of the most competitive marathons in the United States and Canada. Then from those marathons, we selected runners who</td>\n",
       "      <td id=\"T_03ec2_row259_col1\" class=\"data row259 col1\" >What was the criteria for the marathons selected for the sample?</td>\n",
       "      <td id=\"T_03ec2_row259_col2\" class=\"data row259 col2\" >The most competitive marathons in the United States and Canada</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row260\" class=\"row_heading level0 row260\" >260</th>\n",
       "      <td id=\"T_03ec2_row260_col0\" class=\"data row260 col0\" >We first selected a sample of marathons consisting of 22 of the most competitive marathons in the United States and Canada. Then from those marathons, we selected runners who</td>\n",
       "      <td id=\"T_03ec2_row260_col1\" class=\"data row260 col1\" >Who was selected from the marathons?</td>\n",
       "      <td id=\"T_03ec2_row260_col2\" class=\"data row260 col2\" >Runners</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row261\" class=\"row_heading level0 row261\" >261</th>\n",
       "      <td id=\"T_03ec2_row261_col0\" class=\"data row261 col0\" >\\begin{table}\n",
       "\\begin{tabular}{l c c c} men minutes & women minutes & men log minutes & women log minutes \\\\ \\hline \\((-4.410,-2.298)\\) & \\((-3.786,-1.478)\\) & \\((-0.030,-0.016)\\) & \\((-0.024,-0.009)\\) \\\\ \\end{tabular}\n",
       "\\end{table}\n",
       "Table 4: 90% confidence intervals for Vaporfly effects in each model.</td>\n",
       "      <td id=\"T_03ec2_row261_col1\" class=\"data row261 col1\" >What is the 90% confidence interval for Vaporfly effects on men's minutes?</td>\n",
       "      <td id=\"T_03ec2_row261_col2\" class=\"data row261 col2\" >(-4.410, -2.298)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row262\" class=\"row_heading level0 row262\" >262</th>\n",
       "      <td id=\"T_03ec2_row262_col0\" class=\"data row262 col0\" >\\begin{table}\n",
       "\\begin{tabular}{l c c c} men minutes & women minutes & men log minutes & women log minutes \\\\ \\hline \\((-4.410,-2.298)\\) & \\((-3.786,-1.478)\\) & \\((-0.030,-0.016)\\) & \\((-0.024,-0.009)\\) \\\\ \\end{tabular}\n",
       "\\end{table}\n",
       "Table 4: 90% confidence intervals for Vaporfly effects in each model.</td>\n",
       "      <td id=\"T_03ec2_row262_col1\" class=\"data row262 col1\" >What is the range of the 90% confidence interval for Vaporfly effects on women's log minutes?</td>\n",
       "      <td id=\"T_03ec2_row262_col2\" class=\"data row262 col2\" >(-0.024, -0.009)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row263\" class=\"row_heading level0 row263\" >263</th>\n",
       "      <td id=\"T_03ec2_row263_col0\" class=\"data row263 col0\" >\\begin{table}\n",
       "\\begin{tabular}{l c c c} men minutes & women minutes & men log minutes & women log minutes \\\\ \\hline \\((-4.410,-2.298)\\) & \\((-3.786,-1.478)\\) & \\((-0.030,-0.016)\\) & \\((-0.024,-0.009)\\) \\\\ \\end{tabular}\n",
       "\\end{table}\n",
       "Table 4: 90% confidence intervals for Vaporfly effects in each model.</td>\n",
       "      <td id=\"T_03ec2_row263_col1\" class=\"data row263 col1\" >How many models are used to present the Vaporfly effects in Table 4?</td>\n",
       "      <td id=\"T_03ec2_row263_col2\" class=\"data row263 col2\" >Four models: men minutes, women minutes, men log minutes, and women log minutes.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row264\" class=\"row_heading level0 row264\" >264</th>\n",
       "      <td id=\"T_03ec2_row264_col0\" class=\"data row264 col0\" >\\begin{table}\n",
       "\\begin{tabular}{l r r}  & minutes & log minutes \\\\ \\cline{2-3}  & estimate (s.e.) & estimate (s.e.) \\\\ \\hline \\(b_{0}\\) & 140.27 (0.66) & 4.94 (0.0043) \\\\ \\(b_{1}\\) & \\(-3.35\\) (0.64) & \\(-0.0229\\) 0.0041 \\\\ \\(b_{2}\\) & 19.89 (0.53) & 0.1336 0.0035 \\\\ \\(b_{3}\\) & 0.72 (0.86) & 0.0062 0.0055 \\\\ \\(\\sigma_{1}\\) & 5.265 & 0.0350 \\\\ \\(\\sigma_{2}\\) & 2.110 & 0.0137 \\\\ \\(\\sigma_{3}\\) & 2.239 & 0.0144 \\\\ \\(\\sigma_{4}\\) & 4.604 & 0.0292 \\\\ \\end{tabular}\n",
       "\\end{table}\n",
       "Table 3: Table of parameter estimates for the interaction models </td>\n",
       "      <td id=\"T_03ec2_row264_col1\" class=\"data row264 col1\" >What is the estimate and standard error for the parameter \\(b_{0}\\) in the 'minutes' column?</td>\n",
       "      <td id=\"T_03ec2_row264_col2\" class=\"data row264 col2\" >The estimate for \\(b_{0}\\) is 140.27 and the standard error is 0.66.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row265\" class=\"row_heading level0 row265\" >265</th>\n",
       "      <td id=\"T_03ec2_row265_col0\" class=\"data row265 col0\" >\\begin{table}\n",
       "\\begin{tabular}{l r r}  & minutes & log minutes \\\\ \\cline{2-3}  & estimate (s.e.) & estimate (s.e.) \\\\ \\hline \\(b_{0}\\) & 140.27 (0.66) & 4.94 (0.0043) \\\\ \\(b_{1}\\) & \\(-3.35\\) (0.64) & \\(-0.0229\\) 0.0041 \\\\ \\(b_{2}\\) & 19.89 (0.53) & 0.1336 0.0035 \\\\ \\(b_{3}\\) & 0.72 (0.86) & 0.0062 0.0055 \\\\ \\(\\sigma_{1}\\) & 5.265 & 0.0350 \\\\ \\(\\sigma_{2}\\) & 2.110 & 0.0137 \\\\ \\(\\sigma_{3}\\) & 2.239 & 0.0144 \\\\ \\(\\sigma_{4}\\) & 4.604 & 0.0292 \\\\ \\end{tabular}\n",
       "\\end{table}\n",
       "Table 3: Table of parameter estimates for the interaction models </td>\n",
       "      <td id=\"T_03ec2_row265_col1\" class=\"data row265 col1\" >How does the estimate for \\(b_{1}\\) in the 'log minutes' column compare to its standard error?</td>\n",
       "      <td id=\"T_03ec2_row265_col2\" class=\"data row265 col2\" >The estimate for \\(b_{1}\\) in the 'log minutes' column is \\(-0.0229\\), which is smaller in magnitude than its standard error of 0.0041.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row266\" class=\"row_heading level0 row266\" >266</th>\n",
       "      <td id=\"T_03ec2_row266_col0\" class=\"data row266 col0\" >\\begin{table}\n",
       "\\begin{tabular}{l r r}  & minutes & log minutes \\\\ \\cline{2-3}  & estimate (s.e.) & estimate (s.e.) \\\\ \\hline \\(b_{0}\\) & 140.27 (0.66) & 4.94 (0.0043) \\\\ \\(b_{1}\\) & \\(-3.35\\) (0.64) & \\(-0.0229\\) 0.0041 \\\\ \\(b_{2}\\) & 19.89 (0.53) & 0.1336 0.0035 \\\\ \\(b_{3}\\) & 0.72 (0.86) & 0.0062 0.0055 \\\\ \\(\\sigma_{1}\\) & 5.265 & 0.0350 \\\\ \\(\\sigma_{2}\\) & 2.110 & 0.0137 \\\\ \\(\\sigma_{3}\\) & 2.239 & 0.0144 \\\\ \\(\\sigma_{4}\\) & 4.604 & 0.0292 \\\\ \\end{tabular}\n",
       "\\end{table}\n",
       "Table 3: Table of parameter estimates for the interaction models </td>\n",
       "      <td id=\"T_03ec2_row266_col1\" class=\"data row266 col1\" >Which \\(\\sigma\\) value is the largest in the 'minutes' column and what is its value?</td>\n",
       "      <td id=\"T_03ec2_row266_col2\" class=\"data row266 col2\" >\\(\\sigma_{1}\\) has the largest value in the 'minutes' column, which is 5.265.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row267\" class=\"row_heading level0 row267\" >267</th>\n",
       "      <td id=\"T_03ec2_row267_col0\" class=\"data row267 col0\" >achieved a minimum performance standard in 2015 or 2016 in at least one of the 22 selected marathons. Then we selected ALL performances from those runners, regardless of the time, in those 22 marathons over the period 2015 to 2019.</td>\n",
       "      <td id=\"T_03ec2_row267_col1\" class=\"data row267 col1\" >What was the criteria for selecting performances from runners?</td>\n",
       "      <td id=\"T_03ec2_row267_col2\" class=\"data row267 col2\" >Runners who achieved a minimum performance standard in 2015 or 2016 in at least one of the 22 selected marathons.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row268\" class=\"row_heading level0 row268\" >268</th>\n",
       "      <td id=\"T_03ec2_row268_col0\" class=\"data row268 col0\" >achieved a minimum performance standard in 2015 or 2016 in at least one of the 22 selected marathons. Then we selected ALL performances from those runners, regardless of the time, in those 22 marathons over the period 2015 to 2019.</td>\n",
       "      <td id=\"T_03ec2_row268_col1\" class=\"data row268 col1\" >Did the time of the performances affect the selection?</td>\n",
       "      <td id=\"T_03ec2_row268_col2\" class=\"data row268 col2\" >No, all performances were selected regardless of the time.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row269\" class=\"row_heading level0 row269\" >269</th>\n",
       "      <td id=\"T_03ec2_row269_col0\" class=\"data row269 col0\" >achieved a minimum performance standard in 2015 or 2016 in at least one of the 22 selected marathons. Then we selected ALL performances from those runners, regardless of the time, in those 22 marathons over the period 2015 to 2019.</td>\n",
       "      <td id=\"T_03ec2_row269_col1\" class=\"data row269 col1\" >Over what period were the performances of the selected runners considered?</td>\n",
       "      <td id=\"T_03ec2_row269_col2\" class=\"data row269 col2\" >From 2015 to 2019.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row270\" class=\"row_heading level0 row270\" >270</th>\n",
       "      <td id=\"T_03ec2_row270_col0\" class=\"data row270 col0\" >It would probably be better to first create a dataset from all marathons in 2015 and 2016, and then find runners that met the minimum performance standard in any marathon in 2015 and 2016. There is some manual work to be done to get such a dataset, because one needs to know the url on marathonguide.com for every marathon. We selected the 22 marathons to include those where runners had qualified for the U.S. Olympic trials, and marathons that had many Boston Marathon Qualifiers, so we think our method misses very few elite runners.</td>\n",
       "      <td id=\"T_03ec2_row270_col1\" class=\"data row270 col1\" >What is the purpose of creating a dataset from marathons in 2015 and 2016?</td>\n",
       "      <td id=\"T_03ec2_row270_col2\" class=\"data row270 col2\" >The purpose is to find runners that met the minimum performance standard in any marathon in those years.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row271\" class=\"row_heading level0 row271\" >271</th>\n",
       "      <td id=\"T_03ec2_row271_col0\" class=\"data row271 col0\" >It would probably be better to first create a dataset from all marathons in 2015 and 2016, and then find runners that met the minimum performance standard in any marathon in 2015 and 2016. There is some manual work to be done to get such a dataset, because one needs to know the url on marathonguide.com for every marathon. We selected the 22 marathons to include those where runners had qualified for the U.S. Olympic trials, and marathons that had many Boston Marathon Qualifiers, so we think our method misses very few elite runners.</td>\n",
       "      <td id=\"T_03ec2_row271_col1\" class=\"data row271 col1\" >Why is there manual work involved in creating the marathon dataset?</td>\n",
       "      <td id=\"T_03ec2_row271_col2\" class=\"data row271 col2\" >Manual work is needed to know the url on marathonguide.com for every marathon.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row272\" class=\"row_heading level0 row272\" >272</th>\n",
       "      <td id=\"T_03ec2_row272_col0\" class=\"data row272 col0\" >It would probably be better to first create a dataset from all marathons in 2015 and 2016, and then find runners that met the minimum performance standard in any marathon in 2015 and 2016. There is some manual work to be done to get such a dataset, because one needs to know the url on marathonguide.com for every marathon. We selected the 22 marathons to include those where runners had qualified for the U.S. Olympic trials, and marathons that had many Boston Marathon Qualifiers, so we think our method misses very few elite runners.</td>\n",
       "      <td id=\"T_03ec2_row272_col1\" class=\"data row272 col1\" >What was the criteria for selecting the 22 marathons?</td>\n",
       "      <td id=\"T_03ec2_row272_col2\" class=\"data row272 col2\" >The marathons were selected based on whether runners had qualified for the U.S. Olympic trials and whether the marathons had many Boston Marathon Qualifiers.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row273\" class=\"row_heading level0 row273\" >273</th>\n",
       "      <td id=\"T_03ec2_row273_col0\" class=\"data row273 col0\" >The authors refrain from using causal language in the article, probably because the study is observational rather than experimental in nature, and making causal conclusions from observational data is difficult and controversial. However, I don't think there is any doubt that the most relevant scientific question is a causal one: for instance, individual runners might want to know what their time would be wearing Vaporflys (treatment) versus wearing their usual race shoes (control). Such an individual-level causal question is challenging to answer even in the context of randomized trials, but we can and do strive to answer population-average version of such causal questions using observational data. The most common approach is to attempt to adjust for all confounding variables that impact both receipt of treatment (wearing of Vaporflys) and the outcome (marathon time). It would be nice if the authors could comment on what variables might be confounders in this setting.</td>\n",
       "      <td id=\"T_03ec2_row273_col1\" class=\"data row273 col1\" >Why do the authors avoid using causal language in the article?</td>\n",
       "      <td id=\"T_03ec2_row273_col2\" class=\"data row273 col2\" >Because the study is observational rather than experimental, and making causal conclusions from observational data is difficult and controversial.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row274\" class=\"row_heading level0 row274\" >274</th>\n",
       "      <td id=\"T_03ec2_row274_col0\" class=\"data row274 col0\" >The authors refrain from using causal language in the article, probably because the study is observational rather than experimental in nature, and making causal conclusions from observational data is difficult and controversial. However, I don't think there is any doubt that the most relevant scientific question is a causal one: for instance, individual runners might want to know what their time would be wearing Vaporflys (treatment) versus wearing their usual race shoes (control). Such an individual-level causal question is challenging to answer even in the context of randomized trials, but we can and do strive to answer population-average version of such causal questions using observational data. The most common approach is to attempt to adjust for all confounding variables that impact both receipt of treatment (wearing of Vaporflys) and the outcome (marathon time). It would be nice if the authors could comment on what variables might be confounders in this setting.</td>\n",
       "      <td id=\"T_03ec2_row274_col1\" class=\"data row274 col1\" >What is the treatment mentioned in the context?</td>\n",
       "      <td id=\"T_03ec2_row274_col2\" class=\"data row274 col2\" >Wearing Vaporflys</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row275\" class=\"row_heading level0 row275\" >275</th>\n",
       "      <td id=\"T_03ec2_row275_col0\" class=\"data row275 col0\" >The authors refrain from using causal language in the article, probably because the study is observational rather than experimental in nature, and making causal conclusions from observational data is difficult and controversial. However, I don't think there is any doubt that the most relevant scientific question is a causal one: for instance, individual runners might want to know what their time would be wearing Vaporflys (treatment) versus wearing their usual race shoes (control). Such an individual-level causal question is challenging to answer even in the context of randomized trials, but we can and do strive to answer population-average version of such causal questions using observational data. The most common approach is to attempt to adjust for all confounding variables that impact both receipt of treatment (wearing of Vaporflys) and the outcome (marathon time). It would be nice if the authors could comment on what variables might be confounders in this setting.</td>\n",
       "      <td id=\"T_03ec2_row275_col1\" class=\"data row275 col1\" >What is the outcome that the study is looking at?</td>\n",
       "      <td id=\"T_03ec2_row275_col2\" class=\"data row275 col2\" >Marathon time</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row276\" class=\"row_heading level0 row276\" >276</th>\n",
       "      <td id=\"T_03ec2_row276_col0\" class=\"data row276 col0\" >We are not experts in causal inference techniques, so we appreciate your expert opinion on the matter. We list a few possible confounders in the discussion, namely fitness and price. Since Mike Vaporflys are expensive relative to other racing shoes, a runner who is in poor fitness may be unwilling to pay for them to wear in a marathon, but once a runner gets into good shape, they might be more willing to pay for the shoes and race in them. Also, some runners might try the shoes in training and decide that they do not want to race in them because they do not get a benefit from them. A runner's personal objections to the fairness of the shoes might also influence self-assignment of the shoes.</td>\n",
       "      <td id=\"T_03ec2_row276_col1\" class=\"data row276 col1\" >What are the possible confounders mentioned in the discussion?</td>\n",
       "      <td id=\"T_03ec2_row276_col2\" class=\"data row276 col2\" >Fitness and price</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row277\" class=\"row_heading level0 row277\" >277</th>\n",
       "      <td id=\"T_03ec2_row277_col0\" class=\"data row277 col0\" >We are not experts in causal inference techniques, so we appreciate your expert opinion on the matter. We list a few possible confounders in the discussion, namely fitness and price. Since Mike Vaporflys are expensive relative to other racing shoes, a runner who is in poor fitness may be unwilling to pay for them to wear in a marathon, but once a runner gets into good shape, they might be more willing to pay for the shoes and race in them. Also, some runners might try the shoes in training and decide that they do not want to race in them because they do not get a benefit from them. A runner's personal objections to the fairness of the shoes might also influence self-assignment of the shoes.</td>\n",
       "      <td id=\"T_03ec2_row277_col1\" class=\"data row277 col1\" >Why might a runner in poor fitness be unwilling to pay for Mike Vaporflys?</td>\n",
       "      <td id=\"T_03ec2_row277_col2\" class=\"data row277 col2\" >Because they are expensive relative to other racing shoes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_03ec2_level0_row278\" class=\"row_heading level0 row278\" >278</th>\n",
       "      <td id=\"T_03ec2_row278_col0\" class=\"data row278 col0\" >We are not experts in causal inference techniques, so we appreciate your expert opinion on the matter. We list a few possible confounders in the discussion, namely fitness and price. Since Mike Vaporflys are expensive relative to other racing shoes, a runner who is in poor fitness may be unwilling to pay for them to wear in a marathon, but once a runner gets into good shape, they might be more willing to pay for the shoes and race in them. Also, some runners might try the shoes in training and decide that they do not want to race in them because they do not get a benefit from them. A runner's personal objections to the fairness of the shoes might also influence self-assignment of the shoes.</td>\n",
       "      <td id=\"T_03ec2_row278_col1\" class=\"data row278 col1\" >What might influence a runner's decision to not race in Mike Vaporflys despite trying them in training?</td>\n",
       "      <td id=\"T_03ec2_row278_col2\" class=\"data row278 col2\" >Not getting a benefit from them or personal objections to the fairness of the shoes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f3737200040>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "# Extracting context, question, and answer into a DataFrame\n",
    "contexts = []\n",
    "questions = []\n",
    "answers = []\n",
    "\n",
    "error_list = []\n",
    "\n",
    "for item in output:\n",
    "    o = item['output'][0]['response']\n",
    "\n",
    "    for resp in o:\n",
    "        context = resp['context']\n",
    "\n",
    "        for qa in resp['qas']:\n",
    "            questions.append(qa['Question'])\n",
    "            answers.append(qa['Answer'])\n",
    "            contexts.append(context)\n",
    "\n",
    "# Set display options\n",
    "pd.set_option('display.max_colwidth', None)\n",
    "pd.set_option('display.width', 1000)\n",
    "\n",
    "df = pd.DataFrame({\n",
    "    'Context': contexts,\n",
    "    'Question': questions,\n",
    "    'Answer': answers\n",
    "})\n",
    "\n",
    "# Remove any headers or short contexts\n",
    "df_filtered = df[df['Context'].apply(lambda x: len(x) > 100 and not x.startswith('#'))]\n",
    "df_filtered = df_filtered.reset_index(drop=True)\n",
    "\n",
    "styled_df = df_filtered.style.set_properties(**{'text-align': 'left'}).set_table_styles([{\n",
    "    'selector': 'th',\n",
    "    'props': [('text-align', 'left')]\n",
    "}])\n",
    "styled_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can save the `uniflow` output to a `.csv` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_df = df[['Question', 'Answer']]\n",
    "\n",
    "output_dir = 'data/output'\n",
    "\n",
    "uniflow_output_path = f\"{output_dir}/Nike_Research_QApairs.csv\"\n",
    "\n",
    "if not os.path.exists(output_dir):\n",
    "    os.makedirs(output_dir)\n",
    "\n",
    "output_df.to_csv(uniflow_output_path, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Release GPU Memory\n",
    "We'll need to use our GPU for future steps, so let's release the memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPU memory has been released.\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "if torch.cuda.is_available():\n",
    "    torch.cuda.empty_cache()\n",
    "    print(\"GPU memory has been released.\")\n",
    "else:\n",
    "    print(\"No GPU devices found.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Running `pykoi` `SupervisedFineTuning` on the QA pairs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install helper packages\n",
    "If you already have these installed, feel free to skip this step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!{sys.executable} -m pip install -q peft"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import Dependency"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pykoi.rlhf import RLHFConfig\n",
    "from pykoi.rlhf import SupervisedFinetuning\n",
    "from peft import LoraConfig, TaskType"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set the parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "base_model_path = \"meta-llama/Llama-2-7b-chat-hf\"\n",
    "dataset_name = uniflow_output_path\n",
    "peft_model_path = \"./models/rlhf_step1_sft\"\n",
    "dataset_type = \"local_csv\"\n",
    "learning_rate = 1e-3\n",
    "weight_decay = 0.0\n",
    "max_steps = 1600\n",
    "per_device_train_batch_size = 1\n",
    "per_device_eval_batch_size = 4\n",
    "log_freq = 20\n",
    "eval_freq = 2000\n",
    "save_freq = 200\n",
    "train_test_split_ratio = 0.0001\n",
    "dataset_subset_sft_train = 999999999\n",
    "size_valid_set = 0\n",
    "device_map = \"auto\"\n",
    "\n",
    "r = 8\n",
    "lora_alpha = 16\n",
    "lora_dropout = 0.05\n",
    "bias = \"none\"\n",
    "task_type = TaskType.CAUSAL_LM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the parameters in the `LoraConfig` and `RLHFConfig`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lora_config = LoraConfig(\n",
    "    r=r,\n",
    "    lora_alpha=lora_alpha,\n",
    "    lora_dropout=lora_dropout,\n",
    "    bias=bias,\n",
    "    task_type=task_type,\n",
    "    )\n",
    "\n",
    "\n",
    "# run supervised finetuning\n",
    "config = RLHFConfig(\n",
    "    base_model_path=base_model_path,\n",
    "    dataset_type=dataset_type,\n",
    "    dataset_name=dataset_name,\n",
    "    learning_rate=learning_rate,\n",
    "    weight_decay=weight_decay,\n",
    "    max_steps=max_steps,\n",
    "    per_device_train_batch_size=per_device_train_batch_size,\n",
    "    per_device_eval_batch_size=per_device_eval_batch_size,\n",
    "    log_freq=log_freq,\n",
    "    eval_freq=eval_freq,\n",
    "    save_freq=save_freq,\n",
    "    train_test_split_ratio=train_test_split_ratio,\n",
    "    dataset_subset_sft_train=dataset_subset_sft_train,\n",
    "    size_valid_set=size_valid_set,\n",
    "    lora_config_rl=lora_config,\n",
    "    device_map=device_map,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the SupervisedFineTuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rlhf_step1_sft = SupervisedFinetuning(config)\n",
    "rlhf_step1_sft.train_and_save(peft_model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Release GPU Memory\n",
    "We'll need to use our GPU for future steps, so let's release the memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "if torch.cuda.is_available():\n",
    "    torch.cuda.empty_cache()\n",
    "    print(\"GPU memory has been released.\")\n",
    "else:\n",
    "    print(\"No GPU devices found.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Running a `pykoi` `Chatbot` on the fine-tuned model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import pykoi components"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pykoi.application import Application\n",
    "from pykoi.chat import ModelFactory\n",
    "from pykoi.chat import QuestionAnswerDatabase\n",
    "from pykoi.component import Chatbot, Dashboard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Model\n",
    "\n",
    "Here, we create a model from `meta-llama/Llama-2-7b-chat-hf` and the fine-tuned model we created above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = ModelFactory.create_model(\n",
    "    model_source=\"peft_huggingface\",\n",
    "    base_model_path=\"meta-llama/Llama-2-7b-chat-hf\",\n",
    "    lora_model_path=\"/home/ubuntu/pykoi/models/rlhf_step1_sft\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Chatbot with the model\n",
    "Next, we create database, chatbot, and dashboard components via `pykoi`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "database = QuestionAnswerDatabase(debug=True)\n",
    "chatbot = Chatbot(model=model, feedback=\"vote\")\n",
    "dashboard = Dashboard(database=database)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the Chatbot app!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Add `nest_asyncio` \n",
    "Add `nest_asyncio` to avoid error such as `asyncio.run() cannot be called from a running event loop`. Since we're running another interface inside a Jupyter notebook where an asyncio event loop is already running, we'll encounter the error. (since The uvicorn.run() function uses asyncio.run(), which isn't compatible with a running event loop.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install -q nest_asyncio\n",
    "import nest_asyncio\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can run our app!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "app = Application(debug=False, share=False)\n",
    "app.add_component(chatbot)\n",
    "app.add_component(dashboard)\n",
    "app.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congrats! You've just built your own research paper chatbot!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End of the notebook\n",
    "\n",
    "Check more use cases in the [example folder](../../examples/)!\n",
    "\n",
    "<a href=\"https://www.cambioml.com/\" title=\"Title\">\n",
    "    <img src=\"../image/cambioml_logo_large.png\" style=\"height: 100px; display: block; margin-left: auto; margin-right: auto;\"/>\n",
    "</a>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "cookbook-research",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}