code/56-GRU.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 0
   },
   "source": [
    "# 门控循环单元（GRU）\n",
    "\n",
    "### 重置门和更新门\n",
    "\n",
    "$\\mathbf{X}_t \\in \\mathbb{R}^{n \\times d}$\n",
    "（样本个数：$n$，输入个数：$d$），\n",
    "上一个时间步的隐状态是\n",
    "$\\mathbf{H}_{t-1} \\in \\mathbb{R}^{n \\times h}$\n",
    "（隐藏单元个数：$h$）。\n",
    "那么，重置门$\\mathbf{R}_t \\in \\mathbb{R}^{n \\times h}$和\n",
    "更新门$\\mathbf{Z}_t \\in \\mathbb{R}^{n \\times h}$的计算如下所示：\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\mathbf{R}_t = \\sigma(\\mathbf{X}_t \\mathbf{W}_{xr} + \\mathbf{H}_{t-1} \\mathbf{W}_{hr} + \\mathbf{b}_r),\\\\\n",
    "\\mathbf{Z}_t = \\sigma(\\mathbf{X}_t \\mathbf{W}_{xz} + \\mathbf{H}_{t-1} \\mathbf{W}_{hz} + \\mathbf{b}_z),\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "其中$\\mathbf{W}_{xr}, \\mathbf{W}_{xz} \\in \\mathbb{R}^{d \\times h}$\n",
    "和$\\mathbf{W}_{hr}, \\mathbf{W}_{hz} \\in \\mathbb{R}^{h \\times h}$是权重参数，\n",
    "$\\mathbf{b}_r, \\mathbf{b}_z \\in \\mathbb{R}^{1 \\times h}$是偏置参数。\n",
    "请注意，在求和过程中会触发广播机制\n",
    "（请参阅 :numref:`subsec_broadcasting`）。\n",
    "我们使用sigmoid函数（如 :numref:`sec_mlp`中介绍的）\n",
    "将输入值转换到区间$(0, 1)$。\n",
    "\n",
    "### 候选隐状态\n",
    "\n",
    "接下来，让我们将重置门$\\mathbf{R}_t$\n",
    "与 :eqref:`rnn_h_with_state`\n",
    "中的常规隐状态更新机制集成，\n",
    "得到在时间步$t$的*候选隐状态*（candidate hidden state）\n",
    "$\\tilde{\\mathbf{H}}_t \\in \\mathbb{R}^{n \\times h}$。\n",
    "\n",
    "$$\\tilde{\\mathbf{H}}_t = \\tanh(\\mathbf{X}_t \\mathbf{W}_{xh} + \\left(\\mathbf{R}_t \\odot \\mathbf{H}_{t-1}\\right) \\mathbf{W}_{hh} + \\mathbf{b}_h),$$\n",
    ":eqlabel:`gru_tilde_H`\n",
    "\n",
    "其中$\\mathbf{W}_{xh} \\in \\mathbb{R}^{d \\times h}$\n",
    "和$\\mathbf{W}_{hh} \\in \\mathbb{R}^{h \\times h}$是权重参数，\n",
    "$\\mathbf{b}_h \\in \\mathbb{R}^{1 \\times h}$是偏置项，\n",
    "符号$\\odot$是Hadamard积（按元素乘积）运算符。\n",
    "在这里，我们使用tanh非线性激活函数来确保候选隐状态中的值保持在区间$(-1, 1)$中。\n",
    "\n",
    "\n",
    "### 隐状态\n",
    "\n",
    "上述的计算结果只是候选隐状态，我们仍然需要结合更新门$\\mathbf{Z}_t$的效果。\n",
    "这一步确定新的隐状态$\\mathbf{H}_t \\in \\mathbb{R}^{n \\times h}$\n",
    "在多大程度上来自旧的状态$\\mathbf{H}_{t-1}$和\n",
    "新的候选状态$\\tilde{\\mathbf{H}}_t$。\n",
    "更新门$\\mathbf{Z}_t$仅需要在\n",
    "$\\mathbf{H}_{t-1}$和$\\tilde{\\mathbf{H}}_t$\n",
    "之间进行按元素的凸组合就可以实现这个目标。\n",
    "这就得出了门控循环单元的最终更新公式：\n",
    "\n",
    "$$\\mathbf{H}_t = \\mathbf{Z}_t \\odot \\mathbf{H}_{t-1}  + (1 - \\mathbf{Z}_t) \\odot \\tilde{\\mathbf{H}}_t.$$\n",
    "\n",
    "每当更新门$\\mathbf{Z}_t$接近$1$时，模型就倾向只保留旧状态。\n",
    "此时，来自$\\mathbf{X}_t$的信息基本上被忽略，\n",
    "从而有效地跳过了依赖链条中的时间步$t$。\n",
    "相反，当$\\mathbf{Z}_t$接近$0$时，\n",
    "新的隐状态$\\mathbf{H}_t$就会接近候选隐状态$\\tilde{\\mathbf{H}}_t$。\n",
    "这些设计可以帮助我们处理循环神经网络中的梯度消失问题，\n",
    "并更好地捕获时间步距离很长的序列的依赖关系。\n",
    "例如，如果整个子序列的所有时间步的更新门都接近于$1$，\n",
    "则无论序列的长度如何，在序列起始时间步的旧隐状态都将很容易保留并传递到序列结束。\n",
    "\n",
    "\n",
    "## 从零开始实现\n",
    "\n",
    "为了更好地理解门控循环单元模型，我们从零开始实现它。\n",
    "首先，我们读取 :numref:`sec_rnn_scratch`中使用的时间机器数据集：\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "origin_pos": 2,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from d2l import torch as d2l\n",
    "\n",
    "batch_size, num_steps = 32, 35\n",
    "train_iter, vocab = d2l.load_data_time_machine(batch_size, num_steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 4
   },
   "source": [
    "### [**初始化模型参数**]\n",
    "\n",
    "下一步是初始化模型参数。\n",
    "我们从标准差为$0.01$的高斯分布中提取权重，\n",
    "并将偏置项设为$0$，超参数`num_hiddens`定义隐藏单元的数量，\n",
    "实例化与更新门、重置门、候选隐状态和输出层相关的所有权重和偏置。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "origin_pos": 6,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def get_params(vocab_size, num_hiddens, device):\n",
    "    num_inputs = num_outputs = vocab_size\n",
    "\n",
    "    def normal(shape):\n",
    "        return torch.randn(size=shape, device=device)*0.01\n",
    "\n",
    "    def three():\n",
    "        return (normal((num_inputs, num_hiddens)),\n",
    "                normal((num_hiddens, num_hiddens)),\n",
    "                torch.zeros(num_hiddens, device=device))\n",
    "\n",
    "    W_xz, W_hz, b_z = three()  # 更新门参数\n",
    "    W_xr, W_hr, b_r = three()  # 重置门参数\n",
    "    W_xh, W_hh, b_h = three()  # 候选隐状态参数\n",
    "    # 输出层参数\n",
    "    W_hq = normal((num_hiddens, num_outputs))\n",
    "    b_q = torch.zeros(num_outputs, device=device)\n",
    "    # 附加梯度\n",
    "    params = [W_xz, W_hz, b_z, W_xr, W_hr, b_r, W_xh, W_hh, b_h, W_hq, b_q]\n",
    "    for param in params:\n",
    "        param.requires_grad_(True)\n",
    "    return params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 8
   },
   "source": [
    "### 定义模型\n",
    "\n",
    "现在我们将[**定义隐状态的初始化函数**]`init_gru_state`。\n",
    "与 :numref:`sec_rnn_scratch`中定义的`init_rnn_state`函数一样，\n",
    "此函数返回一个形状为（批量大小，隐藏单元个数）的张量，张量的值全部为零。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "origin_pos": 10,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def init_gru_state(batch_size, num_hiddens, device):\n",
    "    return (torch.zeros((batch_size, num_hiddens), device=device), )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 12
   },
   "source": [
    "现在我们准备[**定义门控循环单元模型**]，\n",
    "模型的架构与基本的循环神经网络单元是相同的，\n",
    "只是权重更新公式更为复杂。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "origin_pos": 14,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "def gru(inputs, state, params):\n",
    "    W_xz, W_hz, b_z, W_xr, W_hr, b_r, W_xh, W_hh, b_h, W_hq, b_q = params\n",
    "    H, = state\n",
    "    outputs = []\n",
    "    for X in inputs:\n",
    "        Z = torch.sigmoid((X @ W_xz) + (H @ W_hz) + b_z)\n",
    "        R = torch.sigmoid((X @ W_xr) + (H @ W_hr) + b_r)\n",
    "        H_tilda = torch.tanh((X @ W_xh) + ((R * H) @ W_hh) + b_h)\n",
    "        H = Z * H + (1 - Z) * H_tilda\n",
    "        Y = H @ W_hq + b_q\n",
    "        outputs.append(Y)\n",
    "    return torch.cat(outputs, dim=0), (H,)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 16
   },
   "source": [
    "### [**训练**]与预测\n",
    "\n",
    "训练和预测的工作方式与 :numref:`sec_rnn_scratch`完全相同。\n",
    "训练结束后，我们分别打印输出训练集的困惑度，\n",
    "以及前缀“time traveler”和“traveler”的预测序列上的困惑度。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "origin_pos": 17,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "perplexity 1.1, 32652.0 tokens/sec on cuda:0\n",
      "time traveller for so it will be convenient to speak of himwas e\n",
      "travelleryou can show black is white by argument said filby\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<svg height=\"180.65625pt\" version=\"1.1\" viewBox=\"0 0 252.646875 180.65625\" width=\"252.646875pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <metadata>\r\n",
       "  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\r\n",
       "   <cc:Work>\r\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\r\n",
       "    <dc:date>2022-02-16T16:46:04.433519</dc:date>\r\n",
       "    <dc:format>image/svg+xml</dc:format>\r\n",
       "    <dc:creator>\r\n",
       "     <cc:Agent>\r\n",
       "      <dc:title>Matplotlib v3.4.3, https://matplotlib.org/</dc:title>\r\n",
       "     </cc:Agent>\r\n",
       "    </dc:creator>\r\n",
       "   </cc:Work>\r\n",
       "  </rdf:RDF>\r\n",
       " </metadata>\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 180.65625 \r\n",
       "L 252.646875 180.65625 \r\n",
       "L 252.646875 0 \r\n",
       "L 0 0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 40.603125 143.1 \r\n",
       "L 235.903125 143.1 \r\n",
       "L 235.903125 7.2 \r\n",
       "L 40.603125 7.2 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <path clip-path=\"url(#pc7605b5312)\" d=\"M 76.474554 143.1 \r\n",
       "L 76.474554 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"m11894a15b1\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"76.474554\" xlink:href=\"#m11894a15b1\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- 100 -->\r\n",
       "      <g transform=\"translate(66.930804 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 794 531 \r\n",
       "L 1825 531 \r\n",
       "L 1825 4091 \r\n",
       "L 703 3866 \r\n",
       "L 703 4441 \r\n",
       "L 1819 4666 \r\n",
       "L 2450 4666 \r\n",
       "L 2450 531 \r\n",
       "L 3481 531 \r\n",
       "L 3481 0 \r\n",
       "L 794 0 \r\n",
       "L 794 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-31\" transform=\"scale(0.015625)\"/>\r\n",
       "        <path d=\"M 2034 4250 \r\n",
       "Q 1547 4250 1301 3770 \r\n",
       "Q 1056 3291 1056 2328 \r\n",
       "Q 1056 1369 1301 889 \r\n",
       "Q 1547 409 2034 409 \r\n",
       "Q 2525 409 2770 889 \r\n",
       "Q 3016 1369 3016 2328 \r\n",
       "Q 3016 3291 2770 3770 \r\n",
       "Q 2525 4250 2034 4250 \r\n",
       "z\r\n",
       "M 2034 4750 \r\n",
       "Q 2819 4750 3233 4129 \r\n",
       "Q 3647 3509 3647 2328 \r\n",
       "Q 3647 1150 3233 529 \r\n",
       "Q 2819 -91 2034 -91 \r\n",
       "Q 1250 -91 836 529 \r\n",
       "Q 422 1150 422 2328 \r\n",
       "Q 422 3509 836 4129 \r\n",
       "Q 1250 4750 2034 4750 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-30\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <path clip-path=\"url(#pc7605b5312)\" d=\"M 116.331696 143.1 \r\n",
       "L 116.331696 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"116.331696\" xlink:href=\"#m11894a15b1\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- 200 -->\r\n",
       "      <g transform=\"translate(106.787946 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 1228 531 \r\n",
       "L 3431 531 \r\n",
       "L 3431 0 \r\n",
       "L 469 0 \r\n",
       "L 469 531 \r\n",
       "Q 828 903 1448 1529 \r\n",
       "Q 2069 2156 2228 2338 \r\n",
       "Q 2531 2678 2651 2914 \r\n",
       "Q 2772 3150 2772 3378 \r\n",
       "Q 2772 3750 2511 3984 \r\n",
       "Q 2250 4219 1831 4219 \r\n",
       "Q 1534 4219 1204 4116 \r\n",
       "Q 875 4013 500 3803 \r\n",
       "L 500 4441 \r\n",
       "Q 881 4594 1212 4672 \r\n",
       "Q 1544 4750 1819 4750 \r\n",
       "Q 2544 4750 2975 4387 \r\n",
       "Q 3406 4025 3406 3419 \r\n",
       "Q 3406 3131 3298 2873 \r\n",
       "Q 3191 2616 2906 2266 \r\n",
       "Q 2828 2175 2409 1742 \r\n",
       "Q 1991 1309 1228 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-32\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <path clip-path=\"url(#pc7605b5312)\" d=\"M 156.188839 143.1 \r\n",
       "L 156.188839 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"156.188839\" xlink:href=\"#m11894a15b1\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 300 -->\r\n",
       "      <g transform=\"translate(146.645089 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2597 2516 \r\n",
       "Q 3050 2419 3304 2112 \r\n",
       "Q 3559 1806 3559 1356 \r\n",
       "Q 3559 666 3084 287 \r\n",
       "Q 2609 -91 1734 -91 \r\n",
       "Q 1441 -91 1130 -33 \r\n",
       "Q 819 25 488 141 \r\n",
       "L 488 750 \r\n",
       "Q 750 597 1062 519 \r\n",
       "Q 1375 441 1716 441 \r\n",
       "Q 2309 441 2620 675 \r\n",
       "Q 2931 909 2931 1356 \r\n",
       "Q 2931 1769 2642 2001 \r\n",
       "Q 2353 2234 1838 2234 \r\n",
       "L 1294 2234 \r\n",
       "L 1294 2753 \r\n",
       "L 1863 2753 \r\n",
       "Q 2328 2753 2575 2939 \r\n",
       "Q 2822 3125 2822 3475 \r\n",
       "Q 2822 3834 2567 4026 \r\n",
       "Q 2313 4219 1838 4219 \r\n",
       "Q 1578 4219 1281 4162 \r\n",
       "Q 984 4106 628 3988 \r\n",
       "L 628 4550 \r\n",
       "Q 988 4650 1302 4700 \r\n",
       "Q 1616 4750 1894 4750 \r\n",
       "Q 2613 4750 3031 4423 \r\n",
       "Q 3450 4097 3450 3541 \r\n",
       "Q 3450 3153 3228 2886 \r\n",
       "Q 3006 2619 2597 2516 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-33\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <path clip-path=\"url(#pc7605b5312)\" d=\"M 196.045982 143.1 \r\n",
       "L 196.045982 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"196.045982\" xlink:href=\"#m11894a15b1\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- 400 -->\r\n",
       "      <g transform=\"translate(186.502232 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2419 4116 \r\n",
       "L 825 1625 \r\n",
       "L 2419 1625 \r\n",
       "L 2419 4116 \r\n",
       "z\r\n",
       "M 2253 4666 \r\n",
       "L 3047 4666 \r\n",
       "L 3047 1625 \r\n",
       "L 3713 1625 \r\n",
       "L 3713 1100 \r\n",
       "L 3047 1100 \r\n",
       "L 3047 0 \r\n",
       "L 2419 0 \r\n",
       "L 2419 1100 \r\n",
       "L 313 1100 \r\n",
       "L 313 1709 \r\n",
       "L 2253 4666 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-34\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <path clip-path=\"url(#pc7605b5312)\" d=\"M 235.903125 143.1 \r\n",
       "L 235.903125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"235.903125\" xlink:href=\"#m11894a15b1\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 500 -->\r\n",
       "      <g transform=\"translate(226.359375 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 691 4666 \r\n",
       "L 3169 4666 \r\n",
       "L 3169 4134 \r\n",
       "L 1269 4134 \r\n",
       "L 1269 2991 \r\n",
       "Q 1406 3038 1543 3061 \r\n",
       "Q 1681 3084 1819 3084 \r\n",
       "Q 2600 3084 3056 2656 \r\n",
       "Q 3513 2228 3513 1497 \r\n",
       "Q 3513 744 3044 326 \r\n",
       "Q 2575 -91 1722 -91 \r\n",
       "Q 1428 -91 1123 -41 \r\n",
       "Q 819 9 494 109 \r\n",
       "L 494 744 \r\n",
       "Q 775 591 1075 516 \r\n",
       "Q 1375 441 1709 441 \r\n",
       "Q 2250 441 2565 725 \r\n",
       "Q 2881 1009 2881 1497 \r\n",
       "Q 2881 1984 2565 2268 \r\n",
       "Q 2250 2553 1709 2553 \r\n",
       "Q 1456 2553 1204 2497 \r\n",
       "Q 953 2441 691 2322 \r\n",
       "L 691 4666 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-35\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_6\">\r\n",
       "     <!-- epoch -->\r\n",
       "     <g transform=\"translate(123.025 171.376563)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 3597 1894 \r\n",
       "L 3597 1613 \r\n",
       "L 953 1613 \r\n",
       "Q 991 1019 1311 708 \r\n",
       "Q 1631 397 2203 397 \r\n",
       "Q 2534 397 2845 478 \r\n",
       "Q 3156 559 3463 722 \r\n",
       "L 3463 178 \r\n",
       "Q 3153 47 2828 -22 \r\n",
       "Q 2503 -91 2169 -91 \r\n",
       "Q 1331 -91 842 396 \r\n",
       "Q 353 884 353 1716 \r\n",
       "Q 353 2575 817 3079 \r\n",
       "Q 1281 3584 2069 3584 \r\n",
       "Q 2775 3584 3186 3129 \r\n",
       "Q 3597 2675 3597 1894 \r\n",
       "z\r\n",
       "M 3022 2063 \r\n",
       "Q 3016 2534 2758 2815 \r\n",
       "Q 2500 3097 2075 3097 \r\n",
       "Q 1594 3097 1305 2825 \r\n",
       "Q 1016 2553 972 2059 \r\n",
       "L 3022 2063 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-65\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 1159 525 \r\n",
       "L 1159 -1331 \r\n",
       "L 581 -1331 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2969 \r\n",
       "Q 1341 3281 1617 3432 \r\n",
       "Q 1894 3584 2278 3584 \r\n",
       "Q 2916 3584 3314 3078 \r\n",
       "Q 3713 2572 3713 1747 \r\n",
       "Q 3713 922 3314 415 \r\n",
       "Q 2916 -91 2278 -91 \r\n",
       "Q 1894 -91 1617 61 \r\n",
       "Q 1341 213 1159 525 \r\n",
       "z\r\n",
       "M 3116 1747 \r\n",
       "Q 3116 2381 2855 2742 \r\n",
       "Q 2594 3103 2138 3103 \r\n",
       "Q 1681 3103 1420 2742 \r\n",
       "Q 1159 2381 1159 1747 \r\n",
       "Q 1159 1113 1420 752 \r\n",
       "Q 1681 391 2138 391 \r\n",
       "Q 2594 391 2855 752 \r\n",
       "Q 3116 1113 3116 1747 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-70\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 1959 3097 \r\n",
       "Q 1497 3097 1228 2736 \r\n",
       "Q 959 2375 959 1747 \r\n",
       "Q 959 1119 1226 758 \r\n",
       "Q 1494 397 1959 397 \r\n",
       "Q 2419 397 2687 759 \r\n",
       "Q 2956 1122 2956 1747 \r\n",
       "Q 2956 2369 2687 2733 \r\n",
       "Q 2419 3097 1959 3097 \r\n",
       "z\r\n",
       "M 1959 3584 \r\n",
       "Q 2709 3584 3137 3096 \r\n",
       "Q 3566 2609 3566 1747 \r\n",
       "Q 3566 888 3137 398 \r\n",
       "Q 2709 -91 1959 -91 \r\n",
       "Q 1206 -91 779 398 \r\n",
       "Q 353 888 353 1747 \r\n",
       "Q 353 2609 779 3096 \r\n",
       "Q 1206 3584 1959 3584 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6f\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3122 3366 \r\n",
       "L 3122 2828 \r\n",
       "Q 2878 2963 2633 3030 \r\n",
       "Q 2388 3097 2138 3097 \r\n",
       "Q 1578 3097 1268 2742 \r\n",
       "Q 959 2388 959 1747 \r\n",
       "Q 959 1106 1268 751 \r\n",
       "Q 1578 397 2138 397 \r\n",
       "Q 2388 397 2633 464 \r\n",
       "Q 2878 531 3122 666 \r\n",
       "L 3122 134 \r\n",
       "Q 2881 22 2623 -34 \r\n",
       "Q 2366 -91 2075 -91 \r\n",
       "Q 1284 -91 818 406 \r\n",
       "Q 353 903 353 1747 \r\n",
       "Q 353 2603 823 3093 \r\n",
       "Q 1294 3584 2113 3584 \r\n",
       "Q 2378 3584 2631 3529 \r\n",
       "Q 2884 3475 3122 3366 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-63\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3513 2113 \r\n",
       "L 3513 0 \r\n",
       "L 2938 0 \r\n",
       "L 2938 2094 \r\n",
       "Q 2938 2591 2744 2837 \r\n",
       "Q 2550 3084 2163 3084 \r\n",
       "Q 1697 3084 1428 2787 \r\n",
       "Q 1159 2491 1159 1978 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 4863 \r\n",
       "L 1159 4863 \r\n",
       "L 1159 2956 \r\n",
       "Q 1366 3272 1645 3428 \r\n",
       "Q 1925 3584 2291 3584 \r\n",
       "Q 2894 3584 3203 3211 \r\n",
       "Q 3513 2838 3513 2113 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-68\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\r\n",
       "      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-70\"/>\r\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-6f\"/>\r\n",
       "      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-63\"/>\r\n",
       "      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-68\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_11\">\r\n",
       "      <path clip-path=\"url(#pc7605b5312)\" d=\"M 40.603125 106.749672 \r\n",
       "L 235.903125 106.749672 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_12\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"mffa87bee1e\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"40.603125\" xlink:href=\"#mffa87bee1e\" y=\"106.749672\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_7\">\r\n",
       "      <!-- 5 -->\r\n",
       "      <g transform=\"translate(27.240625 110.548891)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_13\">\r\n",
       "      <path clip-path=\"url(#pc7605b5312)\" d=\"M 40.603125 68.51291 \r\n",
       "L 235.903125 68.51291 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_14\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"40.603125\" xlink:href=\"#mffa87bee1e\" y=\"68.51291\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- 10 -->\r\n",
       "      <g transform=\"translate(20.878125 72.312129)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_15\">\r\n",
       "      <path clip-path=\"url(#pc7605b5312)\" d=\"M 40.603125 30.276148 \r\n",
       "L 235.903125 30.276148 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_16\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"40.603125\" xlink:href=\"#mffa87bee1e\" y=\"30.276148\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- 15 -->\r\n",
       "      <g transform=\"translate(20.878125 34.075366)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_10\">\r\n",
       "     <!-- perplexity -->\r\n",
       "     <g transform=\"translate(14.798437 100.276562)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 2631 2963 \r\n",
       "Q 2534 3019 2420 3045 \r\n",
       "Q 2306 3072 2169 3072 \r\n",
       "Q 1681 3072 1420 2755 \r\n",
       "Q 1159 2438 1159 1844 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2956 \r\n",
       "Q 1341 3275 1631 3429 \r\n",
       "Q 1922 3584 2338 3584 \r\n",
       "Q 2397 3584 2469 3576 \r\n",
       "Q 2541 3569 2628 3553 \r\n",
       "L 2631 2963 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-72\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 603 4863 \r\n",
       "L 1178 4863 \r\n",
       "L 1178 0 \r\n",
       "L 603 0 \r\n",
       "L 603 4863 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6c\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3513 3500 \r\n",
       "L 2247 1797 \r\n",
       "L 3578 0 \r\n",
       "L 2900 0 \r\n",
       "L 1881 1375 \r\n",
       "L 863 0 \r\n",
       "L 184 0 \r\n",
       "L 1544 1831 \r\n",
       "L 300 3500 \r\n",
       "L 978 3500 \r\n",
       "L 1906 2253 \r\n",
       "L 2834 3500 \r\n",
       "L 3513 3500 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-78\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 603 3500 \r\n",
       "L 1178 3500 \r\n",
       "L 1178 0 \r\n",
       "L 603 0 \r\n",
       "L 603 3500 \r\n",
       "z\r\n",
       "M 603 4863 \r\n",
       "L 1178 4863 \r\n",
       "L 1178 4134 \r\n",
       "L 603 4134 \r\n",
       "L 603 4863 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-69\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 1172 4494 \r\n",
       "L 1172 3500 \r\n",
       "L 2356 3500 \r\n",
       "L 2356 3053 \r\n",
       "L 1172 3053 \r\n",
       "L 1172 1153 \r\n",
       "Q 1172 725 1289 603 \r\n",
       "Q 1406 481 1766 481 \r\n",
       "L 2356 481 \r\n",
       "L 2356 0 \r\n",
       "L 1766 0 \r\n",
       "Q 1100 0 847 248 \r\n",
       "Q 594 497 594 1153 \r\n",
       "L 594 3053 \r\n",
       "L 172 3053 \r\n",
       "L 172 3500 \r\n",
       "L 594 3500 \r\n",
       "L 594 4494 \r\n",
       "L 1172 4494 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-74\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2059 -325 \r\n",
       "Q 1816 -950 1584 -1140 \r\n",
       "Q 1353 -1331 966 -1331 \r\n",
       "L 506 -1331 \r\n",
       "L 506 -850 \r\n",
       "L 844 -850 \r\n",
       "Q 1081 -850 1212 -737 \r\n",
       "Q 1344 -625 1503 -206 \r\n",
       "L 1606 56 \r\n",
       "L 191 3500 \r\n",
       "L 800 3500 \r\n",
       "L 1894 763 \r\n",
       "L 2988 3500 \r\n",
       "L 3597 3500 \r\n",
       "L 2059 -325 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-79\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\r\n",
       "      <use x=\"63.476562\" xlink:href=\"#DejaVuSans-65\"/>\r\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-72\"/>\r\n",
       "      <use x=\"166.113281\" xlink:href=\"#DejaVuSans-70\"/>\r\n",
       "      <use x=\"229.589844\" xlink:href=\"#DejaVuSans-6c\"/>\r\n",
       "      <use x=\"257.373047\" xlink:href=\"#DejaVuSans-65\"/>\r\n",
       "      <use x=\"317.146484\" xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "      <use x=\"376.326172\" xlink:href=\"#DejaVuSans-69\"/>\r\n",
       "      <use x=\"404.109375\" xlink:href=\"#DejaVuSans-74\"/>\r\n",
       "      <use x=\"443.318359\" xlink:href=\"#DejaVuSans-79\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_17\">\r\n",
       "    <path clip-path=\"url(#pc7605b5312)\" d=\"M 40.603125 13.377273 \r\n",
       "L 44.588839 27.764183 \r\n",
       "L 48.574554 44.263802 \r\n",
       "L 52.560268 57.708744 \r\n",
       "L 56.545982 61.592855 \r\n",
       "L 60.531696 66.39676 \r\n",
       "L 64.517411 67.871659 \r\n",
       "L 68.503125 71.30948 \r\n",
       "L 72.488839 73.64977 \r\n",
       "L 76.474554 76.451992 \r\n",
       "L 80.460268 79.696987 \r\n",
       "L 84.445982 80.883222 \r\n",
       "L 88.431696 83.811601 \r\n",
       "L 92.417411 86.152282 \r\n",
       "L 96.403125 87.162161 \r\n",
       "L 100.388839 88.894444 \r\n",
       "L 104.374554 90.337371 \r\n",
       "L 108.360268 93.217557 \r\n",
       "L 112.345982 95.538233 \r\n",
       "L 116.331696 97.581647 \r\n",
       "L 120.317411 98.928185 \r\n",
       "L 124.303125 100.825463 \r\n",
       "L 128.288839 103.432887 \r\n",
       "L 132.274554 105.399794 \r\n",
       "L 136.260268 108.019259 \r\n",
       "L 140.245982 110.294972 \r\n",
       "L 144.231696 113.05215 \r\n",
       "L 148.217411 115.743769 \r\n",
       "L 152.203125 118.240279 \r\n",
       "L 156.188839 120.677956 \r\n",
       "L 160.174554 123.284054 \r\n",
       "L 164.160268 126.26667 \r\n",
       "L 168.145982 128.456118 \r\n",
       "L 172.131696 130.276985 \r\n",
       "L 176.117411 132.222306 \r\n",
       "L 180.103125 133.174106 \r\n",
       "L 184.088839 134.2692 \r\n",
       "L 188.074554 135.241492 \r\n",
       "L 192.060268 135.918776 \r\n",
       "L 196.045982 135.956792 \r\n",
       "L 200.031696 136.520685 \r\n",
       "L 204.017411 136.630134 \r\n",
       "L 208.003125 136.621832 \r\n",
       "L 211.988839 136.214975 \r\n",
       "L 215.974554 136.793696 \r\n",
       "L 219.960268 136.808807 \r\n",
       "L 223.945982 136.839185 \r\n",
       "L 227.931696 136.893194 \r\n",
       "L 231.917411 136.7864 \r\n",
       "L 235.903125 136.922727 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 40.603125 143.1 \r\n",
       "L 40.603125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 235.903125 143.1 \r\n",
       "L 235.903125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 40.603125 143.1 \r\n",
       "L 235.903125 143.1 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 40.603125 7.2 \r\n",
       "L 235.903125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"legend_1\">\r\n",
       "    <g id=\"patch_7\">\r\n",
       "     <path d=\"M 173.628125 29.878125 \r\n",
       "L 228.903125 29.878125 \r\n",
       "Q 230.903125 29.878125 230.903125 27.878125 \r\n",
       "L 230.903125 14.2 \r\n",
       "Q 230.903125 12.2 228.903125 12.2 \r\n",
       "L 173.628125 12.2 \r\n",
       "Q 171.628125 12.2 171.628125 14.2 \r\n",
       "L 171.628125 27.878125 \r\n",
       "Q 171.628125 29.878125 173.628125 29.878125 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_18\">\r\n",
       "     <path d=\"M 175.628125 20.298437 \r\n",
       "L 195.628125 20.298437 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_19\"/>\r\n",
       "    <g id=\"text_11\">\r\n",
       "     <!-- train -->\r\n",
       "     <g transform=\"translate(203.628125 23.798437)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 2194 1759 \r\n",
       "Q 1497 1759 1228 1600 \r\n",
       "Q 959 1441 959 1056 \r\n",
       "Q 959 750 1161 570 \r\n",
       "Q 1363 391 1709 391 \r\n",
       "Q 2188 391 2477 730 \r\n",
       "Q 2766 1069 2766 1631 \r\n",
       "L 2766 1759 \r\n",
       "L 2194 1759 \r\n",
       "z\r\n",
       "M 3341 1997 \r\n",
       "L 3341 0 \r\n",
       "L 2766 0 \r\n",
       "L 2766 531 \r\n",
       "Q 2569 213 2275 61 \r\n",
       "Q 1981 -91 1556 -91 \r\n",
       "Q 1019 -91 701 211 \r\n",
       "Q 384 513 384 1019 \r\n",
       "Q 384 1609 779 1909 \r\n",
       "Q 1175 2209 1959 2209 \r\n",
       "L 2766 2209 \r\n",
       "L 2766 2266 \r\n",
       "Q 2766 2663 2505 2880 \r\n",
       "Q 2244 3097 1772 3097 \r\n",
       "Q 1472 3097 1187 3025 \r\n",
       "Q 903 2953 641 2809 \r\n",
       "L 641 3341 \r\n",
       "Q 956 3463 1253 3523 \r\n",
       "Q 1550 3584 1831 3584 \r\n",
       "Q 2591 3584 2966 3190 \r\n",
       "Q 3341 2797 3341 1997 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-61\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3513 2113 \r\n",
       "L 3513 0 \r\n",
       "L 2938 0 \r\n",
       "L 2938 2094 \r\n",
       "Q 2938 2591 2744 2837 \r\n",
       "Q 2550 3084 2163 3084 \r\n",
       "Q 1697 3084 1428 2787 \r\n",
       "Q 1159 2491 1159 1978 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2956 \r\n",
       "Q 1366 3272 1645 3428 \r\n",
       "Q 1925 3584 2291 3584 \r\n",
       "Q 2894 3584 3203 3211 \r\n",
       "Q 3513 2838 3513 2113 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6e\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-72\"/>\r\n",
       "      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-61\"/>\r\n",
       "      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-69\"/>\r\n",
       "      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-6e\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"pc7605b5312\">\r\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"40.603125\" y=\"7.2\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import os\n",
    "os.environ[\"KMP_DUPLICATE_LIB_OK\"]  =  \"TRUE\"\n",
    "\n",
    "vocab_size, num_hiddens, device = len(vocab), 256, d2l.try_gpu()\n",
    "num_epochs, lr = 500, 1\n",
    "model = d2l.RNNModelScratch(len(vocab), num_hiddens, device, get_params,\n",
    "                            init_gru_state, gru)\n",
    "d2l.train_ch8(model, train_iter, vocab, lr, num_epochs, device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 19
   },
   "source": [
    "## [**简洁实现**]\n",
    "\n",
    "高级API包含了前文介绍的所有配置细节，\n",
    "所以我们可以直接实例化门控循环单元模型。\n",
    "这段代码的运行速度要快得多，\n",
    "因为它使用的是编译好的运算符而不是Python来处理之前阐述的许多细节。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "origin_pos": 21,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "perplexity 1.0, 114803.3 tokens/sec on cuda:0\n",
      "time travelleryou can show black is white by argument said filby\n",
      "traveller with a slight accession ofcheerfulness really thi\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<svg height=\"180.65625pt\" version=\"1.1\" viewBox=\"0 0 252.646875 180.65625\" width=\"252.646875pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       " <metadata>\r\n",
       "  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\r\n",
       "   <cc:Work>\r\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\r\n",
       "    <dc:date>2022-02-16T16:48:26.545696</dc:date>\r\n",
       "    <dc:format>image/svg+xml</dc:format>\r\n",
       "    <dc:creator>\r\n",
       "     <cc:Agent>\r\n",
       "      <dc:title>Matplotlib v3.4.3, https://matplotlib.org/</dc:title>\r\n",
       "     </cc:Agent>\r\n",
       "    </dc:creator>\r\n",
       "   </cc:Work>\r\n",
       "  </rdf:RDF>\r\n",
       " </metadata>\r\n",
       " <defs>\r\n",
       "  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\r\n",
       " </defs>\r\n",
       " <g id=\"figure_1\">\r\n",
       "  <g id=\"patch_1\">\r\n",
       "   <path d=\"M 0 180.65625 \r\n",
       "L 252.646875 180.65625 \r\n",
       "L 252.646875 0 \r\n",
       "L 0 0 \r\n",
       "z\r\n",
       "\" style=\"fill:none;\"/>\r\n",
       "  </g>\r\n",
       "  <g id=\"axes_1\">\r\n",
       "   <g id=\"patch_2\">\r\n",
       "    <path d=\"M 40.603125 143.1 \r\n",
       "L 235.903125 143.1 \r\n",
       "L 235.903125 7.2 \r\n",
       "L 40.603125 7.2 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_1\">\r\n",
       "    <g id=\"xtick_1\">\r\n",
       "     <g id=\"line2d_1\">\r\n",
       "      <path clip-path=\"url(#p52c1335e1b)\" d=\"M 76.474554 143.1 \r\n",
       "L 76.474554 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_2\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L 0 3.5 \r\n",
       "\" id=\"m5cdb5ac206\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"76.474554\" xlink:href=\"#m5cdb5ac206\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_1\">\r\n",
       "      <!-- 100 -->\r\n",
       "      <g transform=\"translate(66.930804 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 794 531 \r\n",
       "L 1825 531 \r\n",
       "L 1825 4091 \r\n",
       "L 703 3866 \r\n",
       "L 703 4441 \r\n",
       "L 1819 4666 \r\n",
       "L 2450 4666 \r\n",
       "L 2450 531 \r\n",
       "L 3481 531 \r\n",
       "L 3481 0 \r\n",
       "L 794 0 \r\n",
       "L 794 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-31\" transform=\"scale(0.015625)\"/>\r\n",
       "        <path d=\"M 2034 4250 \r\n",
       "Q 1547 4250 1301 3770 \r\n",
       "Q 1056 3291 1056 2328 \r\n",
       "Q 1056 1369 1301 889 \r\n",
       "Q 1547 409 2034 409 \r\n",
       "Q 2525 409 2770 889 \r\n",
       "Q 3016 1369 3016 2328 \r\n",
       "Q 3016 3291 2770 3770 \r\n",
       "Q 2525 4250 2034 4250 \r\n",
       "z\r\n",
       "M 2034 4750 \r\n",
       "Q 2819 4750 3233 4129 \r\n",
       "Q 3647 3509 3647 2328 \r\n",
       "Q 3647 1150 3233 529 \r\n",
       "Q 2819 -91 2034 -91 \r\n",
       "Q 1250 -91 836 529 \r\n",
       "Q 422 1150 422 2328 \r\n",
       "Q 422 3509 836 4129 \r\n",
       "Q 1250 4750 2034 4750 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-30\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_2\">\r\n",
       "     <g id=\"line2d_3\">\r\n",
       "      <path clip-path=\"url(#p52c1335e1b)\" d=\"M 116.331696 143.1 \r\n",
       "L 116.331696 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_4\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"116.331696\" xlink:href=\"#m5cdb5ac206\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_2\">\r\n",
       "      <!-- 200 -->\r\n",
       "      <g transform=\"translate(106.787946 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 1228 531 \r\n",
       "L 3431 531 \r\n",
       "L 3431 0 \r\n",
       "L 469 0 \r\n",
       "L 469 531 \r\n",
       "Q 828 903 1448 1529 \r\n",
       "Q 2069 2156 2228 2338 \r\n",
       "Q 2531 2678 2651 2914 \r\n",
       "Q 2772 3150 2772 3378 \r\n",
       "Q 2772 3750 2511 3984 \r\n",
       "Q 2250 4219 1831 4219 \r\n",
       "Q 1534 4219 1204 4116 \r\n",
       "Q 875 4013 500 3803 \r\n",
       "L 500 4441 \r\n",
       "Q 881 4594 1212 4672 \r\n",
       "Q 1544 4750 1819 4750 \r\n",
       "Q 2544 4750 2975 4387 \r\n",
       "Q 3406 4025 3406 3419 \r\n",
       "Q 3406 3131 3298 2873 \r\n",
       "Q 3191 2616 2906 2266 \r\n",
       "Q 2828 2175 2409 1742 \r\n",
       "Q 1991 1309 1228 531 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-32\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_3\">\r\n",
       "     <g id=\"line2d_5\">\r\n",
       "      <path clip-path=\"url(#p52c1335e1b)\" d=\"M 156.188839 143.1 \r\n",
       "L 156.188839 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_6\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"156.188839\" xlink:href=\"#m5cdb5ac206\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_3\">\r\n",
       "      <!-- 300 -->\r\n",
       "      <g transform=\"translate(146.645089 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2597 2516 \r\n",
       "Q 3050 2419 3304 2112 \r\n",
       "Q 3559 1806 3559 1356 \r\n",
       "Q 3559 666 3084 287 \r\n",
       "Q 2609 -91 1734 -91 \r\n",
       "Q 1441 -91 1130 -33 \r\n",
       "Q 819 25 488 141 \r\n",
       "L 488 750 \r\n",
       "Q 750 597 1062 519 \r\n",
       "Q 1375 441 1716 441 \r\n",
       "Q 2309 441 2620 675 \r\n",
       "Q 2931 909 2931 1356 \r\n",
       "Q 2931 1769 2642 2001 \r\n",
       "Q 2353 2234 1838 2234 \r\n",
       "L 1294 2234 \r\n",
       "L 1294 2753 \r\n",
       "L 1863 2753 \r\n",
       "Q 2328 2753 2575 2939 \r\n",
       "Q 2822 3125 2822 3475 \r\n",
       "Q 2822 3834 2567 4026 \r\n",
       "Q 2313 4219 1838 4219 \r\n",
       "Q 1578 4219 1281 4162 \r\n",
       "Q 984 4106 628 3988 \r\n",
       "L 628 4550 \r\n",
       "Q 988 4650 1302 4700 \r\n",
       "Q 1616 4750 1894 4750 \r\n",
       "Q 2613 4750 3031 4423 \r\n",
       "Q 3450 4097 3450 3541 \r\n",
       "Q 3450 3153 3228 2886 \r\n",
       "Q 3006 2619 2597 2516 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-33\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_4\">\r\n",
       "     <g id=\"line2d_7\">\r\n",
       "      <path clip-path=\"url(#p52c1335e1b)\" d=\"M 196.045982 143.1 \r\n",
       "L 196.045982 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_8\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"196.045982\" xlink:href=\"#m5cdb5ac206\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_4\">\r\n",
       "      <!-- 400 -->\r\n",
       "      <g transform=\"translate(186.502232 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 2419 4116 \r\n",
       "L 825 1625 \r\n",
       "L 2419 1625 \r\n",
       "L 2419 4116 \r\n",
       "z\r\n",
       "M 2253 4666 \r\n",
       "L 3047 4666 \r\n",
       "L 3047 1625 \r\n",
       "L 3713 1625 \r\n",
       "L 3713 1100 \r\n",
       "L 3047 1100 \r\n",
       "L 3047 0 \r\n",
       "L 2419 0 \r\n",
       "L 2419 1100 \r\n",
       "L 313 1100 \r\n",
       "L 313 1709 \r\n",
       "L 2253 4666 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-34\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"xtick_5\">\r\n",
       "     <g id=\"line2d_9\">\r\n",
       "      <path clip-path=\"url(#p52c1335e1b)\" d=\"M 235.903125 143.1 \r\n",
       "L 235.903125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_10\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"235.903125\" xlink:href=\"#m5cdb5ac206\" y=\"143.1\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_5\">\r\n",
       "      <!-- 500 -->\r\n",
       "      <g transform=\"translate(226.359375 157.698438)scale(0.1 -0.1)\">\r\n",
       "       <defs>\r\n",
       "        <path d=\"M 691 4666 \r\n",
       "L 3169 4666 \r\n",
       "L 3169 4134 \r\n",
       "L 1269 4134 \r\n",
       "L 1269 2991 \r\n",
       "Q 1406 3038 1543 3061 \r\n",
       "Q 1681 3084 1819 3084 \r\n",
       "Q 2600 3084 3056 2656 \r\n",
       "Q 3513 2228 3513 1497 \r\n",
       "Q 3513 744 3044 326 \r\n",
       "Q 2575 -91 1722 -91 \r\n",
       "Q 1428 -91 1123 -41 \r\n",
       "Q 819 9 494 109 \r\n",
       "L 494 744 \r\n",
       "Q 775 591 1075 516 \r\n",
       "Q 1375 441 1709 441 \r\n",
       "Q 2250 441 2565 725 \r\n",
       "Q 2881 1009 2881 1497 \r\n",
       "Q 2881 1984 2565 2268 \r\n",
       "Q 2250 2553 1709 2553 \r\n",
       "Q 1456 2553 1204 2497 \r\n",
       "Q 953 2441 691 2322 \r\n",
       "L 691 4666 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-35\" transform=\"scale(0.015625)\"/>\r\n",
       "       </defs>\r\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_6\">\r\n",
       "     <!-- epoch -->\r\n",
       "     <g transform=\"translate(123.025 171.376563)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 3597 1894 \r\n",
       "L 3597 1613 \r\n",
       "L 953 1613 \r\n",
       "Q 991 1019 1311 708 \r\n",
       "Q 1631 397 2203 397 \r\n",
       "Q 2534 397 2845 478 \r\n",
       "Q 3156 559 3463 722 \r\n",
       "L 3463 178 \r\n",
       "Q 3153 47 2828 -22 \r\n",
       "Q 2503 -91 2169 -91 \r\n",
       "Q 1331 -91 842 396 \r\n",
       "Q 353 884 353 1716 \r\n",
       "Q 353 2575 817 3079 \r\n",
       "Q 1281 3584 2069 3584 \r\n",
       "Q 2775 3584 3186 3129 \r\n",
       "Q 3597 2675 3597 1894 \r\n",
       "z\r\n",
       "M 3022 2063 \r\n",
       "Q 3016 2534 2758 2815 \r\n",
       "Q 2500 3097 2075 3097 \r\n",
       "Q 1594 3097 1305 2825 \r\n",
       "Q 1016 2553 972 2059 \r\n",
       "L 3022 2063 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-65\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 1159 525 \r\n",
       "L 1159 -1331 \r\n",
       "L 581 -1331 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2969 \r\n",
       "Q 1341 3281 1617 3432 \r\n",
       "Q 1894 3584 2278 3584 \r\n",
       "Q 2916 3584 3314 3078 \r\n",
       "Q 3713 2572 3713 1747 \r\n",
       "Q 3713 922 3314 415 \r\n",
       "Q 2916 -91 2278 -91 \r\n",
       "Q 1894 -91 1617 61 \r\n",
       "Q 1341 213 1159 525 \r\n",
       "z\r\n",
       "M 3116 1747 \r\n",
       "Q 3116 2381 2855 2742 \r\n",
       "Q 2594 3103 2138 3103 \r\n",
       "Q 1681 3103 1420 2742 \r\n",
       "Q 1159 2381 1159 1747 \r\n",
       "Q 1159 1113 1420 752 \r\n",
       "Q 1681 391 2138 391 \r\n",
       "Q 2594 391 2855 752 \r\n",
       "Q 3116 1113 3116 1747 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-70\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 1959 3097 \r\n",
       "Q 1497 3097 1228 2736 \r\n",
       "Q 959 2375 959 1747 \r\n",
       "Q 959 1119 1226 758 \r\n",
       "Q 1494 397 1959 397 \r\n",
       "Q 2419 397 2687 759 \r\n",
       "Q 2956 1122 2956 1747 \r\n",
       "Q 2956 2369 2687 2733 \r\n",
       "Q 2419 3097 1959 3097 \r\n",
       "z\r\n",
       "M 1959 3584 \r\n",
       "Q 2709 3584 3137 3096 \r\n",
       "Q 3566 2609 3566 1747 \r\n",
       "Q 3566 888 3137 398 \r\n",
       "Q 2709 -91 1959 -91 \r\n",
       "Q 1206 -91 779 398 \r\n",
       "Q 353 888 353 1747 \r\n",
       "Q 353 2609 779 3096 \r\n",
       "Q 1206 3584 1959 3584 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6f\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3122 3366 \r\n",
       "L 3122 2828 \r\n",
       "Q 2878 2963 2633 3030 \r\n",
       "Q 2388 3097 2138 3097 \r\n",
       "Q 1578 3097 1268 2742 \r\n",
       "Q 959 2388 959 1747 \r\n",
       "Q 959 1106 1268 751 \r\n",
       "Q 1578 397 2138 397 \r\n",
       "Q 2388 397 2633 464 \r\n",
       "Q 2878 531 3122 666 \r\n",
       "L 3122 134 \r\n",
       "Q 2881 22 2623 -34 \r\n",
       "Q 2366 -91 2075 -91 \r\n",
       "Q 1284 -91 818 406 \r\n",
       "Q 353 903 353 1747 \r\n",
       "Q 353 2603 823 3093 \r\n",
       "Q 1294 3584 2113 3584 \r\n",
       "Q 2378 3584 2631 3529 \r\n",
       "Q 2884 3475 3122 3366 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-63\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3513 2113 \r\n",
       "L 3513 0 \r\n",
       "L 2938 0 \r\n",
       "L 2938 2094 \r\n",
       "Q 2938 2591 2744 2837 \r\n",
       "Q 2550 3084 2163 3084 \r\n",
       "Q 1697 3084 1428 2787 \r\n",
       "Q 1159 2491 1159 1978 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 4863 \r\n",
       "L 1159 4863 \r\n",
       "L 1159 2956 \r\n",
       "Q 1366 3272 1645 3428 \r\n",
       "Q 1925 3584 2291 3584 \r\n",
       "Q 2894 3584 3203 3211 \r\n",
       "Q 3513 2838 3513 2113 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-68\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\r\n",
       "      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-70\"/>\r\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-6f\"/>\r\n",
       "      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-63\"/>\r\n",
       "      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-68\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"matplotlib.axis_2\">\r\n",
       "    <g id=\"ytick_1\">\r\n",
       "     <g id=\"line2d_11\">\r\n",
       "      <path clip-path=\"url(#p52c1335e1b)\" d=\"M 40.603125 103.70169 \r\n",
       "L 235.903125 103.70169 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_12\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 0 0 \r\n",
       "L -3.5 0 \r\n",
       "\" id=\"mae635355ce\" style=\"stroke:#000000;stroke-width:0.8;\"/>\r\n",
       "      </defs>\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"40.603125\" xlink:href=\"#mae635355ce\" y=\"103.70169\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_7\">\r\n",
       "      <!-- 5 -->\r\n",
       "      <g transform=\"translate(27.240625 107.500909)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_2\">\r\n",
       "     <g id=\"line2d_13\">\r\n",
       "      <path clip-path=\"url(#p52c1335e1b)\" d=\"M 40.603125 61.839537 \r\n",
       "L 235.903125 61.839537 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_14\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"40.603125\" xlink:href=\"#mae635355ce\" y=\"61.839537\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_8\">\r\n",
       "      <!-- 10 -->\r\n",
       "      <g transform=\"translate(20.878125 65.638756)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"ytick_3\">\r\n",
       "     <g id=\"line2d_15\">\r\n",
       "      <path clip-path=\"url(#p52c1335e1b)\" d=\"M 40.603125 19.977384 \r\n",
       "L 235.903125 19.977384 \r\n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\r\n",
       "     </g>\r\n",
       "     <g id=\"line2d_16\">\r\n",
       "      <g>\r\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"40.603125\" xlink:href=\"#mae635355ce\" y=\"19.977384\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "     <g id=\"text_9\">\r\n",
       "      <!-- 15 -->\r\n",
       "      <g transform=\"translate(20.878125 23.776602)scale(0.1 -0.1)\">\r\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\r\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-35\"/>\r\n",
       "      </g>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "    <g id=\"text_10\">\r\n",
       "     <!-- perplexity -->\r\n",
       "     <g transform=\"translate(14.798437 100.276562)rotate(-90)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 2631 2963 \r\n",
       "Q 2534 3019 2420 3045 \r\n",
       "Q 2306 3072 2169 3072 \r\n",
       "Q 1681 3072 1420 2755 \r\n",
       "Q 1159 2438 1159 1844 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2956 \r\n",
       "Q 1341 3275 1631 3429 \r\n",
       "Q 1922 3584 2338 3584 \r\n",
       "Q 2397 3584 2469 3576 \r\n",
       "Q 2541 3569 2628 3553 \r\n",
       "L 2631 2963 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-72\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 603 4863 \r\n",
       "L 1178 4863 \r\n",
       "L 1178 0 \r\n",
       "L 603 0 \r\n",
       "L 603 4863 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6c\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3513 3500 \r\n",
       "L 2247 1797 \r\n",
       "L 3578 0 \r\n",
       "L 2900 0 \r\n",
       "L 1881 1375 \r\n",
       "L 863 0 \r\n",
       "L 184 0 \r\n",
       "L 1544 1831 \r\n",
       "L 300 3500 \r\n",
       "L 978 3500 \r\n",
       "L 1906 2253 \r\n",
       "L 2834 3500 \r\n",
       "L 3513 3500 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-78\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 603 3500 \r\n",
       "L 1178 3500 \r\n",
       "L 1178 0 \r\n",
       "L 603 0 \r\n",
       "L 603 3500 \r\n",
       "z\r\n",
       "M 603 4863 \r\n",
       "L 1178 4863 \r\n",
       "L 1178 4134 \r\n",
       "L 603 4134 \r\n",
       "L 603 4863 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-69\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 1172 4494 \r\n",
       "L 1172 3500 \r\n",
       "L 2356 3500 \r\n",
       "L 2356 3053 \r\n",
       "L 1172 3053 \r\n",
       "L 1172 1153 \r\n",
       "Q 1172 725 1289 603 \r\n",
       "Q 1406 481 1766 481 \r\n",
       "L 2356 481 \r\n",
       "L 2356 0 \r\n",
       "L 1766 0 \r\n",
       "Q 1100 0 847 248 \r\n",
       "Q 594 497 594 1153 \r\n",
       "L 594 3053 \r\n",
       "L 172 3053 \r\n",
       "L 172 3500 \r\n",
       "L 594 3500 \r\n",
       "L 594 4494 \r\n",
       "L 1172 4494 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-74\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 2059 -325 \r\n",
       "Q 1816 -950 1584 -1140 \r\n",
       "Q 1353 -1331 966 -1331 \r\n",
       "L 506 -1331 \r\n",
       "L 506 -850 \r\n",
       "L 844 -850 \r\n",
       "Q 1081 -850 1212 -737 \r\n",
       "Q 1344 -625 1503 -206 \r\n",
       "L 1606 56 \r\n",
       "L 191 3500 \r\n",
       "L 800 3500 \r\n",
       "L 1894 763 \r\n",
       "L 2988 3500 \r\n",
       "L 3597 3500 \r\n",
       "L 2059 -325 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-79\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\r\n",
       "      <use x=\"63.476562\" xlink:href=\"#DejaVuSans-65\"/>\r\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-72\"/>\r\n",
       "      <use x=\"166.113281\" xlink:href=\"#DejaVuSans-70\"/>\r\n",
       "      <use x=\"229.589844\" xlink:href=\"#DejaVuSans-6c\"/>\r\n",
       "      <use x=\"257.373047\" xlink:href=\"#DejaVuSans-65\"/>\r\n",
       "      <use x=\"317.146484\" xlink:href=\"#DejaVuSans-78\"/>\r\n",
       "      <use x=\"376.326172\" xlink:href=\"#DejaVuSans-69\"/>\r\n",
       "      <use x=\"404.109375\" xlink:href=\"#DejaVuSans-74\"/>\r\n",
       "      <use x=\"443.318359\" xlink:href=\"#DejaVuSans-79\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "   <g id=\"line2d_17\">\r\n",
       "    <path clip-path=\"url(#p52c1335e1b)\" d=\"M 40.603125 13.377273 \r\n",
       "L 44.588839 40.130645 \r\n",
       "L 48.574554 55.514119 \r\n",
       "L 52.560268 61.741891 \r\n",
       "L 56.545982 66.256558 \r\n",
       "L 60.531696 68.450181 \r\n",
       "L 64.517411 72.64912 \r\n",
       "L 68.503125 76.131257 \r\n",
       "L 72.488839 77.645011 \r\n",
       "L 76.474554 80.893915 \r\n",
       "L 80.460268 83.411662 \r\n",
       "L 84.445982 86.542495 \r\n",
       "L 88.431696 89.01358 \r\n",
       "L 92.417411 91.704869 \r\n",
       "L 96.403125 94.807295 \r\n",
       "L 100.388839 99.189316 \r\n",
       "L 104.374554 102.509216 \r\n",
       "L 108.360268 106.29479 \r\n",
       "L 112.345982 110.233361 \r\n",
       "L 116.331696 113.445906 \r\n",
       "L 120.317411 118.596127 \r\n",
       "L 124.303125 121.690043 \r\n",
       "L 128.288839 126.461441 \r\n",
       "L 132.274554 129.224636 \r\n",
       "L 136.260268 132.115361 \r\n",
       "L 140.245982 134.199791 \r\n",
       "L 144.231696 135.063498 \r\n",
       "L 148.217411 135.583499 \r\n",
       "L 152.203125 136.040409 \r\n",
       "L 156.188839 135.627884 \r\n",
       "L 160.174554 136.13718 \r\n",
       "L 164.160268 136.403708 \r\n",
       "L 168.145982 136.489719 \r\n",
       "L 172.131696 136.583862 \r\n",
       "L 176.117411 136.658538 \r\n",
       "L 180.103125 135.97221 \r\n",
       "L 184.088839 136.542983 \r\n",
       "L 188.074554 136.686211 \r\n",
       "L 192.060268 136.766178 \r\n",
       "L 196.045982 136.740577 \r\n",
       "L 200.031696 136.771579 \r\n",
       "L 204.017411 136.77637 \r\n",
       "L 208.003125 136.859085 \r\n",
       "L 211.988839 136.697983 \r\n",
       "L 215.974554 136.922727 \r\n",
       "L 219.960268 136.850836 \r\n",
       "L 223.945982 136.87635 \r\n",
       "L 227.931696 136.815871 \r\n",
       "L 231.917411 136.804752 \r\n",
       "L 235.903125 136.880987 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_3\">\r\n",
       "    <path d=\"M 40.603125 143.1 \r\n",
       "L 40.603125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_4\">\r\n",
       "    <path d=\"M 235.903125 143.1 \r\n",
       "L 235.903125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_5\">\r\n",
       "    <path d=\"M 40.603125 143.1 \r\n",
       "L 235.903125 143.1 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"patch_6\">\r\n",
       "    <path d=\"M 40.603125 7.2 \r\n",
       "L 235.903125 7.2 \r\n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\r\n",
       "   </g>\r\n",
       "   <g id=\"legend_1\">\r\n",
       "    <g id=\"patch_7\">\r\n",
       "     <path d=\"M 173.628125 29.878125 \r\n",
       "L 228.903125 29.878125 \r\n",
       "Q 230.903125 29.878125 230.903125 27.878125 \r\n",
       "L 230.903125 14.2 \r\n",
       "Q 230.903125 12.2 228.903125 12.2 \r\n",
       "L 173.628125 12.2 \r\n",
       "Q 171.628125 12.2 171.628125 14.2 \r\n",
       "L 171.628125 27.878125 \r\n",
       "Q 171.628125 29.878125 173.628125 29.878125 \r\n",
       "z\r\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_18\">\r\n",
       "     <path d=\"M 175.628125 20.298437 \r\n",
       "L 195.628125 20.298437 \r\n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\r\n",
       "    </g>\r\n",
       "    <g id=\"line2d_19\"/>\r\n",
       "    <g id=\"text_11\">\r\n",
       "     <!-- train -->\r\n",
       "     <g transform=\"translate(203.628125 23.798437)scale(0.1 -0.1)\">\r\n",
       "      <defs>\r\n",
       "       <path d=\"M 2194 1759 \r\n",
       "Q 1497 1759 1228 1600 \r\n",
       "Q 959 1441 959 1056 \r\n",
       "Q 959 750 1161 570 \r\n",
       "Q 1363 391 1709 391 \r\n",
       "Q 2188 391 2477 730 \r\n",
       "Q 2766 1069 2766 1631 \r\n",
       "L 2766 1759 \r\n",
       "L 2194 1759 \r\n",
       "z\r\n",
       "M 3341 1997 \r\n",
       "L 3341 0 \r\n",
       "L 2766 0 \r\n",
       "L 2766 531 \r\n",
       "Q 2569 213 2275 61 \r\n",
       "Q 1981 -91 1556 -91 \r\n",
       "Q 1019 -91 701 211 \r\n",
       "Q 384 513 384 1019 \r\n",
       "Q 384 1609 779 1909 \r\n",
       "Q 1175 2209 1959 2209 \r\n",
       "L 2766 2209 \r\n",
       "L 2766 2266 \r\n",
       "Q 2766 2663 2505 2880 \r\n",
       "Q 2244 3097 1772 3097 \r\n",
       "Q 1472 3097 1187 3025 \r\n",
       "Q 903 2953 641 2809 \r\n",
       "L 641 3341 \r\n",
       "Q 956 3463 1253 3523 \r\n",
       "Q 1550 3584 1831 3584 \r\n",
       "Q 2591 3584 2966 3190 \r\n",
       "Q 3341 2797 3341 1997 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-61\" transform=\"scale(0.015625)\"/>\r\n",
       "       <path d=\"M 3513 2113 \r\n",
       "L 3513 0 \r\n",
       "L 2938 0 \r\n",
       "L 2938 2094 \r\n",
       "Q 2938 2591 2744 2837 \r\n",
       "Q 2550 3084 2163 3084 \r\n",
       "Q 1697 3084 1428 2787 \r\n",
       "Q 1159 2491 1159 1978 \r\n",
       "L 1159 0 \r\n",
       "L 581 0 \r\n",
       "L 581 3500 \r\n",
       "L 1159 3500 \r\n",
       "L 1159 2956 \r\n",
       "Q 1366 3272 1645 3428 \r\n",
       "Q 1925 3584 2291 3584 \r\n",
       "Q 2894 3584 3203 3211 \r\n",
       "Q 3513 2838 3513 2113 \r\n",
       "z\r\n",
       "\" id=\"DejaVuSans-6e\" transform=\"scale(0.015625)\"/>\r\n",
       "      </defs>\r\n",
       "      <use xlink:href=\"#DejaVuSans-74\"/>\r\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-72\"/>\r\n",
       "      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-61\"/>\r\n",
       "      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-69\"/>\r\n",
       "      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-6e\"/>\r\n",
       "     </g>\r\n",
       "    </g>\r\n",
       "   </g>\r\n",
       "  </g>\r\n",
       " </g>\r\n",
       " <defs>\r\n",
       "  <clipPath id=\"p52c1335e1b\">\r\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"40.603125\" y=\"7.2\"/>\r\n",
       "  </clipPath>\r\n",
       " </defs>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "num_inputs = vocab_size\n",
    "gru_layer = nn.GRU(num_inputs, num_hiddens)\n",
    "model = d2l.RNNModel(gru_layer, len(vocab))\n",
    "model = model.to(device)\n",
    "d2l.train_ch8(model, train_iter, vocab, lr, num_epochs, device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "origin_pos": 23
   },
   "source": [
    "## 小结\n",
    "\n",
    "* 门控循环神经网络可以更好地捕获时间步距离很长的序列上的依赖关系。\n",
    "* 重置门有助于捕获序列中的短期依赖关系。\n",
    "* 更新门有助于捕获序列中的长期依赖关系。\n",
    "* 重置门打开时，门控循环单元包含基本循环神经网络；更新门打开时，门控循环单元可以跳过子序列。\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}