PyTorch学习率调整策略.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "PyTorch学习率调整策略.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mengcius/PyTorch-Learning-Rate-Scheduler/blob/master/PyTorch%E5%AD%A6%E4%B9%A0%E7%8E%87%E8%B0%83%E6%95%B4%E7%AD%96%E7%95%A5.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mUfV6Xx2JDIM",
        "colab_type": "text"
      },
      "source": [
        "### PyTorch学习率调整策略\n",
        "PyTorch中学习率调整策略通过 torch.optim.lr_scheduler 接口实现，一共9种方法，可分为三大类：\n",
        "\n",
        "a. 有序调整：等间隔Step调整、指定多间隔MultiStep调整学习率、指数衰减调整Exponential、余弦退火CosineAnnealing\n",
        "\n",
        "b. 自适应调整：自适应调整ReduceLROnPlateau\n",
        "\n",
        "c. 自定义调整：自定义lamda调整LambdaLR\n",
        "\n",
        "```\n",
        "torch.optim.lr_scheduler.LambdaLR 自定义lamda函数\n",
        "torch.optim.lr_scheduler.StepLR 等间隔阶梯下降\n",
        "torch.optim.lr_scheduler.MultiStepLR 指定多间隔step_list阶梯下降\n",
        "torch.optim.lr_scheduler.ExponentialLR 指数下降\n",
        "torch.optim.lr_scheduler.CosineAnnealingLR 余弦退火\n",
        "torch.optim.lr_scheduler.CosineAnnealingWarmRestarts 带热启动的余弦退火\n",
        "torch.optim.lr_scheduler.CyclicLR 循环调整\n",
        "torch.optim.lr_scheduler.OneCycleLR 第一次退火到大学习率\n",
        "torch.optim.lr_scheduler.ReduceLROnPlateau 自适应下降\n",
        "```\n",
        "\n",
        "官网：https://pytorch.org/docs/master/search.html?q=lr_scheduler&check_keywords=yes&area=default\n",
        "\n",
        "参考1：https://blog.csdn.net/junqing_wu/article/details/93248190\n",
        "\n",
        "参考2：https://www.cnblogs.com/wanghui-garcia/p/10895397.html"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-IHKzH7uF0ZH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch\n",
        "import torch.optim as optim\n",
        "from torch.optim import lr_scheduler\n",
        "from torchvision.models import AlexNet\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "model = AlexNet(num_classes=2)\n",
        "optimizer = optim.SGD(params=model.parameters(), lr=0.001)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sxKR62MJF3Lj",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 281
        },
        "outputId": "0412a73d-7404-497c-c1c7-a05d27e07e2a"
      },
      "source": [
        "'''\n",
        " 自定义lamda函数\n",
        " 参数 lr_lambda 给定一个lambda函数，将epoch作用于该函数，生成scale*init_lr，可以用来定义自己的lamda函数\n",
        "'''\n",
        "lambda1 = lambda epoch: epoch // 30 # epoch//30 * init_lr（0.05）\n",
        "lambda2 = lambda epoch: 0.95 ** epoch \n",
        "scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda= lambda2)\n",
        "\n",
        "plt.figure()\n",
        "x = list(range(100))\n",
        "y = []\n",
        "for epoch in range(100):\n",
        "    optimizer.step()\n",
        "    scheduler.step()\n",
        "    lr = scheduler.get_lr()\n",
        "    y.append(scheduler.get_lr()[0])   \n",
        "plt.plot(x, y)"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f6b26294f28>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de3xV5Z3v8c9v750LuZCQEEJIAgS5\nBhCQiNfWW1WoF2zrBVsdnerYnmrt1M50cGZOO+OZno6nZ2p1qq31NtpaEW/Isa1WxVawCgRFrgKR\ncEnkEm653/OcP/bCSWISAklYO3t/369XXnvttdZ+8nteS/NlrWft9ZhzDhERkaMCfhcgIiKRRcEg\nIiIdKBhERKQDBYOIiHSgYBARkQ5CfhfQH4YPH+7Gjh3rdxkiIoPKmjVrDjjnsjqvj4pgGDt2LMXF\nxX6XISIyqJjZzq7W61KSiIh0oGAQEZEOFAwiItKBgkFERDpQMIiISAcKBhER6UDBICIiHcR0MCz9\n8BN+816Xt/GKiMSsmA6GVzfs4f43t9HWpjkpRESOiulguKRwJBXVjawtO+J3KSIiESOmg+GCSSMI\nBozXN+3zuxQRkYgR08GQlhTHGQUZCgYRkXZiOhgALi7MpmR/DaUHav0uRUQkIigYCrMBeH3TXp8r\nERGJDDEfDHnDkpiSM1SXk0REPDEfDBA+a1iz8zAHaxr9LkVExHcKBuCSwmzaHLz50X6/SxER8Z2C\nAZg6aiij0hL540ZdThIRUTAAZsbFhdks31ZBbWOL3+WIiPhKweCZOy2HxpY2/rSlwu9SRER8pWDw\nzCnIIDM5nj9s2ON3KSIivlIweIIB45KpI1n20X4amlv9LkdExDcKhnbmTRtJXVMrb2/V5SQRiV0K\nhnbOOiWTtCFxvLpB34IWkdilYGgnLhjg4sJsXt+8j6aWNr/LERHxhYKhk3nTRlLd0MI7Hx/wuxQR\nEV8oGDo5d8JwUhJCvLpel5NEJDYpGDpJCAW5aMoI/rhpL82tupwkIrFHwdCFy6bncLiumb98fNDv\nUkRETjoFQxfOm5RFamKI//fhJ36XIiJy0ikYupAQCnJJ4Uhe27iXxhZ92U1EYouCoRtXzMihuqGF\nt7fq7iQRiS0Khm6cM344w5LidDlJRGKOgqEbccEAc6fl8MbmfdQ36XKSiMQOBUMPrpiRQ11TK8s0\ns5uIxBAFQw/OKMgkKzVBl5NEJKb0KhjMbK6ZbTGzEjNb2MX2BDN71tu+0szGttt2t7d+i5ldeqw2\nzewiM3vfzNaa2QozG9+3Lp64YMC4bHoOy7bsp6qh2a8yREROqmMGg5kFgQeBeUAhcL2ZFXba7Rbg\nsHNuPHAfcK/32UJgATAVmAs8ZGbBY7T5C+BrzrmZwG+Bf+5bF/tm/sxRNLW06REZIhIzenPGMAco\ncc5td841AYuA+Z32mQ886S0/D1xkZuatX+Sca3TOlQIlXns9temAod5yGuDrdZyZ+ekUDE/mpQ/K\n/SxDROSk6U0w5AK7270v89Z1uY9zrgWoBDJ7+GxPbd4K/N7MyoAbgX/vTUcGiplx1cxc3is9yCdH\n6v0sRUTkpIjEwefvAl90zuUBTwA/7WonM7vNzIrNrLiiYmBnXLtq1iicg6UahBaRGNCbYCgH8tu9\nz/PWdbmPmYUIXwI62MNnu1xvZlnADOfcSm/9s8DZXRXlnPuVc67IOVeUlZXVi26cuDGZyZw2Op2X\n3i/HOTegv0tExG+9CYbVwAQzKzCzeMKDyUs77bMUuMlbvhpY5sJ/QZcCC7y7lgqACcCqHto8DKSZ\n2USvrYuBzSfevf7zpdPy2LKvms17qv0uRURkQB0zGLwxgzuA1wj/kV7snNtoZveY2ZXebo8BmWZW\nAtwFLPQ+uxFYDGwCXgVud861dtemt/5vgBfM7EPCYwx/33/dPXGXT88hFDCWrNUgtIhEN4uGSyNF\nRUWuuLh4wH/PrU8Ws67sCO/efRHBgA347xMRGUhmtsY5V9R5fSQOPkesL5+Wy/7qRlaU6ImrIhK9\nFAzH4aIpI0hPiuO54t3H3llEZJBSMByHhFCQq2bm8seN+zhS1+R3OSIiA0LBcJyunp1HU2ubvtMg\nIlFLwXCcpuWmMSVnKM8Vl/ldiojIgFAwnIBrZuexvrySj/ZW+V2KiEi/UzCcgKtm5RIXNJ01iEhU\nUjCcgIzkeL4wJZslH5TT1NLmdzkiIv1KwXCCri3K52BtE29u3ud3KSIi/UrBcII+PzGLUWmJ/HbV\nLr9LERHpVwqGExQMGNedPprl2w6w62Cd3+WIiPQbBUMfXHt6HgGDRat11iAi0UPB0Ac5aUO4cPII\nFheX0dyqQWgRiQ4Khj766hmjOVDTqEFoEYkaCoY+Om/iCHLSEnl6pS4niUh0UDD0UXgQOl+D0CIS\nNRQM/WDB6aMJBoynV+70uxQRkT5TMPSDkWmJXDo1m2eLd9PQ3Op3OSIifaJg6Cc3njmWI3XNehy3\niAx6CoZ+cua4DCZmp/DkX3YQDfNoi0jsUjD0EzPjxrPGsvGTKt7fdcTvckRETpiCoR99eVYuqQkh\nnnp3h9+liIicMAVDP0pOCPGV2Xn8fv0eKqob/S5HROSEKBj62Y1njaG51enWVREZtBQM/eyUrBQu\nmJTFb97bqVtXRWRQUjAMgFs/N44DNU0sXatbV0Vk8FEwDICzT8lk8shUHn+nVLeuisigo2AYAGbG\nLecW8NHeat4pOeh3OSIix0XBMECunDmK4SkJPLZiu9+liIgcFwXDAEkIBbnxzDG8taWCkv3Vfpcj\nItJrCoYBdMOZo0kIBXh0eanfpYiI9JqCYQBlpiRwbVE+L75fzr6qBr/LERHpFQXDAPubz42jpa2N\nx1forEFEBodeBYOZzTWzLWZWYmYLu9ieYGbPettXmtnYdtvu9tZvMbNLj9Wmhf3IzLaa2WYzu7Nv\nXfTX6MwkLjt1FE+v3EVlfbPf5YiIHNMxg8HMgsCDwDygELjezAo77XYLcNg5Nx64D7jX+2whsACY\nCswFHjKz4DHavBnIByY756YAi/rUwwjwzfPGUdPYwm/e02MyRCTy9eaMYQ5Q4pzb7pxrIvyHen6n\nfeYDT3rLzwMXmZl56xc55xqdc6VAiddeT23+D+Ae51wbgHNu/4l3LzJMHZXG5ydm8cQ7O/SYDBGJ\neL0Jhlxgd7v3Zd66LvdxzrUAlUBmD5/tqc1TgOvMrNjM/mBmE3rXlcj2zfPGcaCmkefWlPldiohI\njyJx8DkBaHDOFQGPAI93tZOZ3eaFR3FFRcVJLfBEnDUuk1mj0/nlnz6mqaXN73JERLrVm2AoJ3zN\n/6g8b12X+5hZCEgDDvbw2Z7aLANe9JZfAk7tqijn3K+cc0XOuaKsrKxedMNfZsadF06g/Eg9L32g\nswYRiVy9CYbVwAQzKzCzeMKDyUs77bMUuMlbvhpY5sJPj1sKLPDuWioAJgCrjtHmEuACb/k8YOuJ\ndS3ynD8pi+m5aTz41se0tOqsQUQi0zGDwRszuAN4DdgMLHbObTSze8zsSm+3x4BMMysB7gIWep/d\nCCwGNgGvArc751q7a9Nr69+Br5jZeuDHwK3901X/mRnfvnA8uw7V8bIeyS0iEcqi4bHQRUVFrri4\n2O8yesU5xxcfWEFjcyuv33UewYD5XZKIxCgzW+ON53YQiYPPUe3oWcP2A7W8sk5nDSISeRQMPpg7\ndSSTslO5/81tGmsQkYijYPBBIGB89+KJbK+oZYnGGkQkwigYfHLp1Gym5Q7lZ29s1fcaRCSiKBh8\nYmZ875JJlB2uZ3Hx7mN/QETkJFEw+Oj8iVkUjRnGfy7bpmcoiUjEUDD46OhZw76qRj15VUQihoLB\nZ2edksm544fz4FslVDVovgYR8Z+CIQIsnDeZw3XNPPznj/0uRUREwRAJpuWmMX/mKB5bUcreSs0N\nLSL+UjBEiL+7ZBJtbXDf61HzzEARGaQUDBEiPyOJG84cw3NrdrNtX7Xf5YhIDFMwRJBvXzie5IQQ\nP/7DR36XIiIxTMEQQYYlx3PHBeNZ9tF+3t4a+bPSiUh0UjBEmJvPGcuYzCT+1yub9IA9EfGFgiHC\nJISC/OMXp7Btfw3PrNrldzkiEoMUDBHoksJszhqXyU9f30plnb70JiInl4IhApkZP7iikMr6Zn72\npm5fFZGTS8EQoabkDGXBnNE89e5OPtpb5Xc5IhJDFAwR7O8vmcTQxBA/WLKRaJibW0QGBwVDBBuW\nHM8/zJ3Mqh2HWLK23O9yRCRGKBgi3LVF+czIT+dHv/uIynoNRIvIwFMwRLhAwPi3+dM4WNuo5yiJ\nyEmhYBgEpuelccMZY3jq3R18uPuI3+WISJRTMAwSfz93ElmpCSx8cT3N+ka0iAwgBcMgMTQxjn+9\nchqb91Tx2IpSv8sRkSimYBhE5k4bySWF2fzsja3sPFjrdzkiEqUUDIPMPfOnEQoE+KeXNui7DSIy\nIBQMg8zItET+Yd5kVpQcYNHq3X6XIyJRSMEwCH1tzmjOGpfJj363mfIj9X6XIyJRRsEwCAUCxv+5\n+lTanGPhC+t0SUlE+pWCYZDKz0ji7i9OYfk2XVISkf6lYBjE2l9S2n2ozu9yRCRK9CoYzGyumW0x\nsxIzW9jF9gQze9bbvtLMxrbbdre3fouZXXocbT5gZjUn1q3YEAgYP7nmVAz47rNraW3TJSUR6btj\nBoOZBYEHgXlAIXC9mRV22u0W4LBzbjxwH3Cv99lCYAEwFZgLPGRmwWO1aWZFwLA+9i0m5A1L4p6r\nplK88zC//PPHfpcjIlGgN2cMc4AS59x251wTsAiY32mf+cCT3vLzwEVmZt76Rc65RudcKVDitddt\nm15o/AT4ft+6FjuumpnL5afmcN/rW1lXpmcpiUjf9CYYcoH2o5tl3rou93HOtQCVQGYPn+2pzTuA\npc65PT0VZWa3mVmxmRVXVFT0ohvRy8z40VXTyUpN4G8XraWuqcXvkkRkEIuowWczGwVcA/znsfZ1\nzv3KOVfknCvKysoa+OIiXFpSHP9x7QxKD9byg5c3+l2OiAxivQmGciC/3fs8b12X+5hZCEgDDvbw\n2e7WzwLGAyVmtgNIMrOSXvYl5p19ynC+fcF4nl9Txovvl/ldjogMUr0JhtXABDMrMLN4woPJSzvt\nsxS4yVu+Gljmwt+6Wgos8O5aKgAmAKu6a9M59zvn3Ejn3Fjn3FigzhvQll6686IJzBmbwT8v2cDH\nFbqpS0SO3zGDwRszuAN4DdgMLHbObTSze8zsSm+3x4BM71/3dwELvc9uBBYDm4BXgdudc63dtdm/\nXYtNoWCA+6+fSUIowB2//YCG5la/SxKRQcai4XEKRUVFrri42O8yIsqyj/bx9f8q5rqifO69+lS/\nyxGRCGRma5xzRZ3XR9Tgs/SfCydnc8cF43m2eDfPrt7ldzkiMogoGKLYdy+eyOcmDOd/vryR9WWV\nfpcjIoOEgiGKBQPG/QtmMTw5nm/+Zg2Ha5v8LklEBgEFQ5TLSI7nFzfMpqKmkW89/T7NrW1+lyQi\nEU7BEANm5Kfz4y9N593tB/m3Vzb5XY6IRLiQ3wXIyfGV2Xl8tLeKR5aXMjlnKNfPGe13SSISoXTG\nEEMWzpvCeROz+MHLG3hv+0G/yxGRCKVgiCHBgPHA9bMYk5nMN369Rt+MFpEuKRhiTNqQOJ64+XTi\ngsZfP7GagzWNfpckIhFGwRCD8jOSeOSvithX1cCtTxXrsRki0oGCIUbNGj2M+xfMZO3uI9z5zAe0\n6DZWEfEoGGLY3Gk5/PDyQv64aR//8+UNRMNzs0Sk73S7aoy7+ZwCDtQ08fO3SshKSeCuSyb5XZKI\n+EzBIHzvkolUVDfywLIS0pPi+fq5BX6XJCI+UjBIeM7oL03jSH0T97yyiaT4IAv0BTiRmKUxBgHC\nE/w8cP0szpuYxd0vrefltZ1nbxWRWKFgkE8lhII8fONszijI4K7FH/Lqhj1+lyQiPlAwSAeJcUEe\nvel0ZuSlccdvP+AP6xUOIrFGwSCfkZIQ4smvz2FGfjp3PPMBv1c4iMQUBYN0KTUxjie/PodZ+el8\n+5kPeGXdJ36XJCIniYJBupWSEOK/vj6H2aOHceczH7B49W6/SxKRk0DBID06elnpnPHD+f4L63h8\nRanfJYnIAFMwyDENiQ/y6E1FzJ06knte2cTP3tiqx2eIRDEFg/RKQijIz786i6tn5/GzN7bxT0s2\n0NqmcBCJRvrms/RaKBjgJ1efyojUBB7608ccqG7kgetnkRgX9Ls0EelHOmOQ42JmfH/uZP71yqm8\nvnkfX33kPU32IxJlFAxyQm46eywPffU0Nn5SxVUPvcO2fdV+lyQi/UTBICds3vQcnv3GWdQ3tfHl\nX/yF5dsq/C5JRPqBgkH6ZGZ+OktuP5tRaUO4+YnVPL6iVHcsiQxyCgbps7xhSbzwrbO5cPII7nll\nE3/33DrNIy0yiCkYpF+kJIR4+IbZfOeiCbzwfhnXPfwu5Ufq/S5LRE6AgkH6TSBgfPfiiTx842y2\nV9Ry+QPL+fNWjTuIDDYKBul3l04dydJvn0v20ERufmIVP319q74MJzKI9CoYzGyumW0xsxIzW9jF\n9gQze9bbvtLMxrbbdre3fouZXXqsNs3saW/9BjN73Mzi+tZF8UPB8GRe+tY5fHlWHg+8uY3rH3mP\nPZW6tCQyGBwzGMwsCDwIzAMKgevNrLDTbrcAh51z44H7gHu9zxYCC4CpwFzgITMLHqPNp4HJwHRg\nCHBrn3oovhkSH+Q/rp3Bf1wzgw3llcy7fzl/3LjX77JE5Bh6c8YwByhxzm13zjUBi4D5nfaZDzzp\nLT8PXGRm5q1f5JxrdM6VAiVee9226Zz7vfMAq4C8vnVR/PaV2Xm88u1zyU0fwm2/XsPdL66jtrHF\n77JEpBu9CYZcoP2D+Mu8dV3u45xrASqBzB4+e8w2vUtINwKvdlWUmd1mZsVmVlxRoQHOSDcuK4UX\nv3U23zzvFBat3s28+5ezZuchv8sSkS5E8uDzQ8DbzrnlXW10zv3KOVfknCvKyso6yaXJiUgIBVk4\nbzLP3nYWbc5xzS/f5Ue/26TvPIhEmN4EQzmQ3+59nreuy33MLASkAQd7+GyPbZrZD4Es4K7edEIG\nlzkFGfzhO59jwZzRPLK8lHn3L2f1Dp09iESK3gTDamCCmRWYWTzhweSlnfZZCtzkLV8NLPPGCJYC\nC7y7lgqACYTHDbpt08xuBS4FrnfOtfWtexKpUhPj+N9fms7Tt55Bc2sb1z78Lv/00noq65v9Lk0k\n5h0zGLwxgzuA14DNwGLn3EYzu8fMrvR2ewzINLMSwv/KX+h9diOwGNhEeKzgdudca3dtem39EsgG\n3jWztWb2g37qq0Sgc8YP57W//TxfP6eAZ1bt4gs//TOvrPtEz1sS8ZFFw/+ARUVFrri42O8ypI/W\nl1Vy90vr2FBexecmDOeHV0xl/IgUv8sSiVpmtsY5V9R5fSQPPkuMmZ6XxpJvncO/XFHI2t1HmHf/\n2/z4D5up0a2tIieVgkEiSigY4OZzClj2vfO5amYuD/95O+f/5E8sWrVLj9UQOUkUDBKRslIT+Mk1\nM1hy+zmMyUxi4YvrueyB5by9tULjDyIDTMEgEW1mfjrPf/MsHvzqadQ0tvBXj6/ia4+u5MPdR/wu\nTSRqKRgk4pkZl52aw5vfO48fXlHIR3urmf/gO3zj18V8tLfK7/JEoo7uSpJBp7qhmUeWl/L4ilJq\nm1q4bHoO37loAhOyU/0uTWRQ6e6uJAWDDFpH6pp4ZPl2nnhnB/XNrcybNpLbLxjP1FFpfpcmMigo\nGCRqHapt4vEVpTz5lx1UN7ZwwaQsvnHeKZxRkEH4Ib8i0hUFg0S9yvpmnvrLDp74yw4O1TYxIz+d\n2z43jkunZhMKajhNpDMFg8SMhuZWnltTxqPLt7PzYB256UO46ewxXFc0mrQkTQgocpSCQWJOa5vj\njc37eHxFKStLD5EYF+CqmbnccOYYpuVqHEJEwSAxbUN5Jb95bydL1pbT0NzGrNHpXH/6aC6fkUNS\nfMjv8kR8oWAQASrrmnn+/TKeWbWLkv01pCSEuHLmKK6ZncfM/HQNVktMUTCItOOco3jnYZ5ZtYvf\nr99DQ3Mbp2Ql85XZecyfmUtu+hC/SxQZcAoGkW5UNzTz+/V7eK64jOKdhwGYMzaD+bNGMW9aDhnJ\n8T5XKDIwFAwivbDrYB0vry1nydpyPq6oJRgwzhk/nMun53BxYTbDFBISRRQMIsfBOcemPVW8sm4P\nr6z7hN2H6gkGjDPHZTB36ki+UJhNTpouN8ngpmAQOUHOOdaXV/Lqhr28umEv2w/UAjAtdyhfmJLN\nRZOzmTpqKIGABq5lcFEwiPQD5xwl+2t4Y/N+3ti8j/d3Hca58PwR50/M4vMTszh3/HBdcpJBQcEg\nMgAO1DTy9tYK3tpSwdtbK6isb8YMTs1L59zxmZxzynBOGzOMxLig36WKfIaCQWSAtbY5Piw7wttb\nwyHxYVklrW2OhFCA00YP48xxmZw5LoMZ+ekKCokICgaRk6y6oZlVpYd4p+QgK0sPsmlPFc5BfDDA\njPw0isZmUDRmGKeNHqZLT+ILBYOIzyrrmlm14xCrdxxiVekhNpRX0tIW/v9v3PBkZo0exsz8NGbm\nD2NyTipxeiKsDDAFg0iEqW9qZV3ZEdbsOsz7Ow+zdvcRDtQ0ARAfCjAlZyin5qYxPTeNwlFDmZid\nSnxIYSH9R8EgEuGcc5Qdrmft7iOsL69kXdkRNpRXUdPYAkBc0JgwIpUpOUOZkpNKYc5QJo1MJTMl\nwefKZbBSMIgMQm1tjh0Ha9n4SRUbP6li054qNu+poqK68dN9hqfEMzE7lQkjUpjgvZ4yIoXM5Hg9\nFFB61F0w6HnDIhEsEDDGZaUwLiuFK2aM+nT9gZpGNu+pYsvearbuq2bL3mqeX1NGbVPrp/ukJ8Ux\nbngy47JSKBiezLjhyYwdnsyYzCQ9alx6pDMGkSjhnGNPZQNb91WzvaKWkooaPt5fQ+mBWva3O8MA\nyB6awJiMZPIzkhidkUR+xhDyM5LIH5bEiNQEfYs7RuiMQSTKmRmj0ocwKn0I50/quK2msYUdB2rZ\ncbCWnQfrKD1Qy65DdbxTcoAXqho67BsfDJCTnkiu19ao9CGMSkskx3sdmZZIaqKmSI1mCgaRGJCS\nEGJablqXU5o2NLdSfqSe3Yfq2H24nvLD9ZQfqaf8cB0rth1gf3UDbZ0uLCTHB8kemuj9JDBiaCIj\nUhPISk1gRGoiWakJZKUkMHRISOMcg5CCQSTGJcYFOSUrhVOyUrrc3tzaxr6qBvZUej9H6tlb1cD+\nqkb2VjVQvPMw+6sbaWpp+8xn44MBMlPiGZ6SQGZKPBnJ4eWM5HgyksLvhyXHMSwpnmFJ8QwdEkdQ\nl7F8p2AQkR7FBQPkDUsib1hSt/s456isb2Z/dSMHqhupqGmkwns9UN3EgZpGDtY0sW1fDRU1XYcI\ngBmkDQkHRdqQONKGxJGeFH4dmui9DgkxNDGOoUPiSE0ML6cmhkhNjNP3PPqJgkFE+szMSE+KJz0p\nfOtsT5xz1DW1cqi2iUO1TRyua+JIXTOHaps4UtfE4bpmDtc1UVkffi09UEtVQzNV9c2fuaTVWXwo\nwNDEECkJIZITwq9Hl8Pvg+Hl+BBJCcHwa3yQpPgQQ+KD3nLQWw4xJC4Yk2cwvQoGM5sL3A8EgUed\nc//eaXsC8BQwGzgIXOec2+Ftuxu4BWgF7nTOvdZTm2ZWACwCMoE1wI3Ouaa+dVNEIoWZffqHOj+j\n+7OQztraHDVNLVTVN1NV30JVQzPVDS1Ud3qtamihtrGFGu9nT2UDtU3/va6hueuzle7EhwIMiQsy\nJC5IYlyAxLig9+MthzquTwgFSDj6+ulPkPhQgHjvfXwoQHww0GFdnPf+6Gt8MLzsRzAdMxjMLAg8\nCFwMlAGrzWypc25Tu91uAQ4758ab2QLgXuA6MysEFgBTgVHAG2Y20ftMd23eC9znnFtkZr/02v5F\nf3RWRAavQMDCl5AS42DYibfT2uaoa2qhtrGVuqYW6ppaqW1soa65lfqmVuqaWqlvbqXe21bf3ErD\n0dfmNu+1lcbmNg7WNIWXW9pobAlvb2wJv++vbwIELHw5Lz4YIBQ04rzAiPOWH7vpdEZn9j5ge6M3\nZwxzgBLn3HYAM1sEzAfaB8N84F+85eeBn1v4VoT5wCLnXCNQamYlXnt01aaZbQYuBL7q7fOk166C\nQUT6RTBgpCbGDegtt845mlrbaGppo7GljYbmVppa2mhqbaOxue3TbUe3Nx9939pxuaXVffq+2VsO\n/4SXW9raSIjr/3GV3gRDLrC73fsy4Izu9nHOtZhZJeFLQbnAe50+m+std9VmJnDEOdfSxf4dmNlt\nwG0Ao0eP7kU3RERODjMjIRQkIRSk5xGXyDRoh/Cdc79yzhU554qysrL8LkdEJGr0JhjKgfx27/O8\ndV3uY2YhII3wIHR3n+1u/UEg3Wuju98lIiIDqDfBsBqYYGYFZhZPeDB5aad9lgI3ectXA8tc+CFM\nS4EFZpbg3W00AVjVXZveZ97y2sBr8+UT756IiByvY44xeGMGdwCvEb619HHn3EYzuwcods4tBR4D\nfu0NLh8i/Iceb7/FhAeqW4DbnXOtAF216f3KfwAWmdm/AR94bYuIyEmip6uKiMSo7p6uOmgHn0VE\nZGAoGEREpAMFg4iIdBAVYwxmVgHsPMGPDwcO9GM5g0Us9jsW+wyx2W/1uXfGOOc+80WwqAiGvjCz\n4q4GX6JdLPY7FvsMsdlv9blvdClJREQ6UDCIiEgHCgb4ld8F+CQW+x2LfYbY7Lf63AcxP8YgIiId\n6YxBREQ6UDCIiEgHMR0MZjbXzLaYWYmZLfS7noFgZvlm9paZbTKzjWb2HW99hpm9bmbbvNc+TJYY\nmcwsaGYfmNkr3vsCM1vpHe9nvSf7RhUzSzez583sIzPbbGZnRfuxNrPvev9tbzCzZ8wsMRqPtZk9\nbmb7zWxDu3VdHlsLe8Dr/zozO+14flfMBkO7uaznAYXA9d4c1dGmBfiec64QOBO43evnQuBN59wE\n4E3vfbT5DrC53fuj84mPB6zq78IAAALESURBVA4Tnk882twPvOqcmwzMINz/qD3WZpYL3AkUOeem\nEX5a89F556PtWP8XMLfTuu6O7TzC0xxMIDzT5XFNjxyzwUC7uaydc03A0bmso4pzbo9z7n1vuZrw\nH4pcwn190tvtSeAqfyocGGaWB1wGPOq9N8LziT/v7RKNfU4DPo/3qHrnXJNz7ghRfqwJTx8wxJvg\nKwnYQxQea+fc24SnNWivu2M7H3jKhb1HeAK0nN7+rlgOhq7msu5yfuloYWZjgVnASiDbObfH27QX\nyPaprIHyM+D7QJv3vtfziQ9iBUAF8IR3Ce1RM0smio+1c64c+L/ALsKBUAmsIfqP9VHdHds+/X2L\n5WCIKWaWArwA/K1zrqr9Nm/mvKi5b9nMLgf2O+fW+F3LSRYCTgN+4ZybBdTS6bJRFB7rYYT/dVwA\njAKS+ezllpjQn8c2loOhN3NZRwUziyMcCk875170Vu87emrpve73q74BcA5wpZntIHyJ8ELC196j\nfT7xMqDMObfSe/884aCI5mP9BaDUOVfhnGsGXiR8/KP9WB/V3bHt09+3WA6G3sxlPeh519YfAzY7\n537ablP7ebqjam5t59zdzrk859xYwsd1mXPua0T5fOLOub3AbjOb5K26iPC0ulF7rAlfQjrTzJK8\n/9aP9jmqj3U73R3bpcBfeXcnnQlUtrvkdEwx/c1nM/si4WvRR+ed/pHPJfU7MzsXWA6s57+vt/8j\n4XGGxcBowo8sv9Y513lga9Azs/OBv3POXW5m4wifQWQQnk/8Budco5/19Tczm0l4wD0e2A78NeF/\nAEbtsTazfwWuI3wH3gfArYSvp0fVsTazZ4DzCT9eex/wQ2AJXRxbLyR/TviyWh3w1865Xs9/HNPB\nICIinxXLl5JERKQLCgYREelAwSAiIh0oGEREpAMFg4iIdKBgEBGRDhQMIiLSwf8HffccOsnBxxEA\nAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jCvsuCmEF88b",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 281
        },
        "outputId": "3c2eab1e-1cbb-4c5d-f5d4-449007134271"
      },
      "source": [
        "'''\n",
        " 阶梯递降学习率\n",
        " StepLR:需要指定[step_size,gamma]，step_size为下降频率，gamma为decay衰减系数\n",
        " MultiStepLR:则指定一个step_list，每个指定的step则乘以gamma衰减系数，milestones=[30,60,80]\n",
        "'''\n",
        "# scheduler = lr_scheduler.StepLR(optimizer, step_size=20, gamma=0.5)\n",
        "scheduler = lr_scheduler.MultiStepLR(optimizer, milestones=[30,60,80],gamma= 0.5)\n",
        "\n",
        "plt.figure()\n",
        "x = list(range(100))\n",
        "y = []\n",
        "for epoch in range(100):\n",
        "    optimizer.step()\n",
        "    scheduler.step()\n",
        "    lr = scheduler.get_lr()\n",
        "    y.append(scheduler.get_lr()[0])\n",
        "plt.plot(x, y)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f6b25d6c128>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAWt0lEQVR4nO3df4xd5X3n8fd37vWdAcYMBFzs2Dh2\nayeRadqkdUn2dwqJMGlVr1awNUq3bgtFqkBt0l11TbWiu2gtBe2q7GYL6bLFLUVsDKLQznbd0G7I\nKu0qMZikbWJTNyNoil0DDhDzI/jHjL/7xz0mM9MZz7XnHN85Z94vCXHuc8955nl0rPnM8zznR2Qm\nkiSdMtDvBkiSFhaDQZI0hcEgSZrCYJAkTWEwSJKmaPe7AWW49NJLc82aNf1uhiTVytNPP/2tzFw2\nvbwRwbBmzRr27NnT72ZIUq1ExDdnKncqSZI0hcEgSZrCYJAkTWEwSJKmMBgkSVP0FAwRsSki9kfE\nWERsm+H7wYh4qPh+d0SsmfTdbUX5/oi4ZlL5joh4KSK+Pq2ud0TEn0TEN4r/X3z23ZMknak5gyEi\nWsDdwLXABuCGiNgwbbcbgVczcx1wF3BncewGYAtwBbAJuKeoD+B3irLptgGfz8z1wOeLz5Kkc6SX\n+xiuBMYy81mAiNgJbAb2TdpnM/Dvi+1HgN+IiCjKd2bmMeC5iBgr6vtSZn5x8shiWl0fLrbvB/4v\n8G977tEZeOyrB3ju8JtVVL3ovX/1RVz13sv63QxJZ6GXYFgJPD/p8wHgg7Ptk5njEXEEuKQo//K0\nY1fO8fMuy8xDxfYLwIy/XSLiZuBmgNWrV8/dixn8r784xBf2v3RWx2p2mbDyovO4apvBINXRgr7z\nOTMzImZ8k1Bm3gvcC7Bx48azetvQjp/5kXm0TrP51ce+xh/vfaHfzZB0lnpZfD4IXD7p86qibMZ9\nIqINjAAv93jsdC9GxIqirhWAf9LXTKc1wLHxk/1uhqSz1EswPAWsj4i1EdGhu5g8Om2fUWBrsX0d\n8ER23xk6CmwprlpaC6wHnpzj502uayvwBz20UQtIpz3AcYNBqq05gyEzx4FbgceBZ4CHM3NvRNwR\nET9R7HYfcEmxuPzLFFcSZeZe4GG6C9WfA27JzAmAiPgs8CXgPRFxICJuLOr6FPDRiPgG8JHis2qk\n0xrg+MRJfJ+4VE89rTFk5i5g17Sy2ydtHwWun+XY7cD2GcpvmGX/l4Gre2mXFqZOe4BMGD+ZLGlF\nv5sj6Qx557NK12l3/1k5nSTVk8Gg0nVaBoNUZwaDSvf2iGHCYJDqyGBQ6ZxKkurNYFDpBotg8F4G\nqZ4MBpXONQap3gwGlc41BqneDAaVzjUGqd4MBpXu1FTSCUcMUi0ZDCqdIwap3gwGla7jVUlSrRkM\nKt2gi89SrRkMKl2n1X2tt1NJUj0ZDCqdawxSvRkMKt13g2Gizy2RdDYMBpXOG9ykejMYVDofiSHV\nm8Gg0p16a5vBINWTwaDSRQSd9gDHnEqSaslgUCUGWwOOGKSaMhhUiU7bYJDqymBQJQwGqb4MBlWi\n0x7wclWppgwGVaLjGoNUWwaDKuFUklRfBoMq4VSSVF8GgyrRaQ34PgappgwGVcKpJKm+DAZVYtBg\nkGrLYFAllrRcY5DqymBQJZxKkurLYFAlvI9Bqi+DQZXwclWpvgwGVcKpJKm+DAZVwmCQ6stgUCUG\ni6uSMrPfTZF0hnoKhojYFBH7I2IsIrbN8P1gRDxUfL87ItZM+u62onx/RFwzV50RcXVEfCUi/jwi\n/iwi1s2vi+qHTrt477PrDFLtzBkMEdEC7gauBTYAN0TEhmm73Qi8mpnrgLuAO4tjNwBbgCuATcA9\nEdGao87PAB/PzPcD/xP4d/Provrh7WBwOkmqnV5GDFcCY5n5bGYeB3YCm6ftsxm4v9h+BLg6IqIo\n35mZxzLzOWCsqO90dSZwYbE9Avzd2XVN/dRpGQxSXbV72Gcl8PykzweAD862T2aOR8QR4JKi/MvT\njl1ZbM9W503Aroh4C3gN+NBMjYqIm4GbAVavXt1DN3QuddotAE5MuMYg1c1CXHz+JPCxzFwF/Dbw\n6zPtlJn3ZubGzNy4bNmyc9pAzc2pJKm+egmGg8Dlkz6vKspm3Cci2nSngF4+zbEzlkfEMuAHM3N3\nUf4Q8A976okWlO8uPk/0uSWSzlQvwfAUsD4i1kZEh+5i8ui0fUaBrcX2dcAT2b1OcRTYUly1tBZY\nDzx5mjpfBUYi4t1FXR8Fnjn77qlfTq0x+E4GqX7mXGMo1gxuBR4HWsCOzNwbEXcAezJzFLgPeCAi\nxoBX6P6ip9jvYWAfMA7ckpkTADPVWZT/PPB7EXGSblD8XKk91jkx6FSSVFu9LD6TmbuAXdPKbp+0\nfRS4fpZjtwPbe6mzKH8MeKyXdmnhco1Bqq+FuPisBvAGN6m+DAZVwvsYpPoyGFQJp5Kk+jIYVAmn\nkqT6MhhUCS9XlerLYFAlvFxVqi+DQZVwjUGqL4NBlXCNQaovg0GV8HJVqb4MBlWi3RpgIAwGqY4M\nBlWm0x5wKkmqIYNBlem0BhwxSDVkMKgynXbL+xikGjIYVJnBtiMGqY4MBlXGNQapngwGVWZJKzg+\n7qs9pboxGFSZjlNJUi0ZDKpMp+VUklRHBoMq44hBqieDQZXptFsGg1RDBoMq02kNeB+DVEMGgyoz\n6OWqUi0ZDKqMawxSPRkMqozPSpLqyWBQZbzzWaong0GV6bQHOOGIQaodg0GVccQg1ZPBoMp0WgOc\nmEhOnsx+N0XSGTAYVJlOu3jvs6MGqVYMBlVm0GCQaslgUGXeHjG4AC3VisGgynRaBoNURwaDKuOI\nQaong0GVcfFZqieDQZVxKkmqJ4NBlTk1YvDR21K99BQMEbEpIvZHxFhEbJvh+8GIeKj4fndErJn0\n3W1F+f6IuGauOqNre0T8dUQ8ExG/OL8uql9cY5DqqT3XDhHRAu4GPgocAJ6KiNHM3DdptxuBVzNz\nXURsAe4EfjIiNgBbgCuAdwL/JyLeXRwzW50/A1wOvDczT0bE95TRUZ173scg1VMvI4YrgbHMfDYz\njwM7gc3T9tkM3F9sPwJcHRFRlO/MzGOZ+RwwVtR3ujp/AbgjM08CZOZLZ9899VOn1QIcMUh100sw\nrASen/T5QFE24z6ZOQ4cAS45zbGnq/P76I429kTEH0XE+t66ooXGqSSpnhbi4vMgcDQzNwL/A9gx\n004RcXMRHnsOHz58Thuo3nz3ctWJPrdE0pnoJRgO0p3zP2VVUTbjPhHRBkaAl09z7OnqPAA8Wmw/\nBvzATI3KzHszc2Nmbly2bFkP3dC55ohBqqdeguEpYH1ErI2IDt3F5NFp+4wCW4vt64AnMjOL8i3F\nVUtrgfXAk3PU+fvAjxbb/wz467PrmvrN+xikeprzqqTMHI+IW4HHgRawIzP3RsQdwJ7MHAXuAx6I\niDHgFbq/6Cn2exjYB4wDt2TmBMBMdRY/8lPAgxHxSeAN4KbyuqtzyfsYpHqaMxgAMnMXsGta2e2T\nto8C189y7HZgey91FuXfBn6sl3ZpYfNyVameFuLisxpiiVNJUi0ZDKpMayBoDYTBINWMwaBKdVoD\nBoNUMwaDKtVpD7jGINWMwaBKddqOGKS6MRhUKaeSpPoxGFSpwfYAx5xKkmrFYFClnEqS6sdgUKUM\nBql+DAZVyjUGqX4MBlXKy1Wl+jEYVKlOe4ATBoNUKwaDKuVUklQ/BoMq5eKzVD8GgyrVaQ/4Pgap\nZgwGVWrQxWepdnp6UY90tjqtAY68dYL/+If7+t2URvueCwf5+X/yvUREv5uiBjAYVKn3rbqIR79y\nkM8++bf9bkpjnTiZHB8/ycfet4JVF5/f7+aoAQwGVeq6H17FdT+8qt/NaLQ/+tohfuHBr/DGsfF+\nN0UN4RqDVHPDQ92/7944ajCoHAaDVHPDg91geN0Rg0piMEg1t9QRg0pmMEg1Nzy4BMA1BpXGYJBq\nzjUGlc1gkGru/CUtIlxjUHkMBqnmBgaC4U7bEYNKYzBIDTA81OaNYyf63Qw1hMEgNcDwYNvFZ5XG\nYJAa4ILBNq87laSSGAxSAywdavOmIwaVxGCQGsCpJJXJYJAaYHjQq5JUHoNBaoDhobb3Mag0BoPU\nAEuLqaTM7HdT1AAGg9QAw0NtMuE7xyf63RQ1gMEgNYAP0lOZDAapAU49SM97GVSGnoIhIjZFxP6I\nGIuIbTN8PxgRDxXf746INZO+u60o3x8R15xBnZ+OiDfOrlvS4rK0eFmPIwaVYc5giIgWcDdwLbAB\nuCEiNkzb7Ubg1cxcB9wF3FkcuwHYAlwBbALuiYjWXHVGxEbg4nn2TVo0fPS2ytTLiOFKYCwzn83M\n48BOYPO0fTYD9xfbjwBXR0QU5Tsz81hmPgeMFfXNWmcRGv8J+JX5dU1aPIbfHjH4ID3NXy/BsBJ4\nftLnA0XZjPtk5jhwBLjkNMeers5bgdHMPHS6RkXEzRGxJyL2HD58uIduSM319nufHTGoBAtq8Tki\n3glcD/y3ufbNzHszc2Nmbly2bFn1jZMWsLff++wag0rQSzAcBC6f9HlVUTbjPhHRBkaAl09z7Gzl\nHwDWAWMR8TfA+REx1mNfpEXrgkHXGFSeXoLhKWB9RKyNiA7dxeTRafuMAluL7euAJ7J7C+YosKW4\namktsB54crY6M/N/Z+byzFyTmWuA7xQL2pJOY0lrgMH2gCMGlaI91w6ZOR4RtwKPAy1gR2bujYg7\ngD2ZOQrcBzxQ/HX/Ct1f9BT7PQzsA8aBWzJzAmCmOsvvnrR4LB3yCasqx5zBAJCZu4Bd08pun7R9\nlO7awEzHbge291LnDPsM99I+ST56W+VZUIvPks7e8JCP3lY5DAapIYYHffS2ymEwSA0xPLjEEYNK\nYTBIDeHis8piMEgN4eKzymIwSA3h4rPKYjBIDTE82Ob4xEmOjfsWN82PwSA1xFIfva2SGAxSQwz7\nsh6VxGCQGsJHb6ssBoPUEMM+elslMRikhlg6uARwjUHzZzBIDXHBYAtwxKD5MxikhnAqSWUxGKSG\neHsqyWDQPBkMUkMMLRmgNRCuMWjeDAapISLC5yWpFAaD1CDDg23vY9C8GQxSg3QfvX2i381QzRkM\nUoM4laQyGAxSg/jobZXBYJAaxPc+qwwGg9QgSx0xqAQGg9QgrjGoDAaD1CDDg0v4zvEJJk5mv5ui\nGjMYpAbxeUkqQ7vfDZBUnuHiCasHX32Lkw0fNUTAyHlLiIh+N6VxDAapQUbO6wDwsU//aZ9bcm58\n4iPr+cRH3t3vZjSOwSA1yIffs4xP/Yv3cfTERL+bUrn//sVn2ft3r/W7GY1kMEgNMrSkxZYrV/e7\nGefEF/Yf5oUjR/vdjEZy8VlSLa0YGeKQwVAJg0FSLS0fGeLlN49xfPxkv5vSOAaDpFpaMTJEJrz0\nuqOGshkMkmpp+ch5AK4zVMBgkFRLK0aGAFxnqIDBIKmWLruwGwyOGMrXUzBExKaI2B8RYxGxbYbv\nByPioeL73RGxZtJ3txXl+yPimrnqjIgHi/KvR8SOiFgyvy5KaqILh9qc32k5YqjAnMEQES3gbuBa\nYANwQ0RsmLbbjcCrmbkOuAu4szh2A7AFuALYBNwTEa056nwQeC/wPuA84KZ59VBSI0UEy0eGeOG1\nt/rdlMbpZcRwJTCWmc9m5nFgJ7B52j6bgfuL7UeAq6P7AJPNwM7MPJaZzwFjRX2z1pmZu7IAPAms\nml8XJTWV9zJUo5dgWAk8P+nzgaJsxn0ycxw4AlxymmPnrLOYQvpXwOdmalRE3BwReyJiz+HDh3vo\nhqSmWX7hea4xVGAhLz7fA3wxM2d8Glhm3puZGzNz47Jly85x0yQtBCtGhnjp9WO+f6JkvQTDQeDy\nSZ9XFWUz7hMRbWAEePk0x562zoj4NWAZ8Mu9dELS4rR8ZIiJk8m33jjW76Y0Si/B8BSwPiLWRkSH\n7mLy6LR9RoGtxfZ1wBPFGsEosKW4amktsJ7uusGsdUbETcA1wA2Z6b3ukma1/ELvZajCnE9Xzczx\niLgVeBxoATsyc29E3AHsycxR4D7ggYgYA16h+4ueYr+HgX3AOHBLZk4AzFRn8SN/E/gm8KXiBRyP\nZuYdpfVYUmMsHzl1L8NbcPlFfW5Nc/T02O3M3AXsmlZ2+6Tto8D1sxy7HdjeS51FuY8Cl9QT736u\nxkJefJak03rHBR06rQGvTCqZwSCptk7d5OaIoVwGg6RaWz4y5IihZAaDpFpbMTLEIR+LUSqDQVKt\nLb9wiBePHKN7hbzKYDBIqrXlI0McnzjJK28e73dTGsNgkFRrXrJaPoNBUq35is/yGQySau3tEcNr\nBkNZDAZJtXbp8CCtgeg+FkOl8PETkmqtNRBctnSQB770Tf5474v9bs45d9/WH2H1JeeXWqfBIKn2\nbrlqHf9v7Fv9bkZfdNrlT/wYDJJq7+MffBcf/+C7+t2MxnCNQZI0hcEgSZrCYJAkTWEwSJKmMBgk\nSVMYDJKkKQwGSdIUBoMkaYpowsstIuIw8M2zPPxSYDHeMrkY+70Y+wyLs9/2uTfvysxl0wsbEQzz\nERF7MnNjv9txri3Gfi/GPsPi7Ld9nh+nkiRJUxgMkqQpDAa4t98N6JPF2O/F2GdYnP22z/Ow6NcY\nJElTOWKQJE1hMEiSpljUwRARmyJif0SMRcS2frenChFxeUR8ISL2RcTeiPilovwdEfEnEfGN4v8X\n97utZYuIVkR8NSL+sPi8NiJ2F+f7oYjo9LuNZYuIiyLikYj4q4h4JiL+QdPPdUR8svi3/fWI+GxE\nDDXxXEfEjoh4KSK+PqlsxnMbXZ8u+v+XEfFDZ/KzFm0wREQLuBu4FtgA3BARG/rbqkqMA/86MzcA\nHwJuKfq5Dfh8Zq4HPl98bppfAp6Z9PlO4K7MXAe8CtzYl1ZV678Cn8vM9wI/SLf/jT3XEbES+EVg\nY2Z+P9ACttDMc/07wKZpZbOd22uB9cV/NwOfOZMftGiDAbgSGMvMZzPzOLAT2NznNpUuMw9l5leK\n7dfp/qJYSbev9xe73Q/88/60sBoRsQr4MeC3is8BXAU8UuzSxD6PAP8UuA8gM49n5rdp+Lmm+4ri\n8yKiDZwPHKKB5zozvwi8Mq14tnO7Gfjd7PoycFFErOj1Zy3mYFgJPD/p84GirLEiYg3wAWA3cFlm\nHiq+egG4rE/Nqsp/AX4FOFl8vgT4dmaOF5+beL7XAoeB3y6m0H4rIi6gwec6Mw8C/xn4W7qBcAR4\nmuaf61NmO7fz+v22mINhUYmIYeD3gE9k5muTv8vuNcuNuW45In4ceCkzn+53W86xNvBDwGcy8wPA\nm0ybNmrgub6Y7l/Ha4F3Ahfw96dbFoUyz+1iDoaDwOWTPq8qyhonIpbQDYUHM/PRovjFU0PL4v8v\n9at9FfhHwE9ExN/QnSK8iu7c+0XFdAM083wfAA5k5u7i8yN0g6LJ5/ojwHOZeTgzTwCP0j3/TT/X\np8x2buf1+20xB8NTwPri6oUO3QWr0T63qXTF3Pp9wDOZ+euTvhoFthbbW4E/ONdtq0pm3paZqzJz\nDd3z+kRmfhz4AnBdsVuj+gyQmS8Az0fEe4qiq4F9NPhc051C+lBEnF/8Wz/V50af60lmO7ejwE8X\nVyd9CDgyacppTov6zueI+BjduegWsCMzt/e5SaWLiH8M/CnwNb473/6rdNcZHgZW031k+b/MzOkL\nW7UXER8G/k1m/nhEfC/dEcQ7gK8CP5WZx/rZvrJFxPvpLrh3gGeBn6X7B2Bjz3VE/AfgJ+legfdV\n4Ca68+mNOtcR8Vngw3Qfr/0i8GvA7zPDuS1C8jfoTqt9B/jZzNzT889azMEgSfr7FvNUkiRpBgaD\nJGkKg0GSNIXBIEmawmCQJE1hMEiSpjAYJElT/H8W8OXMihL1HAAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wnHzAia0I3-q",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 281
        },
        "outputId": "d773286d-51da-44d2-f1f0-820c2ec1444e"
      },
      "source": [
        "'''\n",
        " 指数衰减学习率\n",
        " gamma是衰减系数，越大衰减的越慢\n",
        "'''\n",
        "scheduler = lr_scheduler.ExponentialLR(optimizer, gamma=0.9)\n",
        "\n",
        "plt.figure()\n",
        "x = list(range(100))\n",
        "y = []\n",
        "for epoch in range(100):\n",
        "    optimizer.step()\n",
        "    scheduler.step()\n",
        "    lr = scheduler.get_lr()\n",
        "    y.append(scheduler.get_lr()[0])\n",
        "plt.plot(x, y)"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f6b25d4c320>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAfgklEQVR4nO3de3hV9b3n8fd3752E3EgghABJMCBB\nBBTQgNpq26N2hNan6NRWtNOxrdaejo6255ynR3vmmTPjOZ1Tezq1N7Xj1Gtbi9RWpRf1GavWSxUJ\nRSvIxXBR7gQICSSEXPZ3/tgLTWIuGxKydvb+vB55stZv/dZvf9ezfPLJWr+19zZ3R0RE5JhI2AWI\niEhqUTCIiEg3CgYREelGwSAiIt0oGEREpJtY2AUMhXHjxnlVVVXYZYiIjCirVq3a5+6lPdvTIhiq\nqqqora0NuwwRkRHFzN7prV23kkREpBsFg4iIdKNgEBGRbhQMIiLSjYJBRES6UTCIiEg3CgYREekm\no4Ph8dU7+PmrvT7GKyKSsTI6GJ5as5v7X94SdhkiIiklo4Nhamk+7+xvob0zHnYpIiIpI6OD4dTS\nAjrizrsHWsIuRUQkZWR2MIwvAGBzfXPIlYiIpI6MDoappfkAbKo/HHIlIiKpI6ODYfSoLMYX5rBp\nr4JBROSYjA4GSMwz6IpBROR9GR8MU0vz2VTfjLuHXYqISErI+GA4tbSAxiPt7G9uC7sUEZGUoGAI\nnkzSPIOISIKCIXgyafM+PbIqIgIKBiYV5TIqK6IrBhGRQMYHQyRiTBmnJ5NERI7J+GCAxO2kTXr3\ns4gIoGAAEk8mbWtoobW9M+xSRERCp2Ag8WSSO2zdr6sGEREFA12eTNLtJBERBQPA1HF6L4OIyDEK\nBiA3O0p5ca6eTBIRIclgMLOFZrbBzOrM7JZetueY2SPB9hVmVtVl261B+wYzu2SgMc3sIjP7i5m9\nbmYvmdm0wR1icqaW5lOnYBARGTgYzCwK3AksAmYCV5nZzB7drgUa3H0acAdwe7DvTGAJMAtYCNxl\nZtEBxrwb+Jy7zwUeBv7b4A4xOdXjC9m0t5l4XB+mJyKZLZkrhgVAnbtvdvc2YCmwuEefxcCDwfKj\nwEVmZkH7Unc/6u5bgLpgvP7GdGB0sFwE7DyxQzs+p00o4Eh7J9sa9DWfIpLZYkn0KQe2dVnfDpzT\nVx937zCzRqAkaH+1x77lwXJfY14H/MHMjgBNwLm9FWVm1wPXA0yePDmJw+jf9LJCADbsPsQpJfmD\nHk9EZKRKxcnnrwOfcPcK4H7ge711cvd73L3G3WtKS0sH/aLVQTBs3HNo0GOJiIxkyQTDDqCyy3pF\n0NZrHzOLkbgFtL+ffXttN7NSYI67rwjaHwE+lNSRDFJBToyKMbls2KMJaBHJbMkEw0qg2symmFk2\nicnk5T36LAeuCZavAJ71xFeiLQeWBE8tTQGqgdf6GbMBKDKz6cFYHwfWnfjhHZ8ZEwrZsLtpuF5O\nRCQlDTjHEMwZ3Ag8DUSB+9x9rZndBtS6+3LgXuBnZlYHHCDxi56g3zLgLaADuMHdOwF6GzNo/zLw\nazOLkwiKLw3pEfdjelkhz2+op60jTnYsFe+yiYicfJYO33VcU1PjtbW1gx7nidd3cPPS13n6ax/h\ntAmFQ1CZiEjqMrNV7l7Ts11/Fnfx3pNJmoAWkQymYOhiamk+0YixcbeCQUQyl4Khi5xYlKnj8lmv\nYBCRDKZg6GH6hEK9l0FEMpqCoYfTygp590ALLW0dYZciIhIKBUMPxyag39Yb3UQkQykYejj2mOoG\nzTOISIZSMPQweWweo7IiemRVRDKWgqGHaMSoHq8JaBHJXAqGXkwvK9QjqyKSsRQMvTh9YiH1h45S\nf+ho2KWIiAw7BUMvZk0qAmDtzsaQKxERGX4Khl7MnJT4ZtG1O/UR3CKSeRQMvSjKzaJybC5vKRhE\nJAMpGPowa2KRbiWJSEZSMPRhdvlotu5voam1PexSRESGlYKhD8cmoNfpdpKIZBgFQx9maQJaRDKU\ngqEP40ePYlxBjoJBRDKOgqEfsyaN1gS0iGQcBUM/Zk0aTd3ewxzt6Ay7FBGRYaNg6Mfs8iI64s7G\n3fpuBhHJHAqGfrw/Aa3bSSKSORQM/agck0dhTow1CgYRySAKhn5EIsbpk0brySQRySgKhgHMmjSa\ndbua6OiMh12KiMiwUDAM4MyKIlrb47y9VxPQIpIZFAwDmFs5BoA3th0MuRIRkeGhYBhAVUkeRblZ\nvK5gEJEMoWAYgJkxp7JYwSAiGUPBkIS5lcVs3HOI5qMdYZciInLSKRiSMK+ymLjDmzv0fgYRSX8K\nhiScWZH4bgbdThKRTKBgSEJJQQ6Tx+bpySQRyQgKhiRpAlpEMkVSwWBmC81sg5nVmdktvWzPMbNH\ngu0rzKyqy7Zbg/YNZnbJQGNawrfMbKOZrTOzmwZ3iENjbmUxuxpb2dPUGnYpIiIn1YDBYGZR4E5g\nETATuMrMZvbodi3Q4O7TgDuA24N9ZwJLgFnAQuAuM4sOMOYXgEpghrufDiwd1BEOkbmVxYDmGUQk\n/SVzxbAAqHP3ze7eRuIX9eIefRYDDwbLjwIXmZkF7Uvd/ai7bwHqgvH6G/OrwG3uHgdw970nfnhD\nZ9ak0cQipmAQkbSXTDCUA9u6rG8P2nrt4+4dQCNQ0s++/Y15KnClmdWa2ZNmVt1bUWZ2fdCntr6+\nPonDGJxRWVFOnzia199VMIhIekvFyeccoNXda4D/C9zXWyd3v8fda9y9prS0dFgKm1tZzJs7GumM\n+7C8nohIGJIJhh0k7vkfUxG09drHzGJAEbC/n337G3M78Jtg+THgzCRqHBZzK4s5fLSDjXsOhV2K\niMhJk0wwrASqzWyKmWWTmExe3qPPcuCaYPkK4Fl396B9SfDU0hSgGnhtgDEfB/4mWP4osPHEDm3o\nza8aC0Dt1gMhVyIicvIMGAzBnMGNwNPAOmCZu681s9vM7FNBt3uBEjOrA/4OuCXYdy2wDHgLeAq4\nwd07+xozGOvbwKfN7E3g34DrhuZQB69ybC5lo3N4bWtD2KWIiJw0lvjDfmSrqanx2traYXmtGx/+\nC7VbG3jl1gtJPHglIjIymdmqYD63m1ScfE5p86vGsruple0NR8IuRUTkpFAwHKdj8wwrNc8gImlK\nwXCcTptQSOGomIJBRNKWguE4RSPG2aeMYaUmoEUkTSkYTsD8qrHU7T3Mgea2sEsRERlyCoYTsGCK\n5hlEJH0pGE7AmRVFZMcieqObiKQlBcMJyIlFmVNRpDe6iUhaUjCcoPlVY1m7o5GWto6wSxERGVIK\nhhO0YMpYOuJOra4aRCTNKBhO0PyqsWRFjZc37Qu7FBGRIaVgOEH5OTHmVY7hz3X7wy5FRGRIKRgG\n4cPTxrFmZyMHW/R+BhFJHwqGQfjwtBLc4ZVNumoQkfShYBiEOZXF5GdHNc8gImlFwTAIWdEI50wt\n4WXNM4hIGlEwDNKHTi1hy75mdh7U9zOISHpQMAzS+dXjAHi5TreTRCQ9KBgG6bSyQsYVZCsYRCRt\nKBgGycw479RxvLxpP+nw/dkiIgqGIXD+tBLqDx3l7b2Hwy5FRGTQFAxD4ILqUgCe37A35EpERAZP\nwTAEJhXnMmNCIc+uVzCIyMinYBgifzNjPLVbG2hqbQ+7FBGRQVEwDJELZ4ynI+68uFFPJ4nIyKZg\nGCLzKospys3S7SQRGfEUDEMkFo3w0eml/GnjXuJxPbYqIiOXgmEIXThjPPsOt/HXHY1hlyIicsIU\nDEPoo9NLiRi6nSQiI5qCYQiNyc9m3uQxPKdgEJERTMEwxC6cMZ43dzSyt6k17FJERE6IgmGIXThj\nPAB/1FWDiIxQCoYhNmNCIaeU5PHkmt1hlyIickIUDEPMzFg4ewJ/rttHY4veBS0iI4+C4SRYNHsi\nHXHnmXV7wi5FROS4JRUMZrbQzDaYWZ2Z3dLL9hwzeyTYvsLMqrpsuzVo32BmlxzHmD80sxH5OdZz\nKoqYVDSKJ9fsCrsUEZHjNmAwmFkUuBNYBMwErjKzmT26XQs0uPs04A7g9mDfmcASYBawELjLzKID\njWlmNcCYQR5baBK3kybywtv7OHy0I+xyRESOSzJXDAuAOnff7O5twFJgcY8+i4EHg+VHgYvMzIL2\npe5+1N23AHXBeH2OGYTGvwPfGNyhhWvRGRNo64jrzW4iMuIkEwzlwLYu69uDtl77uHsH0AiU9LNv\nf2PeCCx3937vw5jZ9WZWa2a19fX1SRzG8Dp78hhKC3N4SreTRGSESanJZzObBHwG+NFAfd39Hnev\ncfea0tLSk1/ccYpEjEtmlfHc+nqOtHWGXY6ISNKSCYYdQGWX9Yqgrdc+ZhYDioD9/ezbV/s8YBpQ\nZ2ZbgTwzq0vyWFLOJ2ZP5Eh7p77yU0RGlGSCYSVQbWZTzCybxGTy8h59lgPXBMtXAM+6uwftS4Kn\nlqYA1cBrfY3p7r939wnuXuXuVUBLMKE9Ii2YMpZxBTk88frOsEsREUlabKAO7t5hZjcCTwNR4D53\nX2tmtwG17r4cuBf4WfDX/QESv+gJ+i0D3gI6gBvcvROgtzGH/vDCFYtG+NScSfz81XdobGmnKC8r\n7JJERAZkiT/sR7aamhqvra0Nu4xerdnRyKU/eon/dfkZXH3O5LDLERF5j5mtcveanu0pNfmcjmZN\nGs208QU8tnp72KWIiCRFwXCSmRmXzytn5dYGth1oCbscEZEBKRiGweK5kwB4fHXPh7lERFKPgmEY\nVIzJ45wpY3ns9R2kw5yOiKQ3BcMwuXxeOZvrm/nr9sawSxER6ZeCYZgsOmMiObEIv1q1beDOIiIh\nUjAMk6LcLD55xkQeX72TZn3iqoikMAXDMLr6nMkcPtrB7/6qd0KLSOpSMAyjs08Zw/SyAh5e8W7Y\npYiI9EnBMIzMjKsXTOaN7Y2s2aFJaBFJTQqGYXb5vApyYhEefk1XDSKSmhQMw6woL4tLz5zEE6t3\naBJaRFKSgiEEV58zmea2Tn0ct4ikJAVDCM6aXMzMiaN54M9b9E5oEUk5CoYQmBnXXTCFjXsO88Lb\n+8IuR0SkGwVDSC49cxLjC3P46Yubwy5FRKQbBUNIsmMRrvlQFS++vY8Nuw+FXY6IyHsUDCH63DmT\nyc2Kcu9LumoQkdShYAhRcV42V5xdweOrd1J/6GjY5YiIAAqG0H3p/Cm0x+M89MrWsEsREQEUDKGb\nMi6fhbMm8MCft9J4pD3sckREFAyp4L9eWM2h1g7uf3lL2KWIiCgYUsHMSaP5DzPLuO+lLTS16qpB\nRMKlYEgRN11UTVNrBw+8vDXsUkQkwykYUsTs8iIuPn089760hUO6ahCRECkYUsjNF02n8Ui7rhpE\nJFQKhhRyRkXiquGeFzZzoLkt7HJEJEMpGFLMPy6cQXNbBz/849thlyIiGUrBkGKqywq5cv5kfv7q\nO2zZ1xx2OSKSgRQMKejrH68mOxbhO0+tD7sUEclACoYUNL5wFNd/ZCpPrtnNqncOhF2OiGQYBUOK\n+vIFUyktzOFffreOeFzf8iYiw0fBkKLyc2LcumgGr287yCO128IuR0QyiIIhhV0+r5xzpozl20+u\nZ/9hfSy3iAwPBUMKMzP+9bLZNB/t4NtPaiJaRIZHUsFgZgvNbIOZ1ZnZLb1szzGzR4LtK8ysqsu2\nW4P2DWZ2yUBjmtkvgvY1ZnafmWUN7hBHtuqyQq67YCq/WrWdlVs1ES0iJ9+AwWBmUeBOYBEwE7jK\nzGb26HYt0ODu04A7gNuDfWcCS4BZwELgLjOLDjDmL4AZwBlALnDdoI4wDdx00TTKi3P55m/epLW9\nM+xyRCTNJXPFsACoc/fN7t4GLAUW9+izGHgwWH4UuMjMLGhf6u5H3X0LUBeM1+eY7v4HDwCvARWD\nO8SRLy87xrcun83bew9zxzMbwy5HRNJcMsFQDnR9LGZ70NZrH3fvABqBkn72HXDM4BbS54GneivK\nzK43s1ozq62vr0/iMEa2j502nqsWVHLPC5v13gYROalSefL5LuAFd3+xt43ufo+717h7TWlp6TCX\nFo5/+uRMyotz+ftlb9DS1hF2OSKSppIJhh1AZZf1iqCt1z5mFgOKgP397NvvmGb2z0Ap8HfJHESm\nKMiJ8e9XzGHr/hY9pSQiJ00ywbASqDazKWaWTWIyeXmPPsuBa4LlK4BngzmC5cCS4KmlKUA1iXmD\nPsc0s+uAS4Cr3D0+uMNLP+edWsK150/hoVfe4em1u8MuR0TS0IDBEMwZ3Ag8DawDlrn7WjO7zcw+\nFXS7FygxszoSf+XfEuy7FlgGvEViruAGd+/sa8xgrJ8AZcArZva6mf33ITrWtPGNhadxZkUR//Cr\nN3h3f0vY5YhImrHEH/YjW01NjdfW1oZdxrDadqCFT/zwRapK8nn0q+eRE4uGXZKIjDBmtsrda3q2\np/Lks/Sjcmwe3/3MHN7c0ci3fr8u7HJEJI0oGEawS2ZN4LpgvuGRle+GXY6IpAkFwwh3y6IZXFA9\njn96bA2vbNofdjkikgYUDCNcLBrhx1efxSkleXz1F6vYqq8DFZFBUjCkgaLcLO77wnwM+NIDKznQ\n3BZ2SSIygikY0sQpJfn8n8/XsP3gEb5w/2scam0PuyQRGaEUDGlkwZSx3HX1Wazd2cSXH6rVJ7GK\nyAlRMKSZi2eW8b8/M4cVWw5w48Orae/Um8dF5PgoGNLQZfPKue1Ts3hm3R6++vNVunIQkeOiYEhT\nnz+vin+5bDbPrNvLlx+q5UibwkFEkqNgSGOfP/cUvvPpM3mpbp8mpEUkaQqGNPfZ+ZV8/8q51L7T\nwGd+8gq7Go+EXZKIpDgFQwZYPLec+78wn+0NR7jszpdZu7Mx7JJEJIUpGDLER6aX8qu/PY+IGZ/9\nySv6LgcR6ZOCIYOcPnE0j/2XD3Pq+AK+8rNVfOep9XTGR/7HrovI0FIwZJgJRaNY9pXzuGrBZO56\nfhPX3Pca+w4fDbssEUkhCoYMNCoryr/9xzP4zqfP5LWtB1j4/Rd4dv2esMsSkRShYMhgn51fyW9v\nPJ9xBTl86YFa/umxN2lp6wi7LBEJmYIhw502oZAnbvww139kKg+/9i4f/94LPLdhb9hliUiIFAxC\nTizKNz9xOsu+ch6jsiJ88f6V3PTL1ew91Bp2aSISAgWDvGd+1Vj+cPMFfO3iap5as5sLv/sn7n5+\nkz5rSSTDKBikm5xYlK9dPJ2nvnYB504t4fan1vPxO/7EE6/vIK5HW0UygoJBejW1tICfXlPDL647\nh/zsGDcvfZ1FP3iRp9bswl0BIZLOFAzSrw9PG8cfbrqAH101j/Z4nL/9+V9Y9IMXeWz1dn3Xg0ia\nsnT466+mpsZra2vDLiPtdXTGWf7GTu5+fhNv7z1MeXEuX/hQFZ+pqaA4Lzvs8kTkOJnZKnev+UC7\ngkGOVzzuPL9xLz/502Ze23KAnFiExXMncfU5pzCnoggzC7tEEUlCX8EQC6MYGdkiEePCGWVcOKOM\ndbua+Nmr7/D46h0sq91O9fgCrji7gsvmlVM2elTYpYrICdAVgwyJptZ2fv/XXTy6ajur3mnALPH4\n66VnTmTh7AmML1RIiKQa3UqSYbO5/jC/fWMXv39zJxv3HMYM5lQUc/Hp47no9DJmTCjU7SaRFKBg\nkFBs3HOIp9fs5pn1e3lj20EASgtzuGDaOM6vHse5U0uYVJwbcpUimUnBIKHb29TK8xvqebFuHy+9\nXU9DS+I7qCvH5rKgqoSzTinmrMljmF5WSDSiKwqRk03BICklHnfW7W5ixeYDrNiyn5VbGzjQ3AZA\nfnaUWZOKmF1exBkVozl94mimjisgO6a33YgMJQWDpDR3590DLfzl3QZWv3uQN3c0sm5XE63tiTfR\nZUWNU0sLqC4rZFppAdPGFzC1NJ+qknxys6MhVy8yMulxVUlpZsYpJfmcUpLP5fMqgMQb6jbVN7N+\ndxPrdh1i/e4mVr/bwG/f2Nlt3wmjRzG5JI/KMXlMHptHxZhcJhXnUl6cS1lRDjkxBYfI8VAwSMqK\nRSOcNqGQ0yYUsnju++0tbR1srm9my75mtu5rZsv+ZrYdaOHlun38uumDHxU+Nj+bstGjKBudw/jC\nHEoLcygtyKGkIIeSgmxK8nMYk5dFcV62bleJkGQwmNlC4AdAFPipu3+7x/Yc4CHgbGA/cKW7bw22\n3QpcC3QCN7n70/2NaWZTgKVACbAK+Ly7tw3uMCWd5GXHmF2emIPoqbW9k92Nrew4eIQdDUfY3dTK\n7qZW9jS2svfQUdbtamLf4TY6+/ik2IKcGEW5WRTnZVGUm/g3elQWhaNiFI7KomBUjMKcGPk5MfJz\nohTkxMjLjpGXHSUvO0pudpTcrCixqAJGRq4Bg8HMosCdwMeB7cBKM1vu7m916XYt0ODu08xsCXA7\ncKWZzQSWALOAScAzZjY92KevMW8H7nD3pWb2k2Dsu4fiYCX9jcqKUjUun6px+X32icedhpY2DjS3\nse9wG/ubj9LQ0s7B5jYOtLTReKSdxpZ2Dh5pZ1P9YZqOdNDU2k5LW/LfS5EdjTAqK8KorGjwL7Gc\nE4uQEwt+ZkXIjkbIjiX+ZR1bjiaWY1EjOxohFjFi0QhZUSMWSbTHIhGiEQu2JdYjEYJ2iEYiRM2I\nRCAasWA5+Bm0R8yIRoJ1I/hpWIRubXbsJ++v630o6S2ZK4YFQJ27bwYws6XAYqBrMCwG/kew/Cjw\nY0v8n7MYWOruR4EtZlYXjEdvY5rZOuBC4Oqgz4PBuAoGGTKRiAW3kXKoLkt+v47OOM1tnRxqbaf5\naCeHj3bQfLSDlrZOWto6aG7rpLWtM7He3sHR9jit7Z0cae9MLHd00tqe6NvQEudoR5y2Y/8647QH\nP9s644yEZ0LM6B4WJBrsvW2JcLEgVI5tS+xr7+1/LGSO7Rf0eG+5a7t1aT+2DT4YVD1zq/s+1kd7\n1/69B1+fcdjHhmTjczBBe98185lcknfC+/cmmWAoB7Z1Wd8OnNNXH3fvMLNGEreCyoFXe+xbHiz3\nNmYJcNDdO3rp342ZXQ9cDzB58uQkDkNkcGLRCEW5EYpys076a3V0xumIO+2dcdo7nY7OOO1xf6+9\nM9jWGXc64k5HZ6It7on1eNB+rO3Yz8Ry4qop7k6nJ/o6ibZOTzwh5g6dwc94lz7H1t17rAfLjhP8\n99448aC9a9h13R+6b+/aTtf2Y+Pz/voH+3fv816HDy52+16R7u29n5O+srqvJzuTzvZB/hFwMubF\nRuzks7vfA9wDicdVQy5HZEjFohFi0cStMZHhlkzU7AAqu6xXBG299jGzGFBEYhK6r337at8PFAdj\n9PVaIiJyEiUTDCuBajObYmbZJCaTl/fosxy4Jli+AnjWE9dXy4ElZpYTPG1UDbzW15jBPs8FYxCM\n+cSJH56IiByvAW8lBXMGNwJPk3i09D53X2tmtwG17r4cuBf4WTC5fIDEL3qCfstITFR3ADe4eydA\nb2MGL/mPwFIz+1dgdTC2iIgME30khohIhurrIzH0LhwREelGwSAiIt0oGEREpBsFg4iIdJMWk89m\nVg+8c4K7jwP2DWE5I0UmHncmHjNk5nHrmJNziruX9mxMi2AYDDOr7W1WPt1l4nFn4jFDZh63jnlw\ndCtJRES6UTCIiEg3Cobgg/gyUCYedyYeM2TmceuYByHj5xhERKQ7XTGIiEg3CgYREekmo4PBzBaa\n2QYzqzOzW8Ku52Qws0oze87M3jKztWZ2c9A+1sz+n5m9HfwcE3atQ83Moma22sx+F6xPMbMVwfl+\nJPjI97RiZsVm9qiZrTezdWZ2XrqfazP7evD/9hoz+6WZjUrHc21m95nZXjNb06Wt13NrCT8Mjv+v\nZnbW8bxWxgaDmUWBO4FFwEzgKjObGW5VJ0UH8PfuPhM4F7ghOM5bgD+6ezXwx2A93dwMrOuyfjtw\nh7tPAxqAa0Op6uT6AfCUu88A5pA4/rQ912ZWDtwE1Lj7bBIf47+E9DzXDwALe7T1dW4Xkfj+m2oS\nX4F89/G8UMYGA7AAqHP3ze7eBiwFFodc05Bz913u/pdg+RCJXxTlJI71waDbg8Bl4VR4cphZBfBJ\n4KfBugEXAo8GXdLxmIuAjxB8h4m7t7n7QdL8XJP4Xpnc4Jsf84BdpOG5dvcXSHzfTVd9ndvFwEOe\n8CqJb8acmOxrZXIwlAPbuqxvD9rSlplVAfOAFUCZu+8KNu0GykIq62T5PvANIB6slwAH3b0jWE/H\n8z0FqAfuD26h/dTM8knjc+3uO4DvAu+SCIRGYBXpf66P6evcDur3WyYHQ0YxswLg18DX3L2p67bg\nK1XT5rllM7sU2Ovuq8KuZZjFgLOAu919HtBMj9tGaXiux5D463gKMAnI54O3WzLCUJ7bTA6GHUBl\nl/WKoC3tmFkWiVD4hbv/Jmjec+zSMvi5N6z6ToIPA58ys60kbhFeSOLee3FwuwHS83xvB7a7+4pg\n/VESQZHO5/piYIu717t7O/AbEuc/3c/1MX2d20H9fsvkYFgJVAdPL2STmLBaHnJNQy64t34vsM7d\nv9dl03LgmmD5GuCJ4a7tZHH3W929wt2rSJzXZ939c8BzwBVBt7Q6ZgB33w1sM7PTgqaLSHzfetqe\naxK3kM41s7zg//Vjx5zW57qLvs7tcuA/B08nnQs0drnlNKCMfuezmX2CxL3oKHCfu38r5JKGnJmd\nD7wIvMn799u/SWKeYRkwmcRHln/W3XtObI14ZvYx4B/c/VIzm0riCmIssBr4T+5+NMz6hpqZzSUx\n4Z4NbAa+SOIPwLQ912b2P4ErSTyBtxq4jsT99LQ612b2S+BjJD5eew/wz8Dj9HJug5D8MYnbai3A\nF929NunXyuRgEBGRD8rkW0kiItILBYOIiHSjYBARkW4UDCIi0o2CQUREulEwiIhINwoGERHp5v8D\nQFQVxYptRIEAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-KcNcj_3I34o",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 281
        },
        "outputId": "cf65b358-2cce-48fd-afbd-e0be132ce44f"
      },
      "source": [
        "'''\n",
        " 余弦退火\n",
        " T_max： 每次cosine的epoch数，既cos函数周期\n",
        " eta_min: 每个周期衰减的最小学习率，一般设置为1e-5之类的小学习率\n",
        " 注意，这只实现了SGDR的余弦退火部分，而没有重新启动。\n",
        "'''\n",
        "scheduler = lr_scheduler.CosineAnnealingLR(optimizer, T_max=20, eta_min = 1e-5)\n",
        "\n",
        "plt.figure()\n",
        "x = list(range(100))\n",
        "y = []\n",
        "for epoch in range(100):\n",
        "    optimizer.step()\n",
        "    scheduler.step()\n",
        "    lr = scheduler.get_lr()\n",
        "    y.append(scheduler.get_lr()[0])\n",
        "plt.plot(x, y)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f6b25cad2e8>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de3xcV3no/d8zI43uGt1lWRdLluSL\nbMmyLYfEDiEXmySUEggxOAUOtKH00FDa0ltyPv1wevg058A53EoL5ZMXKOl5eQlJSMGlISFOKJc4\nF0u+W4pt2ZYtybbuGt0vI633j9njyIpkjaTRXPY8338sjfbsWduPtJ+917PXWmKMQSmllPJzhLsB\nSimlIosmBqWUUtfRxKCUUuo6mhiUUkpdRxODUkqp68SFuwHBkJOTY0pLS8PdDKWUiioNDQ3dxpjc\n2a/bIjGUlpZSX18f7mYopVRUEZGLc72uXUlKKaWuo4lBKaXUdTQxKKWUuo4mBqWUUtfRxKCUUuo6\nASUGEblHRE6LSLOIPDLHzxNE5EfWz18XkdIZP3vUev20iNw94/XviUiniJycta8sEXlRRM5a/2Yu\n/fCUUkot1oKJQUScwDeBe4Eq4EERqZq12UNAnzGmAvga8CXrvVXAPmATcA/wLWt/AN+3XpvtEeAl\nY0wl8JL1vVJKqRAJ5I7hJqDZGHPeGDMBPAncN2ub+4AnrK+fAe4SEbFef9IYM26MuQA0W/vDGPNr\noHeOz5u5ryeA9y/ieJQCYHjcy48b2vjZ8cuMe6fC3Ry1gobGvTxd38pzJ64w4Z0Od3NsIZABboVA\n64zv24B3zLeNMcYrIh4g23r9tVnvLVzg8/KNMVesr68C+XNtJCKfAj4FUFJSsvBRzOFC9zC9wxNs\nX6O9VXZx+uog3z94gf1HLzM84UsImcnx3L+tiE/sLKU4KznMLVTB0nRlgO+/0sK/H7/MiBXrrBQX\nH9xWyMd3llKUqbFeqoge+WyMMSIy50pCxpjHgccB6urqFr3akDGGv3z6GOe7hvi3P95FaU7KMlur\nwu1ku4e9334Vg+G9NavZt6OY0ckpfvjGJZ442MJPjrTzk4d3aXKwgSOX+tj3+Gs4RPjdLQV8eEcJ\ng2OTPPlGK//ySgs/OXqZnz68i9UZSeFualQKpCupHSie8X2R9dqc24hIHOAGegJ872wdIlJg7asA\n6AygjYsmInxl7xYM8AffP4RnZHIlPkaFyBXPKA89cYisFBe/+qs7+PLeLdSVZvHOyly+9ZHtPP9n\n72RyapqHnjjEwJjGOpq19Y3wh//aQF56Ar/+6zv43w9sYfuaTG5fn8e3P7adn332VsYmpnjoiXqG\nxr3hbm5UCiQxHAIqRaRMRFz4isn7Z22zH/i49fUDwMvGt2bofmCf9dRSGVAJvLHA583c18eBnwbQ\nxiUpzUnh8Y/V0do3wqd/0MDklPZPRqPhcS8Pfb+e4fEpvvuJOvLTE9+2TUVeGt/+6HbOdw3z8A8O\n49VYR6XBsUke+n49494p/uUTO8hNS3jbNhtWpfNPH9nGmY5BPvvDI0xN6/LFi7VgYjDGeIHPAC8A\nTcBTxphTIvIFEXmftdl3gWwRaQY+h/UkkTHmFPAU0Ag8DzxsjJkCEJEfAq8C60WkTUQesvb1RWCP\niJwFdlvfr5ibyrL40gdrOHiuh7/bf2olP0qtAGMMn3vqKG9eHeAff28rG1alz7vtzooc/v79m/nN\n2W7+/j+aQthKFQzGGP70yaM0dw3xzx/ZTkVe2rzbvmtdLn/3vk28/GYnX3r+zRC20h4CqjEYY54D\nnpv12udnfD0G7J3nvY8Bj83x+oPzbN8D3BVIu4Ll/m1FNF4e4Du/vcDvvaOETavdofx4tQy/be7m\nhVMd/M09G7hjfd6C2++7qYTGKwM88WoLH3lHCZX5859cVGT55elOXn6zk7/9nY3cWpmz4PYfu3kN\njZc9fPe3F9i3o5i1uakhaKU96Mhny5/cVUl6Yhxf/cWZcDdFBcgYw5dfOE1hRhJ/cGtpwO/7893r\nSHHF8dUXNdbRYnra8OUXzlCSlczHd5YG/L7P7VmPy+ng6wfOrlzjbEgTg8WdFM8fvaucl97s5PCl\nvnA3RwXgQFMnx9o8fPauChLinAu/wZKZ4uIPbi3j5yevcrLds4ItVMHy/KmrNF4Z4M92VxLvDPy0\nlZuWwCd2lfLvxy/z5tWBFWyhvWhimOETO0vJTnHxlV+cDndT1AKmpw1f+cVpSrOT+eC2okW//5Pv\nLMOdFK+xjgJT04avvniGirxU7qtdaBjU2/3RbWtJdWlvwGJoYpghJSGOT99ezivNPRw81x3u5qgb\n+I8TV3jz6iB/vmcdcYu4gvRLT4znj961ll+e7qLh4lwD8FWk+OnRdpo7h/jcnnU4HbLo92cku/jk\nO9fyi8YOjrf1r0AL7UcTwywfvXkNq9IT9eoigk1NG7524Azr89P43ZrVS97PJ3aWkpOawFc01hHL\nOzXN1w+cZdPqdO7ZtGrJ+/mDW0vJTI7XWAdIE8MsifFO/vC2tdRf7KPpivZJRqKD57o53zXMH99R\njmMJV5B+ya44Hrq1jIPnemjuHApiC1Ww/PpsF5d6R/jMHRXLinVaYjy/v6uMX53p4lLPSBBbaE+a\nGObwga2FxDuFp+vbwt0UNYen69tIT4zj7mVcQfp9cHshTofwTIPGOhI9Xd9GVoqLuzbOOWXaojyw\nvQgReKahdeGNY5wmhjlkpbjYvTGfnxxt19kaI4xndJIXTl3lvtpCEuMDfxJpPnlpidy+LpdnD7fp\naOgI0zs8wYGmDt5fW4grbvmnqtUZSdxakcOPD7czraOhb0gTwzz21hXROzzBy2+uyFRNaon+/dhl\nxr3TfKiueOGNA7S3rpjOwXF+fbYraPtUy/fTo+1MThk+tGPxT53N50N1xbT3j3LwXE/Q9mlHmhjm\ncVtlLnlpCXrbGWGebmhjw6o0NhfOP/XFYt25IY+sFJd2HUaYp+rbqC5033Cak8XaU5VPemIcT+vf\n9Q1pYphHnNPBB7YV8svTXXQOjoW7OQo42zHIsdZ+q6946YXI2VxxDt5fW8iBpg56hyeCtl+1dCfb\nPTRdGWBvXfDuFsD3cMl9tYU8f/IqnlGdZXc+mhhuYO/2YqamDT85stBM4SoUnm5oI84hfGDr4gc5\nLWRvXRGTU4afHtVYR4JnGtpwOR28b8vSH0eez966Isa90/zs+OWg79suNDHcQEVeKltLMniqvg3f\nLOIqXLxT0zx7uJ07N+SRnfr2qZaXa2NBOpsL07U7KQKMe6f4ydF29mzKJyPZFfT9Vxe6WZ+fxlMa\n63lpYljA/duKaO4c0ufcw+yNll66h8a5f1vw7xb87t9aROOVAS50D6/YZ6iFvXquh/6RSe5fgTtD\n8C3Sdf+2Qo619tPeP7oinxHtNDEsYPdG31TOLzZ1hLklse1AYyeuOAe3rctdsc/YU+V7Vv4ljXVY\nHWjqICneya6KhafWXiqN9Y1pYlhAgTuJ6kI3LzbqL1C4GGN4sekqt1bkkOxauWXKi7OS2bAqjV9o\nrMPGGMOBxk5uW5cTlHEq81mbm8ra3BT9u56HJoYA7KnK52hrvz6dFCZnOoZo7R29dpW3kvZU5VPf\n0kufPp0UFifbB7g6MMaequWPal/Inqp8Xjvfo2uAz0ETQwB2b8zHGPilDnYLiwPW7f5dGxZeoW25\ndm/MZ9r4VgtTofdiUwcOgTvWr1yXod+ejflMThl+fUYHNs6miSEAGwvSKMxI4sVGPVmEw4uNHWwp\nziAvPXHFP6u60E1eWsK1ZKRC60BjB9vXZK7Ik2ezbS3JJCvFxQHtTnobTQwBEBF2b8zjt81djE5M\nhbs5MaVzcIyjrf3s2bjydwsADodw18Z8fnW6i3GvxjqU2vtHabwywO4gTJgXCKdDuHNDHi+/2cmk\nzpN1HU0MAdpdlc/Y5DSvNOsCPqH0cpPvLm13COoLfnuq8hiemOK187qATyj5nxAKZax3b8xnYMxL\nfYsu5zuTJoYAvaMsm7SEOO1iCLEDTR0UZSaxPj8tZJ+5szyHpHindjGE2IuNHazNSaE8NzVkn/nO\nyhxccQ79u55FE0OAXHEO3rU+lwNNnTplb4iMTkzxm7Pd7KnKD+rcSAtJjHdy27ocDjR16Ij3EBkc\nm+S18z0hefJsppSEOG6tyOHFRo31TJoYFuHODXl0D43TqCu7hcRrF3oY905zZwieRprtzg15XPGM\ncaZDR7yHwsFzPUxOGe4IQ6zv2JDHpd4RWnRlt2s0MSyCfySm1hlC42BzN644BztKs0L+2Rrr0DrY\n3E1SvJNtJZkh/+xbNdZvo4lhEfLTE6nIS+UVXeQjJF5p7mF7SeaKjoCdT1FmMmuyk3VBlxA5eK6H\nHWVZQVmpbbFKs5NZ7U7k4DlNDH6aGBZpV3k2hy706pKfK6xveILGKwPsLM8OWxt2lmfz+vkeXfJz\nhXUOjHG2cyhssRYRbinP4dVzPVo/tGhiWKRbynMYnZziaGt/uJtia6+e912p71zBidQWsrM8h8Fx\nLyfaPWFrQyzw35XtKg9frHdVZNM3MknTVa0fgiaGRbtlbTYO0f7IlfZKczepCXFsKXKHrQ23WFew\n2p20sg6e68adFE/V6uAt4blYO62kdLBZYw2aGBbNnRzP5kK39keusIPneripLIs4Z/h+RXNSE9iw\nKk1jvYKMMbzS3MPNa7NwOkL3SPJsq9yJrM1N4RWNNaCJYUl2ludw5FI/IxPecDfFli73j3Khezis\n9QW/neU51Lf0MTap02OshEu9I7T3j67o2guB2lWewxsXenV6DAJMDCJyj4icFpFmEXlkjp8niMiP\nrJ+/LiKlM372qPX6aRG5e6F9ishdInJYRI6KyG9FpGJ5hxh8O8uz8U4b3rigUyashGt9zpFwsqjI\nZtw7zeFLOmXCSvDHemcY6wt+uyqyGZmY4pjWDxdODCLiBL4J3AtUAQ+KSNWszR4C+owxFcDXgC9Z\n760C9gGbgHuAb4mIc4F9/jPwEWNMLfD/AX+7vEMMvh2lWbicDu17XiEHm7vJSnGFdBqM+dxU5uvi\n0L7nlfFKczd5aQmU56aEuyncvDYbEd9j0rEukDuGm4BmY8x5Y8wE8CRw36xt7gOesL5+BrhLfHMY\n3Ac8aYwZN8ZcAJqt/d1onwbwV6HcwOWlHdrKSXI52VqSoQXoFWCM4ZVz3dxSno0jjH3OfmmJ8dQU\nubXveQVMTxtePdfDroqckE55Mp+MZBebVqdrrAksMRQCrTO+b7Nem3MbY4wX8ADZN3jvjfb5SeA5\nEWkDPgZ8ca5GicinRKReROq7ukK/0MauihwarwzoSl9Bdr57mI6B8bA+ujjbrvIcjrd5GNSVvoLq\nTOcgPcMTEVFL8ttVnsORS30xXz+MxOLznwPvMcYUAf8CfHWujYwxjxtj6owxdbm5K7/a02w3r83G\nGKi/qH3PweSv29y8NvTTYMzn5rXZTE0bDl/SvudgeivWkZMYbl6bzeSUiflxSoEkhnageMb3RdZr\nc24jInH4uoB6bvDeOV8XkVxgizHmdev1HwE7AzqSEKspcuNyOqhv0QJ0MB1q6SU7xUVZTvj7nP1q\nSzJwOkRjHWSHWvpYlZ5IUWZSuJtyzbaSTESI+fUZAkkMh4BKESkTERe+YvL+WdvsBz5uff0A8LLx\nzWG7H9hnPbVUBlQCb9xgn32AW0TWWfvaAzQt/fBWTmK8k82F6XrHEGQNF/vYviYzIvqc/VIT4thY\nkBbzJ4tga2jpZXtpZMXanRzPury0mP+7XjAxWDWDzwAv4DtJP2WMOSUiXxCR91mbfRfIFpFm4HPA\nI9Z7TwFPAY3A88DDxpip+fZpvf6HwI9F5Bi+GsNfBe9wg2tHaRYn2jz6jHuQdA6OcbFnJCyzqS6k\nbk0WR1r79Bn3IGnvH+WyZ4wda0I/m+pC6kozOXyxj6kYnjcpoBqDMeY5Y8w6Y0y5MeYx67XPG2P2\nW1+PGWP2GmMqjDE3GWPOz3jvY9b71htjfn6jfVqv/5sxptoYs8UYc/vMfUWa7WsymZia1rl0gqTB\nuiLfXhp5J4sdpVmMTU7TeFnn0gkGf7dcXSReBJRmMjTu5fTVwXA3JWwisfgcNbZbVzuHtO85KA61\n9JEQ52Dz6vDNjzSfulKNdTDVt/SR4nKyYVX4x6rMVrfGl6zqL8ZurDUxLEN2agJrc1OuXemq5Wm4\n2MuW4oywzMm/kPz0RIqzkmiI8b7nYKm/2Me2NZlhnQtrPkWZSeSnJ8R0TSnyohJldqzJov5in87j\nvkwjE15OXh5gRwR2I/nVrcniUEufrg28TANjk7x5deDaHXekERHqSrNi+ik0TQzLtL00E8/oJOe6\ndG3g5Tja2s/UtLl2Gx+J6koz6R4a51Kvrg28HEcu9WMMEfmQgV/dmkwue8Zo7x8Nd1PCQhPDMvl/\nuQ/F8G1nMNS39CFCWNb8DZTGOjjqW3pxOoTa4oxwN2Ve/ljH6l2DJoZlKs1OJjvFFdOFqmCov9jH\nurw03Mnx4W7KvCpyU0lPjKNBY70s9S19VBWkk5IQF+6mzGvDqjSSXc6YrSlpYlgmX39kZkwXqpZr\natpw+GLftSd/IpXD4et71juGpZucmuZIa+THOs7pYFtJZszGWhNDENStyeJS7widA2PhbkpUOn11\nkKFxb8SfLMD3iHJz55BOnrhEjZcHGJucjuhakt/2NZmcvjrAQAxOnqiJIQj8A7J0MZelabD+37aX\nRP7Jos56kuZIq8Z6KfxdM5H6RNJMdaWZTBticuEeTQxBUFWQTrxTOBKDv0DBcPRSP9kpLoqzImcy\ntflUF7lxOoSjOtPqkhxt7afAncgqd2K4m7KgmiJfcTwWY62JIQgS451UFaTH5C9QMBxt7aO2OCOi\nJlObT7IrjnX5aXoRsERHW/sj+mmkmdxJ8ZTnpsTkFNyaGIKktjiDE+2emJ54ayl8Y0CGo+ZkAb5Y\nH2vt10GNi9RjjQGJrlhncrS1P+YGNWpiCJLakgxGJqY42xm7E28txfE239VYbUn0nCy2FmcwMObl\nQs9wuJsSVY75Yx1NiaEkg57hCdr6YmugmyaGIKkt9hXTtDtpcfz/X/7+3GjgT2Ia68U5eqkfh/jq\nNNFiq5XEYq3rUBNDkJRmJ+NOio/J/sjlONraT3luCu6kyB3YNlt5biopLqfGepGOtPazLj+NZFfk\nDmybbf2qNBLiHDF3EaCJIUhEhC3FGXqyWARjjFWMjPxHF2dyOoSaIo31YkxPG4619rM1iroMAeKd\nDqoL3RyNsceTNTEEUW1xBmc6Bhke94a7KVGhrW+UnuGJqKov+NWWZNB0ZUBX7wvQhZ5hBsa8UVVf\n8KstzuDk5QEmvLGzep8mhiDaWpzBtIHjbbqiWyD8/bZbo/Rk4Z02nLqssQ6Evysm2u4OwXcRMOGd\n5s2rsbN6nyaGINpineC0iyEwRy/1kxDnYH0EruK1kGtFyRjre16qo639pLicVOSlhrspi1Ybg3/X\nmhiCKCvFxZrs5Jjrj1yqo619VBe6iY/AVbwWkpeeyGp3YkydLJbjaGs/NUUZOB2RP4hxtsKMJHJS\nE2KqAB19f5ERrlYL0AGZnJrm5OWBqOxz9qstybj2bL6a39jkFE1XBqKylgS+B0tqizM4GkOx1sQQ\nZLXFGXQMjHPVozOt3sibVwaZ8E5H7ckCfLFu7R2lZ2g83E2JaKcuD+CdNlF9EbC1JIPzXcN4RmJj\nplVNDEH2Vn+kdifdiP//Z0sUDWyb7dqgRr1DvCH//080JwZ/22PlDlETQ5BtLEgnziH6ZNICjrd5\nyE5xUZQZ+TOqzmfT6nQcok+hLeREWz+r0hPJT4/8GVXns7nQN1r7RHtsxFoTQ5AlxjtZvyotZn6B\nlupEu4fqIndUzKg6n5SEOCryUjXWCzhuxTqauZPiKctJuTa3l91pYlgBNUVujrd5Ym5GxkCNTkxx\npmOQmsLoPlkAVBdmaKxvYHBskvNdwzaJtZsTMXJ3qIlhBVQXZuAZneRS70i4mxKRGq94mDZQHcX1\nBb+aIjfdQ+Nc1WVd53Sy3TcoLNrvGMAX68ueMboG7f+wgSaGFVBj/RFo3/PcjrX6/l9qbHCy8J/w\n/Mekrufveomm2XPnU32tzmD/7iRNDCtgXX4arjiH9j3P40S7h7y0hKguRvpVWQ8bxMLJYimOt3so\nykwiK8UV7qYs26ZCNxIjDxtoYlgBrjgHGwvSY6ZQtVjH2/ptcbcAvocN1uWnxcTJYilOtHlsE+vU\nhDjKc1Njos6giWGF1BS6Odk+oMs/zjI4Nsn57mFbdC341RS5OdGuBejZ+kcmuNQ7QnWhvWJ9PAZi\nHVBiEJF7ROS0iDSLyCNz/DxBRH5k/fx1ESmd8bNHrddPi8jdC+1TfB4TkTMi0iQin13eIYZHdZGb\noXEv57t1+ceZTl0ewBh7FCP9qovc9I9Mxtzyjwvxd6Xa5Y4BfBd8XYPjdAzYuwC9YGIQESfwTeBe\noAp4UESqZm32ENBnjKkAvgZ8yXpvFbAP2ATcA3xLRJwL7PMTQDGwwRizEXhyWUcYJv4/Bu17vp7/\nNrzaBo8v+tVYV8TanXQ9///HZhvF2v8knd27iQO5Y7gJaDbGnDfGTOA7Ud83a5v7gCesr58B7hLf\nyKX7gCeNMePGmAtAs7W/G+3z08AXjDHTAMaYzqUfXvhU5KaSFO/Uk8Usx9r6r81WaRfrV6Xhcjps\nf7JYrONt/ZTlRNeyrQupKkjHGQMzGwSSGAqB1hnft1mvzbmNMcYLeIDsG7z3RvssBz4sIvUi8nMR\nqZyrUSLyKWub+q6urgAOI7TinA42rU63/S/QYp1o99jqbgH8DxtoAXq2E232i3WSy0llXirHbf7E\nYSQWnxOAMWNMHfD/AN+bayNjzOPGmDpjTF1ubm5IGxio6iI3py578E7FzpKAN+IZmeRiz4it6gt+\n1UVuTrZ79GEDS9fgOJc9Y7aqL/jVFLk50dZv6wJ0IImhHV+fv1+R9dqc24hIHOAGem7w3hvtsw14\n1vr634CaANoYkWqK3IxNTtPcNRTupkQEOxYj/WoKMxgc99LSow8bAJxst18tya+6KIM+mz9sEEhi\nOARUikiZiLjwFZP3z9pmP/Bx6+sHgJeNL53uB/ZZTy2VAZXAGwvs8yfAHdbX7wLOLO3Qws//mN5x\nHRULwHGrEG/Pk4WOdp/peJsHEXsVnv388z7ZOdYLJgarZvAZ4AWgCXjKGHNKRL4gIu+zNvsukC0i\nzcDngEes954CngIageeBh40xU/Pt09rXF4EPisgJ4H8BnwzOoYbe2pwUUlxOTuqC8YDvKrI4K4mM\n5OgfBTtbZV4qCXGOa1fKse5Eu8f3+58QF+6mBN36VWnEOcTWf9cBRc0Y8xzw3KzXPj/j6zFg7zzv\nfQx4LJB9Wq/3A78TSLsincMhbFrt1qkxLHYsPPvFOX2j3TXWPifbPdy8NivczVgR/tHudr4IiMTi\ns61sLnTTdGUg5gvQ/SMTtPaO2rJrwa+60M2pyzravWvQN9us3WNt59HumhhWWHVRuhagmTH9ss1P\nFkNagLZ14dlvs81Hu2tiWGHXpuq1caEqEP4uls2rbXyyiLHlH+dzot1XeN5k48Tg/7u2a3eSJoYV\nVpaTSrLLadtfoECdtKZfzrTB9MvzqcxPxaUFaE60eyjLSSHVhoVnvw1WAdquFwGaGFaY0yFsWq1F\nSTsXnv3itQAN+C4C7B7rxHgnlfn2XdtdE0MIbC500xjDBWjPiG+ZUzsXI/2qC9M5FcPTrXcPjXPF\nM2b7xAC+WJ+0aQFaE0MIVBf6RkCf64rNoqT/ee/YOFm4GRz3cjFG1/u+VkuKkVj3jUzS3m+/ArQm\nhhCwe6FqISdj6GSxOcZjfco67k2r08PckpX3VqwHwtyS4NPEEAJrrSm47dofuZAT7R4KM+yx7u9C\nKvN8U3DHamLwF57TEu0z1fZ8NlpTcNsx1poYQsDpEKpWp9vyFygQJ9s9bC60/xUk+Kbg3lBg36Lk\nQk62D8TEnSFYBei8VFvGWhNDiPhHxU7FWFFyYGySlp6RmKgv+G0udNu2KHkjvcMTtPePUh0jFwFg\n31hrYgiRzYVuRienOB9jI6Bjqb7gV13oZmDMy6UYK0DHUuHZr7rQTc/wBFc8Y+FuSlBpYgiR6hgd\nFRsL0yPMFuuxjqXEYNfR7poYQqQ8N4XEeIftfoEWcqJ9gNXuRLJttMbzQtbl+wrQMRfrNg+l2cmk\nx0Dh2a+qIB2H2O8pNE0MIRLndFBVEHsFaF/hOXauIMFXgF6/yt7TMs/lRAzG2rcGtP0eNtDEEEKx\nVoAeGJvkQvdwTHUj+fmKkgO2K0rOp+9a4TlWY22vArQmhhDaXOhmZGKKC92xMQK68bJv4M9mG67x\nvJDqQjeeUftOyzxbLI1un626MJ3uoQk6BsbD3ZSg0cQQQv51gWOliyEWC89+sVaA9h+nnafano//\n79pOsdbEEEIVub51ge30C3QjJ9o9FLgTyYmhwrPfulWpxDvtOy3zbCfbPZRkJeNOip3Cs19VgRuH\naGJQSxRr6wLHYjHSLyHO/usCzxQL06rPJ8nlpCIv1Vax1sQQYtWFbhpjYF3goXFvzBae/ey+LrBf\nLKznvZDNVqztQhNDiPnXBb5g83WBT7V7MCY26wt+mwvtvS6wXyys572Q6kI3XYPjdAzYYwS0JoYQ\ni5VpmWNxeoTZYmW69bdiHTtzJM1mt7XdNTGEmH9dYLv8As3nZLuH/PQEctNir/Dst97m6wL7nWz3\nUJyVREay/adVn8/GgnTERgVoTQwhFivrAsdyMdIvMd5XgNZY219KQhzlufYpQGtiCIPqwnRO2bgA\nPTTu5Xz3cEx3I/lV23BU7EyxtJ73QqptVIDWxBAG/gJ0i00L0I2XB2K+8Oy3uci+6wJDbI94nm1z\noZvOwXE6bVCA1sQQBptWW0XJy/ZbKxZie8TzbJuttY/tuC4wzJhqe7XG+trDBpej/65BE0MYrMtP\nswrQ/eFuyoo42e4hLy2BvPTEcDcl7DYWpFsFaHvG2r+ed2YMrOe9kE2rfQXo4zZ4sEQTQxi44uxd\ngD7e7qGmKCPczYgIbxWg7R/xmc4AAByHSURBVHnHcKLdw5ZivVsAXwG6wiYFaE0MYVJjTctstwL0\n0LiXc11D1MTgjKrzqSlyc7yt33YF6P6RCS72jFBdqBcBftVFbo61Rf/DBgElBhG5R0ROi0iziDwy\nx88TRORH1s9fF5HSGT971Hr9tIjcvYh9fkNEbLtAcnWRPUdAn/SPeNbEcE11kT1HQPvvePUi4C01\n10ZAR/cU3AsmBhFxAt8E7gWqgAdFpGrWZg8BfcaYCuBrwJes91YB+4BNwD3At0TEudA+RaQOyFzm\nsUU0/x/TcZvVGfwD97Tw/JYa64raDn3PM/mPRwvPb6ku8sc6uv+uA7ljuAloNsacN8ZMAE8C983a\n5j7gCevrZ4C7RESs1580xowbYy4Azdb+5t2nlTT+D/DXyzu0yFaRm0pSvNN+JwurGBmLU23PZ92q\nVFxOB8dtVoD2r/HsTo69qbbnU1WQjtMGo90DSQyFQOuM79us1+bcxhjjBTxA9g3ee6N9fgbYb4y5\ncqNGicinRKReROq7uroCOIzIEud0sGl1uu2mxjjR1q93C7MkxDnZUJBmv1i3e65dISsf3xrQqVF/\nwRdRxWcRWQ3sBf5xoW2NMY8bY+qMMXW5ubkr37gVUF3kWwPaOzUd7qYEhWdkkpaeEWr0KZW3qSly\nc6LNY5uHDbqHxmnvH2WL1hfeZktRRtQ/bBBIYmgHimd8X2S9Nuc2IhIHuIGeG7x3vte3AhVAs4i0\nAMki0hzgsUSdmiI3o5NTnOuyRwH6WjFSn1J5m5rCDAZtNNpda0nzq7ZGu0fzwwaBJIZDQKWIlImI\nC18xef+sbfYDH7e+fgB42fjS5X5gn/XUUhlQCbwx3z6NMf9hjFlljCk1xpQCI1ZB25aqC+1RqPLz\n96HryeLt7LYu8PE2DyKxucbzQmpsEOsFE4NVM/gM8ALQBDxljDklIl8QkfdZm30XyLau7j8HPGK9\n9xTwFNAIPA88bIyZmm+fwT20yLc2J4XUhLio/gWa6USbhzVajJxTZZ5vve9o73v2O9HeT3luKqkJ\nceFuSsRZvyqNeKdEdawDiqox5jnguVmvfX7G12P4agNzvfcx4LFA9jnHNqmBtC9aORzC5sJ0jkXx\nL9BMx9s8bC3RbqS52O1hg+NtHm6tyAl3MyJSQpyTDavSo3oalIgqPseimqIMmq4MMOGN7gJ0j1WM\n1MFO86spyuDkZQ9TUV6AvuoZo3NwXAcx3kB1kZvjUfywgSaGMKsudDPhneZMx2C4m7Isx6+NgtU7\nhvnUFLkZmZjiXFd0D+j318Q01vPbUuRmcMzLxd6RcDdlSTQxhNkW648r2usMJ/zFyNWxu+7vQt4a\n7R7lsW734HQIVQUa6/lE+4MlmhjCzLdWbnzU/gL5HW/rpywnhbRELTzPpywnlRSXM+pjfazNQ2Ve\nKkkuZ7ibErEq86P7YQNNDGEmItQUZXDkUvSeLIwxHG3tp7ZYuxZuxOkQqovcHG2N7lgf01gvKN7p\nYHNh9MZaE0MEqC3O4EzHIMPj3nA3ZUna+0fpHppgq54sFlRbnEnTlQHGJqfC3ZQludA9jGd0UhND\nAGqLMzjZ7mEyCmc20MQQAbYWZzBtorfO4L8qqi229YS4QVFbnMHklOFUlC7rei3W+ljygmqLMxj3\nTnP6avQ9WKKJIQL4i5LRett59FI/rjgH61elhbspEc8/ziNqY93aT4rLSWWexnoh/ruqI1EYa00M\nESA7NYGSrGSOReEvEPhOFptXp+OK01+nheSnJ1LgTozaWB9r7ae6yI3TIeFuSsQrykwiO8XF0Sis\nH+pfcoSoLc6IyqvIyalpTrR7tBtpEaI11mOTUzReGdBYB0hErFj3hbspi6aJIULUFmdwxTNGx8BY\nuJuyKKevDjLundY+50WoLc7gUu8IPUPRtfxj45UBJqeMFp4XobY4g3NdvoJ9NNHEECH8J9Zoe2zV\n33+qTyQFzn9iPRZl4xn8XSI6H1bg/H/X0TZ2RRNDhKgqSCfeKVHXxXD0Uj/ZKS6KMpPC3ZSosbnQ\njUOIur7no639rEpPJD89MdxNiRr+aUOiLdaaGCJEYryTqoL0qOuPPNraR21xBr4lvlUgUhLiWJef\nFnVPq+ggxsVzJ8VTnpsSdRd8mhgiSG1xBifaomf2zYGxSc51DevJYgm2lmRwrLU/ambf7Bka51Lv\niNaSlqC2OJNjUbbUpyaGCFJbksHwxBRnO6NjQMzxVt+APD1ZLF5tcQYDY14uRMlSn/56iF4ELF5t\nSQbdQxNRtdSnJoYIsiXK+iP93V46/fLi+R/5jJpYX+rHIbps61LU+v+uo6g7SRNDBCnLScGdFB81\nv0BHW/tZm+trs1qcijzfTKvREusjrf2sy08jRZfyXLQNBWkkxDmiJtagiSGiiAhbSzJouBj5BWhj\nDA0X+9hWooOdlsLpEGqjJNZT04ajl/rZtkZjvRTxTgc1Re6oiLWfJoYIs6M0i7OdQ/SPTIS7KTd0\nrmuYvpFJdpTqyWKp6tZk8ebVAQbHInvw0+mrgwyOezXWy1BXmsXJdg+jE9Exq64mhgiz3boqi/Sr\ni4aLvQBsX5MV5pZEr7rSTKZN5A9q9Me6TmO9ZHVrMvFOm6gZ1KiJIcJsKcog3ikcaonsxHCopY/M\nZN8z2mpptpZk4hCob+kNd1Nu6FBLH/npCTqIcRn8F3yRHms/TQwRJsnlZNNq97WrtEjVcLGP7Wuy\ndGDbMqQmxLGxIJ36iL877KNOY70sGckuKvNSIz7WfpoYItCO0kyOtXkY90Zmf2TX4DgXuoe1zzkI\ndpRmceRSf8Su8nW5f5T2/lHqNNbLVleaRcPFvqgY1KiJIQJtX5PFhHeakxG6opu//qEni+XbviaT\n0ckpmq5E5opu/itcrS8sX92aTAbHvJyJggGsmhgikP+EG6l1hvqWXlxxvsXO1fJEQ6yTXU42FuiK\nbcu1o9SXXCM11jNpYohAOakJlOWkUB+hv0D1F/vYUuQmIc4Z7qZEvQJ3EoUZSRFbU6pv6WNrSQZx\nTj1VLFdxVhK5aQk0REEBWqMdoerWZNJwsTfiJt4anZjiZLuHulLtWgiWHaWZHGrpi7hYD45N8ubV\nAe1GChIRuRbrSKeJIULVlWbSN+KbvTSSHGvrxzttqNNRsEGzvTSLrsFxWnsja5K1I5f6mTZaSwqm\n7WuyaO8f5YonsmI9myaGCOW/Io+055797dmuiSFodlyrM0RerB3iG2+hgsMf60jtJvbTxBCh1uak\nkJXiirjbzvqLfVTmpZKR7Ap3U2xjXV4aaYlx1EdYnaH+Yh8bC9JJ1YnzgmZjQTpJ8c6Iu+CbLaDE\nICL3iMhpEWkWkUfm+HmCiPzI+vnrIlI642ePWq+fFpG7F9qniPzAev2kiHxPRGJy6k4RoW5NJm+0\n9IS7Kdd4p6apb+ljR5n2OQeTw+GL9esXIudkMe6d4vClvmtP0qjgiHc62LYmI6JiPZcFE4OIOIFv\nAvcCVcCDIlI1a7OHgD5jTAXwNeBL1nurgH3AJuAe4Fsi4lxgnz8ANgDVQBLwyWUdYRTbVZFDa+8o\nrb0j4W4KAMfbPQyNe9lVnhPuptjOzvIczncNc9UzFu6mAL76wtjkNLsqNNbBtrM8hzevDtI9NB7u\npswrkDuGm4BmY8x5Y8wE8CRw36xt7gOesL5+BrhLfOPn7wOeNMaMG2MuAM3W/ubdpzHmOWMB3gCK\nlneI0WtneTYArzR3h7klPgetdty8Vq8ig21nReTF2iFwk94dBp3/7/rVc5HTGzBbIImhEGid8X2b\n9dqc2xhjvIAHyL7Bexfcp9WF9DHg+bkaJSKfEpF6Eanv6uoK4DCiT0VeKnlpCbwSIb9ArzT3sLEg\nnezUhHA3xXY2rkonK8XFK+ciIzG8cq6H6qIMXYRpBVQXuklLiONghMR6LpFcfP4W8GtjzG/m+qEx\n5nFjTJ0xpi43NzfETQsNEWFneTavnusO+zPuY5NTNFzqY5d1taOCy+EQblmbzavnesIe66FxL8da\n+zXWKyTO6eAda7N5pTkyLvjmEkhiaAeKZ3xfZL025zYiEge4gZ4bvPeG+xSR/w7kAp8L5CDsbGdF\nDt1DE5zpGAprOxou9jHh1T7nlXRLeTZXPGNc6A7v2JVDF3rxThuN9QraVZHNpd6RiKkfzhZIYjgE\nVIpImYi48BWT98/aZj/wcevrB4CXrRrBfmCf9dRSGVCJr24w7z5F5JPA3cCDxpjInHIyhCKlzvBK\nczdxDtEnklaQ/0Qc7q7DV5q7ccU5dKzKCtppPcARqXWGBRODVTP4DPAC0AQ8ZYw5JSJfEJH3WZt9\nF8gWkWZ8V/mPWO89BTwFNOKrFTxsjJmab5/Wvr4N5AOvishREfl8kI41KhVlJrMmO5mD4T5ZnOth\nS3GGPtO+gkqzk1ntTuTVMPc9HzzXw/aSTBLjdS6slbIuP5Wc1ISIqSnNFtBfuTHmOeC5Wa99fsbX\nY8Deed77GPBYIPu0Xtczzyw7y3P42bHLeKemwzKZmWd0khNt/XzmjoqQf3YsERFuKc/h5Tc7mJ42\nOByhXxind3iCxisD/OW714X8s2OJv3540KopRdoiSJFcfFaWXRXZDI57ORGm9RleP9/DtPHVO9TK\n2lWRTd/IJI1hWp/B37WhsV55uyqy6Roc52xneOuHc9HEEAVuWeurM4SrO+nguR4S4x1sLckIy+fH\nknD3PR88101qQhw1utbGivPH+mCEjF2ZSRNDFMhOTWDDqrSwFaAPnutmR2mWrr8QAqvciazNTeG3\nYYt1D+8oy9L1F0KgOCuZ4qwkfhuBj61q9KPEu9blcqill8GxyZB+bmvvCGc6hnhnpXYthMptlbm8\ndr6HkQlvSD/3fNcQF7qHNdYhdFtlLgfPdTM2GVnru2tiiBK7q/KZnDL8+kxoryRfauoAYE/VqpB+\nbizbU5XPuHea354Ndaw7Ad/vmgqN3VX5jExM8er5yLpr0MQQJbaVZJKV4uKAdaIOlQNNnZTnplCW\nkxLSz41lN5VlkZYYF/JYv9jUwcaCdIoyk0P6ubHslrXZJLucHGgMbawXookhSjgdwp0b8nj5zU68\nU6EZ9zcwNslr53v0biHE4p0O7lifx0tNnUxNh2Z6jL7hCepbetmjdwshlRjv5F3rcjnQ1BH2qVBm\n0sQQRXZvzMczOhmyxXv+83QX3mnDnqq8kHyeesvuqnx6hic42hqaWL/8ZifTBvZs1MQQars35tMx\nMB62x9HnookhiryzMgdXnCNkXQwHGjvITnFRW6xTI4Tau9blEucQXmzsDMnnHWjqID89gc2F6SH5\nPPWWOzbk4RAiqjtJE0MUSUmIY1d5dkhuOyenpvnl6U7u3JCHMwwjcGOdOymem9dmh+QiYGxyil+d\n6WL3xvyIG4EbC7JSXNStyeLFptBcBARCE0OU2V2Vz8WeEZpXeLTkoQu9DI559QmVMNq9MY/mzqEV\nn23V92jslMY6jHZX5dF0ZYC2vsiYbVUTQ5S5a4Pvj/fFFb6SfLGpA1ecQ59pD6O7rP7+l1Y41gea\nOkh2Oa+NsFeht/tarCPjrkETQ5RZ5U6kpsjNiyvYH2mM4UBTB7dW5JDs0jkNw6U4K5kNq9L4xQrG\nenracKCxk9sqc3U21TBam5vK2tyUFf27XgxNDFHons2rOHKpn4s9K9PFcPhSH629o9yzWR9TDbd7\nNxdwqKWX9v7RFdn/6xd6uTowxr3VGutwu3fzKg6e66ZjYCzcTdHEEI3u31qEQ+CZhrYV2f/T9W0k\nu5y8p7pgRfavAnf/tkKMgR+vVKwbWklLiOPdOlYl7B7YXsy0gWcPz14gM/Q0MUShVe5E3lmZyzMN\nbUEfADUy4eXfj13mPdUFuihPBCjOSmZneTZPN7QyHeRYD45N8tyJK7x3y2qSXNqNFG5lOSnsKM3k\n6frWsA9208QQpfbWFXHFMxb0GVd/fuIqwxNT7N1eFNT9qqXbW1dEa+8or1/oDep+/+P4FcYmp9lb\np7GOFHu3F3O+e5jDl0IzsHE+mhii1J6qfDKS43k6yF0MTze0siY7mZt0beeIcc+mAtIS4ni6oTWo\n+326oY2KvFS2Fus6G5HiPTUFJMU7ebp+ZboOA6WJIUolxDm5b8tqXjh1Fc9IcKbivtQzwmvne3lg\nW5EOdIogSS4n792ymp+fuMrQeHCm4j7XNUTDxT72btdYR5LUhDjeU13Az45fCfm06zNpYohie+uK\nmfBOs//45aDs75nDbYjAB7UbKeLsrStidHKK/whWrBvacDqED2wrDMr+VPDsrStiaNzL8yevhq0N\nmhii2KbV6WxYlcbT9cvvYpiaNvy4oY1bK3JYnZEUhNapYNpanEF5bgpPBaGLwTs1zbOH27h9XS55\naYlBaJ0KpneUZbEmO5mngvB3vVSaGKKYiPB77yjheJtn2UXofz92mfb+UX7vppIgtU4Fk4jw4E0l\nNFzs441lFqGfPdJOx8A4D2qsI5KIsG9HCa+d7w1bEVoTQ5T7UF0xBe5EvvyL00t+xG1yapqvHzjD\nxoJ07t6kz7NHqo+8Yw25aQl8+YWlx3rCO80/HDhLTZGbuzbqdOqR6r/csobsFBdf/cWZsHy+JoYo\nlxjv5LN3VXLkUj8vv7m0eVZ+3NBGS88If7FnHQ6dSTViJbmc/MmdFbzR0stvlrjs548OXaK9f5S/\nePd6LTpHsJSEOD59ezm/be7m1XOhX/ZTE4MNPLC9iDXZyXzlF2cWPQhq3DvFN146y5biDL2CjAIf\n3lFMYUYSX1nCHeLY5BT/+HIzO0ozuU0nR4x4H715DfnpCUuK9XJpYrCBeKeDP9tdSeOVAX6+yCcZ\nfvj6JS57xvgrvYKMCglxTv70rkqOtXkWPeHa/331Ip2D4/ylxjoqJMY7+ZM7K6m/2Md/nukK6Wdr\nYrCJ920ppDIvla+8eJqxyamA3jMwNsk//fIcN6/NYleFTrkcLe7fVkhZTgpf+cUZJryBrf/dPzLB\nP//qHO+szOEdOr121PhQXTHFWUl8+YXTTIZorXfQxGAbTofw335nI+e7hvmbHx9f8NZzcmqah39w\nmP6RCR65d6NeQUaROKeDR+/dwOmOQR599sSCsZ7wTvPp//cwQ2Ne/uaeDSFqpQoGV5yDR+/dyKnL\nA3z+pydD1qWkicFG7lifx1/dvZ6fHr3MP7x0dt7tjDH83f5T/OZsN//z/mpqdUqEqPPuTav4s92V\n/PhwG9/6z3PzbmeM4W9/coJXz/fwvx+oYXOhO4StVMHwnuoCHr6jnB++0cp3fnMhJJ+p02fazB/f\nXs75rmG+fuAsZTkp3Ff79pGt33ulhR+8fon/+q5yPlRXHIZWqmD407squdA9zP954TSl2Sn8Ts3b\np0n/9q/O81R9G5+9s4L3b9VRztHqL/as50L3MP/z502syU7m3Sv8WLmEe3rXYKirqzP19fXhbkbE\nmPBO87Hvvk7DxT7evSmffTtKuHltNv95upMnD7Xyy9Od3F21im99ZJs+nhrlxian+Mh3XudYaz93\nb17F791Uwo7SLF5+s4MfvtHKr8508btbVvONfbXaXRjlxian+PDjr9F42cO9mwt48KYSbl6btay4\nikiDMabuba8HkhhE5B7gHwAn8B1jzBdn/TwB+FdgO9ADfNgY02L97FHgIWAK+Kwx5oUb7VNEyoAn\ngWygAfiYMWbiRu3TxPB2npFJvvHyWZ493EbfyCQup4OJqWny0hL4UF0xD99RoXPw20T/yARfP3CW\nfzvSjmf0rVivSk/kwzuK+fTt5bpsp030DI3zjZfO8uyRdgbHvJTlpPDtj25n/aq0Je1vyYlBRJzA\nGWAP0AYcAh40xjTO2OaPgRpjzH8VkX3AB4wxHxaRKuCHwE3AauAAsM5625z7FJGngGeNMU+KyLeB\nY8aYf75RGzUxzG/cO8ULpzp47XwPt6/L5c4NecQ5tbRkR2OTUzx/8ipvtPRy5/o8bl+fq7G2qdGJ\nKZ47cYWfHrvMtz+6bclrsy8nMdwC/J0x5m7r+0cBjDH/a8Y2L1jbvCoiccBVIBd4ZOa2/u2st71t\nn8AXgS5glTHGO/uz56OJQSmlFm++xBDI5UQhMHOavzbrtTm3McZ4AQ++rqD53jvf69lAv7WP+T7L\nf0CfEpF6Eanv6grt4A+llLKzqL3PNMY8boypM8bU5ebmhrs5SillG4EkhnZg5jONRdZrc25jdSW5\n8RWh53vvfK/3ABnWPub7LKWUUisokMRwCKgUkTIRcQH7gP2zttkPfNz6+gHgZeMrXuwH9olIgvW0\nUSXwxnz7tN7zS2sfWPv86dIPTyml1GItWMq2isCfAV7A92jp94wxp0TkC0C9MWY/8F3g/4pIM9CL\n70SPtd1TQCPgBR42xkwBzLVP6yP/BnhSRP4eOGLtWymlVIjoADellIpRy3kqSSmlVAzRxKCUUuo6\ntuhKEpEu4OIS354DLG2dxOgWi8cdi8cMsXncesyBWWOMedvz/rZIDMshIvVz9bHZXSwedyweM8Tm\ncesxL492JSmllLqOJgallFLX0cQAj4e7AWESi8cdi8cMsXnceszLEPM1BqWUUtfTOwallFLX0cSg\nlFLqOjGdGETkHhE5LSLNIvJIuNuzEkSkWER+KSKNInJKRP7Uej1LRF4UkbPWv5nhbmuwiYhTRI6I\nyM+s78tE5HUr3j+yJnC0FRHJEJFnRORNEWkSkVvsHmsR+XPrd/ukiPxQRBLtGGsR+Z6IdIrIyRmv\nzRlb8fmGdfzHRWTbYj4rZhODtWTpN4F7gSrgQWspUrvxAn9hjKkCbgYeto7zEeAlY0wl8JL1vd38\nKdA04/svAV8zxlQAffjWIrebfwCeN8ZsALbgO37bxlpECoHPAnXGmM34JuXchz1j/X3gnlmvzRfb\ne/HNZl0JfAq44fLIs8VsYsC3DnWzMea8MWYCeBK4L8xtCjpjzBVjzGHr60F8J4pCfMf6hLXZE8D7\nw9PClSEiRcDvAN+xvhfgTuAZaxM7HrMbuA1rRmJjzIQxph+bxxrfLNFJ1jouycAVbBhrY8yv8c1e\nPdN8sb0P+Ffj8xq+dW4KAv2sWE4MgSxZaisiUgpsBV4H8o0xV6wfXQXyw9SslfJ14K+Baev7gJeN\njWJl+NZM/xerC+07IpKCjWNtjGkHvgxcwpcQPEAD9o+133yxXdb5LZYTQ0wRkVTgx8CfGWMGZv7M\nWiDJNs8ti8h7gU5jTEO42xJiccA24J+NMVuBYWZ1G9kw1pn4ro7LgNVACm/vbokJwYxtLCeGQJYs\ntQURiceXFH5gjHnWernDf2tp/dsZrvatgF3A+0SkBV8X4Z34+t7tvmxsG9BmjHnd+v4ZfInCzrHe\nDVwwxnQZYyaBZ/HF3+6x9psvtss6v8VyYghkydKoZ/WtfxdoMsZ8dcaPZi7HaqslVI0xjxpjiowx\npfji+rIx5iPYfNlYY8xVoFVE1lsv3YVv9UTbxhpfF9LNIpJs/a77j9nWsZ5hvtjuB/6L9XTSzYBn\nRpfTgmJ65LOIvAdfX7R/edHHwtykoBORW4HfACd4q7/9v+GrMzwFlOCbsvxDxpjZha2oJyK3A39p\njHmviKzFdweRhW/Z2I8aY8bD2b5gE5FafAV3F3Ae+H18F4C2jbWI/A/gw/iewDsCfBJff7qtYi0i\nPwRuxze9dgfw34GfMEdsrST5T/i61UaA3zfGBLzMZUwnBqWUUm8Xy11JSiml5qCJQSml1HU0MSil\nlLqOJgallFLX0cSglFLqOpoYlFJKXUcTg1JKqev8/1iu+eMC7ZcBAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7MxmYxMmec75",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 281
        },
        "outputId": "aff65a44-9a61-44dc-9e53-4789510507fc"
      },
      "source": [
        "'''\n",
        " 带热启动的余弦退火\n",
        " T_0：第一次重启的迭代次数\n",
        " T_mult：增大周期\n",
        "'''\n",
        "scheduler = lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0=20, T_mult=2, eta_min = 1e-5)\n",
        "\n",
        "plt.figure()\n",
        "x = list(range(100))\n",
        "y = []\n",
        "for epoch in range(100):\n",
        "    optimizer.step()\n",
        "    scheduler.step()\n",
        "    lr = scheduler.get_lr()\n",
        "    y.append(scheduler.get_lr()[0])\n",
        "plt.plot(x, y)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f6b25c886a0>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dZ2Ac1bnw8f+zuyqW1WxJlq1muchF\nuGK5gsGACSYUX7oNoffyQkJyKfem3HBDAjcJJKEkIZSYakyNCQ4EMMWAm9y7LdwkV7nJvezueT/s\nyshGtlbW7M7M7vP7wmp2dvYMsz7POec5c0aMMSillFL1PHYXQCmllLNoYFBKKXUEDQxKKaWOoIFB\nKaXUETQwKKWUOoLP7gJYITc315SWltpdDKWUcpVZs2ZtMcbkHb09LgJDaWkplZWVdhdDKaVcRUTW\nNLZdh5KUUkodQQODUkqpI2hgUEopdQQNDEoppY6ggUEppdQRIgoMIjJKRJaJSJWIPNDI+yki8nr4\n/ekiUtrgvQfD25eJyDkNtj8vIptFZOFRx2orIh+JyIrwf9uc+OkppZRqriYDg4h4gaeAc4FyYKyI\nlB+1243AdmNMV+Bx4NHwZ8uBMcBJwCjg6fDxAP4e3na0B4BPjDFlwCfhv5VSSsVIJPcxDAKqjDEr\nAURkPDAaWNxgn9HA/4Rfvwk8KSIS3j7eGHMAWCUiVeHjTTXGfNGwZ3HUsUaEX48DPgPuj/iM1Hfs\nOxjg5WlrSE320i4jhQ5ZqfTskEmSV0cS483uA35enLqaQMCQ7POQ7POQ1SqJNmnJtGmdTEFWKrnp\nKXg8YndRlYNFEhgKgeoGf9cAg4+1jzHGLyJ1QE54+7SjPlvYxPflG2M2hF9vBPIb20lEbgFuASgp\nKWn6LBqxessetu45yICO8T1aNW3lVh6etOSIbZmpPk7v3o6zerRjVK/2pCZ5j/Fp5SZfVW3h/z5Y\ndtx9kn0eirJb0aVdOt3zM+jePoM+RVmUtE0j1J5Tic7Rdz4bY4yINPokIWPMM8AzABUVFc1+2pAx\nhh+/MY+Vtbt5545TKM1t3cLSOtcBfxCAcTcMom1aMqu37uGL5bV8umwz781bT96kFG49rTNXDe5I\nq2QNEG52MHyt/3XPcDrmpLH/UJAdew+yfe8htu05yIa6fdRs30f1tr2s2LybyUs3EwiG/vnktE6m\nf0kbhnRuy6lluXTPz9BAkaAiCQzrgOIGfxeFtzW2T42I+IAsYGuEnz3aJhHpYIzZICIdgM0RlLHZ\nRITfX9aX/3j6K24YN5N3bj+FrLSkaHyV7er/4bfPTKV7+wx6F2VxQd8CgkHDtJVbeWJyFb96fwl/\n+fwbHji3J5ecXKgVgkvVX+vUJC9pyT7SkqFt6+Rj7r//UICqzbuZV7OD2Wt2MHvtdj5esgmAvIwU\nzuiex6he7Tmlay4pPm00JIpIAsNMoExEOhGq1McAVx61z0TgWmAqcCkwOdzanwi8KiKPAQVAGTCj\nie+rP9Yj4f/+I8JzabbS3Nb89QcD+MFz07n9lVmMu2FQXI67+4OhVqT3qHFlj0cY1jWXYV1zmbFq\nG49+sJSfvDGPDxdt5DcX9yY3PcWO4qoW8IcDgy/CHEJqkpdehVn0KsziqsEdAVi/Yx9frtjCFytq\n+deCjUyorCE9xcfInu24+OQiTuma+53fkoovTdaCxhg/cBfwIbAEmGCMWSQiD4nIheHdngNywsnl\newnPJDLGLAImEEpUfwDcaYwJAIjIa4QCSXcRqRGRG8PHegQ4W0RWACPDf0fN4M45PHJxH77+Ziu/\nfG9RNL/KNoEIKotBndoy4dah/Nf3e/D5slrOefwLpqyojVURlUUCx2gENEdBdisuH1jMk1eeTOXP\nRvLC9QM5v08HJi/dzDXPz2DYI5/w6AdLqd6216piK4cRY5o9PO84FRUVpqWrqz703mKe/2oVk+4e\nTnlBpkUlc4YJldXc9+Z8ptx3BsVt05rcf9nGXdz92hyqanfzm4t6c/nA4iY/o5zhlelr+O93FjLj\nv86iXWaqpcfefyjAp0s38+asGj5dthkDnNm9HdcMK+W0slwdfnQhEZlljKk4env8jZucoHtGlpGZ\n6uOxj5bbXRTLHe4xeCP7h9u9fQZv3j6UYV1yuO+t+Tz20XLioQGRCILhax2NoZ7UJC/n9u7Ac9cN\n5Mv7z+SuM7oyr6aOa5+fwbl/nMLbs2s4FAha/r0q9jQwhGW1SuKW0zrz8ZJNzFm73e7iWMp/ApVF\nRmoSz183kMsrivjTJyv473cXanBwgW9zDNH9p12Q3Yoff687Xz9wJr+7rC+BoOHeCfMY8dvPeG3G\nWg0QLqeBoYHrTulE29bJcddrCIT/kTa3skjyenj0kj7cdnoXXp2+ll9PWqLBweHqe4feCHuHLZXs\n83DpgCI+/OFpPHdtBXkZKTz49gLO/P1nTKisPlwe5S4aGBpIT/Fxx4guTFmxhWkrt9pdHMucSI+h\nnohw/6juXDO0I3+bsoonJldZXTxloebOSrKKxyOc1TOfd+4YxgvXDSS7VTL3vTmf7/9xCp8t26wN\nCpfRwHCUHwzpSH5mCo/9O356DZHMSjoeEeF/LjiJi/sX8thHy3lx6mrrCqcsFYhijiESIsIZPdox\n8a5TeOrKk9l3KMB1L8zkmudnsGLTLlvKpJpPA8NRUpO83Dy8MzNWb2PZxvj4Ibekx1DP4xH+79I+\njOzZjl++t5ivqrZYVTxlIX8gfK1tniEkIpzXpwMf3XsaPzu/nHnVOzj3j1N4+P3F7D7gt7Vsqmka\nGBpxUf9CfB7hjcrqpnd2gZb2GOr5vB7+MKY/XfJac+ers1m7VeexO00gGMQjOGaRvBSflxtP7cSn\nPxnBJScX8bcpqzjzd5/xrwUbmv6wso0GhkbkpKdwVs92vDt3XVzMrrCix1AvPcXHM1dXEAwabn6x\nkj3a+nMUf9BEfUbSichJT+HRS/vw7p2nkJuewu2vzOaWFyvZWLff7qKpRjjvF+QQlw0oZsvug3y6\nNCpLNcVUIBjE6xHLbkAqzW3Nk1eezIrNu7j/rfmaWHSQQNA4ermKfsXZ/OOuU3jg3B58vryWsx/7\nnAmV1fobchgNDMcwonseuekpvDGrxu6itJg/CpXFad3y+PH3uvPP+Rt4a3ZT6yKqWAn1GJwbGCA0\nDfq207vw4Q9Po2dBJve9OZ+bX6xk8y7tPTiFBoZj8Hk9XHxyIZ8u3cyW3QfsLk6LBALRqSxuO70L\ngzu15Rf/WMiarXssP75qvkDQxOwehpYqzW3N+JuH8LPzy5myYgvnPP4FHy7aaHexFBoYjuuyAUX4\ng4Z357i7RRyNHgOEchaPX9EPr0e4Z/zcuMjHuJ0/GHR8j6Ehj0e48dROvH/3cIrapHHrS7P473cW\nsO9gwO6iJTQNDMdRlp9B3+Js3qiscfUYaCCKwwsF2a349cW9mVu9gyc+WRGV71CRc3qO4Vi6tkvn\nrduHcetpnXll+loufPLLuJku7kYaGJpwcf9Clm3axTe17h0qCfUYonepz+9TwMUnF/LUZ9+weP3O\nqH2Papo/4MxZSZFI9nl48Ps9eenGQWzfe4jRT33J27Pdn+NzI3f+gmLo7PLQI6frn2rlRoEYDC/8\n/Pxy2qQl8cDb83V9HBu5tcfQ0PCyPCbdfSp9i7K5d8I8Hnx7PvsP6dBSLGlgaEJBditOKsjk48Xu\nDQzRyjE0lJ2WzC8uOIn5NXW88NWqqH6XOjY3zEqKRLvMVF65aTC3j+jCazOqufyvU1m/Y5/dxUoY\nGhgiMLJnPrPWbmerS2cnBYIm4mcxtMT5fTpwZo92/P7fy/XpXjaJhx5DPZ/Xw/2jevDM1QNYWbuH\nC5/8kulxtLilk2lgiMDZ5fkYA5NderNbLHoMEFof53//oxcegZ/q8xts4Q/fzBhPvndSe969cxiZ\nqUlc9ex0Xp62xu4ixT0NDBE4qSCT9pmprs0zROs+hsYUZrfi3u915/PltXzk4uE3t4pV7zDWurbL\n4N27TmF4WS4/fXch/zNxEX6dHh01GhgiICKMLG/HF8u3uDIJFu1ZSUe7ZmhHytql86v3l7jy/5eb\nxfpax1JmahLPXjuQG0/txN+/Xs2N4yrZuf+Q3cWKS/H5C4qCkT3z2XcowNRv3DfGGYtZSQ0leT38\n/IJy1m7by3NfaiI6lqJ5z4oTeD3Cz84v59cX9earqi1c9mdNSkeDBoYIDe2SQ+tkLx+5cDgpVjmG\nhoaX5fG98nye+rRKV9CMIX8gfpLPx3Pl4BL+fv0g1u/Yx0VPf6X3z1hMA0OEUnxeTuuWxydLNhF0\n2Tx9u1qRPz2vHH/Q8Mi/lsT8uxNVvPcYGjq1LJc3bh+KIFz+16l8sbzW7iLFDQ0MzXBmj3Zs2nmA\npS67Vd+OHgNASU4aNw/vxLtz17Ogpi7m35+I4nFW0vH0aJ/JO3cOo6hNK24cN5N/zHX3umZOoYGh\nGU7pmgvA19+467GWds5Uue30LrRJS+LRD5ba8v2JJpF6DPU6ZLViwm1D6V/ShnvGz9UbLC2ggaEZ\nCrJb0Sm3NV+7LAFt50yVjNQk/t+ZZXxZtYUpK7SrH23xPCvpeDJTk3jxhkF8rzyfX763mN//e5ne\nR9MCifcLaqFhXXKYvnKrq5aYjvWspKNdNaSEojateORfS12Xn3GbROwx1EtN8vL0VSdzRUUxT0yu\n4pfvLdbf2wnSwNBMp3TNZc/BAPNrdthdlIjZPVMlxeflJ9/rzqL1O3lv/nrbypEI/C56UE80+Lwe\nHrmkNzeF73W47635eiPcCdDA0ExDO+cA8HWVe4aTnNCKvLBvAT07ZPK7fy/joF//oUaLE6613USE\n/z6vJz8cWcabs2q4e/wcV/XwnUADQzO1aZ1MeYdMvnJRAtoJC6t5PMJ953Snets+3tI19qMm0WYl\nHYuI8MOR3fjpeT2ZtGAjd7wymwN+vQs/UhoYTsApXXOYvWaHax4/6JSlmEd0z6NfcTZPTq7SXkOU\nxHJdLDe4aXhnHhp9Eh8t3sTtL8/WJVoiFFFgEJFRIrJMRKpE5IFG3k8RkdfD708XkdIG7z0Y3r5M\nRM5p6pgicpaIzBaRuSLypYh0bdkpWm9Y11wOBoLMWrPd7qJEJOCQmSqhVlwZ63bs441Z1XYXJy4l\n6qyk47lmaCm/vqg3k5du5paXZmlwiECTvyAR8QJPAecC5cBYESk/arcbge3GmK7A48Cj4c+WA2OA\nk4BRwNMi4m3imH8GrjLG9ANeBX7aslO03qDStvg84prhJCc9IP70bnn0L8nmqclV2rWPAs0xNO7K\nwSX83yV9mLKills1ODQpkqbFIKDKGLPSGHMQGA+MPmqf0cC48Os3gbNERMLbxxtjDhhjVgFV4eMd\n75gGyAy/zgIcN42ldYqPfsXZfF3ljsAQcNBMFRHhRyO7sb5uPxMqNddgNbvucneDywcW88jFvfl8\neS23vazB4XgiCQyFQMN+f014W6P7GGP8QB2Qc5zPHu+YNwGTRKQGuBp4pLFCicgtIlIpIpW1tbG/\ncWpY11wWrKujbp/zl/11So6h3vCyXAZ0bMPTn2qvwWraYzi+KwaW8JuLe/PZslpuf3mW/v6OwYmD\nkT8Cvm+MKQJeAB5rbCdjzDPGmApjTEVeXl5MCwgwpHNbggZmuyDPEHDYipv1uYYNdft5e7aubWMl\nfzDomN6hU40dVMLDF/Xi02W13P2aTmVtTCSBYR1Q3ODvovC2RvcRER+hIaCtx/lso9tFJA/oa4yZ\nHt7+OjAsojOJsX7F2fg8wszV2+wuSpOc1mMAOLVrLn2KsvjL59/oDUgW0h5DZK4a3JGfn1/Oh4s2\nce+EeQT0DukjRBIYZgJlItJJRJIJJZMnHrXPRODa8OtLgckmtFDJRGBMeNZSJ6AMmHGcY24HskSk\nW/hYZwOOXLM5LdnHSQWZVLqhx+DAmSoiwh0jurJm614mLdxod3Hihs5KitwNp3bivlHdeW/eeu5/\na74un9GAr6kdjDF+EbkL+BDwAs8bYxaJyENApTFmIvAc8JKIVAHbCFX0hPebACwG/MCdxpgAQGPH\nDG+/GXhLRIKEAsUNlp6xhSpK2/LytDUc9AdJ9jn3H6OTZiU19L3yfLq2S+fpT6u4oE8HQvMV1IkK\nBg3G4Mhr7VR3jOjK/kNB/vTJCtJTfPzignL9HRJBYAAwxkwCJh217ecNXu8HLjvGZx8GHo7kmOHt\n7wDvRFIuu1V0bMNzX65i4fo6Ti5pY3dxGhUMGoIGR+UY6nk8wu2nd+HHb8xj8tLNnNUz3+4iuZo/\n3OJ14rV2sh+NLGPPAT/PfbmKzFZJ3Ht2t6Y/FOec28x1gQGloWAwa7Vzh5MC4aWHndqKvLBfAYXZ\nrXjy0ypdJrmF6sfJnXqtnUpE+Ol5Pbm8oog/fbKCZ6estLtIttPA0ALtMlLpmJPm6AR0fWXh1Jkq\nSV4Pt57emTlrdzB9lXP/P7qBPxhK4muPoflEhN9c3IfzenfgV+8vYUJlYt+Zr4GhhQZ0bMOsNdsd\n29r1u6AVedmAYtq2TtaWWgtpj6FlvB7h8Sv6MbwslwffXsBHizfZXSTbaGBooYGlbdm65yCrtuyx\nuyiNCgTqx52de6lbJXu5ekhHPl6ymarNu+0ujmsdzjF4nXutnS7Z5+EvPxhAr8Is7nx1NtNXumd5\nfSvpL6iFKjqG8gxOnbZaP7zg9Fbk1UM7kuLz8NyX2ms4UdpjsEbrFB8vXDeQ4jatuGlcJYvX77S7\nSDGngaGFuuSlk52WRKVD8wwBl8xUyU1P4ZIBRbw1ex21uw7YXRxX0llJ1mnbOpmXbhxM6xQf170w\ng+pte+0uUkxpYGghj0cYUNLGwT0G97Qibzy1E4cCQV6autruorjS4WFDnYdviYLsVrx44yD2Hwpw\n7Qsz2L7noN1FihkNDBYYUNqGlbV72LrbeS1dt/QYINT7GtkznxenrWHvQb/dxXGdw8OGDp2B5kbd\n8jN49tqB1Gzfxw3jZrrm4VwtpYHBAhUd2wIwZ+0Om0vyXYd7DC6pLG4e3pkdew/p4nonwE2NADcZ\n1KktfxrTj7nVO7h7/JyEWFdJA4MFehdm4fUIc6udFxgCh+e2u+NSDyxtQ+/CLF74apWuXdNMbho2\ndJtRvTrwi/PL+WjxJh56b5Fjp6dbxR21hcO1SvbSPT/DkYHBbZWFiHD9KaV8U7uHKS55EJJTfNtj\n0H/W0XDdKZ24eXgnxk1dw9/i/J4b/QVZpF9JNvOqdziulesPuG944bw+HcjLSOGFr1bZXRRXcVsj\nwI0ePLcn5/XpwK8nLeWf8x33cEnLaGCwSL/ibHYd8LPSYTe6uXFue4rPyw8Gd+SzZbV6w1szBHRJ\njKjzeITfX9aXgaVtuHfCPGatceY09ZbSwGCR/sXZAI4bTnLr3ParhpSQ7PUw7uvVdhfFNep7h25q\nBLhRapKXv15dQUFWKje/OIvVDmsMWkEDg0U656WTnuJjbrWz7mf4tsfgrkudm57Chf0KeHNWDXV7\nnf9cbSfQWUmx07Z1Mi9cPwhjDNf/fWbc3ePgrtrCwbweoU9RlgN7DO4dXrj+lFL2HQrweuVau4vi\nCm6bmux2nXJb88w1Fazbvo9bX57FQX/8PKJWA4OF+hVns3TDLvYfcs5NMAEXVxYnFWQxsLQNL09b\n67ikvhPprKTYG1jalt9e1ocZq7bx4NsL4mYaq/6CLNSvOBt/0LBwXZ3dRTnMrTmGetcMLWXttr18\nvrzW7qI4ns5KssfofoXcc1YZb82u4enPvrG7OJbQwGChfg5MQAdcnpA856T2tMtIYdzU1XYXxfF0\nVpJ9fjiyjAv7FvDbD5cxacEGu4vTYhoYLNQuM5WCrFRHBQa39xiSfR6uHFzCZ8tq43L2h5W0x2Af\nEeH/Lu3DgI5tuHfCXBbUOGfU4ERoYLBYv5JsRwUGt85KaujKQSX4PMJL09bYXRRH01lJ9gpNYx1A\nTusUbnpxJpt27re7SCfMvbWFQ/UrzqZm+z62OGSlVTfPSqrXLjOVc3t3YEJlta66ehzf3seg/6zt\nkpuewrPXVrB7v5+bX6x07Wqs+guyWL/i0BPd5jpkpVU33vncmGuHdmTXfj/vzonfZQha6nCPwYUz\n0OJJzw6Z/HFMfxasq+Mnb8xz5UwlDQwWO6kgE4/AAofMTHJ7jqHegI5t6NE+g5enrXHlP7RY0ByD\nc4wsz+fBc3vw/oINPDG5yu7iNJsGBou1TvHRtV26YwKDm+9jaEhE+MGQjizesJM5DsrhOInOSnKW\nm4d35uKTC3nso+X8y2UzlTQwREHvwmzm1+xwRMs2XnoMAP/Rv5D0FB8vT9UkdGO0x+AsIsKvL+rN\nySXZ3DthnqPub2qKBoYo6FOUxZbdB9lQZ/+shEAg/LjHOEhIpqf4uKh/If9csCHu1qaxgs5Kcp7U\nJC9/uXoAbdKSuOXFSsdMSmmK+2sLB+pTlAXAfAfMZY6nHgPAD4Z05KA/yBuzqu0uiuP442Bqcjxq\nl5HKM9dUsG3vQW53yZpK+guKgp4dMvF5hAXr7B8Lj5dZSfW6t89gYGkbXpmu6ycdTXsMztWrMIvf\nXtqXmau384uJCx0xzHw8GhiiIDXJS7f8DO0xRMkPhnRkzda9fKmP/jyCPo/B2S7oW8AdI7rw2oxq\nx9+sqYEhSvoUZbFgXZ3tLYN46zEAjOrVnpzWybwy3dn/uGItEAwiEnrKmHKmn3yvOyN7tuOX7y1m\n6jdb7S7OMUUUGERklIgsE5EqEXmgkfdTROT18PvTRaS0wXsPhrcvE5FzmjqmhDwsIstFZImI3N2y\nU7RH76Isduw9RPW2fbaWIx57DCk+L5dWFPHxks2uXnbAav6giasGQDzyeITHr+hHaU4ad746m5rt\ne+0uUqOaDAwi4gWeAs4FyoGxIlJ+1G43AtuNMV2Bx4FHw58tB8YAJwGjgKdFxNvEMa8DioEexpie\nwPgWnaFN+hSGVlqdb3OeIRAM4vUIIvFVYYwdWEIgaHh9piah6wWMiasGQLzKSE3ib9dUcCgQ5JYX\nZzly2YxIegyDgCpjzEpjzEFCFfXoo/YZDYwLv34TOEtCNdFoYLwx5oAxZhVQFT7e8Y55O/CQMSYI\nYIzZfOKnZ59u7dNJ9npsX2XRH4zPyqI0tzWnds1l/Iy1h4fLEl0gYHRGkkt0zkvnT2P7s2TjTv7z\nTectmxHJr6gQaNgsqwlva3QfY4wfqANyjvPZ4x2zC3CFiFSKyL9EpKyxQonILeF9KmtrnfcQlxSf\nlx4d7E9AhyqL+AsMAFcOLmF93X4+X+7KtoPl4rUREK/O6N6O+87pwT/nb+CvX6y0uzhHcGLzIgXY\nb4ypAP4GPN/YTsaYZ4wxFcaYiry8vJgWMFJ9irJYuK7O1mmV8VxZnF2eT256Cq9O12dCQ2iiQbw2\nAuLVbad35rw+HXj0g6V8tsw5DZxIAsM6QmP+9YrC2xrdR0R8QBaw9TifPd4xa4C3w6/fAfpEUEZH\n6lOYza4DflZvte8BM/FcWSR5PVwxsIjJSzezfoe9SX4niOdGQLwSEX57aR+652dw92tzHPMwqkgC\nw0ygTEQ6iUgyoWTyxKP2mQhcG359KTDZhAbNJgJjwrOWOgFlwIwmjvkucEb49enA8hM7Nfv1dsAd\n0KHKwokdQ2uMGViCAcZrEppAMBi3jYB4lpbs45mrK/B4hFteqmTPAfufOdJkjRHOGdwFfAgsASYY\nYxaJyEMicmF4t+eAHBGpAu4FHgh/dhEwAVgMfADcaYwJHOuY4WM9AlwiIguA3wA3WXOqsVfWLp0U\nn8fWlVbjvbIobpvGaWV5TJhZjT/g/KUGoskfNPosBpcqyUnjybEnU7V5tyOS0b5IdjLGTAImHbXt\n5w1e7wcuO8ZnHwYejuSY4e07gPMiKZfT+bweenbItDUwJMLwwthBJdz28iw+W1bLyPJ8u4tjm9Cw\nYfz2DuPdqWW5PHBuD349aSl//vwb7hjR1bay6K8oynoXZrF4/U7bEtCBoHH9sxiaclbPduRlpPDa\njMROQidCIyDe3Ty8Mxf0LeC3Hy6zNRmtgSHKehVmstvGBHQiVBZJXg+XVxTx6bLETkLH89TkRCEi\nPHpJ78PJ6DU21RsaGKKsV2EoAW3XcFKiVBZjBpYQNDChMnGT0InQCEgE9cloEeHWl2ax92Dsk9Ea\nGKKsW34GyT6PbU9vivdZSfWK26YxvCyXCTOrE/ZO6HifaJBISnLS+NPY/izbtIv731oQ82R0/NcY\nNkvyeujZPsO+HkMCVRZXDgrdCf3FcufdCR8L2mOIL6d3y+M/z+nOe/PW89yXq2L63RoYYqBXYRaL\n1tmTgE6kymJk+E7oVxL0TmidlRR/bj+9C+f2as+vJy3h6xg+f0R/RTHQqzCLXQf8rN0W+yV24/nO\n56MleT1cOiCUhE7E5bgTqRGQKESE317Wly556dz12hzWxWhyhQaGGOhtYwI60SqLMQOLCQQNbyRg\nEjoRpiYnovQUH3+9egCH/EFue2kW+w9Ff5luDQwx0C0/g2SvPQnoRKssSnNbM6xLDuNnVifcM6ET\nrRGQSDrnpfPYFf1YsK6On74b/WdGa2CIgWSfh+42JaATZVZSQ2MGlVCzfV/CPRM6kSYaJKKzy/O5\n+6wy3pxVw8tRzqMlVo1ho16FoSW4Yz3tLBEri3NOyqdNWhLjZyZWEtof0B5DvPvhWWWc0T2Ph95b\nxKw126L2PRoYYqR3YRY79/tj/gzoRKwsUnxeLjm5iH8v2kTtrgN2FydmdFZS/PN4hD9c0Z+C7Fbc\n/vJsNkdpkoX+imKkV2EmEPsEdCLNSmpozKBi/EHDW7Nr7C5KzAQ0x5AQstKS+OvVA9i1388dr8zm\noN/6VYU1MMRI9/YZJHnFlsCQiJVF13YZDCxtw+szq21fwjhW/AnaCEhEPdpn8uilfViyYSfLNu6y\n/PgaGGIkxeelW35GzGcmJXJlMXZQCau27GHayuiNxTpJojYCEtWFfQv4/L4zDj8QzEoaGGKod2EW\nC2KcgA4k4Kyket/v3YHMVF/CLMftDwYTamqygtz0lKgcNzFrDJv0Ksyibt8harbHLgHtT8BZSfVS\nk7xc1L+QDxZuZPueg3YXJ9Ac7q0AABfUSURBVOq0x6CsooEhhuqX4I7lcFIgwR/3OHZwCQcDQd6e\ns87uokSdX2clKYvoryiGerTPwOeJbQI6kXMMEErS9SvO5rUZa+M+CR1IwKnJKjo0MMRQapKXsvzY\n3gGtlQWMHVRM1ebdzFqz3e6iRJUuiaGsooEhxnoXZsb0DuhE7zEAnN+ngNbJXl6N8yS05hiUVTQw\nxFjvwiy27z0Us+VzE3lWUr3WKT5G9y/k/fkbqNt7yO7iRE0iTzRQ1krsGsMG3yagd8bk+7SyCLly\nUAkH/EHenRufSehg0BA0aI9BWUIDQ4z17JCJ1yMxmZmklcW3ehVm0bswK26T0IHwOWkjQFlBA0OM\npSZ5KWuXHpMEtFYWRxo7qISlG3cxp3qH3UWxXCD87IlEHzZU1tBfkQ1itQT34coige9jaOjCfqEk\n9Gtx+Exof1AbAco6Ghhs0Lswi617DrKhLrrPJdbK4kjpKT4u7FfIe/PXs3N/fCWhA4H6HoNea9Vy\nGhhs0CtGz4D+trLQy1zvykEl7D8U5B9xdie0PxhaelnXSlJW0BrDBuUdMvFI9JfGOFxZaCvysN5F\nWfQqzOSV6fGVhP42x6DXWrWcBgYbtEr2UtYu+ndAa2XRuHhMQuuwobKSBgab9CnKYkFNdBPQWlk0\nbnS/wtCd0HGUhNZZScpKEf2KRGSUiCwTkSoReaCR91NE5PXw+9NFpLTBew+Gty8TkXOaccw/icju\nEzst5+tTFEpAr49iAlp7DI2rT0L/c/566vbFRxJaGwHKSk0GBhHxAk8B5wLlwFgRKT9qtxuB7caY\nrsDjwKPhz5YDY4CTgFHA0yLibeqYIlIBtGnhuTla76JsABbURG8443BloQnJ77hqcCgJ/U6cPBM6\nEM4naSNAWSGSHsMgoMoYs9IYcxAYD4w+ap/RwLjw6zeBs0REwtvHG2MOGGNWAVXh4x3zmOGg8Vvg\nvpadmrPVL8E9vyZ6eYZvKwsdXjhar8Is+hZlxU0SWnsMykqR1BiFQHWDv2vC2xrdxxjjB+qAnON8\n9njHvAuYaIzZcLxCicgtIlIpIpW1tbURnIazpCZ56d4+uglorSyO78rBJazYvJvKOFiO26/3MSgL\nOaopKSIFwGXAE03ta4x5xhhTYYypyMvLi37hoqBPURbzo5iA1sri+C7oW0BGii8uktABHTZUFook\nMKwDihv8XRTe1ug+IuIDsoCtx/nssbb3B7oCVSKyGkgTkaoIz8V1ehdmU7fvENXborMEd0B7DMeV\nluzjopMLeX/BBra5/JnQfp2VpCwUya9oJlAmIp1EJJlQMnniUftMBK4Nv74UmGxCzeCJwJjwrKVO\nQBkw41jHNMa8b4xpb4wpNcaUAnvDCe241KcodAf0/HXRSUD7dVZSk64a3JGD/iBvzqpuemcH00aA\nslKTgSGcM7gL+BBYAkwwxiwSkYdE5MLwbs8BOeHW/b3AA+HPLgImAIuBD4A7jTGBYx3T2lNzvm75\nGST7PCyIUgL628pCW5HH0r19BoNK2/LK9LUEg+5NQvt1VpKykC+SnYwxk4BJR237eYPX+wnlBhr7\n7MPAw5Ecs5F90iMpn1sl+zz07JDJvChNWdXKIjJXDSnhnvFzmVK1hdO7uTNfpT0GZSVtStqsT2EW\nC9ftjEprVROSkRnVqz05rZN5edoau4tywnTYUFlJA4PNehdlsfuAn1Vb91h+bK0sIpPi83L5wGI+\nWbKJ9TF6FrfV6lfS1WFDZQX9FdmsPgEdjTzDt5WFBoamXDmoBAOMn+HOqavaCFBW0sBgs6556bRK\n8kYlz6CVReSK26ZxRvd2vDazmoP+oN3FaTYdNlRW0sBgM5/XQ6/CTOZFYQlonZXUPFcP7UjtrgN8\nuGij3UVpNp1ooKykNYYD9CvOZuH6nRwKWNtS1cqieU4vy6NjThovTl1td1GaTWclKStpYHCAvsXZ\nHPQHWbphl6XH1cqieTwe4eohHZm5ejuL1++0uzjNosOGykoaGBygX3FoCe651dYu5qaVRfNdNqCY\n1CQPL01bbXdRmkWHDZWV9FfkAIXZrchNT7H8UZOakGy+rLQkLupfyDtz1lG31z0P8dFGgLKSBgYH\nEBH6FWcz1+LAoJXFibl6SCn7DwV5w0XrJwXC+SkdNlRW0MDgEP1LsllZu8fSVuq3lYVe5uYoL8hk\nYGkbXpy6xjXrJx1uBGjvUFlAawyH6Bt+1KeV9zNoj+HEXTO0lLXb9jJ56Wa7ixIRnWigrKSBwSH6\nFGchgqXDSVpZnLhRvdrTISuVF75eZXdRIqKNAGUlDQwOkZmaRJe8dEsDg1YWJy7J6+HqoR35qmor\nSzc6f+qqzkpSVtJfkYP0K85mXvUOyx71qT2Glhk7sITUJA9//2q13UVpUn0jQC+1soIGBgfpV5zN\n1j0HqdluzQqf2mNomTatk7mofxHvzFnn+Ed/BoJBfB5BRK+1ajkNDA5Sf6ObVfczBIJBvFpZtMgN\np5RywB/kNYevuuoPGm0AKMtoYHCQ7u0zSPF5mLvWmsCglUXLleVnMLwslxenrnb0qquBgNEhQ2UZ\nDQwOkuT10Kcoi9lrrVkaQysLa9xwSic27TzA+wvW212UY9JGgLKSBgaHGdCxLYvW17HvYKDFx9LK\nwhqnd8ujrF06z3yxyrKJAVYLBA0+r/5zVtbQX5LDDCxtw6GAseRGt0BQewxW8HiEm4d3ZsmGnXxV\ntdXu4jRKGwHKShoYHGZAxzYAzFrT8uGkUGWhl9gKo/sXkJeRwjNTVtpdlEbVz0pSygpaazhMdloy\nZe3Smbl6W4uPpZWFdVJ8Xq4bVsoXy2tZssF5N7xpj0FZSQODA1WUtmH2mu0tXsBNKwtrXTW4hLRk\nL39zYK9Bhw2VlTQwONCAjm3Zud/Pis27W3ScUEJSKwurZKclc3lFMRPnrmdDnTU3IVoloI0AZSEN\nDA40sDSUZ2jpcJL2GKx346mdMMCzU5y1uF6ox6D/nJU19JfkQCVt08hNT2lxAlrvY7Becds0Rvct\n4NXpa9nuoGUytBGgrKSBwYFEhIGlbSzqMeglttrtI7qw71CAF75ebXdRDtNhQ2UlrTUcakDHNtRs\n38fGuv0nfAydlRQdZfkZnHNSPn//ahW7D/jtLg6gPQZlLQ0MDjWwtC0AlWtOvNeglUX03DGiKzv3\n+3ll2hq7iwJoI0BZSwODQ5UXZNIqyUvl6hPPM+gUxujpW5zN8LJcnv1yFfsPtXz5kpbyB7QRoKwT\nUWAQkVEiskxEqkTkgUbeTxGR18PvTxeR0gbvPRjevkxEzmnqmCLySnj7QhF5XkSSWnaK7pTk9dCv\nOJsZq7TH4FR3jOhK7a4DTKistrsoOitJWarJX5KIeIGngHOBcmCsiJQftduNwHZjTFfgceDR8GfL\ngTHAScAo4GkR8TZxzFeAHkBvoBVwU4vO0MWGdslhycadJzz7RROS0TWkc1sGlbblqU+rbO81aCNA\nWSmSJsYgoMoYs9IYcxAYD4w+ap/RwLjw6zeBsyT0dJjRwHhjzAFjzCqgKny8Yx7TGDPJhAEzgKKW\nnaJ7ndI1B2Ng2soTW7hNZyVFl4jww7PL2LTzAONtfpCP3uCmrBRJrVEINOwr14S3NbqPMcYP1AE5\nx/lsk8cMDyFdDXzQWKFE5BYRqRSRytra2ghOw336FGXTOtnLV99sOaHPa0Iy+oZ1yWVwp7Y8/dk3\ntvYatMegrOTk5uTTwBfGmCmNvWmMecYYU2GMqcjLy4tx0WIjyethUKe2fH2CSz1rQjI2fnR2Nzbv\nOsCr0+3rNWgjQFkpksCwDihu8HdReFuj+4iID8gCth7ns8c9poj8AsgD7o3kJOLZKV1zWbllzwmt\nzaOzkmJjSOcchnbO4c+f29dr0B6DslIkgWEmUCYinUQkmVAyeeJR+0wErg2/vhSYHM4RTATGhGct\ndQLKCOUNjnlMEbkJOAcYa4xx7kN2Y2RYl1yAE+o16Lhz7Pzo7G7U7jrAi1NX2/L92ghQVmoyMIRz\nBncBHwJLgAnGmEUi8pCIXBje7TkgR0SqCLXyHwh/dhEwAVhMKFdwpzEmcKxjho/1FyAfmCoic0Xk\n5xadqyv1aJ9B29bJJ5Rn8GtlETODOrVlRPc8nvr0G+r2HYr594eGDZ08MqzcxBfJTsaYScCko7b9\nvMHr/cBlx/jsw8DDkRwzvD2iMiUKj0cY2jmHqd9sxRhDaLJXZAI6Kymm7junB+c9MYW/fP4N94/q\nEdPv1h6DspLWGi4wrGsOG+r2s2rLnmZ9zq8JyZgqL8jkon6FPP/lqhatcXUi/EGDV+9ZURbRwOAC\nh/MM3zQvzxDQyiLmfnR2N4yBP3y8PKbfq7OSlJU0MLhAaU4aBVmpfN3MPIPmGGKvuG0aVw/tyITK\nalZs2hWz79VZScpKGhhcQEQ4tSyXL1ds4VAg8olaAb2PwRZ3ntGV1ik+fvX+EkKT86JPcwzKShoY\nXOKsnvns3O9v1sN7tMdgj7atk7nnrDI+X17L5KWbY/KduvyJspL+klxieFkuyT4PHy+OvKLRWUn2\nuXZYKV3yWvO//1zMAX/0b3rTHoOyktYaLpGW7OPUrrl8tGRjxMMTOivJPkleDz+/4CRWb93LC1+t\njup3GWP0ZkZlKQ0MLjKyZz7V2/axYvPuJvcNBg1Bg1YWNjq9Wx4je7bjiU9WsHln9KavBoKhhoI2\nApRVNDC4yFk92wHw0eJNTe4bMFpZOMFPzyvnUMDwq/eXRO07/OHAoFOTlVU0MLhIfmYqfYuy+HhJ\nBIFBKwtHKM1tzR1ndGHivPV8GqVEtPYYlNU0MLjMyJ75zK3eweZdxx+a8Gtl4Ri3j+hCWbt0fvru\nQvYc8Ft+/MM9Bp1ooCyivySXGVmejzE02foMBLSycIoUn5dHLunD+rp9/O7fyyw/vvYYlNW01nCZ\nHu0zKMxuxUdNTFv1B0M3wmll4QwDOrbh6iEd+fvXq5mzdrulx66/1jrRQFlFA4PLiAhnl+fzxYra\n4y7vfDjHoJWFY/znOd3pkJnKj9+Yx96D1g0paY9BWU0DgwtdcnIRB/1B3pu3/pj7aI7BeTJSk/jd\n5X1ZtWUP//tP62Yp+QPaCFDW0sDgQr0KM+nRPoM3ZtUccx/tMTjTsC653HpaF16bsZYPF2205JiH\neww6A01ZRAODC4kIlw4oYl71jmOu4OnXysKx7j27G70KM3ngrflssuDGN52VpKymvySXuqh/IT6P\nHLPXEDickNRL7DTJPg9/HNOffYcC/L/X5jRrxdzGaI5BWU1rDZfKSU/hzB7teHv2ukYrFs0xOFuX\nvHR+c3FvZqzaxv/+c3GLjqWzkpTVNDC42GUVxWzZfYDPl9V+5z1NSDrfRf2LuHl4J16cuobXZ649\n4eNoj0FZTQODi43onkduejJvzKr+zntaWbjD/aN6MLwsl5++u5BZayJ/1kZDfp1ooCymgcHFkrwe\nLj65iE+WbGbt1r1HvKeVhTv4vB6eGNufguxW3DSu8oQeB/ptI0D/OStr6C/J5W48tRNej/CHT458\n+LxWFu6RnZbMuOsH4fN6uOrZ6d8J8k3RYUNlNa01XC4/M5Vrhnbk3TnrqNr8bWtTE5LuUprbmpdv\nHMzBQJArn53GxrrIp7HqfQzKahoY4sBtp3ehVZKXxz9ecXibVhbu0719BuOuH8SOvYcY88xUVm/Z\nE9HntBGgrKaBIQ7kpKdww6mdeH/+BhatrwM0x+BWfYuzGXfDIOr2HeLiP38d0YJ7OtFAWU0DQ5y4\naXhnMlN9PPbvUK6hftltrSzcZ0DHNrx1+zDSU3yM/ds0Pli44bj7ayNAWU0DQ5zIapXEbSO68MnS\nzbw2Y61WFi7XOS+dt+8YRo/2mdz28mwefHvBMR/yoxMNlNX0lxRHbhnemdO75fGzdxfyxYrQTW9a\nWbhXbnoK428Zwq2ndWb8zLWM+uMXTFu59Tv7aSNAWU1rjTji83p44sr+dM5rzavTQ3fSamXhbqlJ\nXh78fk9ev2UoAGOemcY1z89g1ppvcw8BfSiTspgGhjiTmZrEc9cOJDc9GdDKIl4M6tSWD+45jftG\ndWfhujou+fPXXPHXqYz7ejXV2/YB2ghQ1okoMIjIKBFZJiJVIvJAI++niMjr4feni0hpg/ceDG9f\nJiLnNHVMEekUPkZV+JjJLTvFxFPcNo3nrh3Ixf0LKchuZXdxlEVap/i4Y0RXptx3Bg+e24PNuw7w\ni4mLeOyj0IQDDQzKKmKMOf4OIl5gOXA2UAPMBMYaYxY32OcOoI8x5jYRGQNcZIy5QkTKgdeAQUAB\n8DHQLfyxRo8pIhOAt40x40XkL8A8Y8yfj1fGiooKU1lZ2dxzV8r1VtbuZvLSzdTuPsD95/TAo8FB\nNYOIzDLGVBy93RfBZwcBVcaYleEDjQdGAw3XCh4N/E/49ZvAkyIi4e3jjTEHgFUiUhU+Ho0dU0SW\nAGcCV4b3GRc+7nEDg1KJqnNeOp3z0u0uhoozkQwlFQINl++sCW9rdB9jjB+oA3KO89ljbc8BdoSP\ncazvAkBEbhGRShGprK397rLTSimlToxrk8/GmGeMMRXGmIq8vDy7i6OUUnEjksCwDihu8HdReFuj\n+4iID8gCth7ns8favhXIDh/jWN+llFIqiiIJDDOBsvBsoWRgDDDxqH0mAteGX18KTDahrPZEYEx4\n1lInoAyYcaxjhj/zafgYhI/5jxM/PaWUUs3VZPLZGOMXkbuADwEv8LwxZpGIPARUGmMmAs8BL4WT\ny9sIVfSE95tAKFHtB+40xgQAGjtm+CvvB8aLyK+AOeFjK6WUipEmp6u6gU5XVUqp5jvWdFXXJp+V\nUkpFhwYGpZRSR4iLoSQRqQXWnODHc4EtFhbHLRLxvBPxnCExz1vPOTIdjTHfme8fF4GhJUSksrEx\ntniXiOediOcMiXnees4to0NJSimljqCBQSml1BE0MMAzdhfAJol43ol4zpCY563n3AIJn2NQSil1\nJO0xKKWUOoIGBqWUUkdI6MDQ1CNL44GIFIvIpyKyWEQWicg94e1tReQjEVkR/m8bu8tqNRHxisgc\nEfln+O+4f2ysiGSLyJsislRElojI0Hi/1iLyo/Bve6GIvCYiqfF4rUXkeRHZLCILG2xr9NpKyJ/C\n5z9fRE5uznclbGAIP7L0KeBcoBwYG34UabzxAz82xpQDQ4A7w+f5APCJMaYM+CT8d7y5B1jS4O9H\ngceNMV2B7cCNtpQquv4IfGCM6QH0JXT+cXutRaQQuBuoMMb0IrQo5xji81r/HRh11LZjXdtzCa1m\nXQbcQjOfgpmwgYEGjyw1xhwE6h9ZGleMMRuMMbPDr3cRqigKCZ3ruPBu44D/sKeE0SEiRcB5wLPh\nv4XQY2PfDO8Sj+ecBZxGeEViY8xBY8wO4vxaE1olulX4OS5pwAbi8FobY74gtHp1Q8e6tqOBF03I\nNELPuekQ6XclcmCI5JGlcUVESoH+wHQg3xizIfzWRiDfpmJFyx+A+4Bg+O+IHxvrYp2AWuCF8BDa\nsyLSmji+1saYdcDvgLWEAkIdMIv4v9b1jnVtW1S/JXJgSCgikg68BfzQGLOz4XvhByTFzbxlETkf\n2GyMmWV3WWLMB5wM/NkY0x/Yw1HDRnF4rdsQah13AgqA1nx3uCUhWHltEzkwRPLI0rggIkmEgsIr\nxpi3w5s31Xctw//dbFf5ouAU4EIRWU1oiPBMQmPv8f7Y2BqgxhgzPfz3m4QCRTxf65HAKmNMrTHm\nEPA2oesf79e63rGubYvqt0QODJE8stT1wmPrzwFLjDGPNXir4eNY4+oRqsaYB40xRcaYUkLXdbIx\n5iri/LGxxpiNQLWIdA9vOovQ0xPj9loTGkIaIiJp4d96/TnH9bVu4FjXdiJwTXh20hCgrsGQU5MS\n+s5nEfk+obHo+seLPmxzkSwnIqcCU4AFfDve/l+E8gwTgBJCS5Zfbow5OrHleiIyAviJMeZ8EelM\nqAfRltBjY39gjDlgZ/msJiL9CCXck4GVwPWEGoBxe61F5JfAFYRm4M0BbiI0nh5X11pEXgNGEFpe\nexPwC+BdGrm24SD5JKFhtb3A9caYiB9zmdCBQSml1Hcl8lCSUkqpRmhgUEopdQQNDEoppY6ggUEp\npdQRNDAopZQ6ggYGpZRSR9DAoJRS6gj/H9Sw9SP1fuWvAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EvP6OQgXL7IF",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 281
        },
        "outputId": "6f224ad2-0f43-4acc-d7c3-faddb58daea2"
      },
      "source": [
        "'''\n",
        " 循环学习率,以恒定的频率循环两个边界之间的学习率\n",
        "'''\n",
        "scheduler = lr_scheduler.CyclicLR(optimizer,base_lr=0.05,max_lr=0.1,step_size_up=10,step_size_down=20)\n",
        "\n",
        "plt.figure()\n",
        "x = list(range(100))\n",
        "y = []\n",
        "for epoch in range(100):\n",
        "    optimizer.step()\n",
        "    scheduler.step()\n",
        "    lr = scheduler.get_lr()\n",
        "    y.append(scheduler.get_lr()[0])\n",
        "\n",
        "plt.plot(x, y)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f6b25c66a58>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO29eZAk153f9/nV1fd9ZM19X1kkARLD\nAcjFtZjpWlJ2ELaDlElvyHQEw9RGmCF5KYdMhRxUiI79g2FZlBzBkMQQd0Vr5SXX9FqGZYaAwQxx\ncEEcAwIEOBems+bGzFT23VXV3XU9/1GVPYVG93R1dx2ZWe8TMTHdWdlVrzKrvu+97/u930+UUmg0\nGo3GvwSa3QCNRqPR1Bct9BqNRuNztNBrNBqNz9FCr9FoND5HC71Go9H4nFCzG7CS4eFhtXfv3mY3\nQ6PRaDzF22+/PaGUGlntMdcJ/d69ezl37lyzm6HRaDSeQkSur/WYtm40Go3G52ih12g0Gp+jhV6j\n0Wh8jhZ6jUaj8Tla6DUajcbnVCX0IvIFEbksIuMi8p1VHn9SRH4jInkR+fKKx74uIlfK/75eq4Zr\nNBqNpjrWFXoRCQI/BL4ImMDXRMRccdoN4L8B/o8VfzsI/CPgUeAE8I9EZGDrzdZoNBpNtVQzoj8B\njCulEkqpLPBT4NnKE5RS15RS7wHFFX/7B8BppdSUUmoaOA18oQbtbjo3pzK8cP5us5vRMizmCvz0\nzRsUijqtdqP4v9+5xVQ62+xmtAx/+qur/OL9O3V57mqEfgdws+L3W+Vj1VDV34rIN0XknIics227\nyqduLj96JcHf/vO3seeXmt2UluD583f5zl+9z4sX7zW7KS3BrekMf/yz3/IvX7aa3ZSW4V+9YnH2\nUrIuz+2KxVil1I+UUseVUsdHRlbdwes6xpMplIIzWngawngyBcDpC/p6N4LK662LE9Wf+cUc9+aW\n2D/SVZfnr0bobwO7Kn7fWT5WDVv5W1dj2Vp4Golzvc9cvEe+sNIh1NQay04DcHUivXztNfUjUb7e\nB0a66/L81Qj9W8AhEdknIhHgq8BzVT7/80BcRAbKi7Dx8jFPM7eYIzm/REc4yKvjE6SX8s1uku+x\nkmk6wkGmMznevj7d7Ob4HstO0R4uycPz5/Vgpt4kJkqdadOEXimVB75FSaAvAn+plDovIt8TkS8B\niMhnReQW8BXgX4nI+fLfTgH/M6XO4i3ge+Vjnsbpfb92YjfZfJFXr3hjXcGrFIqKq5Np/ovP7CAS\nDOhZVAOwkili2/t4aGefvt4NwEqmCQWEPUOddXn+qjx6pdQvlFKHlVIHlFJ/Uj72XaXUc+Wf31JK\n7VRKdSmlhpRSsYq//VOl1MHyvz+ry7toMInyVParJ3bR3xnmBT3iqSu3pxfI5os8tLOfzx8c4gXt\nG9edxESaAyNdjJkG796c4d7cYrOb5GssO8XuwU7Cwfosm7piMdZrWHaKUEDYN9zFM0dHOXMpSU77\nxnXD8YgPjHYRN6PcmMpw+d58k1vlX2YXctjzSxwY6SYeiwJ6LareWHaK/XWybUAL/aawkmn2DJV6\n37hpMLuQ461rnnekXIsj9PuHuzl1bBSA03oWVTecGev+kW4OjXazd6hTC30dyReKXJvIcGC0PhE3\noIV+U1T2vk8eHqEtFND2TR2x7BSDXREGuiKM9rbz6d39vKCFp25YyxEgXYgI8ViU16wJ5hdzTW6Z\nP7k1vUC2UKzbQixood8w+UKRa5Pp5ZvSGQnxxKFhHW9cR6xkyS92iJtR3r89y4czC01slX+x7BTh\noLBrsLQwOGYa5AqKly7roIN6UO+IG9BCv2FuTi+QK6iPCM+YaXB7ZoELd+aa2DL/Ytmpj3wJxkwD\nQO+SrRNWMsWeoa7lhcHP7B5gqCui7Zs6YSXvz6DqhRb6DZJYXhi8LzwnjxmIoO2bOjCTyTKZzn5E\n6A+OdrN/pEtf7zqRmEizf/i+6AQDwqljBr+8lCSb10EHtcayUwx1RejvjNTtNbTQb5DlCJDh+8Iz\n3N3GI7sH9IinDiz7xSsWqsZMg9cTk8wuaN+4luQKRa5Ppj8ykIHS9Z5fyvPG1ckmtcy/rJyx1gMt\n9BvESqYZ7m6jrzP8kePxmMGFO3PcnMo0qWX+ZLljXfFFiJtR8kXFS5frkwSqVbk5lSlbkx+93o8f\nGqYjHNSzqDpg2em6RtyAFvoNU4q4+fhNGTNL8cbaN64tlp0iEgywc+CjOwY/vauf4e42LTw1pjLi\nppL2cJCnDo/ooIMaM5XOMrXCmqwHWug3yFrTrH3DXRwa7dbCU2OsZJq9w50EA/KR44GAMGaO8tLl\nJEv5QpNa5z+sihj6lYyZBnfnFnn/9myjm+Vb7u9Z0CN61zCVzjKdya25Oh6PGbx5bYqZjC7WUCsS\nE2v7l3EzSjpb4DVL+8a1ImGnStZkR/hjjz1zdJRgQPRgpobUO2ulgxb6DbBaxE0lcTNKoajqVjyg\n1cgVityYzKz5JfjcgSG6IkG9CF5DLDu95kBmoCvCib2D+nrXkLWsyVqjhX4DONPag2sIzyd39GH0\nat+4VlyfzJAvqjUXqtrDQZ46UvKNi7rE4JZRSjGeTK05kIGSfXP53jzXJtINbJl/sewU+4a7PmZN\n1hot9BvAstO0hQJs7+9Y9fGSb2zw8gc2izntG2+VtSJuKombUez5Jd69NdOoZvmWqXSW2YXcA6+3\ns1lNj+prQyMibkAL/Yawkuv3vmNmlIVcgb8en2hgy/yJI/T7htf+Ivz+kVFCAdHCUwOciJsHLQzu\nGuzk2LZefb1rwFK+wI2pDPuH6+vPgxb6DVHNxobP7R+ipy2k7ZsaYCXTGL1t9LR/fGHQoa8zzKP7\nB3nh/N0GtsyfrGdNOsRNg3PXp5hILTWiWb7lxmSGwgOsyVqihb5KlvIFbk4vrJuPIhIK8PTRUV68\neI+C9o23xIMibiqJm1EsW9c23SoJO/VAa9IhHjMoKjh7UQcdbAWrQRE3oIW+au73vuvflDHTYDKd\n5Z0burbpZlFKYSWrE/pT2jeuCZadrmph0NzWy47+Dp0qeos8aM9CrdFCXyXVLAw6PH1khHBQ9Bdh\nC0yksswt5qvK6Lejv4NP7OjV9s0WsewHR9w4iJSCDl69YpPJ5hvQMn9i2Smive10t4Xq/lpa6KvE\nmWY9aGHQobc9zOcODPPC+bt6u/gm2ehoJ25GeefmDMl5Xdt0MyzmCtycynCgis83lOybpXyRV6/o\noIPNYtnpuu+IddBCXyVWMsW2vna6qux9x0yDa5MZxpPaN94M1jqb01YyZhooBWe0b7wprk9mKKrq\nr/eJvYP0dYR10MEmUUqRqNKarAVa6KvEmkhv6KaMHSv5xtq+2RwJO01HOMi23vaqzj8a7WHXYIe2\nbzZJYgPWJEAoGODk0VHOXLpHvqBz1G8UO7XE/FJ11mQt0EJfBfd73+pvSrSvnYd26dqmm8XJEhqo\ncsegiBA3o/y1NUlqSfvGG6WaPQsriccMZjI5zl3XQQcbZbmqVJUzqK2ihb4K7Ply77vBmxI3DX57\nc4a7s9o33iibKcYwZhpk80Ve1rVNN4xlp9m+AWsS4IlDI0RCAZ7Xs6gNs5Hgjlqghb4Kxp2FwQ3u\nYIs7YX86R/2GWMwVuDW9sOGFquN7BhjoDHP6ghaejVKaQW3s893VFuKJg8M6R/0msOwUHeEg0Sqt\nya2ihb4K1ipntx4HR7vZO9Sp47s3yNWJNEptfLQTCgZ45qjB2UtJcto3rpr7exY27hePmQa3phe4\ndHe+Di3zL07ETbXW5FbRQl8FCTtFZ2Tjva+IEI9F+bU1wdyirm1aLVvJ0R2PGcwt5nnz6lStm+Vb\nkvNLpLOFTfnFJ48ZiKCjbzZIogF1YivRQl8FpRzd3YhsvPeNmwa5guIl7RtXjWWnENnYwqDDk4dG\naA8HdPTNBrCSm/eLR3raeGT3AC9ou6xqFrIFbs8saKF3G5ud1gJ8evcAQ10Rbd9sAMtOsaO/g45I\ncMN/2xEJ8vhBXdt0I2x1YXDMNDj/4Ry3ZxZq2SzfsmxNNiCZmYMW+nVwet/N5qMIBoRTxwxeupQk\nm9e+cTVsZmGwknjM4MPZRc5/OFfDVvkXy07TFQli9LZt6u/jsSgAp/UsqiqsTQZ3bAUt9OuQmNh6\nGFQ8ZjC/lOf1hK5tuh5KKRIPKGdXDSePjhIQvVmtWpyOdTPWJJQstkOj3Tq6rEq2Yk1ulqqEXkS+\nICKXRWRcRL6zyuNtIvKz8uNviMje8vGIiPyZiLwvIr8Vkadr2voGkNhkxE0lv3dwmI5wUPuYVXB3\nbpFMtrCljnWou43je3SO+mrZascKJfvm9cQUsxkddLAeCTu9aWtys6wr9CISBH4IfBEwga+JiLni\ntG8A00qpg8APgO+Xj/+3AEqpTwJjwP8qIp6aRTi9796hzX8R2sNBnjqsa5tWw/KOwS0uVMVjBpfu\nznNzKlOLZvmWTDZfk4XBeCxKoag4e1mP6tdjM5sBt0o1onsCGFdKJZRSWeCnwLMrznkW+En5558D\nJ6U0DzSBswBKqSQwAxyvRcMbhWWn2TnQQXt4a71vPGZwb26J927P1qhl/uT+wuDWR5iA3rW5Donl\n8oFbE55P7ejD6G3TYZbrUCyWrMlGZa10qEbodwA3K36/VT626jlKqTwwCwwBvwW+JCIhEdkHPALs\nWvkCIvJNETknIuds211hiFYytW5ptWp45ugowYDoXZvrYNkpetpDjPRsbmHQYc9QF0eMHh3ttA7L\n5QO3mHMlUA46ePkDm8VcoRZN8yV35hZZyBW2fL03Sr1tlD+l1DGcA/4Z8BrwsU+BUupHSqnjSqnj\nIyMjdW5S9RSLisTE1iJAHPo7I5zYO6hHPOuw1YXBSuIxg7euTTGVztagZf7EstOIwJ6hzi0/VzwW\nJZMt8Jqlc9SvhbNnoZERN1Cd0N/mo6PwneVjq54jIiGgD5hUSuWVUn+slHpYKfUs0A98sPVmN4YP\nZxdYzBVr5qeNmQZXkimuTqRr8nx+xEpufWHQIW5GS7VNL+kc9Wth2Sl2DXRu2ZoEeGz/IN1tIT2L\negD36yy4z7p5CzgkIvtEJAJ8FXhuxTnPAV8v//xl4KxSSolIp4h0AYjIGJBXSl2oUdvrzv2t+LW5\nKWPLtU21fbMaqaU8d+cWa9axfmJHL9v62nX0zQOoRcSNQ1soyNNHRjh9IamDDtYgYadL1mT31qzJ\njbKu0Jc9928BzwMXgb9USp0Xke+JyJfKp/0YGBKRceDbgBOCOQr8RkQuAv8j8Ldq/QbqyUarHK3H\nrsFOzG29esSzBle3kONmNZzapq9csVnIat94JaWFwdpGgMRjUSZSS7xzc6Zmz+knnIibWliTG6Eq\nj14p9Qul1GGl1AGl1J+Uj31XKfVc+edFpdRXlFIHlVInlFKJ8vFrSqkjSqljSqlTSqnr9Xsrtcey\nU/R1hBnqitTsOcdMg3PXp5lILdXsOf3C/YXB2k1r42aUxVyRX41r33glt2cWWMoXa1r84ukjI4SD\noveMrEEzQitB74x9IFayFAZVy943HnNqm+pR/UosO0UwIOwerJ3QP7p/kJ72kLZvVuH+VvzaXe/e\n9jCP7R/itA46+BjziznuzS01PLQStNA/kHr0vua2Xnb0d2j7ZhUsO8XuwU4iodp9LMPBAM8cHeXM\npSQF7Rt/hPt1Fmr7GY/HoiQm0oyXI0w0JbaSfnuraKFfg/nFHMn5pZrflPu+8QRpXdv0I9RyYbCS\nuBllKp3lbV3b9CMk6mBNAowdKwUdaPvmozh5s2ppTVaLFvo1qHXETSXxWKm26atX3LU5rJkUiorE\nRLouo52njowQCeoc9SspzVhra00CRPvaeWhnn94zsgIrma65NVktWujXoNYRN5Wc2DtIX0dYZ1es\n4Pb0Atl87fYsVNLdFuLzB4c4fVHnqK/EKahTD8ZMg3dvzpCcW6zL83sRy06xp8bWZLVooV8Dy04R\nCgi7B7e+Y3AloWCAk0dHOXspSV7XNgUqFgbrtFAVN6Ncn8zwwT3tGwPMLuSw55dqsut7NZZz1Oug\ng2VKu74bP5oHLfRrYiXT7B7qJByszyWKxwxmMjneuqZ9Y9h6laP1OGWOIqI3qzkkapQ8bi0OjXaz\nd6hTBx2UKRQV1yYyTVmIBS30a5KYqG+86xOHRoiEAnrBqoxlpxnsijBQ44VBh9Gedh7e1a/tsjKJ\nOkXcODhBB6+NTzK/qHPU35rOkC3Ux5qsBi30q5AvFOve+3a1hXji4DAvnNe+MdxfGKwncTPKe7dm\nuTOra5vW05p0iMeiZAtFXv5ABx00K8eNgxb6Vbg1vVDufet7U8ZMg9szC1y8M1/X1/ECtd6KvxpO\nrqEX9ai+tDBYR2sS4DO7Bxjqimj7hvsFdRqdtdJBC/0q1DPippKTxwxEdLzxTCbLRCpbd6E/ONrN\n/pEubd9Q34gbh2BAOHmsFHSQa/GgA8tOMVRHa3I9tNCvwrLQ17n3Help45HdAy0/4rGWqxzVf1ob\nN6P82ppkdqF1feNcocj1yXTdIm4qiZtR5hfzvJGYqvtruZlmRtyAFvpVsZJphrsj9HWG6/5aY6bB\n+Q/nuDXdurVN6x1xU8mYaZAvKl663Lo56m9OZcgVVN2tSYDHDw3TEQ62/Ky1ETOoB6GFfhUSE6mG\neWnL8cYtPKpP2GkiwQA7Bzrq/lqf3tXPSE9bS9s3taoTWw3t4SBPHh7m9IXWDTqYyWSZSmf1iN5t\nWHa67v68w77hLg6Odre00Ft2ir3DnYTquDDoUKptOsrLl22W8q2Zo345HXSDRphjZpQ7s4v87vZc\nQ17PbdSqLu9W0EK/gql0qfdtxLTWIW4avHF1iplMa9Y2bXSO7rgZJbWU59fWZMNe001Ydorh7raG\nWJMAJ4+OEmjhoAMn4kZbNy4i0UC/2CEei1IoqpasbZorFLkxmWnotPZzB4boigRb1r6x6pQldC0G\nuiKc2DfYsknOLDtVtibrt2dhPbTQr6CRC4MOn9rRx2hPW0vaN9cnM+SLqqHXuz0c5KkjI7x44V5L\n1jYtRYA0dnQ5Zka5fG+e65Pphr6uG3CsyWCgseUDK9FCv4KEnSYSCrCjAQuDDoFAabv4yx/YLOZa\nyzduxgwKSvZNcn6J395qrdqmU+ksM5lcQ0f0ULInoTWDDhJNjrgBLfQfw7JT7B/uanjvG49FyWQL\nvGa1Vm3TRsbQV/L7R0YJBaTl7JtGbQZcya7BTo5t6205+yabL3J9qnnJzBy00K+gWfGuj+0fpLst\n1HJfBMtOYfS20dPemIVBh77OMI/uH2y5EaaVbGzETSVjpsG561NMpJYa/trN4sZUmkJRNS3HjYMW\n+gqW8gVuTDV2YdChLRTk6SMjvHjxXkvVNi3NoJoz2ombUcaTqeVRbitg2SnaQgG29zfOmnSImwZF\nBWcvtk7QwXiTc9w4aKGv4MZkptT7NmmaFY9FmUhlefdma+SoV0phJVNNG+2cakHf2LLT7GuCNQkQ\n297Ljv6OlrLL6l1Qp1q00FdgNbFKO8DTR0YIB4XnW8S+mUxnmVvMN+167+jv4BM7eltK6BN2quH+\nvIOTo/7VKzaZbL4pbWg0CTvdFGtyJVroK2h279vbHuax/UMts13c8YubuVAVN6P85sY0yXn/1zZ1\nrMkDw80bXcZNg6V8kVc+aI2gg0ZvBlwLLfQVWHaKbX3tdLWFmtaGuGlwdSLdEr6xVecqR9UwZhoo\nBWdawDe+PpmhqJp7vT+7b5De9lBLzKKUUlro3UizM8zBfd+4Fewby07REQ6yrbe9aW04Gu1h12BH\nSwiPG2ZQ4WCAk8cMzly6R97nOert1BLzi/mG71lYDS30ZZRSJJLNzRkNsK2vg4d29rXEgpVlp9g3\n3EWgiTsGRYSxY1F+dWWC1JK/fWNnlrividYNlGatM5kcb13zd9DBclUpPaJ3D/b8EvNLzVsYrGTM\nNPjtzRnuzfnbN7aauDBYSTxmkC0UecXntU0tO832JluTAE8eHiESCvh+FtWszWmroYW+TLMjbipp\nhRz1i7kCt6YXXDGtPb5ngIHOsK+vNzQ34qaSrrYQjx8c5oULd30ddJCw0023Jh2qEnoR+YKIXBaR\ncRH5ziqPt4nIz8qPvyEie8vHwyLyExF5X0Quisg/qG3za0ezq7RXcmi0m71Dnb4WnmuTaZRyR8ca\nCgZ45qjBmYv3fFvbtLQw2Pw1KIe4aXBreoFLd+eb3ZS64ZQPbKY16bCu0ItIEPgh8EXABL4mIuaK\n074BTCulDgI/AL5fPv4VoE0p9UngEeBvO52A27DsFJ2RIFEX9L5OvPFr1gTzi/6sbeqGHN2VxGMG\nc4t53rzqz9qmyfklUkvuWBgEOHnMQARfp/xwS8QNVDeiPwGMK6USSqks8FPg2RXnPAv8pPzzz4GT\nIiKAArpEJAR0AFnAlWVmLDvN/pEuSs1uPvFYlFxB8dJlf/rGblkYdHjy0AjtYf/6xk7EjRsWBgFG\netr4zO4B3xYjWcgWuD2z0PTgDodqhH4HcLPi91vlY6ueo5TKA7PAECXRTwN3gBvAP1FKfWzIJCLf\nFJFzInLOtpsjbFayeTlXVuMzuwcY6or4NvrGslPs6O+gIxJsdlMA6IgEefzgCC+c96dv3OzNgKsR\nNw3OfzjHrelMs5tSc65OlKxJt3Ss9V6MPQEUgO3APuDvicj+lScppX6klDqulDo+MjJS5yZ9nIVs\ngQ9nF5pa03ElwYBw8tgoL11Kks37zzdONLAub7XEYwYfzi5y/kNXTjq3hGWn6XKJNekwVt4z8qIP\nBzOJieZlCV2NaoT+NrCr4ved5WOrnlO2afqASeC/Av6jUiqnlEoCfw0c32qja43T+7rFT3OIm1Hm\nl/K8nvBXbdP7OwbdM7qEytqm/hMeJ5TVLdYklEa7B0e7/Xm9k2lE3GNNViP0bwGHRGSfiESArwLP\nrTjnOeDr5Z+/DJxVpfnvDeAZABHpAh4DLtWi4bXETRE3lTx+aJiOcNB3vvHduUUy2YLrOtah7jaO\n7xnkhfP+843dUOVoNcZMgzeuTjGb8VfQgdusyXWFvuy5fwt4HrgI/KVS6ryIfE9EvlQ+7cfAkIiM\nA98GnBDMHwLdInKeUofxZ0qp92r9JraKZacQgb1D7hL69nCQJw8Pc9pntU3v7xh01/WGkn1z6e48\nN6f84xtnsvnSwqBLRpeVxE2DQlFx9rK/BjPNqMv7IKry6JVSv1BKHVZKHVBK/Un52HeVUs+Vf15U\nSn1FKXVQKXVCKZUoH0+Vj8eUUqZS6n+p31vZPJadZudAB+1hd/S+lcTNKHfnFnn/9myzm1IznBmU\nW/zLShzf2E92QsIFyePW4qGd/Yz2tPkqzLJYVOUZlHs6Vr0zlvKOQReKDsAzR0cJBsRX9k3CTtHT\nFmKkp63ZTfkYe4a6OGL0cNpHYX+JCXftWagkEBBOmQYvf2CzmCs0uzk14e7cIgs5d1mTLS/093tf\n99yUSga6Inx2r7/ijS07zX6XLQxWEo8ZvHl1iul0ttlNqQlWMkVAYM9QZ7Obsipx0yCTLfCa5Y8c\n9ctrfi7SlJYX+jsu7H1XEjejfHAvxbXyyMzruDHippK4GaWo4Mwlf+Sot+wUuwY7XWlNAnzuwBDd\nbf7JUb+cDtpFwR0tL/T3c3S756as5L5v7P1RfWopz53ZRVd3rJ/Y0cu2vnbfRN+4KcfNarSFgjx1\nZITTF+5R8EHQgWWn6WkPMdLtHmtSC73trq3hq7FrsJNj2/xR2/TqcpZQ93as92ubTnjeNy5ZkylX\nRtxUEjcNJlJZ3r3p/Rz1TsSNm6xJLfR2it72EMPdkWY35YHETYO3r08zkVpqdlO2hBv9y9UYMw0W\ncgV+dcXbvvHtmQWW8kVXRtxU8vSRUUIB8UW0kxutyZYXemcrvpt639UYMw2KCs56vLZpwk4RDAi7\nXbow6PDoviF62kOet8vcHHFTSV9HmM8dGOKF8/c8nWsotZTn3tyS6653ywu9m1KJPojY9l529Hd4\nXngsO83uwU7aQu5cGHSIhAI8c3SUMxeTnvaNvbAG5RA3Da5OpJdnfV4k4dIZa0sL/fxizpW972pU\n+saZrHdrm7pxWrsWY6bBZDrLb2541ze27BT9nWEGu9xtTQKc8sFmteXNgC6KuIEWF3pnx6Abt+Kv\nRtw0WMoXeeUDb/rGhaIiMZF29cJ3JU8dHiESDHg6+sYqL8S63ZoE2NbXwad29nl6l6yVTJesyUF3\naUpLC71XFgYdPrtvkL4O79Y2vT29QDZf9MyIvqe97Btf8K5v7PbQypWMHTN49+YMybnFZjdlU1h2\nit2DnURC7pJWd7WmwSTsNKGAuHbH4ErCwQAnj45y5tI98h6sbWpNeKtjhdIu2euTGa4kvecbzy3m\nsOeXXB9xU0k8FgXg9EVvDmbcluPGoaWF3rJT7B7qJBz0zmWIxwxmMjneuuY93/j+wqB3hGfsWNk3\n9qB9s5zMzEPX+7DRzZ6hTk/aN4Wi4uqEO2dQ3lG4OuCViJtKnjg0QiQU8GT0jWWnGeyKMOCBhUGH\n0d52Ht7V78kFQi9F3DiICGPHDH5tTTK/6K0c9bemM2QLRVdqSssKfb5Q5NpExjMLsQ5dbSGeOFjK\nUe8139jywA7N1YjHDN67Ncud2YVmN2VDWHaKUEDYNegNa9IhHouSLRR5+YPm1I/eLG6sy+vQskJ/\na3rBtb3vesRjBremF7h4Z77ZTdkQbk4H/SDiZsk39lptU8tOscdj1iTAI3sGGOyKeM6+cQrquPEz\n7q1PQA3xWsRNJc8cNRDBU9E3s5kcE6msqzL6VcvB0W72D3d5zr7xWsSNQzAgnDw6yi8vJ8nmvRN0\nYNkp11qTWuhdOM1aj5GeNh7Z7a0c9V6MuKlkLGbwemKSOY/4xvlCkeuTaU9F3FQSj0WZX8zzxtXJ\nZjelatwacQMtLPQJO81wd4T+Tvf1vtUwZhqc/3CO2zPe8I29GHFTSdw0yBUUL132hm98c3qBXEF5\n9no/fnCY9nDAU7NWNwd3tKzQu61470ZZjjf2SNifZaeJBAPsHOhodlM2xcO7BhjubvNMmKUXI24q\n6YgEefLQiGeSnE2ns0yms2p3+VgAAB8USURBVFro3Ybl4mlWNewb7uLQaLdnfGNnYTDksYVBh2BA\nGDNHeemyzVLe/TnqvVBnYT3isSh35xZ5//Zss5uyLokJ90bcQIsK/VQ6y5SLe99qGTMN3rg6xWzG\n/b6xm6e11TJmGqSW8ryemGp2U9bFslMMd7fR1xFudlM2zTNHRwl4JOjAzRE30KJC79ZUohslHotS\nKCrOXnb3FyFXKHJjMuPJiJtKPn9gmM5I0BP2jZsXBqtlsCvCZ/cOeiLM0ppIudqabEmh93JoZSWf\n2tGH0dvm+i/CjakM+aJ3FwYd2sNBni7XNi26PEe9Zac8G3FTSTwW5fK9ea5PppvdlAdiJdPsHXav\nNenOVtUZy04TCQXY4dLet1oCAeHUMYOXP7BdXdvU6xE3lYyZBsn5Jd5zsW88lc4yncn54nrHyznq\n3W7fuH0zYEsKvVMsORhwf47u9YjHomSyBV6z3Juj3vJY3v8H8cwRg2BAXG3feHmPyEp2DXZyNNrj\n6llrNl/k+lRGC73bsOy0L0QH4LH9g3S3hVw94rHsFKM9bfS0e3dh0KGvM8yj+wbdfb19NIOC0qj+\n3PUpJlNLzW7KqtyYSlMoKldrSssJ/VK+wA2X974boS103zd2a21TP0TcVBI3Da4kU8uL+m7DslO0\nhQJs7/e2NekQj0UpKjhzKdnspqzKuMsjbqAFhf7GZIaCDxYGKxkzDSZSWd696b4c9UqpUgSIxyNu\nKhlzNqu5dFSfsNPs84k1CRDb3sv2vnbX2jduj6GHFhR6v0TcVPL7R0cJB8WVm6cm01lmF/yxMOiw\no7+D2PZe1wq9XyJuHESEMdPgV+M2C1n3BR1YyTRGr7utyRYUev8sDDr0tod5bP+QK7eL+80vdoib\nUd6+MY097y7f2G/WpEM8FmUxV+SVK+7LNeQFa7IqoReRL4jIZREZF5HvrPJ4m4j8rPz4GyKyt3z8\nD0Xk3Yp/RRF5uLZvYWNYyRTR3na62kLNbEbNiZsGVyfSyzMWt+DHjhVKNQGUgrOX3DWqvz6Zoaj8\nEXFTyYl9g/S2uy/oQClVzpvl7uu9rtCLSBD4IfBFwAS+JiLmitO+AUwrpQ4CPwC+D6CU+ndKqYeV\nUg8Dfwu4qpR6t5ZvYKOUprXuvimb4VQ53tht9o1lp2gPB9je54+FQYej0R52DnS4zjf26wwqHAzw\nzNFRzly8R77gnhz1dmqJ+cW86693NSP6E8C4UiqhlMoCPwWeXXHOs8BPyj//HDgpIitXgr5W/tum\n4SwM7h92903ZDNv6OvjUzj7XCU/CTrFvuJuATxYGHUSEuBnl1fEJ0kv5ZjdnmcREaQa1z4MlG9cj\nHosynclx7rp7gg4SyzNWd2tKNUK/A7hZ8fut8rFVz1FK5YFZYGjFOf8l8BervYCIfFNEzonIOduu\nnwdnzy8xv5TnoI8WqiqJmwbv3pzh3txis5uyjGWn/Xu9YwbZfJFXXeQbW8kU2/v8Z00CPHV4hEjI\nXTnqHavU7Z/xhizGisijQEYp9bvVHldK/UgpdVwpdXxkZKRu7Rj3YcRNJU6O+hcvuuOLsJgrcHM6\n4zu/2OH4ngH6O8OumkX5LeKmkq62EL93YIgXLtx1TdCBlUzTEQ6yrbe92U15INUI/W1gV8XvO8vH\nVj1HREJAH1BZA+yrrDGabyR+XRh0ODTazZ6hTtcIz7XJNEq5f1q7WULBACePGpy5lCTnAt+4tDCY\nZr8PbRuHeCzKzakFLt2db3ZTgFLHum+4y/XWZDVC/xZwSET2iUiEkmg/t+Kc54Cvl3/+MnBWlbtc\nEQkAf5Mm+/NQmtZ2RoJEXd77bpaSb2zwmjXBvAtqm97P0e1f4RkzDWYXcrx1tfk56pPzS6SW8r4d\n0QOcPDaKuChHvVdmUOsKfdlz/xbwPHAR+Eul1HkR+Z6IfKl82o+BIREZB74NVIZgPgncVEolatv0\njZOYKOW4cXvvuxXGzKhraps6KQL8uPjt8OThYdpCAVdEO/lxM+BKRnva+fSufl640Pykcou5Ardn\nFjwxkKnKo1dK/UIpdVgpdUAp9SflY99VSj1X/nlRKfUVpdRBpdSJSlFXSr2klHqsPs3fGFbS/Rsb\ntsojewYY7Iq4YsRj2Sl29HfQEQk2uyl1ozMS4olDpVxDzfaNHWvS75/xeCzK727PcXtmoantuDpR\nsia9cL1bZmfsQtbpfd1/U7ZCMCCcOjbKLy8lyeab6xtbdtoT09qtEjcNbs8scP7Duaa2w0qm6IoE\nMXrbmtqOejNW3jPyYpMHM16aQbWM0DuJh7xwU7bKmBllfinP64nJ9U+uE86OQS9Ma7fKyWOl2qbN\ntm8cv/jjW1j8xYGRbg6MdDXdvrGSaUS8sWehZYTe7xE3lTxxaJiOcLCp9s3duUUy2YJvI24qGepu\n4/ie5ueoT/g84qaSeCzKG4kpZjPNCzqw7BTb+7xhTbaO0CdTnul9t0p7OMgTh4ab6hu3QsRNJWOm\nwcU7c9ycyjTl9TPZfEtYkw5jpkG+qDh7uXmdq1cibqCFhD4xkWbnQAftYff3vrUgHotyd26R95tU\n29Sxyg62kPBA8+ybq+XUB14Rnq3y8M5+RnramjaLKhbLdRY8MpBpGaFvhYibSk4eLfvGTdo8ZSVT\n9LSFGOnx98Kgw97hLg4b3Zxukm/cKhE3DoGAcOqYwcuXbRZzjc9Rf3dukYVcwTPXuyWEvlhUJCZa\nS+gHuiJ8dm/zfGPLTrO/BRYGK4mbUd68OsV0Otvw17aSKQICe4Y6G/7azSIeM0hnC/zaanzQgZci\nbqBFhP7D2QUWc8WWWIitJB6LcvnePNfK0/pGYtkpDrTAekgl8ZjRtNqmlp1i50Bny1iTAJ8/MERX\nJNiU6Jv76aC98RlvCaFvtWmtQ7zsGzd6VJ9eynNndrFl/GKHT+7oI9rb3hT7xvKQX1wr2kJBnj4y\nyukLSYrFxgYdWHbaU9ZkSwh9wmPTrFqxa7CTY9saX9t0eWGwxYTHqW36ygcTDfWNi0XF1RazJh3i\nMYOJ1BLv3Jxp6OsmJlKesiZbQugtO0Vve4jh7kizm9JwxkyDc9enmEw1rrap1/zLWhKPGSzkCvzq\nykTDXtOxJlttBgXw9JFRQgFp+GDGSnprBtUaQp9Mt8SOwdWIm433ja1kimBA2N1CC4MOj+4boqct\n1FDfuFWtSYC+jjCP7R9q6PVOLeW5O7foqevdGkJvp3ydQfFBxLb3sqO/sbVNLTvNroEO2kKtszDo\nEAkF+P2jo5y5mKTQIN/YWRhstWADh3jMIGGnGS9fh3pz3wr2zvX2vdDPL+ZIzi+17JfA8Y1fvWKT\nyTamtqllp1oi9cFaxGMGk+ksbzeotqljTQ51tZ41CXDqmLNZrTGjesea9NJn3PdC7xTvdXtNx3oS\nNw2W8kVebYBvXCgqrk74t05sNTx1eIRwUBoWfZMo1+VtRWsSYHt/B5/c0dcwnz5hpwkGxFN7Fnwv\n9K28MOjw2X2D9HU0prbphzMLLOWLnprW1pqe9jCfPzDMCw3KNVTKEtq6n28oDWbeuTFDcm6x7q9l\n2Sl2D3Z6yppsCaEPeaz3rTXhYIBnjo5y9tI98nWuber3AuzVMmYaXJ/McKXOvvFc2ZpsxYibSsZi\n5Rz1F+sfdOC1iBtoBaFPptk92Ek46Pu3+kDipsF0Jse5OvvG9xcGW1x4nCRn5+tr3zjWZKukJ16L\nI0YPuwc76+7TO9ak1z7fvle/Vl8YdHjy8AiRUKDuPqZlpxnoDDPYoguDDkZvOw/t6q//9Xa24rf4\niF5EiJsGr41PklqqX9DBrekM2YL3rElfC32+UOT6ZIYDo966KfWgqy3E4weHeeHC3br6xgntFy8T\nNw1+e2uWu7P1840TEyVrcvdg61qTDmOmQbZQ5OXLdt1eI+HRPQu+Fvpb0wvl3tdbN6VexE2Dm1ML\nXLo7X7fXKOVc0dcb4A/KvvHpi/Ub1VvJNHuGtDUJ8MieAQa7InW1b7wa3OHrT4dXb0q9OHnMQKR+\nSc5mMzkmUkt6BlXmwEg3+4a76urT64ib+4SWgw6S5OoUdGDZKQa7Igx4zJpsEaHXwgMw0tPGZ3YP\n1G3EY7VQAfZqcHzj1xOTzC3WvrZpvlDk2mS65f35SuKmwfxinjcSU3V5fi9G3IDfhT6ZZqgrQn+n\nt3rfehI3DX53e47bMws1f24dcfNxxkyDXEHxUh1845vTC+QKquUjbip54tAI7eFA/QYzHk2n4m+h\n19Paj+GE/b1YB/vGstOEg8KugY6aP7dX+fTuAYa7I3Wxb3TEzcfpiAR54tAIp+uwWW0mk2UynfWk\nNelroU9MpD15U+rJ/pFuDo5212XEk7BT7B3qIqQXBpcJlmubvnTZZilf2xz1TgH2Ax4cYdaTuGlw\nZ3aR392eq+nzejlLqG+/kVPpLFPprCdvSr2JmwZvJKaYzdTWN9YzqNUZMw1SS3ler7FvbCXTDHe3\n0dcZrunzep2TxwwCQs1zDXk5uMO3Qt+qVaWqYcw0yBcVv7xcu+3iOb1nYU1+7+AwnZFgXYTHiwuD\n9WawK8LxvYO8UGN70rJTRIIBdnrQmvSt0N9PJaq/CCt5aGc/oz1tNbVvbkxlyBeVJxeq6k17OMiT\nZd+4lrVN9a7vtYmbBpfuznNjMlOz53T2LHjRmvRei6vEstPl3lfvGFxJICCcMg1evmzXrLapXhh8\nMPGYwb25Jd67PVuT55tKZ5nO5PSIfg3iZhSobY56L+/6rkroReQLInJZRMZF5DurPN4mIj8rP/6G\niOyteOxTIvJrETkvIu+LSHvtmr82CTvFvuEugoHWzNG9HnHTIJ0t8GtrsibPlygXBNczqNV55ugo\nwUDtctQvW5O6Y12V3UOdHI321My+yRWK3JjyrjW5rtCLSBD4IfBFwAS+JiLmitO+AUwrpQ4CPwC+\nX/7bEPDnwB8ppWLA00Dtd46sgmXriJsH8bkDQ3S3hWr2RbCSKUZ72uht1wuDq9HfGeHE3sGa1QRw\nrMmDHh1hNoK4aXDu2hRT6eyWn+v6ZMma9POI/gQwrpRKKKWywE+BZ1ec8yzwk/LPPwdOSqncTRx4\nTyn1WwCl1KRSqrYxZquwlC+Uel+P3pRG0BYK8tSR2vnGOuJmfeIxgyvJFFfLs5+tYNlp2kIBtvd7\nb2GwUYyZUYoKztQg15CXI26gOqHfAdys+P1W+diq5yil8sAsMAQcBpSIPC8ivxGRv7/aC4jIN0Xk\nnIics+2t7yC8MZmhUFTaRliHuGkwkVrinZszW3oepRSWndbXex2czWq1sG+spLYm1+MTO3rZ3tde\nk1mrI/T7PPoZr/dibAh4HPjD8v//uYicXHmSUupHSqnjSqnjIyMjW35Rr/e+jeL3j46Wa5tu7Ysw\nmc4yu5DT13sddg50Ym7rrYl9o2dQ6yMijJkGr16xWchuzUiwkmlPW5PVCP1tYFfF7zvLx1Y9p+zL\n9wGTlEb/ryilJpRSGeAXwGe22uj1cHaw6dCzB9PbHuax/UNbjkzQETfVE48ZvH1jmonU0qaf4741\n6c3RZSMZM6Ms5oq8emVrToHXO9ZqhP4t4JCI7BORCPBV4LkV5zwHfL3885eBs6qUaOJ54JMi0lnu\nAJ4CLtSm6Wtj2Smive10t4Xq/VKeJ24aJOw041uobepE3GjhWZ+4GUWpreUaujGZoah0x1oNj+4f\npKd9a0EHSqlSaKWHgzvWFfqy5/4tSqJ9EfhLpdR5EfmeiHypfNqPgSERGQe+DXyn/LfTwD+l1Fm8\nC/xGKfX/1f5tfBQdcVM9p5zaplsY1VvJFO3hANv79MLgehzb1sOO/o4t2WXamqyecDlH/ZmL98hv\nMkf9RCrL3GLe09e7Ko9eKfULpdRhpdQBpdSflI99Vyn1XPnnRaXUV5RSB5VSJ5RSiYq//XOlVEwp\n9Qml1KqLsbVEKUUi6e1pViPZ1tfBp3b2bVl49g93E9ALg+siIsRjBq+OT5DeZG1Tx5rcp9MTV0Xc\njDKdyfH29elN/b0fOlbf7Yy155eYX/J279to4qbBOzdmSM5trrapjrjZGHEzSja/ed/YSqbY1tdO\nl7Ymq+KpIyNEgoFN2zeWDzan+U7o7y/EauGplrHl7eIb/yIs5grcnNZ7FjbCZ/cO0N8Z3nT0jdcX\nBhtNd1uIzx8c2nSOeiuZpiMcZFtvQzb11wUfCr33p1mN5rDRzZ6hzk3ZN9cnMyi9MLghnNqmZy4l\nN+wblxYGvVnOrpnEzSg3pjJcvje/4b9NTJT2LHjZmvSl0HdGgkQ93Ps2Gqe26a+tSeY3WNtU1+Xd\nHHEzyuxCjjevbSxH/bI1qTvWDXHKHEUETm9iFmXZKc9fbx8Kfckv9nLv2wzGzCjZQpGXP9iYb7xc\nJ1anJ94QTx4epi0U2LB9M65nrJtitKedh3f18/wGo8sWcwVuTS94fiDjP6HXETeb4pE9Awx2RTYs\nPJadYkd/Bx2RYJ1a5k86IyGeODS8Yd9Yr0FtnrgZ5Xe35/hwZqHqv7k6kS5Zkx7XFF8J/UK2wIez\nC56/Kc2gVNt0lF9eTpLNV+8b64ibzRM3o9yeWeDCneprm1pJbU1ulnistGfkxQ0kOfPLmp+vhN7p\nfbXwbI4xM8r8Yp43rlaXo355x6DHvwTN4pljJd94I7OoxESaAyPdlJLDajbCgZFu9o90bex622lE\nvL9nwVdC75fet1k8cWiYjnCw6i/Cvbkl0tmC5xeqmsVwdxvH9wxsKNqpZE16W3SaSdyM8npiktmF\n6oIO/GJN+k7o/dD7Nov2cHBDvrGOuNk6cTPKhTtz3Jxav7bpQrbA7RltTW6FeMwgX1S8dDlZ1fl+\nqcvrM6FPs3Ogg/awt3vfZhKPRbk7t8j7VdQ21TOorXM/R/36o/rEhFPwXl/vzfLwzn5Getqq2hxY\nLCqspD/2LPhK6LVfvHVOHh0lINUJj5VM0d0WYrSnrQEt8yd7h7s4bHRXd73LETc6Yd/mCQSEU8cM\nXrqUZCn/4Bz1d+cWWcgVfKEpvhH6YtHZMej9m9JMBroifLbK2qalhcEuvTC4ReJmlDevTTG9Tm3T\nRNma3DukhX4rxGMG6WyB16wHBx0knI7VB5riG6G/46Pet9nEY1Eu35vn2jq1TfWehdowZhoUioqz\nlx7sG1t2ml0Dndqa3CKfPzBEVyS47izqfjIz73esvhH6ifklRnvadGhlDYib68cbp5fyfDi7qK93\nDfjkjj6ive3rC08ypa93DWgLBXn6yCinL9yjWFw76MCyU/S0hRjp9r416Ruhf2hXP2/+w1M8um+w\n2U3xPLsGOzka7XmgfXO1PNo/qEMrt0wgIJwyR3n5A5vF3Oq+cbGoSEykOKhnUDUhHjOw55d499bM\nmuc4OW78YE36Rugd/HBT3EA8FuXc9Skm16ht6kxrdQRIbYibURZyBf56fGLVxz+cXWAxV9TXu0Y8\nfWSUUEAeOIuykv7Z9e07odfUhrhpUFRwZg3f2EqmCAjsGepscMv8yWP7h+hpC605i1qOuPGJ8DSb\nvo4wj+0f4oXzqyc5Sy3luTu36Js1KC30mlWJbe9lR3/H2sIzkWb3YCdtIb0wWAsioQBPHx3lxYv3\nKKziGyd8UOXIbcRjBpadXp6dVnLVRxE3oIVeswYiwphp8Ktxm4Xsx31jHXFTe+KmwWQ6y29ufLy2\nqWWn6OsIM9QVaULL/MmpY2tvVnPE/6APIm5AC73mAcRNg8VckVdW1DYtFBVXJ9J6dFljnj4yQji4\num/s7NDUa1C1Y3t/B5/c0beqfWPZKYIBYfegFnqNz/nsvkF62z/uG384s8BSvsh+nVOopvS0h/nc\ngWFeOH/3Y7mG/JJzxW3ETYN3bs6QnF/8yHHLTrF7sJNIyB8S6Y93oakL4WCAk8cMzl6695HapuPa\nL64bcdPg2mSG8eR933huMUdyfklbZXVgLGagFLx44aNBB37JceOghV7zQMZMg+lMjnPX7/vGTvlA\nLTy1x0ly9nyFnZDQETd144jRw+7BTk5XlBgsFBVXJ/2VTkULveaBPHl4hMiK2qaJiTQDnWEG9cJg\nzTF623loV/9HfHodcVM/nKCDvx6fJLWUB+D29ALZfFELvaZ16G4L8fjBYU5fvO8b64ib+hI3DX57\na5a7syXf2LJThALC7kG9Z6EexE2DbKHIKx+Ugg78lOPGQQu9Zl3GTIObUwtcujsPlDbvaKGvH06u\nodPlXENWMs2eoU7CQf11rQeP7BlgsCuyHH2zvOt72D+fcf3J0azLyXJt09MX7jGbyTGRWvLN1nA3\ncnC0m33DXR8RHh1xUz9CwQDPHB3l7KUkuUIRy04x2BVhwEfWpBZ6zbqM9rTzmd0DvHDhLtaEXoit\nN45v/Hpikul0lms+Wxh0I3HTYG4xz5tXp3wXcQNa6DVVMmYa/O72HL+6Ukq6pRcG60vcNMgVFH/+\n+nVyBeU74XEbTxwaoT0c4IXzd0tZK33WsWqh11SF4xv/5LVrhIPCroGOJrfI33x69wDD3RH+zWvX\nAN2x1puOSJAnDo3w/753h8l0tjWFXkS+ICKXRWRcRL6zyuNtIvKz8uNviMje8vG9IrIgIu+W//3L\n2jZf0yj2j3RzYKSLyXSWvUNdhPTCYF0JBoSTR0u5bwAO+Ghh0K2MmQZTzvX2UcQNVCH0IhIEfgh8\nETCBr4mIueK0bwDTSqmDwA+A71c8ZimlHi7/+6MatVvTBOKxKIBeiG0Q8VhpFjXcHaGvM9zk1vif\nk0dHCZRTCfkp4gaqG9GfAMaVUgmlVBb4KfDsinOeBX5S/vnnwEnR2Zd8h7Nr02/TWrfyeweH6QgH\ndcRNgxjqbuP4nkEiwQA7fWZNhqo4Zwdws+L3W8Cja52jlMqLyCwwVH5sn4i8A8wB/5NS6tWVLyAi\n3wS+CbB79+4NvQFN43h4Zz9/5+Qhnn14e7Ob0hK0h4P842djvqhZ6hX+hz84woUPZ31nTVYj9Fvh\nDrBbKTUpIo8A/15EYkqpucqTlFI/An4EcPz48bWr9WqaSiAgfHvscLOb0VL8zeO7mt2EluLEvkFO\n+LDudDXd1m2g8tO2s3xs1XNEJAT0AZNKqSWl1CSAUuptwAK0Umg0Gk0DqUbo3wIOicg+EYkAXwWe\nW3HOc8DXyz9/GTirlFIiMlJezEVE9gOHgERtmq7RaDSaaljXuil77t8CngeCwJ8qpc6LyPeAc0qp\n54AfA/9WRMaBKUqdAcCTwPdEJAcUgT9SSk3V441oNBqNZnVkZSWbZnP8+HF17ty5ZjdDo9FoPIWI\nvK2UOr7aY/5aWtZoNBrNx9BCr9FoND5HC71Go9H4HC30Go1G43NctxgrIjZwfQtPMQxM1Kg5XqEV\n3zO05vvW77l12Oj73qOUGlntAdcJ/VYRkXNrrTz7lVZ8z9Ca71u/59ahlu9bWzcajUbjc7TQazQa\njc/xo9D/qNkNaAKt+J6hNd+3fs+tQ83et+88eo1Go9F8FD+O6DUajUZTgRZ6jUaj8Tm+Efr1Cpj7\nARHZJSK/FJELInJeRP5u+figiJwWkSvl/wea3dZ6ICJBEXlHRP5D+fd95WL04+Xi9JFmt7GWiEi/\niPxcRC6JyEUR+Vwr3GsR+ePy5/t3IvIXItLux3stIn8qIkkR+V3FsVXvr5T438rv/z0R+cxGXssX\nQl9lAXM/kAf+nlLKBB4D/rvy+/wOcEYpdQg4U/7dj/xd4GLF798HflAuSj9NqUi9n/jnwH9USh0F\nHqL03n19r0VkB/B3gONKqU9QSo3+Vfx5r/8N8IUVx9a6v1+kVM/jEKWyq/9iIy/kC6GnugLmnkcp\ndUcp9Zvyz/OUvvg7+Ghx9p8A/1lzWlg/RGQn8J8A/7r8uwDPUCpGDz573yLSR6mew48BlFJZpdQM\nLXCvKdXJ6ChXq+ukVJLUd/daKfUKpfodlax1f58F/ndV4nWgX0S2VftafhH61QqY72hSWxqCiOwF\nPg28ARhKqTvlh+4CRpOaVU/+GfD3KRWwgVLx+RmlVL78u9/u+T7ABv6sbFf9axHpwuf3Wil1G/gn\nwA1KAj8LvI2/73Ula93fLWmcX4S+pRCRbuD/Av77VQqtK8BXMbMi8p8CyXLd4VYhBHwG+BdKqU8D\naVbYND691wOURq/7gO1AFx+3N1qCWt5fvwh9NQXMfYGIhCmJ/L9TSv1V+fA9ZxpX/j/ZrPbVid8D\nviQi1yjZcs9Q8q/7y9N78N89vwXcUkq9Uf7955SE3+/3+hRwVSllK6VywF9Ruv9+vteVrHV/t6Rx\nfhH6agqYe56yL/1j4KJS6p9WPFRZnP3rwP/T6LbVE6XUP1BK7VRK7aV0b88qpf4Q+CWlYvTgs/et\nlLoL3BSRI+VDJ4EL+PxeU7JsHhORzvLn3Xnfvr3XK1jr/j4H/Nfl6JvHgNkKi2d9lFK++Af8DeAD\nwAL+YbPbU6f3+Dilqdx7wLvlf3+Dkl99BrgCvAgMNrutdbwGTwP/ofzzfuBNYBz4P4G2Zrevxu/1\nYeBc+X7/e2CgFe418I+BS8DvgH8LtPnxXgN/QWkdIkdpBveNte4vIJQiCy3gfUpRSVW/lk6BoNFo\nND7HL9aNRqPRaNZAC71Go9H4HC30Go1G43O00Gs0Go3P0UKv0Wg0PkcLvUaj0fgcLfQajUbjc/5/\nJEnKxXezEskAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t8VfXH-_gVTB",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 281
        },
        "outputId": "6a9bb85f-0c9c-4f06-a6a7-e056dbb3b7ff"
      },
      "source": [
        "'''\n",
        " 第一次退火到大学习率\n",
        " 将学习率从一个初始学习率退火到一个最大学习率，然后从这个最大学习率退火到一个比初始学习率低很多的最小学习率。\n",
        "'''\n",
        "scheduler = lr_scheduler.OneCycleLR(optimizer, max_lr=0.1, steps_per_epoch=10, epochs=10)\n",
        "# (optimizer, max_lr=0.01, steps_per_epoch=len(data_loader), epochs=10)\n",
        "\n",
        "plt.figure()\n",
        "x = list(range(100))\n",
        "y = []\n",
        "for epoch in range(100):\n",
        "    optimizer.step()\n",
        "    scheduler.step()\n",
        "    lr = scheduler.get_lr()\n",
        "    y.append(scheduler.get_lr()[0])\n",
        "\n",
        "plt.plot(x, y)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f6b25bc2da0>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3yV5f3/8dfnZCckIRtIgLBDGGGE\noSAyRHEgIojgrKNYRx20VbqtrVXrrIpW68KBYJ1UEJChIioQNiEBwgqBbDIgIfv6/ZHj95fGAIGc\n5D7j83w8eJBzzp3kfXvLOyfXfd/XJcYYlFJKuS+b1QGUUkq1Li16pZRyc1r0Sinl5rTolVLKzWnR\nK6WUm/O2OkBjkZGRJj4+3uoYSinlUjZt2lRgjIlq6jWnK/r4+HhSUlKsjqGUUi5FRA6d6jUdulFK\nKTenRa+UUm5Oi14ppdycFr1SSrk5LXqllHJzzSp6EZkkIrtFJENE5jbx+hgR2SwiNSIyvdFrN4vI\nXvufmx0VXCmlVPOcsehFxAuYB1wKJAKzRCSx0WaZwM+ABY0+Nxz4MzACGA78WUTCWh5bKaVUczXn\nOvrhQIYxZj+AiCwEpgC7ftzAGHPQ/lpdo8+9BPjSGHPM/vqXwCTg/RYnV63KGENW0Ul2ZZeSkXeC\nujqDv48X/j42woJ8iQ72JzrYj07tA/D11hFApZxZc4o+Fjjc4HEW9e/Qm6Opz41tvJGIzAZmA3Tp\n0qWZX1q1horqWuZ/d5BXvtnPsbKqM27vbRO6RATSK7od/TuFMqRrGEmd29POz+nuxVPKYznFv0Zj\nzKvAqwDJycm6EooFjDH8d3s2/1iWTlbRScb2iWJiYgyJHUPo0yEYb5uNyppaTlbXcqysirzSSnJL\nKzhYWEZG3gn25p1geWouADaBgXHtGdcnmnEJUfTvFIrNJhbvoVKeqzlFfwTo3OBxnP255jgCjG30\nuV8183NVG6mtM/zxs50sWJ9J344hvHf7QEb1jPzJdr7eNoL9fYgO9iehw0+/TsnJarYeLmbTwWOs\nzSjguVV7eHblHjqE+DM5qSNTBsXSr1MIIlr6SrUlOdNSgiLiDewBJlBf3BuB64wxqU1s+xbwuTHm\nQ/vjcGATMMS+yWZg6I9j9k1JTk42OtdN26msqWXOom0s2ZHNHRd258FLEvBy0LvvwhOVfLM3nyXb\ns/lqdz41dYbeMe24YWRXpg6OJdjfxyHfRykFIrLJGJPc5GvNWTNWRC4DngO8gDeMMY+KyCNAijFm\nsYgMAz4BwoAKIMcY08/+ubcCv7N/qUeNMW+e7ntp0bed8qoa7nhnE2v3FvC7yxKYPaZHq32vorIq\nlu7MZuGGw+w4UkKQrxdXD4lj9pjudA4PbLXvq5SnaHHRtyUt+rZhjOGe97fwxY5sHp82kBnJnc/8\nSQ6y7XAxb39/iMXbjlBnYEpSJ+4a14Oe0cFtlkEpd6NFr37itbX7+duSNB6alMCdY1vvnfzpZJec\n5LW1B1iwPpOKmlqmDYnjgYm9iW0fYEkepVyZFr36Hz/sL+T619ZzUd9o/nXDUMtPjh4rq+LlrzKY\n/90hEPjZ+fHcM74nITqGr1Szna7o9U4XD5NbWsE9CzbTNTyQp65JsrzkAcKDfPn95Yms+c1YJg/s\nxL/X7mf8U1/z8eYsnO2NiFKuSIvew/zh052cqKzhXzcOdbqrXmLbB/D0jCQ+vWsUsWEBzPlgGzNe\n+Z6MvBNWR1PKpWnRe5AVqTl8uSuX+y/qTe8Y5z3xmdS5PZ/ceT5PTBvA3rwTXPb8WuatyaCmtvEM\nG0qp5tCi9xBllTU8vDiVhA7B3Da6m9VxzshmE64d1oUvH7iQi/pG8+Ty3Vz10jr25h63OppSLkeL\n3kM8t3IPR0sqeHRqf3y8XOewRwX78dL1Q3n5+iEcLa7gihe+5Z3vD+rYvVJnwXX+xatztutoKW+s\nO8is4Z0Z2jXc6jjn5NIBHVl2/wWM7B7BHz9L5fb5Kc2adE0ppUXvER77Io0Qf28empRgdZQWiQ72\n561bhvHw5ETWZhRwxfNr2ZJZZHUspZyeFr2bW7+/kLV7C7hrbE/aB/paHafFRISfjerGR784H5tN\nmPHK98z/TodylDodLXo3Zozh6RV7iA7244aRXa2O41AD4kJZ8ssLGNMrij8vTuVX/9lGRXWt1bGU\nckpa9G5s7d4CNhw8xj3jexLg62V1HIcLDfTh3zclM2dibz7efISZr/5AXmmF1bGUcjpa9G6q/t38\nbmLbB3DtsLabsKyt2WzCvRN68a8bhrIn9ziTX/yWHVklVsdSyqlo0buplWl5bMsq4b4JvfDzdr93\n841N6t+Bj+48H2+bjWtf/Z416XlWR1LKaWjRuyFjDPPWZNA1IpCrh/xkiV631bdjCJ/cdT7do4K4\nbf5GFqzPtDqSUk5Bi94NbTpUxNbDxdw+uhveLnRzlCNEh/izaPZ5XNg7it99soOnV+zWK3KUx/Os\nFvAQ/167n9AAH6YNjbM6iiWC/Lz5903JzBzWmRdWZ/DHz3ZSW6dlrzxXcxYHVy7kUGEZK3blctfY\nHgT6eu7h9fay8djVA2gf6Mu/vt5HcXk1z8wYhK+3vrdRnsdzm8BNvfHtAbxtws3nxVsdxXIiwtxL\nEwgL9OGxL9Ipq6zh5RuG4u/j/ienlWpI3964keLyKj5IyeLKpFiiQ/ytjuM07riwB49dPYCv9uRz\n+/wUTlbpjVXKs2jRu5EFGzI5WV3L7Rc4/zTEbW3W8C48OT2J7/YVcMtbGyirrLE6klJtRoveTdTW\nGd77IZPze0TQt2OI1XGc0vShcTx77SA2Hiziljc3Ul6lZa88gxa9m1i7N58jxSe5bkQXq6M4tSmD\nYvnnzEGkHDrGrW9t1GEc5RG06N3EgvWZRAT5cnFiB6ujOL0rBnbimRmDWH/gGD9/O0UnQ1NuT4ve\nDeSWVrAqPY/pyXF6+WAzXTU4lienJ7FuXwF3vLOJqhpdj1a5L20FN/CflMPU1hlmDdNhm7MxfWgc\nf586gK/35PPAoq16U5VyW3odvYurqzO8v+Ewo3pGEB8ZZHUclzNreBdOVNTw6NI02vl58/i0AYiI\n1bGUcigtehe3NqOAI8Un+e1lrr1MoJV+PqY7pRXVvLA6g5AAb35/eaLVkZRyKC16F7dwg56EdYQ5\nE3tTerKaf689QFSwH7PH9LA6klIOo0XvworLq1iZlstN58XrSdgWEhH+PLkfBWVV/H1pOpHt/Lh6\niGdOCqfcjxa9C/t8ezbVtcaj5pxvTTab8MyMJIrKqnjww+2EBfkyrk+01bGUarFmvQ0UkUkisltE\nMkRkbhOv+4nIIvvr60Uk3v68j4jMF5EdIpImIr91bHzP9vHmLPrEBJOod8I6jJ+3F6/cOJTeMcHc\n/d5mXZZQuYUzFr2IeAHzgEuBRGCWiDQ+W3UbUGSM6Qk8Czxhf/4awM8YMwAYCtzx4w8B1TIHC8rY\nnFnM1CGxepWIgwX7+/DWLcMIC/Tl1vkbySoqtzqSUi3SnHf0w4EMY8x+Y0wVsBCY0mibKcB8+8cf\nAhOkvn0MECQi3kAAUAWUOiS5h/tkyxFE4KpBOmzTGqJD/HnzlmFUVNdyy5sbKTlZbXUkpc5Zc4o+\nFjjc4HGW/bkmtzHG1AAlQAT1pV8GZAOZwFPGmGONv4GIzBaRFBFJyc/PP+ud8DTGGD7ZcoRRPSLp\nEKrTEbeW3jHBvHLDUA4WlnHnu5uortW7Z5Vrau1LNYYDtUAnoBvwKxHp3ngjY8yrxphkY0xyVFRU\nK0dyfZsOFZF5rJypg/XdfGs7v2ckj189kO/2FfLHT3fq+rPKJTWn6I8AnRs8jrM/1+Q29mGaUKAQ\nuA5YZoypNsbkAeuA5JaG9nQfbzlCgI8Xk/rrtfNtYdrQOO4e14OFGw/z+rcHrI6j1FlrTtFvBHqJ\nSDcR8QVmAosbbbMYuNn+8XRgtal/65MJjAcQkSBgJJDuiOCeqrq2jqU7srm4XwxBfnp1bFv51cQ+\nXNq/A48uTWPlrlyr4yh1Vs5Y9PYx93uA5UAa8IExJlVEHhGRK+2bvQ5EiEgGMAf48RLMeUA7EUml\n/gfGm8aY7Y7eCU/ybUYBxeXVTB7YyeooHqX+GvtB9O8Uyr0Lt5Ceo9cUKNchzjbmmJycbFJSUqyO\n4bR+9cE2VuzKIeUPF+HnrYtct7Xc0gomv/Atfj42Prt7NOFBvlZHUgoAEdlkjGlyaFzvm3chlTW1\nrNiVw8WJHbTkLRIT4s8rNw4lt7SSu97TK3GUa9CidyFr9xRwvKKGK5I6Wh3Fow3uEsbjVw/gh/3H\neOS/u6yOo9QZ6dk8F/L59qO0D/RhdM9Iq6N4vKuHxJGec5xXv9lPv04hzByui74o56Xv6F1ERXUt\nX+7KZVK/Dvh46WFzBg9NSuCCXpH86bNUtmQWWR1HqVPSxnARX+3Oo6yqlssH6rCNs/CyCc/PHEx0\niB93vruZ/OOVVkdSqkla9C7i8+3ZRAT5cl73CKujqAbCgnx55cahFJ+s4u73NuvJWeWUtOhdQEV1\nLavT87ikfwe8ddjG6fTrFMoT0way4eAxHluq9wMq56Ot4QLW7i2gvKqWS3XKA6c1ZVAsPzs/njfW\nHeDz7UetjqPU/9CidwHLduYQ4u/NSB22cWq/u6wvQ7uG8eCH29mbe9zqOEr9Hy16J1ddW8fKtFwu\nSozRq22cnK+3jZeuH0Kgrzd3vLuJE5U1VkdSCtCid3rr9x+j5GQ1l/TTYRtXEBPiz4vXDeZgQRlz\nP9qu0xorp6BF7+SWp+YQ4OPFmF46T7+rGNk9gl9f0ofPt2fzzg+HrI6jlBa9M6urMyxPzWFsnygC\nfHVuG1fyizE9mJAQzV8/38W2w8VWx1EeToveiW05XEze8UpdYMQF2WzC0zOSiA725673NlNcXmV1\nJOXBtOid2IrUHHy8hHEJ0VZHUeegfaAv864fQt7xCn79Hx2vV9bRondSxhiWpeYwqmckIf4+VsdR\n52hQ5/b89tK+rEzL1WUIlWW06J3U3rwTHCosZ2JijNVRVAvdMiqeixNjeGJZOlt1vF5ZQIveSX1p\nX5f0or5a9K5ORHhyev14/T0LNlNSXm11JOVhtOid1Je7cknq3J6YEH+roygHCA304cXrBpNTUsHc\nj3W8XrUtLXonlHe8gq2Hi5nYV0/CupPBXcJ4cFIfvtiZw3vrM62OozyIFr0TWpWWB8BFOj7vdm4f\n3Z0xvaN45PNdpGWXWh1HeQgteie0clcucWEB9IkJtjqKcjCbTXhmRhKhAT788v0tlFfpfDiq9WnR\nO5nyqhq+zShgYmIMImJ1HNUKItv58dy1g9iXf4K/LNbFxVXr06J3Mmv3FlBZU8dEvdrGrY3qGcmd\nF/ZgUcphlmzPtjqOcnNa9E7my125hPh7M6xbuNVRVCt7YGJvBnVuz9yPt5NVVG51HOXGtOidSG2d\nYXV6HuMSonXueQ/g42Xj+ZmDMQbuX7iVGl1vVrUSbRMnsvVwEcfKqpigwzYeo0tEII9O7U/KoSJe\nWJ1hdRzlprTonciqtDy8bMKFvXXueU8yZVAsVw+J5YXVe0k5eMzqOMoNadE7kdXpeQyLDyM0QCcx\n8zR/ubIfcWGB3LdwK6UVOkWCciwteieRVVROes5xJiTosI0nCvb34bmZg8gpreCPn+60Oo5yM80q\nehGZJCK7RSRDROY28bqfiCyyv75eROIbvDZQRL4XkVQR2SEiOnlLE9ak198NO16nPfBYQ7qEcd+E\nXny29SifbMmyOo5yI2csehHxAuYBlwKJwCwRSWy02W1AkTGmJ/As8IT9c72Bd4FfGGP6AWMB/b20\nCavS84iPCKR7ZJDVUZSF7h7Xk2HxYfzp01QOH9NLLpVjNOcd/XAgwxiz3xhTBSwEpjTaZgow3/7x\nh8AEqb+t82JguzFmG4AxptAYU+uY6O6jvKqG7/YVMj5B74b1dF424ZkZgwCY88FWaut0lkvVcs0p\n+ljgcIPHWfbnmtzGGFMDlAARQG/AiMhyEdksIg829Q1EZLaIpIhISn5+/tnug8tbl1FIVU0dE3TY\nRgGdwwP561X92XiwiJe/0ksuVcu19slYb2A0cL3976kiMqHxRsaYV40xycaY5Kgoz7u0cFVaLu38\nvBkWr3fDqnpXDY7lyqROPLdyr65KpVqsOUV/BOjc4HGc/bkmt7GPy4cChdS/+//GGFNgjCkHlgJD\nWhrandTZ74Yd0zsSX2+9CEr9f3+9qj8xIf48sGirznKpWqQ5zbIR6CUi3UTEF5gJLG60zWLgZvvH\n04HVpn4JneXAABEJtP8AuBDQ6foa2JVdSt7xSsbrZZWqkdAAH56ekcTBwjL+tiTN6jjKhZ2x6O1j\n7vdQX9ppwAfGmFQReURErrRv9joQISIZwBxgrv1zi4BnqP9hsRXYbIxZ4vjdcF2r0/MQgbF9PG/I\nSp3ZyO4RzB7TnQXrM1lpX0dYqbMlzrZ2ZXJysklJSbE6Rpu5at46DPDZ3aOsjqKcVGVNLVPnfUdu\naQXL7h9DVLCf1ZGUExKRTcaY5KZe00FhCxWeqGRbVjHj9N28Og0/by+emzmI45U1PPSRLiyuzp4W\nvYW+3pOPMTA+QS+rVKfXOyaYuZMSWJ2ex/sbDp/5E5RqQIveQqvT84hs50f/TqFWR1Eu4GfnxzO6\nZyR//XwXBwrKrI6jXIgWvUVqauv4Zk8+4/pEYbPp3bDqzGw24clrBuLjJTywSBcqUc2nRW+RzZnF\nlFbUME6HbdRZ6BgawKNTB7D1cDHz1uyzOo5yEVr0Flmdnoe3TRjdK9LqKMrFTE7qxFWDOvH86r1s\n07tmVTNo0VtkTXoew+LDCfHXRUbU2fvLlP5EB/vxwAdbOVml8wSq09Oit8CR4pPszj3OuAS9rFKd\nm9AAH56+Jon9+WU8/oXeNatOT4veAj8uMjKuj47Pq3N3fs9Ibh3VjfnfH+LrPZ4366tqPi16C3y1\nO4+4sAB6RrezOopycQ9O6kOv6Hb85j/bKC6vsjqOclJa9G2sorqWdRmFjE+I1kVGVIv5+3jx7LWD\nKCqv4vef7tS7ZlWTtOjb2IYDxzhZXavDNsph+seGcv9FvVmyPZvF245aHUc5IS36NrY6PQ8/bxsj\nu0dYHUW5kTvGdGdo1zD++OlOjhaftDqOcjJa9G3sq915nN8jggBfL6ujKDfi7WXjmRlJ1NQZfvPh\nNup0rVnVgBZ9GzpQUMbBwnK9G1a1iq4RQfzxikTWZRQy//uDVsdRTkSLvg2t1ssqVSubOawzExKi\nefyLdDLyjlsdRzkJLfo29NXuPHpGt6NzeKDVUZSbEhEemzaAID9v7l+0laoanfhMadG3mbLKGtbv\nP6aLjKhWFx3sz9+nDmDnkVJeWL3X6jjKCWjRt5F1GQVU1dbpsI1qE5P6d2D60Djmrclg06Eiq+Mo\ni2nRt5E1u/No5+dNcny41VGUh/jz5EQ6hgbwqw+2UlZZY3UcZSEt+jZgjGFNej6je0bi663/yVXb\nCPb34ZkZSRw6Vs6jS3XiM0+mrdMG0rKPk1NaoWvDqjY3onsEsy/ozoL1maxOz7U6jrKIFn0bWLO7\n/rLKsXoiVllgzsW9SegQzIMf7qDwRKXVcZQFtOjbwJr0PPrHhhAd4m91FOWB/Ly9eG7mIEpPVvPb\nj3foxGceSIu+lRWVVbE5s4jxerWNslBChxB+fUlvVuzK5T+bsqyOo9qYFn0r+2ZvPnUGnfZAWe72\n0d0Z2T2cvyxOJbOw3Oo4qg1p0bey1el5hAf5MjCuvdVRlIez2YSnrknCJsKcD7ZSqxOfeQwt+lZU\nW2f4ek8+Y3tH4WXTRUaU9eLCAnnkqn6kHCriX1/vszqOaiNa9K1o6+EiisurddhGOZWrBsVy+cCO\nPPvlHnZklVgdR7UBLfpWtDo9Dy+bMKaXXlapnIeI8OhV/Yls58d9i7ZwsqrW6kiqlWnRt6JVaXkM\niw8jNNDH6ihK/Y/2gb48PSOJ/fllPPaF3jXr7ppV9CIySUR2i0iGiMxt4nU/EVlkf329iMQ3er2L\niJwQkV87JrbzyyoqJz3nOBMSYqyOolSTRvWM5PbR3Xj7+0Ossa+VoNzTGYteRLyAecClQCIwS0QS\nG212G1BkjOkJPAs80ej1Z4AvWh7Xdfz4D2d8Xx2fV87r15f0IaFDML/5cBsFetes22rOO/rhQIYx\nZr8xpgpYCExptM0UYL794w+BCSIiACJyFXAASHVMZNewKj2P+IhAukcGWR1FqVPy9/HinzMHU1pR\nw0Mfbte7Zt1Uc4o+Fjjc4HGW/bkmtzHG1AAlQISItAMeAv5yum8gIrNFJEVEUvLz85ub3WmVV9Xw\n3b5CxifEYP95p5TT6tMhmLmTEliVnsd76zOtjqNaQWufjH0YeNYYc+J0GxljXjXGJBtjkqOiXP8K\nlXUZhVTV1DFBh22Ui/jZ+fFc0CuSvy3ZRUbeaf+5KhfUnKI/AnRu8DjO/lyT24iINxAKFAIjgH+I\nyEHgfuB3InJPCzM7vdXpubTz82aYLjKiXITNJjx9TRIBPl7ct3ALlTV6yaU7aU7RbwR6iUg3EfEF\nZgKLG22zGLjZ/vF0YLWpd4ExJt4YEw88B/zdGPOig7I7JWMMq9LyGNNbFxlRriU6xJ9/TE8i9Wgp\nT6/YY3Uc5UBnbCL7mPs9wHIgDfjAGJMqIo+IyJX2zV6nfkw+A5gD/OQSTE+RerSUvOOVelmlckkT\nE2O4fkQXXv1mP9/uLbA6jnIQ7+ZsZIxZCixt9NyfGnxcAVxzhq/x8Dnkczkr03IR0UVGlOv6w+WJ\nrD9wjDkfbGXZ/WMID/K1OpJqIR1bcLCVabkM7RJGRDs/q6ModU4CfL3458xBFJdX86BecukWtOgd\n6GjxSXYeKeWiRB22Ua6tX6dQHro0gZVpubz7wyGr46gW0qJ3oFVp9YsvX9RXi165vltHxTO2TxR/\nXZJGek6p1XFUC2jRO9CKXbl0jwyiZ3Q7q6Mo1WIi9QuVhPj7cO/7W6io1ksuXZUWvYMcr6jmh/2F\nOmyj3EpkOz+emZHEntwTPPL5LqvjqHOkRe8g3+wpoLrW6LCNcjtjekdxx4XdWbA+ky92ZFsdR50D\nLXoH+XJXDmGBPgztGmZ1FKUc7tcX92FQ5/Y8+NF2Dh/ThcVdjRa9A1TX1rE6PY/xCTG6NqxySz5e\nNl6YNRgM3LdwC9W1dVZHUmdBi94BNh48RmlFDRMTdRIz5b46hwfy+LSBbM4s5pkvdYoEV6JF7wAr\nUnPx9bZxga4Nq9zc5QM7Mmt4F17+ah9f7dZVqVyFFn0LGWNYkZrDmF6RBPk1a0YJpVzanycnktAh\nmDkfbCOnpMLqOKoZtOhbaMeREo6WVHBJvw5WR1GqTfj7eDHv+iFUVNdy7/tbqNHxeqenRd9Cy3bm\n4GUTvaxSeZQeUe34+9QBbDh4jGdX6ni9s9OibwFjDMt25jCyezhhOsOf8jBXDY5l5rDOzFuzjzU6\nXu/UtOhbICPvBPsLypikwzbKQz18ZT/6dgzhgUVbOVJ80uo46hS06FtgeWoOABdr0SsP5e/jxUvX\nD6Gm1nD3e5upqtHxemekRd8Cy1JzGNKlPTEh/lZHUcoy3SKD+Mf0gWw9XMxjX6RZHUc1QYv+HB0+\nVs7OI6VM6q/v5pW6bEBHbhkVz5vrDrJ421Gr46hGtOjP0Y/DNnpZpVL1fndZX5K7hjH3o+3szT1u\ndRzVgBb9OVqyI5u+HUPoGhFkdRSlnIKPl4151w8h0NebO97dxPGKaqsjKTst+nOQVVTOlsxirhjY\n0eooSjmVmBB/XrxuMIcKy3W9WSeiRX8Oltrn5J48sJPFSZRyPiO7RzB3UgJf7Mzh5a/3WR1HoUV/\nTj7fns3AuFC6RARaHUUpp3T7Bd2YnNSJJ5fv1snPnIAW/Vk6VFjG9qwSLh+gwzZKnYqI8MS0AfSJ\nCebe97dwqLDM6kgeTYv+LH2+vX7Y5nIdn1fqtAJ9vXn1xmREhNlvb+JEZY3VkTyWFv1ZWrI9m8Fd\n2hMXpsM2Sp1Jl4hAXrxuMHvzjjNn0Vbq6vTkrBW06M/C/vwT7Mou1WEbpc7CBb2i+P3liazYlctz\nq/ZaHccj6UoZZ0GHbZQ6N7eOiic9u5TnV+2lT0yw/htqY/qOvpmMMXy69QjD48PpGBpgdRylXIqI\n8Lep/RnSpT2//s82dh4psTqSR9Gib6ZtWSXszy9j6pBYq6Mo5ZL8vL145cZkwoN8uW3+Rl2GsA01\nq+hFZJKI7BaRDBGZ28TrfiKyyP76ehGJtz8/UUQ2icgO+9/jHRu/7XyyOQtfbxuX6fi8UucsKtiP\n125O5kRFDbe/vZHyKr0Spy2csehFxAuYB1wKJAKzRCSx0Wa3AUXGmJ7As8AT9ucLgMnGmAHAzcA7\njgrelqpr6/jv9mwm9o0hNMDH6jhKubS+HUN4ftZgUo+WMmfRNr0Spw005x39cCDDGLPfGFMFLASm\nNNpmCjDf/vGHwAQREWPMFmPMj3OWpgIBIuLniOBt6evd+Rwrq2LqYB22UcoRJvSN4feX9WVZao7O\nYd8GmlP0scDhBo+z7M81uY0xpgYoASIabTMN2GyMqWz8DURktoikiEhKfn5+c7O3mY+3ZBEe5MuF\nfaKsjqKU27htdDduPq8r/157gLe/P2h1HLfWJidjRaQf9cM5dzT1ujHmVWNMsjEmOSrKucq05GQ1\nK9PyuDKpEz5eeu5aKUcREf40uR8X9Y3h4cWprNyVa3Ukt9Wc5joCdG7wOM7+XJPbiIg3EAoU2h/H\nAZ8ANxljXG4qu6U7sqmqqeNqvdpGKYfzsgkvzBrMgNhQfvn+FrZkFlkdyS01p+g3Ar1EpJuI+AIz\ngcWNtllM/clWgOnAamOMEZH2wBJgrjFmnaNCt6UPN2XRIyqIAbGhVkdRyi0F+Hrx+s+GER3ix61v\nbWRf/gmrI7mdMxa9fcz9HmA5kAZ8YIxJFZFHRORK+2avAxEikgHMAX68BPMeoCfwJxHZav8T7fC9\naCW7c46z6VARM4d1QUSsjqOU24ps58fbtw7Hyybc9PoGckv1GntHEmdbASY5OdmkpKRYHQOAhxen\nsmB9Jj/8bgLhQb5Wx1HK7Xr9eD0AAAwiSURBVO08UsK1r3xP5/BAFs0+j9BAvZy5uURkkzEmuanX\n9OziKZysquWjzVlM6t9BS16pNtI/NpRXbkxmf34Zt7y1gTKd2tghtOhPYcmObI5X1HDdiC5WR1HK\no4zuFcnzswax9XAxd7yzicqaWqsjuTwt+lN4f0Mm3SODGNEt3OooSnmcSf078o/pSXybUcC972+h\nprbO6kguTYu+CT+ehJ01XE/CKmWV6UPj+PPkRJan5jLng23U6lQJ50zno2/CgvWH8PWyMW1onNVR\nlPJot4zqRkV1HU8sS8fbJjx5TRJeNn3zdba06Bsprajmo81HuGyAnoRVyhncObYHNbV1PP3lHrxs\nwhPTBmLTsj8rWvSNLNyQyYnKGm6/oLvVUZRSdr+c0IvqOsPz9qUIH582UN/ZnwUt+gaqa+t4a91B\nRnYPp7/eCauUU3ngol4APL9qL9W1dTx1TRLeOv9Us2jRN7B0RzZHSyp4ZEp/q6MopRoREeZM7I2f\nt40nl++mutbw3MxBOtlgM2jR2xljeG3tAbpHBTE+wWVmaVDK49w9rie+XjYeXZrGyepa5l03hABf\nL6tjOTX9UWi34cAxdhwp4bbR3fREj1JO7udjuvO3q/qzZnceN72xnpKT1VZHcmpa9Hb/Xruf8CBf\npg3RSyqVcgU3jOzK8zMHs/VwMTNf/YG84zoR2qlo0QM7skpYmZbHzefF4++jvwIq5SomJ3XitZuH\ncbCgjKtf+o6MPJ3iuCla9MBTK3YTFujDraPjrY6ilDpLF/aOYuHskVRU1zLt5e9Yv7/Q6khOx+OL\nfuPBY3y9J59fXNiDYH+dElUpV5TUuT2f3DWKiHa+3Pj6Bj7d0ngRPM/m0UVvjOGp5buJbOfHTefF\nWx1HKdUCncMD+fjO8xnUpT33L9rKE8vSqdP5cQAPL/p1GYWsP3CMe8b10MuzlHID7QN9efe2Ecwa\n3oWXv9rH7HdSOF6hV+R4bNHX1RmeXJ5Op1B/Zumc80q5DV9vG3+f2p+/TunHmt35TJm3jj25x62O\nZSmPLfpFKYfZllXCry7ug5+3vptXyp2ICDeeF897t4+g9GQNU15cxydbsqyOZRmPLPqCE5U8/kU6\nI7qFc/WQWKvjKKVaycjuESy9dzQD4kJ5YNE2fvvxdsqrPG95Qo8s+r8vSaO8qoZHpw7QhUWUcnPR\nIf4suH0Ed47twcKNh7nihW/ZeaTE6lhtyuOK/ruMAj7ecoRfXNiDntHtrI6jlGoD3l42HpqUwHu3\njaC8spapL61j3poMj1mi0KOKvqyyht9/upOuEYHcPa6n1XGUUm3s/J6RLLv/AiYmxvDk8t1Mfek7\n0rJLrY7V6jym6I0xPPjRdg4VlvH41QN1qgOlPFT7QF/mXTeEedcN4WjxSSa/8C1Pr9hNRXWt1dFa\njccU/evfHmDJ9mwenJTAeT0irI6jlLKQiHD5wI58OedCrhjYkRdWZ3DRM1+zIjUHY9zvJiuPKPrv\n9xXy2BfpTOrXgTvG6BKBSql64UG+PDdzMAt+PoIAHy9mv7OJm9/cyK6j7jWc4/ZFvzvnOPcs2Ex8\nRCBPXjNQr7JRSv3E+T0iWXrfBfzh8r5szSzi8hfW8sCirRw+Vm51NIcQZ/s1JTk52aSkpDjka206\ndIxb3txIgK8XC34+kh5RepWNUur0SsqreenrDN5ad5DaOsPUwbH8YmwPp+8PEdlkjElu8jV3Lfo1\nu/O4891NdAwN4O1bh9M5PNAB6ZRSniK75CSvfL2f9zdkUlVbx6X9O3DzefEM7xbulCMDHlX0BScq\n+efKvSzYkElCh2Dm3zqcyHZ+DkyolPIkBScqeePbA7z7wyFKK2roHdOOG0Z2ZfLAToQF+Vod7/94\nRNFXVNfyxroDvLRmHyera7l+RBd+c0kfnWNeKeUQJ6tq+e+2o7z9w0F2HinF2yZc2DuKKwd1Ymyf\naEIDrO2a0xW9dzO/wCTgn4AX8Jox5vFGr/sBbwNDgULgWmPMQftrvwVuA2qBe40xy89xP05re1YJ\n/1i2m4v6xvDbyxKcfjxNKeVaAny9mDGsM9ckx5F6tJTF246yeOtRVqXn4WUThnYNY2yfKEZ0C6df\np1CnulfnjO/oRcQL2ANMBLKAjcAsY8yuBtvcBQw0xvxCRGYCU40x14pIIvA+MBzoBKwEehtjTnln\nQkuGbtKyS+nbMeScPlcppc5WXZ1hc2YRa3bnsSY9n132u2x9vWz0iw0hoUMIPaPb0TO6HbHt/YkK\n9ifE37tVxvhb+o5+OJBhjNlv/2ILgSnArgbbTAEetn/8IfCi1O/JFGChMaYSOCAiGfav9/257MiZ\naMkrpdqSzSYkx4eTHB/Oby5JIP94JZszi9h8qIgtmcV8sTOb4vL/XfjEz9tGOz9v/Lxt+Pl4UWcM\nldV1VNbUMjExhn9MT3J4zuYUfSxwuMHjLGDEqbYxxtSISAkQYX/+h0af+5N5gUVkNjAboEsXXQRE\nKeWaooL9uKRfBy7p1wGon3qlsKyKfXknyCmtIK+0kvwTlZRV1lBhL3cvm+DnbcPfx4uBce1bJVez\nxuhbmzHmVeBVqB+6sTiOUko5hIgQ2c7P8iv/mnNn7BGgc4PHcfbnmtxGRLyBUOpPyjbnc5VSSrWi\n5hT9RqCXiHQTEV9gJrC40TaLgZvtH08HVpv6s7yLgZki4ici3YBewAbHRFdKKdUcZxy6sY+53wMs\np/7yyjeMMaki8giQYoxZDLwOvGM/2XqM+h8G2Lf7gPoTtzXA3ae74kYppZTjuc0NU0op5clOd3ml\n289eqZRSnk6LXiml3JwWvVJKuTkteqWUcnNOdzJWRPKBQy34EpFAgYPiuApP3GfwzP3WffYcZ7vf\nXY0xUU294HRF31IiknKqM8/uyhP3GTxzv3WfPYcj91uHbpRSys1p0SullJtzx6J/1eoAFvDEfQbP\n3G/dZ8/hsP12uzF6pZRS/8sd39ErpZRqQIteKaXcnNsUvYhMEpHdIpIhInOtztMaRKSziKwRkV0i\nkioi99mfDxeRL0Vkr/3vMKuztgYR8RKRLSLyuf1xNxFZbz/mi+zTaLsNEWkvIh+KSLqIpInIeZ5w\nrEXkAfv/3ztF5H0R8XfHYy0ib4hInojsbPBck8dX6j1v3//tIjLkbL6XWxS9fQHzecClQCIwy74w\nubupAX5ljEkERgJ32/dzLrDKGNMLWGV/7I7uA9IaPH4CeNYY0xMoAm6zJFXr+SewzBiTACRRv+9u\nfaxFJBa4F0g2xvSnfmr0mbjnsX4LmNTouVMd30upX8+jF/XLrr58Nt/ILYqeBguYG2OqgB8XMHcr\nxphsY8xm+8fHqf+HH0v9vs63bzYfuMqahK1HROKAy4HX7I8FGE/9YvTgZvstIqHAGOrXesAYU2WM\nKcYDjjX162QE2FerCwSyccNjbYz5hvr1Oxo61fGdArxt6v0AtBeRjs39Xu5S9E0tYP6TRcjdiYjE\nA4OB9UCMMSbb/lIOEGNRrNb0HPAgUGd/HAEUG2Nq7I/d7Zh3A/KBN+3DVa+JSBBufqyNMUeAp4BM\n6gu+BNiEex/rhk51fFvUce5S9B5FRNoBHwH3G2NKG75mX8LRra6ZFZErgDxjzCars7Qhb2AI8LIx\nZjBQRqNhGjc91mHUv3vtBnQCgvjp8IZHcOTxdZei95hFyEXEh/qSf88Y87H96dwff42z/51nVb5W\nMgq4UkQOUj8sN5768ev29l/vwf2OeRaQZYxZb3/8IfXF7+7H+iLggDEm3xhTDXxM/fF352Pd0KmO\nb4s6zl2KvjkLmLs8+7j060CaMeaZBi81XJz9ZuCzts7WmowxvzXGxBlj4qk/tquNMdcDa6hfjB7c\nbL+NMTnAYRHpY39qAvVrL7v1saZ+yGakiATa/3//cb/d9lg3cqrjuxi4yX71zUigpMEQz5kZY9zi\nD3AZsAfYB/ze6jyttI+jqf9Vbjuw1f7nMurHq1cBe4GVQLjVWVvxv8FY4HP7x92BDUAG8B/Az+p8\nDt7XQUCK/Xh/CoR5wrEG/gKkAzuBdwA/dzzWwPvUn4eopv43uNtOdXwBof7Kwn3ADuqvSmr299Ip\nEJRSys25y9CNUkqpU9CiV0opN6dFr5RSbk6LXiml3JwWvVJKuTkteqWUcnNa9Eop5eb+H5Eukpfp\nBOl+AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "El6y4uUUI30i",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 950
        },
        "outputId": "a4cdd3d9-70bc-432c-9bfd-f7ca4db3086e"
      },
      "source": [
        "'''\n",
        " 自适应下降,当loss不减时降低学习率\n",
        " 这个调度器读取一个指标数量，如果没有看到“容忍间隔”时间内的改善，学习率就会降低\n",
        " factor：衰减因子\n",
        " patience：容忍epoch间隔，当patience个epoch损失没有改进时使学习率会降低。\n",
        "'''\n",
        "import torchvision.models as models\n",
        "import torch.nn as nn\n",
        "model = models.resnet18(pretrained=False)\n",
        "fc_features = model.fc.in_features\n",
        "model.fc = nn.Linear(fc_features, 2)\n",
        "criterion = nn.CrossEntropyLoss()\n",
        "optimizer = optim.SGD(params = model.parameters(), lr=10)\n",
        "\n",
        "scheduler = lr_scheduler.ReduceLROnPlateau(optimizer, factor=0.1, patience=3)\n",
        "\n",
        "inputs = torch.randn(4,3,224,224)\n",
        "labels = torch.LongTensor([1,1,0,1])\n",
        "plt.figure()\n",
        "x = list(range(20))\n",
        "y = []\n",
        "for epoch in range(20):\n",
        "    optimizer.zero_grad()\n",
        "    outputs = model(inputs)\n",
        "    #print(outputs)\n",
        "    loss = criterion(outputs, labels)\n",
        "    print(loss)\n",
        "    loss.backward()\n",
        "    scheduler.step(loss) # 传入loss\n",
        "    optimizer.step()    \n",
        "    lr = optimizer.param_groups[0]['lr'] # ReduceLROnPlateau里没有scheduler.get_lr()\n",
        "    print('epoch',epoch,',lr',lr)\n",
        "    y.append(lr)\n",
        "plt.plot(x,y)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor(0.6010, grad_fn=<NllLossBackward>)\n",
            "epoch 0 ,lr 10\n",
            "tensor(229.9586, grad_fn=<NllLossBackward>)\n",
            "epoch 1 ,lr 10\n",
            "tensor(39.6332, grad_fn=<NllLossBackward>)\n",
            "epoch 2 ,lr 10\n",
            "tensor(0.0642, grad_fn=<NllLossBackward>)\n",
            "epoch 3 ,lr 10\n",
            "tensor(0.0395, grad_fn=<NllLossBackward>)\n",
            "epoch 4 ,lr 10\n",
            "tensor(2.9802e-06, grad_fn=<NllLossBackward>)\n",
            "epoch 5 ,lr 10\n",
            "tensor(2.9802e-06, grad_fn=<NllLossBackward>)\n",
            "epoch 6 ,lr 10\n",
            "tensor(2.9802e-06, grad_fn=<NllLossBackward>)\n",
            "epoch 7 ,lr 10\n",
            "tensor(2.9802e-06, grad_fn=<NllLossBackward>)\n",
            "epoch 8 ,lr 10\n",
            "tensor(2.9802e-06, grad_fn=<NllLossBackward>)\n",
            "epoch 9 ,lr 1.0\n",
            "tensor(2.9802e-06, grad_fn=<NllLossBackward>)\n",
            "epoch 10 ,lr 1.0\n",
            "tensor(2.9802e-06, grad_fn=<NllLossBackward>)\n",
            "epoch 11 ,lr 1.0\n",
            "tensor(2.9802e-06, grad_fn=<NllLossBackward>)\n",
            "epoch 12 ,lr 1.0\n",
            "tensor(2.9802e-06, grad_fn=<NllLossBackward>)\n",
            "epoch 13 ,lr 0.1\n",
            "tensor(2.9802e-06, grad_fn=<NllLossBackward>)\n",
            "epoch 14 ,lr 0.1\n",
            "tensor(2.9802e-06, grad_fn=<NllLossBackward>)\n",
            "epoch 15 ,lr 0.1\n",
            "tensor(2.9802e-06, grad_fn=<NllLossBackward>)\n",
            "epoch 16 ,lr 0.1\n",
            "tensor(2.9802e-06, grad_fn=<NllLossBackward>)\n",
            "epoch 17 ,lr 0.010000000000000002\n",
            "tensor(2.9802e-06, grad_fn=<NllLossBackward>)\n",
            "epoch 18 ,lr 0.010000000000000002\n",
            "tensor(2.9802e-06, grad_fn=<NllLossBackward>)\n",
            "epoch 19 ,lr 0.010000000000000002\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f6b25b96780>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAUnUlEQVR4nO3dfXBcV3nH8d8jrXZla9eO7btxnBfj\npGQCFBowSspLYBhCaAg0KUymY1qaBKdjaAuFDh1Ihw4wlOkAbZnQloExwSSkGQgEUlIIhfBW6NCY\nyMZJ7BjiEEJw4tiyTfxuy5Ke/rFXlixLtrJ3tXfPud/PjEar3bu6j69WP1+dvc855u4CAISnK+8C\nAADNIcABIFAEOAAEigAHgEAR4AAQqFI7d5YkiS9btqyduwSA4K1bt26nu9cn39/WAF+2bJkGBgba\nuUsACJ6Z/Xqq+xlCAYBAEeAAECgCHAACRYADQKAIcAAI1CkD3MzWmNkOM9s44b6FZnaPmW1JPy+Y\n3TIBAJPN5Az8ZkmXT7rvBknfc/fzJX0v/RoA0EanvA7c3X9kZssm3X2VpFelt2+R9ENJ72thXce5\n82db9avBA7P17RGAC86Yp9f/3pK8ywA6SrONPIvdfVt6+ylJi6fb0MxWSVolSUuXLm1qZ/91/zb9\n4Bc7mnouwucuVUpduuIFZ8jM8i4H6BiZOzHd3c1s2lUh3H21pNWS1N/f39TqEWuuu6jJ6hCDm378\nqD7yzc3ad2RY83p78i4H6BjNXoWy3cyWSFL6mdNjzJqkWpEkDe47knMlQGdpNsDvknRtevtaSV9v\nTTnAicYCfCcBDhxnJpcRflHS/0m6wMy2mtn1kj4q6TIz2yLpNenXwKxIamVJ0s79QzlXAnSWmVyF\n8uZpHrq0xbUAUzp2Br6fM3BgIjox0fEWzC2rywhwYDICHB2vu8u0sK9CgAOTEOAIQlIta3AfY+DA\nRAQ4glCvcQYOTEaAIwhJlQAHJiPAEYSkWtbO/Ufk3lQzLxAlAhxBSKoVHT46qgNDI3mXAnQMAhxB\noBsTOBEBjiAkNZp5gMkIcAQhqY610xPgwBgCHEGoj81IyHwowDEEOIKwsK8sM8bAgYkIcASh1N2l\nBXPLDKEAExDgCMbYteAAGghwBKPRjckYODCGAEcwaKcHjkeAIxhJtcKbmMAEBDiCkdTKOjA0okO0\n0wOSCHAEhKXVgOMR4AjGeDMPAQ5IBDgCwoRWwPEIcAQjqY3Nh8KlhIBEgCMgi/oYAwcmIsARjHKp\nS/Pn9BDgQIoAR1BopwfGEeAISr1W0c59jIEDEgGOwNBOD4wjwBGUpFrRIJcRApIIcASmXqto35Fh\nHT5KOz1AgCMorI0JjCPAEZTx+VB4IxPIFOBm9jdmtsnMNprZF82st1WFAVOhnR4Y13SAm9lZkv5a\nUr+7P19St6QVrSoMmEpSoxsTGJN1CKUkaY6ZlSTNlfRk9pKA6S3qYwwcGNN0gLv7E5L+WdLjkrZJ\n2uPu35m8nZmtMrMBMxsYHBxsvlJAUm9Pt2q9JcbAAWUbQlkg6SpJ50o6U1Kfmb1l8nbuvtrd+929\nv16vN18pkKpXK8wJDijbEMprJP3K3Qfd/aikr0l6WWvKAqbH2phAQ5YAf1zSS8xsrpmZpEslbW5N\nWcD0khoTWgFStjHwtZLukLRe0oPp91rdorqAaTXmQ2EMHChlebK7f1DSB1tUCzAjSbWiPYeOamh4\nVOUSvWgoLl79CM5YM8+uAwyjoNgIcATn2HwozAuOgiPAERy6MYEGAhzBqadDKFwLjqIjwBGc8RkJ\nCXAUGwGO4Mwpd6uv3M0YOAqPAEeQkhprYwIEOILE4sYAAY5AJVXa6QECHEGinR4gwBGopFrRbw8O\naXhkNO9SgNwQ4AhSUqvIXdp9gLNwFBcBjiDV03Z6mnlQZAQ4gjTezMMZOIqLAEeQjgU4K/OgwAhw\nBIkJrQACHIHqK3ert6eLAEehEeAIkplxLTgKjwBHsJJqRYOMgaPACHAEi/lQUHQEOIJVrzEfCoqN\nAEewkmpFuw8MaWTU8y4FyAUBjmAl1YpGaadHgRHgCFada8FRcAQ4gsXamCg6AhzBStIJrQhwFBUB\njmAda6dncWMUFAGOYNUqJZVLtNOjuAhwBMvMVK9WmBMchUWAI2iNxY0ZQkExZQpwMzvNzO4ws5+b\n2WYze2mrCgNmIqlWmBMchZX1DPyTkv7b3Z8j6UJJm7OXBMwc86GgyErNPtHM5kt6paTrJMndhyTx\ntyzaKqmVtevAkEZHXV1dlnc5QFtlOQM/V9KgpM+b2c/M7CYz65u8kZmtMrMBMxsYHBzMsDvgREm1\nopFR19OHjuZdCtB2WQK8JGm5pE+7+4skHZB0w+SN3H21u/e7e3+9Xs+wO+BEdGOiyLIE+FZJW919\nbfr1HWoEOtA2LG6MIms6wN39KUm/MbML0rsulfRQS6oCZqhea7TTcy04iqjpNzFT75R0m5mVJT0q\n6a3ZSwJmbnwIhffPUTyZAtzdN0jqb1EtwDM2f06PerqNMXAUEp2YCJqZaVEfzTwoJgIcwUtYGxMF\nRYAjeI1uTMbAUTwEOIJHOz2KigBH8JJqRbv2D8md1elRLAQ4gpdUyxoaGdXeQ8N5lwK0FQGO4I2t\nTk8zD4qGAEfwmA8FRUWAI3gEOIqKAEfwkmpjPhSaeVA0BDiCt2BuWd1dxhg4CocAR/C6ukwL+8ra\nuY9mHhQLAY4o0MyDIiLAEYWkynwoKB4CHFGoMx8KCogARxSSWkWD+4/QTo9CIcARhaRa1tDwqPYd\noZ0exUGAIwosbowiIsARBdbGRBER4IgC7fQoIgIcUUhqaTs9AY4CIcARhUV9FXUZY+AoFgIcUehO\n2+kHGQNHgRDgiAbt9CgaAhzRIMBRNAQ4osF8KCgaAhzRSKoVppRFoRDgiEZSq+jQ0REdoJ0eBUGA\nIxo086BoCHBE49jamAQ4CoIARzTGzsAHGQdHQWQOcDPrNrOfmdk3WlEQ0Kx6jSEUFEsrzsDfJWlz\nC74PkMnCPoZQUCyZAtzMzpb0ekk3taYcoHk93V1aMLeHAEdhZD0Dv1HSeyWNTreBma0yswEzGxgc\nHMy4O+DkuBYcRdJ0gJvZGyTtcPd1J9vO3Ve7e7+799fr9WZ3B8wI7fQokixn4C+XdKWZPSbpS5Je\nbWb/0ZKqgCYlNQIcxdF0gLv737n72e6+TNIKSd9397e0rDKgCY35UBhCQTFwHTiiklQr2n9kWIeP\njuRdCjDrWhLg7v5Dd39DK74XkEX9WDMPwyiIH2fgiMrY2piDjIOjAAhwROXYhFacgaMACHBEZXxG\nQt7IRPwIcERlETMSokAIcESlUurWvN4SAY5CIMARHZp5UBQEOKLDfCgoCgIc0akzHwoKggBHdJJq\nmevAUQgEOKKTVCvad5h2esSPAEd0knRptV0HGAdH3AhwRIduTBQFAY7oJDTzoCAIcERnvJ2eAEfc\nCHBEp15jPhQUAwGO6PT2dKtaKTEnOKJHgCNKjaXVCHDEjQBHlFidHkVAgCNK9VqFMXBEjwBHlDgD\nRxEQ4IhSUq3o6YNHdXRkNO9SgFlDgCNKY4sb72IYBREjwBElmnlQBAQ4ojQW4Ewri5gR4IhSnQmt\nUAAEOKI0NgbOpYSIGQGOKM0tlzS33M0YOKJGgCNaXAuO2BHgiBbzoSB2BDiilVQr2rmPMXDEq+kA\nN7NzzOwHZvaQmW0ys3e1sjAgq6TGEAriVsrw3GFJ73H39WZWk7TOzO5x94daVBuQSVKtaPfBIQ2P\njKrUzR+biE/Tr2p33+bu69Pb+yRtlnRWqwoDsqpXy3KXdrM6PSLVktMSM1sm6UWS1rbi+wGtQDcm\nYpc5wM2sKumrkt7t7nuneHyVmQ2Y2cDg4GDW3QEzlrA2JiKXKcDNrEeN8L7N3b821Tbuvtrd+929\nv16vZ9kd8IwktNMjclmuQjFJn5O02d0/0bqSgNZIqmPt9AQ44pTlDPzlkv5M0qvNbEP6cUWL6gIy\nq1ZKqpS6CHBEq+nLCN39fyVZC2sBWsrM0nZ6xsARJy6ORdRo5kHMCHBErV4ta5A3MREpAhxRYwgF\nMSPAEbWkWtHuA0c0Mup5lwK0HAGOqCXVskZd+u1BzsIRHwIcURvvxmQcHPEhwBG18W5MzsARHwIc\nUTsW4JyBI0IEOKJWJ8ARMQIcUZs3p6RydxdTyiJKBDiiZmZaVC0zBo4oEeCIXqOZhzNwxIcAR/SS\napkAR5SyLGoMBCGpVrT+8ad1y08ey7uUpp29YI4ufe7ivMtAhyHAEb3nLpmnr6zbqg/etSnvUjL5\n+9c/V3/+ivPyLgMdhABH9FZecq7etPwshTodirvr/Xdu1Ee+uVmnz+vVlReemXdJ6BAEOArhtLnl\nvEvI5MYVL9Q1a36q93x5gxbOLeuS85O8S0IH4E1MIAC9Pd367DX9Oi+p6m23DmjjE3vyLgkdgAAH\nAjF/To9uXnmR5s/p0XWfv0+P7zqYd0nIGQEOBGTJ/Dn6wvUX6+jIqK5Zs5bLIwuOAAcC8+zTa1pz\nXb+27Tms62++TweODOddEnJCgAMBevGzFurf/2S5Hnxij/7itvU6OjKad0nIAQEOBOqy5y3WP77x\nBfrRw4N63x0PyD3Q6yTRNC4jBAK24uKl2rHviD5xz8M6fV6vbnjdc/IuCW1EgAOBe+ern63tew/r\nM//zS51eq2jlJefmXRLahAAHAmdm+vBVz9eu/UP6h28+pHqtoj+kW7MQGAMHItDdZbpxxQt10bMW\n6j1fvl8/eWRn3iWhDQhwIBJj3ZrnJn1ades6bXqSbs3YEeBARObPbXRrzust6brP36ff7KZbM2YE\nOBCZJfPn6JaVF2toeFTXrPmpdtGtGS0CHIjQ+Ysb3ZpPPn1IK28Z0MEhujVjlOkqFDO7XNInJXVL\nusndP9qSqgBkNtat+bZbB/Tm1ffqgjNqeZfUtFJ3lxbXerVkfq+WnJZ+nj9HfZViX0jX9L/ezLol\nfUrSZZK2SrrPzO5y94daVRyAbC573mJ9/OoLdeN3H9aPt4R7ZcrQ8Kh2HRg64f5ab0lnzp+jM+b3\n6szTenXGvDmFCvks/7KLJT3i7o9Kkpl9SdJVkghwoINc/eKzdfWLz867jMyODI9ox94j2rbnsLbt\nOdT4/HT6ec9hbXpy75SzM9Z6S6rXKuo2y6HqcZ+79iItXTS3pd8zS4CfJek3E77eKun3J29kZqsk\nrZKkpUuXZtgdgCKrlLp1zsK5Omfh9CE4NDyq7XsPHxfyT+05rMF9R+TKd66Ycqn1bznO+t8W7r5a\n0mpJ6u/vZ7YdALOmXOo6ZcjHJMt/CU9IOmfC12en9wEA2iBLgN8n6XwzO9fMypJWSLqrNWUBAE6l\n6SEUdx82s3dI+rYalxGucfdNLasMAHBSmcbA3f1uSXe3qBYAwDNAJyYABIoAB4BAEeAAECgCHAAC\nZe1cydrMBiX9usmnJ5I6eTIH6suG+rKhvmw6vb5nuXt98p1tDfAszGzA3fvzrmM61JcN9WVDfdl0\nen3TYQgFAAJFgANAoEIK8NV5F3AK1JcN9WVDfdl0en1TCmYMHABwvJDOwAEAExDgABCojgtwM7vc\nzH5hZo+Y2Q1TPF4xs9vTx9ea2bI21naOmf3AzB4ys01m9q4ptnmVme0xsw3pxwfaVV+6/8fM7MF0\n3wNTPG5m9q/p8XvAzJa3sbYLJhyXDWa218zePWmbth4/M1tjZjvMbOOE+xaa2T1mtiX9vGCa516b\nbrPFzK5tY33/ZGY/T39+d5rZadM896SvhVms70Nm9sSEn+EV0zz3pL/rs1jf7RNqe8zMNkzz3Fk/\nfpm5e8d8qDEt7S8lnSepLOl+Sc+btM1fSvpMenuFpNvbWN8SScvT2zVJD09R36skfSPHY/iYpOQk\nj18h6VuSTNJLJK3N8Wf9lBoNCrkdP0mvlLRc0sYJ931c0g3p7RskfWyK5y2U9Gj6eUF6e0Gb6nut\npFJ6+2NT1TeT18Is1vchSX87g5//SX/XZ6u+SY//i6QP5HX8sn502hn4sYWS3X1I0thCyRNdJemW\n9PYdki41a89qpe6+zd3Xp7f3SdqsxtqgIblK0he84V5Jp5nZkhzquFTSL9292c7clnD3H0naPenu\nia+xWyT90RRP/QNJ97j7bnf/raR7JF3ejvrc/TvuPpx+ea8aq2HlYprjNxMz+V3P7GT1pbnxx5K+\n2Or9tkunBfhUCyVPDshj26Qv4j2SFrWlugnSoZsXSVo7xcMvNbP7zexbZva7bS1McknfMbN16YLS\nk83kGLfDCk3/i5Pn8ZOkxe6+Lb39lKTFU2zTKcdxpRp/UU3lVK+F2fSOdIhnzTRDUJ1w/F4habu7\nb5nm8TyP34x0WoAHwcyqkr4q6d3uvnfSw+vVGBa4UNK/SfrPNpd3ibsvl/Q6SX9lZq9s8/5PKV2C\n70pJX5ni4byP33G88bd0R15ra2bvlzQs6bZpNsnrtfBpSb8j6YWStqkxTNGJ3qyTn313/O9SpwX4\nTBZKPraNmZUkzZe0qy3VNfbZo0Z43+buX5v8uLvvdff96e27JfWYWdKu+tz9ifTzDkl3qvGn6kSd\nsBj16yStd/ftkx/I+/ilto8NK6Wfd0yxTa7H0cyuk/QGSX+a/idzghm8FmaFu2939xF3H5X02Wn2\nm/fxK0l6k6Tbp9smr+P3THRagM9koeS7JI2943+1pO9P9wJutXTM7HOSNrv7J6bZ5oyxMXkzu1iN\nY9yW/2DMrM/MamO31Xiza+Okze6SdE16NcpLJO2ZMFzQLtOe+eR5/CaY+Bq7VtLXp9jm25Jea2YL\n0iGC16b3zTozu1zSeyVd6e4Hp9lmJq+F2apv4nsqb5xmv3kviv4aST93961TPZjn8XtG8n4XdfKH\nGldJPKzGO9TvT+/7sBovVknqVeNP70ck/VTSeW2s7RI1/px+QNKG9OMKSW+X9PZ0m3dI2qTGu+r3\nSnpZG+s7L93v/WkNY8dvYn0m6VPp8X1QUn+bf759agTy/An35Xb81PiPZJuko2qMw16vxnsq35O0\nRdJ3JS1Mt+2XdNOE565MX4ePSHprG+t7RI3x47HX4NhVWWdKuvtkr4U21Xdr+tp6QI1QXjK5vvTr\nE37X21Ffev/NY6+5Cdu2/fhl/aCVHgAC1WlDKACAGSLAASBQBDgABIoAB4BAEeAAECgCHAACRYAD\nQKD+H4rPEJOrflJgAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    }
  ]
}