12_卷积神经网络.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "12_卷积神经网络.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mengcius/pytorch-learn/blob/master/12_%E5%8D%B7%E7%A7%AF%E7%A5%9E%E7%BB%8F%E7%BD%91%E7%BB%9C.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IJhnBzUljKKK",
        "colab_type": "text"
      },
      "source": [
        "## 12_卷积神经网络"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gAwmT4BCIRGH",
        "colab_type": "text"
      },
      "source": [
        "### 卷积网络层"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Kt0lJLZXDVL0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FTxGP0MIjnAz",
        "colab_type": "text"
      },
      "source": [
        "类风格实现（nn.Xxx）：要先实例化再调用"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5rtOBQFKLGeP",
        "colab_type": "code",
        "outputId": "1de3a0f4-150c-4d15-9587-420982195510",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 33
        }
      },
      "source": [
        "x=torch.rand(1,1,28,28) #核个数，输入通道数，核大小\n",
        "layer=nn.Conv2d(1,3,kernel_size=3,stride=1,padding=0) #卷积层，类风格实现：输入通道数，核数量\n",
        "\n",
        "out=layer.forward(x) #卷积前向运算（不建议用）\n",
        "out.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([1, 3, 26, 26])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Nrf4O7PwLIMA",
        "colab_type": "code",
        "outputId": "c79e2881-dd87-478a-c9be-b3310264b1d3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 33
        }
      },
      "source": [
        "out=layer(x) #卷积前向运算，用封装好的方法，直接是实例+()\n",
        "out.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([1, 3, 26, 26])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c1vCMXmALGx4",
        "colab_type": "code",
        "outputId": "e06f253b-d2bf-4c5e-e590-444ea0488397",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 251
        }
      },
      "source": [
        "layer.weight #查看网络层里的参数，[3,1,3,3]核数量 输入通道 核大小，w b会自动跟新"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Parameter containing:\n",
              "tensor([[[[-0.3300,  0.0198, -0.1243],\n",
              "          [ 0.0597, -0.0743,  0.2338],\n",
              "          [-0.2747,  0.2101,  0.2337]]],\n",
              "\n",
              "\n",
              "        [[[-0.2405,  0.1064,  0.0402],\n",
              "          [ 0.3295,  0.1337,  0.0645],\n",
              "          [ 0.0190, -0.0427,  0.1558]]],\n",
              "\n",
              "\n",
              "        [[[-0.3241,  0.0522, -0.0070],\n",
              "          [ 0.0812, -0.2457, -0.1421],\n",
              "          [ 0.1151,  0.0057, -0.1088]]]], requires_grad=True)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_wnwntTvfg17",
        "colab_type": "text"
      },
      "source": [
        "函数风格实现（F.xxx）：自己管理这里tensor、运算过程，仅仅用了它GUP加速的过程，不用实例化。\n",
        "\n",
        "pytorch默认大写是类，小写是接口"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bTGcgMjsfEaj",
        "colab_type": "code",
        "outputId": "f98e2b78-bcdc-4a91-ecb7-9a07046c5493",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 33
        }
      },
      "source": [
        "w=torch.rand(16,3,5,5) #核个数，输入通道数，核大小\n",
        "b=torch.rand(16)\n",
        "x=torch.randn(1,3,28,28) #核个数，输入通道数，核大小\n",
        "\n",
        "out=F.conv2d(x,w,b,stride=1,padding=1) #CNN的一层，底层函数风格实现\n",
        "out.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([1, 16, 26, 26])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g4jjUtXhkrHH",
        "colab_type": "text"
      },
      "source": [
        "### 上下采样"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z8SBvZsWuBGw",
        "colab_type": "text"
      },
      "source": [
        "池化"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T_NLxKaQhvNy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x=torch.rand(1,16,14,14)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MP8K20vMllkW",
        "colab_type": "code",
        "outputId": "0af53c62-d266-4cf5-f1cb-985cc6171d5c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 33
        }
      },
      "source": [
        "layer=nn.MaxPool2d(2,stride=2) #最大值池化，类风格实现：窗口大小，步长\n",
        "out=layer(x)\n",
        "out.shape #减半了"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([1, 16, 7, 7])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TmrCHIp6sy1J",
        "colab_type": "code",
        "outputId": "5c29f13a-8421-4893-f7ef-69ad7c10a209",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 33
        }
      },
      "source": [
        "layer=F.avg_pool2d(x,2,stride=2) #最大值池化，函数风格实现\n",
        "out.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([1, 16, 7, 7])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rm3q9v03t9r-",
        "colab_type": "text"
      },
      "source": [
        "上采样"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RHnFO52etNu3",
        "colab_type": "code",
        "outputId": "232a45a4-c41f-45b8-b005-4cfe3ae75dbc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 33
        }
      },
      "source": [
        "out=F.interpolate(x,scale_factor=3,mode='nearest') #插值法上采样：tensor，放大倍数，插值方法（近邻、双性、三性）\n",
        "out.shape #"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([1, 16, 42, 42])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e9HjPBYYuFAs",
        "colab_type": "text"
      },
      "source": [
        "### ReLU函数"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OMpiiwO4tNk8",
        "colab_type": "code",
        "outputId": "dc1c35f5-aa1c-4712-db81-bbeecfab1282",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 33
        }
      },
      "source": [
        "x.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([1, 16, 14, 14])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yMmq3ww5uJtd",
        "colab_type": "code",
        "outputId": "5fe2010d-2d8e-48aa-a80a-d9a307c129bc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 33
        }
      },
      "source": [
        "layer=nn.ReLU(inplace=True) #ReLU函数，维度不变，inplace=True就是让结果替换原来的x的地址，节省内存\n",
        "out=layer(x)\n",
        "out.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([1, 16, 14, 14])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MnOg9YbDuWDw",
        "colab_type": "code",
        "outputId": "9a7f361b-af8a-4798-98d7-339d45bca04f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 33
        }
      },
      "source": [
        "layer=F.relu(x)\n",
        "out.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([1, 16, 14, 14])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dDwGj_PTKXL7",
        "colab_type": "text"
      },
      "source": [
        "### Batch Normalization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Uun8f0kUX2Rp",
        "colab_type": "text"
      },
      "source": [
        "一维"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D34Ig_Xluxtt",
        "colab_type": "code",
        "outputId": "6fc78d3f-3d94-4494-f48f-d457da6c0d71",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 33
        }
      },
      "source": [
        "x=torch.rand(100,16,784) #均匀分布的均值是0.5，方差为1\n",
        "layer=nn.BatchNorm1d(16) #Batch Normal;用1d是因为1维的feature 28*28，16就是通道数\n",
        "out=layer(x) #统计出当前x的u和o^2\n",
        "out.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([100, 16, 784])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c8Fh5JpyKsT2",
        "colab_type": "code",
        "outputId": "97e99f00-0aaa-4ccc-aa5e-06230a5cd4ab",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 50
        }
      },
      "source": [
        "layer.running_mean #更新全局的running_u，当只有一次运算时就是u，都在0.5附近"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([0.0500, 0.0499, 0.0498, 0.0500, 0.0500, 0.0499, 0.0497, 0.0499, 0.0498,\n",
              "        0.0500, 0.0500, 0.0498, 0.0499, 0.0498, 0.0499, 0.0500])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sllk9Nr1L7ml",
        "colab_type": "code",
        "outputId": "bf95629d-1a47-4d41-cb22-114cbdd0fa4f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 50
        }
      },
      "source": [
        "layer.running_var #更新全局的running_o^2，方差为1左右"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([0.9083, 0.9084, 0.9083, 0.9083, 0.9083, 0.9083, 0.9084, 0.9083, 0.9083,\n",
              "        0.9083, 0.9083, 0.9083, 0.9083, 0.9083, 0.9084, 0.9083])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DvwOjNUIRIFo",
        "colab_type": "text"
      },
      "source": [
        "二维"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-TfCeeCiL-vb",
        "colab_type": "code",
        "outputId": "0bda665c-6d19-41c5-afdf-e20f2eabd9b7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 33
        }
      },
      "source": [
        "x=torch.rand(1,16,7,7)\n",
        "x.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([1, 16, 7, 7])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tjf3xBHaQJ-5",
        "colab_type": "code",
        "outputId": "3ab350b4-31b3-401a-8145-563e1d5cccec",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 33
        }
      },
      "source": [
        "layer=nn.BatchNorm2d(16)\n",
        "out=layer(x)\n",
        "out.shape #大小不变"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([1, 16, 7, 7])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "padUzDWgMkhl",
        "colab_type": "code",
        "outputId": "8100974b-da78-4710-a04d-8b75d24ebfea",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 84
        }
      },
      "source": [
        "layer.weight #相当于r，需要梯度"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Parameter containing:\n",
              "tensor([0.3847, 0.9785, 0.7068, 0.7617, 0.5684, 0.7968, 0.0674, 0.8322, 0.9962,\n",
              "        0.2803, 0.3346, 0.1811, 0.4323, 0.6628, 0.6418, 0.6469],\n",
              "       requires_grad=True)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v5zwLZ6KRv4w",
        "colab_type": "code",
        "outputId": "20bb9289-8e76-4b41-a66a-8ecbaf911ff3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67
        }
      },
      "source": [
        "layer.bias #相当于B"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Parameter containing:\n",
              "tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       requires_grad=True)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V8op_RJyR0sx",
        "colab_type": "code",
        "outputId": "d6bda720-ef16-4f2d-c47f-5278ac719a6e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 472
        }
      },
      "source": [
        "vars(layer) #Class variables"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'_backend': <torch.nn.backends.thnn.THNNFunctionBackend at 0x7f5edd095668>,\n",
              " '_backward_hooks': OrderedDict(),\n",
              " '_buffers': OrderedDict([('running_mean',\n",
              "               tensor([0.0450, 0.0558, 0.0435, 0.0471, 0.0516, 0.0510, 0.0461, 0.0421, 0.0493,\n",
              "                       0.0516, 0.0499, 0.0498, 0.0552, 0.0517, 0.0464, 0.0504])),\n",
              "              ('running_var',\n",
              "               tensor([0.9090, 0.9087, 0.9091, 0.9075, 0.9097, 0.9088, 0.9078, 0.9065, 0.9085,\n",
              "                       0.9097, 0.9072, 0.9109, 0.9090, 0.9070, 0.9080, 0.9054])),\n",
              "              ('num_batches_tracked', tensor(1))]),\n",
              " '_forward_hooks': OrderedDict(),\n",
              " '_forward_pre_hooks': OrderedDict(),\n",
              " '_load_state_dict_pre_hooks': OrderedDict(),\n",
              " '_modules': OrderedDict(),\n",
              " '_parameters': OrderedDict([('weight', Parameter containing:\n",
              "               tensor([0.3847, 0.9785, 0.7068, 0.7617, 0.5684, 0.7968, 0.0674, 0.8322, 0.9962,\n",
              "                       0.2803, 0.3346, 0.1811, 0.4323, 0.6628, 0.6418, 0.6469],\n",
              "                      requires_grad=True)), ('bias', Parameter containing:\n",
              "               tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "                      requires_grad=True))]),\n",
              " '_state_dict_hooks': OrderedDict(),\n",
              " 'affine': True,\n",
              " 'eps': 1e-05,\n",
              " 'momentum': 0.1,\n",
              " 'num_features': 16,\n",
              " 'track_running_stats': True,\n",
              " 'training': True}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fHz9aF4lTTEx",
        "colab_type": "text"
      },
      "source": [
        "测试里的Batch Norm"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V8otessgSGI-",
        "colab_type": "code",
        "outputId": "dec3f847-06d8-46b1-cbe9-e7c94d4884fb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 33
        }
      },
      "source": [
        "layer.eval()\n",
        "nn.BatchNorm1d(16,eps=1e-05,momentum=0.1,affine=True,track_running_stats=True)\n",
        "out=layer(x)\n",
        "out.shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([1, 16, 7, 7])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R17F4Qx8YetA",
        "colab_type": "text"
      },
      "source": [
        "### ResNet的残差单元\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1vOyNV5MYf-c",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import  torch\n",
        "from    torch import  nn\n",
        "from    torch.nn import functional as F\n",
        "\n",
        "# from    torchvision.models import resnet18\n",
        "\n",
        "# 残差单元（适用于浅层网络，中间有两层卷积层）\n",
        "class ResBlk(nn.Module):\n",
        "    \"\"\"\n",
        "    resnet block\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, ch_in, ch_out):\n",
        "        \"\"\"\n",
        "        :param ch_in:\n",
        "        :param ch_out:\n",
        "        \"\"\"\n",
        "        super(ResBlk, self).__init__()\n",
        "\n",
        "        self.conv1 = nn.Conv2d(ch_in, ch_out, kernel_size=3, stride=1, padding=1)\n",
        "        self.bn1 = nn.BatchNorm2d(ch_out)\n",
        "        self.conv2 = nn.Conv2d(ch_out, ch_out, kernel_size=3, stride=1, padding=1)\n",
        "        self.bn2 = nn.BatchNorm2d(ch_out)\n",
        "\n",
        "        self.extra = nn.Sequential()\n",
        "        if ch_out != ch_in: # 确保x跳连的输出是ch_out，以便与卷积层像素相加\n",
        "            # [b, ch_in, h, w] => [b, ch_out, h, w]\n",
        "            self.extra = nn.Sequential(\n",
        "                nn.Conv2d(ch_in, ch_out, kernel_size=1, stride=1),\n",
        "                nn.BatchNorm2d(ch_out)\n",
        "            )\n",
        "\n",
        "    def forward(self, x):\n",
        "        \"\"\"\n",
        "        :param x: [b, ch, h, w]\n",
        "        :return:\n",
        "        \"\"\"\n",
        "        out = F.relu(self.bn1(self.conv1(x))) # 这里加relu\n",
        "        out = self.bn2(self.conv2(out))\n",
        "        # short cut，extra module: [b, ch_in, h, w] => [b, ch_out, h, w]\n",
        "        out = self.extra(x) + out # 元素相加（element-wise add）\n",
        "\n",
        "        return out"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tZaju0Oziocs",
        "colab_type": "text"
      },
      "source": [
        "### ResNet10删减版网络\n",
        "1、跑CIFAR10，test acc几乎没变，train loss在260 loss就为0了，0.58：\n",
        "\n",
        "259 loss: 0.0\n",
        "259 acc: 0.5816\n",
        "260 loss: 0.0\n",
        "260 acc: 0.5812\n",
        "261 loss: 0.0\n",
        "261 acc: 0.579\n",
        "\n",
        "699 loss: 0.0\n",
        "699 acc: 0.5886\n",
        "700 loss: 0.0\n",
        "700 acc: 0.5784\n",
        "701 loss: 0.0\n",
        "701 acc: 0.582\n",
        "702 loss: 2.384185791015625e-06\n",
        "702 acc: 0.5793\n",
        "\n",
        "2、数据增强全加，效果不好，0.55：\n",
        "\n",
        "94 loss: 1.1804752349853516 94 acc: 0.5292 \n",
        "95 loss: 0.8475046157836914 95 acc: 0.5538 \n",
        "96 loss: 1.3808478116989136 96 acc: 0.5405 \n",
        "97 loss: 1.1416053771972656 97 acc: 0.5345\n",
        "\n",
        "3、数据增强只用水平翻转，有效，0.61：\n",
        "\n",
        "98 acc: 0.6101\n",
        "99 loss: 0.2222895622253418\n",
        "99 acc: 0.6141\n",
        "100 loss: 0.017084121704101562\n",
        "\n",
        "199 loss: 0.0027701258659362793\n",
        "199 acc: 0.6124\n",
        "200 loss: 0.18000590801239014\n",
        "200 acc: 0.6092\n",
        "\n",
        "500 loss: 7.152557373046875e-07\n",
        "500 acc: 0.6159\n",
        "501 loss: 0.3125884532928467\n",
        "501 acc: 0.6105\n",
        "\n",
        "595 loss: 0.0\n",
        "595 acc: 0.612\n",
        "596 loss: 0.0\n",
        "596 acc: 0.6182\n",
        "597 loss: 0.0\n",
        "597 acc: 0.6139\n",
        "\n",
        "945 loss: 0.0\n",
        "945 acc: 0.6157\n",
        "946 loss: 0.0\n",
        "946 acc: 0.6143"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lOj-v6Jkim-0",
        "colab_type": "code",
        "outputId": "4e436f96-e248-47fa-ee60-08ad87aa559c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "import  torch\n",
        "from    torch import  nn\n",
        "from    torch.nn import functional as F\n",
        "from    torch.utils.data import DataLoader\n",
        "from    torchvision import datasets\n",
        "from    torchvision import transforms\n",
        "from    torch import nn, optim\n",
        "\n",
        "# from    torchvision.models import resnet18\n",
        "\n",
        "# 残差单元（适用于浅层网络，中间有两层卷积层）\n",
        "class ResBlk(nn.Module):\n",
        "    \"\"\"\n",
        "    resnet block\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, ch_in, ch_out):\n",
        "        \"\"\"\n",
        "        :param ch_in:\n",
        "        :param ch_out:\n",
        "        \"\"\"\n",
        "        super(ResBlk, self).__init__()\n",
        "\n",
        "        self.conv1 = nn.Conv2d(ch_in, ch_out, kernel_size=3, stride=1, padding=1)\n",
        "        self.bn1 = nn.BatchNorm2d(ch_out)\n",
        "        self.conv2 = nn.Conv2d(ch_out, ch_out, kernel_size=3, stride=1, padding=1)\n",
        "        self.bn2 = nn.BatchNorm2d(ch_out)\n",
        "\n",
        "        self.extra = nn.Sequential()\n",
        "        if ch_out != ch_in: # 确保x跳连的输出是ch_out，以便与卷积层像素相加\n",
        "            # [b, ch_in, h, w] => [b, ch_out, h, w]\n",
        "            self.extra = nn.Sequential(\n",
        "                nn.Conv2d(ch_in, ch_out, kernel_size=1, stride=1),\n",
        "                nn.BatchNorm2d(ch_out)\n",
        "            )\n",
        "\n",
        "    def forward(self, x):\n",
        "        \"\"\"\n",
        "        :param x: [b, ch, h, w]\n",
        "        :return:\n",
        "        \"\"\"\n",
        "        out = F.relu(self.bn1(self.conv1(x))) # 这里加relu\n",
        "        out = self.bn2(self.conv2(out))\n",
        "        # short cut，extra module: [b, ch_in, h, w] => [b, ch_out, h, w]\n",
        "        out = self.extra(x) + out # 元素相加（element-wise add）\n",
        "\n",
        "        return out\n",
        "\n",
        "\n",
        "# ResNet10删减版网络结构\n",
        "class ResNet18(nn.Module):\n",
        "\n",
        "    def __init__(self):\n",
        "        super(ResNet18, self).__init__()\n",
        "\n",
        "        self.conv1 = nn.Sequential(\n",
        "            nn.Conv2d(3, 16, kernel_size=3, stride=1, padding=1),\n",
        "            nn.BatchNorm2d(16)\n",
        "        )\n",
        "        # followed 4 blocks\n",
        "        # [b, 64, h, w] => [b, 128, h ,w]\n",
        "        self.blk1 = ResBlk(16, 16)\n",
        "        # [b, 128, h, w] => [b, 256, h, w]\n",
        "        self.blk2 = ResBlk(16, 32)\n",
        "        # # [b, 256, h, w] => [b, 512, h, w]\n",
        "        # self.blk3 = ResBlk(128, 256)\n",
        "        # # [b, 512, h, w] => [b, 1024, h, w]\n",
        "        # self.blk4 = ResBlk(256, 512)\n",
        "\n",
        "        self.outlayer = nn.Linear(32*32*32, 10)\n",
        "\n",
        "    def forward(self, x):\n",
        "        \"\"\"\n",
        "        :param x:\n",
        "        :return:\n",
        "        \"\"\"\n",
        "        x = F.relu(self.conv1(x))\n",
        "\n",
        "        # [b, 64, h, w] => [b, 1024, h, w]\n",
        "        x = self.blk1(x)\n",
        "        x = self.blk2(x)\n",
        "        # x = self.blk3(x)\n",
        "        # x = self.blk4(x)\n",
        "\n",
        "        # print(x.shape)\n",
        "        x = x.view(x.size(0), -1)\n",
        "        x = self.outlayer(x)\n",
        "\n",
        "        return x\n",
        "\n",
        "\n",
        "def main():\n",
        "    batchsz = 32\n",
        "\n",
        "    cifar_train = datasets.CIFAR10('cifar', True, transform=transforms.Compose([\n",
        "            # transforms.RandomHorizontalFlip(), # 水平翻转，Random是随机做                                                                               transforms.RandomHorizontalFlip(), # 水平翻转，Random是随机做\n",
        "#            transforms.RandomVerticalFlip(), # 竖直翻转\n",
        "#            transforms.RandomRotation(15), # 随机旋转-15~15度之间任意角度\n",
        "#            transforms.RandomRotation([90, 270]), # 随机旋转这几个选项中的一个角度\n",
        "#            transforms.Resize([32, 32]), # 缩放\n",
        "#            transforms.RandomCrop([28, 28]), # 裁剪\n",
        "        transforms.Resize((32, 32)),\n",
        "        transforms.ToTensor()\n",
        "    ]), download=True)\n",
        "    cifar_train = DataLoader(cifar_train, batch_size=batchsz, shuffle=True)\n",
        "\n",
        "    cifar_test = datasets.CIFAR10('cifar', False, transform=transforms.Compose([\n",
        "        transforms.Resize((32, 32)),\n",
        "        transforms.ToTensor()\n",
        "    ]), download=True)\n",
        "    cifar_test = DataLoader(cifar_test, batch_size=batchsz, shuffle=True)\n",
        "\n",
        "\n",
        "    x, label = iter(cifar_train).next()\n",
        "    print('x:', x.shape, 'label:', label.shape)\n",
        "\n",
        "    device = torch.device('cuda')\n",
        "    # model = Lenet5().to(device)\n",
        "    model = ResNet18().to(device)\n",
        "\n",
        "    criteon = nn.CrossEntropyLoss().to(device)\n",
        "    optimizer = optim.Adam(model.parameters(), lr=1e-3)\n",
        "    print(model)\n",
        "\n",
        "    for epoch in range(1000):\n",
        "\n",
        "        model.train()\n",
        "        for batchidx, (x, label) in enumerate(cifar_train):\n",
        "            # [b, 3, 32, 32]\n",
        "            # [b]\n",
        "            x, label = x.to(device), label.to(device)\n",
        "\n",
        "            logits = model(x)\n",
        "            # logits: [b, 10]\n",
        "            # label:  [b]\n",
        "            # loss: tensor scalar\n",
        "            loss = criteon(logits, label)\n",
        "\n",
        "            # backprop\n",
        "            optimizer.zero_grad()\n",
        "            loss.backward()\n",
        "            optimizer.step()\n",
        "\n",
        "        #\n",
        "        print(epoch, 'loss:', loss.item())\n",
        "\n",
        "\n",
        "        model.eval()\n",
        "        with torch.no_grad():\n",
        "            # test\n",
        "            total_correct = 0\n",
        "            total_num = 0\n",
        "            for x, label in cifar_test:\n",
        "                # [b, 3, 32, 32]\n",
        "                # [b]\n",
        "                x, label = x.to(device), label.to(device)\n",
        "\n",
        "                # [b, 10]\n",
        "                logits = model(x)\n",
        "                # [b]\n",
        "                pred = logits.argmax(dim=1)\n",
        "                # [b] vs [b] => scalar tensor\n",
        "                correct = torch.eq(pred, label).float().sum().item()\n",
        "                total_correct += correct\n",
        "                total_num += x.size(0)\n",
        "                # print(correct)\n",
        "\n",
        "            acc = total_correct / total_num\n",
        "            print(epoch, 'acc:', acc)\n",
        "\n",
        "\n",
        "if __name__ == '__main__':\n",
        "    main()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\r0it [00:00, ?it/s]"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to cifar/cifar-10-python.tar.gz\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "170500096it [00:06, 27783743.02it/s]                               \n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Files already downloaded and verified\n",
            "x: torch.Size([32, 3, 32, 32]) label: torch.Size([32])\n",
            "ResNet18(\n",
            "  (conv1): Sequential(\n",
            "    (0): Conv2d(3, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "  )\n",
            "  (blk1): ResBlk(\n",
            "    (conv1): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (bn1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (conv2): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (bn2): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (extra): Sequential()\n",
            "  )\n",
            "  (blk2): ResBlk(\n",
            "    (conv1): Conv2d(16, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (extra): Sequential(\n",
            "      (0): Conv2d(16, 32, kernel_size=(1, 1), stride=(1, 1))\n",
            "      (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (outlayer): Linear(in_features=32768, out_features=10, bias=True)\n",
            ")\n",
            "0 loss: 1.5689066648483276\n",
            "0 acc: 0.5463\n",
            "1 loss: 0.4720211923122406\n",
            "1 acc: 0.5897\n",
            "2 loss: 1.0085821151733398\n",
            "2 acc: 0.615\n",
            "3 loss: 0.6026808619499207\n",
            "3 acc: 0.5989\n",
            "4 loss: 0.833814263343811\n",
            "4 acc: 0.5939\n",
            "5 loss: 0.5865252614021301\n",
            "5 acc: 0.6259\n",
            "6 loss: 0.29089391231536865\n",
            "6 acc: 0.6246\n",
            "7 loss: 0.3529692590236664\n",
            "7 acc: 0.6145\n",
            "8 loss: 0.32114699482917786\n",
            "8 acc: 0.6095\n",
            "9 loss: 0.26456528902053833\n",
            "9 acc: 0.5967\n",
            "10 loss: 0.05757981538772583\n",
            "10 acc: 0.6057\n",
            "11 loss: 0.6052832007408142\n",
            "11 acc: 0.6022\n",
            "12 loss: 0.2594645023345947\n",
            "12 acc: 0.6056\n",
            "13 loss: 0.3285031318664551\n",
            "13 acc: 0.6026\n",
            "14 loss: 0.1915207803249359\n",
            "14 acc: 0.5982\n",
            "15 loss: 0.18464431166648865\n",
            "15 acc: 0.5958\n",
            "16 loss: 0.3418786823749542\n",
            "16 acc: 0.5944\n",
            "17 loss: 0.059285640716552734\n",
            "17 acc: 0.5957\n",
            "18 loss: 0.07600599527359009\n",
            "18 acc: 0.5932\n",
            "19 loss: 0.6340038180351257\n",
            "19 acc: 0.59\n",
            "20 loss: 0.17945337295532227\n",
            "20 acc: 0.6049\n",
            "21 loss: 0.5185438990592957\n",
            "21 acc: 0.5972\n",
            "22 loss: 0.14132356643676758\n",
            "22 acc: 0.5882\n",
            "23 loss: 0.21194368600845337\n",
            "23 acc: 0.6079\n",
            "24 loss: 0.028404533863067627\n",
            "24 acc: 0.5945\n",
            "25 loss: 0.07482045888900757\n",
            "25 acc: 0.5976\n",
            "26 loss: 0.5836591720581055\n",
            "26 acc: 0.595\n",
            "27 loss: 0.09832996129989624\n",
            "27 acc: 0.588\n",
            "28 loss: 0.006471216678619385\n",
            "28 acc: 0.5962\n",
            "29 loss: 0.01897108554840088\n",
            "29 acc: 0.5987\n",
            "30 loss: 0.003093123435974121\n",
            "30 acc: 0.5907\n",
            "31 loss: 0.1614762544631958\n",
            "31 acc: 0.6028\n",
            "32 loss: 0.19719654321670532\n",
            "32 acc: 0.5857\n",
            "33 loss: 0.10160613059997559\n",
            "33 acc: 0.5913\n",
            "34 loss: 0.011950850486755371\n",
            "34 acc: 0.5916\n",
            "35 loss: 0.22899967432022095\n",
            "35 acc: 0.5988\n",
            "36 loss: 0.24871325492858887\n",
            "36 acc: 0.6021\n",
            "37 loss: 0.20580178499221802\n",
            "37 acc: 0.5962\n",
            "38 loss: 0.0013948678970336914\n",
            "38 acc: 0.6078\n",
            "39 loss: 0.03174859285354614\n",
            "39 acc: 0.591\n",
            "40 loss: 0.16682755947113037\n",
            "40 acc: 0.5941\n",
            "41 loss: 0.029500842094421387\n",
            "41 acc: 0.5936\n",
            "42 loss: 0.0936427116394043\n",
            "42 acc: 0.5978\n",
            "43 loss: 0.004338622093200684\n",
            "43 acc: 0.5917\n",
            "44 loss: 0.002698659896850586\n",
            "44 acc: 0.5948\n",
            "45 loss: 0.02468717098236084\n",
            "45 acc: 0.5858\n",
            "46 loss: 0.06288212537765503\n",
            "46 acc: 0.5839\n",
            "47 loss: 0.02251267433166504\n",
            "47 acc: 0.5989\n",
            "48 loss: 0.013461947441101074\n",
            "48 acc: 0.5992\n",
            "49 loss: 0.005137324333190918\n",
            "49 acc: 0.6009\n",
            "50 loss: 0.13865137100219727\n",
            "50 acc: 0.592\n",
            "51 loss: 0.010742902755737305\n",
            "51 acc: 0.5978\n",
            "52 loss: 0.00018608570098876953\n",
            "52 acc: 0.6005\n",
            "53 loss: 0.22536420822143555\n",
            "53 acc: 0.5973\n",
            "54 loss: 7.081031799316406e-05\n",
            "54 acc: 0.5925\n",
            "55 loss: 0.34509992599487305\n",
            "55 acc: 0.5941\n",
            "56 loss: 0.010912775993347168\n",
            "56 acc: 0.5933\n",
            "57 loss: 0.48967766761779785\n",
            "57 acc: 0.5924\n",
            "58 loss: 0.08834528923034668\n",
            "58 acc: 0.5951\n",
            "59 loss: 0.00023746490478515625\n",
            "59 acc: 0.5948\n",
            "60 loss: 0.00027740001678466797\n",
            "60 acc: 0.5973\n",
            "61 loss: 8.857250213623047e-05\n",
            "61 acc: 0.6037\n",
            "62 loss: 0.0011403560638427734\n",
            "62 acc: 0.5849\n",
            "63 loss: 0.008451581001281738\n",
            "63 acc: 0.5901\n",
            "64 loss: 0.0028405189514160156\n",
            "64 acc: 0.5994\n",
            "65 loss: 0.0020815134048461914\n",
            "65 acc: 0.5996\n",
            "66 loss: 0.0005661249160766602\n",
            "66 acc: 0.5962\n",
            "67 loss: 0.0010734796524047852\n",
            "67 acc: 0.6028\n",
            "68 loss: 0.0004298686981201172\n",
            "68 acc: 0.6033\n",
            "69 loss: 6.592273712158203e-05\n",
            "69 acc: 0.5962\n",
            "70 loss: 0.009637594223022461\n",
            "70 acc: 0.5986\n",
            "71 loss: 0.009062051773071289\n",
            "71 acc: 0.5937\n",
            "72 loss: 0.045945167541503906\n",
            "72 acc: 0.5964\n",
            "73 loss: 0.17322731018066406\n",
            "73 acc: 0.5948\n",
            "74 loss: 0.0021420717239379883\n",
            "74 acc: 0.5974\n",
            "75 loss: 0.2164599895477295\n",
            "75 acc: 0.5953\n",
            "76 loss: 0.008026838302612305\n",
            "76 acc: 0.5929\n",
            "77 loss: 0.06652522087097168\n",
            "77 acc: 0.595\n",
            "78 loss: 0.02453482151031494\n",
            "78 acc: 0.5992\n",
            "79 loss: 5.7220458984375e-06\n",
            "79 acc: 0.6006\n",
            "80 loss: 0.21974515914916992\n",
            "80 acc: 0.5899\n",
            "81 loss: 0.3036484718322754\n",
            "81 acc: 0.6006\n",
            "82 loss: 0.0033943653106689453\n",
            "82 acc: 0.595\n",
            "83 loss: 7.450580596923828e-05\n",
            "83 acc: 0.5895\n",
            "84 loss: 8.189678192138672e-05\n",
            "84 acc: 0.584\n",
            "85 loss: 0.0001373291015625\n",
            "85 acc: 0.5846\n",
            "86 loss: 0.6518764495849609\n",
            "86 acc: 0.5901\n",
            "87 loss: 9.417533874511719e-06\n",
            "87 acc: 0.6029\n",
            "88 loss: 0.0003987550735473633\n",
            "88 acc: 0.5964\n",
            "89 loss: 0.4133231043815613\n",
            "89 acc: 0.6069\n",
            "90 loss: 0.07649719715118408\n",
            "90 acc: 0.5949\n",
            "91 loss: 0.0001932382583618164\n",
            "91 acc: 0.5932\n",
            "92 loss: 0.005640864372253418\n",
            "92 acc: 0.5945\n",
            "93 loss: 8.225440979003906e-06\n",
            "93 acc: 0.6019\n",
            "94 loss: 0.0025731325149536133\n",
            "94 acc: 0.596\n",
            "95 loss: 2.574920654296875e-05\n",
            "95 acc: 0.6021\n",
            "96 loss: 0.002440214157104492\n",
            "96 acc: 0.5954\n",
            "97 loss: 0.0442044734954834\n",
            "97 acc: 0.5941\n",
            "98 loss: 0.5975921154022217\n",
            "98 acc: 0.6003\n",
            "99 loss: 6.794929504394531e-06\n",
            "99 acc: 0.5978\n",
            "100 loss: 0.03245651721954346\n",
            "100 acc: 0.595\n",
            "101 loss: 0.00026094913482666016\n",
            "101 acc: 0.5935\n",
            "102 loss: 0.6049680709838867\n",
            "102 acc: 0.5967\n",
            "103 loss: 0.002384662628173828\n",
            "103 acc: 0.6005\n",
            "104 loss: 1.4066696166992188e-05\n",
            "104 acc: 0.5937\n",
            "105 loss: 0.0019692182540893555\n",
            "105 acc: 0.5948\n",
            "106 loss: 0.14223015308380127\n",
            "106 acc: 0.5908\n",
            "107 loss: 0.09893786907196045\n",
            "107 acc: 0.5922\n",
            "108 loss: 4.76837158203125e-06\n",
            "108 acc: 0.5949\n",
            "109 loss: 3.421306610107422e-05\n",
            "109 acc: 0.5935\n",
            "110 loss: 0.0002447366714477539\n",
            "110 acc: 0.591\n",
            "111 loss: 0.15205717086791992\n",
            "111 acc: 0.5953\n",
            "112 loss: 2.6226043701171875e-06\n",
            "112 acc: 0.5966\n",
            "113 loss: 0.00018537044525146484\n",
            "113 acc: 0.5932\n",
            "114 loss: 0.0010901689529418945\n",
            "114 acc: 0.5919\n",
            "115 loss: 0.09571123123168945\n",
            "115 acc: 0.5997\n",
            "116 loss: 0.2117455005645752\n",
            "116 acc: 0.5927\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a9ZbZtj21FXx",
        "colab_type": "text"
      },
      "source": [
        "### nn.Module"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VmQOFi9jrc8h",
        "colab_type": "code",
        "outputId": "27b41be9-5776-4c96-86b3-f62380f69d7c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 569
        }
      },
      "source": [
        "import  torch\n",
        "from    torch import nn\n",
        "from    torch import optim\n",
        "\n",
        "# 自定义线性层\n",
        "class MyLinear(nn.Module):\n",
        "\n",
        "    def __init__(self, inp, outp):\n",
        "        super(MyLinear, self).__init__()\n",
        "\n",
        "        # requires_grad = True\n",
        "        self.w = nn.Parameter(torch.randn(outp, inp))\n",
        "        self.b = nn.Parameter(torch.randn(outp))\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = x @ self.w.t() + self.b\n",
        "        return x\n",
        "\n",
        "# 展平\n",
        "class Flatten(nn.Module):\n",
        "\n",
        "    def __init__(self):\n",
        "        super(Flatten, self).__init__()\n",
        "\n",
        "    def forward(self, input):\n",
        "        return input.view(input.size(0), -1)\n",
        "\n",
        "class TestNet(nn.Module):\n",
        "\n",
        "    def __init__(self):\n",
        "        super(TestNet, self).__init__()\n",
        "\n",
        "        self.net = nn.Sequential(nn.Conv2d(1, 16, stride=1, padding=1),\n",
        "                                 nn.MaxPool2d(2, 2),\n",
        "                                 Flatten(), # 展平\n",
        "                                 nn.Linear(1*14*14, 10))\n",
        "\n",
        "    def forward(self, x):\n",
        "        return self.net(x)\n",
        "\n",
        "\n",
        "class BasicNet(nn.Module):\n",
        "\n",
        "    def __init__(self):\n",
        "        super(BasicNet, self).__init__()\n",
        "\n",
        "        self.net = nn.Linear(4, 3)\n",
        "\n",
        "    def forward(self, x):\n",
        "        return self.net(x)\n",
        "\n",
        "class Net(nn.Module):\n",
        "\n",
        "    def __init__(self):\n",
        "        super(Net, self).__init__()\n",
        "\n",
        "        self.net = nn.Sequential(BasicNet(),\n",
        "                                 nn.ReLU(),\n",
        "                                 nn.Linear(3, 2))\n",
        "\n",
        "    def forward(self, x):\n",
        "        return self.net(x)\n",
        "\n",
        "\n",
        "def main():\n",
        "    device = torch.device('cuda')\n",
        "    net = Net()\n",
        "    net.to(device)\n",
        "\n",
        "    net.train()\n",
        "\n",
        "    net.eval()\n",
        "\n",
        "    # net.load_state_dict(torch.load('ckpt.mdl'))\n",
        "    #\n",
        "    #\n",
        "    # torch.save(net.state_dict(), 'ckpt.mdl')\n",
        "\n",
        "    for name, t in net.named_parameters():\n",
        "        print('parameters:', name, t.shape)\n",
        "\n",
        "    for name, m in net.named_children():\n",
        "        print('children:', name, m)\n",
        "\n",
        "    for name, m in net.named_modules():\n",
        "        print('modules:', name, m)\n",
        "\n",
        "\n",
        "if __name__ == '__main__':\n",
        "    main()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "parameters: net.0.net.weight torch.Size([3, 4])\n",
            "parameters: net.0.net.bias torch.Size([3])\n",
            "parameters: net.2.weight torch.Size([2, 3])\n",
            "parameters: net.2.bias torch.Size([2])\n",
            "children: net Sequential(\n",
            "  (0): BasicNet(\n",
            "    (net): Linear(in_features=4, out_features=3, bias=True)\n",
            "  )\n",
            "  (1): ReLU()\n",
            "  (2): Linear(in_features=3, out_features=2, bias=True)\n",
            ")\n",
            "modules:  Net(\n",
            "  (net): Sequential(\n",
            "    (0): BasicNet(\n",
            "      (net): Linear(in_features=4, out_features=3, bias=True)\n",
            "    )\n",
            "    (1): ReLU()\n",
            "    (2): Linear(in_features=3, out_features=2, bias=True)\n",
            "  )\n",
            ")\n",
            "modules: net Sequential(\n",
            "  (0): BasicNet(\n",
            "    (net): Linear(in_features=4, out_features=3, bias=True)\n",
            "  )\n",
            "  (1): ReLU()\n",
            "  (2): Linear(in_features=3, out_features=2, bias=True)\n",
            ")\n",
            "modules: net.0 BasicNet(\n",
            "  (net): Linear(in_features=4, out_features=3, bias=True)\n",
            ")\n",
            "modules: net.0.net Linear(in_features=4, out_features=3, bias=True)\n",
            "modules: net.1 ReLU()\n",
            "modules: net.2 Linear(in_features=3, out_features=2, bias=True)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cxc0sPytYqzl",
        "colab_type": "text"
      },
      "source": [
        "### 数据增强\n",
        "翻转、旋转、缩放、剪裁\n",
        "\n",
        "效果并不好"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iR--FnuxaCKR",
        "colab_type": "code",
        "outputId": "2425d2e0-f28c-4c65-8aa1-f10d8ad547bf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 100
        }
      },
      "source": [
        "! npm install -g localtunnel\n",
        "# 8097可修改为自己要用的端口号\n",
        "get_ipython().system_raw('python3 -m pip install visdom')\n",
        "get_ipython().system_raw('python3 -m visdom.server -port 8097 >> visdomlog.txt 2>&1 &')\n",
        "get_ipython().system_raw('lt --port 8097 >> url.txt 2>&1 &')\n",
        "! cat url.txt\n",
        "# import visdom\n",
        "# vis = visdom.Visdom(port='8097')\n",
        "# vis.text('testing')\n",
        "# ! cat visdomlog.txt"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[K\u001b[?25h/tools/node/bin/lt -> /tools/node/lib/node_modules/localtunnel/bin/lt.js\n",
            "+ localtunnel@2.0.0\n",
            "added 35 packages from 21 contributors in 2.024s\n",
            "/bin/bash: lt: command not found\n",
            "your url is: https://big-eagle-26.localtunnel.me\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HcNNtOLkrc6G",
        "colab_type": "code",
        "outputId": "2b9fa925-1f4c-40bd-e517-a6b39a650a76",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "import  torch\n",
        "import  torch.nn as nn\n",
        "import  torch.nn.functional as F\n",
        "import  torch.optim as optim\n",
        "from    torchvision import datasets, transforms\n",
        "\n",
        "from visdom import Visdom\n",
        "\n",
        "batch_size=200\n",
        "learning_rate=0.01\n",
        "epochs=10\n",
        "\n",
        "train_loader = torch.utils.data.DataLoader(\n",
        "    datasets.MNIST('mnist_data', train=True, download=True,\n",
        "                   transform=transforms.Compose([\n",
        "                       transforms.RandomHorizontalFlip(), # 水平翻转，Random是随机做\n",
        "                       transforms.RandomVerticalFlip(), # 数值翻转\n",
        "                       transforms.RandomRotation(15), # 随机旋转-15~15度之间任意角度\n",
        "                       transforms.RandomRotation([90, 270]), # 随机旋转这几个选项中的一个角度（只能有两个值）\n",
        "                    #    transforms.Resize([32, 32]), # 缩放\n",
        "                    #    transforms.RandomCrop([28, 28]), # 裁剪\n",
        "                       transforms.ToTensor(), # numpy格式转为Tensor\n",
        "                       # transforms.Normalize((0.1307,), (0.3081,)) # 归一化\n",
        "                   ])),\n",
        "    batch_size=batch_size, shuffle=True) # 一次加载batch_size张图片，随机打散\n",
        "\n",
        "test_loader = torch.utils.data.DataLoader(\n",
        "    datasets.MNIST('mnist_data', train=False, transform=transforms.Compose([\n",
        "        transforms.ToTensor(),\n",
        "        # transforms.Normalize((0.1307,), (0.3081,))\n",
        "    ])),\n",
        "    batch_size=batch_size, shuffle=True)\n",
        "\n",
        "\n",
        "# 全连接网络\n",
        "class MLP(nn.Module):\n",
        "\n",
        "    def __init__(self):\n",
        "        super(MLP, self).__init__()\n",
        "\n",
        "        self.model = nn.Sequential(\n",
        "            nn.Linear(784, 200),\n",
        "            nn.LeakyReLU(inplace=True),\n",
        "            nn.Linear(200, 200),\n",
        "            nn.LeakyReLU(inplace=True),\n",
        "            nn.Linear(200, 10),\n",
        "            nn.LeakyReLU(inplace=True),\n",
        "        )\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.model(x)\n",
        "\n",
        "        return x\n",
        "\n",
        "device = torch.device('cuda:0')\n",
        "net = MLP().to(device)\n",
        "optimizer = optim.SGD(net.parameters(), lr=learning_rate)\n",
        "criteon = nn.CrossEntropyLoss().to(device)\n",
        "\n",
        "viz = Visdom(port='8097') # Visdom实例化，加上端口号\n",
        "# 先创建一条曲线，后面再把最新数据添加到上面去。参数：初始点坐标y x，win是小窗口ID，窗口名称\n",
        "viz.line([0.], [0.], win='train_loss', opts=dict(title='train loss'))\n",
        "# 一个窗口同时显示两条曲线，参数：初始点坐标y1 y2 x，ID，两个窗口名称，两个图例说明\n",
        "viz.line([[0.0, 0.0]], [0.], win='test', opts=dict(title='test loss&acc.',legend=['loss', 'acc.'])) \n",
        "global_step = 0 # x轴\n",
        "\n",
        "for epoch in range(epochs):\n",
        "\n",
        "    for batch_idx, (data, target) in enumerate(train_loader):\n",
        "        data = data.view(-1, 28*28)\n",
        "        data, target = data.to(device), target.cuda()\n",
        "\n",
        "        logits = net(data)\n",
        "        loss = criteon(logits, target)\n",
        "\n",
        "        optimizer.zero_grad()\n",
        "        loss.backward()\n",
        "        # print(w1.grad.norm(), w2.grad.norm())\n",
        "        optimizer.step()\n",
        "\n",
        "        #添加新数据到曲线上：loss是numpy类型，x轴时间戳，ID，更新方式为往后添加，否则就会覆盖掉\n",
        "        global_step += 1 #x加1，这里有3000个,3000*200=60000*10\n",
        "        viz.line([loss.item()], [global_step], win='train_loss', update='append') \n",
        "\n",
        "        if batch_idx % 100 == 0:\n",
        "            print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n",
        "                epoch, batch_idx * len(data), len(train_loader.dataset),\n",
        "                       100. * batch_idx / len(train_loader), loss.item()))\n",
        "\n",
        "    test_loss = 0\n",
        "    correct = 0\n",
        "    for data, target in test_loader:\n",
        "        data = data.view(-1, 28 * 28)\n",
        "        data, target = data.to(device), target.cuda()\n",
        "        logits = net(data)\n",
        "        test_loss += criteon(logits, target).item()\n",
        "\n",
        "        pred = logits.argmax(dim=1)\n",
        "        correct += pred.eq(target).float().sum().item()\n",
        "\n",
        "    #添加新数据到曲线上，y1 y2 x，最好y1和y2的范围接近\n",
        "    viz.line([[test_loss, correct/len(test_loader.dataset)]], [global_step], win='test', update='append')\n",
        "\n",
        "    #图片可视化，这里显示test_loader最近一批的200张\n",
        "    viz.images(data.view(-1, 1, 28, 28), win='x') #变形为图片，图像可以直接传tensor类型不用转numpy\n",
        "    viz.text(str(pred.detach().cpu().numpy()), win='pred', opts=dict(title='pred')) #预测的标签，要转numpy\n",
        "\n",
        "    test_loss /= len(test_loader.dataset)\n",
        "    print('\\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\\n'.format(\n",
        "        test_loss, correct, len(test_loader.dataset),\n",
        "        100. * correct / len(test_loader.dataset)))\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Setting up a new session...\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Train Epoch: 0 [0/60000 (0%)]\tLoss: 2.304445\n",
            "Train Epoch: 0 [20000/60000 (33%)]\tLoss: 2.292063\n",
            "Train Epoch: 0 [40000/60000 (67%)]\tLoss: 2.285799\n",
            "\n",
            "Test set: Average loss: 0.0113, Accuracy: 2999.0/10000 (30%)\n",
            "\n",
            "Train Epoch: 1 [0/60000 (0%)]\tLoss: 2.275401\n",
            "Train Epoch: 1 [20000/60000 (33%)]\tLoss: 2.239706\n",
            "Train Epoch: 1 [40000/60000 (67%)]\tLoss: 2.201200\n",
            "\n",
            "Test set: Average loss: 0.0108, Accuracy: 3252.0/10000 (33%)\n",
            "\n",
            "Train Epoch: 2 [0/60000 (0%)]\tLoss: 2.152547\n",
            "Train Epoch: 2 [20000/60000 (33%)]\tLoss: 2.089677\n",
            "Train Epoch: 2 [40000/60000 (67%)]\tLoss: 2.045973\n",
            "\n",
            "Test set: Average loss: 0.0096, Accuracy: 4201.0/10000 (42%)\n",
            "\n",
            "Train Epoch: 3 [0/60000 (0%)]\tLoss: 1.900767\n",
            "Train Epoch: 3 [20000/60000 (33%)]\tLoss: 1.835191\n",
            "Train Epoch: 3 [40000/60000 (67%)]\tLoss: 1.728176\n",
            "\n",
            "Test set: Average loss: 0.0080, Accuracy: 4638.0/10000 (46%)\n",
            "\n",
            "Train Epoch: 4 [0/60000 (0%)]\tLoss: 1.646666\n",
            "Train Epoch: 4 [20000/60000 (33%)]\tLoss: 1.635103\n",
            "Train Epoch: 4 [40000/60000 (67%)]\tLoss: 1.531050\n",
            "\n",
            "Test set: Average loss: 0.0073, Accuracy: 4868.0/10000 (49%)\n",
            "\n",
            "Train Epoch: 5 [0/60000 (0%)]\tLoss: 1.386552\n",
            "Train Epoch: 5 [20000/60000 (33%)]\tLoss: 1.405610\n",
            "Train Epoch: 5 [40000/60000 (67%)]\tLoss: 1.421683\n",
            "\n",
            "Test set: Average loss: 0.0071, Accuracy: 4897.0/10000 (49%)\n",
            "\n",
            "Train Epoch: 6 [0/60000 (0%)]\tLoss: 1.398590\n",
            "Train Epoch: 6 [20000/60000 (33%)]\tLoss: 1.448791\n",
            "Train Epoch: 6 [40000/60000 (67%)]\tLoss: 1.388239\n",
            "\n",
            "Test set: Average loss: 0.0070, Accuracy: 4909.0/10000 (49%)\n",
            "\n",
            "Train Epoch: 7 [0/60000 (0%)]\tLoss: 1.404058\n",
            "Train Epoch: 7 [20000/60000 (33%)]\tLoss: 1.550699\n",
            "Train Epoch: 7 [40000/60000 (67%)]\tLoss: 1.357575\n",
            "\n",
            "Test set: Average loss: 0.0069, Accuracy: 4909.0/10000 (49%)\n",
            "\n",
            "Train Epoch: 8 [0/60000 (0%)]\tLoss: 1.424706\n",
            "Train Epoch: 8 [20000/60000 (33%)]\tLoss: 1.370729\n",
            "Train Epoch: 8 [40000/60000 (67%)]\tLoss: 1.455225\n",
            "\n",
            "Test set: Average loss: 0.0068, Accuracy: 5077.0/10000 (51%)\n",
            "\n",
            "Train Epoch: 9 [0/60000 (0%)]\tLoss: 1.475395\n",
            "Train Epoch: 9 [20000/60000 (33%)]\tLoss: 1.338458\n",
            "Train Epoch: 9 [40000/60000 (67%)]\tLoss: 1.359877\n",
            "\n",
            "Test set: Average loss: 0.0067, Accuracy: 5168.0/10000 (52%)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}