Using_More_Ways_pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. (torch)  squeeze() 与 unsqueeze()\n",
    "\n",
    "- squeeze() 压缩一个维度，仅当当前维度为1，否则不做修改\n",
    "- unsqueeze(num) 增加一个维度，num可以为负数，表示在倒数第num个维度增加一个维度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([3, 1, 2]),\n",
       " torch.Size([3, 1, 2]),\n",
       " torch.Size([3, 2]),\n",
       " torch.Size([3, 1, 2]))"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "x = torch.randn(3, 1, 2)\n",
    "y = x.squeeze(0)  # not modify\n",
    "z = x.squeeze(1)  # modified\n",
    "w = z.unsqueeze(-2)\n",
    "x.shape, y.shape, z.shape, w.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. (torch) nn.ReLU()\n",
    "    相比于max(0, value)，nn.ReLU()可以用于处理批量数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([-0.3862,  0.3717,  1.3680,  0.5028,  0.0278]),\n",
       " tensor([0.0000, 0.3717, 1.3680, 0.5028, 0.0278]))"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "m = torch.nn.ReLU()\n",
    "x = torch.randn(5)\n",
    "y = m(x)\n",
    "x, y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4 `linalg.norm()`计算矩阵的二范式值\n",
    "    [linalg.norm()](https://pytorch.org/docs/stable/linalg.html?highlight=norm#torch.linalg.norm) 计算矩阵二范式，返回值是一个数\n",
    "\n",
    "#### 4.1 P范数的一般形式\n",
    "若 $x=[x_1, x_2, ..., x_n]$，则\n",
    "$$\n",
    "    \\|x\\|_p = (|x_1|^p + |x_2|^p + ... + |x_n|^p)^{(\\frac1p)}\n",
    "$$\n",
    "\n",
    "当p取0, 1, ∞时，有以下三种情形：\n",
    "- p=1，一范式：$\\|x\\|_1 = (|x_1| + |x_2| + ... + |x_n|)$\n",
    "- p=2，二范式：$ \\|x\\|_2 = (|x_1|^2 + |x_2|^p + ... + |x_n|^2)^{(\\frac12)}$\n",
    "- p=∞，三范式：$\\|x\\|_∞ = max(|x_1| + |x_2| + ... + |x_n|)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.2 `linalg.norm()` 计算矩阵的二范式，返回值是一个数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1.],\n",
       "        [1., 1.]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "data = torch.ones(2, 2)\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 4.2.1 linalg.norm() 返回值是二维矩阵的二范式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(2.),\n",
       " tensor([[0.5000, 0.5000],\n",
       "         [0.5000, 0.5000]]))"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "norm_data = torch.linalg.norm(data)  # torch.linalg.norm(data, p=2, ord=None)\n",
    "result1 = data / norm_data\n",
    "norm_data, result1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 4.2.2 linalg.norm(data, ord=1) 不是返回二维矩阵的一范式！！！ 而是 max(sum(abs(data), dim=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(2.),\n",
       " tensor([[0.5000, 0.5000],\n",
       "         [0.5000, 0.5000]]))"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "norm_data = torch.linalg.norm(data, ord=1)\n",
    "result1 = data / norm_data\n",
    "norm_data, result1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(2.)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "norm_data = max(torch.sum(abs(data), dim=0))\n",
    "norm_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.3 手动计算矩阵的二范式，具有可导性????有问题"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[2., 2.],\n",
       "         [2., 2.],\n",
       "         [2., 2.],\n",
       "         [2., 2.]]),\n",
       " torch.Size([4, 2]))"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datax = torch.full((4, 2), 2.)\n",
    "datax, datax.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[4., 4.],\n",
       "        [4., 4.],\n",
       "        [4., 4.],\n",
       "        [4., 4.]])"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datax.pow(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[4.],\n",
       "        [4.],\n",
       "        [4.],\n",
       "        [4.]])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datax.pow(2).mean(1, keepdim=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. `nn.functional.normalize() ` 返回沿某个dim维度的二范式矩阵\n",
    "    dim=0，表示在第一个维度上作二范式，如果keepdim=True，将在第一维度压缩成大小为1\n",
    "\n",
    "#### 5.1 nn.functional.normalize(input, p, dim, eps) 参数细节\n",
    "\n",
    "normalize(`input`: Tensor, `p`: float = 2, `dim`: int = 1, `eps`: float = 1e-12, `out`: Optional[Tensor] = None) -> Tensor\n",
    "- `input`: input tensor of any shape\n",
    "- `p` (float): the exponent value in the norm formulation. Default: 2\n",
    "- `dim` (int): the dimension to reduce. Default: 1\n",
    "- `eps` (float): small value to avoid division by zero. Default: 1e-12\n",
    "- `out` (Tensor, optional): the output tensor. If :attr:`out` is used, this operation won't be differentiable.\n",
    "\n",
    "##### 注意：可用于二维矩阵的规范化（大于维度2，则仅用于沿某个维度的二范式计算）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn.functional as fn\n",
    "\n",
    "data3 = torch.ones(2, 4)\n",
    "data3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.2 dim=1（默认值），沿着第二个维度计算二范式矩阵\n",
    "- norm = sqrt(1^2 + 1^2 + 1^2 + 1^2) = 2\n",
    "- result = data / norm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.5000, 0.5000, 0.5000, 0.5000],\n",
       "        [0.5000, 0.5000, 0.5000, 0.5000]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result3 = fn.normalize(data3, p=2, dim=1)\n",
    "result3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.3 dim=0, 沿着第一个维度上计算二范式矩阵\n",
    "- norm0 = sqrt(1^2) = 1\n",
    "- norm1 = sqrt(1^2) = 1\n",
    "- norm2 = sqrt(1^2) = 1\n",
    "- norm3 = sqrt(1^2) = 1\n",
    "- result = data / [norm0, norm1, norm2, norm3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.7071, 0.7071, 0.7071, 0.7071],\n",
       "        [0.7071, 0.7071, 0.7071, 0.7071]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result4 = fn.normalize(data3, p=2, dim=0)  # 第一个维度\n",
    "result4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. (torch) Variable\n",
    "\n",
    "Variable是对Tensor的一个封装，可以理解为含有计算图的Tensor。每个Variable都有三个属性：\n",
    "- Varibale的Tensor本身的`.data`\n",
    "- 对应Tensor的梯度`.grad`\n",
    "- Variable是通过什么方式得到的`.grad_fn`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.1 Tensor 与 Variable\n",
    "    Varibale(tesnor, requires_grad=True) 默认情况下不求梯度的，如果要求梯度，需要说明。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 2.],\n",
       "        [3., 4.]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "from torch.autograd import Variable\n",
    "\n",
    "tensor = torch.FloatTensor([[1, 2], [3, 4]])\n",
    "tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 2.],\n",
       "        [3., 4.]], requires_grad=True)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "variable = Variable(tensor, requires_grad=True)\n",
    "variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.2 Variable 求梯度\n",
    "    Variable计算时，会生成计算图，将所有的计算节点都连接起来。在进行误差反向传递时，一次性将所有Variable梯度都计算出来。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(28.5746),\n",
       " <SumBackward0 at 0x2299a10f648>,\n",
       " tensor([[1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1.]]),\n",
       " tensor([[2., 2., 2., 2., 2.],\n",
       "         [2., 2., 2., 2., 2.],\n",
       "         [2., 2., 2., 2., 2.],\n",
       "         [2., 2., 2., 2., 2.],\n",
       "         [2., 2., 2., 2., 2.]]))"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from torch.autograd import Variable\n",
    "import torch\n",
    "\n",
    "x_tensor = torch.randn(5,5)\n",
    "y_tensor = torch.ones(5,5)\n",
    "\n",
    "x = Variable(x_tensor,requires_grad=True) \n",
    "y = Variable(y_tensor,requires_grad=True)\n",
    "z = torch.sum(x + y**2)\n",
    "\n",
    "z.backward()\n",
    "\n",
    "z.data, z.grad_fn, x.grad, y.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. torch的排序和比较\n",
    "\n",
    "#### 7.1 torch的排序\n",
    "    `sort()`返回从小到大的序列和元素在原序列的位置索引"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([1, 2, 3, 4, 5, 6]), tensor([0, 3, 1, 4, 2, 5]))"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "arr = [1, 3, 5, 2, 4, 6]\n",
    "array = torch.tensor(arr)\n",
    "y, i = torch.sort(array)\n",
    "y, i"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7.2 torch的比较\n",
    "- `gt` (greater than) 大于\n",
    "- `lt` (less than) 小于\n",
    "- `eq` (equal) 等于\n",
    "- `ge` (greater and equal) 大于等于\n",
    "- `le` (less and equal) 小于等于\n",
    "\n",
    "布尔类型转换为浮点型： result.float() 返回 [1.0, 0.0, ..., 0.0]\n",
    "\n",
    "布尔类型转换为整型： result.int() 返回 [0, 0, 0, 1, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([False, False, False,  True,  True]),\n",
       " tensor([0., 0., 0., 1., 1.]),\n",
       " tensor([0, 0, 0, 1, 1], dtype=torch.int32))"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mask, mask.float(), mask.int()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7.3 torch的求和 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(2.)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.sum(mask.float())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. nn.Sequential() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (0): Conv2d(1, 20, kernel_size=(5, 5), stride=(1, 1))\n",
       "  (1): ReLU()\n",
       "  (2): Conv2d(20, 64, kernel_size=(5, 5), stride=(1, 1))\n",
       "  (3): ReLU()\n",
       ")"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "model = nn.Sequential(\n",
    "          nn.Conv2d(1,20,5),\n",
    "          nn.ReLU(),\n",
    "          nn.Conv2d(20,64,5),\n",
    "          nn.ReLU())\n",
    "\n",
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.0433),\n",
       " tensor([[0.6995, 0.7147],\n",
       "         [0.6625, 0.7491],\n",
       "         [0.9186, 0.3952]]))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn.functional as F\n",
    "\n",
    "def at(x):\n",
    "    return F.normalize(x.pow(2).mean(1).view(x.size(0), -1))\n",
    "\n",
    "def at_loss(x, y):\n",
    "    return (at(x) - at(y)).pow(2).mean()\n",
    "\n",
    "x = torch.randn(3, 5, 2)\n",
    "y = torch.randn(3, 5, 2)\n",
    "value = at_loss(x, y)\n",
    "value, at(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(3.2966)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "norm_data = torch.linalg.norm(value)\n",
    "norm_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([-4., -3., -2., -1.]),\n",
       " tensor([[-4., -3.],\n",
       "         [-2., -1.]]),\n",
       " tensor(5.4772))"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "a = torch.arange(4, dtype= torch.float) - 4\n",
    "b = a.reshape((2, 2))\n",
    "a, b, torch.norm(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[-4., -3.],\n",
       "         [-2., -1.]]),\n",
       " tensor([[-0.8000, -0.6000],\n",
       "         [-0.8944, -0.4472]]))"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch.nn.functional as F\n",
    "c = F.normalize(b)\n",
    "b, c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1000.0"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "a = 1e+3\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.9349,  0.0000, -0.4164],\n",
       "        [ 1.7015,  0.0000,  0.3736],\n",
       "        [-0.5406,  0.8038,  1.0288],\n",
       "        [-1.3739,  0.2597, -0.7225]])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "x = torch.randn(4, 3)\n",
    "x[0][1] = 0\n",
    "x[1][1] = 0\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[0, 0],\n",
       "         [0, 2],\n",
       "         [1, 0],\n",
       "         [1, 2],\n",
       "         [2, 0],\n",
       "         [2, 1],\n",
       "         [2, 2],\n",
       "         [3, 0],\n",
       "         [3, 1],\n",
       "         [3, 2]]),\n",
       " torch.Size([10, 2]))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = torch.nonzero(x)\n",
    "y, y.shape"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:python36]",
   "language": "python",
   "name": "conda-env-python36-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}