Introduction.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 00 预告\n",
    "<img src=\"https://s2.loli.net/2022/02/19/w7pbgVvONoA1U4I.png\" width=70%>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 04 数据操作 + 数据预处理\n",
    "\n",
    "## 数据操作\n",
    "\n",
    "\n",
    "## 数据操作实现\n",
    "- 创建tensor\n",
    "- 运算符运算$\\rightarrow$逐个元素操作;逻辑运算符：X==Y 二元张量\n",
    "- torch.cat(); x.sum() $\\rightarrow$ 始终是一个张量，可设置维度\n",
    "- 广播机制：3 * 1+1 * 2 $\\rightarrow$3 * 2\n",
    "- x[-1], x[1:3] 左闭右开，获取最后一个/XXX个元素\n",
    "- id(Y)// Z[:]=X+Y, X+=Y\n",
    "- A = X.numpy(), B = torch.tensor(A)\n",
    "- a, a.item()\n",
    "\n",
    "## 数据预处理\n",
    "- 写数据\n",
    "```python\n",
    "import os \n",
    "os.makedirs(os.path.join('...', '...'), exist_ok=True)\n",
    "path = os.path.join(os.path.join('', '', '...csv'))\n",
    "with open(path, 'w') as f:\n",
    "    f.write('this is a sentence\\n')  # 可以设置列名，以及数据样本\n",
    "```\n",
    "\n",
    "- 读数据并预处理\n",
    "\n",
    "```python\n",
    "import pandas as pd\n",
    "data = pd.read_csv('...csv')\n",
    "\n",
    "input_data, output_data = data.iloc[:, 0:2], data.iloc[2] # iloc   \n",
    "input_data = input_data.fillna(input_data.mean())   # fillna, mean\n",
    "# 法二：类别值或离散值\n",
    "input_data = pd.get_dummies(input_data, dummy_na=True)\n",
    "\n",
    "print(data.values)\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 05 线性代数\n",
    "## 线性代数实现\n",
    "- 矩阵转置 A.T, A==A.T\n",
    "- B = A.clone()\n",
    "- 哈达玛积 A * B\n",
    "- A.sum(axis=0//axis=[0, 1])   (2,5,4)->(5,4)//(4) \n",
    "- A.mean(), A.sum()/A.numel()  ||  A.mean(axis=0) ../A.shape[0]\n",
    "- 保持维度：keepdim=True  $\\rightarrow$ 保存维度从而***广播***\n",
    "- A.cumsum(axis=0)\n",
    "- torch.dot(x,y)\n",
    "- torch.mv(A,x)//torch.mm(A,B)//torch.norm(u)、\n",
    "\n",
    "## 按特定轴求和\n",
    "keepdim $\\rightarrow$ 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 06 矩阵运算\n",
    "- 亚导数：将导数拓展到不可微的函数\n",
    "\n",
    "<img src=\"https://s2.loli.net/2022/02/08/UFCj8LgbItqxh9z.png\" width=40%>\n",
    "\n",
    "- 梯度/偏导\n",
    "- 标量$\\rightarrow$向量：注意内积\n",
    "$\\frac{\\partial y}{\\partial \\textbf{x}}$  $\\leftarrow$ ***分子布局***\n",
    "<img src=\"https://s2.loli.net/2022/02/08/m8k5rEZR1XD6H2u.png\" width=70%>\n",
    "- 向量求导\n",
    "<img src=\"https://s2.loli.net/2022/02/08/BHMbOWntpSZIkrf.png\" width=60%>\n",
    "\n",
    "## 矩阵布局相关内容\n",
    "相关重要公式：\n",
    "$$\\frac{\\partial (\\textbf x^T\\textbf a)}{\\partial \\textbf x}=\\frac{\\partial (\\textbf a^T\\textbf x)}{\\partial \\textbf x}=\\textbf a$$ \n",
    "$$\\frac{\\partial (\\textbf x^T\\textbf x)}{\\partial \\textbf x}=2\\textbf x$$\n",
    "<font color=\"#cf6923\" size=4 face=\"黑体\">不是很明白的推导：</font>\n",
    "$$\\frac{\\partial(\\textbf x^T\\textbf A\\textbf x)}{\\partial \\textbf x}=\\textbf A\\textbf x+\\textbf A^T\\textbf x$$\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 07 自动求导\n",
    "## 自动求导\n",
    "- 向量链式求导法则\n",
    "- 自动求导\n",
    "  - 符号求导/数值求导\n",
    "  - 正向/反向 $\\rightarrow$ 链式反则：【计算与内存复杂度】\n",
    "- 计算图\n",
    "  - 无环图：隐式构造/显式构造\n",
    "  \n",
    "## 自动求导实现\n",
    "- **【注意】：标量求导**\n",
    "\n",
    "```python\n",
    "x.requires_grad(True)\n",
    "x.grad\n",
    "y = torch.dot(x, x)  #这里标量\n",
    "y = x * x\n",
    "y = y.sum()  #转化为标量 ## u=y.detach() 在保存参数中常用！分出计算图\n",
    "\n",
    "y.backward()  \n",
    "x.grad\n",
    "```\n",
    "- 控制流\n",
    "\n",
    "## QA：\n",
    "*梯度累加*：[掘金：梯度累加Gradient Accumulation](https://juejin.cn/post/7041121387418746910)  -->batch_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 08 线性回归+基础优化算法\n",
    "## 线性回归\n",
    "- 凸函数-->梯度0***最***优解\n",
    "- 显式解//NPC问题\n",
    "\n",
    "## 基础优化算法\n",
    "- 梯度下降：沿反梯度方向更新参数求解\n",
    " - 超参数：学习率\n",
    " - 小批量随机梯度下降：超参数：批量大小\n",
    " \n",
    "## 线性回归从0实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## 流水线、模型、损失函数、优化器(小批量随机梯度下降)、（训练）\n",
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import torch \n",
    "import d2l.torch as d2l\n",
    "import random\n",
    "# ---------生成数据---------\n",
    "def synthetic_data(w, b, num):\n",
    "    X = torch.normal(0, 1, (num, len(w)))\n",
    "    y = torch.matmul(X, w) + b        ## 我猜想这里有广播机制\n",
    "    #y = torch.mul(w, X) + b   ## 这个确确实实用了广播，mul与matmul\n",
    "                                # 我是脑残，区别mm,matmul,mul(对应位)\n",
    "    \n",
    "    y +=torch.normal(0, 0.01, y.shape)\n",
    "    return X, y.reshape(-1, 1)\n",
    "\n",
    "true_w = torch.tensor([2, -3.4])\n",
    "ture_b = 4.2\n",
    "feats, labels = synthetic_data(true_w, ture_b, 1000)\n",
    "# 绘图展示\n",
    "d2l.set_figsize()\n",
    "# d2l.plt.scatter(feats[:,1].detach().numpy(), labels.detach().numpy(), 1)\n",
    "#d2l.plt.show()\n",
    "\n",
    "# --------读取小批量--------\n",
    "def data_iter(batch_size, feats, labels):  ## 切片使用，转化list\n",
    "    num = len(feats)   # len与shape不同\n",
    "    indices = list(range(num))\n",
    "    random.shuffle(indices)\n",
    "    for i in range(0, num, batch_size):\n",
    "        batch_indices = torch.tensor(indices[i:min(i+batch_size, num)])\n",
    "        yield feats[batch_indices], labels[batch_indices] ## 需要在函数里\n",
    "\n",
    "batch_size = 10                                     ## 生成器\n",
    "for X, y in data_iter(batch_size, feats, labels):   ## 这里它是迭代器！\n",
    "   # print(X,'\\n',y)\n",
    "    break\n",
    "\n",
    "# 初始化模型参数\n",
    "w = torch.normal(0, 0.01, size=(2,1), requires_grad=True)  ##需要梯度哦\n",
    "b = torch.zeros(1, requires_grad=True)\n",
    "# --------定义模型--------\n",
    "def linreg(X, w, b):\n",
    "    \"\"\"线性回归模型\"\"\"\n",
    "    return torch.matmul(X, w) + b\n",
    "\n",
    "# ------定义损失函数-------\n",
    "def squared_loss(y_hat, y):\n",
    "    return (y_hat-y.reshape(y_hat.shape))**2 / 2\n",
    "\n",
    "# ------定义优化算法--------\n",
    "def sgd(params, lr, batch_size):\n",
    "    \"\"\"小批量随机梯度下降\"\"\"\n",
    "    with torch.no_grad():\n",
    "        for param in params:\n",
    "            param -= lr * param.grad / batch_size\n",
    "            param.grad.zero_()\n",
    "\n",
    "# --------训练------------\n",
    "lr = 0.03\n",
    "num_epochs = 3\n",
    "net = linreg\n",
    "loss = squared_loss\n",
    "\n",
    "for epoch in range(num_epochs):\n",
    "    for X, y in data_iter(batch_size, feats, labels):\n",
    "        l = loss(net(X, w, b), y)\n",
    "        l.sum().backward()     # 需要对向量累加\n",
    "        sgd([w, b], lr, batch_size)  # 可能batch不能整除\n",
    "    \n",
    "    with torch.no_grad():\n",
    "        train_l = loss(net(feats, w, b), labels)\n",
    "        print(f'epoch {epoch + 1}, loss {float(train_l.mean()):f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Pytorch中的广播机制](https://blog.csdn.net/qq_42890800/article/details/115558389)\n",
    "- 广播：\n",
    "  - mul:对应位不同才可broadcast\n",
    "  - matmul:\n",
    "    - (2,5,3)×(1,3,4)-->(2,5,4) \n",
    "    - (5,3,4)×(4,2)-->(5,3,2) ; (2,1,3,4)×(5,4,2)-->(2,5,3,2) 最外：batch\n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 线性回归的简洁实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "from torch.utils import data\n",
    "from d2l import torch as d2l\n",
    "import torch.nn as nn\n",
    "\n",
    "true_w = torch.tensor([2, -3.4])\n",
    "ture_b = 4.2\n",
    "feats, labels = d2l.synthetic_data(true_w, ture_b, 1000)\n",
    "print('----------')\n",
    "print(feats)\n",
    "\n",
    "def load_array(data_arrays, batch_size, is_train=True): ## 注意这里的使用\n",
    "    \"\"\"构造一个Pytorch数据迭代器\"\"\"\n",
    "    dataset = data.TensorDataset(*data_arrays)   ## 以及这里的*：作用于可迭代对象\n",
    "    return data.DataLoader(dataset, batch_size, shuffle=is_train)\n",
    "\n",
    "batch_size = 10\n",
    "data_iter = load_array((feats,labels), batch_size)\n",
    "\n",
    "for data in data_iter:\n",
    "    print('---------------')\n",
    "    print(data)\n",
    "\n",
    "    \n",
    "# next(iter(data_iter))    ## iter：生成迭代器\n",
    "\n",
    "# net = nn.Sequential(nn.Linear(2,1))\n",
    "\n",
    "# net[0].weight.data.normal_(0, 0.01)  ##初始化 单层网络因此可以直接索引\n",
    "# net[0].bias.data.fill_(0)\n",
    "\n",
    "# loss = nn.MSELoss()\n",
    "# trainer = torch.optim.SGD(net.parameters(), lr=0.03)\n",
    "\n",
    "# num_epochs = 3\n",
    "# for epoch in range(num_epochs):\n",
    "#     for X, y in data_iter:\n",
    "#         l = loss(net(X), y)\n",
    "#         trainer.zero_grad()\n",
    "#         l.backward()\n",
    "#         trainer.step()\n",
    "#     l = loss(net(feats), labels)\n",
    "#     print(f'epoch {epoch + 1}, loss {l:f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 梯度下降时，注意除以batch_size，避免由于这个参数导致的数值差异过大；或者让lr来除调节学习率\n",
    "- 合理初始化/Adam \n",
    "- batch_size：小则更利于收敛--->噪音，鲁棒\n",
    "  - 更新：batch中每一个样本对应参数求梯度求和取平均值更新\n",
    "- 一阶导向量，二阶导梯度/矩阵；  \n",
    "   - 统计模型与优化模型-->收敛速度快与泛化性等无关；saddle point\n",
    "- load可能存在内存爆掉；硬盘与内存，prefetch\n",
    "- SGD：本质原因：NPC，没有显示解，导数为0；batch逐一逼近"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 09 softmax回归+损失函数+图片分类数据集\n",
    "## softmax回归\n",
    "- 从回归到多类分类\n",
    "- 分类\n",
    "  - 均方损失\n",
    "    - 对类别一位有效编码\n",
    "    - 均方损失训练\n",
    "  - 无校验比例\n",
    "    $O_y - O_i >= \\Delta(y,i)$\n",
    "  - 校验比例\n",
    "     - 匹配概率，softmax，概率区别\n",
    "- softmax与交叉熵损失: 预测置信度\n",
    " $$l(\\textbf{y},\\hat{\\textbf{y}}) = -\\sum_{\\substack{i}}y_i\\log{\\hat{y}_i} = -\\log{\\hat{y}_y}$$\n",
    " \n",
    "## 损失函数\n",
    "损失函数，似然函数，导数\n",
    "- L2 Loss: \n",
    "- L1 Loss：梯度常数，权重更新稳定\n",
    "- Huber's Robust Loss\n",
    "$\\rightarrow$ 依据距离远近，函数、梯度形状\n",
    "\n",
    "## 图像分类数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#### 这个代码块没有具体实现所以不能运行 不要运行不然报一堆错\n",
    "%matplotlib inline\n",
    "import torch\n",
    "import torchvision\n",
    "from torch.utils import data\n",
    "from torchvision import transforms\n",
    "import d2l.torch as d2l\n",
    "\n",
    "d2l.use_svg_display()    ## 使用svg， 据说图像更加清晰\n",
    "\"\"\"加载数据集\"\"\"\n",
    "trans = transforms.ToTensor()       ##-----这里 转化\n",
    "mnist_train = torchvision.datasets.MNIST(\n",
    "root=\"../data\", train=True, transform=trans, download=True)\n",
    "mnist_test = torchvisionhvision.datasets.MNIST(\n",
    "root=\"../data\", train=False, transform=trans, download=True)\n",
    "\n",
    "len(mnist_test),len(mnist_train)\n",
    "mnist_train[0][0].shape\n",
    "\n",
    "\"\"\"绘制\"\"\"\n",
    "def get_fashion_mnist_labels(labels):\n",
    "    \"\"\"返回Fashion-MNIST数据集的文本标签\"\"\"\n",
    "    text_labels = [\n",
    "        't-shirt', 'trouser', 'pullover', 'dress', 'coat',\n",
    "        'sandal', 'sneaker', 'bag', 'ankle boot']\n",
    "    return [text_labels[int(i)] for i in labels]\n",
    "    \n",
    "def show_images(img, num_rows, num_cols, titles=None, scale=1.5):\n",
    "    figsize = (num_cols * scale, num_rows * scale)\n",
    "    _, axes = d2l.plt.subplot(num_rows, num_cols, figsize)\n",
    "    axes = axes.flatten()\n",
    "    \n",
    "    for i, (ax, img) in enumerate(zip(axes, imgs)):\n",
    "        if torch.is_tensor(img):\n",
    "            # 图片张量\n",
    "            ax.imshow(img.numpy())\n",
    "        else:\n",
    "            # PIL图片\n",
    "            ax.imshow(img)\n",
    "\n",
    "\"\"\"图像与标签\"\"\"\n",
    "X, y = next(iter(data.DataLoader(mnist_train, batch_size=18)))\n",
    "show_images(X.reshape(18, 28, 28), 2, 9, titles=get_fashion_mnist_labels)   \n",
    "\n",
    "\"\"\"批量读取\"\"\"\n",
    "batch_size = 256\n",
    "\n",
    "def get_dataloder_workers():\n",
    "    return 4\n",
    "\n",
    "train_iter = data.DataLoader(mnist_train, batch_size, shuffle=True,\n",
    "                            num_workers=get_dataloder_workers())#训练集随机\n",
    "timer = d2l.Timer()\n",
    "for X, y in train_iter:\n",
    "    continue\n",
    "f'{timer.stop():.2f} sec'              #  使用计时器什么的\n",
    "\n",
    "\"\"\"函数定义\"\"\"\n",
    "def load_data_fashion_mnist(batch_size, resize=None):\n",
    "    \"\"\"下载数据集并加载到内存中\"\"\"\n",
    "    trans = [transforms.ToTensor()]       # 注意transforms的使用，处理图像\n",
    "    if resize:                                    # 直到这里才第一次理解到transforms到底是在干啥，组合操作\n",
    "        trans.insert(0, transforms.Resize(resize))\n",
    "    trans = transforms.Compose(trans)\n",
    "    \n",
    "    mnist_train = torchvision.datasets.FashionMNIST(root='../data', train=True, transform=trans, download=True)\n",
    "    mnist_test = torchvision.datasets.FashionMNIST(root='../data', train=False, transform=trans, download=True)\n",
    "    \n",
    "    return(data.DataLoader(mnist_train, batch_size, shuffle=True, num_workers=get_dataloder_workers),\n",
    "          data.DataLoader(mnist_test, batch_size, shuffle=True, num_workers=get_dataloder_workers))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## softmax回归从零开始实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## 这个也没有实现\n",
    "import torch\n",
    "import numpy as np\n",
    "from IPython import display \n",
    "from d2l import torch as d2l\n",
    "\n",
    "batch_size = 256\n",
    "train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size)\n",
    "\n",
    "num_inputs = 784    # 28*28\n",
    "num_outputs = 10\n",
    "\n",
    "\"\"\"初始化参数\"\"\"\n",
    "W = torch.normal(0, 0.01, size=(num_inputs, num_outputs), requires_grad=True)\n",
    "b = torch.zeros(num_outputs, requires_grad=True)\n",
    "\n",
    "def softmax(X):\n",
    "    X_exp = torch.exp(X)\n",
    "    partition = X_exp.sum(1, keepdim=True)\n",
    "    return X_exp / partition    # 广播机制\n",
    "\n",
    "\"\"\"实现softmax回归\"\"\"\n",
    "def net(X):                     # 由于reshape感觉这个地方变得有点怪只能说\n",
    "    return softmax(torch.matmul(X.reshape(-1, W.shape[0]), W) + b)   # 好了我现在理解了，应该是：图片个数×784 (原先纠结于特征)\n",
    "\n",
    "\"\"\"实现交叉熵损失函数\"\"\"\n",
    "def cross_entropy(y_hat, y):\n",
    "    return -torch.log(y_hat[range(len(y_hat)), y])   ## 注意这个用法，很奇妙的\n",
    "\n",
    "\"\"\"计算预测正确的数量\"\"\"\n",
    "def accuracy(y_hat, y):\n",
    "    if len(y_hat.shape) > 1 and y_hat.shape[1] > 1:\n",
    "        y_hat = y_hat.argmax(axis = 1)\n",
    "    cmp = y_hat.type(y.dtype) == y\n",
    "    return float(cmp.type(y.dtype).sum())\n",
    "\n",
    ".  \n",
    "\"\"\"计算在指定数据集上模型的精度\"\"\"\n",
    "def evaluate_accuracy(net, data_iter):\n",
    "    if isinstance(net, torch.nn.Module):  # 这个东西是区别于model.train()和model.eval()的\n",
    "        # 将模型设置为评估模型\n",
    "        net.eval()  \n",
    "    # 正确预测数、预测总数\n",
    "    metric = Accumulator(2)      # 具体累加器实现，感觉挺好用的\n",
    "    for X, y in data_iter:\n",
    "        metric.add(accuracy(net(X), y), y.numel())\n",
    "    return metric[0] / metric[1]\n",
    "\n",
    "class Accumulator:\n",
    "    def _init_(self, n):\n",
    "        self.data = [0.0] * n\n",
    "    \n",
    "    def add(self, *arg):\n",
    "        self.data = [a + float(b) for a, b in zip(self.data, args)]\n",
    "    \n",
    "    def reset(self):\n",
    "        self.data = [0.0] * len(self.data)\n",
    "    \n",
    "    def _getitem_(self, idx):\n",
    "        return self.data[idx]\n",
    "\n",
    "    \n",
    "evaluate_accuracya(net, test_ite r)\n",
    "\n",
    "\"\"\"训练\"\"\"\n",
    "def train_epoch_ch3(net, train_iter, loss, update):  ## 这次代码中很多判断这种\n",
    "    if isinstance(net, torch.nn.Module):\n",
    "        net.train()\n",
    "    metric = Accumulator(3)\n",
    "    for X, y in train_iter:\n",
    "        y_hat = net(X)\n",
    "        l = loss(y_hat, y)\n",
    "        if isinstance(update, torch.optim.Optimizer):\n",
    "            update.zero_grad()\n",
    "            l.backward()\n",
    "            update.step()\n",
    "            metric.add(\n",
    "               float(l) * len(y), accuracy(y_hat, y),\n",
    "                y.size().numel()\n",
    "            )\n",
    "        else:\n",
    "            l.sum().backward()\n",
    "            update(X.shape[0])\n",
    "            metric.add(float(l.sum()), accuracy(y_hat, y), y.size().numel())\n",
    "    \n",
    "    return metric[0] / metric[2], metric[1] / metric[2]\n",
    "\n",
    "\n",
    "def train_ch3(net, train_iter, test_iter, loss, num_epochs, update):\n",
    "    for epoch in range(num_epochs):\n",
    "        train_metrics = train_epoch_ch3(net, train_iter, loss, update)\n",
    "        test_acc = evaluate_accuracy(net, test_iter)\n",
    "    \n",
    "    train_loss, train_acc = train_metrics\n",
    "\n",
    "\n",
    "lr = 0.1\n",
    "def update(batch_size):\n",
    "    return d2l.sgd([W, b], lr, batch_size)\n",
    "\n",
    "num_epochs = 10\n",
    "train_ch3(net, train_iter, test_iter, cross_entropy, num_epochs,update)\n",
    "\n",
    "\"\"\"PS部分\"\"\"\n",
    "# sum求和注意keepdim=True，保持维度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import numpy as np\n",
    "from torchvision import transforms\n",
    "from d2l import torch as d2l\n",
    "\n",
    "a = torch.tensor([[1, 2, 3], [2, 3, 4]])\n",
    "# a.size().numel()\n",
    "#batch_size = 256\n",
    "#train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size)\n",
    "\n",
    "# next(iter(train_iter))\n",
    "# image = torch.tensor(next(iter(train_iter)))\n",
    "\n",
    "# image = image.cpu().clone()\n",
    "# image = image.squeeze(0) # 压缩一维\n",
    "# image = transforms.ToPILImage(image) # 自动转换为0-255"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## softmax回归的简洁实现\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import d2l.torch as d2l\n",
    "\n",
    "batch_size = 256\n",
    "train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size)\n",
    "\n",
    "\"\"\"搭建网络\"\"\"           ## 这里注意\n",
    "net = nn.Sequential(nn.Flatten(), nn.Linear(784, 10)) ##展平向量\n",
    "\n",
    "def init_weights(m):\n",
    "    if type(m) == nn.Linear:\n",
    "        nn.init.normal_(m.weight, std=0.01)\n",
    "        \n",
    "net.apply(init_weights)     # 设置模型参数\n",
    "\n",
    "loss = nn.CrossEntropyLoss()\n",
    "trainer = torch.optim.SGD(net.parameters(), lr=0.1)\n",
    "\n",
    "num_epochs = 10\n",
    "d2l.train_ch3(net, train_iter, test_iter, loss, num_epochs, trainer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- softlabel\n",
    "- logistic regression 逻辑回归\n",
    "- 交叉熵，相对熵，互信息\n",
    "- 似然函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 10 多层感知机+代码实现\n",
    "## 感知机\n",
    "- 二分类：-1/1\n",
    " - 回归：实数\n",
    " - softmax回归：概率\n",
    " \n",
    "- 训练感知机\n",
    "   - 损失函数：$l(y,\\textbf{x},\\textbf{w}) = max(0, -y<\\textbf{w}, \\textbf{x}>)$  批量大小为1的梯度下降\n",
    "\n",
    "- 收敛定理\n",
    "  - 数据在r内\n",
    "  - 余量：$y(\\textbf{x}^T+\\textbf{b})\\ge \\rho$ （有限制条件）\n",
    "  - 感知机保证在$\\frac{r^2+1}{\\rho^2}$步后收敛\n",
    "\n",
    "- XOR问题：感知机不能拟合XOR函数，只能产生线性分割面\n",
    "\n",
    "\n",
    "## 多层感知机\n",
    "\n",
    "- 单隐藏层\n",
    "  - 非线性激活函数必须有：避免层数塌陷\n",
    "<img src=\"https://s2.loli.net/2022/02/09/aTU8mSJv5OhHi2I.png\" width=70%>\n",
    "  - sigmoid\n",
    "  - tanh\n",
    "  - ReLU\n",
    "\n",
    "- 多类分类\n",
    "<img src=\"https://s2.loli.net/2022/02/09/MnVRhjbkA3yGcLo.png\" width=70%>\n",
    "- 多隐藏层\n",
    "  - 超参数：隐层层数与隐藏层大小\n",
    "  - 选择：128-64-32-8-输出 (底层可以先expand)\n",
    "\n",
    "## 代码实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import d2l.torch as d2l\n",
    "\n",
    "batch_size = 256\n",
    "train_iter, test_iter = d2l.load_data_fashion_mnist(batch_sizeh_size)\n",
    "\n",
    "num_inputs, num_hiddens, num_outputs = 784, 256, 10\n",
    "W1 = nn.Parameter(torch.randn(num_inputs, num_hiddens, requires_grad=True))\n",
    "W2 = nn.Parameter(torch.randn(num_hiddens, num_outputs, requires_grad=True))\n",
    "b1 = nn.Parameter(torch.zeros(num_hiddens, requires_grad=True))\n",
    "b2 = nn.Parameter(torch.zeros(num_outputs, requires_grad=True))\n",
    "\n",
    "params = [W1, b1, W2, b2]   # 用nn.Parameter设定网络参数\n",
    "\n",
    "def relu(X):\n",
    "    a = torch.zeros_like(X)\n",
    "    return torch.max(X, a)  ##居然能比较大小属于是太牛了\n",
    "\n",
    "def net(X):\n",
    "    X = X.reshape((-1,num_inputs))  ## reshape遗漏\n",
    "    H = relu(X @ W1 + b1)           ## @：简写\n",
    "    return (H @ W2 + b2)\n",
    "\n",
    "loss = nn.CrossEntropyLoss()\n",
    "epoch, lr = 10, 0.1\n",
    "trainer = torch.optim.SGD(params, lr=lr)\n",
    "d2l.train_ch3(net, train_itera, test_iter, loss, epoch, trainer)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import d2l.torch as d2l\n",
    "\n",
    "batch_size = 256\n",
    "train_iter, test_iter = d2l.load_data_fashion_mnist(batch_sizeh_size)\n",
    "\n",
    "net = nn.Sequential(\n",
    "    nn.Flatten(), nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10))\n",
    "\n",
    "def init_weight(m):\n",
    "    if type(m) == nn.Linear:\n",
    "        nn.init.normal_(m.weight, std=0.01)\n",
    "\n",
    "net.apply(init_weight)\n",
    "\n",
    "loss = nn.CrossEntropyLoss()\n",
    "epoch, lr = 10, 0.1\n",
    "trainer = torch.optim.SGD(net.parameters(), lr=lr)\n",
    "d2l.train_ch3(net, train_itera, test_iter, loss, epoch, trainer)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- svm与mlp、多层mlp\n",
    "- \"深度学习\"与\"浅度学习\"：理论上一样，实际一般深度较好\n",
    "- relu：引入非线性\n",
    "- 鲁棒、泛化、稳定性"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 11 模型选择 + 过拟合和欠拟合\n",
    "## 模型选择\n",
    "- 训练误差$\\rightarrow$训练数据 和 泛化误差$\\rightarrow$新数据\n",
    "- 验证数据集$\\rightarrow$评估模型好坏 和 测试数据集$\\rightarrow$只用一次的数据集\n",
    "  - 验证：不要和训练混在一起；常拿出50%的训练数据\n",
    "  - 测试只能用一次！（举例数据：虚高）\n",
    "- K-折交叉验证\n",
    "  - 非大数据集上\n",
    "  - validation与train\n",
    "  - k个验证的精度/误差平均得loss\n",
    "     - 极端算法：尽可能使用数据集：n个数据，n折交叉验证\n",
    "\n",
    "## 过拟合和欠拟合\n",
    "- 过拟合和欠拟合\n",
    "  - 模型容量||数据 $\\rightarrow$ 正常/过拟合/欠拟合\n",
    "    - 过拟合：没有泛化性\n",
    "- 模型复杂度\n",
    "  - 模型容量：\n",
    "    - 拟合各种函数的能力\n",
    "    - 最优情况：泛化误差最小，训练误差还行\n",
    "      - 有时会选择承受一定的过拟合\n",
    "    - 估计模型容量：同种类算法：①参数个数 ②参数值范围 -->控制复杂度\n",
    "  - 统计学习：VC维\n",
    "     - 支持n维输入的感知机的VC维是N+1\n",
    "- 数据复杂度\n",
    "  - 个数，时空结构，多样性\n",
    "- 两个复杂度需要匹配\n",
    " \n",
    "\n",
    "## 代码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 12 权重衰退\n",
    "处理过拟合\n",
    "- 使用均方误差范数作为硬性限制\n",
    " - $\\textbf{w}的L2 Loss\\le \\theta$; 更强的正则项\n",
    "- 使用均方误差作为柔性限制\n",
    "  $$minL(\\textbf{w},b) + \\frac{\\lambda}{2}\\textbf{w}的L2 Loss$$\n",
    "  - 超参数$\\lambda$控制了正则项的重要程度\n",
    "    - $\\lambda$ = 0：无作用\n",
    "    "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}