Akankhya_Homework_1_PyTorch.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "Akankhya_Homework_1_PyTorch.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z_LTBtfMBvW_"
      },
      "source": [
        "# Machine Learning II: Deep Learning and Applications"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T33dD1e8tii2"
      },
      "source": [
        "Install PyTorch and Skorch."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pJB3VQYDCUmh",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "9d542c87-4c45-42e3-8778-100d8f03ef98"
      },
      "source": [
        "!pip install -q torch skorch torchvision torchtext"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[?25l\r\u001b[K     |███                             | 10kB 28.5MB/s eta 0:00:01\r\u001b[K     |██████▏                         | 20kB 33.2MB/s eta 0:00:01\r\u001b[K     |█████████▎                      | 30kB 35.8MB/s eta 0:00:01\r\u001b[K     |████████████▍                   | 40kB 39.2MB/s eta 0:00:01\r\u001b[K     |███████████████▌                | 51kB 43.3MB/s eta 0:00:01\r\u001b[K     |██████████████████▌             | 61kB 47.2MB/s eta 0:00:01\r\u001b[K     |█████████████████████▋          | 71kB 48.7MB/s eta 0:00:01\r\u001b[K     |████████████████████████▊       | 81kB 50.4MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▉    | 92kB 52.7MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████ | 102kB 54.1MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 112kB 54.1MB/s \n",
            "\u001b[?25h"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3l_Dl6qxCXmv",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        },
        "outputId": "6488a7a7-5dad-4ac8-d5f2-e150b3123d1a"
      },
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import torch.optim as optim\n",
        "import torchvision\n",
        "import skorch\n",
        "import sklearn\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:144: FutureWarning: The sklearn.metrics.scorer module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.metrics. Anything that cannot be imported from sklearn.metrics is now part of the private API.\n",
            "  warnings.warn(message, FutureWarning)\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q9GLqo7ZDUh-",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 125
        },
        "outputId": "454414e3-9232-4b38-f8c0-f59590483c4d"
      },
      "source": [
        "from google.colab import drive \n",
        "drive.mount('/content/gdrive/')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/gdrive/\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fAEjIi3ODVgU",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "e00f39a8-39c7-46da-8bd6-4826272ccbaa"
      },
      "source": [
        "%cd /content/gdrive/My\\ Drive/"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/gdrive/My Drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S5hNbX_wDX6V",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "35dfb5f6-1323-4e6a-eab4-6908c2f4eda9"
      },
      "source": [
        "!pwd"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/gdrive/My Drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uevQtU7NtZ_-"
      },
      "source": [
        "## 1. Tensor Operations (20 points)\n",
        "\n",
        "Tensor operations are important in deep learning models. In this part, you are required to implement some common tensor operations in PyTorch."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5DeQOItkeQCx"
      },
      "source": [
        "### 1) Tensor squeezing, unsqueezing and viewing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pAOmBE5ODwpP"
      },
      "source": [
        "Tensor squeezing, unsqueezing and viewing are important methods to change the dimension of a Tensor, and the corresponding functions are [torch.squeeze](https://pytorch.org/docs/stable/torch.html#torch.squeeze), [torch.unsqueeze](https://pytorch.org/docs/stable/torch.html#torch.unsqueeze) and [torch.Tensor.view](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.view). Please read the documents of the functions, and finish the following practice."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hVrM80YxFSjb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "outputId": "d720a645-b117-4fcc-b893-bc876657b467"
      },
      "source": [
        "# x is a tensor with size being (3, 2)\n",
        "x = torch.Tensor([[1, 2], [3, 4], [5, 6]])\n",
        "print(\"size of x:\",x.size())\n",
        "\n",
        "# Add two new dimensions to x by using the function torch.unsqueeze, so that the size of x becomes (3, 1, 2, 1).\n",
        "x = torch.unsqueeze(torch.unsqueeze(x,1),-1)\n",
        "print(\"size of x after addition of two dimensions :\",x.shape)\n",
        "\n",
        "# Remove the two dimensions justed added by using the function torch.squeeze, and change the size of x back to (3, 2).\n",
        "x = torch.squeeze(torch.squeeze(x, -1), 1)\n",
        "print(\"size of x after removing two dimensions\",x.shape)\n",
        "\n",
        "# x is now a two-dimensional tensor, or in other words a matrix. Now use the function torch.Tensor.view and change x to a one-dimensional vector with size being (6).\n",
        "x=torch.Tensor.view(x,6)\n",
        "print(\"modified tensor x:\",x)\n",
        "print(\"new size of x:\",x.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "size of x: torch.Size([3, 2])\n",
            "size of x after addition of two dimensions : torch.Size([3, 1, 2, 1])\n",
            "size of x after removing two dimensions torch.Size([3, 2])\n",
            "modified tensor x: tensor([1., 2., 3., 4., 5., 6.])\n",
            "new size of x: torch.Size([6])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "liuR-U0wea0n"
      },
      "source": [
        "### 2) Tensor concatenation and stack"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pkbnt6v8Bo-j"
      },
      "source": [
        "Tensor concatenation and stack are operations to combine small tensors into big tensors. The corresponding functions are [torch.cat](https://pytorch.org/docs/stable/torch.html#torch.cat) and [torch.stack](https://pytorch.org/docs/stable/torch.html#torch.stack). Please read the documents of the functions, and finish the following practice."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b9KqXu3Stfjh",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 193
        },
        "outputId": "d8458adf-3401-4d9a-d20c-a671a29708f4"
      },
      "source": [
        "# x is a tensor with size being (3, 2)\n",
        "x = torch.Tensor([[1, 2], [3, 4], [5, 6]])\n",
        "\n",
        "# y is a tensor with size being (3, 2)\n",
        "y = torch.Tensor([[-1, -2], [-3, -4], [-5, -6]])\n",
        "\n",
        "# Our goal is to generate a tensor z with size as (2, 3, 2), and z[0,:,:] = x, z[1,:,:] = y.\n",
        "\n",
        "# Use torch.stack to generate such a z\n",
        "z = torch.stack([x, y,],dim=0)\n",
        "print(\"concatenated tensor z shape:\",z.shape)\n",
        "print(\"concatenated tensor z:\\n\",z)\n",
        "\n",
        "# Use torch.cat and torch.unsqueeze to generate such a z\n",
        "z = torch.cat([torch.unsqueeze(x, 0),torch.unsqueeze(y, 0)], dim=0)\n",
        "print(\"concatenated tensor z shape:\",z.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "concatenated tensor z shape: torch.Size([2, 3, 2])\n",
            "concatenated tensor z:\n",
            " tensor([[[ 1.,  2.],\n",
            "         [ 3.,  4.],\n",
            "         [ 5.,  6.]],\n",
            "\n",
            "        [[-1., -2.],\n",
            "         [-3., -4.],\n",
            "         [-5., -6.]]])\n",
            "concatenated tensor z shape: torch.Size([2, 3, 2])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WGw4eEo-eeHm"
      },
      "source": [
        "### 3) Tensor expansion"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KAII9eJgJFK2"
      },
      "source": [
        "Tensor expansion is to expand a tensor into a larger tensor along singleton dimensions. The corresponding functions are [torch.Tensor.expand](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.expand) and [torch.Tensor.expand_as](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.expand_as). Please read the documents of the functions, and finish the following practice. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sQbFte-AJzVL",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "outputId": "a6b350b2-d78f-4b01-b3aa-adf2b15c6c38"
      },
      "source": [
        "# x is a tensor with size being (3)\n",
        "x = torch.Tensor([1, 2, 3])\n",
        "\n",
        "# Our goal is to generate a tensor z with size (2, 3), so that z[0,:] = x, z[1,:] = x.\n",
        "\n",
        "# Change the size of x into (1, 3) by using torch.unsqueeze.\n",
        "x = torch.unsqueeze(x,dim=0)\n",
        "print(\"changed size of x:\",x.shape)\n",
        "\n",
        "# Then expand the new tensor to the target tensor by using torch.Tensor.expand.\n",
        "z = x.expand(2,-1)\n",
        "print(\"expanded tensor z shape:\",z.shape)\n",
        "print(\"expanded tensor z:\\n\",z)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "changed size of x: torch.Size([1, 3])\n",
            "expanded tensor z shape: torch.Size([2, 3])\n",
            "expanded tensor z:\n",
            " tensor([[1., 2., 3.],\n",
            "        [1., 2., 3.]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0rFL_Shoef3m"
      },
      "source": [
        "### 4) Tensor reduction in a given dimension"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fmEoJVw0LL9H"
      },
      "source": [
        "In deep learning, we often need to compute the mean/sum/max/min value in a given dimension of a tensor. Please read the document of [torch.mean](https://pytorch.org/docs/stable/torch.html#torch.mean), [torch.sum](https://pytorch.org/docs/stable/torch.html#torch.sum), [torch.max](https://pytorch.org/docs/stable/torch.html#torch.max), [torch.min](https://pytorch.org/docs/stable/torch.html#torch.min), [torch.topk](https://pytorch.org/docs/stable/torch.html#torch.topk), and finish the following practice."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A7dlZwe4MNxo",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "outputId": "d4428fb2-6e55-4f97-ddf1-da05d17bbc7f"
      },
      "source": [
        "# x is a random tensor with size being (10, 50)\n",
        "x = torch.rand(10, 50)\n",
        "\n",
        "# Compute the mean value for each row of x.\n",
        "# You need to generate a tensor x_mean of size (10), and x_mean[k] is the mean value of the k-th row of x.\n",
        "x_mean=torch.mean(x,dim=1)\n",
        "print(\"mean value for each row of x:\",x_mean.shape)\n",
        "\n",
        "# Compute the sum value for each row of x.\n",
        "# You need to generate a tensor x_sum of size (10).\n",
        "x_sum = torch.sum(x,dim=1)\n",
        "print(\"sum value for each row of x:\",x_sum.shape)\n",
        "\n",
        "# Compute the max value for each row of x.\n",
        "# You need to generate a tensor x_max of size (10).\n",
        "x_max = torch.max(x,dim=1).values\n",
        "print(\"max value for each row of x:\",x_max.shape)\n",
        "\n",
        "# Compute the min value for each row of x.\n",
        "# You need to generate a tensor x_min of size (10).\n",
        "x_min = torch.min(x,dim=1).values\n",
        "print(\"min value for each row of x:\",x_min.shape)\n",
        "\n",
        "# Compute the top-5 values for each row of x.\n",
        "# You need to generate a tensor x_mean of size (10, 5), and x_top[k] is the top-5 values of each row in x.\n",
        "x_top = torch.topk(x,5,dim=1).values\n",
        "print(\"top-5 values for each row of x:\",x_top.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "mean value for each row of x: torch.Size([10])\n",
            "sum value for each row of x: torch.Size([10])\n",
            "max value for each row of x: torch.Size([10])\n",
            "min value for each row of x: torch.Size([10])\n",
            "top-5 values for each row of x: torch.Size([10, 5])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I49qjiqHB9oa"
      },
      "source": [
        "## Convolutional Neural Networks (40 points)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JePbG5pSt1xv"
      },
      "source": [
        "Implement a convolutional neural network for image classification on CIFAR-10 dataset.\n",
        "\n",
        "CIFAR-10 is an image dataset of 10 categories. Each image has a size of 32x32 pixels. The following code will download the dataset, and split it into `train` and `test`. For this question, we use the default validation split generated by Skorch."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sQxOUQ29BuMB",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "3a9fbd9a-8a6d-4576-c8c9-619f15fc4ea9"
      },
      "source": [
        "train = torchvision.datasets.CIFAR10(\"./data\", train=True, download=True, transform=torchvision.transforms.Compose([torchvision.transforms.ToTensor(),]))\n",
        "test = torchvision.datasets.CIFAR10(\"./data\", train=False, download=True, transform=torchvision.transforms.Compose([torchvision.transforms.ToTensor(),]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Files already downloaded and verified\n",
            "Files already downloaded and verified\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ieBpiwMwi6wD"
      },
      "source": [
        "The following code visualizes some samples in the dataset. You may use it to debug your model if necessary."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cU5HrxybupyJ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 108
        },
        "outputId": "c7297f11-e758-438d-abb5-70ca3b9963d7"
      },
      "source": [
        "def plot(data, labels=None, num_sample=5):\n",
        "  n = min(len(data), num_sample)\n",
        "  for i in range(n):\n",
        "    plt.subplot(1, n, i+1)\n",
        "    plt.imshow(data[i], cmap=\"gray\")\n",
        "    plt.xticks([])\n",
        "    plt.yticks([])\n",
        "    if labels is not None:\n",
        "      plt.title(labels[i])\n",
        "\n",
        "train.labels = [train.classes[target] for target in train.targets]\n",
        "plot(train.data, train.labels)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWUAAABbCAYAAACrgpTSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO29aZRd13Ue+J07vXmoeUIBBYAABM7U\nQEnURFlyJNm0JctW3OkVJ+5O0kl3x3a3e3USJ+5ur8Tu4YeTdqLV9lpWZC9ZScduyZI1RoMlaiIl\nmhAnECQBEEChUPP06s13PP1jf+dWAeKAB4lCSb57LfKh3rvjueee8529v/1tpbVGZplllllm+8Os\nm30BmWWWWWaZ7Vo2KGeWWWaZ7SPLBuXMMssss31k2aCcWWaZZbaPLBuUM8sss8z2kWWDcmaZZZbZ\nPrKbMigrpU4opR5XSrWUUr96M67hx92UUlopdcvNvo79ZD9ubaKU+mOl1G/f7Ov4cTel1INKqb//\nIr8dVEq1lVL2y217vXazkPI/AfBVrXVFa/1vb9I13HRTSl1SSr3zZl/HfrKsTTJ7IftBDHavhGmt\nL2uty1rr+Ad1zJs1KB8C8PQL/WBmnL/uppRybvY17DfL2uSVt6yNb7790AdlpdRXALwdwAcJ+/+j\nUur3lVKfU0p1ALxdKVVTSn1EKbWulJpXSv2mUsri/rZS6neVUhtKqYtKqX/MZemPVGdSSv0JgIMA\nPs12+Ce8j7+nlLoM4CtKqfuVUleu2S9FkmyLf66Uep6uoFNKqdkXONeblVILSqn7fxj3dqOWtclL\nm1LqHqXUd3lffwogv+e3B+gSbCilHlJK3bnnt2ml1Mf5Pl3c6zJUSv2WUupjSqmPKqWaAH75Fb6H\nf7bn2ZxRSv3cnuv46J7t5sx7rZT6HQBvwe6Y8UFuc59S6q+UUjv8vG/P/g8qpX6bbdFWSn1aKTWi\nlPoPSqkmt5/bs/2LHot2VCn1CPf9C6XU8LXX+SL3+18rpZ5RSm0rpb6glDr0so2ktf6h/wfgQQB/\nn//+YwA7AN4EmSTyAD4C4C8AVADMATgL4O9x+38E4AyAAwCGAHwZgAbg3Ix7+T7b4RKAd/Lfc7yP\njwAoASgAuB/AlZfY538G8BSAEwAUgLsAjPA3DeAWAO8GsADg3pt9v1mbfF/t4gGYB/A/AnAB/AKA\nEMBvA7gHwBqA1wOwAfxdtkmO79QpAP8rj3EEwAUA7+Jxf4vHeR+3LbzC9/EBANM81y8C6ACY4nV8\ndM925tk7/PtBcMzg38MAtgH8EgAHwN/i3yN7tj8P4CiAGmTMOAvgndz+IwD+aIBjLQK4nf3w4+Za\nX+o6AbyX13CSx/1NAA+9bBvdpA6298L/GMBH9vxmAwgA3Lrnu38I4EH++ysA/uGe396JH69B+cie\n3+/HSw9AzwF474scWwP4DciLfPvNvtesTb7vdnkrgCUAas93D0EG5d8H8K+u2f45AG+DDNSXr/nt\nN/YMSL8F4Os38b4e5+D1WxhsUP4lAI9cc6yHAfzynu3/xZ7ffhfA5/f8/TMAHh/gWP/nnt9u5Rhl\nv9R1Avg8CCb5twWgC+DQS7XJflnyL+z59ygECczv+W4ewAz/PX3N9nv//eNgg9zPLIDnX+L3/wEy\n4Z3+/i7pplvWJtLvFzXfbpp5Rw4B+LtKqV/Z85vHfWIA00qpxp7fbADf2PP3D+0dUkr9HQC/DhnM\nAKAMeecHtWlcPUYAV48TALC659+9F/i7PMCxFq75zcXLX/chAL+nlPrdPd8pHvfa86W2X3jKezva\nBmQ5tdf3chCyfACAZYjrwtj3+At/hOyFJPr2ftcBUDR/KAmCju35fQGyPHsx+wCA9ymlfu37ucgf\nsmVt8sK2DGBGKaX2fHeQnwsAfkdrXd/zX1Fr/f/yt4vX/FbRWv/UnuP8UKQi6U/9QwD/GOIaqAM4\nDRmornquACav2f3aa1zC1WMEcPU4MYhdz7Fmr/kthIxVL2ULkFX93rYvaK0feqmd9sugnJoWasmf\nAfgdpVSFD/LXAZggwJ8B+DWl1IxSqg7gn96kS/1B2CrEx/didhZAXin100opF+KTyu35/UMA/pVS\n6pgSu1MpNbLn9yUA74C013/7g774V8iyNnlhexhABOBXlVKuUur9AO7lb38I4B8ppV7Pey6xfSoA\nHgHQUkr9U6VUgYHQ25VSr7sJ91CCDK7rAKCU+q8gflpA3BhvVcL7rUFcLHvt2n7xOQDHlVL/JYOB\nvwhxK3zmBq7reo71t5VStyqligD+JYCP6Zenwf0BgN9QSt0GAEoIDB94uYvZd4My7VcgM+cFAN8E\n8B8BfJi//SGALwJ4EsBjkAaNIMu0HzX7PwD8JpeWv3Dtj1rrHQD/HWSgWYS0yV7mwb+GTFJfBNAE\n8O8hwbC9x7gMGYT+mdqHPM8XsKxNXsC01gGA90PYEVuQINmf87dHAfwDAB+EBKjOczsDch4AcDeA\nixB09yFI8OuHalrrMxDf7sOQQfYOAN/ib18C8KeQ9/oUvndw/T0Av0AWw7/VWm9C7ut/ArAJyX14\nQGv9cuj1ha7reo71J5D41wqEjPCySW9a608A+L8A/CcyW04DeM/L7aeudlH96JlS6j0A/kBr/fJU\nk8wyyyyzfW77FSm/qHEJ9lNcZswA+N8AfOJmX1dmmWWW2Q/CfuSQMn06XwPwKkgE9bMAfk1r3byp\nF5ZZZpll9gOwH7lBObPMMsvsx9l+5NwXmWWWWWY/zpYNyplllllm+8gGyuhzXVfn8nnEsbDPLGjY\npLJ7jozvLj8dW8TelFKglpBQxAFEkexvHCe22VZrJDqR3xL5VVl7ufJAksTp9ul33EfxBAoKFrex\nLSu9DtlW89y7x9Xpd8BWo4V2t3/1SV/CKtWaHhmfQNDvyr0FfWgtu7ue6MV4Ofm0XQ8AYFkK/V4b\nABD4PTk329Tcg7l+ZVkolSsAgByPo+MIANDrdc0dpG3Q78nxYm6T3psGoki2SRLznfztOA4/bR4t\nhvFqJbIJdhrNDa313iSNF7XR0VE9Nzd3PZv+wCzhhUaR3Lfj2GkfskwfSPuSfOr0Xzdmp06duu42\nketQ2nEsWCb/Q6k95zf/utqdGMUxLL4/BkGZPgzejzmeZVmwbXmW5vmbZ21MJzo9Vdoa3N/my+w6\nDsIwlOPwXOa4u/1C+qvnWun+Sil0ugH8ILruZq3W6npsfCq9b6UsWJZ5PtZVLZK+33rvc9PmJq7a\nNt1H7z5lc+1pi+y9ypdx475gRpN+4V/13m+40fzzZ667rww0KGsAbj6PzobQ94bzwOFhGSiOTZYA\nAHOHxgEA+RwbNNbQSgajbj+Qz54PAAhjDgrsDHlHpwOHbcml5XI57tsBAERJgNFRyQWwODaHvhyv\n4Mi1+H6QdspiUa5LWa582vIJvqjdfoiIHdB2cvjwF58cpElQHxnFr/wvvw3X3wEAbFw5D+1UAQDT\ncyd5nXKu8ckpuV8V4fR3vwUAWLkiGcET9WEAwMFZ0WCfvUUYftMzBzA+PgEAcF1piyiSdjT3GEUB\n+n0ZjD/2sY8BAJ588hTvSTpFFNoIfPl3j9vuNLcBACMjdQBAvmAG5wSbm5KVa/b5wue/8aJpodfa\n3NwcHn300evd/Adi26tLAIDL558DAHRaO9hpSp9500+8AwBQHTVZsdJxEqjva6molLruNgEEsMxM\nlNLJIYkT5Dx5N8xkYnNiRDrQKlRL0oeDltzPRkP6WnlIqMZ1TtrFYhEV/vvSpcsAgDCWdyOfl75T\nrdawvS3PPV+Q92V6St5Zm0PJ1PgwtncENFxcWEqPDQDFknz2u/L75GgtHSw73Q6+/ND5QZoEY+NT\n+N9/78O7gCDRCEMZA9ySvEe2KzRzW8s2llYcbIG4J++C6f8e7zOGHKPda8NS8l25JO2lLZlQzMSi\nlNodzHlcM16bSS2B2jMhJVdta0BqCvyg08nMbPvfvP+u6+4rmfsis8wyy2wf2UBI2QJQcFSa1Hpo\nJI+5CZl9xscE6RUMMuWs0fP76IcyW2t+5xWYYBVxGZ3I77XhIqJQvvM4O3ISgu3JSf2gj5CroyK/\nc0qybZ5/R6oDi0vziHOecbOUOdO3O7L0D6PQrALRau4giQdMDNQaCEMEvqDtbjfA3PEZnkOQTRD2\nAQDDo9JWjmvh2LHjAID73vBaAMDMhMh51GqywgkduY5iPgfHrNCIpnodQSk+EX6xUMRQXdDO0SO3\nAgCeeeY57iTb+H4XteoQAIBeFOw0RZ9FQ9CGQQXb2x30un56ezdqrySzxxzbUvK5snARAPDkw18H\nAIS9Ltyy3G+vKciyOix9NDHLWWX9cEQfaEopeI6VuvOGRkfQoQvKjQUFGsRskODU5Dgmx2RlePG8\nrKpGHelHk9MiD2FFdG8ohSrR70hNELO25d2o1WSfYqkI25JzjE3IyiHvyeqxxXaKdIhaXbaf4TtK\n7wUcV/7O2fKuJUGMakUQrQ6TXdfMdVqcxGh22qm7ZGN9E1cW1+SceRlLyhV5jjlLzqkVEESyfRLK\nvXRb8k4UuJqEJe9/K2ghCOSajhw+BgC45aisQgt5aaskSVJEayCycW8matfnsesKfOFeY8Y8CwoJ\nkfqN2ECDslIaeRWhUpHdjs8MYaQgnclNZOBpb3FpnUhH6XUjWBwEqnURZXI4eDZ2WvI3r2K4UkSL\nS86A7opeXxrfNFK5VEIYyFLFimVHly6OOJZtHVvB5yDpGT9uIg/Pb8vSDbHpXEDEB7LT8REng72m\nOkkQ9XtQ9JPnvAJ26N4ZmZSB9uBt4pIYn52W63U9gJ0qjKTdnl3eBAB0L6zL95a043NPPYHXnZSB\n9q33ilyB6RRNvkSX55fgufRfe/KCjI7JxHB54Zx8ny+i3etwP7k+x5U2rVZlojI+6jja9T/nct5A\n7bHX1IAv6CCm2elDX655aUFWh9WiDELFegVr29K/NpdFV2Zilvo9lvGdf2/M4pU027ZQq1ZSt8H4\n+DjWNuW559mHd7bFbTQxKpNzLmejUJBBc2ZWBuES3RlhIH3agzyjnJdDlzGF2WmZpLUr7eTxOQZB\ngNERggMOXL4v/aJi+oHfQ2tnm79Jvx4ZlUG+UKKLS8n3TuCh35FzRn448ETc7nTw0LcfRptAw4KL\nHl1m/VjaxvXk0+aYEiugryP+W7YtMX5ToNZ8PifPOLYCdDryrj365GMAgLUNcckcOXwYADA6OooC\n3TMmDmFcEmm8KrFe3u9s3BpKpe2QDvYDWOa+yCyzzDLbRzYQUnaUwlDOQYGzeq1UwFhVZvGYTnOz\n+E8DFpYFPyGCNVF+zj6xYR4wELa21kAcyhFaXUFA3VgQY7kgCBB+DJsoySxdbbISeh1BnUW3Cocz\nVZ/BxR6XOQkXrI22bNvohmh35bd+aCGIBpvZdJLA73ZQJvqpDo/h1XfdDQCYPSLLpRaXpM9dEEnW\nZreLdkMQ0WZDUMDyiiCTKt0XsMR98Jk//Tjcvynt87Y3vhkA4LrSnpOT07yIDTSICr/7mAQqHS7j\nSlxaRrFG0JZzsrkxRpdTzDbe3BIEbaGYPqt6/YeuW/OyprVOn/36lrSfCWz5/LuS99BtS5Lns08I\nQpqcE0XP+uSMOVAKfl5JVG/McRyMjo6k6Cno9zExKYi2mBeEnyPrZmqMbqywi80NWc5XqoJWHVce\nYBIwIOaYoKBGr8vEVt6OlZfj+Vxd+oGfBs/bTekzpbKgRIMON7e2kXONG1KOEwTSH1ttg2jlh6AZ\nIwikP5ZLpRRpXq/FcYJGu5cylhQ0HLpTikS9JuhvVgR9xIiIJ1tdrqjpKsyxxGdZyz3aDuDmpG37\nfOefX5CV0/zyCgCgXq1h9oCsasdIIqgPicvE4arK1sn3rAJiE/hL2R27rKYkRcqDO8gypJxZZpll\nto9sMKRsK4zV86i4Mnvk8zYsW2aCAoN3IX2ru7NHgIDBgpgzaqLpJyZC047MgK2gg5gBjy7pchE/\nW/QLLW514NIXVm3LOcIVQXi9HUHXB0dvwfi4zHyqIn5Xf1sQVLstM+pOS2bNjZ0eLi3INrHtwKef\n7npNWQq5nIvQFhTTK5RxsSmo5PFvPgIA2NoUdLG4JIE111bpPfiRofTI59SYPJK1FfpIcx5aDUE/\nZy9KMGtqSgI0rivbTs1OYpr+xssrgsafe0o+x6cEcV26vAGQamQQVsxgogmQ5hxBKL1+jGpVELbj\n7JUq3i+mYaRsF6+IaufFy/K5cP4CAGC0UsaBUUF7y5elLZ969K8AAK+9XyiAxWrt+yMqD2gKgIUE\ngS99Lw58RKYfkOfucBnTbGxxnzjlsC8uLwMAaoYCx/emSTqm1hpeXvpEaGIWRLiKFNAkipHYJv5B\neijBnKGqerkiPK60inlpIBNb2OEKb8fQ8vI1KKL7YrWW8uuv1xKt0QuStC8DCjo2cST5VLxeE3ML\nwj5Cbl4pSpyq1ZT2a5oVAVcjnueh4hmutdxDJ5L7ND5qf2MHjYa8o6WyjGNTU7IKPXpYJJzLXi6l\nL5qgJF8naEOx1LtUOQOq48GBcoaUM8sss8z2kw2W0efYmB4roeoJmiwXPSii3jQjh7OFz0i+BYWR\nivglSyXxuzZ3BNnWiMZaZFjML26g7cus43EWminSD+3KDHhpswGfJHKXU2eNvrb7bhV6WXM5hu7y\nt1FBA35XjtNuyzyUc+X72clKmpyx2uxj8+zKIE0Cy3JQLE5grSFtcn5hAWeelvJvFmf/mEyQHsn/\ntpWg5wv6bbTks8Xo86Urz0hbFeSeThw9ARBNf+sbDwIADjFqfPyE0OpGRmrIESHVqoJwrEiQTMc3\nLBgfvYb4EONYkFqeUX3jWzTUplzeTv2E3a7JGrwRM/75vXD0RaCpBrSBbNrQk4yf8VrsoJCQTWMQ\nYasr93RlVRDm6uoW4lj8tQfGZf9n/0pWLiaJ5/jr7oV5BSzj0zTIhqfUJvtN3zjFadc0FDQ8z2TH\naUREhT6TH4YKgu5dskIcy0U/4DvB2EngS38IyFTyiO48z4PiKjYmGizQVx3yeVaqdeRJBVNkUBg/\ncRgQkbq5dBsQFfqkSMaBNIznCEKtDg8jZLym2ekiHpBtkGiNnt+HH+5m3ppzpwkdfAaGnpYojQ7f\nl3yBSN7cdyh/9xmvilQCzf08k22Wdif53nHsdJsWk2J2zsl7uLEpY1UlX8OBGVl9D9Hf7OVM7QTS\n5xg7ihKkPu/4ZYuTfK9lSDmzzDLLbB/ZwD7l4UoBTiB+pZzroJiTyK3fM34WmS3qdZlNtNYIYhn7\nQyZRFMsyyy6ty+z7/LyguvVWBBIhcIj85/e9RZgMB6Zkn4+duoCHzwuajRJBDI7FWa4hHN9u20el\nQn9ZzBTuvPztMRpdVPJ3FEc4SP5wZauFJy+uD9IksG0H9eFRnF84CwBYvnQRRVfua6cjjIp2U6Ln\niiii0Wqj0ZO2cHJyHaMTguoKXFXMzN0FAJjN27j4xMNyLiX3G9LHuL4hfvI77jiJW46J72uWPuTy\nG+4BADz5LFkJ/Tx8clYTCCJOyPVcWRHepmdYNUPjkCpLQI+81xuz73Wo6WuR8m5+6x4NEiZQGMyQ\nImbzufv/g9TYKBLlN8mZhbJwekHavUC/uEO//dMPfQ0AMDIzgaED0m4qMis9o6FA9MO+Zf1AskwU\nLMtKGQqFUgF9Rd8nucdxx+em8mpOTkwg2uTJuWIqmUQqJkzUJoVFs3dVMzoh/cBvyz42+7vr5pA3\nbATqr+Q8+dvy5B3b6fgIyYKymcrf52oWibw/JvHC8Tz0QznH+sY6wmiwmIzWGoFOoGKjy5IguZY7\nnuMzob89saI0t8HkLHiUWCgXKOkQyPsVIQJpz/D5jHNkc9gwfHUrHbci8seMXsrKlvShJX8T5+fl\nXRobk5jO9LTUUi3Tx5832jSWjVATKQ+ajIZBB2XHwfjwCHpbcsOWctDucmnOAJlDSko3NKJFQI8P\nrT4kL05A7/eFKzIYbDUpUOR4sNnw1bx8N+7I0jq/JZ31WHUSy8OyzWpDGszvyvEfOysDoxUlCJk3\nj5q4JsAHUavJJFLhi9EPQuhAXAhzYyXk3MEWD77fwfPPP4Jnn5ec/6Xl5xHTTVGpyYt24tgcAOD2\nk1Ijcnm9h/l12WZsUq7v0FFxSVRGZHBe3Zbf9cZFXGZnWCd9jrkk+Mnjoq3RafdARiJ0wIHn2zKQ\nHzshk9rETB3ffkSy3VZW5X7NsrNP/YBt0uoK5XoatOiQcnRj9r1tqa4Z3FKaUaKRcKkXcvDxGFhR\n6U57RGfYz4aG5AV581vvBwA89fizAIBLF+cRM+h83pZJPD8nk2/8nCTUPPW1b+H1PyODV4EBo3hX\nK2jPGYFozwSjbjA6GEYxFtd30nsu+QnK7CN9ug7KtrzYM1PMYisq2Mx3GipKe9SLsk1lUu7d54xx\ndmUJ9br0e5+AoE+U4/K4YTNCn1oxCdvQ5tK/3ZbnH/V239Gxurwvw8wGPdeSQOoIl/DKBqrMqE3C\nChx7c6A20QCiPa6hOInQb5ukMrr/2NwOE6q0AlwmPjlmCEsz8uS6ywxiRhbAeB5CI1pFgoERetJR\ngpiDcWybCB2vL6VMuogY2WsuSdvOL18CAOSYuGL0QfL5fBoUdOkmHcQy90VmmWWW2T6yAZGyi6HR\nMQwxsGBZLhpUGgtNmqRJT2SQR7sOymXO0pDPZy4Iou0wvdMoWOU9BwVqUwzZMsOfOi80siiQS/Vr\nkxgbYqCCy3CTqtzlUqbT1QiMhgBRugE3JoCi6fR3HQcRkYOO9cBaD512E9/++pfgTJwAABw9eQcK\npJydvFWSR04clwBB3OdyyeqhA5PqTFlPu857kbbotCRgVQsiREQtl9eo7lUW8rvRsjhydC6VOew1\nZAn77Hcel3P15Fpuf9e7ccedslTvPSpI+fnzlwAARaLEWn2EdxWj2TRptt9HoE9fAzeBFHqYoF6K\nRHWEc+cFwfaYDv6qk7ISyDFldq+uQqKN0pv0i/ve9BYAwOWL0jYf+oMPIeIK4PI63W1FadtjXGk9\n941HMUb3xavedC8AoEvXiUt45fGcW90d+KSXGQQ+qGmt4UcJtrbk2Ra7fQyzf7q8j3yZyJlJIO1u\nlDaSzT7tt+Q6xiry3J47J1TJcr6IMqmpPgNdQ1Pi2lAxkWPXB2PCaPVJjeOye4VKe0gKKNekP/YZ\nsDdKigW6/yolQYJbrTb6pPhVyuWBKXFaa/hhsKuwluy6sSLeQ4990CX6tZWV0jc13T/K9AeT6pyY\nNGmgSxdMYJLO+O4HPKerFTSpiSEV5FJ1VHM/qm+EJdM1U8I+EtAN1OywX8QB4Mt3N5KUlCHlzDLL\nLLN9ZAMhZUABlgu1x0+SYwCtiBIPSMUqTishEuQKErzaWBFfUXdDUNgRajFzokW+VMSJo5ICa/HL\niPrHBrk59g4qnpxrZEjSZo8eE6GZi5clOeDZs4vwHKNyJjNWFJH6RMK9mXWTJEkTXZSyBvYWhkGE\ntYUN3HPXT0t75MYwzMl1alqQ/BapaAvnBSEFSQ4W6Ui2w0QOzQBPZGh0Rvw+QbkmvsNNJr5YvP9U\n7Bw6ZZ+V83LOOQYh8vSRWWjjjtvFb12vCwr6VO+LAICVZWnbmXH6XFU/JfM3m6Ye7TMDtszu9am9\nCltELabugXHeLixexqc/9xmeUwK/9zG9+O1v+wkAu9raidYp2c4kF5UrEmx54L0PAADOP3cWX/78\nl+R49J0/uyi+5SFFVcG+hW//Z2kDZ0RQpzUhbdNhcoRLxLXcvIKdlnzX7/cHbgtAqFfjwxVEfemT\nlXIOmv5zm8UhCgxUmUfb7QUIqAJnaI8nT4jA1cqKrCJ9RrJGx8ZSil0CvpdE3kGXOuUFBZtosLMl\n97PTlU9DUW13NWJKIxjqqAngzRyc5fHluW032yk6rQ+Ppe/99VqSJOj2+3DMfomT+od7Hbk/j8kf\nw1RSLMSAxX5km/ay5Hp3mCTWY4r9ocMn0AqlDba35T5zJCeEXKUoxLvvEuOU5m+T/OEhhMXVe0Ta\nnRFdM51Zc+WfNBawuSi+d+jBcW+GlDPLLLPM9pENhJQTrdHrh1ChoUlF6HRkRgpI/o4sQb/trqDD\nZreFmVmS5SP57tCozDRHp2UWNtWXZo7fBU8LCtneoQ/L+Dk3BX7OTk6hQfGRI68Sn211qMhP8UFu\nr7ewvUOkQ1RpUaAkNL4mQq04jFI9Za31C5C4Xtosy0GxPAzKzKLRWENuWNBWl+JGBlgVhljWKVFA\n3zBO5Ld+KH4zU/3DIv0tsRyURwTBelqQtl0g3dCjH011oWLeJ4VvXfr8CmVWtvBb2FwU5DFSEsbB\ne3/qXQCAR5+4BABomyoO/jp8UuHqlfqALbLX6GNTFraJYHa2mT5MgeuVdUHDDz/6CE49/QQAoLkl\nPmCfSOa2O4S1Mk4qkm07aLakvRpM+52joMz0AWGv/PI/+NtYWBT94e88ISJNfkfa69wVQczFSRub\npyXRp/vncqlH3/RqAMA2GQBd+nZ91UBAXfAbEZkBxCdeztk4eVRWdoViMX1eKwuSQh0x6aNUlvto\ntPuwWbnHsD5alLxdX5O4RGjyt+CizUQQI2XQJXum3ZROWC1WEDB9WSuiTaLUKlcbhaKTVgKpVBjz\nsK722V68LGn8yvHg0e/a6vYHl76FRhxFqaN2KFdAlXGlHhPHwHfBbUufzEcWxselffoUAjP6yoW8\n7GsbCddqFfWSJAtNjprnx/eSaLibJFhZl3cj7Eh/ctl+DuNVdhIgDMkKseUcCWNkCZld6HHMW7oE\nf1uO1277A7UHkCHlzDLLLLN9ZQPW6NOIVZz6BbXWaRpnuSKzx9K6zGYXr0gShuNqeIzq9lflu2Pj\ngpDfcb8g3ecXBT1VZsYwOiLCOmucuep1IsCEyR+WjbV1ibA7eZnV1huCMhaXBSW4bhH1Kv1SPfoy\nnauLZ5r6XJZSqVjLjYiHeF4OUwcPp8fo95tYbVJqsC7ILoyIdOif67XbKbncCP5ErORQpF9vfETu\nTW/1ENAnqhLjdyT7hb7rREe7xWzJOTVE+3anxX0T5HiNTbZtoSiR+be+8U4AwHPPi3DP6TMraJsU\nXrJDBjMNwE/bGArYobD+NxXDtAcAACAASURBVB76JgBgfkkEhDaacp/bnRYsovu8L898bdPs8w0A\nwNyc+DNzuRwW2b9C8rJ7XTlOu0Wk4wAnXyfMisfPPwUACFrygK9Q4Kno5XCgJvd38dHvAgBs1pa0\npqVtdiJB5DYAaLk+3x8c/QBS/abs2SixOo/ruaixNiOzhbFN0funnxGGUpRYyDGpY7gkK6SlRen/\nmyym0I+MfEFr17/JlWCjQXYUSUiBH6BYlD4yTLF7UwnFNwWNE53WcdSQezUVUcy9G6leU2kIABzX\nG5xtoDUQBagVBaXXiw4Wl4WX3zNJMiYOQZGuwyPjGJ+V2NOzS0u8ZlYjYvJQjZIOTy08gfKk9OUy\nE7Uunj0j98D2rB+7E+Vp8dN35iV2YtMnXWVMqttuoNuSVZ3nyvNokk1VqMvKc4QPsY1wtzht6mO/\n/vTzDClnlllmme0jGwgp27aFer2MyJGZq93uQzNzz0Sm5y8bXwprZuUtLF+UWWciL0hjZkZqZNWn\nhQ3gtjiL5F0cuEv4ovkVQQOFSBBRDPHtdDp9TBVlZgoYeVclmbkOlJguXZ9Ea1P8hmurgjxCppn2\nyTU1ebOlXD7lGbqeO/BMrxWglZ1mx3VbLeSIZFtNsi36cs4uhX9cBVRKggLGhgQpVYcFcYzVWZuQ\nddh6uQhbh+S+/FhWBAhN2SZTW08hJs/SCNLUhwUFJDG3DSPUahSuYdZTg6hSh3L/d5+UVUq9ksNn\nPiOshPXVjYHaAwB6/S6efuYJOOSShkGAbfp+G22WsGKJptq4xAyGawWMsATS+vNyn8+cFoT7pS8L\ni6JWZVVjx4YfyD0YGcz//AX5NAmZ0wfGUaQY1V13vwoA8Ng3pW5hl6jl7OYqCvTFD0WC1M5/W6qA\nN8YEaW2xXd0gj8g84xsUafJcFwcmx1OUOVQfgs2sOndUnpepx/eXX5VU8CSxUa/Q/74s9zhBnn69\nJv2+sSbocGNtJc2aLXHVUePflZL0s0qthlKZnGXGDS6cFwRqk5nU9QMEXIEELAdlMm0V265gqkYr\nd1fK0u+nJZGu27SGFYeYpPTC6vYaQt6vQx+3xTaKQkH9h159G7Z5HQHjSTbT0q2qtE2D71qr30PC\nVZTfl+dX4zYLHKM665s4REbS9AlZNTbOcLxZlLbZXp1HsyNjSUw2zE5PrrMwJP22MiufUbeJPmVQ\nDSd6EBss0BdHaDU24QRmcLFMtXY4dPZ3+dINVaSz10t59LZlUB6flg43c+fbAACnr8iDP3tePu+b\nGkajIf+eOCraDxbkBQh8GZzrOkFzTRqnQOWrKRbEbMTSUdw7h9CjS+Nbn/sUAODKguxvGw1Zri96\nGggNjS8cvMaYWX451OGo5YHZmhz7VUfkQZfp4rG5TOw0G+iThlQoyT2cOCb3MHtIAlaWKxNXu9HA\n7JQEKk5clOVTlVTC4SGjeezBxFfIoUeewZKIHdHSgGtcLFySjozKi9DmINNpyEQ2MzaG9/3M3wAA\nfPKzXx6sPQB0Om089MhD6NEFUsqX8MAD75XrYcD11FOSDl1jUcxe0sc01frCVRksdljctntOBtMh\nuhZKtRLKfBHyJXk5a3W5cUPrqlbLKLCixv0/8Xo53oa0+enTQleKQ4XLDQ7mdC05K9JerW3Snyp0\nFRVGschgXLN5Y6nnGhpaJ6mOsW1bCE3FDAY+tXs13cqy3N3lLGlqhw6xthyrkxyg2y6Xc1Fl2rbN\n462tyeR33+sF7ExOTyNiML25Ke/ENimqmw25FsfWGGORXxPUNAWFaxw8txls1JZCQB2XOIygB1TT\nc2wbw9UKRqkf0dhaxTBptjm2hZkMx49KgtaRqVk8fVmeYZ06zxH9M+OT8s5Z7Nsdx4JVkW2216V/\nH6LWetejvEDcwda2tIU1JUHYA7e+AQCweEX6ab/XhWueEf2cNp+HT7mHdTBNvduFZRvti4GaQ65h\n8F0yyyyzzDJ7pWzA5BEJVsRc7msoWDBVZQWpbJOe02wywOYHmOLs/bq3vx0AcOCEzEJ//kcfBgBM\n0v1gBz0sXhAa0+QRUd3Jj4gDvqRJUdpaQyERdBUwBXSD9Kj6mCCIkck59NqCmCzqEsWezOYm0JcS\nx6MYikI4UeQMjJQrpSLe9sbX4MitguyXFhcxwyDR8WOS3DI5JvQdm2nHrVYDPl0Q5nrKVAkzKek2\nlbvcJECvI7P4q28X9Dx3fE7uITEVGixEVLnSnM1tJn+EfSKdMIJlgp35VOEFAOCHpgo46y0GDYwR\nabz5LVJB+88+/qXrbhPfD3Dh0gXsMC382OFjKFAneGlJUMX8xcu8b6YFh10oVmzpUZvacBVvOSoB\nu6Njgt4qQ1WsrXFFxpTpqVk5vqlA4SVAnopmVe73k++W/rfFldvqlTVs+ILsijtczZmKK3TxzFTk\nWZYmJrF46RIAICDdc1ALghCXF66kz7rV6qRIz9DUYrp8ily6B70I42MUJ7KkfY4ekSCXqQZiuXRL\n5dy08rVl5ARI0/Kb1Eyu9TAyJe1hRXK8Q7OCHHN5aYNmp5FqPjt0C5g0a1N7M6ams50vQZPGVy4N\nI+fOD9Qmnmvj0OQw3v8eSRCavzCHFpNrfKr6Rb70h7lpQbE60dCj4mrb4XvcoQ7ygVF514zIUbvT\nh6arpaylHW26jybozuusraO9KP0mZH8oMVFl+jZJ30/CHawtydjUJV3SqIBVS9ImDhgcdYCwy6Dp\nDYhXZUg5s8wyy2wf2UBIWUFSZmPOmsqyDNiCpp4y9UEwPCL+vMlihFe/VipknLxPEPL2Gn1grI5x\nhMT/RCWYHKeznL7QLn3MRmAo7DmIISju+UWhVT11+lEAwH1vkG1HJkfQJH3FlcvA6BxTkw39jVKJ\nkR9gh4I1fquYSlZerxWLBbzmzlfhtnsEKfduP4pSjXrF3EYzeGgRiQ6XJtPsy5QwY2QF6T9LKz74\nPRy9hckGTITpdViTzZDWlZNWTkjTQ/cIvABA0OshTkgvNNWPTUXgTUEJ8xclIeBNb74HXRLlTY22\nQSyJY3R2dtAlrSpXzO8GghcuAQDqbKOYFchV38fyisifLi9JcFGxovff/Pn3y3HbEjj9yjcfxPyT\n4isdqQlaXDnHBCSiqZ1wFXClDwyPiK/6jhOShBK8T9rtw//+T9BjrcYl1mgDg10+RaXa1KyerlXh\nEYWOjovf8vKlAdslSdDt+WmKchDFGGZFcVNJxegWz84K/e/M6efSatVTk/JujBE523zZjOqBl3NQ\nLBqBKz63niDKHtPlt9bXoC255wKfrdmnWpG+0uxupXXyDOVVOaY+nbxj1QIrYDsKVUqKuvau5On1\nmq00qnYfb3y1PLd7b5tBi1VODG00pA5y1OVKqu/jcCCrhS4DkW1S4Yw8wDbvN3/YQ88IjpGiurgi\nsYFzXK3dOjSOy+vSt4xedJyXlUr5kCQTveXoHLYWBCk/910JBq+tSKyjpKityjTrfmyn2ukOA+99\nBuWvxzKknFlmmWW2j2yw5BEt1XB7vqmWUE5pTzYFqG+ZlFk8X5Dxfu7QLO56s/jypkg3efzhPwIA\nHJwlDei2O+R4Y0fhFMXf1aVfqUdqy+qSoLjt1SuI6Y8tMAV0lNSnhaXHAAATUzOI6GPSpKYoin7H\nmn4fIstCzoU3SdGjnDJa+NdtlmWhUCqhTL9VqegAjqluK9sYmp2Rnkx0gsRUljaiPUTwkZEXNKnf\nykKZCQZGfCdOTNaIEX2PUx+iUQQ3vsk0cTwKoOgDy3F/lxVhSkZSlKyH9QurOHBCVi8bVnuwBuH9\nBX4PXSKH8xfP4xOf/DgA4JtfE6qXqfCxSl/n+vwCWBglTYX3JqUvfOvrkjziMwHlzLmz6KwKsmys\ny7b1EekL62RPNHc6GCK9MIglEePBByVBpFAVFtDQ6Dg2QkHCXfotF4mcNatdFHdYV3F9DXUmW9hM\njf7uI48P1C5KKVi2m/pKc46XyoHm8oYBxGdMGdrWdgNdJjIcPigxigKvrcyEi9qQqSQfIqaAu6Gw\njY7KNmukzS2vb+HUaUk7v4UrsLV1Of7SssQuIvios+6ly/5o5D0j9m2f2gGJAoqUFWi22wN7UJMo\nQntrG1cuSrr7gZnDmJmSlY3D+0vo124yWabR2MbIsDzDDlfoXUoEdCja1WrLszpx9Ag6ZLj0SQEc\nK5ClxdqZr3n9fdhisY5LK7KiCygXEZNZgqExTN8pMauxO39S2oKp1FvPfAcAcPG0CKJtPH8Wlkfx\nMAqOwc+QcmaZZZbZj6QN5lNWCq7tYJtsh7ivUKDwh81kjHH6kheWxU979NXvxoE73s0jCDIOWS6p\nxnp0Y8elZFHHGcbTj8ls41PovMk03I1F8f/YcYA8JQxnDotf6c7jwtCIbKav2nW4HhkFnNG78+KD\nTJhKSv432raN4ojsNzE9AnfAclC2baNSG4amv7jrB9D0Yfn+1bN3QH+c74eIKFaUEu/5m0lM6DI9\nOkoSVIbJOqDweL0ivrE8S87ESQBQXMawYYyQzOYaRYZ6bSRkrSjIfkks11mtCHI4dFAQSq/bgaaP\ns1bZTaO97jZxbNSGa6BGFZrtJs48Lqhy9eJFXqc8w6Jj0ue9tJSVEVA9MCXPd5hc5m36FI/MncB8\nLCufxhYJ/Tlpm1X6qLvdGI0tQTKKHPo+fX+NrvgGLa+AxKb0I8WdTGJJzOdTIgumXBtK0WdyAxWK\nAcB1XEyOTqYlx4o5D4UiubhEuC6XV9W8tP/RmQnU+Y5N05ddpuh/lanEfYvsi8RDc0f2y5PV4hal\nfVfWZUWysNXFcywcsbJGvvIOmRlMIrr15BTK5ArH9O8aX6thJ+XJtY6jGIorhyiOMKikl23ZqBdK\nabLXcpJgdFLapMbjlowoVk2Qs61CkD6OGvnN2rqar/zMGeEXj42NoViUFUGX7+Fdc9Kv3vZa8Rf3\nIp3WBj02K892dVP62tKK+JpXLi7gMvnJfSL4Ql1Wk/XbZXy7+8QbAQAzF5/Ekw99DgCwvnKRd2ok\ncF/eBnNfJAn8Xh/FnOym8jZci1Qs5qcXyvLwfvYXfxYAcN973oHqqLzsqxeYV859Ggz+rF8Sh/lS\nK8aDn/wkAKDMoEqfCv6TEzIwVSslXLwiroyAxxmengMAHL/jNXKhcQ5bDQkCGgW67R7z5ynL1mdF\njrbW0G3pnCfruy6H67VGo4lPfurziF1ZYm9vr6K9I8ssU2zTDM6rq/IyxInGMGlyQ6OyDMuxA3ao\nkHaWJc6b7TZmDwsVzmZEp1qRfQ4fJtF9dhKHSZMa5tK2wpcqYUANto3QaNAyOmtz24k5DvJVKunp\nGByrMDxcHaxBIBNVebgGhwN6sNnBxll5ZrNl6i3wJWpxedi3IigqfuVIr1xflRfi1HdEPW6CNLHN\n7QZ2uBRtc3XY2zCdntlgtoeCa+owsrAnswpjZlkVnULqNrLyJvOKB6RKWIcBpGazh6ERDg7JjdXo\n0wrQloU8g2SuY8FlQkyf1URMwdIaC8Heffdoeh+uK23mOGYy5rUycJfzHJSZreeZYqMJVQN5n2ee\nfQ4dLtURyyDlcxnv2YZOl0uD0wm1l5ts71aXBX/ZQYIgQsSsysD3B87oc20bU8M1KCaCba2u4Ykn\nJeD72GkZFyZmJOj5lre9FQAwM1ZDf5uaJA5HZ8u0jdzvwWmZyAt5FzmPKngeo/5MJglj2abVC9Gj\n2++Zc5cAANtMVnv1EQmutscdXFyWieOZeRnwn7gg19kiIBityvFvnZjBa98qLo7HHqamNxNMrscy\n90VmmWWW2T6ygVXiEh2kpGkVJYiIKEzF4XyOM/xrBLXmXBdnHpcA3DbJ1z5n1ha1dRfOi2pTWxfg\nxvJbmQGFap6aEEOCsJZXV1Iie5cl1hdIbQGeluO0W8g7pNHkBJFuRnJdBaKxItc/BSeHFjVzoyQa\nWE+52WrjS199CPUDkgKq4zYee+irAIBDpPqNjgiyXaSOb5TEaXAkoLbCKtH/O+6VJdDdd94m9+j3\nYZHmc/GyEPPPnpN2fOq0tGu9VsbP/8LPAQDedJvQDz3SiQ5MCcoIbHtXIY9L0NCUU3cYAKxL2xQs\nC4ltascNbloBiWdBE314tgWXCPBglUFLotUWEZhdLcNiVeDeqqygfNYbbG2KK2eDqccNv4u5V0vQ\neGVd3BcNVpUopzXuOgipcNdnEK8Xmhptcl15Lw+tmLRBhGwTaVmkYRmq4tp6A6Y0n+PdIFJOgCCM\n0GL6uFUposeqNCH1gIsFLtGJ/BqbO/CJlHeoJ2wQnuZ9Gcqca9no0iVlpKwDBrrN6nZlZRm+ZsUf\nmwiZyNvmaqHbjRFxdWGqMu+Q3riyKS4gbfQVtIJiFZ1CzhmYE9frdvDkY38FvSl9uzYyhlNPCxJ9\nlqj1TW9/BwDgo//hTwAAP/OON2Moz/GG7eWQ+9rrS9uOsSp8kith+xpVP2WbqkhMpnLzOD8vK+t/\n86//DQBgY03Gpte/4c0AgAc+8EsYJyWxxKSb6Uju9ekGg/Zcua9dnscxugKPnJAkuLNPfee62yRD\nyplllllm+8gGJIBpAAkSEqEdt5hW9g0YYJqoySz+hU9JvbXhiacxbtAaRXhcl2mPJaa0EjWVXBeT\nVA3rtWRGLlBneHOdVRaCGBUS2gOqPJ17TJJHlp8V6pMf9YTJjl3/YekAA1YlBpNygsjzSYQhyPFO\n3nYYhfyFgVpkaHgEH/hbfwe5cdGG7rZWcO4p8YFOTcp9m7plBdbPC5Iejt8u2w9NyYzepUrYA+95\nJ4BdJN/x+6kL06SO9lkNYY2z+fzFJRSLcuyVK4IcLz0tlaEtBjovrKzh3r/xWgDAoTlRnTM+Zovq\nfXC5AkoigOjHU4P5CAEgjhM0Gi34XWnrUmBjbFLOuTkvvrXzlwQZrYdyfcPDw7D4XDsJ6YushRYx\n2NRnokCkNNZXpD902oKMdCjIqcj6a0GvD8WafhFV+jymN2tSC/t+gISO/4B9Oke/rWdSc1npu1As\nI+Q5Bq1DZyyKI2xsNzDNPt7qdBElvP8RWUGYNPGIOs5+EKRxjmfPM0jKZ+IR8R3k87TKOfRZUTkm\n0o1Irctx28b2Ds5S+ezwmAhdDTPg7jB+0OmE2GZil8N0a+P73+Znoo1qnAOXQeZO10c0YKXvME6w\n3ujiWZeCYWubuLwsyR1vfcf9AIB//pv/AgDw7z74/wAAPvvpT+FVM9KGLgO0Jfrgja74cE3ac2x4\nIvUze0T9Fil2bVPl2rHw+38gNN0zz4oyoekHn/jU/wcAOHDiDtxxTFahBdIDq1r2n5Yugoixmk6s\noEl1PDRzcKD2ADKknFlmmWW2r2wwpKwVkkTBo7837yRploMmHS1hFHVjQ/yn7fUVFELx2Sb0Qw0P\nySxXn2ZKNf1gi0srKaXGYhaHSa+2qYdcyhdBthJs8w/6s+NAZncrUWh2BW0FOUEKlWk5R6cgEfgW\npTb7HQsjVRG8GR0fgeMO1iRKATnPwtlnhfze3FlJaUOmKkabVByTRJLPuQgparOzLtuusubZ57/w\neQDAdou/t3dQoUhOjdrLJbIkrlyRqgvjozPIVwVxf+Ozsv/WuSfZJvI8zq+s4gppdsdOCkqvMVpc\no7++wHTbWsmFS/9isZgbqD0ACDuh5wKmQLfywPJ4WCazYpnPrs10ZmzuwHZJB6Qf11ST6EWm0g3R\nu+thkSsnk1Bj6tetbzPlVSlooiaX+tZVQyE0FTa03q0iTe+5kVw0TAfFfXSSpL5Ig7QGtSAMsbC0\nBJeruCjoYXZW0qA7XA002wYpUx7SstElin/mvKzizMpyiVKio9TOrtXqOHdOGAHmPfrZn5YYRU5L\nHxqqV1BoSp/YJBslCUy6ts1rKKLDxJ8ukbbFKiB9+uUNDS5JEmxTrnfU8NQGMC+Xw8zcLYgpexmG\nfXgUKJtidRGT6DU7LTGaL//Fx9FakXsuMhHEaJgb9k2OVMtysYwi2S4en2meNEcjVLTea+HpZySu\n9c53iv/6rrtFNuEPPyQI+uGvfx5HKAvqsXLLxoqMcU+ckxW6SxriRLWOuEc/u5dVs84ss8wy+5G2\nAad8BUvlkM9xpkGEEmehEhMauvQRjpAL6CBCsCP83IQR5S7zaScmJG0xIaI8cecBPPTVvwQABFoQ\ng0t02SOCqFaq8OgjMoIsbfpNLy4zOaARwVcy048dl3lnxqTcss7a9gYlHvsuSvRP9boxBi2ckEQh\nWpsr+MpffBYAsLByBRarfT/5JLmzvAdT5wwqwZc+8xU5P/3rd98jRPbAk2hy05fru3B5DZubwlkO\n+nJxSyuX5H4vyfevvec1+NX//tcBAI98+2E5184mjyMIrAeNC48KGv/GKUFYJUcQk/HL2fTBVkou\nDhyaAwC89+f/i8EaBLIicJSLkCuGds/HlhHEoa8tck2FcyZ29PpQ5HOH2rAk6C8k19pmEojtOKmg\nk1mVpL/x07IUjOvXiFBZ6f4mXT2GttRVvxl/cVqBhoUJkiSGeXzpcxzQNIBIa2yy0nq1mE+RsWF9\nmNVkh7K0lgXohKu9gvy2tiW/Pf6U+IZLBfHH+v0QhmftcaXzzDnZZqIo72el5GJyUv69OS9IT5G9\nsbYuxzlwYAQxVyk+EftuMpMR4ec1VcsI6PTuBMnAdS41NCLEaRVsL1cEQ01p26yuyXVtbMn7fWVl\nE5psFTMWGX63OX2O/auUc1O50UJeVoJ5VrxOKNp0eX1VKEMA3vdzwmK67777AAALC8LK+MSnPo3H\nnpB8gZhp8ttkCQWbrBkay7vbjdq4sC3vWpHyqoPYQIOypQDPsdDli27nS0gYiOtyILJJ38kZPWC3\nBI96FrWquDhMOe/ujCxHxmclI29xbQO3ve5NAID2uizNL5wVmlunLUstx+6hxpfUlKZZXpRtL8/T\nfZEroTohDT/G4IXiwK22WP5nm1mB48M4wMyc82dW4PfSeu3XZa7rYWpiCsfmZILRSOCQ5man6nCm\nmCU7Xr4EkK41PS1LtPvf9S4AQKVIl0JelmdnTj+Bs+epMT0zBwDoc0SyOSGePvsszpyVJVRx7iQA\nYGlJ9h+qy+e456FYlmeyxQKUm4uy1F3fkOfR5xsVJgrLDWmf+95xYypx7VY7rdDRaffQMWpwPFy1\nLs8lV9h1j5hEjgIpWi6XzGagNQpgtuOkiRO7+tfU7+aftmXturVio5cdXbVPGEWIuZ95cU1QKM1c\n44uccx1oox2SuwGXDgDHdjA0Mooq34O862CL2i4FPsuQ6oUBXSyOa8EzmstUblvbkn36TEsdZsbb\ngSOjaVmyJkt9XWKBWW+M7hkdoUxVNzUufaNakGfRZkHZS/OXcPS4BKgCDlYBqaomt8YM0geHqygw\nUOz3AmBA9YsoirHR2ETI4LVjWdC898eeFJfgHXe9hn9LEC6EhYBJI0FId9gyi8iSbmuAm2vvXpHL\nLETTj2KjudzvYZgJboa+2iKImJwS99LW9jq++EXJ0uvTHbm5yXJSnLgd9mVbKwxNiGt2fGJyoPYA\nMvdFZpllltm+soGQsuMoTIxZCFkGvRcnoAATNNMxDdKoUonLc130OjLrFEwQLZDPRx96CABw5IQg\ntStXVlJifzFn6pjJ7GMqV3TaPfSYcBAxAFLmDHXfPUJZyVeqiGxWRKGiXG9BZlCrJchnnPnr9xy/\nDeN1mSVPLV/c1TO+TouiCFvrW3jD62W5c9/b3oYctQkcExiyjGYCETTsFBH1Arm+zStCd9qinu7W\nhtDdLpx/Hktrsswsjwv1CaTkKKaNBpGPL33tmwCAQ0dFcW92WBB4ngHTopuD3xd0c6Epq4+yoRGR\n2rOyLTP/6OgcugzofOVrjwzUHqZNNjY303vs93cLcbpM/3aJrsyztGxrt8ikZXQWjC4EqXsmKFfM\n7ZZuN/rR1/idlFJp8M+Y0RUxyNlxndR9oa5xW+wicCPXB+RJ2btRpBwnCVrdLhJWjJmeGIdHhNyl\n66ZEaqNiQo+yNVyPwTUi4y6DSB4TocojEhgLrSgtapyvc4nOgFeLroBjRw4hWpHnHDGFfIc61cdu\nkQDwlYVzCIlWFYeINql6CXFcmSu6ctFDh8kwdrGSPrvrNa00YpVAMW273e2iR6qrSQz6v//dBwEA\n8yzw2g5inF+UFYBZfZpnGprAL8kDNqy0Hyi2myaFL+0dWqNQku03ObaZpJkmK9L4foRLl8SVodg2\nocnIpzvE9BjP9VDKyTPpdgbXScmQcmaZZZbZPrKBkLLnKRyc9VBTMkOfX+hilZSugJWky2UK6zBR\nJE7asDn2b3Hma7VZZSGUbWwtn5XyEFapynSlY0jqFM1h6XWVhNhuiMM/V5Jz1qkeZcj0fhADRAgd\nX74L2qTUMVX3FlKRpidHsHBFkPrmejdVb7tesyyFUjGHzaZc72NPnsI4fXUT4xJQMUpw29vi50O/\nD4doaeawoN/ZIbmHxbMShOu0ZeYen5hEkUI4NpNPuiTwT7Hy7srSFWxsShtOTZN+Z4JsFJuBk0t1\ninNcdeSICgNWNYYlbTQxM4eAyG3Q4t6ApHGHYQATjXMcFwZcptQlE0djD7RtO02SiPnMDfqxib5s\nBiQt14Jn9KL11Uhpb41F3m66UqmzjLx5Hn4QIKbf+VqEbPzPUWTEe0IYLBTfSIliyGqgWCoi5grP\nD8O0MoWh4Bn/ucFLlgs47tV90mffUfSDF1l9pdVqocD2XWclDceRfjVEffNivYpyXhDyBGsXbmh5\nn4pUlBsfH0l9qoFpQxMLoFJhpSrnae40sEGdY22VBw6COo7DxBm5l167A5+UOEuZhBd5b0Yo4lUb\nHkNkqmxrJsmE8r7EpkIRA39JqNPnZYTBjMwATEAZFhq832899C0AwNtZT/TpMxJMj2OkAU2b15rw\n+gw6j827FmgszEugz85VBmoPuZ7MMssss8z2jQ2ElG1HoTrkorcuPqShcRsoiT9lY5WpsPQdOh5T\nigMgCY2/R7bZ6cnMZH0DogAACgVJREFUXKIvuE85wF5/AwG3jQ3FRcusZHxa1WoB1arM8D1TzZoi\nKUaMRlkWFKk8HqO0dMPCI9qau2VOjtHV+PrXhTj+5Nk19PqDzfSWAnJuAr8vs/lDD/0lNGmB1aKh\n63BlQP+pAwuH5iQF+/Y3iGDJ0YOCmBuk4KxsC/rwCjkcHRFUv05NXFNr7rY7RATpP330I3CokRxy\nhREErKBh0l7zUUp5mzssyTJrC8/xJkgZ4srj5Mnj6LNyyyzTwAcxx3EwMjICiwkZcawRcgVikGmf\nAjeKtCSlrFT8JyDysJOr/ZMGRSY6To93rd/YsDuSRKcpv0l8NcPCoLkwihBSN9pQ4q5FzCnFDruI\nKxmUN0mzlEK+4MFS9KcHfloFpkCGhaJcgUcEDVuhypThfpMULIfvWC7hcfq8Vg8EjAh6cv3LfelH\nwzMSYwiX11Aw4mEVOcdYTZ7xxqYIew3Xqumqqc1K1SempH8m2ogWURSsE2KY6DmMANsajH2hoREj\nSdvUyXnIMVXexKeGhmTFaRShkkSnzytiTCahHnXMvpOk7JzdupftDqtk+0YmlftEcfrdZz4r1NbT\nZ2RMePSUVKtRlouYfS0yqzMibc2+mDD2EWGXzpnXg7G5gAwpZ5ZZZpntKxu48oiTd5Cvyqw+XLbg\nUBrQLchs0ST/F7ER4RlH7Bqfi6BJr0gOoZEMtGVm9HWSVucwkXdl3D9EA3FfKjjIgQTZNZha22NK\nca1ehWMSBniOLhHI6oYwELbp1251dvDlB0UqcLUL9IPBkLJUKO7CZCq86z0PIAlY140zdMLZW6fJ\nCx7yXGGsNCge3hCe8ZYR4yc/9rnHL2DzYfH5HjksyPh1jJIH9C0XvBy0kTPld5ZJgzVFjZMEDmfy\nQwcEKffb4uO/lbzZR06JFOjS/HPokVajma4+iNm2jWq1iiQ2zAULPp9Nkwjc+FJtIxwVx6ncpBFk\njxKDQIiQTMUPZaU1/q6tSmD8hUmcQONq1kvAOm7Gp5xA78oEmP0NwuI3RT4Hz7HTGosGwQ1qSil4\ntoUimQtxHMPmTZvq0zG5yIZZpG0LrRb9rfR7mn1MBZ4gNJXeI3R3KL7EFWKFErHmXQm7PdieSdRg\nCjlZUcZPnHNs1IeFZ6ub4ptWZFf1WTWo1+U1FIu7iTZap9VZrrtNoKCUnVb8UbZK60y6pky3GQN4\nnpy9WzbbMwXdwRqCbIuUjaN3UfXIqKw4zMpVs1/EcYKEAQjDJFlhQYo55h+0OiG6XOmaC/oexMxz\nWradxjEMm6zb3LzuNsmQcmaZZZbZPrKBpvwkUWi3XcCW6Gi51IdbkNmiRKdtrcYsmSbL9TRX0eas\nGvbls+IJkyLPmTCiP8dxLBj9DjdnfHyMGpPVYTm7vFWvQE40OZlbzHRq6QRVVrs1Yi7nLslM9exT\nEhWdYKbfxIEiwAy80VoFq63BfECWpVAqe6hxNq+MHU/9U3nOeR59iJqR8VzRQ8Jq3a0W0Q/5qeNH\nBdkcLYov8NzF5wGK+LgUB1pcFt/fCOU+R0aHEPRMaR/xO5oMOp/INPS7cMinnKAQ1PyyoIHVy5Ix\n2KewzPNPP46RESIliiANagpWWvggCH30fQq0cyVk0ItZ0eg4ScWnfMORvYZDbJCqZVlIGDO4hk1s\nEs6gldrN+mM6vmXE4O1d6X6dgjzDrDBRfbOBqS5upf+OwhtkXyiFkpeDw6u1sJsx2CY31/iwPfr/\nC6Xi7r/5bvR2ZMU5MS7smz6Rc72UhztmBJRk25CqUFFarq0Elxl9ptFCtuvomLzXXuLA5mrUVLHW\nmmL5qZQpj2HbKde81+tdxX65HtNQ0NpOxacEOctvZtWSImZn1+9v+oL5zmYfMTUOzWoojuNdqrlh\nT1DczLSJbe+uzgrMjpw5yDqW3KcXxCnCNtdlBKrMPZvvbdvew/ig2Nq8qdX38jbQoBwEwJV5wG/I\ng6qMRcgX6DKgpujwMMnmXAY0Gl1sb1JvggjeBHCSa+hMSOIUupsX0mgC9OgO0RHgkhIUdWVpFTPg\nF7MjNdrdlMqzxcnh0nlWqNhkEVOSuidrkzh5SIIgzR5wbuX6CxwCQJL00W2dBUi1c1UZq8yJP3fm\nEgAgz6Wkx4DI6PgQpkepYcvOMFKTSYSeDvQZDB0fr2JmWgbGZapSnT0rNJ25QJZWvu+jxXqH3a4M\ntCnpnYNyHPRg58RN8fRpCZwY2tv4uCTPzNwpAcTxsQmMjklwMZ8bvHAqtHRQQ0EKwyANPJpzmiW3\ncS0oqHRAynMQsvjCxdekRydJAsVAign0meWiZ+8GB/tMrTeBPfPimvNordOXpsuirKmSHwdLs08U\n+ClFK5+/seQRBcDVGpYBFbbzPddvXmzPAJYoQsJgZJ7b1CqGMibHzTOJKAliFMtM1/ZNwVzqMnOi\nK3pOmr7eYTJNnklEPbruen4AVzN5i+1s2dIefA3RZY3LRmM7bV/RKx4w0JdoBP04bXfb2h0g00GO\nY4ChAGpocT1hF7RZHGhd1vbUrKqSu8qd8sK0xzAI0n5ovusGxrVBt00U7rpp6GoyKoTGbWH0mve6\nt4yrahDL3BeZZZZZZvvIBqvRpxzE7ihCTypY+IkPK5Jldr4ms0d9TGbUIVNpupugsSVIsbHBgEWH\ngiARl0BMMkiiBH0GqsysY2hMLSqk9dp9uCSMVywhZieWoMIwlOPmShp5qq/VPdn2CASl3nGXIL8T\nd4pe6twtt+DeNwhiuLLUxree3xikSYBEIwn6sDi/OaGNKgObp779NQDAyqocU/Ga7r33NXjzG6UN\nd6gY9uR3pYZXh+juLPWVL1y6hB4RjQl+5qviWmhSzKa1vYFOU5C1wSkOZ/NaRWbq6cOHMTQilSbG\np5k4c4+kZA8z0OftVVpTpgbb4PO21hphGKauiiiK0iyUFEWkSBfpOQ1aNKnPIVGL2cegFgUNO628\n/MJUNp0ku5UmeNxrkbPruntU5ayrzpGiP6LiYq6YXmuKmAY0SykUPHc30SWJ0/uoUjM7XRrzHI3G\nNjSRco3urzKjW5orzp5vKsZoJKH0tQoTMIw3wThcOoEPN5Rz9hikjyxZJWzsSH9qbzZRr1NJriP9\nKl8wS3U59zaV6lrdbpqwUigUboguKP2agc4oTqvemHT2XVeEUTV0dyl0hnbJlVdkgoJmVQWdBttM\nmxp3mHGR2q6X/nYt7dHUTrSSKA04Ryapie9jcs1Kbq8L50aq1GRIObPMMstsH5kaxDGvlFoHMP/K\nXc6+sENa67Hr3fivSZsAA7RL1iYvbH9N2iVrkxe2639/Bo2WZpZZZpll9spZ5r7ILLPMMttHlg3K\nmWWWWWb7yLJBObPMMstsH1k2KGeWWWaZ7SPLBuXMMssss31k2aCcWWaZZbaPLBuUM8sss8z2kWWD\ncmaZZZbZPrJsUM4ss8wy20f2/wObcV05GGeeKQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 5 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JwzKmdcuCv1D"
      },
      "source": [
        "### 1) Basic CNN implementation\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TbEYo5WgjTtm"
      },
      "source": [
        "Consider a basic CNN model\n",
        "\n",
        "- It has 3 convolutional layers, followed by a linear layer.\n",
        "- Each convolutional layer has a kernel size of 3, a padding of 1.\n",
        "- ReLU activation is applied on every hidden layer.\n",
        "\n",
        "Implement this model in the following section. We will need to tune the hyperparameters and fill the results in the table."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OZKyE2GUfL-Z"
      },
      "source": [
        "#### a) Implement convolutional layers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4P_aYytExtq9"
      },
      "source": [
        "Implement the initialization function and the forward function of the CNN."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sDmCKUD1LBFk"
      },
      "source": [
        "import torch\n",
        "class CNN(nn.Module):\n",
        "  def __init__(self, channels = 256):\n",
        "    super(CNN, self).__init__()\n",
        "    # implement parameter definitions here\n",
        "    self.conv1 = nn.Conv2d(3, channels, kernel_size = 3, padding = 1,stride=1)\n",
        "    self.conv2 = nn.Conv2d(channels, channels, kernel_size = 3, padding = 1, stride=1)\n",
        "    self.conv3 = nn.Conv2d(channels, channels, kernel_size = 3, padding = 1, stride=1)\n",
        "    self.fc1 = nn.Linear(channels * 32 * 32 , len(set(train.targets)))\n",
        "  \n",
        "  def forward(self, images):\n",
        "    # implement the forward function here\n",
        "    images = F.relu(self.conv1(images))\n",
        "    images = F.relu(self.conv2(images))\n",
        "    images = F.relu(self.conv3(images))\n",
        "    images = images.view(-1, images.size(1) * images.size(2) * images.size(3))\n",
        "    images = self.fc1(images)\n",
        "    return images"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dU23nACgzn_P",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "53582cdd-dc22-4ea0-a0b2-70abd5f4b911"
      },
      "source": [
        "len(set(train.targets))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "10"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E_YaASPpgRiL"
      },
      "source": [
        "#### b) Tune hyperparameters"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ygMcDdpy6XWP"
      },
      "source": [
        "Train the CNN model on CIFAR-10 dataset. Tune the number of channels, optimizer, learning rate and the number of epochs for best validation accuracy."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w-EHKzozkRbD"
      },
      "source": [
        "Write down **validation accuracy** of your model under different hyperparameter settings. Note the validation set is automatically split by Skorch during `model.fit()`.\n",
        "\n",
        " We need more epochs for SGD than Adam.\n",
        "\n",
        "| #channel for each layer \\ optimizer | SGD   | Adam  |\n",
        "|-------------------------------------|-------|-------|\n",
        "| (128, 128, 128)                     |0.3990 |0.6533 |\n",
        "| (256, 256, 256)                     |0.4193 |0.6493 |\n",
        "| (512, 512, 512)                     |0.4482 |0.6538 |\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JUaguxFA5xOp",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "def634a6-90b9-4294-c050-31bfa1460047"
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "# Hyperparameters to search\n",
        "lr = [1e-6,1e-5, 1e-4, 1e-3]\n",
        "opt = [optim.SGD, optim.Adam]\n",
        "ch = [128, 256, 512]\n",
        "\n",
        "# Empty dataframe to store results\n",
        "results = pd.DataFrame()\n",
        "\n",
        "# Transform to the right type\n",
        "X = torch.Tensor(train.data/255)\n",
        "X = X.permute(0,3,1,2)\n",
        "\n",
        "for l in lr:\n",
        "  for o in opt:\n",
        "    for c in ch:\n",
        "      print(\"training for : {0} channels, {1} learning rate, {2} optimizer\".format(c, l, str(o)) )\n",
        "      \n",
        "      # Define number of channels\n",
        "      cnn = CNN(channels = c)\n",
        "      \n",
        "      # implement hyperparameters here\n",
        "      model = skorch.NeuralNetClassifier(cnn, criterion=torch.nn.CrossEntropyLoss,\n",
        "                                        device=\"cuda\",\n",
        "                                        max_epochs = 50, batch_size=64,\n",
        "                                        optimizer=o,\n",
        "                                        lr = l,\n",
        "                                        callbacks = [skorch.callbacks.EarlyStopping(lower_is_better=True)])\n",
        "      \n",
        "      # implement input normalization & type cast here\n",
        "      model.fit(X, np.array(train.targets))\n",
        "\n",
        "      # Store results for this hyperparameters\n",
        "      result = pd.DataFrame(pd.DataFrame(model.history).sort_values('valid_loss').loc[0, [\"epoch\", \"train_loss\", \"valid_loss\", \"valid_acc\"]]).T\n",
        "      result.assign(learning_rate = str(l), inplace = True)\n",
        "      result.assign(channels = str(c), inplace = True)\n",
        "      result.assign(optimizer = str(o), inplace = True)\n",
        "      \n",
        "      # concat to results table\n",
        "      results = pd.concat([results, result])\n",
        "results.to_csv(\"CNN_1b_results.csv\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "training for : 128 channels, 0.0001 learning rate, <class 'torch.optim.sgd.SGD'> optimizer\n",
            "  epoch    train_loss    valid_acc    valid_loss      dur\n",
            "-------  ------------  -----------  ------------  -------\n",
            "      1        \u001b[36m2.3001\u001b[0m       \u001b[32m0.1008\u001b[0m        \u001b[35m2.2972\u001b[0m  15.4241\n",
            "      2        \u001b[36m2.2946\u001b[0m       \u001b[32m0.1043\u001b[0m        \u001b[35m2.2914\u001b[0m  15.3893\n",
            "      3        \u001b[36m2.2882\u001b[0m       \u001b[32m0.1146\u001b[0m        \u001b[35m2.2841\u001b[0m  15.0981\n",
            "      4        \u001b[36m2.2800\u001b[0m       \u001b[32m0.1388\u001b[0m        \u001b[35m2.2745\u001b[0m  15.0585\n",
            "      5        \u001b[36m2.2690\u001b[0m       \u001b[32m0.1713\u001b[0m        \u001b[35m2.2615\u001b[0m  15.0326\n",
            "      6        \u001b[36m2.2539\u001b[0m       \u001b[32m0.1976\u001b[0m        \u001b[35m2.2435\u001b[0m  15.1531\n",
            "      7        \u001b[36m2.2334\u001b[0m       \u001b[32m0.2168\u001b[0m        \u001b[35m2.2196\u001b[0m  15.2558\n",
            "      8        \u001b[36m2.2068\u001b[0m       \u001b[32m0.2276\u001b[0m        \u001b[35m2.1896\u001b[0m  15.2023\n",
            "      9        \u001b[36m2.1749\u001b[0m       \u001b[32m0.2395\u001b[0m        \u001b[35m2.1555\u001b[0m  15.1070\n",
            "     10        \u001b[36m2.1405\u001b[0m       \u001b[32m0.2520\u001b[0m        \u001b[35m2.1206\u001b[0m  15.0899\n",
            "     11        \u001b[36m2.1068\u001b[0m       \u001b[32m0.2658\u001b[0m        \u001b[35m2.0877\u001b[0m  15.1055\n",
            "     12        \u001b[36m2.0752\u001b[0m       \u001b[32m0.2796\u001b[0m        \u001b[35m2.0570\u001b[0m  15.2047\n",
            "     13        \u001b[36m2.0453\u001b[0m       \u001b[32m0.2915\u001b[0m        \u001b[35m2.0277\u001b[0m  15.1804\n",
            "     14        \u001b[36m2.0169\u001b[0m       \u001b[32m0.3010\u001b[0m        \u001b[35m2.0001\u001b[0m  15.1790\n",
            "     15        \u001b[36m1.9907\u001b[0m       \u001b[32m0.3101\u001b[0m        \u001b[35m1.9750\u001b[0m  15.1664\n",
            "     16        \u001b[36m1.9677\u001b[0m       \u001b[32m0.3186\u001b[0m        \u001b[35m1.9535\u001b[0m  15.1888\n",
            "     17        \u001b[36m1.9482\u001b[0m       \u001b[32m0.3236\u001b[0m        \u001b[35m1.9354\u001b[0m  15.1523\n",
            "     18        \u001b[36m1.9321\u001b[0m       \u001b[32m0.3291\u001b[0m        \u001b[35m1.9204\u001b[0m  15.1271\n",
            "     19        \u001b[36m1.9187\u001b[0m       \u001b[32m0.3348\u001b[0m        \u001b[35m1.9078\u001b[0m  15.0861\n",
            "     20        \u001b[36m1.9074\u001b[0m       \u001b[32m0.3381\u001b[0m        \u001b[35m1.8970\u001b[0m  15.1480\n",
            "     21        \u001b[36m1.8976\u001b[0m       \u001b[32m0.3430\u001b[0m        \u001b[35m1.8876\u001b[0m  15.1055\n",
            "     22        \u001b[36m1.8888\u001b[0m       \u001b[32m0.3451\u001b[0m        \u001b[35m1.8791\u001b[0m  15.0996\n",
            "     23        \u001b[36m1.8809\u001b[0m       \u001b[32m0.3471\u001b[0m        \u001b[35m1.8713\u001b[0m  15.1107\n",
            "     24        \u001b[36m1.8736\u001b[0m       \u001b[32m0.3500\u001b[0m        \u001b[35m1.8641\u001b[0m  15.1711\n",
            "     25        \u001b[36m1.8668\u001b[0m       \u001b[32m0.3528\u001b[0m        \u001b[35m1.8573\u001b[0m  15.1414\n",
            "     26        \u001b[36m1.8604\u001b[0m       \u001b[32m0.3538\u001b[0m        \u001b[35m1.8509\u001b[0m  15.1306\n",
            "     27        \u001b[36m1.8542\u001b[0m       \u001b[32m0.3568\u001b[0m        \u001b[35m1.8447\u001b[0m  15.1935\n",
            "     28        \u001b[36m1.8483\u001b[0m       \u001b[32m0.3579\u001b[0m        \u001b[35m1.8388\u001b[0m  15.2021\n",
            "     29        \u001b[36m1.8426\u001b[0m       \u001b[32m0.3599\u001b[0m        \u001b[35m1.8332\u001b[0m  15.1972\n",
            "     30        \u001b[36m1.8371\u001b[0m       \u001b[32m0.3622\u001b[0m        \u001b[35m1.8277\u001b[0m  15.1737\n",
            "     31        \u001b[36m1.8318\u001b[0m       \u001b[32m0.3636\u001b[0m        \u001b[35m1.8225\u001b[0m  15.2643\n",
            "     32        \u001b[36m1.8267\u001b[0m       \u001b[32m0.3677\u001b[0m        \u001b[35m1.8174\u001b[0m  15.1768\n",
            "     33        \u001b[36m1.8217\u001b[0m       \u001b[32m0.3702\u001b[0m        \u001b[35m1.8125\u001b[0m  15.2195\n",
            "     34        \u001b[36m1.8168\u001b[0m       \u001b[32m0.3718\u001b[0m        \u001b[35m1.8076\u001b[0m  15.2102\n",
            "     35        \u001b[36m1.8119\u001b[0m       \u001b[32m0.3737\u001b[0m        \u001b[35m1.8029\u001b[0m  15.1990\n",
            "     36        \u001b[36m1.8071\u001b[0m       \u001b[32m0.3769\u001b[0m        \u001b[35m1.7981\u001b[0m  15.2040\n",
            "     37        \u001b[36m1.8024\u001b[0m       \u001b[32m0.3788\u001b[0m        \u001b[35m1.7935\u001b[0m  15.2283\n",
            "     38        \u001b[36m1.7976\u001b[0m       \u001b[32m0.3797\u001b[0m        \u001b[35m1.7888\u001b[0m  15.1941\n",
            "     39        \u001b[36m1.7929\u001b[0m       \u001b[32m0.3806\u001b[0m        \u001b[35m1.7842\u001b[0m  15.1891\n",
            "     40        \u001b[36m1.7882\u001b[0m       \u001b[32m0.3811\u001b[0m        \u001b[35m1.7795\u001b[0m  15.1592\n",
            "     41        \u001b[36m1.7834\u001b[0m       \u001b[32m0.3833\u001b[0m        \u001b[35m1.7749\u001b[0m  15.1406\n",
            "     42        \u001b[36m1.7787\u001b[0m       \u001b[32m0.3845\u001b[0m        \u001b[35m1.7702\u001b[0m  15.1331\n",
            "     43        \u001b[36m1.7739\u001b[0m       \u001b[32m0.3862\u001b[0m        \u001b[35m1.7655\u001b[0m  15.1854\n",
            "     44        \u001b[36m1.7690\u001b[0m       \u001b[32m0.3871\u001b[0m        \u001b[35m1.7608\u001b[0m  15.1811\n",
            "     45        \u001b[36m1.7642\u001b[0m       \u001b[32m0.3893\u001b[0m        \u001b[35m1.7561\u001b[0m  15.1542\n",
            "     46        \u001b[36m1.7593\u001b[0m       \u001b[32m0.3905\u001b[0m        \u001b[35m1.7513\u001b[0m  15.1297\n",
            "     47        \u001b[36m1.7544\u001b[0m       \u001b[32m0.3915\u001b[0m        \u001b[35m1.7465\u001b[0m  15.1692\n",
            "     48        \u001b[36m1.7495\u001b[0m       \u001b[32m0.3939\u001b[0m        \u001b[35m1.7417\u001b[0m  15.1939\n",
            "     49        \u001b[36m1.7445\u001b[0m       \u001b[32m0.3959\u001b[0m        \u001b[35m1.7369\u001b[0m  15.1685\n",
            "     50        \u001b[36m1.7396\u001b[0m       \u001b[32m0.3990\u001b[0m        \u001b[35m1.7321\u001b[0m  15.2324\n",
            "training for : 256 channels, 0.0001 learning rate, <class 'torch.optim.sgd.SGD'> optimizer\n",
            "  epoch    train_loss    valid_acc    valid_loss      dur\n",
            "-------  ------------  -----------  ------------  -------\n",
            "      1        \u001b[36m2.2939\u001b[0m       \u001b[32m0.1109\u001b[0m        \u001b[35m2.2844\u001b[0m  37.7099\n",
            "      2        \u001b[36m2.2750\u001b[0m       \u001b[32m0.1680\u001b[0m        \u001b[35m2.2629\u001b[0m  37.6306\n",
            "      3        \u001b[36m2.2500\u001b[0m       \u001b[32m0.2018\u001b[0m        \u001b[35m2.2327\u001b[0m  37.6743\n",
            "      4        \u001b[36m2.2155\u001b[0m       \u001b[32m0.2184\u001b[0m        \u001b[35m2.1931\u001b[0m  37.7305\n",
            "      5        \u001b[36m2.1741\u001b[0m       \u001b[32m0.2322\u001b[0m        \u001b[35m2.1501\u001b[0m  37.6633\n",
            "      6        \u001b[36m2.1335\u001b[0m       \u001b[32m0.2472\u001b[0m        \u001b[35m2.1120\u001b[0m  37.7704\n",
            "      7        \u001b[36m2.0988\u001b[0m       \u001b[32m0.2625\u001b[0m        \u001b[35m2.0801\u001b[0m  37.5900\n",
            "      8        \u001b[36m2.0685\u001b[0m       \u001b[32m0.2766\u001b[0m        \u001b[35m2.0508\u001b[0m  37.8284\n",
            "      9        \u001b[36m2.0393\u001b[0m       \u001b[32m0.2889\u001b[0m        \u001b[35m2.0217\u001b[0m  37.6271\n",
            "     10        \u001b[36m2.0104\u001b[0m       \u001b[32m0.3018\u001b[0m        \u001b[35m1.9930\u001b[0m  37.8035\n",
            "     11        \u001b[36m1.9827\u001b[0m       \u001b[32m0.3147\u001b[0m        \u001b[35m1.9663\u001b[0m  37.6617\n",
            "     12        \u001b[36m1.9580\u001b[0m       \u001b[32m0.3230\u001b[0m        \u001b[35m1.9432\u001b[0m  37.8003\n",
            "     13        \u001b[36m1.9374\u001b[0m       \u001b[32m0.3289\u001b[0m        \u001b[35m1.9243\u001b[0m  37.6853\n",
            "     14        \u001b[36m1.9208\u001b[0m       \u001b[32m0.3324\u001b[0m        \u001b[35m1.9092\u001b[0m  37.7681\n",
            "     15        \u001b[36m1.9075\u001b[0m       \u001b[32m0.3371\u001b[0m        \u001b[35m1.8968\u001b[0m  37.7850\n",
            "     16        \u001b[36m1.8964\u001b[0m       \u001b[32m0.3408\u001b[0m        \u001b[35m1.8863\u001b[0m  37.7506\n",
            "     17        \u001b[36m1.8869\u001b[0m       \u001b[32m0.3460\u001b[0m        \u001b[35m1.8771\u001b[0m  37.7707\n",
            "     18        \u001b[36m1.8783\u001b[0m       \u001b[32m0.3490\u001b[0m        \u001b[35m1.8688\u001b[0m  37.7659\n",
            "     19        \u001b[36m1.8705\u001b[0m       \u001b[32m0.3527\u001b[0m        \u001b[35m1.8611\u001b[0m  37.6450\n",
            "     20        \u001b[36m1.8632\u001b[0m       \u001b[32m0.3551\u001b[0m        \u001b[35m1.8539\u001b[0m  37.8596\n",
            "     21        \u001b[36m1.8562\u001b[0m       \u001b[32m0.3574\u001b[0m        \u001b[35m1.8470\u001b[0m  37.7827\n",
            "     22        \u001b[36m1.8495\u001b[0m       \u001b[32m0.3594\u001b[0m        \u001b[35m1.8404\u001b[0m  37.7393\n",
            "     23        \u001b[36m1.8431\u001b[0m       \u001b[32m0.3625\u001b[0m        \u001b[35m1.8339\u001b[0m  37.7540\n",
            "     24        \u001b[36m1.8367\u001b[0m       \u001b[32m0.3651\u001b[0m        \u001b[35m1.8276\u001b[0m  37.8070\n",
            "     25        \u001b[36m1.8306\u001b[0m       \u001b[32m0.3689\u001b[0m        \u001b[35m1.8215\u001b[0m  37.8394\n",
            "     26        \u001b[36m1.8245\u001b[0m       \u001b[32m0.3715\u001b[0m        \u001b[35m1.8155\u001b[0m  37.8827\n",
            "     27        \u001b[36m1.8185\u001b[0m       \u001b[32m0.3734\u001b[0m        \u001b[35m1.8096\u001b[0m  37.9061\n",
            "     28        \u001b[36m1.8126\u001b[0m       \u001b[32m0.3747\u001b[0m        \u001b[35m1.8037\u001b[0m  37.8600\n",
            "     29        \u001b[36m1.8067\u001b[0m       \u001b[32m0.3758\u001b[0m        \u001b[35m1.7979\u001b[0m  37.8590\n",
            "     30        \u001b[36m1.8008\u001b[0m       \u001b[32m0.3777\u001b[0m        \u001b[35m1.7921\u001b[0m  37.8848\n",
            "     31        \u001b[36m1.7950\u001b[0m       \u001b[32m0.3793\u001b[0m        \u001b[35m1.7862\u001b[0m  37.8963\n",
            "     32        \u001b[36m1.7891\u001b[0m       \u001b[32m0.3813\u001b[0m        \u001b[35m1.7804\u001b[0m  37.8199\n",
            "     33        \u001b[36m1.7831\u001b[0m       \u001b[32m0.3839\u001b[0m        \u001b[35m1.7745\u001b[0m  37.7920\n",
            "     34        \u001b[36m1.7772\u001b[0m       \u001b[32m0.3868\u001b[0m        \u001b[35m1.7685\u001b[0m  37.7822\n",
            "     35        \u001b[36m1.7712\u001b[0m       \u001b[32m0.3893\u001b[0m        \u001b[35m1.7625\u001b[0m  37.8100\n",
            "     36        \u001b[36m1.7651\u001b[0m       \u001b[32m0.3914\u001b[0m        \u001b[35m1.7566\u001b[0m  37.8647\n",
            "     37        \u001b[36m1.7591\u001b[0m       \u001b[32m0.3937\u001b[0m        \u001b[35m1.7506\u001b[0m  37.9175\n",
            "     38        \u001b[36m1.7531\u001b[0m       \u001b[32m0.3959\u001b[0m        \u001b[35m1.7446\u001b[0m  37.9371\n",
            "     39        \u001b[36m1.7471\u001b[0m       \u001b[32m0.3991\u001b[0m        \u001b[35m1.7387\u001b[0m  37.9015\n",
            "     40        \u001b[36m1.7411\u001b[0m       \u001b[32m0.4013\u001b[0m        \u001b[35m1.7328\u001b[0m  37.9034\n",
            "     41        \u001b[36m1.7353\u001b[0m       \u001b[32m0.4030\u001b[0m        \u001b[35m1.7270\u001b[0m  37.9109\n",
            "     42        \u001b[36m1.7295\u001b[0m       \u001b[32m0.4057\u001b[0m        \u001b[35m1.7214\u001b[0m  37.9253\n",
            "     43        \u001b[36m1.7238\u001b[0m       \u001b[32m0.4087\u001b[0m        \u001b[35m1.7158\u001b[0m  37.9387\n",
            "     44        \u001b[36m1.7183\u001b[0m       \u001b[32m0.4098\u001b[0m        \u001b[35m1.7104\u001b[0m  37.8680\n",
            "     45        \u001b[36m1.7129\u001b[0m       \u001b[32m0.4111\u001b[0m        \u001b[35m1.7052\u001b[0m  37.8379\n",
            "     46        \u001b[36m1.7077\u001b[0m       \u001b[32m0.4120\u001b[0m        \u001b[35m1.7001\u001b[0m  37.7865\n",
            "     47        \u001b[36m1.7026\u001b[0m       \u001b[32m0.4136\u001b[0m        \u001b[35m1.6952\u001b[0m  37.8861\n",
            "     48        \u001b[36m1.6977\u001b[0m       \u001b[32m0.4176\u001b[0m        \u001b[35m1.6905\u001b[0m  37.9428\n",
            "     49        \u001b[36m1.6928\u001b[0m       \u001b[32m0.4189\u001b[0m        \u001b[35m1.6859\u001b[0m  37.9599\n",
            "     50        \u001b[36m1.6881\u001b[0m       \u001b[32m0.4193\u001b[0m        \u001b[35m1.6814\u001b[0m  37.8356\n",
            "training for : 512 channels, 0.0001 learning rate, <class 'torch.optim.sgd.SGD'> optimizer\n",
            "  epoch    train_loss    valid_acc    valid_loss       dur\n",
            "-------  ------------  -----------  ------------  --------\n",
            "      1        \u001b[36m2.2810\u001b[0m       \u001b[32m0.1603\u001b[0m        \u001b[35m2.2581\u001b[0m  109.6618\n",
            "      2        \u001b[36m2.2323\u001b[0m       \u001b[32m0.2240\u001b[0m        \u001b[35m2.1987\u001b[0m  109.7115\n",
            "      3        \u001b[36m2.1636\u001b[0m       \u001b[32m0.2585\u001b[0m        \u001b[35m2.1212\u001b[0m  109.9599\n",
            "      4        \u001b[36m2.0871\u001b[0m       \u001b[32m0.2836\u001b[0m        \u001b[35m2.0475\u001b[0m  109.8832\n",
            "      5        \u001b[36m2.0219\u001b[0m       \u001b[32m0.3017\u001b[0m        \u001b[35m1.9903\u001b[0m  110.1601\n",
            "      6        \u001b[36m1.9735\u001b[0m       \u001b[32m0.3145\u001b[0m        \u001b[35m1.9493\u001b[0m  110.3293\n",
            "      7        \u001b[36m1.9393\u001b[0m       \u001b[32m0.3228\u001b[0m        \u001b[35m1.9202\u001b[0m  109.9912\n",
            "      8        \u001b[36m1.9146\u001b[0m       \u001b[32m0.3307\u001b[0m        \u001b[35m1.8985\u001b[0m  109.8355\n",
            "      9        \u001b[36m1.8955\u001b[0m       \u001b[32m0.3402\u001b[0m        \u001b[35m1.8812\u001b[0m  109.9748\n",
            "     10        \u001b[36m1.8799\u001b[0m       \u001b[32m0.3465\u001b[0m        \u001b[35m1.8667\u001b[0m  110.4222\n",
            "     11        \u001b[36m1.8665\u001b[0m       \u001b[32m0.3521\u001b[0m        \u001b[35m1.8540\u001b[0m  110.2467\n",
            "     12        \u001b[36m1.8546\u001b[0m       \u001b[32m0.3575\u001b[0m        \u001b[35m1.8426\u001b[0m  110.1328\n",
            "     13        \u001b[36m1.8438\u001b[0m       \u001b[32m0.3603\u001b[0m        \u001b[35m1.8322\u001b[0m  110.3447\n",
            "     14        \u001b[36m1.8338\u001b[0m       \u001b[32m0.3635\u001b[0m        \u001b[35m1.8225\u001b[0m  110.1058\n",
            "     15        \u001b[36m1.8245\u001b[0m       \u001b[32m0.3685\u001b[0m        \u001b[35m1.8133\u001b[0m  110.1682\n",
            "     16        \u001b[36m1.8155\u001b[0m       \u001b[32m0.3720\u001b[0m        \u001b[35m1.8045\u001b[0m  110.2053\n",
            "     17        \u001b[36m1.8068\u001b[0m       \u001b[32m0.3748\u001b[0m        \u001b[35m1.7960\u001b[0m  110.2266\n",
            "     18        \u001b[36m1.7983\u001b[0m       \u001b[32m0.3771\u001b[0m        \u001b[35m1.7876\u001b[0m  110.3165\n",
            "     19        \u001b[36m1.7899\u001b[0m       \u001b[32m0.3798\u001b[0m        \u001b[35m1.7794\u001b[0m  110.3171\n",
            "     20        \u001b[36m1.7816\u001b[0m       \u001b[32m0.3830\u001b[0m        \u001b[35m1.7712\u001b[0m  110.2716\n",
            "     21        \u001b[36m1.7733\u001b[0m       \u001b[32m0.3875\u001b[0m        \u001b[35m1.7630\u001b[0m  110.3013\n",
            "     22        \u001b[36m1.7650\u001b[0m       \u001b[32m0.3918\u001b[0m        \u001b[35m1.7549\u001b[0m  110.3286\n",
            "     23        \u001b[36m1.7568\u001b[0m       \u001b[32m0.3954\u001b[0m        \u001b[35m1.7468\u001b[0m  110.3487\n",
            "     24        \u001b[36m1.7486\u001b[0m       \u001b[32m0.3990\u001b[0m        \u001b[35m1.7387\u001b[0m  110.4210\n",
            "     25        \u001b[36m1.7405\u001b[0m       \u001b[32m0.4012\u001b[0m        \u001b[35m1.7308\u001b[0m  110.3963\n",
            "     26        \u001b[36m1.7325\u001b[0m       \u001b[32m0.4038\u001b[0m        \u001b[35m1.7230\u001b[0m  110.3282\n",
            "     27        \u001b[36m1.7247\u001b[0m       \u001b[32m0.4076\u001b[0m        \u001b[35m1.7154\u001b[0m  110.3701\n",
            "     28        \u001b[36m1.7171\u001b[0m       \u001b[32m0.4103\u001b[0m        \u001b[35m1.7080\u001b[0m  110.3695\n",
            "     29        \u001b[36m1.7098\u001b[0m       \u001b[32m0.4120\u001b[0m        \u001b[35m1.7009\u001b[0m  110.4490\n",
            "     30        \u001b[36m1.7027\u001b[0m       \u001b[32m0.4129\u001b[0m        \u001b[35m1.6942\u001b[0m  110.3808\n",
            "     31        \u001b[36m1.6960\u001b[0m       \u001b[32m0.4157\u001b[0m        \u001b[35m1.6877\u001b[0m  110.2915\n",
            "     32        \u001b[36m1.6895\u001b[0m       \u001b[32m0.4172\u001b[0m        \u001b[35m1.6815\u001b[0m  110.3676\n",
            "     33        \u001b[36m1.6832\u001b[0m       \u001b[32m0.4184\u001b[0m        \u001b[35m1.6756\u001b[0m  110.3925\n",
            "     34        \u001b[36m1.6771\u001b[0m       \u001b[32m0.4204\u001b[0m        \u001b[35m1.6699\u001b[0m  110.4754\n",
            "     35        \u001b[36m1.6713\u001b[0m       \u001b[32m0.4221\u001b[0m        \u001b[35m1.6645\u001b[0m  110.4739\n",
            "     36        \u001b[36m1.6656\u001b[0m       \u001b[32m0.4241\u001b[0m        \u001b[35m1.6593\u001b[0m  110.4317\n",
            "     37        \u001b[36m1.6601\u001b[0m       \u001b[32m0.4259\u001b[0m        \u001b[35m1.6542\u001b[0m  110.4361\n",
            "     38        \u001b[36m1.6547\u001b[0m       \u001b[32m0.4272\u001b[0m        \u001b[35m1.6493\u001b[0m  110.4516\n",
            "     39        \u001b[36m1.6495\u001b[0m       \u001b[32m0.4298\u001b[0m        \u001b[35m1.6446\u001b[0m  110.4535\n",
            "     40        \u001b[36m1.6444\u001b[0m       \u001b[32m0.4314\u001b[0m        \u001b[35m1.6400\u001b[0m  110.4641\n",
            "     41        \u001b[36m1.6393\u001b[0m       \u001b[32m0.4343\u001b[0m        \u001b[35m1.6355\u001b[0m  110.4667\n",
            "     42        \u001b[36m1.6344\u001b[0m       \u001b[32m0.4361\u001b[0m        \u001b[35m1.6311\u001b[0m  110.5059\n",
            "     43        \u001b[36m1.6295\u001b[0m       \u001b[32m0.4388\u001b[0m        \u001b[35m1.6267\u001b[0m  110.5408\n",
            "     44        \u001b[36m1.6247\u001b[0m       \u001b[32m0.4399\u001b[0m        \u001b[35m1.6225\u001b[0m  110.4947\n",
            "     45        \u001b[36m1.6199\u001b[0m       \u001b[32m0.4413\u001b[0m        \u001b[35m1.6183\u001b[0m  110.5102\n",
            "     46        \u001b[36m1.6152\u001b[0m       \u001b[32m0.4415\u001b[0m        \u001b[35m1.6142\u001b[0m  110.4671\n",
            "     47        \u001b[36m1.6106\u001b[0m       \u001b[32m0.4426\u001b[0m        \u001b[35m1.6101\u001b[0m  110.4532\n",
            "     48        \u001b[36m1.6060\u001b[0m       \u001b[32m0.4441\u001b[0m        \u001b[35m1.6061\u001b[0m  110.4496\n",
            "     49        \u001b[36m1.6014\u001b[0m       \u001b[32m0.4461\u001b[0m        \u001b[35m1.6021\u001b[0m  110.4864\n",
            "     50        \u001b[36m1.5969\u001b[0m       \u001b[32m0.4482\u001b[0m        \u001b[35m1.5981\u001b[0m  110.4409\n",
            "training for : 128 channels, 0.0001 learning rate, <class 'torch.optim.adam.Adam'> optimizer\n",
            "  epoch    train_loss    valid_acc    valid_loss      dur\n",
            "-------  ------------  -----------  ------------  -------\n",
            "      1        \u001b[36m1.5966\u001b[0m       \u001b[32m0.5395\u001b[0m        \u001b[35m1.3118\u001b[0m  15.5354\n",
            "      2        \u001b[36m1.2406\u001b[0m       \u001b[32m0.5877\u001b[0m        \u001b[35m1.1688\u001b[0m  15.5381\n",
            "      3        \u001b[36m1.0886\u001b[0m       \u001b[32m0.6143\u001b[0m        \u001b[35m1.1107\u001b[0m  15.5554\n",
            "      4        \u001b[36m0.9545\u001b[0m       \u001b[32m0.6279\u001b[0m        \u001b[35m1.0756\u001b[0m  15.6295\n",
            "      5        \u001b[36m0.8395\u001b[0m       \u001b[32m0.6416\u001b[0m        \u001b[35m1.0519\u001b[0m  15.5810\n",
            "      6        \u001b[36m0.7387\u001b[0m       \u001b[32m0.6520\u001b[0m        \u001b[35m1.0427\u001b[0m  15.5885\n",
            "      7        \u001b[36m0.6479\u001b[0m       \u001b[32m0.6533\u001b[0m        1.0661  15.5904\n",
            "      8        \u001b[36m0.5630\u001b[0m       0.6515        1.1125  15.6275\n",
            "      9        \u001b[36m0.4838\u001b[0m       0.6500        1.1857  15.6003\n",
            "     10        \u001b[36m0.4082\u001b[0m       0.6460        1.2828  15.6274\n",
            "Stopping since valid_loss has not improved in the last 5 epochs.\n",
            "training for : 256 channels, 0.0001 learning rate, <class 'torch.optim.adam.Adam'> optimizer\n",
            "  epoch    train_loss    valid_acc    valid_loss      dur\n",
            "-------  ------------  -----------  ------------  -------\n",
            "      1        \u001b[36m1.5486\u001b[0m       \u001b[32m0.5620\u001b[0m        \u001b[35m1.2495\u001b[0m  37.8505\n",
            "      2        \u001b[36m1.1692\u001b[0m       \u001b[32m0.6145\u001b[0m        \u001b[35m1.1022\u001b[0m  37.9456\n",
            "      3        \u001b[36m0.9647\u001b[0m       \u001b[32m0.6310\u001b[0m        \u001b[35m1.0830\u001b[0m  37.9306\n",
            "      4        \u001b[36m0.8018\u001b[0m       \u001b[32m0.6473\u001b[0m        \u001b[35m1.0712\u001b[0m  37.9702\n",
            "      5        \u001b[36m0.6585\u001b[0m       \u001b[32m0.6493\u001b[0m        1.1148  37.9831\n",
            "      6        \u001b[36m0.5230\u001b[0m       0.6436        1.2308  38.0031\n",
            "      7        \u001b[36m0.4002\u001b[0m       0.6416        1.3512  38.0444\n",
            "      8        \u001b[36m0.3048\u001b[0m       0.6233        1.6042  38.0395\n",
            "Stopping since valid_loss has not improved in the last 5 epochs.\n",
            "training for : 512 channels, 0.0001 learning rate, <class 'torch.optim.adam.Adam'> optimizer\n",
            "  epoch    train_loss    valid_acc    valid_loss       dur\n",
            "-------  ------------  -----------  ------------  --------\n",
            "      1        \u001b[36m1.5393\u001b[0m       \u001b[32m0.5619\u001b[0m        \u001b[35m1.2471\u001b[0m  107.0762\n",
            "      2        \u001b[36m1.1418\u001b[0m       \u001b[32m0.6221\u001b[0m        \u001b[35m1.0731\u001b[0m  107.0671\n",
            "      3        \u001b[36m0.9113\u001b[0m       \u001b[32m0.6393\u001b[0m        \u001b[35m1.0591\u001b[0m  107.0786\n",
            "      4        \u001b[36m0.7236\u001b[0m       \u001b[32m0.6538\u001b[0m        1.0721  107.1170\n",
            "      5        \u001b[36m0.5475\u001b[0m       0.6514        1.1754  107.4268\n",
            "      6        \u001b[36m0.4034\u001b[0m       0.6496        1.3261  107.4634\n",
            "      7        \u001b[36m0.2891\u001b[0m       0.6431        1.4896  107.3924\n",
            "Stopping since valid_loss has not improved in the last 5 epochs.\n",
            "training for : 128 channels, 0.001 learning rate, <class 'torch.optim.sgd.SGD'> optimizer\n",
            "  epoch    train_loss    valid_acc    valid_loss      dur\n",
            "-------  ------------  -----------  ------------  -------\n",
            "      1        \u001b[36m2.2253\u001b[0m       \u001b[32m0.2631\u001b[0m        \u001b[35m2.0878\u001b[0m  15.3117\n",
            "      2        \u001b[36m1.9779\u001b[0m       \u001b[32m0.3266\u001b[0m        \u001b[35m1.9087\u001b[0m  15.3597\n",
            "      3        \u001b[36m1.8786\u001b[0m       \u001b[32m0.3505\u001b[0m        \u001b[35m1.8443\u001b[0m  15.3756\n",
            "      4        \u001b[36m1.8269\u001b[0m       \u001b[32m0.3691\u001b[0m        \u001b[35m1.7989\u001b[0m  15.3336\n",
            "      5        \u001b[36m1.7841\u001b[0m       \u001b[32m0.3831\u001b[0m        \u001b[35m1.7566\u001b[0m  15.3467\n",
            "      6        \u001b[36m1.7421\u001b[0m       \u001b[32m0.4021\u001b[0m        \u001b[35m1.7132\u001b[0m  15.3492\n",
            "      7        \u001b[36m1.7029\u001b[0m       \u001b[32m0.4145\u001b[0m        \u001b[35m1.6758\u001b[0m  15.3348\n",
            "      8        \u001b[36m1.6699\u001b[0m       \u001b[32m0.4249\u001b[0m        \u001b[35m1.6465\u001b[0m  15.3530\n",
            "      9        \u001b[36m1.6416\u001b[0m       \u001b[32m0.4346\u001b[0m        \u001b[35m1.6214\u001b[0m  15.3485\n",
            "     10        \u001b[36m1.6159\u001b[0m       \u001b[32m0.4425\u001b[0m        \u001b[35m1.5983\u001b[0m  15.3491\n",
            "     11        \u001b[36m1.5918\u001b[0m       \u001b[32m0.4489\u001b[0m        \u001b[35m1.5763\u001b[0m  15.3335\n",
            "     12        \u001b[36m1.5685\u001b[0m       \u001b[32m0.4581\u001b[0m        \u001b[35m1.5546\u001b[0m  15.3393\n",
            "     13        \u001b[36m1.5456\u001b[0m       \u001b[32m0.4638\u001b[0m        \u001b[35m1.5330\u001b[0m  15.3319\n",
            "     14        \u001b[36m1.5229\u001b[0m       \u001b[32m0.4704\u001b[0m        \u001b[35m1.5120\u001b[0m  15.3763\n",
            "     15        \u001b[36m1.5010\u001b[0m       \u001b[32m0.4746\u001b[0m        \u001b[35m1.4921\u001b[0m  15.3800\n",
            "     16        \u001b[36m1.4804\u001b[0m       \u001b[32m0.4826\u001b[0m        \u001b[35m1.4741\u001b[0m  15.4116\n",
            "     17        \u001b[36m1.4614\u001b[0m       \u001b[32m0.4864\u001b[0m        \u001b[35m1.4582\u001b[0m  15.3411\n",
            "     18        \u001b[36m1.4441\u001b[0m       \u001b[32m0.4918\u001b[0m        \u001b[35m1.4441\u001b[0m  15.3578\n",
            "     19        \u001b[36m1.4283\u001b[0m       \u001b[32m0.4959\u001b[0m        \u001b[35m1.4316\u001b[0m  15.3451\n",
            "     20        \u001b[36m1.4138\u001b[0m       \u001b[32m0.4992\u001b[0m        \u001b[35m1.4201\u001b[0m  15.3267\n",
            "     21        \u001b[36m1.4001\u001b[0m       \u001b[32m0.5029\u001b[0m        \u001b[35m1.4094\u001b[0m  15.3312\n",
            "     22        \u001b[36m1.3869\u001b[0m       \u001b[32m0.5040\u001b[0m        \u001b[35m1.3990\u001b[0m  15.3229\n",
            "     23        \u001b[36m1.3740\u001b[0m       \u001b[32m0.5081\u001b[0m        \u001b[35m1.3887\u001b[0m  15.3577\n",
            "     24        \u001b[36m1.3611\u001b[0m       \u001b[32m0.5104\u001b[0m        \u001b[35m1.3783\u001b[0m  15.3077\n",
            "     25        \u001b[36m1.3480\u001b[0m       \u001b[32m0.5149\u001b[0m        \u001b[35m1.3677\u001b[0m  15.3143\n",
            "     26        \u001b[36m1.3346\u001b[0m       \u001b[32m0.5173\u001b[0m        \u001b[35m1.3567\u001b[0m  15.2964\n",
            "     27        \u001b[36m1.3209\u001b[0m       \u001b[32m0.5223\u001b[0m        \u001b[35m1.3454\u001b[0m  15.3502\n",
            "     28        \u001b[36m1.3069\u001b[0m       \u001b[32m0.5261\u001b[0m        \u001b[35m1.3340\u001b[0m  15.3317\n",
            "     29        \u001b[36m1.2929\u001b[0m       \u001b[32m0.5303\u001b[0m        \u001b[35m1.3224\u001b[0m  15.3685\n",
            "     30        \u001b[36m1.2790\u001b[0m       \u001b[32m0.5339\u001b[0m        \u001b[35m1.3111\u001b[0m  15.3390\n",
            "     31        \u001b[36m1.2656\u001b[0m       \u001b[32m0.5381\u001b[0m        \u001b[35m1.3000\u001b[0m  15.3526\n",
            "     32        \u001b[36m1.2527\u001b[0m       \u001b[32m0.5427\u001b[0m        \u001b[35m1.2893\u001b[0m  15.2978\n",
            "     33        \u001b[36m1.2404\u001b[0m       \u001b[32m0.5449\u001b[0m        \u001b[35m1.2789\u001b[0m  15.3284\n",
            "     34        \u001b[36m1.2288\u001b[0m       \u001b[32m0.5487\u001b[0m        \u001b[35m1.2691\u001b[0m  15.3116\n",
            "     35        \u001b[36m1.2177\u001b[0m       \u001b[32m0.5523\u001b[0m        \u001b[35m1.2596\u001b[0m  15.3504\n",
            "     36        \u001b[36m1.2071\u001b[0m       \u001b[32m0.5545\u001b[0m        \u001b[35m1.2511\u001b[0m  15.3220\n",
            "     37        \u001b[36m1.1969\u001b[0m       \u001b[32m0.5566\u001b[0m        \u001b[35m1.2430\u001b[0m  15.3259\n",
            "     38        \u001b[36m1.1871\u001b[0m       \u001b[32m0.5592\u001b[0m        \u001b[35m1.2357\u001b[0m  15.3439\n",
            "     39        \u001b[36m1.1776\u001b[0m       \u001b[32m0.5618\u001b[0m        \u001b[35m1.2288\u001b[0m  15.3443\n",
            "     40        \u001b[36m1.1683\u001b[0m       \u001b[32m0.5639\u001b[0m        \u001b[35m1.2223\u001b[0m  15.3332\n",
            "     41        \u001b[36m1.1592\u001b[0m       \u001b[32m0.5664\u001b[0m        \u001b[35m1.2162\u001b[0m  15.3148\n",
            "     42        \u001b[36m1.1503\u001b[0m       \u001b[32m0.5689\u001b[0m        \u001b[35m1.2102\u001b[0m  15.3562\n",
            "     43        \u001b[36m1.1414\u001b[0m       \u001b[32m0.5720\u001b[0m        \u001b[35m1.2044\u001b[0m  15.2854\n",
            "     44        \u001b[36m1.1326\u001b[0m       \u001b[32m0.5740\u001b[0m        \u001b[35m1.1990\u001b[0m  15.3432\n",
            "     45        \u001b[36m1.1238\u001b[0m       \u001b[32m0.5749\u001b[0m        \u001b[35m1.1938\u001b[0m  15.3076\n",
            "     46        \u001b[36m1.1151\u001b[0m       \u001b[32m0.5771\u001b[0m        \u001b[35m1.1886\u001b[0m  15.3467\n",
            "     47        \u001b[36m1.1063\u001b[0m       \u001b[32m0.5774\u001b[0m        \u001b[35m1.1836\u001b[0m  15.3087\n",
            "     48        \u001b[36m1.0975\u001b[0m       \u001b[32m0.5786\u001b[0m        \u001b[35m1.1785\u001b[0m  15.3070\n",
            "     49        \u001b[36m1.0887\u001b[0m       \u001b[32m0.5795\u001b[0m        \u001b[35m1.1737\u001b[0m  15.3104\n",
            "     50        \u001b[36m1.0798\u001b[0m       \u001b[32m0.5820\u001b[0m        \u001b[35m1.1689\u001b[0m  15.3532\n",
            "training for : 256 channels, 0.001 learning rate, <class 'torch.optim.sgd.SGD'> optimizer\n",
            "  epoch    train_loss    valid_acc    valid_loss      dur\n",
            "-------  ------------  -----------  ------------  -------\n",
            "      1        \u001b[36m2.1475\u001b[0m       \u001b[32m0.2985\u001b[0m        \u001b[35m1.9777\u001b[0m  37.7090\n",
            "      2        \u001b[36m1.9131\u001b[0m       \u001b[32m0.3487\u001b[0m        \u001b[35m1.8591\u001b[0m  37.8887\n",
            "      3        \u001b[36m1.8275\u001b[0m       \u001b[32m0.3752\u001b[0m        \u001b[35m1.7846\u001b[0m  37.8102\n",
            "      4        \u001b[36m1.7606\u001b[0m       \u001b[32m0.3999\u001b[0m        \u001b[35m1.7189\u001b[0m  37.7864\n",
            "      5        \u001b[36m1.7046\u001b[0m       \u001b[32m0.4205\u001b[0m        \u001b[35m1.6694\u001b[0m  37.8386\n",
            "      6        \u001b[36m1.6637\u001b[0m       \u001b[32m0.4300\u001b[0m        \u001b[35m1.6355\u001b[0m  37.8885\n",
            "      7        \u001b[36m1.6310\u001b[0m       \u001b[32m0.4413\u001b[0m        \u001b[35m1.6076\u001b[0m  37.8981\n",
            "      8        \u001b[36m1.6020\u001b[0m       \u001b[32m0.4468\u001b[0m        \u001b[35m1.5823\u001b[0m  37.9393\n",
            "      9        \u001b[36m1.5751\u001b[0m       \u001b[32m0.4562\u001b[0m        \u001b[35m1.5583\u001b[0m  37.9011\n",
            "     10        \u001b[36m1.5494\u001b[0m       \u001b[32m0.4626\u001b[0m        \u001b[35m1.5352\u001b[0m  37.9884\n",
            "     11        \u001b[36m1.5245\u001b[0m       \u001b[32m0.4690\u001b[0m        \u001b[35m1.5128\u001b[0m  37.9559\n",
            "     12        \u001b[36m1.5004\u001b[0m       \u001b[32m0.4759\u001b[0m        \u001b[35m1.4914\u001b[0m  37.9611\n",
            "     13        \u001b[36m1.4772\u001b[0m       \u001b[32m0.4834\u001b[0m        \u001b[35m1.4715\u001b[0m  37.9878\n",
            "     14        \u001b[36m1.4552\u001b[0m       \u001b[32m0.4904\u001b[0m        \u001b[35m1.4535\u001b[0m  37.9932\n",
            "     15        \u001b[36m1.4347\u001b[0m       \u001b[32m0.4952\u001b[0m        \u001b[35m1.4372\u001b[0m  37.9635\n",
            "     16        \u001b[36m1.4156\u001b[0m       \u001b[32m0.4997\u001b[0m        \u001b[35m1.4224\u001b[0m  37.8645\n",
            "     17        \u001b[36m1.3976\u001b[0m       \u001b[32m0.5043\u001b[0m        \u001b[35m1.4088\u001b[0m  37.9347\n",
            "     18        \u001b[36m1.3803\u001b[0m       \u001b[32m0.5082\u001b[0m        \u001b[35m1.3957\u001b[0m  38.0075\n",
            "     19        \u001b[36m1.3633\u001b[0m       \u001b[32m0.5121\u001b[0m        \u001b[35m1.3825\u001b[0m  38.0016\n",
            "     20        \u001b[36m1.3462\u001b[0m       \u001b[32m0.5163\u001b[0m        \u001b[35m1.3690\u001b[0m  38.0269\n",
            "     21        \u001b[36m1.3287\u001b[0m       \u001b[32m0.5212\u001b[0m        \u001b[35m1.3551\u001b[0m  38.0114\n",
            "     22        \u001b[36m1.3109\u001b[0m       \u001b[32m0.5242\u001b[0m        \u001b[35m1.3408\u001b[0m  38.0105\n",
            "     23        \u001b[36m1.2931\u001b[0m       \u001b[32m0.5287\u001b[0m        \u001b[35m1.3266\u001b[0m  38.0065\n",
            "     24        \u001b[36m1.2757\u001b[0m       \u001b[32m0.5332\u001b[0m        \u001b[35m1.3128\u001b[0m  37.9939\n",
            "     25        \u001b[36m1.2591\u001b[0m       \u001b[32m0.5365\u001b[0m        \u001b[35m1.2999\u001b[0m  37.9813\n",
            "     26        \u001b[36m1.2436\u001b[0m       \u001b[32m0.5413\u001b[0m        \u001b[35m1.2875\u001b[0m  37.9483\n",
            "     27        \u001b[36m1.2291\u001b[0m       \u001b[32m0.5467\u001b[0m        \u001b[35m1.2761\u001b[0m  37.9319\n",
            "     28        \u001b[36m1.2156\u001b[0m       \u001b[32m0.5496\u001b[0m        \u001b[35m1.2659\u001b[0m  37.9672\n",
            "     29        \u001b[36m1.2029\u001b[0m       \u001b[32m0.5530\u001b[0m        \u001b[35m1.2565\u001b[0m  37.9740\n",
            "     30        \u001b[36m1.1909\u001b[0m       \u001b[32m0.5561\u001b[0m        \u001b[35m1.2479\u001b[0m  37.9989\n",
            "     31        \u001b[36m1.1794\u001b[0m       \u001b[32m0.5604\u001b[0m        \u001b[35m1.2401\u001b[0m  38.0002\n",
            "     32        \u001b[36m1.1683\u001b[0m       \u001b[32m0.5625\u001b[0m        \u001b[35m1.2330\u001b[0m  38.0070\n",
            "     33        \u001b[36m1.1574\u001b[0m       \u001b[32m0.5649\u001b[0m        \u001b[35m1.2262\u001b[0m  38.0161\n",
            "     34        \u001b[36m1.1468\u001b[0m       \u001b[32m0.5667\u001b[0m        \u001b[35m1.2200\u001b[0m  38.0727\n",
            "     35        \u001b[36m1.1363\u001b[0m       \u001b[32m0.5713\u001b[0m        \u001b[35m1.2140\u001b[0m  38.0517\n",
            "     36        \u001b[36m1.1259\u001b[0m       \u001b[32m0.5740\u001b[0m        \u001b[35m1.2081\u001b[0m  37.9876\n",
            "     37        \u001b[36m1.1155\u001b[0m       \u001b[32m0.5760\u001b[0m        \u001b[35m1.2027\u001b[0m  37.9986\n",
            "     38        \u001b[36m1.1051\u001b[0m       \u001b[32m0.5779\u001b[0m        \u001b[35m1.1973\u001b[0m  38.0045\n",
            "     39        \u001b[36m1.0948\u001b[0m       \u001b[32m0.5795\u001b[0m        \u001b[35m1.1922\u001b[0m  37.9478\n",
            "     40        \u001b[36m1.0845\u001b[0m       \u001b[32m0.5806\u001b[0m        \u001b[35m1.1872\u001b[0m  37.9538\n",
            "     41        \u001b[36m1.0743\u001b[0m       \u001b[32m0.5824\u001b[0m        \u001b[35m1.1825\u001b[0m  37.9003\n",
            "     42        \u001b[36m1.0641\u001b[0m       \u001b[32m0.5839\u001b[0m        \u001b[35m1.1780\u001b[0m  37.9774\n",
            "     43        \u001b[36m1.0540\u001b[0m       \u001b[32m0.5855\u001b[0m        \u001b[35m1.1737\u001b[0m  38.0165\n",
            "     44        \u001b[36m1.0440\u001b[0m       \u001b[32m0.5876\u001b[0m        \u001b[35m1.1696\u001b[0m  38.0606\n",
            "     45        \u001b[36m1.0340\u001b[0m       \u001b[32m0.5885\u001b[0m        \u001b[35m1.1657\u001b[0m  38.0022\n",
            "     46        \u001b[36m1.0242\u001b[0m       \u001b[32m0.5908\u001b[0m        \u001b[35m1.1621\u001b[0m  37.9891\n",
            "     47        \u001b[36m1.0144\u001b[0m       \u001b[32m0.5930\u001b[0m        \u001b[35m1.1587\u001b[0m  37.9961\n",
            "     48        \u001b[36m1.0048\u001b[0m       \u001b[32m0.5943\u001b[0m        \u001b[35m1.1555\u001b[0m  38.0089\n",
            "     49        \u001b[36m0.9953\u001b[0m       \u001b[32m0.5956\u001b[0m        \u001b[35m1.1526\u001b[0m  38.0295\n",
            "     50        \u001b[36m0.9859\u001b[0m       \u001b[32m0.5965\u001b[0m        \u001b[35m1.1499\u001b[0m  38.0106\n",
            "training for : 512 channels, 0.001 learning rate, <class 'torch.optim.sgd.SGD'> optimizer\n",
            "  epoch    train_loss    valid_acc    valid_loss       dur\n",
            "-------  ------------  -----------  ------------  --------\n",
            "      1        \u001b[36m2.0573\u001b[0m       \u001b[32m0.3382\u001b[0m        \u001b[35m1.8932\u001b[0m  110.0193\n",
            "      3        \u001b[36m1.7471\u001b[0m       \u001b[32m0.4093\u001b[0m        \u001b[35m1.6988\u001b[0m  110.4477\n",
            "      4        \u001b[36m1.6824\u001b[0m       \u001b[32m0.4248\u001b[0m        \u001b[35m1.6474\u001b[0m  110.4264\n",
            "      5        \u001b[36m1.6340\u001b[0m       \u001b[32m0.4377\u001b[0m        \u001b[35m1.6064\u001b[0m  110.4202\n",
            "      6        \u001b[36m1.5929\u001b[0m       \u001b[32m0.4514\u001b[0m        \u001b[35m1.5691\u001b[0m  110.4495\n",
            "      7        \u001b[36m1.5551\u001b[0m       \u001b[32m0.4618\u001b[0m        \u001b[35m1.5329\u001b[0m  110.6337\n",
            "      8        \u001b[36m1.5184\u001b[0m       \u001b[32m0.4727\u001b[0m        \u001b[35m1.4976\u001b[0m  110.5261\n",
            "      9        \u001b[36m1.4826\u001b[0m       \u001b[32m0.4830\u001b[0m        \u001b[35m1.4647\u001b[0m  110.4747\n",
            "     10        \u001b[36m1.4487\u001b[0m       \u001b[32m0.4930\u001b[0m        \u001b[35m1.4361\u001b[0m  110.7655\n",
            "     11        \u001b[36m1.4174\u001b[0m       \u001b[32m0.5030\u001b[0m        \u001b[35m1.4120\u001b[0m  110.4649\n",
            "     12        \u001b[36m1.3884\u001b[0m       \u001b[32m0.5081\u001b[0m        \u001b[35m1.3909\u001b[0m  110.4448\n",
            "     13        \u001b[36m1.3609\u001b[0m       \u001b[32m0.5124\u001b[0m        \u001b[35m1.3712\u001b[0m  110.4960\n",
            "     14        \u001b[36m1.3344\u001b[0m       \u001b[32m0.5187\u001b[0m        \u001b[35m1.3517\u001b[0m  110.8759\n",
            "     15        \u001b[36m1.3094\u001b[0m       \u001b[32m0.5254\u001b[0m        \u001b[35m1.3329\u001b[0m  110.7514\n",
            "     16        \u001b[36m1.2861\u001b[0m       \u001b[32m0.5303\u001b[0m        \u001b[35m1.3152\u001b[0m  110.4941\n",
            "     17        \u001b[36m1.2648\u001b[0m       \u001b[32m0.5362\u001b[0m        \u001b[35m1.2988\u001b[0m  110.4972\n",
            "     18        \u001b[36m1.2454\u001b[0m       \u001b[32m0.5425\u001b[0m        \u001b[35m1.2838\u001b[0m  110.5009\n",
            "     19        \u001b[36m1.2276\u001b[0m       \u001b[32m0.5484\u001b[0m        \u001b[35m1.2699\u001b[0m  110.4954\n",
            "     20        \u001b[36m1.2111\u001b[0m       \u001b[32m0.5528\u001b[0m        \u001b[35m1.2577\u001b[0m  110.9214\n",
            "     21        \u001b[36m1.1955\u001b[0m       \u001b[32m0.5565\u001b[0m        \u001b[35m1.2467\u001b[0m  110.5036\n",
            "     22        \u001b[36m1.1806\u001b[0m       \u001b[32m0.5584\u001b[0m        \u001b[35m1.2366\u001b[0m  110.5295\n",
            "     23        \u001b[36m1.1662\u001b[0m       \u001b[32m0.5624\u001b[0m        \u001b[35m1.2270\u001b[0m  110.7380\n",
            "     24        \u001b[36m1.1522\u001b[0m       \u001b[32m0.5652\u001b[0m        \u001b[35m1.2183\u001b[0m  110.7440\n",
            "     25        \u001b[36m1.1384\u001b[0m       \u001b[32m0.5694\u001b[0m        \u001b[35m1.2100\u001b[0m  110.9048\n",
            "     26        \u001b[36m1.1248\u001b[0m       \u001b[32m0.5734\u001b[0m        \u001b[35m1.2021\u001b[0m  111.2386\n",
            "     27        \u001b[36m1.1114\u001b[0m       \u001b[32m0.5752\u001b[0m        \u001b[35m1.1946\u001b[0m  110.9912\n",
            "     28        \u001b[36m1.0981\u001b[0m       \u001b[32m0.5782\u001b[0m        \u001b[35m1.1875\u001b[0m  111.0640\n",
            "     29        \u001b[36m1.0850\u001b[0m       \u001b[32m0.5805\u001b[0m        \u001b[35m1.1809\u001b[0m  110.8392\n",
            "     30        \u001b[36m1.0720\u001b[0m       \u001b[32m0.5824\u001b[0m        \u001b[35m1.1746\u001b[0m  110.8584\n",
            "     31        \u001b[36m1.0592\u001b[0m       \u001b[32m0.5850\u001b[0m        \u001b[35m1.1687\u001b[0m  110.6405\n",
            "     32        \u001b[36m1.0467\u001b[0m       \u001b[32m0.5898\u001b[0m        \u001b[35m1.1630\u001b[0m  110.5864\n",
            "     33        \u001b[36m1.0342\u001b[0m       \u001b[32m0.5916\u001b[0m        \u001b[35m1.1576\u001b[0m  110.5746\n",
            "     34        \u001b[36m1.0220\u001b[0m       \u001b[32m0.5927\u001b[0m        \u001b[35m1.1527\u001b[0m  110.6105\n",
            "     35        \u001b[36m1.0099\u001b[0m       \u001b[32m0.5945\u001b[0m        \u001b[35m1.1480\u001b[0m  110.5982\n",
            "     36        \u001b[36m0.9980\u001b[0m       \u001b[32m0.5966\u001b[0m        \u001b[35m1.1437\u001b[0m  110.5520\n",
            "     37        \u001b[36m0.9862\u001b[0m       \u001b[32m0.5983\u001b[0m        \u001b[35m1.1395\u001b[0m  110.8625\n",
            "     38        \u001b[36m0.9744\u001b[0m       \u001b[32m0.6007\u001b[0m        \u001b[35m1.1356\u001b[0m  110.6520\n",
            "     39        \u001b[36m0.9628\u001b[0m       \u001b[32m0.6018\u001b[0m        \u001b[35m1.1319\u001b[0m  110.5320\n",
            "     40        \u001b[36m0.9512\u001b[0m       \u001b[32m0.6034\u001b[0m        \u001b[35m1.1284\u001b[0m  110.5010\n",
            "     41        \u001b[36m0.9396\u001b[0m       \u001b[32m0.6038\u001b[0m        \u001b[35m1.1249\u001b[0m  110.5573\n",
            "     42        \u001b[36m0.9280\u001b[0m       \u001b[32m0.6062\u001b[0m        \u001b[35m1.1217\u001b[0m  110.5510\n",
            "training for : 128 channels, 0.001 learning rate, <class 'torch.optim.adam.Adam'> optimizer\n",
            "  epoch    train_loss    valid_acc    valid_loss      dur\n",
            "-------  ------------  -----------  ------------  -------\n",
            "      1        \u001b[36m1.5485\u001b[0m       \u001b[32m0.5754\u001b[0m        \u001b[35m1.2160\u001b[0m  15.0222\n",
            "      2        \u001b[36m1.1238\u001b[0m       \u001b[32m0.6161\u001b[0m        \u001b[35m1.1024\u001b[0m  15.1097\n",
            "      3        \u001b[36m0.8903\u001b[0m       \u001b[32m0.6176\u001b[0m        1.1344  15.0436\n",
            "      4        \u001b[36m0.6909\u001b[0m       0.5931        1.3639  15.0826\n",
            "      5        \u001b[36m0.5256\u001b[0m       0.5940        1.5818  15.0317\n",
            "      6        \u001b[36m0.3993\u001b[0m       0.5884        1.7692  15.0935\n",
            "Stopping since valid_loss has not improved in the last 5 epochs.\n",
            "training for : 256 channels, 0.001 learning rate, <class 'torch.optim.adam.Adam'> optimizer\n",
            "  epoch    train_loss    valid_acc    valid_loss      dur\n",
            "-------  ------------  -----------  ------------  -------\n",
            "      1        \u001b[36m1.7967\u001b[0m       \u001b[32m0.5024\u001b[0m        \u001b[35m1.3882\u001b[0m  36.3632\n",
            "      2        \u001b[36m1.2401\u001b[0m       \u001b[32m0.5986\u001b[0m        \u001b[35m1.1405\u001b[0m  36.4417\n",
            "      3        \u001b[36m0.9719\u001b[0m       \u001b[32m0.6264\u001b[0m        \u001b[35m1.1058\u001b[0m  36.4436\n",
            "      4        \u001b[36m0.7613\u001b[0m       0.6065        1.2211  36.4474\n",
            "      5        \u001b[36m0.5845\u001b[0m       0.5951        1.4564  36.4795\n",
            "      6        \u001b[36m0.4487\u001b[0m       0.5920        1.6230  36.4023\n",
            "      7        \u001b[36m0.3493\u001b[0m       0.5669        2.0429  36.4087\n",
            "Stopping since valid_loss has not improved in the last 5 epochs.\n",
            "training for : 512 channels, 0.001 learning rate, <class 'torch.optim.adam.Adam'> optimizer\n",
            "  epoch    train_loss    valid_acc    valid_loss       dur\n",
            "-------  ------------  -----------  ------------  --------\n",
            "      1        \u001b[36m2.0993\u001b[0m       \u001b[32m0.4151\u001b[0m        \u001b[35m1.6298\u001b[0m  102.6764\n",
            "      2        \u001b[36m1.4408\u001b[0m       \u001b[32m0.5437\u001b[0m        \u001b[35m1.2858\u001b[0m  103.0963\n",
            "      3        \u001b[36m1.1922\u001b[0m       \u001b[32m0.5714\u001b[0m        \u001b[35m1.2155\u001b[0m  102.9424\n",
            "      4        \u001b[36m1.0232\u001b[0m       \u001b[32m0.5880\u001b[0m        \u001b[35m1.1975\u001b[0m  102.8897\n",
            "      5        \u001b[36m0.8413\u001b[0m       0.5786        1.3375  102.8098\n",
            "      6        \u001b[36m0.6616\u001b[0m       0.5454        1.9111  103.0538\n",
            "      7        \u001b[36m0.5089\u001b[0m       0.5501        2.0487  102.7659\n",
            "      8        \u001b[36m0.3915\u001b[0m       0.5472        2.2861  102.6992\n",
            "Stopping since valid_loss has not improved in the last 5 epochs.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "go55LVSJd-vG"
      },
      "source": [
        "### 2) Full CNN implementation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6G0eCj6OmOEE"
      },
      "source": [
        "Based on the CNN in the previous question, implement a full CNN model with max pooling layer.\n",
        "\n",
        "- Add a max pooling layer after each convolutional layer.\n",
        "- Each max pooling layer has a kernel size of 2 and a stride of 2.\n",
        "\n",
        "Implement this model in the following section. We will need to tune the hyperparameters and fill the results in the table."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bMrKGlMQhCa0"
      },
      "source": [
        "#### a) Implement max pooling layers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g2INt6P3Myd1"
      },
      "source": [
        "Implement max pooling layers."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DHu3Ic2dM1S9"
      },
      "source": [
        "class CNN_MaxPool(nn.Module):\n",
        "  \n",
        "  def __init__(self,channels_1=512,channels_2=1024,channels_3=2048):\n",
        "    super(CNN_MaxPool, self).__init__()\n",
        "    # implement parameter definitions here\n",
        "    self.conv1 = nn.Conv2d(3, channels_1, kernel_size = 3, padding = 1,stride=1)\n",
        "    self.conv2 = nn.Conv2d(channels_1, channels_2, kernel_size = 3, padding = 1,stride=1)\n",
        "    self.conv3 = nn.Conv2d(channels_2, channels_3, kernel_size = 3, padding = 1,stride=1)\n",
        "    self.fc1 = nn.Linear(channels_3 * 4 * 4 , len(set(train.targets)))\n",
        "  \n",
        "  def forward(self, images):\n",
        "    # implement the forward function here\n",
        "    \n",
        "    images = F.relu(F.max_pool2d(self.conv1(images), stride = 2, kernel_size=2))\n",
        "    images = F.relu(F.max_pool2d(self.conv2(images), stride = 2, kernel_size=2))\n",
        "    images = F.relu(F.max_pool2d(self.conv3(images), stride = 2, kernel_size=2))\n",
        "    images = images.view(-1, images.size(1) * images.size(2) * images.size(3))\n",
        "    images = self.fc1(images)\n",
        "    return images\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-A6AEOoigq68"
      },
      "source": [
        "#### b) Tune hyperparameters"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "drH4MHSVNqwz"
      },
      "source": [
        "Based on best optimizer you found in the previous problem, tune the number of channels and learning rate for best validation accuracy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7povzg-4Nhrr",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "3444ffa5-ed18-48a2-bf59-f29a2cdfe9ff"
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "# Hyperparameters to search\n",
        "lr=[1e-6,1e-5,1e-4,1e-3]\n",
        "opt = [optim.Adam]\n",
        "ch = [[128, 128, 128], [128, 256, 512], [256, 256, 256], [256, 512, 1024], [512, 512, 512], [512, 1024, 2048],]\n",
        "\n",
        "# Empty dataframe to store results\n",
        "results = pd.DataFrame()\n",
        "\n",
        "# Transform to the right type\n",
        "X = torch.FloatTensor(train.data.reshape(-1, 3, 32, 32) / 255.0)\n",
        "\n",
        "\n",
        "for l in lr:\n",
        "  for o in opt:\n",
        "    for c in ch:\n",
        "      print(\"training for : {0} channels | {1} learning rate | {2} optimizer\".format(str(c), l, str(o)) )\n",
        "      \n",
        "      # Define number of channels\n",
        "      cnn = CNN_MaxPool(channels_1=c[0],channels_2=c[1],channels_3=c[2])\n",
        "      \n",
        "      # implement hyperparameters here\n",
        "      model = skorch.NeuralNetClassifier(cnn, criterion=torch.nn.CrossEntropyLoss,\n",
        "                                        device=\"cuda\",\n",
        "                                        max_epochs = 20,\n",
        "                                        optimizer = o,\n",
        "                                        lr = l,\n",
        "                                        callbacks = [skorch.callbacks.EarlyStopping(lower_is_better=True)])\n",
        "      \n",
        "      # implement input normalization & type cast here\n",
        "      model.fit(X, np.array(train.targets))\n",
        "\n",
        "      # Store results for this hyperparameters\n",
        "      result = pd.DataFrame(pd.DataFrame(model.history).sort_values('valid_loss').loc[0, [\"epoch\", \"train_loss\", \"valid_loss\", \"valid_acc\"]]).T\n",
        "      result.assign(learning_rate = str(l), inplace = True)\n",
        "      result.assign(optimizer = str(o), inplace = True)\n",
        "      result.assign(channels_3 = str(c), inplace = True)\n",
        "      \n",
        "      # concat to results table\n",
        "      results = pd.concat([results, result])\n",
        "results.to_csv(\"CNN_q2b_results.csv\")\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "training for : [128, 128, 128] channels | 0.0001 learning rate | <class 'torch.optim.adam.Adam'> optimizer\n",
            "  epoch    train_loss    valid_acc    valid_loss     dur\n",
            "-------  ------------  -----------  ------------  ------\n",
            "      1        \u001b[36m2.0693\u001b[0m       \u001b[32m0.3174\u001b[0m        \u001b[35m1.9151\u001b[0m  6.3785\n",
            "      2        \u001b[36m1.8444\u001b[0m       \u001b[32m0.3614\u001b[0m        \u001b[35m1.7865\u001b[0m  6.3543\n",
            "      3        \u001b[36m1.7563\u001b[0m       \u001b[32m0.3836\u001b[0m        \u001b[35m1.7245\u001b[0m  6.3718\n",
            "      4        \u001b[36m1.7048\u001b[0m       \u001b[32m0.3965\u001b[0m        \u001b[35m1.6834\u001b[0m  6.3179\n",
            "      5        \u001b[36m1.6689\u001b[0m       \u001b[32m0.4075\u001b[0m        \u001b[35m1.6533\u001b[0m  6.3395\n",
            "      6        \u001b[36m1.6408\u001b[0m       \u001b[32m0.4143\u001b[0m        \u001b[35m1.6297\u001b[0m  6.3382\n",
            "      7        \u001b[36m1.6167\u001b[0m       \u001b[32m0.4213\u001b[0m        \u001b[35m1.6089\u001b[0m  6.3642\n",
            "      8        \u001b[36m1.5956\u001b[0m       \u001b[32m0.4311\u001b[0m        \u001b[35m1.5904\u001b[0m  6.3651\n",
            "      9        \u001b[36m1.5766\u001b[0m       \u001b[32m0.4388\u001b[0m        \u001b[35m1.5740\u001b[0m  6.3764\n",
            "     10        \u001b[36m1.5590\u001b[0m       \u001b[32m0.4443\u001b[0m        \u001b[35m1.5591\u001b[0m  6.3363\n",
            "     11        \u001b[36m1.5428\u001b[0m       \u001b[32m0.4487\u001b[0m        \u001b[35m1.5450\u001b[0m  6.3105\n",
            "     12        \u001b[36m1.5275\u001b[0m       \u001b[32m0.4509\u001b[0m        \u001b[35m1.5326\u001b[0m  6.3131\n",
            "     13        \u001b[36m1.5129\u001b[0m       \u001b[32m0.4562\u001b[0m        \u001b[35m1.5209\u001b[0m  6.4280\n",
            "     14        \u001b[36m1.4991\u001b[0m       \u001b[32m0.4627\u001b[0m        \u001b[35m1.5099\u001b[0m  6.3559\n",
            "     15        \u001b[36m1.4858\u001b[0m       \u001b[32m0.4663\u001b[0m        \u001b[35m1.4994\u001b[0m  6.3650\n",
            "     16        \u001b[36m1.4729\u001b[0m       \u001b[32m0.4689\u001b[0m        \u001b[35m1.4896\u001b[0m  6.3465\n",
            "     17        \u001b[36m1.4606\u001b[0m       \u001b[32m0.4708\u001b[0m        \u001b[35m1.4808\u001b[0m  6.3580\n",
            "     18        \u001b[36m1.4488\u001b[0m       \u001b[32m0.4740\u001b[0m        \u001b[35m1.4724\u001b[0m  6.3706\n",
            "     19        \u001b[36m1.4376\u001b[0m       \u001b[32m0.4769\u001b[0m        \u001b[35m1.4642\u001b[0m  6.3303\n",
            "     20        \u001b[36m1.4267\u001b[0m       \u001b[32m0.4819\u001b[0m        \u001b[35m1.4566\u001b[0m  6.3221\n",
            "training for : [128, 256, 512] channels | 0.0001 learning rate | <class 'torch.optim.adam.Adam'> optimizer\n",
            "  epoch    train_loss    valid_acc    valid_loss     dur\n",
            "-------  ------------  -----------  ------------  ------\n",
            "      1        \u001b[36m2.0062\u001b[0m       \u001b[32m0.3510\u001b[0m        \u001b[35m1.8242\u001b[0m  9.9374\n",
            "      2        \u001b[36m1.7742\u001b[0m       \u001b[32m0.3843\u001b[0m        \u001b[35m1.7231\u001b[0m  9.9063\n",
            "      3        \u001b[36m1.6828\u001b[0m       \u001b[32m0.4071\u001b[0m        \u001b[35m1.6595\u001b[0m  9.9405\n",
            "      4        \u001b[36m1.6210\u001b[0m       \u001b[32m0.4243\u001b[0m        \u001b[35m1.6061\u001b[0m  9.9461\n",
            "      5        \u001b[36m1.5723\u001b[0m       \u001b[32m0.4404\u001b[0m        \u001b[35m1.5619\u001b[0m  9.9437\n",
            "      6        \u001b[36m1.5310\u001b[0m       \u001b[32m0.4543\u001b[0m        \u001b[35m1.5255\u001b[0m  9.8955\n",
            "      7        \u001b[36m1.4942\u001b[0m       \u001b[32m0.4645\u001b[0m        \u001b[35m1.4971\u001b[0m  9.9482\n",
            "      8        \u001b[36m1.4614\u001b[0m       \u001b[32m0.4729\u001b[0m        \u001b[35m1.4742\u001b[0m  9.9116\n",
            "      9        \u001b[36m1.4317\u001b[0m       \u001b[32m0.4787\u001b[0m        \u001b[35m1.4559\u001b[0m  9.9001\n",
            "     10        \u001b[36m1.4045\u001b[0m       \u001b[32m0.4858\u001b[0m        \u001b[35m1.4415\u001b[0m  9.9651\n",
            "     11        \u001b[36m1.3787\u001b[0m       \u001b[32m0.4912\u001b[0m        \u001b[35m1.4286\u001b[0m  9.9368\n",
            "     12        \u001b[36m1.3541\u001b[0m       \u001b[32m0.4971\u001b[0m        \u001b[35m1.4166\u001b[0m  9.8980\n",
            "     13        \u001b[36m1.3306\u001b[0m       \u001b[32m0.5018\u001b[0m        \u001b[35m1.4049\u001b[0m  9.9388\n",
            "     14        \u001b[36m1.3078\u001b[0m       \u001b[32m0.5065\u001b[0m        \u001b[35m1.3939\u001b[0m  9.8991\n",
            "     15        \u001b[36m1.2859\u001b[0m       \u001b[32m0.5106\u001b[0m        \u001b[35m1.3839\u001b[0m  9.9385\n",
            "     16        \u001b[36m1.2651\u001b[0m       \u001b[32m0.5155\u001b[0m        \u001b[35m1.3771\u001b[0m  9.9447\n",
            "     17        \u001b[36m1.2490\u001b[0m       0.5125        1.3887  9.8972\n",
            "     18        \u001b[36m1.2306\u001b[0m       \u001b[32m0.5196\u001b[0m        \u001b[35m1.3657\u001b[0m  9.9092\n",
            "     19        \u001b[36m1.2079\u001b[0m       \u001b[32m0.5231\u001b[0m        \u001b[35m1.3566\u001b[0m  9.9173\n",
            "     20        \u001b[36m1.1856\u001b[0m       \u001b[32m0.5260\u001b[0m        \u001b[35m1.3496\u001b[0m  9.8811\n",
            "training for : [256, 256, 256] channels | 0.0001 learning rate | <class 'torch.optim.adam.Adam'> optimizer\n",
            "  epoch    train_loss    valid_acc    valid_loss      dur\n",
            "-------  ------------  -----------  ------------  -------\n",
            "      1        \u001b[36m2.0156\u001b[0m       \u001b[32m0.3482\u001b[0m        \u001b[35m1.8213\u001b[0m  14.6430\n",
            "      2        \u001b[36m1.7653\u001b[0m       \u001b[32m0.3886\u001b[0m        \u001b[35m1.7037\u001b[0m  14.6534\n",
            "      3        \u001b[36m1.6769\u001b[0m       \u001b[32m0.4100\u001b[0m        \u001b[35m1.6510\u001b[0m  14.6016\n",
            "      4        \u001b[36m1.6222\u001b[0m       \u001b[32m0.4282\u001b[0m        \u001b[35m1.6061\u001b[0m  14.6427\n",
            "      5        \u001b[36m1.5792\u001b[0m       \u001b[32m0.4400\u001b[0m        \u001b[35m1.5663\u001b[0m  14.6336\n",
            "      6        \u001b[36m1.5414\u001b[0m       \u001b[32m0.4550\u001b[0m        \u001b[35m1.5331\u001b[0m  14.6787\n",
            "      7        \u001b[36m1.5078\u001b[0m       \u001b[32m0.4654\u001b[0m        \u001b[35m1.5066\u001b[0m  14.6364\n",
            "      8        \u001b[36m1.4770\u001b[0m       \u001b[32m0.4723\u001b[0m        \u001b[35m1.4820\u001b[0m  14.6111\n",
            "      9        \u001b[36m1.4489\u001b[0m       \u001b[32m0.4794\u001b[0m        \u001b[35m1.4630\u001b[0m  14.6332\n",
            "     10        \u001b[36m1.4231\u001b[0m       \u001b[32m0.4836\u001b[0m        \u001b[35m1.4461\u001b[0m  14.6668\n",
            "     11        \u001b[36m1.3990\u001b[0m       \u001b[32m0.4906\u001b[0m        \u001b[35m1.4313\u001b[0m  14.6060\n",
            "     12        \u001b[36m1.3761\u001b[0m       \u001b[32m0.4957\u001b[0m        \u001b[35m1.4184\u001b[0m  14.6622\n",
            "     13        \u001b[36m1.3540\u001b[0m       \u001b[32m0.5003\u001b[0m        \u001b[35m1.4068\u001b[0m  14.6228\n",
            "     14        \u001b[36m1.3328\u001b[0m       \u001b[32m0.5055\u001b[0m        \u001b[35m1.3954\u001b[0m  14.7046\n",
            "     15        \u001b[36m1.3127\u001b[0m       \u001b[32m0.5101\u001b[0m        \u001b[35m1.3855\u001b[0m  14.6011\n",
            "     16        \u001b[36m1.2946\u001b[0m       \u001b[32m0.5116\u001b[0m        \u001b[35m1.3835\u001b[0m  14.6164\n",
            "     17        \u001b[36m1.2778\u001b[0m       \u001b[32m0.5155\u001b[0m        \u001b[35m1.3747\u001b[0m  14.6427\n",
            "     18        \u001b[36m1.2595\u001b[0m       \u001b[32m0.5212\u001b[0m        \u001b[35m1.3603\u001b[0m  14.6791\n",
            "     19        \u001b[36m1.2398\u001b[0m       \u001b[32m0.5255\u001b[0m        \u001b[35m1.3500\u001b[0m  14.6524\n",
            "     20        \u001b[36m1.2204\u001b[0m       \u001b[32m0.5285\u001b[0m        \u001b[35m1.3408\u001b[0m  14.6322\n",
            "training for : [256, 512, 1024] channels | 0.0001 learning rate | <class 'torch.optim.adam.Adam'> optimizer\n",
            "  epoch    train_loss    valid_acc    valid_loss      dur\n",
            "-------  ------------  -----------  ------------  -------\n",
            "      1        \u001b[36m1.9699\u001b[0m       \u001b[32m0.3590\u001b[0m        \u001b[35m1.7828\u001b[0m  25.0279\n",
            "      2        \u001b[36m1.7249\u001b[0m       \u001b[32m0.3981\u001b[0m        \u001b[35m1.6783\u001b[0m  25.0559\n",
            "      3        \u001b[36m1.6262\u001b[0m       \u001b[32m0.4220\u001b[0m        \u001b[35m1.6096\u001b[0m  24.9948\n",
            "      4        \u001b[36m1.5570\u001b[0m       \u001b[32m0.4461\u001b[0m        \u001b[35m1.5482\u001b[0m  24.9829\n",
            "      5        \u001b[36m1.4958\u001b[0m       \u001b[32m0.4598\u001b[0m        \u001b[35m1.5035\u001b[0m  25.0450\n",
            "      6        \u001b[36m1.4387\u001b[0m       \u001b[32m0.4734\u001b[0m        \u001b[35m1.4651\u001b[0m  25.1005\n",
            "      7        \u001b[36m1.3863\u001b[0m       \u001b[32m0.4880\u001b[0m        \u001b[35m1.4350\u001b[0m  25.0394\n",
            "      8        \u001b[36m1.3377\u001b[0m       \u001b[32m0.4996\u001b[0m        \u001b[35m1.4068\u001b[0m  25.0524\n",
            "      9        \u001b[36m1.2917\u001b[0m       \u001b[32m0.5101\u001b[0m        \u001b[35m1.3826\u001b[0m  25.0790\n",
            "     10        \u001b[36m1.2485\u001b[0m       \u001b[32m0.5202\u001b[0m        \u001b[35m1.3620\u001b[0m  25.0628\n",
            "     11        \u001b[36m1.2089\u001b[0m       0.5179        1.3712  25.0881\n",
            "     12        \u001b[36m1.1714\u001b[0m       \u001b[32m0.5250\u001b[0m        \u001b[35m1.3468\u001b[0m  25.0389\n",
            "     13        \u001b[36m1.1289\u001b[0m       \u001b[32m0.5356\u001b[0m        \u001b[35m1.3148\u001b[0m  25.0765\n",
            "     14        \u001b[36m1.0863\u001b[0m       \u001b[32m0.5435\u001b[0m        \u001b[35m1.2992\u001b[0m  25.0600\n",
            "     15        \u001b[36m1.0428\u001b[0m       \u001b[32m0.5483\u001b[0m        \u001b[35m1.2889\u001b[0m  25.0939\n",
            "     16        \u001b[36m0.9996\u001b[0m       \u001b[32m0.5506\u001b[0m        \u001b[35m1.2841\u001b[0m  25.0663\n",
            "     17        \u001b[36m0.9566\u001b[0m       0.5499        \u001b[35m1.2823\u001b[0m  25.0838\n",
            "     18        \u001b[36m0.9143\u001b[0m       0.5489        1.2882  25.1075\n",
            "     19        \u001b[36m0.8727\u001b[0m       0.5409        1.3214  25.1243\n",
            "     20        \u001b[36m0.8314\u001b[0m       0.5410        1.3416  25.0989\n",
            "training for : [512, 512, 512] channels | 0.0001 learning rate | <class 'torch.optim.adam.Adam'> optimizer\n",
            "  epoch    train_loss    valid_acc    valid_loss      dur\n",
            "-------  ------------  -----------  ------------  -------\n",
            "      1        \u001b[36m1.9813\u001b[0m       \u001b[32m0.3575\u001b[0m        \u001b[35m1.7952\u001b[0m  33.2091\n",
            "      2        \u001b[36m1.7327\u001b[0m       \u001b[32m0.3933\u001b[0m        \u001b[35m1.6952\u001b[0m  33.2452\n",
            "      3        \u001b[36m1.6373\u001b[0m       \u001b[32m0.4211\u001b[0m        \u001b[35m1.6168\u001b[0m  33.3273\n",
            "      4        \u001b[36m1.5747\u001b[0m       \u001b[32m0.4410\u001b[0m        \u001b[35m1.5603\u001b[0m  33.2583\n",
            "      5        \u001b[36m1.5216\u001b[0m       \u001b[32m0.4566\u001b[0m        \u001b[35m1.5178\u001b[0m  33.3179\n",
            "      6        \u001b[36m1.4733\u001b[0m       \u001b[32m0.4716\u001b[0m        \u001b[35m1.4838\u001b[0m  33.3811\n",
            "      7        \u001b[36m1.4291\u001b[0m       \u001b[32m0.4806\u001b[0m        \u001b[35m1.4575\u001b[0m  33.3827\n",
            "      8        \u001b[36m1.3873\u001b[0m       \u001b[32m0.4909\u001b[0m        \u001b[35m1.4335\u001b[0m  33.3302\n",
            "      9        \u001b[36m1.3479\u001b[0m       \u001b[32m0.4987\u001b[0m        \u001b[35m1.4107\u001b[0m  33.3957\n",
            "     10        \u001b[36m1.3109\u001b[0m       \u001b[32m0.5058\u001b[0m        \u001b[35m1.3933\u001b[0m  33.3664\n",
            "     11        \u001b[36m1.2776\u001b[0m       \u001b[32m0.5086\u001b[0m        \u001b[35m1.3922\u001b[0m  33.3765\n",
            "     12        \u001b[36m1.2467\u001b[0m       \u001b[32m0.5090\u001b[0m        \u001b[35m1.3918\u001b[0m  33.3368\n",
            "     13        \u001b[36m1.2145\u001b[0m       \u001b[32m0.5174\u001b[0m        \u001b[35m1.3685\u001b[0m  33.4396\n",
            "     14        \u001b[36m1.1807\u001b[0m       \u001b[32m0.5249\u001b[0m        \u001b[35m1.3505\u001b[0m  33.3597\n",
            "     15        \u001b[36m1.1469\u001b[0m       \u001b[32m0.5287\u001b[0m        \u001b[35m1.3389\u001b[0m  33.4028\n",
            "     16        \u001b[36m1.1134\u001b[0m       \u001b[32m0.5315\u001b[0m        \u001b[35m1.3277\u001b[0m  33.3532\n",
            "     17        \u001b[36m1.0800\u001b[0m       \u001b[32m0.5368\u001b[0m        \u001b[35m1.3165\u001b[0m  33.4248\n",
            "     18        \u001b[36m1.0469\u001b[0m       \u001b[32m0.5397\u001b[0m        \u001b[35m1.3085\u001b[0m  33.4061\n",
            "     19        \u001b[36m1.0139\u001b[0m       \u001b[32m0.5415\u001b[0m        \u001b[35m1.3032\u001b[0m  33.4075\n",
            "     20        \u001b[36m0.9804\u001b[0m       0.5396        1.3047  33.4156\n",
            "training for : [512, 1024, 2048] channels | 0.0001 learning rate | <class 'torch.optim.adam.Adam'> optimizer\n",
            "  epoch    train_loss    valid_acc    valid_loss      dur\n",
            "-------  ------------  -----------  ------------  -------\n",
            "      1        \u001b[36m1.9392\u001b[0m       \u001b[32m0.3672\u001b[0m        \u001b[35m1.7434\u001b[0m  72.5097\n",
            "      2        \u001b[36m1.6858\u001b[0m       \u001b[32m0.4174\u001b[0m        \u001b[35m1.6248\u001b[0m  72.5668\n",
            "      3        \u001b[36m1.5750\u001b[0m       \u001b[32m0.4416\u001b[0m        \u001b[35m1.5553\u001b[0m  72.5532\n",
            "      4        \u001b[36m1.4903\u001b[0m       \u001b[32m0.4636\u001b[0m        \u001b[35m1.4902\u001b[0m  72.6250\n",
            "      5        \u001b[36m1.4243\u001b[0m       \u001b[32m0.4839\u001b[0m        \u001b[35m1.4349\u001b[0m  72.4904\n",
            "      6        \u001b[36m1.3613\u001b[0m       \u001b[32m0.4966\u001b[0m        \u001b[35m1.4086\u001b[0m  72.6382\n",
            "      7        \u001b[36m1.3003\u001b[0m       \u001b[32m0.5102\u001b[0m        \u001b[35m1.3798\u001b[0m  72.7860\n",
            "      8        \u001b[36m1.2396\u001b[0m       \u001b[32m0.5227\u001b[0m        \u001b[35m1.3530\u001b[0m  72.7674\n",
            "      9        \u001b[36m1.1815\u001b[0m       \u001b[32m0.5275\u001b[0m        \u001b[35m1.3356\u001b[0m  72.6696\n",
            "     10        \u001b[36m1.1246\u001b[0m       \u001b[32m0.5285\u001b[0m        1.3389  72.6912\n",
            "     11        \u001b[36m1.0731\u001b[0m       \u001b[32m0.5383\u001b[0m        \u001b[35m1.3174\u001b[0m  72.7692\n",
            "     12        \u001b[36m1.0141\u001b[0m       \u001b[32m0.5439\u001b[0m        \u001b[35m1.3024\u001b[0m  72.7590\n",
            "     13        \u001b[36m0.9470\u001b[0m       \u001b[32m0.5454\u001b[0m        1.3098  72.7959\n",
            "     14        \u001b[36m0.8781\u001b[0m       0.5434        1.3347  72.6329\n",
            "     15        \u001b[36m0.8088\u001b[0m       0.5423        1.3630  72.6315\n",
            "     16        \u001b[36m0.7394\u001b[0m       0.5393        1.3963  72.7296\n",
            "Stopping since valid_loss has not improved in the last 5 epochs.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H7Mu2ZZHoZU0"
      },
      "source": [
        "Write down the **validation accuracy** of your model under different hyperparameter settings.\n",
        "\n",
        "| #channel for each layer | validation accuracy |\n",
        "|-------------------------|---------------------|\n",
        "| (128, 128, 128)         |   0.4819            |\n",
        "| (128, 256, 512)         |   0.5260            |\n",
        "| (256, 256, 256)         |   0.5285            |\n",
        "| (256, 512, 1024)        |   0.5506            |\n",
        "| (512, 512, 512)         |   0.5415            |\n",
        "| (512, 1024, 2048)       |   0.5454            |\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8UCaz8nWoWWS"
      },
      "source": [
        "For the best model you have, test it on the test set.\n",
        "\n",
        "It is fine if you found some hyperparameter combination better than those listed in the tables."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bTtBk22OECDD",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "5f2b0c54-326e-4795-b5cd-25b256ca55e0"
      },
      "source": [
        "# implement the same input normalization & type cast here\n",
        "test.predictions = model.predict(torch.Tensor(test.data.reshape(-1,3,32,32)/255.0))\n",
        "sklearn.metrics.accuracy_score(test.targets, test.predictions)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.5231"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kSbhC8f1or6_"
      },
      "source": [
        "How much **test accuracy** do you get?\n",
        "\n",
        "**Your Answer:** 0.5213"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nG3E9ZckomkX"
      },
      "source": [
        "What can you conclude for the design of CNN structure?\n",
        "\n",
        "**Your Answer:** Increase in number of layers results in extracting more features until the model starts to overfit the data. Therefore, with the accurate variation of hyperparameters, the CNN structure will be able to achieve an optimum validation accuracy and test accuracy. In the above model, we find learning rate = 1e-4 or 0.0001, channels for each layer = (512,1024,2048) respectively, optimizer = Adam, epoch = 20, we achieve a validation accuracy = 0.5454 and test accuracy = 0.5213"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AWO5rjXuPIH5"
      },
      "source": [
        "## Recurrent Neural Networks (40 points)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D2AmIPwBJt9j"
      },
      "source": [
        "Next, let's use PyTorch to implement a recurrent neural network for sentiment analysis, i.e., classifying sentences into given sentiment labels, including positive, negative and neutral.\n",
        "\n",
        "We use a benckmark dataset (i.e., SST) for this task. First, let's download the SST dataset, and do some preprocessing to build vocabulary and split the dataset into training/validation/test sets. Also, let's define the training and evaluation function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HT8b2nr7Kq73"
      },
      "source": [
        "import copy\n",
        "import torch\n",
        "from torch import nn\n",
        "from torch import optim\n",
        "import torchtext\n",
        "from torchtext import data\n",
        "from torchtext import datasets\n",
        "\n",
        "TEXT = data.Field(sequential=True, batch_first=True, lower=True)\n",
        "LABEL = data.LabelField()\n",
        "\n",
        "# load data splits\n",
        "train_data, val_data, test_data = datasets.SST.splits(TEXT, LABEL)\n",
        "\n",
        "# build dictionary\n",
        "TEXT.build_vocab(train_data)\n",
        "LABEL.build_vocab(train_data)\n",
        "\n",
        "vocab_size = len(TEXT.vocab)\n",
        "label_size = len(LABEL.vocab)\n",
        "padding_idx = TEXT.vocab.stoi['<pad>']\n",
        "embedding_dim = 64\n",
        "hidden_dim = 256\n",
        "\n",
        "# build iterators\n",
        "train_iter, val_iter, test_iter = data.BucketIterator.splits(\n",
        "    (train_data, val_data, test_data), \n",
        "    batch_size=64)\n",
        "\n",
        "# train a model\n",
        "def train(model, iterator, optimizer, criterion):\n",
        "    total_loss, total_correct, total_prediction = 0.0, 0.0, 0.0\n",
        "    model.train()\n",
        "    for batch in iterator:\n",
        "        optimizer.zero_grad()\n",
        "        logits = model(batch.text.cuda())\n",
        "        predictions = torch.max(logits, dim=-1)[1]\n",
        "        loss = criterion(logits, batch.label.cuda())\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "        \n",
        "        total_loss += loss.item()\n",
        "        total_correct += torch.eq(predictions, batch.label.cuda()).sum().item()\n",
        "        total_prediction += batch.label.size(0)\n",
        "    return total_loss / len(iterator), total_correct / total_prediction\n",
        "\n",
        "# evaluate a model\n",
        "def evaluate(model, iterator, criterion):  \n",
        "    total_loss, total_correct, total_prediction = 0.0, 0.0, 0.0\n",
        "    model.eval()\n",
        "    with torch.no_grad():\n",
        "        for batch in iterator:\n",
        "            logits = model(batch.text.cuda())\n",
        "            predictions = torch.max(logits, dim=-1)[1]\n",
        "            loss = criterion(logits, batch.label.cuda())\n",
        "\n",
        "            total_loss += loss.item()\n",
        "            total_correct += torch.eq(predictions, batch.label.cuda()).sum().item()\n",
        "            total_prediction += batch.label.size(0)\n",
        "    return total_loss / len(iterator), total_correct / total_prediction"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pbWxFDZdK6rf"
      },
      "source": [
        "Next, we are ready to build our RNN model for sentiment analysis. In the following codes, we have provided several hyperparameters we needed for building the model, including vocabulary size (vocab_size), the word embedding dimension (embedding_dim), the hidden layer dimension (hidden_dim), the number of layers (num_layers) and the number of sentence labels (label_size). We implement an LSTM model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kWUKPgDGNQSr"
      },
      "source": [
        "class RNNClassifier(nn.Module):\n",
        "    def __init__(self, vocab_size, embedding_dim, hidden_dim, label_size, padding_idx):\n",
        "        super(RNNClassifier, self).__init__()\n",
        "        self.vocab_size = vocab_size\n",
        "        self.embedding_dim = embedding_dim\n",
        "        self.hidden_dim = hidden_dim\n",
        "        self.label_size = label_size\n",
        "        self.num_layers = 1\n",
        "\n",
        "\n",
        "        # add the layers required for sentiment analysis.\n",
        "        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=padding_idx)\n",
        "        self.lstm=nn.LSTM(input_size=embedding_dim,hidden_size=hidden_dim,num_layers=self.num_layers)\n",
        "        self.fc=nn.Linear(hidden_dim,label_size)\n",
        "\n",
        "    def zero_state(self, batch_size): \n",
        "        # implement the function, which returns an initial hidden state.\n",
        "          h=torch.zeros(self.num_layers,batch_size,self.hidden_dim)\n",
        "          c=torch.zeros(self.num_layers,batch_size,self.hidden_dim)\n",
        "          return h, c\n",
        "\n",
        "    def forward(self, text):\n",
        "        # implement the forward function of the model.\n",
        "        h,c=self.zero_state(text.size(0))\n",
        "        embedding=self.embedding(text)\n",
        "        lstm_out, _=self.lstm(embedding)\n",
        "        lstm_out = F.softmax(lstm_out, dim=-1)\n",
        "        return self.fc(lstm_out[:, 1, :])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EBX_xc9MN0gw"
      },
      "source": [
        "Finally, we are ready to train the model and compute the accuracy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KQrU0wuUOIgb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 897
        },
        "outputId": "3d87dadf-22ec-4928-cfd1-8bb7b3e20bb7"
      },
      "source": [
        "model = RNNClassifier(vocab_size, embedding_dim, hidden_dim, label_size, padding_idx)\n",
        "# tune the optimizer type and hyperparameters here\n",
        "optimizer = optim.Adam(model.parameters(), lr=0.01)\n",
        "criterion = nn.CrossEntropyLoss()\n",
        "model.cuda()\n",
        "criterion.cuda()\n",
        "\n",
        "# train and test the model\n",
        "best_valid_acc = 0.0\n",
        "best_state_dict = copy.deepcopy(model.state_dict())\n",
        "for epoch in range(50):\n",
        "    train_loss, train_acc = train(model, train_iter, optimizer, criterion)\n",
        "    valid_loss, valid_acc = evaluate(model, val_iter, criterion)\n",
        "\n",
        "    print('Epoch {} | Train loss {:.3f} | Train Accuracy {:.3f} | Valid loss {:.3f} | Valid acc {:.3f}'.format(epoch, train_loss, train_acc, valid_loss, valid_acc))\n",
        "\n",
        "    if valid_acc > best_valid_acc:\n",
        "        best_valid_acc = valid_acc\n",
        "        best_state_dict = copy.deepcopy(model.state_dict())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 0 | Train loss 1.050 | Train Accuracy 0.416 | Valid loss 1.063 | Valid acc 0.403\n",
            "Epoch 1 | Train loss 1.048 | Train Accuracy 0.422 | Valid loss 1.060 | Valid acc 0.403\n",
            "Epoch 2 | Train loss 1.048 | Train Accuracy 0.418 | Valid loss 1.062 | Valid acc 0.403\n",
            "Epoch 3 | Train loss 1.047 | Train Accuracy 0.423 | Valid loss 1.061 | Valid acc 0.403\n",
            "Epoch 4 | Train loss 1.040 | Train Accuracy 0.457 | Valid loss 1.059 | Valid acc 0.406\n",
            "Epoch 5 | Train loss 1.005 | Train Accuracy 0.549 | Valid loss 1.065 | Valid acc 0.421\n",
            "Epoch 6 | Train loss 0.949 | Train Accuracy 0.584 | Valid loss 1.085 | Valid acc 0.420\n",
            "Epoch 7 | Train loss 0.920 | Train Accuracy 0.590 | Valid loss 1.088 | Valid acc 0.431\n",
            "Epoch 8 | Train loss 0.898 | Train Accuracy 0.590 | Valid loss 1.081 | Valid acc 0.425\n",
            "Epoch 9 | Train loss 0.880 | Train Accuracy 0.589 | Valid loss 1.090 | Valid acc 0.426\n",
            "Epoch 10 | Train loss 0.864 | Train Accuracy 0.589 | Valid loss 1.094 | Valid acc 0.410\n",
            "Epoch 11 | Train loss 0.853 | Train Accuracy 0.586 | Valid loss 1.089 | Valid acc 0.416\n",
            "Epoch 12 | Train loss 0.842 | Train Accuracy 0.595 | Valid loss 1.099 | Valid acc 0.444\n",
            "Epoch 13 | Train loss 0.834 | Train Accuracy 0.594 | Valid loss 1.096 | Valid acc 0.428\n",
            "Epoch 14 | Train loss 0.824 | Train Accuracy 0.597 | Valid loss 1.092 | Valid acc 0.444\n",
            "Epoch 15 | Train loss 0.817 | Train Accuracy 0.596 | Valid loss 1.102 | Valid acc 0.433\n",
            "Epoch 16 | Train loss 0.808 | Train Accuracy 0.609 | Valid loss 1.103 | Valid acc 0.428\n",
            "Epoch 17 | Train loss 0.801 | Train Accuracy 0.619 | Valid loss 1.103 | Valid acc 0.421\n",
            "Epoch 18 | Train loss 0.793 | Train Accuracy 0.635 | Valid loss 1.109 | Valid acc 0.427\n",
            "Epoch 19 | Train loss 0.783 | Train Accuracy 0.635 | Valid loss 1.108 | Valid acc 0.442\n",
            "Epoch 20 | Train loss 0.777 | Train Accuracy 0.634 | Valid loss 1.109 | Valid acc 0.451\n",
            "Epoch 21 | Train loss 0.770 | Train Accuracy 0.640 | Valid loss 1.124 | Valid acc 0.440\n",
            "Epoch 22 | Train loss 0.765 | Train Accuracy 0.637 | Valid loss 1.114 | Valid acc 0.453\n",
            "Epoch 23 | Train loss 0.760 | Train Accuracy 0.643 | Valid loss 1.118 | Valid acc 0.444\n",
            "Epoch 24 | Train loss 0.754 | Train Accuracy 0.638 | Valid loss 1.127 | Valid acc 0.438\n",
            "Epoch 25 | Train loss 0.750 | Train Accuracy 0.640 | Valid loss 1.127 | Valid acc 0.433\n",
            "Epoch 26 | Train loss 0.748 | Train Accuracy 0.642 | Valid loss 1.135 | Valid acc 0.441\n",
            "Epoch 27 | Train loss 0.743 | Train Accuracy 0.645 | Valid loss 1.136 | Valid acc 0.438\n",
            "Epoch 28 | Train loss 0.742 | Train Accuracy 0.639 | Valid loss 1.138 | Valid acc 0.432\n",
            "Epoch 29 | Train loss 0.736 | Train Accuracy 0.645 | Valid loss 1.141 | Valid acc 0.422\n",
            "Epoch 30 | Train loss 0.736 | Train Accuracy 0.639 | Valid loss 1.134 | Valid acc 0.412\n",
            "Epoch 31 | Train loss 0.734 | Train Accuracy 0.637 | Valid loss 1.132 | Valid acc 0.421\n",
            "Epoch 32 | Train loss 0.731 | Train Accuracy 0.640 | Valid loss 1.138 | Valid acc 0.436\n",
            "Epoch 33 | Train loss 0.728 | Train Accuracy 0.642 | Valid loss 1.141 | Valid acc 0.435\n",
            "Epoch 34 | Train loss 0.726 | Train Accuracy 0.636 | Valid loss 1.138 | Valid acc 0.435\n",
            "Epoch 35 | Train loss 0.723 | Train Accuracy 0.640 | Valid loss 1.154 | Valid acc 0.425\n",
            "Epoch 36 | Train loss 0.724 | Train Accuracy 0.637 | Valid loss 1.151 | Valid acc 0.440\n",
            "Epoch 37 | Train loss 0.722 | Train Accuracy 0.638 | Valid loss 1.145 | Valid acc 0.437\n",
            "Epoch 38 | Train loss 0.721 | Train Accuracy 0.643 | Valid loss 1.146 | Valid acc 0.438\n",
            "Epoch 39 | Train loss 0.719 | Train Accuracy 0.641 | Valid loss 1.152 | Valid acc 0.426\n",
            "Epoch 40 | Train loss 0.718 | Train Accuracy 0.635 | Valid loss 1.148 | Valid acc 0.429\n",
            "Epoch 41 | Train loss 0.717 | Train Accuracy 0.641 | Valid loss 1.158 | Valid acc 0.440\n",
            "Epoch 42 | Train loss 0.716 | Train Accuracy 0.636 | Valid loss 1.157 | Valid acc 0.443\n",
            "Epoch 43 | Train loss 0.714 | Train Accuracy 0.643 | Valid loss 1.169 | Valid acc 0.434\n",
            "Epoch 44 | Train loss 0.714 | Train Accuracy 0.640 | Valid loss 1.175 | Valid acc 0.432\n",
            "Epoch 45 | Train loss 0.713 | Train Accuracy 0.638 | Valid loss 1.170 | Valid acc 0.411\n",
            "Epoch 46 | Train loss 0.713 | Train Accuracy 0.643 | Valid loss 1.164 | Valid acc 0.418\n",
            "Epoch 47 | Train loss 0.713 | Train Accuracy 0.640 | Valid loss 1.174 | Valid acc 0.414\n",
            "Epoch 48 | Train loss 0.710 | Train Accuracy 0.640 | Valid loss 1.172 | Valid acc 0.414\n",
            "Epoch 49 | Train loss 0.709 | Train Accuracy 0.644 | Valid loss 1.178 | Valid acc 0.428\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4EgT69I1reZ4"
      },
      "source": [
        "Once we find the best hyperparameters for the validation set, we can now evaluate our model on the test set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rPDvglccrdWt",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "403f029a-506c-49f5-c85a-e1de614fbea7"
      },
      "source": [
        "model.load_state_dict(best_state_dict)\n",
        "test_loss, test_acc = evaluate(model, test_iter, criterion)\n",
        "print('Test loss {:.3f} | Test acc {:.3f}'.format(test_loss, test_acc))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Test loss 1.040 | Test acc 0.476\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PbqSAz90zBYi"
      },
      "source": [
        "### 1) Implement the RNN model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y_5KEDzgzVxT"
      },
      "source": [
        "The current codes of the RNN model are not complete, so let's first complete the codes to implement a standard RNN model by filling in the [block](https://colab.research.google.com/drive/1mhhF9FPHSmePtVQrhNBwRujfUkOjUspj#scrollTo=kWUKPgDGNQSr)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UiDCl9J0zIKO"
      },
      "source": [
        "- **Subtask 1-1: Creating all the Required Layers in Your Model**\n",
        "\n",
        "Remember that when building a deep learning model, we first need to complete the **init** function by creating all the required layers. In our case, since we are using RNNs for sentence classification, we need an embedding layer to transform words into word embeddings, a RNN layer to transform word embeddings into sentence encodings, an activation function, and a linear layer as well as a softmax function for sentence classification.\n",
        "\n",
        "Based on that, we create all the necessary layers of your RNN model in the **init** function. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tF_dE-TqzL0A"
      },
      "source": [
        "- **Subtask 1-2: Implementing the Function for Initializing Hidden States**\n",
        "\n",
        "Remember that when applying a RNN unit to transform word embeddings into sentence encodings, the RNN unit starts from an initial hidden vector with all zero values, and sequentially read each word to update the hidden vector. Finally, the hidden vector obtained after reading the last word is treated as the sentence encoding.\n",
        "\n",
        "In this step, we implement the **zero_state** function, which returns a batch of initial hidden vectors given a batch size. Our function returns a tensor with all the values being zero. Refer to the [official document](https://pytorch.org/docs/stable/nn.html#rnn) for the correct shape of the tensor."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kGiPTOnY28Iq"
      },
      "source": [
        "- **Subtask 1-3: Implementing the Forward Function**\n",
        "\n",
        "Finally, we are ready to build the forward function, which takes a batch of sentences as inputs and returns the a batch of logits. To be more specific, the input is given by the tensor called $\\text{text}$, and the size of the tensor is $(B, L)$, with $B$ being the batch size, $L$ being the maximum length of sentencees in this batch and $\\text{text}[i, j]$ being the interger id of the $j$-th word in the $i$-th sentence. Given this tensor as input, your forward function should return a logit tensor of size $(B, C)$, with $B$ being the batch size and $C$ being the number of possible classes.\n",
        "\n",
        "We implement the forward function based on the above instructions. We have applied the word embedding layer to the text input, and obtained a tensor called $\\text{embedding}$, and the size of the tensor is $(B, L, D)$, where $D$ is the word embedding dimension."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nUL9zYw4y_IZ"
      },
      "source": [
        "### 2) Compare Different Optimizers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SeKKFSkzzUA_"
      },
      "source": [
        "In the previous task, we have implemented a RNN model for sentiment analysis, or more generally sentence classification.\n",
        "\n",
        "To better understand several concepts in deep learning, let's do some ablation studies by using the model we have just implemented.\n",
        "\n",
        "The first task is to try different optimizers for your model, where for each optimizer, you may also try different options of learning rate. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kNlnDbZtqHgi"
      },
      "source": [
        "- **Subtask 2-1: Completing the Table**\n",
        "\n",
        "We have provided the following table for different combinations of optimizers and learning rate, please write down the **validation accuracy** of your model with different optimizers and learning rates.\n",
        "\n",
        "|         | 0.1  | 0.01 | 0.001|0.0001|\n",
        "|---------|------|------|------|------|\n",
        "| SGD     | 0.403|0.403 |0.403 |0.403 |\n",
        "| Adam    | 0.444|0.449 |0.445 |0.403 |\n",
        "| RMSprop | 0.419|0.441 |0.441 |0.403 |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nVVd5b9rzSFA"
      },
      "source": [
        "- **Subtask 2-2: Explaining your Observations**\n",
        "\n",
        "Based on your results, briefly explain your observations, e.g., which optimizer works the best, what is the optimal learning rate for each optimizer?\n",
        "\n",
        "*Your Answer:*\n",
        "Based on the observations, learning rate of 0.01 is optimal for the above deep learning model. Given the learning rate, SGD optimizer gives a validation accuracy of 0.403, Adam yields a validation accuracy of 0.449 and RMSprop of 0.441. Although the training accuracy yield of Adam and RMSprop are comparable, it is known that Adam is much faster and therefore is the best optimizer for the above model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dyy7qpHtzYJn"
      },
      "source": [
        "### 3) Compare the Results under Different Epoches"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mhXQERRizZO6"
      },
      "source": [
        "In this task, we hope to compare the results of our model under different training epoches, and answer a question."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9JBBjTGd1zis"
      },
      "source": [
        "- **Subtask 3-1: Completing the Table**\n",
        "\n",
        "We have provided the following table, please write down the **training accuracy** and **validation accuracy** of your model under different epoches.\n",
        "\n",
        "|                    |  10  |  20  |  30  |  40  |  50  |\n",
        "|--------------------|------|------|------|------|------|\n",
        "| Training Accuracy  |0.596 |0.637 |0.641 |0.641 |0.644 |\n",
        "| Validation Accuracy|0.441 |0.451 |0.445 |0.445 |0.447 |\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l82_FjXazdod"
      },
      "source": [
        "- **Subtask 3-2: Answering the Question**\n",
        "\n",
        "Is it always better to train a model for more epoches? How can we decide when should we stop training?\n",
        "\n",
        "*Your Answer:* As long as the validation loss and training loss keeps dropping, running the model for more epochs is better. Once the validation error starts increasing, it is an indication of overfitting the data and we should stop training. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "066JcRvAze7f"
      },
      "source": [
        "### 4) Compare Different Model Capacities/Configurations"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "agxhZpHYzjIR"
      },
      "source": [
        "In practice, we may also vary the capacity of our model to find the optimal choice. In this part, please try different configurations of your model, which have different model capacities. Based on your observation, please also answer a question."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3fp2QC2Pzkja"
      },
      "source": [
        "- **Subtask 4-1: Completing the Table**\n",
        "\n",
        "Please write down the **validation accuracy** of your model under different model capacities (i.e., specified by the word embedding dimension and the hidden layer dimension).\n",
        "\n",
        "|Embedding dim / Hidden dim |  64  |  128  |  256 |\n",
        "|---------------------------|------|-------|------|\n",
        "| 64                        |0.447 |0.450  |0.451 |\n",
        "| 128                       |0.440 |0.449  |0.437 |\n",
        "| 256                       |0.430 |0.431  |0.441 |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mY-OzP6E4JWU"
      },
      "source": [
        "- **Subtask 4-2: Answering the Question**\n",
        "\n",
        "Is it always better to increase model capacities in this case? Is it always better to increase model capacities in general? How to decide a proper model capacity in practice?\n",
        "\n",
        "*Your Answer:* In the above model, validation accuracy decreases with increase in embedding dimension keeping the hidden dimension fixed and increases with increase in hidden dimension keeping the embedding dimension fixed. Therefore, (embedding dim, hidden dim)=(64,256) yields the best validation accuracy for the above deep learning model. In general, increasing dimensionality could mean increasing sparsity resulting in allowing data points to differ and increase validation error. However, it is not applicable to all cases in general. A good model is one which is able to generalize well. In practice, we can decide a proper model capacity by starting with default values and then increasing to realize a pattern until we determine the best capacity for the model.\n"
      ]
    }
  ]
}