lab2/Lab 2.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "This project was developed by Peter Chen, Rocky Duan, Pieter Abbeel for the Berkeley Deep RL Bootcamp, August 2017.\n",
    "Bootcamp website with slides and lecture videos: https://sites.google.com/view/deep-rl-bootcamp/.\n",
    "\n",
    "Code adapted from Stanford CS231N materials: http://cs231n.stanford.edu/\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "# note to properly run this lab, you should execute all code blocks sequentially\n",
    "import chainer\n",
    "import chainer.functions as F\n",
    "import chainer.links as L\n",
    "from chainer import training\n",
    "from chainer.training import extensions\n",
    "\n",
    "from collections import namedtuple, defaultdict, deque\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Chainer\n",
    "\n",
    "Chainer can be understood as Numpy plus the ability to record the computation graph of numerical operations to enable Automatic Differentiation. (Chainer actually also offers many other things; for example, a Numpy equivalent library that runs on GPU, but we will ignore them for now)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's illustrate how Chainer works by a simple 1D regression task.\n",
    "\n",
    "Suppose we have observations from the following model $y = w x + b + \\epsilon$ where $\\epsilon \\sim \\mathcal{N}(0, 0.1)$ and the task is to estimate the linear model parameters $w, b$ from data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAHGVJREFUeJzt3W+MHVd5BvDn8boGr0gIXm9DFNu7ITiohqKAVxZ8KKAmaY0/2JVIQ4JtkirUgigVUkorV5YiFGQ1gEpDS1QwFAFe0xAiVd0KoxRCIiSURN40IcKJTDZu7DiEZlmjSFVM0zhvP8xccvd65s6ZO2dmzsw8P2nke+/Ozp+s8s6557znPTQziIhIt6yo+wJERKR6Cv4iIh2k4C8i0kEK/iIiHaTgLyLSQQr+IiIdpOAvItJBCv4iIh2k4C8i0kEr676ANGvXrrXp6em6L0NEpFEeeeSRX5nZZNZ+wQb/6elpzM/P130ZIiKNQvKEy37q9hER6SAFfxGRDlLwFxHpIAV/EZEOUvAXEekgBX8RkQ5S8BcR6SAFfxGRGh06BExPAytWRP8eOlTNeYOd5CUi0naHDgF79gAvvRS9P3Eieg8AO3eWe261/EVEarJv32uBv+ell6LPy6bgLyIyoIyumKRjnjyZvG/a5z6p20dEpE8ZXTFpx1yzBlhaOnf/DRtGO08eavmLiPQpoysm7ZgAMD6+/PPxcWD//tHP5UrBX0SkTxldMWm/e/o0cOAAMDUFkNG/Bw6UP9gLKPiLiCyT1uXi2hWT1Lc/7Jg7dwLPPAO8+mr0bxWBH1DwFxFZZv/+0btien37J04AZq/17W/bVl/3ThoFfxGRPjt3jt4Vk9a3f/hwfd07aWhm9Z19iJmZGdNKXiLSJCtWRC3+QWTUrVMFko+Y2UzWfmr5i4h4UnS8oEoK/iIi8DOxq8h4QdW8BH+SXyf5AsmfpfycJP+B5ALJx0m+28d5RUR8SBuozfsAKDJeUDVfLf9vANg65OcfBLAx3vYA+CdP5xURySWphe9zYlddqZt5eSnvYGY/Jjk9ZJcdAL5l0ejyQyQvIHmRmT3v4/wiIi7SyiwMBv6eKmrs1KWqPv+LATzb9/5U/JmISGXSWvhjY8n7hzhQ60tQA74k95CcJzm/uLhY9+WISMukteTPnm3OQK0vVQX/5wCs73u/Lv5sGTM7YGYzZjYzOTlZ0aWJSFekteR7A7NNGKj1pargPwfgo3HWz3sAvKj+fhGp2rBUzCIDtXUtxViEr1TPfwHwIIC3kTxF8kaSHyf58XiXwwCOA1gA8FUAN/k4r4hIHmWkYo6SJhrCw0LlHUSkcXrpmSdPRl05vZZ7Haano4A/aGoq+gYxaDDjCIi+ffjqZnIt76DgLyKNUnbwzCtvPZ+8D4u8VNtHRFopLV1z1656ulDy1vOpc93efgr+ItIow4LkqGUZishbzyeU4m8K/iLSKFlBsuh6u3nlHUQOpfibgr+INEpS8Bzk2oXiK+smT5poKMXfvNT2ERGpSi9I7tuXPHAKuHWhpNX56T9HWXburH8CmVr+ItI4vZb27OzoXSg+K3k2kYK/iDRWkS4U16ybECZklUHBX0SC5RJ4Ry3L4JJ142uRlxAp+ItIkMoOvC5ZN23uGlLwF5EgFQ28Wd8aXLqMQpmQVQaVdxCRIOUtm9AvqQQEGR1vasq9FlDZpRjKoPIOItJoRWbCJn1r6D1I8nQfhTIhqwwK/iISpCKBN6tbxrX7KJQJWWVQ8BeRIBUJvC7fDlz77Yss8hIyBX8RCdaogdelBMQohdTalPOv4C8irdP/rQGIvjn0G6Xfvm05/wr+IuJFaK3i3rcGM+DgweL99m3L+fe1hu9WksdILpDcm/DzDSTvJ/koycdJbvNxXhEJg2uruMgDosjv+ui3b13Ov5kV2gCMAXgawFsArALwUwCbBvY5AOAT8etNAJ7JOu7mzZtNRJphasosCvvLt6mp1/aZnTUbH1/+8/Hx6PMsRX7XF5d7HDQ7G/2cjP6t4noBzJtD7PbR8t8CYMHMjpvZywDuArBj8BkD4Pz49RsB/MLDeUWkZr3WeFpp5f5WcZFukxC6XPKmnoY+RuAj+F8M4Nm+96fiz/p9GsAukqcAHAbwFx7OKyI16g9uafozaop0m4TQ5ZI39TSEB9YwVQ34XgfgG2a2DsA2AAdJnnNukntIzpOcX1xcrOjSRGQUScGt32CruMiM3VDWvc0zdhDCA2sYH8H/OQDr+96viz/rdyOAuwHAzB4E8HoAawcPZGYHzGzGzGYmJyc9XJqIlGVYEEtqFReZsdvEMguhPLDS+Aj+RwBsJHkJyVUArgUwN7DPSQBXAADJ30MU/NW0F2mwtCDWK3o22CouMmO3iWUWgn9guYwKZ22IunJ+jijrZ1/82W0AtttrGT4/QZQJ9BiAP8o6prJ9RMIWQgZOnVwyeULO9vES/MvYFPxFhqsjsIR4DXUI+cHnGvw1w1ekgUJJIyyj6FloM4WThJ7J40LBX6SBXINPXTNq8+qdiwR2767/oZYl9EweFwr+Ig3kEnyKfDuo8pvF4HwBG1i9K8QWdeiZPC4U/EUayCX4FPl2UEW3Ru+8u3YNny8AuLeoq/q2EnwmjwuXgYE6Ng34iqRzGXAkl/+8t5HZx0n6vcHf9X39w7Zh9XPy/DfxKdTBbjgO+GoBd5GG6rXQT56MWvyDi5K7LD6ets/YGHD27PDfLWJYPaBB4+NuOf1NXGy9DFrAXaTlsjJtXLom0rpTzp4tt1sjqxunt/hKnslcbRiErZKCv0hLucyKHTZLt8wZtcMGRqemosVXzPKlj7ZhELZKCv4iLZb27aC/FHPaEodlLlye9q1kdnb0c7ViELZCCv4iHZOUWjlKN0sRZdTqaWL9nzppwFekY6oeGM0amBa/XAd8V1ZxMSISjioHRnvfMnp5/L3JYoAeAHVTt49Ix1QxMDpsAleIM3a7SMFfpGPKHhh1Wd5R6Zf1U/AX6ZiyB0azlncElH4ZAvX5i3TQzp3l9blnteqVfhkGtfxFWqbuevhZE7iUfhkGBX+RnOoOrsNUUYo56/7LmMAlJXCp/pa1AdgK4BiABQB7U/a5BsATAI4C+HbWMVXVU0IU8vJ9ZlF1yVGrYrpwvf9QK152Aaqq6klyDNHi7VcBOAXgCIDrzOyJvn02ArgbwB+a2a9J/q6ZvTDsuJrkJSEKvXLkihXnLoYCRAO7r75a/Pih379UW9VzC4AFMztuZi8DuAvAjoF9/hzAnWb2awDICvwioQq9cmTZOfyh37+48xH8LwbwbN/7U/Fn/S4DcBnJn5B8iORWD+cVqVzolSPLzuEP/f7FXVUDvisBbATwAQDXAfgqyQsGdyK5h+Q8yfnFxcWKLk3EXaiVI3uDsLt3A6tXAxMTfnP4XaqASrP4CP7PAVjf935d/Fm/UwDmzOz/zOy/EI0RbBw8kJkdMLMZM5uZnJz0cGkifoVYOXIww2dpCThzJqqJ7yO7JoQqoOKfjwHflYiC+RWIgv4RAB8xs6N9+2xFNAh8Pcm1AB4FcLmZLaUdVwO+Im7KHoTVIG+zVDbga2avALgZwL0AngRwt5kdJXkbye3xbvcCWCL5BID7AfzVsMAvIu7KHoTVIG87eenzN7PDZnaZmV1qZvvjz241s7n4tZnZLWa2ycx+38zu8nFekbbKM5HMdRB21MlpGuRtJ83wFamQSwDOO0vXZRC6yMzfUAe5pSCXmWB1bJrhK23jOjt2lFm6WTNqi8781Yzd5kBVM3zLogFfaRvXgdNhs3QPHhxtScSyZ/5KOKqc4SsiDlwHTtP60tesGb3rRv32MkjBX6QirgE4rY8dGH1JRPXbyyAFf5GKuAbgtIlkp08nH9cl5TLEyWlSL/X5i1To0KHR+uwBTbYSN+rzFwnQzp1RoH711fylF9R1Iz4p+Is0hLpuxCct4C7SIGUuvC7dopa/SE1cyy1UuWZwyOsTi19q+YvUoFduoZe62cvZB5a37F33q/KapB2U7SNSA9fMnSozfJRN1A7K9hEJmOts3yrLKat0c7co+IvUwHW2b5VlGVQColsU/EVq4JqzX2Vuv+YRdIuCv0gNXHP2q8zt1zyCbtGAr4hIi1Q64EtyK8ljJBdI7h2y34dIGsnMCxNpGuXIS5MUzvMnOQbgTgBXATgF4AjJOTN7YmC/8wB8EsDDRc8pEhrlyEvT+Gj5bwGwYGbHzexlAHcB2JGw32cAfBbAbzycUyQIvdb+rl2j19oXqYOP4H8xgGf73p+KP/stku8GsN7MvufhfCJB6F8UPY1y5CVUpWf7kFwB4AsA/tJh3z0k50nOLy4uln1pUpMQ+sZ9XMO+fee29geZqf9fwuQj+D8HYH3f+3XxZz3nAXgHgAdIPgPgPQDmkgZ9zeyAmc2Y2czk5KSHS5PQ9LeW865DW/U1ZD0gXFv1ddyjSJbCqZ4kVwL4OYArEAX9IwA+YmZHU/Z/AMCnzGxoHqdSPdsphPoxLtcwOIALRBOe+vPe046TRjVypAqVpXqa2SsAbgZwL4AnAdxtZkdJ3kZye9HjS7uEUD/G5RqSunQGB3CHLbSe57widfDS529mh83sMjO71Mz2x5/damZzCft+IKvVL+0VQv0Yl2tweUCkzYidmsp3XpE6qLyDVGqU+jG+B4hdrsH1IZW0Jq9q5EgjmFmQ2+bNm03aaXbWbGrKjIz+nZ1N/3x21mx83Cwamo228fHXfsf3NfT/vMh5s44vUhYA8+YQY2sP8mmbgn+3pAXbiYnln/W2qSm/53Z9GImEzjX4q9tHANSfe582wLq0lLz/yZN+rnlY2mdSl06Z6v4bSMe4PCHq2NTyr05ZXSt5kMkt/LRtYsLPNU9N1ffNYnCfuv8G0g5wbPmrpLMEnXs/MQGcOXNuvv3q1cnfCvJe84oVUagdREYt/qJc5gsAYfwNpB20hq84CyH3Pi1D5otfTE6nPH06+Th5r7ns1FOX+QJAGH8D6RYFfwki937YKlJJfe++rrnstEzXoB7C30C6RcFfgslLzzPA6uuaXZcuHHUw1jWoh/I3kA5xGRioY9OAb7WamNZY1TUXGYzN87tN/BtIeKABX5H8Dh2K+uNPnoxa5/v3R++LDMYmHVOre0lZXAd8FfxFYmmZOWk1+31lBIn4pGwfaYSQJjalZeaMjSXvr8FYabLCC7iLjCq0Rc/TMnPOnj33G4AGY6Xp1PKX2rjmwLvw8Q0irSXfX6p5WEaQSJMo+EthowZeXxObfC0NOSzdsuo6PyJlU/CXobICe5HAOywHPs8DxfUbRNYxXXP+RVrBJR+0jk15/vVzyVEvUhgt7fif+ES+vPq0onBkvnsRaQNUWdKZ5FaSx0gukNyb8PNbSD5B8nGS95FMWehOQuLSoi7SdZPW0j58ON9YgMssWp/jCyJtUDj4kxwDcCeADwLYBOA6kpsGdnsUwIyZvRPAPQA+V/S8Uj6XwF60Jk1SX3reB4pLaQQVThNZzkfLfwuABTM7bmYvA7gLwI7+HczsfjPrtbseArDOw3mlZC6BvYyaNHnHAlz66lU4TWQ5H8H/YgDP9r0/FX+W5kYA3/dwXimZS2AvY5A07bzbto2+6pYKp4kMcBkYGLYBuBrA1/re7wbwpZR9dyFq+b8u5ed7AMwDmN+wYUNJwyGSR13FxpLOW3TVLRVOky5AVYXdSL4XwKfN7I/j938TP1T+dmC/KwH8I4D3m9kLWcdVbR8ZVPaqWyJtUGVtnyMANpK8hOQqANcCmBu4mHcB+AqA7S6BXySJ+u1F/Ckc/M3sFQA3A7gXwJMA7jazoyRvI7k93u3zAN4A4LskHyM5l3I4kVTqtxfxx0uev5kdNrPLzOxSM9sff3armc3Fr680swvN7PJ42z78iNIG/Zk5a9dGW5HaO2WvuiXSJSrv4IkCznKDZR+WlqKtSO0dIDurx1edH5G2U/D3IPSA4+vBlKclnzSjtl9Zs2s1k1fEjVby8mB6utgyf2VKW50qby5+0nH6DR4zLTOnXxlZOsoIkq7TSl4VCrl0gK+WcN6WvEsGThlZOsoIEnGj4O9ByAHH14PJZf/+fZIyc/qVlaWjjCARNwr+HoQccHw9mPK25AczcyYmoq3sOvmqyS/iyGUacB1b0+r5V1k6IO1cSZ8Pq2Of55qTjqPa+CLhgWN5h9qDfNrWtOBflVEWQMn7UBh27t5xJiaiTXVyRMLiGvyV7dMwaZlFY2PA2bPnfp6WcRRyhpKIjE7ZPi2VNvCaFPiH7R9yhpKIlE/Bv2HSBl7HxvLtH3KGkk+aeS2STMG/YdIyi/bsyZdxFHKGki+hz7wWqZOCf8MkpVCuXg18+cvRv67plF1IiVSpB5F0GvBtMF+lG9pKpR6kizTg2wFq2Q7XlXENkVEo+DeYMnaG68K4hsioFPwDlpWpopbtcF0Y1xAZlYJ/oFwyVdSyzZa1+ItIV3kJ/iS3kjxGcoHk3oSfv47kd+KfP0xy2sd5myyrVe/Snx9iy1Z59SIN4VIDYtgGYAzA0wDeAmAVgJ8C2DSwz00Avhy/vhbAd7KO2+baPi51dcjkAmpkNdc3SpG6UeoFiYhfcKzt46PlvwXAgpkdN7OXAdwFYMfAPjsAfDN+fQ+AK0jSw7kbyaVVX1d/fpGJUco+EmkOH8H/YgDP9r0/FX+WuI+ZvQLgRQATHs7dKL0ukaSCakD2YihV9OcXCeDKPhJpjqAGfEnuITlPcn5xcbHuy/Gqv0WdZsOG1x4Qu3fnm7HrS5EAruwjkebwEfyfA7C+7/26+LPEfUiuBPBGAEuDBzKzA2Y2Y2Yzk5OTHi4tHFlr4I6PA9u2Le9yWVoCzpwBDh6sLlOlSABX9pFIc/gI/kcAbCR5CclViAZ05wb2mQNwffz6agA/igcmOmNYy7nXqj98uP4+8yIBPMTsIxFJ5qW2D8ltAO5AlPnzdTPbT/I2RKPOcyRfD+AggHcBOA3gWjM7PuyYbavt47J4Sii1aA4dih44J09GLf79+xXARZqi0to+ZnbYzC4zs0vNbH/82a1mNhe//o2Z/amZvdXMtmQF/jZyaVFX0WfukoeviVEi7RfUgG+buXSJDHtA+Jg8pfr2ItKjks6BSepyAfyUbta6vSLt59rto+DfAL6CdihjCiJSHtXzb5CsLh1fk6eUhy8iPQr+NXPph/cVtJWHLyI9Cv41cymn4CtoKw9fRHrU518z13545d6LiAvXPv+VVVyMpNuwIXkwd7BLZ+dOBXsR8UfdPjVTP7yI1EHBv2bqhxeROij451DWEoUqpyAiVVOfv6NeSmYvM6eXkgkoWItI86jl70hLFIpImyj4J0jq3mnaEoVldVGJSDuo22dAWvfOmjXRylqDQiyNoC4qEcmiln+s11LetSu5ewdoTkqmuqhEJIuCP9wWVz99ujkpmU3rohKR6qnbB9mLqwNR905TZtm6zhoWke4q1PInuYbkD0g+Ff/7poR9Lif5IMmjJB8n+eEi5yxDVos41O6dNJo1LCJZinb77AVwn5ltBHBf/H7QSwA+amZvB7AVwB0kLyh4Xq+GtYhD7t5Jo1nDIpKlUFVPkscAfMDMnid5EYAHzOxtGb/zUwBXm9lTw/arsqrnYHYMMNoyiSIidatqJa8Lzez5+PUvAVyYcVFbAKwC8HTB83rRy/DZvRtYvRqYmFBLWUS6ITP4k/whyZ8lbDv697PoK0Tq14j4m8FBAH9mZokrxpLcQ3Ke5Pzi4mLOW8lncAWtpSXgzBng4EH3+jouE6k02UpEgmRmI28AjgG4KH59EYBjKfudD+A/EXX3OB178+bNVqapKbMo7C/fpqbcfn921mx8fPnvjo9Hn+fZR0TEJwDz5hBji/b5fx7AkpndTnIvgDVm9tcD+6wC8H0A/25md7geu+w+f9cVtNJMTyenU05NRd8cXPcREfGpqj7/2wFcRfIpAFfG70FyhuTX4n2uAfA+ADeQfCzeLi943sKKLoruMpFKk61EJFSFgr+ZLZnZFWa20cyuNLPT8efzZvax+PWsmf2OmV3etz3m4+KLKJoL7/LwKPqAEREpS2fLOxTNhXd5eGiylYiEqrPBHyi2gpbLw0OTrUQkVIUGfMtU5SQvEZG2qGrAt/WUpy8ibaSqnkNoURQRaatOtPz7W+9r10abS0tei6KISFu1PvgnlXFYWope91ryaQ+AUfP01VUkIqFrffDPWqhlWEt+lDz9wYdN1gNGRKQOrQ/+LrNp0/YZJU9fXUUi0gStD/4us2nT9hklT18lHUSkCVob/Hv97idORIE7TVZLPu9EMJV0EJEmaGXw7+93B6K+994DYGKi3EVbVNJBRJqglXn+Sf3uZtWUUu49SPbti7p6NmyIAr/mBYhISFpZ3qForX4RkabqdHkH9buLiAzXyuCvfncRkeFaGfxVSllEZLhWDvgCUaBXsBcRSVao5U9yDckfkHwq/vdNQ/Y9n+Qpkl8qck4RESmuaLfPXgD3mdlGAPfF79N8BsCPC55PREQ8KBr8dwD4Zvz6mwD+JGknkpsBXAjgPwqeT0REPCga/C80s+fj179EFOCXIbkCwN8B+FTBc4mIiCeZA74kfwjgzQk/Wlan0syMZNKMsZsAHDazUxxWZCc61x4AewBgg5LyRURKkxn8zezKtJ+R/G+SF5nZ8yQvAvBCwm7vBfAHJG8C8AYAq0j+j5mdMz5gZgcAHACiGb6uNyEiIvkUTfWcA3A9gNvjf/9tcAcz+23CJckbAMwkBX4REalOodo+JCcA3A1gA4ATAK4xs9MkZwB83Mw+NrD/DYiC/80Ox16Mj9lUawH8qu6LqJDut926dL9Nv9cpM5vM2inYwm5NR3LepbhSW+h+261L99uVe21leQcRERlOwV9EpIMU/MtzoO4LqJjut926dL+duFf1+YuIdJBa/iIiHaTg70nXKpy63C/Jy0k+SPIoycdJfriOay2C5FaSx0gukDxnfgrJ15H8Tvzzh0lOV3+Vfjjc6y0kn4j/lveRnKrjOn3Jut++/T5E0uIU9tZQ8PenaxVOXe73JQAfNbO3A9gK4A6SF1R4jYWQHANwJ4APAtgE4DqSmwZ2uxHAr83srQD+HsBnq71KPxzv9VFE83TeCeAeAJ+r9ir9cbxfkDwPwCcBPFztFZZPwd+frlU4zbxfM/u5mT0Vv/4FovIfmZNPArIFwIKZHTezlwHchei++/X/d7gHwBXMKmIVpsx7NbP7zeyl+O1DANZVfI0+ufxtgaih9lkAv6ny4qqg4O9P1yqcZt5vP5JbAKwC8HTZF+bRxQCe7Xt/Kv4scR8zewXAiwAmKrk6v1zutd+NAL5f6hWVK/N+Sb4bwHoz+16VF1aV1i7jWIYqK5yGwMP99o5zEYCDAK43s1f9XqVUjeQuADMA3l/3tZQlbqh9AcANNV9KaRT8c6iywmkIPNwvSJ4P4HsA9pnZQyVdalmeA7C+7/26+LOkfU6RXAngjQCWqrk8r1zuFSSvRPTwf7+Z/W9F11aGrPs9D8A7ADwQN9TeDGCO5HYzm6/sKkukbh9/ehVOgSEVTs1sg5lNI+r6+Vaogd9B5v2SXAXgXxHd5z0VXpsvRwBsJHlJfC/XIrrvfv3/Ha4G8CNr5uSZzHsl+S4AXwGw3cwSH/YNMvR+zexFM1trZtPx/68PIbrvVgR+QMHfp9sBXEXyKQBXxu9Bcobk12q9snK43O81AN4H4AaSj8Xb5fVcbn5xH/7NAO4F8CSAu83sKMnbSG6Pd/tnABMkFwDcguFZXsFyvNfPI/rG+t34bzn4IGwMx/ttNc3wFRHpILX8RUQ6SMFfRKSDFPxFRDpIwV9EpIMU/EVEOkjBX0SkgxT8RUQ6SMFfRKSD/h+AYzc6eImHoQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# first generate some observations\n",
    "true_a = 1.3\n",
    "true_b = 0.4\n",
    "data_x = (np.arange(100) / 99.0 - .5).astype(np.float32) # Chainer assumes all the cpu computation is done in float32\n",
    "data_y = (data_x * true_a + true_b + np.random.randn(*data_x.shape) * 0.1).astype(np.float32)\n",
    "_ = plt.scatter(data_x, data_y, c='b')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Chainer provides an abstraction called `Link` that describe some computation and keeps track of parameters for it. For instance, a `Linear` link describes a linear map on input and keeps track of `w` and bias `b`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w: variable W([[ 0.98645675]])\n",
      "b: variable b([ 0.])\n",
      "(1, 1)\n",
      "Operations on chainer.Variable: variable([[ 123.98645782]]), type: <class 'chainer.variable.Variable'>\n",
      "numpy arrays: [[ 0.98645675]] [[ 123.98645782]]\n"
     ]
    }
   ],
   "source": [
    "model = L.Linear(in_size=1, out_size=1) # input is 1D data and output is also 1D data\n",
    "\n",
    "# Chainer will randomly initialize `w` and `b` for us.\n",
    "# we can take a look at their values\n",
    "print(\"w:\", model.W)\n",
    "print(\"b:\", model.b)\n",
    "\n",
    "print(model.W.shape)\n",
    "\n",
    "# model.W and model.b have type `chainer.Variable`, \n",
    "#   which is a wrapper around Numpy array\n",
    "assert isinstance(model.W, chainer.Variable)\n",
    "\n",
    "# operations that involve `chainer.Variable` will produce\n",
    "# `chainer.Variable` and this records the computation graph\n",
    "var_result = model.W + 123 # some random computation\n",
    "print(\"Operations on chainer.Variable: %s, type: %s\" % (var_result, type(var_result)))\n",
    "\n",
    "# the underlying numpy array can be accessed by `data` attribute\n",
    "print(\"numpy arrays:\", model.W.data, var_result.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEICAYAAAC3Y/QeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xu8XPO9//HXRyIIgkSqIjeOFNGLEinV1qVS4RCtqlviUtK4n6rjFL+c9qCU6qnS6rFF6h51ax3XNkqltK5xPRINkeuORK6UJkGSz++P7+xmMuayZs+atdbMvJ+PxzyyZ2btme/am8+s/fl+vp+vuTsiItJa1kt7ACIikjwFfxGRFqTgLyLSghT8RURakIK/iEgLUvAXEWlBCv7SVMzs92Z2fJnn28zsBxFfa5KZjY5vdNGY2T5m1h7x2AvM7NZ6j0maT9e0ByBSiZnNAka7+yOVjnX3A/O+74Tc930p7/lT6jFGkUajK38RkRak4C8NxcxOMLO/mNl/m9kyM5tpZvlX+5PMbLSZ7QS0AXua2ftm9k7u+RvN7OLc11uY2QNmtij3Wg+YWd+I47jAzO4ys1vN7D0z+z8z+5SZnW9mC81srpl9Le/4PmZ2n5ktNbPpZvadvOc2yo1rmZlNBXYveK8+Zvbb3Dhnmtm/1fRDFEHBXxrTF4BpwJbA5cCvzczyD3D314BTgKfcfRN337zI66wH3AAMAPoDK4CrqxjHIcAtwBbAi8DE3GtuA1wEXJt37O1AO9AHOBz4sZntl3vuv4B/yd0OAP45Z2Fm6wH3Ay/nXverwFlmdkAV4xT5GAV/aUSz3f06d18N3ARsDWxV7Yu4+xJ3/627L3f394BLgL2reIkn3H2iu68C7gJ6A5e5+0eEYD/QzDY3s37AXsC57r7S3V8CxgPH5V7nCOASd1/q7nOBX+S9x+5Ab3e/yN0/dPcZwHXAUdWer0g+TfhKI1rQ8YW7L89d9G9S7YuYWXfg58BwwtU7wKZm1iX3wVLJ23lfrwAW533fity/mxCu9pfmPmA6zAaG5L7uA8wteK7DAKBPR9oqpwvwRITxiZSk4C/NrFLL2n8HdgC+4O4LzGwXQvrGyn9b1d4CeprZpnkfAP2Bebmv5wP9gCl5z3WYC8x090Exj0lanNI+0szeBvqaWbcSz29KuEJ/x8x6EnLvsculcp4ELjWzDc3ss8BJQEd9/p3A+bkJ6L7AmXnf/izwnpmdm5sY7mJmnzazdSaFRaql4C/N7E+Eq+kFZra4yPNXAhsBi4GngT/UcSxHAwMJfwXcA/xX3rqFCwmpnpnAw4RJZAByaaSDgV1yzy8mzBdsVsexSgswbeYiItJ6dOUvItKCYgn+ZnZ9bmHLqyWeH2lmr+QWwjxpZp+L431FRKRz4rryv5FQLlfKTGBvd/8M8CNgXEzvKyIinRBLqae7P25mA8s8/2Te3aeBSEvoRUSkPtKo8z8J+H2xJ8xsDDAGYOONN95txx13THJcIiIN7/nnn1/s7r0rHZdo8DezfQnB/0vFnnf3ceRSQkOGDPHJkycnODoRkcZnZrMrH5Vg8M8tbBkPHOjuS5J6XxER+bhESj3NrD/wO+BYd389ifcUEZHSYrnyN7PfAPsAW+a2n/svYH0Ad28Dfgj0Av4n14RrlbsPKf5qIiJSb3FV+xxd4fnRQOJ7oYqISHFa4Ssi0oIU/EVEWpCCv4hIC1LwFxFJ0YQJMHAgrLde+HfChGTeVzt5iYikZMIEGDMGli8P92fPDvcBRo6s73vryl9EpEA9rsaLvebYsWsDf4fly8Pj9aYrfxGRPPW4Gi/1moWBv8OcOZ17n2royl9EJE89rsZLvWaXLsWP79+/8+8VlYK/iEieUlfdtVyNl/re1auhe/d1H+veHS65pPPvFZWCv4hInlJX3VGvxovl9kt974ABMG5c+Nds7f16T/aCgr+IyDouuaTzV+Mduf3Zs8F9bW7/oINKv+bIkTBrFqxZE/5NIvCDgr+IyDpGjuz81Xip3P5DD6V3hV+KuXt6716GNnMRkUaz3nrhir+QWbiyT4KZPR+la7Ku/EVEiKe2v9b5giQp+ItIyyuVq6/2A6CW+YKkxRL8zex6M1toZq+WeN7M7BdmNt3MXjGzXeN4XxGROMRV21/LfEHS4rryvxEYXub5A4FBudsY4JqY3ldEpCrF0jtx1vanVb1TrViCv7s/Diwtc8ihwM0ePA1sbmZbx/HeIiJRlUrv9OxZ/Pgs5urjklTOfxtgbt799txj6zCzMWY22cwmL1q0KKGhiUirKJXegcbJ1cclUxO+7j7O3Ye4+5DevXunPRwRaTKl0jhLlzZOrj4uSXX1nAf0y7vfN/eYiEhi+vcPqZ5ij48c2dzBvlBSV/73Acflqn72AN519/kJvbeICFC/Usxq1wiktXtXvrhKPX8DPAXsYGbtZnaSmZ1iZqfkDnkImAFMB64DTovjfUWkNXU2eNajFLPaNQJxrSmoldo7iEhDKdwYBcLVe1o5+oEDi6eSBgwIpZ61Hl8ttXcQkaZUqmJn1Kh0UijVrhGox34BnaHgLyINpVyQTCOFUm0/n6z0/1HwF5GGUilIVtOWIY6J12onkbPS/0fBX0QaSrHgWShKCiWuiddqJ5Gz0v9HE74i0nAmTAhX98UmTiHa5Gm9J17ToglfEWlaHc3Tbr218ymUqBOvWajJrwcFfxHJrEqBt5YUSpSJ16zU5NeD0j4ikkn1rueP8vqNmBpS2kdEGlqtG6zE8VdDVmry60FX/iKSSbVshl7sqt4svN6AAWFOIMpfD7ryFxFJWC2LoYr91dDxQVJN3j4rNfn1oOAvIplUS+CtlJaJmj7KSk1+PSj4i0gm1aOSJ1/UvH2j7MlbLQV/EcmszgbeKKuAO9NLp5lq/hX8RaTp5P/VAOEvh3ydyds3W82/gr+IxCLKVXGSV84dfzW4wy231J63r7X0NHPcveYbMByYRtip67wiz/cHHgNeBF4BDqr0mrvttpuLSGO49Vb37t3dQ6gNt+7dw+PVHFPpPQYMcDcL/0b9vriYrTv2jptZ6e9JY8zAZI8St6McVPYFoAvwJrAd0A14GRhccMw44NTc14OBWZVeV8FfJPs6gluxoAjhuQ6ljss/ptz71PLBEYdqx5/WmKMG/zjSPkOB6e4+w90/BG4HDi38AwPokft6M+CtGN5XRFKUnwMvJb+ippbVsllIuVRbepqFMZcTR/DfBpibd78991i+C4BRZtZO2Mz9zGIvZGZjzGyymU1etGhRDEMTkXopFtwK5VfU1LJoKwttFqotPc3CmMtJasL3aOBGd+8LHATcYmYfe293H+fuQ9x9SO/evRMamoh0RqUgVnhVXMuiraxsfVhN6WlWxlxKHMF/HtAv737f3GP5TgLuBHD3p4ANgS1jeG8RSUm5IFbsqriWRVuN2GYh82OOMjFQ7gZ0BWYA27J2wnfngmN+D5yQ+3onQs7fyr2uJnxFsi3pCc20q306M56mrvYJ78VBwOuEqp+xuccuAkb42gqfv+Y+GF4CvlbpNRX8RcrLQjDMwhjSkIXqo1KiBn+1dBZpQPXe6CRNHfvzzpkTUktR2y8nKcutntXSWaSJRS0jrGVFbZKrcTveywyOPTb7LRSyXskThYK/SAOKEnyi9qIpFuST7GNTuF6gMBkR5+5dccl6JU8kUXJDadyU8xcpLcpq0yjHlMpd9+pV+XtrVWl1cNQWCpXOpR55+GbI+ace5EvdFPxFSosSfKL0ookafKsJwp0df7lblA+dWtpHdPYcsjjZHTX4K+0j0oCi1MxHSU1Um6OOK60RZXVwh1p376pXHr7RN3lR8BdpUJWCT7lFRh25cS9R7NerV30XKFUKyB399+PYvauh8vAJUvAXaVKl/jqA8g3ZuneHq66q7961lVYH33JL+GCqdfeuTK2ozRjV+Yu0mFI16hACbxJ19fVap9AIawTqTXX+IlJUqZSLWX1y18XKL2vp81NOo+fhk9Q17QGISLL69y9+5R9nbrzjCnz27BDcOxIMHesFIARmBef06MpfpMXUOzce56ItqR8Ff5EWU6+US4coZZyN1AahWSntI9KC6plyiRLYVX6ZPl35i0isKgV2lV9mg4K/SJWS7HbZGfUeX6XXLzan0JlFW1JnUXpAVLoBw4FpwHTgvBLHHAFMBaYAt1V6TfX2kSzKckMv9/qPL+rrZ7XvTSsgqc1czKwLYRevYUA78BxwtLtPzTtmEGEP3/3cfZmZfcLdF5Z7XS3ykizK8iYeUP/xZf38JdlFXkOB6e4+w90/BG4HDi045jvAr9x9GUClwC+SVVnfxKPe48v6+Ut0cQT/bYC5effbc4/l+xTwKTP7q5k9bWbDY3hfkcRltXlYpUZtcY0vq+cv1UtqwrcrMAjYBzgauM7MNi88yMzGmNlkM5u8aNGihIYmEl0Wm4cVLqoqFMf4Oj5cOlbsxv36krw4gv88oF/e/b65x/K1A/e5+0fuPpMwRzCo8IXcfZy7D3H3Ib17945haCLxqvcCqc4ot6gqjvEVW7Gr6p3GF8eEb1dCMP8qIeg/Bxzj7lPyjhlOmAQ+3sy2BF4EdnH3JaVeVxO+ItGst17xdI9ZaHBWK03yNpbEJnzdfRVwBjAReA24092nmNlFZjYid9hEYImZTQUeA/6jXOAXaWXV1ulHzcN3tv5fk7xNKko9aBo31flLM6pU/96ZOv0o31NL/X/Se+NKbdAG7iLZEiUAdzbQVvpQqSWAZ31hm6wravDXTl4iCYmSO6+Uv+/sTlW1zgtoh6zGETXnr66eIgmJkjsvt9FK4daHhRujlFPrBi7aeKX5qLGbSEKiTMyWW0dQrKQz6sYoWVyfIOlS8BdJSJQAXG4dQS1VN1lcnyDpUs5fJEG15M5Vby9RJNnYTUQiGjkyBOo1a8K/1Vx5K3XTxNxhYbL9LhX8RRqEUjdNaNkyuOoq2HlnGD68dGe+OlDwF0lJlBW3hcdA5/9yiGtMUiN3eOYZ+Pa3oU8fOOss6NED/u3fEg3+qS/mKnXTIi9pZvVelVuvMUkN/v5397Y29112CT/cTTZxP/lk9xdeiPVt0CIvkeyKMnmb9ASvJpTr5KWX4Npr4dZb4f334XOfg1NPhWOOgU03jf3ttMhLJMOilG0m3VBNDdxitHw53HkntLWFFM+GG8JRR8Epp8DQoR/fFCEFyvmLpCDKgq+kd83SLl0xeO21kMPfZpuQ03/3Xfj5z2HePLjhBvjCFzIR+EHBXyQVUco2ky7tVClpJ33wAdx+O+y7LwweDP/zP6Fy57HHYOrU8GHQs2fao/y4KBMDadw04SvNrlInzqjHJD0myXnzTfdzz3Xv3TtM4G67rfull7q//Xaqw0ITviIiMVu1Ch54IOTyJ06ELl3g4IPDBO6wYaFGNmWJrvA1s+FmNs3MppvZeWWO+6aZuZlVHJhIo1GNfBNrb4cLLgilT9/4BkyZAhdeGMqj/vd/4YADMhH4q1FztY+ZdQF+BQwjbNT+nJnd5+5TC47bFPgu8Eyt7ymSNbW0W5aMWr0aHn44XOU/8EBY+jB8OFxzDRx0EHRt7GLJOD6qhgLT3X2Gu38I3A4cWuS4HwE/AVbG8J4imdBxtT9qVOfbLUvGvP02XHopbL99CPJPPQXf/z5Mnw4PPQQjRjR84Id4gv82wNy8++25x/7JzHYF+rn7g+VeyMzGmNlkM5u8aNGiGIYmWZSF9EgcY+i42i+2MKrD7NlKATUEd5g0KdTi9+sH/+//wbbbhiqe9vbwYbDddmmPMl5RZoXL3YDDgfF5948Frs67vx4wCRiYuz8JGFLpdVXt05yy0EIg6hg6uy9usZvaJGTUkiXuV1zhvsMO4Re1xRbuZ53l/tpraY+s00hqA3dgT2Bi3v3zgfPz7m8GLAZm5W4rgbcqfQAo+DenWjYST3IMUT4gzKIH/6TPUcpYs8b9ySfdjzvOfYMNwi9njz3cb7rJffnytEdXs6jBv+ZSTzPrCrwOfBWYBzwHHOPuU0ocPwk4x93L1nGq1LM51bqReFJjqKX3TilJnqMU8fe/h/xbWxu88kroqzNqFJx8cui30yQSK/V091XAGcBE4DXgTnefYmYXmdmIWl9fmksWWghEGUOUPjelVsT26lXd+0qdvfBCCPB9+sBpp4Xa/HHj4K23wmrcJgr8VYny50EaN6V9mlNncv5xrzqNMoao6aliY8vCvEbLe/9991//2n333cMvYKON3E880f2ZZ0Lap4mRVM6/XjcF/+ZVKpgnGUgrfaDU+r5qk5CSV191P/NM9802C7+0nXZyv+oq92XL0h5ZYhT8paGUCra9epW+Ao8rwFbzYSQZtHKl+223uX/5y+E/jm7d3I8+2v3xx5v+Kr8YBX+pStqBrpqyyfwPh1r/IshKiibtn39DeuMN9//4D/cttwy/uH/5F/fLL3dfuDDtkaVKwV8iy0IArLZsskuX4o9XW06ZROlpvVNMLeXDD93vvtt92LC1/yEcdpj7ww+7r16d9ugyIWrwV1dPycT2faXG0KsXrFixbuuE7t0/3kqhQ7XllPUuPS3s+QNh/OPGre35k4Wff+bNmQPjx4fb/PlhFe53vgMnnRSqeOSfEu3qKY0tC9v3lSqbvOqqECgHDAgBecCAtfeLqbacst6lp2PHVu75k4WffyatXr22l86228LFF8Ouu8J998HMmfCDHyjw1yLKnwdp3JT2SU4WVt26V5f3jitVElerh1JKpbPM1h6TlZ9/Zsyf737xxe79+4cfxFZbuZ9/vvvMmWmPrCGgnL9E1ag553pU+/TqFW5xlZvG1Uqi6a1e7f7II+6HH+7etWv4Iey3n/tdd7l/8EHao2soCv5SFVWbdK7ctLOvGddfFg1v8WL3//5v90GDwg+nZ0/3f/9392nT0h5Zw1Lwl4aQpaBXbblpfuqmnCydYyasWeP+l7+4jxq1trHaXnu533KL+4oVaY+u4UUN/o2/I4E0rKztflXtBGvUSeGRI7WbFwDvvgu33hoaq736KvToAaNHh747n/lM2qNrOar2kdREqYRJUqlg3qtX8UqkSy6p/5iawvPPh7LMPn3gjDNgww3huutCY7Wrr1bgT4mCv6QmzhLHOHbmqrbcVFfzZfzjH6Emf/fdYcgQuO02OPpoeO65cBs9GjbeOO1RtrYouaE0bsr5Z0OUfHVnc9pxlTjGOamq/HyNXnnF/fTT3Xv0CL+InXd2v/pq93feSXtkLQNN+EqtogTVWsoUy31vNUFY5ZQpW7EiTNbutVf4wW6wQZjMfeKJlmysljYFf6lZlKBa69V7HG2ctZAqJa+/HsoyO2phBw0KZZuLFqU9spYWNfjH0tvHzIYDVwFdCJu5X1bw/NnAaGAVsAg40d3LboCn3j7pi9L3ph69cartdRPl+CxsH9kUPvoI7r03VOw8+ih07Qpf/zqceirsu2/4gUqqEuvtY2ZdgF8BBwKDgaPNbHDBYS8SNmz/LHA3cHmt7yv1F6XvTT1645SbCC42sVtqoja/GicL20c2tFmzQhlWv37wrW/BG2+EXjtz58Jdd8F++ynwN5g4qn2GAtPdfYa7fwjcDhyaf4C7P+buHUV9TwN9Y3hfqbMoQTXKMdUqFZB79gzrAGbPDlfx+esCKlXj1GOcTW/1anjgATj4YNhuO7jsMhg6FB58EGbMCB8Gn/xk2qOUzoqSGyp3Aw4npHo67h8LXF3m+KuB/yzx3BhgMjC5f//+8SfDpGppVMjUo81CPcbZtObNc7/oIvd+/cIPeOut3X/wA/fZs9MemURAUjl/MzscGO7uo3P3jwW+4O5nFDl2FHAGsLe7f1DudZXzb20TJoQLyzlzwl8Cl1wCxx6rvH3drFkTcvhtbSGnv3o1DBsWVt+OGAHrr5/2CCWiqDn/ONo7zAP65d3vm3uscED7A2OJEPhFirVEGDu2+MSu8vY1WLQIbrwx5MmmTw/Lmb/3vRD0t98+7dFJHcWR838OGGRm25pZN+Ao4L78A8zs88C1wAh3XxjDe0oDiGPVbT7l7WPiDk88ET5d+/aF738ftt46/ILmzYOf/lSBvxVEyQ1VugEHAa8DbwJjc49dRAj2AI8AbwMv5W73VXpN1fk3tmJ5+456/Lh672uVbpWWLXP/xS/cBw8Ov4gePdzPPNP91VfTHpnECC3ySlaWA049Nj3p7Krbeq+u1UreAmvWuD/7rPuJJ7pvtFH4gQwd6v7rX7u//37ao5M6UPBPUJYDTj23Oyx3JV9q1W29V9dqJW/Oe++5X3ut+667hh/Axhu7jxnj/vzzaY9M6ixq8I9lhW89NFK1T7UrUpMU19hKvU6H7t3Xra2vdDzUp0qn5VfyvvJKqNi59VZ4773QLvmUU2DUqNA/X5peYit8Jd7WxHGLa2yVji/sw19scrZQPap0WnIl74oVcPPN8MUvwuc+B9dfH1ouPPkkvPwynHaaAr98jIJ/DLIccOIaW5Tj8z8gRo5cu+oWPr7yv15VOi1VETRtGpx9NmyzDRx/PCxZAldcETZJuflm2HNPtVyQ0qLkhtK4Kedf/v2KTbxW2yGzmgncYq9TTV49yQnxLE++1+yDD9zvuMN9333DD3399d2PPNL9scfUPlncPXrOP/UgX+rWSMHfPbmAUyqYn3pqdUG+Mx9YHa+TP9mbtQnupjVjhvv557t/4hPhBz5woPull7ovWJD2yCRjogZ/Tfg2mFITqV26hBX5hWppg1xOsfYL2tYwZqtWhSZqbW0wcWJI4RxySFh9e8ABYXZbpEDUCV8F/wZTqpqllFJVLq1SFdOQH1Lz5oX9b8ePh/b2sPH56NFhE/S+aogr5SXZ20cS1L9/dVf+5SZ8m71PzoQJoeXz8lwz8fwW0Jn7AFizBv74x3CVf//94Zf5ta/BL38ZWip31f+qEi/93dhgSlWzjBlTXZVLK1TFjB27NvB3KCxJTd3ChfCTn4ReOsOHw1//CuecE5qsTZwYSjYV+KUeokwMpHFrtAnfJOVP4PbqFW6FX0eZdG7qqhiPtrdvKtascZ80yf2oo0K1Drjvs4/7b37jvnJlyoOTRocmfJtfYVoDPr7StpVlbuX1smWh/r6tDf72N9h8czjhhPBL3GmnFAYkzUgrfFtAQ6Q1UpSJ1JY7PPMMfPvbYeL2rLNgs83CKtx58+DnP1fgl1Qo+GdYpX74WW4rkQX5q4xL7e1bN++9B9deC7vuCnvsAXffHVbhvvgiPP10+DCo1P9CpI40k5RRUSpVWqFip1bFdgSrq5deCmmdCRPg/fdhl13C/WOOgU03TXAgIuXFcuVvZsPNbJqZTTez84o8v4GZ3ZF7/hkzGxjH+zaySlf1UVI69Upr1LIDV9y7dzWE5cvDVoh77AGf/zzcdBMcfni4wn/hhbAoS4FfsibKrHC5G9CFsIPXdkA34GVgcMExpwFtua+PAu6o9LrNXO0TpbVC1EqVuCt2aulTlOV9Depi6lT3737XffPNw8nuuKP7lVe6L12a9sikhZFUbx9gT2Bi3v3zgfMLjpkI7Jn7uiuwmNzq4lK3Zg7+UTYcSWtTklretyU2Ulm50v2229z33tv/2VjtqKPUWE0yI2rwjyPtsw0wN+9+e+6xose4+yrgXaBX4QuZ2Rgzm2xmkxctWhTD0LKlIyVSapOTOXPWPSapNsiFY6jm8bi+N/PefBPOPRf69Qv5+7lz4bLLQvuF3/wG9tlH7ZOloWRqwtfdxwHjINT5pzycWBWryS/Us+e6x7iHeOIeKlWS6EtTyyRy001Ar1oVWi20tcHDD4ceGoccAqeeCvvvr8Zq0tDi+K93HtAv737f3GNFjzGzrsBmwJIY3rthFJvAzdcxcVt4TEfgnzUrmaqVWiaRM1FXH4f2drjggvCDP+wwmDoVLrwwfLLdc0/ouaPAL40uSm6o3I3w18MMYFvWTvjuXHDM6aw74Xtnpddttpx/uQ3NOyZqs9KOoJZJ5IZtGbFqlftDD7mPGOG+3nrhBA480P3ee90/+ijt0YlERpKbuQAHAa8Tqn7G5h67CBiR+3pD4C5gOvAssF2l12y24J+VSd6GDc71smCB+49/HDZHgbBZyvnnh81TRBpQosG/HrdmC/5RyiDj2nKxljG0hDVr3P/0J/cjjnDv2jX8IPbd1/3228M2iSINTME/g6IE8Li2XCymJUoxy1myxP2KK9w/9alw4lts4f6977n/7W9pj0wkNlGDv7p6NoC4ulO2yu5d63APK23b2uCOO+CDD2DPPeGUU+Bb34KNNkp7hCKxUlfPBpJUA7dyu3o1nb//Ha65JvTW+eIXQ5XOSSfByy/Dk0/Ccccp8EtLU/BPWUf9/+zZ4SK1o4Fb/gdAXEG7aUoxy3nhhfAD7NMHTjst7II1bhy89Rb86lfw2c+mPUKRTFDwT1mSDdxSbXFcT//4R+iPP3Qo7LYb3HorHHUUPPssPP982Ph8k03SHqVIpijnn7KoefgJE8IHwpw54Yo/idW+mTdlSuiZf/PN8O67MHhw6KB53HFhlyyRFhQ155+p9g6tKGpLhMT70mfVBx/Ab38bJnCfeAK6dQvtk089FfbaS/11RCJS2idlLZGHj8P06fD970PfvuFT8K234Kc/DVshTpgAX/qSAr9IFRT8qxT3ZiVNm4ePw0cfhav8YcNg0CC44grYe2/44x/h9dfhnHNgyy3THqVIQ1LOvwrFOnN2765gHbs5c+C662D8eFiwILRRHjMGTjwxVPGISElRc/4K/lWIa7GVFLF6NfzhDyGX/9BDYRb8oIPCBO5BB4V2yiJSkRZ51ahYeqeRNitpmL10588PExzbbQcHHwyTJ8P558PMmfDAA6F/vgK/SOxU7ZOno5yyYxetjj+KOhZe9ewJS4rsQpC1FbKF6amO8UNG0lNr1sCf/hSu8u+9N2ya8tWvhpz+iBGw/vppj1Ck6enKPyd/pS18vPa+I5A2QmVOlIVjqViyBH72M9hxxzCJO2kSnHVWmLx95BH45jcV+EUSouCfU2mnLYClSxujMidT6Sl3+MtfYNSoMFl7zjmw1VZhFW57eyjXHDQohYGJtDalfXKiBMb+/RtjsVUm9tJ991245ZYDciR4AAALlUlEQVSQ2pkyBXr0CG0WTj4ZPvOZBAciIsXUdOVvZj3N7I9m9kbu3y2KHLOLmT1lZlPM7BUzO7KW96yXSoExi+mdUlJdODZ5cgjyffrAmWeGzpnXXRcWZV19tQK/SEbUmvY5D3jU3QcBj+buF1oOHOfuOwPDgSvNLHONV4oFzI4Fo1lN75SS+MKxf/wj1OQPGQK77w633QbHHAPPPRduo0fDxhvX6c1FpDNqqvM3s2nAPu4+38y2Bia5+w4Vvudl4HB3f6PccUnV+ec3TOvZMzy2dKmap0Xyf/8XGqvdckvon//pT4dNUkaNgs02S3t0Ii0pqTr/rdx9fu7rBcBWFQY1FOhG2Oi92PNjzGyymU1etGhRjUOrrLCX/pIlsGJFiGWzZlUO/FFq6Rum3j6qlSvDZO2XvhR6448fD4ceCn/9K7zyCpx+ugK/SCOotM8j8AjwapHbocA7BccuK/M6WwPTgD2i7C+ZxB6+texpW+uG7A1n2jT3s89279kznMigQe4/+5n74sVpj0xE8pDEHr5R0z5m1gOYBPzY3e+O8tpJpH1q2dM2SquHhm8H8eGHYRFWW1tYlNW1K3zjG6FiZ7/91EVTJIOSSvvcBxyf+/p44N4iA+kG3APcHDXwJ6WW7RGj1NJnqt6+GrNmhYmQ/v3hiCPgzTfh4oth7ly4886wGleBX6Sh1Rr8LwOGmdkbwP65+5jZEDMbnzvmCOArwAlm9lLutkuN7xuLWkoio3xwNNSG6atXw/33w7/+a+izc9lloXLnwQdD8B87Fj75ybRHKSJxiZIbSuOWRM7fPeTfBwxwNwv/Rs3HN03Of94894sucu/XLwxw663df/hD9zlz0h6ZiHQCEXP+qQf5Urekgn8tonxwdPbDpa5Wr3afONH9sMPcu3QJ/xl87Wvuv/ud+4cfpj06EalB1OCvfv4RNM3m6YsWwQ03hNr8GTPCLlgnnhhW5G6/fdqjE5EYaAP3mGS+PXIl7mGj82uvhbvvDhU8X/kK/OhHoYvmBhukPUIRSUHLdPXMX2y15ZbhFmXhVWbbI1eybBn84hew885h39sHHwyfWq++Cn/+c2i/oMAv0rJaIvgXW8m7ZEn4uuNKvtQHQGfKNVNb1esOzz4bUjnbbAPf/S5ssglcf31orPbLX4YPAxFpeS2R8y+12CpfqYVX1S7USmWT9/ffD83U2trgxRdDE7WRI8NirF13rdObikgWaQ/fPFEWVZU6ptq1AImmiV5+GU47LbRPPvnkUKt/zTXhKv/aaxX4RaSklpjwLbW5SeExxXRcrUet9qn7qt4VK+Cuu8JV/lNPhbz9kUeGbpp77KGVtyISSVMH/1IbsheqtKq3mt276raL1rRp4Wr+xhvDZO4OO4QNz48/fm0vahGRiJo2+Bfm3t3XfgD06hUeq0ff/ksuKZ7z79QuWh9+CPfcE67yJ00Km5sfdli4yt97b13li0inNW3wL5Z7d69/R81q00RFzZwZZoivvx4WLgyzzpdeCt/+dtj8XESkRk1b7VNLu+ZUrFoVavHb2mDixDDQQw4JV/nDhkGXLmmPUEQaQMuv8K1b7j1u8+aF3bDGj4f29lCf/8Mfhn1v+/ZNe3Qi0qSaNvjHmnuP25o18PDDYQL3/vvD/QMOCIuwDj44bJoiIlJHTRtlYsm9x23hwrWN1WbOhN694ZxzwqfUdtulODARaTU1BX8z6wncAQwEZgFHuPuyEsf2AKYC/+vuZ9TyvlFVU6JZN+7w+OMhl//b38JHH4VKnUsvha9/Xf11RCQVta7wPQ941N0HAY/m7pfyI+DxGt+vcSxdCldeCYMHwz77wB/+AKefDlOnhrLNI49U4BeR1NSa9jkU2Cf39U2ETdrPLTzIzHYDtgL+AFSchW5Y7vD00yGtc8cdsHJlWHV7ww1hL9zCPhEiIimpNfhv5e7zc18vIAT4dZjZesDPgFGEfX5LMrMxwBiA/pkryynjvffCqrK2ttBvZ5NN4IQTQr+dXTKxXbGIyDoqBn8zewQotnP3Oq3K3N3NrNiigdOAh9y93SqsSHX3ccA4CHX+lcaWuhdfDFf5EyaEzpq77BI+AI45BjbdNO3RiYiUVDH4u3vJq3Uze9vMtnb3+Wa2NbCwyGF7Al82s9OATYBuZva+u5ebH8iu5cvhzjtDkH/mGdhww5C/P/VUGDpULRdEpCHUmva5DzgeuCz3772FB7j7P+ttzOwEYEhDBv7XXgtX+TfdBO+8AzvtBFddBcceC1tskfboRESqUmvwvwy408xOAmYDRwCY2RDgFHcfXePrp+uDD+B3vwtB/89/hm7dwr63J58c9sHVVb6INKim7e1TkzffXNtYbfHisADr5JNDY7XevdMZk4hIBC3f26dqH30EDzwQcvkPPxwaqR1ySMjl779/6BQnItIkFPznzl3bWO2tt0IztQsvhJNOCk3WRESaUGsG/9Wrw9V9W1u42neHAw8M9w88UI3VRKTptVaUW7Ag5PHHjQv9nj/xCTjvPPjOd8KGKSIiLaL5g787PPZYuKq/556wacp++8Hll4fGat26pT1CEZHENXfwnzoVvvENeP31UIt/5pmhameHHdIemYhIqpo7+G+7bSjT/M//hMMPh402SntEIiKZ0NzBf6ON4Pe/T3sUIiKZo+J1EZEWpOAvItKCFPxFRFqQgr+ISAtS8BcRaUEK/iIiLUjBX0SkBSn4i4i0oMxu5mJmiwi7gzWqLYHFaQ8iQTrf5tZK59vo5zrA3SvuOpXZ4N/ozGxylN10moXOt7m10vm2yrkq7SMi0oIU/EVEWpCCf/2MS3sACdP5NrdWOt+WOFfl/EVEWpCu/EVEWpCCv4hIC1Lwj5GZ9TSzP5rZG7l/tyhzbA8zazezq5McY1yinKuZ7WJmT5nZFDN7xcyOTGOstTCz4WY2zcymm9l5RZ7fwMzuyD3/jJkNTH6U8Yhwrmeb2dTc7/JRMxuQxjjjUul88477ppm5mTVV+aeCf7zOAx5190HAo7n7pfwIeDyRUdVHlHNdDhzn7jsDw4ErzWzzBMdYEzPrAvwKOBAYDBxtZoMLDjsJWObu2wM/B36S7CjjEfFcXwSGuPtngbuBy5MdZXwini9mtinwXeCZZEdYfwr+8ToUuCn39U3A14sdZGa7AVsBDyc0rnqoeK7u/rq7v5H7+i1gIVBx5WGGDAWmu/sMd/8QuJ1w3vnyfw53A181M0twjHGpeK7u/pi7L8/dfRrom/AY4xTldwvhIu0nwMokB5cEBf94beXu83NfLyAE+HWY2XrAz4BzkhxYHVQ813xmNhToBrxZ74HFaBtgbt799txjRY9x91XAu0CvREYXryjnmu8koJE3yK54vma2K9DP3R9McmBJae4N3OvAzB4BPlnkqbH5d9zdzaxYHe1pwEPu3p71C8QYzrXjdbYGbgGOd/c18Y5SkmZmo4AhwN5pj6VechdpVwAnpDyUulHwr5K771/qOTN728y2dvf5uYC3sMhhewJfNrPTgE2Abmb2vruXmx9IRQznipn1AB4Exrr703Uaar3MA/rl3e+be6zYMe1m1hXYDFiSzPBiFeVcMbP9CR/+e7v7BwmNrR4qne+mwKeBSbmLtE8C95nZCHefnNgo60hpn3jdBxyf+/p44N7CA9x9pLv3d/eBhNTPzVkM/BFUPFcz6wbcQzjHuxMcW1yeAwaZ2ba5czmKcN758n8OhwN/8sZcOVnxXM3s88C1wAh3L/ph30DKnq+7v+vuW7r7wNz/q08TzrspAj8o+MftMmCYmb0B7J+7j5kNMbPxqY4sflHO9QjgK8AJZvZS7rZLOsOtXi6HfwYwEXgNuNPdp5jZRWY2InfYr4FeZjYdOJvyFV6ZFfFcf0r4a/Wu3O+y8IOwYUQ836am9g4iIi1IV/4iIi1IwV9EpAUp+IuItCAFfxGRFqTgLyLSghT8RURakIK/iEgL+v8IEgy3NeKlCgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# A chainer link is a callable object. calling it performs the\n",
    "#   forward computation. (in this case, it performs Wx + b)\n",
    "model_y = model(data_x[:, None]) # chainer's link usually assumes input is [Batch Size, Input Dimension]\n",
    "# `model_y` is a chainer variable so we use `.data` to access its numpy array for plotting\n",
    "\n",
    "# we can plot the model's current fit in red. it should be terrible because we haven't trained it yet\n",
    "_ = plt.plot(data_x, model_y.data[:,0], c='r')\n",
    "_ = plt.scatter(data_x, data_y, c='b')\n",
    "_ = plt.title(\"Initial model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss variable(0.17373330891132355)\n",
      "dloss/dW [[-0.05188771]]\n"
     ]
    }
   ],
   "source": [
    "# now let's walk through how to perform forward computation\n",
    "# and use AD to get gradients\n",
    "\n",
    "# first we clear the gradients that are stored in the model\n",
    "model.cleargrads()\n",
    "# as we have seen we can perform forward computation by calling the link\n",
    "model_y = model(data_x[:, None])\n",
    "\n",
    "# remember that `model_y` is a chainer variable. to operate on chainer variable\n",
    "# we will use functions from chainer.functions to operate on those objects.\n",
    "loss = F.mean(F.square(model_y - data_y[:, None]))\n",
    "# `loss` is a scalar chainer variable\n",
    "assert isinstance(loss, chainer.Variable)\n",
    "print(\"loss\", loss)\n",
    "# calculating gradients d loss /d params is as simple as\n",
    "loss.backward()\n",
    "\n",
    "# we can inspect the gradient of loss with respect to W\n",
    "print(\"dloss/dW\", model.W.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know how to calculate gradients, we can code up a simple loop to perform gradient descent to train this model:\n",
    "\n",
    "(Hint: if you run into weird problems, maybe the state has been messed up and you can try re-runing all the code blocks from the beginning)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Itr 0 loss: variable(0.18160659074783325)\n",
      "Itr 25 loss: variable(0.1658174842596054)\n",
      "Itr 50 loss: variable(0.16581565141677856)\n",
      "Itr 75 loss: variable(0.16581538319587708)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEICAYAAABfz4NwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XuUXnV97/H3Z+6ZhEwGGBLIhQQIYgoKOImoNUjUY6ItYE+wRI41Ksajsry0atPVHqqssuqt6rEHPUQB5dRwEUFTDQa0FKsiZgjhkkR0CCGZEGDAhJDrZGa+54+9J+wMzzPzzC1PMvvzWutZPPv++2Wzns/s32/v/VNEYGZmVlHuApiZ2ZHBgWBmZoADwczMUg4EMzMDHAhmZpZyIJiZGeBAMOuTpDslvbfc5TA7HOTnEOxIJGkTcHlE/KzcZTHLC18hWG5Jqip3GYZqNNTBjhwOBDvqSPozSWsl7ZD0a0mvyixbKulxSS9KWi/pnZlliyX9StJXJT0PfDad90tJX5a0XdITkhZktvlPSZdntu9r3RmSfpEe+2eSrpH0b33U46K0HjvTMs9P52+S9JbMep/t2Y+k6ZJC0gckbQb+I23WuqLXvh+S9Bfp9zMk3S3pj5Iek/Suwf/r22jmQLCjiqRzgOuBDwHHAdcCKyTVpqs8DrwRaAA+B/ybpBMzu3gtsBGYCFydmfcYcDzwReA6SSpShL7WXQ78Ni3XZ4H39FGPOcCNwKeBCcBcYFN/9c84H3gl8DbgJmBRZt+zgJOBn0gaC9ydlu0E4FLgG+k6ZodwINjRZglwbUTcHxFdEfFdYD9wHkBEfD8inoqI7oi4BfgDMCez/VMR8a8R0RkRe9N5T0bEtyKiC/gucCJJYBRScF1J04DZwJUR0RERvwRW9FGPDwDXR8TdaVm3RsTvBvDv8NmI2J3W4Q7gbEknp8suA26PiP3AnwGbIuKGtM4PAj8ALhnAsSwnHAh2tDkZ+Ju0uWiHpB3AVOAkAEl/lWlO2gGcSfLXfI8tBfb5dM+XiNiTfh1X5PjF1j0J+GNmXrFj9ZhKcjUzWAf3HREvAj8h+esfkquF76XfTwZe2+vf6zJg0hCObaOUO6TsaLMFuDoiru69IP0L+VvAm4H7IqJL0log2/wzUrfVbQOOlVSfCYWpfay/BTi1yLLdQH1mutCPd+963AT8o6RfAHXAPZnj3BsRb+2r8GbgKwQ7slVLqst8qkh+8P+npNcqMVbSOyQdA4wl+aFsB5D0PpIrhBEXEU8CLSQd1TWSXgf8eR+bXAe8T9KbJVVImizpjHTZWuBSSdWSmoGFJRRhJcnVwFXALRHRnc7/MXC6pPek+6uWNFvSKwdTTxvdHAh2JFsJ7M18PhsRLcAHgf8DbAdagcUAEbEe+BfgPuAZ4CzgV4exvJcBrwOeB/4JuIWkf+NlIuK3wPuArwIvAPeS/KAD/C+Sq4ftJB3jy/s7cNpfcDvwluz6aXPSfyNpTnqKpMnrC0Btgd1YzvnBNLMRIukW4HcR8Y/lLotZKXyFYDZM0qaYU9MmoPnARcAPy10us1K5U9ls+EwiabY5DmgDPpze5ml2VHCTkZmZAW4yMjOz1FHVZHT88cfH9OnTy10MM7OjygMPPPBcRDT1t95RFQjTp0+npaWl3MUwMzuqSHqylPVKajKSND99S2KrpKUFls+VtEZSp6SFvZZ9UdI6SRskfT19mKhe0k8k/S5d9vnSqmVmZiOl30CQVAlcAywAZgGLCrwpcTPJw0HLe237euANwKtInhidTfKWRoAvR8QZwDnAG7KvETYzs8OvlCajOUBrRGwEkHQzyf3V63tWiIhN6bLuXtsGyXtVakjeJ1MNPJO+6+WedNsOSWuAKUOqiZmZDUkpTUaTOfStjW3pvH5FxH0kP/zb0s+qiNiQXUfSBJJ3vvy80D4kLZHUIqmlvb29lMOamdkgjOhtp5JOIxnEYwpJiMyT9MbM8iqStzR+vecKpLeIWBYRzRHR3NTUbye5mZkNUimBsJVDX+M7JZ1XincCv4mIXRGxC7iT5OVfPZYBf4iIr5W4PzMzGyGlBMJqYGY6XmwNyVsT+xoJKmszcL6kKknVJB3KGwAk/RPJMIefGHixzcxsuPUbCBHRCVwBrCL5Mb81ItZJukrShXDwpV5tJMPyXStpXbr5bSSjQj0CPAQ8FBH/LmkK8Pckdy2tSUe4uny4K9fjjgfb+LfflHQbrplZbpX0YFpErCR5N3123pWZ76spcJdQOu7shwrMb+PQUaxG1I8f2sbTO/fxP847uf+VzcxyKhfvMhpbW8Xu/Z3lLoaZ2REtN4Gwa39XuYthZnZEy0UgjKut9BWCmVk/chEIY2ur2Hugi65uj/1gZlZMPgKhJuk739PhqwQzs2LyEQi1SSDsdj+CmVlROQmESgB2uR/BzKyoXATCuINXCA4EM7NichEIYx0IZmb9ykcgpJ3Kuzvch2BmVkw+AiHtQ/AVgplZcbkIhJ4+BHcqm5kVl4tAcB+CmVn/chEIY6rdZGRm1p9cBEJFhRhbU+lOZTOzPuQiEMCvwDYz609JgSBpvqTHJLVKWlpg+VxJayR1SlrYa9kXJa2TtEHS1yUpnf8aSY+k+zw4f6SMq61yp7KZWR/6DQRJlcA1wAKSIS8XSZrVa7XNwGJgea9tXw+8AXgVcCYwm2RcZYBvAh8EZqaf+YOtRCl8hWBm1rdSrhDmAK0RsTEiOoCbgYuyK0TEpoh4GOjutW0AdUANUAtUA89IOhEYHxG/iYgAbgQuHlpV+lZfU+mX25mZ9aGUQJgMbMlMt6Xz+hUR9wH3ANvSz6qI2JBu31bKPiUtkdQiqaW9vb2UwxY0rraK3X79tZlZUSPaqSzpNOCVwBSSH/x5kt44kH1ExLKIaI6I5qampkGXxU1GZmZ9KyUQtgJTM9NT0nmleCfwm4jYFRG7gDuB16XbTxnkPgfF4yqbmfWtlEBYDcyUNENSDXApsKLE/W8GzpdUJamapEN5Q0RsA3ZKOi+9u+ivgB8Novwl87jKZmZ96zcQIqITuAJYBWwAbo2IdZKuknQhgKTZktqAS4BrJa1LN78NeBx4BHgIeCgi/j1d9hHg20Brus6dw1etl6uv8bjKZmZ9qSplpYhYCazsNe/KzPfVHNoE1DO/C/hQkX22kNyKelj0vOBuT0cnx9RVH67DmpkdNXL1pDJ4XGUzs2JyFAgeV9nMrC/5CYQavwLbzKwv+QkEj4lgZtan3ARCT6eyX4FtZlZYbgLB4yqbmfUtN4HgcZXNzPqWm0Codx+CmVmf8hMIHlfZzKxPuQkEj6tsZta33AQC+BXYZmZ9yVUgeFxlM7PichUI9X4FtplZUbkKhLE1VX65nZlZEbkKBI+rbGZWXK4CwZ3KZmbFlRQIkuZLekxSq6SlBZbPlbRGUqekhZn5F0ham/nsk3RxuuzN6TZrJf1S0mnDV63CxtZWelxlM7Mi+g0ESZXANcACYBawSNKsXqttBhYDy7MzI+KeiDg7Is4G5gF7gLvSxd8ELkuXLQf+YQj1KEnSh+ArBDOzQkoZQnMO0BoRGwEk3QxcBKzvWSEiNqXLuvvYz0LgzojY07MZMD793gA8NaCSD8LY2pfGVa6s0EgfzszsqFJKIEwGtmSm24DXDuJYlwJfyUxfDqyUtBfYCZxXaCNJS4AlANOmTRvEYV/icZXNzIo7LJ3Kkk4EzgJWZWZ/Enh7REwBbuDQsDgoIpZFRHNENDc1NQ2pHB5X2cysuFICYSswNTM9JZ03EO8C7oiIAwCSmoBXR8T96fJbgNcPcJ8D5nGVzcyKKyUQVgMzJc2QVEPS9LNigMdZBNyUmd4ONEg6PZ1+K7BhgPscMI+rbGZWXL99CBHRKekKkuaeSuD6iFgn6SqgJSJWSJoN3AE0An8u6XMR8ScAkqaTXGHc22ufHwR+kHZEbwfeP7xVezmPq2xmVlwpncpExEpgZa95V2a+ryZpSiq07SaSjune8+8gCZHDxuMqm5kVl7MnlT1IjplZMTkLBI+rbGZWTC4DwVcIZmYvl6tA8LjKZmbF5SoQesZV9gvuzMxeLleBAFBfW8Uej4lgZvYyuQsEj6tsZlZY7gJhrMdVNjMrKH+B4HGVzcwKyl0guMnIzKyw3AVCfW0Vu92pbGb2MrkLhIYxVbyw90C5i2FmdsTJXSA01tfwwt4DdHdHuYtiZnZEyV0gTKivIQJ27vNVgplZVv4CYUwylvKOPQ4EM7Os3AVC49gkELbv6ShzSczMjiwlBYKk+ZIek9QqaWmB5XMlrZHUKWlhZv4FktZmPvskXZwuk6SrJf1e0gZJHxu+ahXXMKYGgB3uWDYzO0S/I6ZJqgSuIRn3uA1YLWlFRKzPrLYZWAx8KrttRNwDnJ3u51igFbgrXbyYZGjNMyKiW9IJQ6pJiRrre5qMfIVgZpZVyhCac4DWiNgIIOlm4CLgYCCkw2SSjo9czELgzojYk05/GHh3RHSn+3h2wKUfhAn1yRXC9t2+QjAzyyqlyWgysCUz3UaBMZJLcClwU2b6VOAvJbVIulPSzEIbSVqSrtPS3t4+iMMeqmFMNZKbjMzMejssncqSTgTOAlZlZtcC+yKiGfgWcH2hbSNiWUQ0R0RzU1PTkMtSWSHG11W7ycjMrJdSAmErSVt/jynpvIF4F3BHRGT/LG8Dbk+/3wG8aoD7HLQJ9dW+7dTMrJdSAmE1MFPSDEk1JE0/KwZ4nEUc2lwE8EPggvT7+cDvB7jPQZtQX+PbTs3Meuk3ECKiE7iCpLlnA3BrRKyTdJWkCwEkzZbUBlwCXCtpXc/2kqaTXGHc22vXnwf+u6RHgH8GLh96dUozYUy132dkZtZLKXcZERErgZW95l2Z+b6apCmp0LabKNAJHRE7gHcMoKzDprG+mo3P7SrHoc3Mjli5e1IZkiYj9yGYmR0qp4FQzYv7Ouns6uuxCTOzfMllIDTW+/UVZma95TIQJtT7jadmZr3lNBDSKwTfempmdlAuA6HRVwhmZi+Ty0CYkL4C2w+nmZm9JJ+BkA6S44fTzMxekstAOKa2isoK+QrBzCwjl4EgiQljqtnuPgQzs4NyGQiQ3Hr6ggPBzOygHAeC33hqZpaV20Bo9JgIZmaHyG0gNIyp8YNpZmYZuQ2Exvpqv8vIzCwjt4Ewob6aPR1d7O/sKndRzMyOCCUFgqT5kh6T1CppaYHlcyWtkdQpaWFm/gWS1mY++yRd3Gvbr0s67KPV9LzPyHcamZkl+g0ESZXANcACYBawSNKsXqttBhYDy7MzI+KeiDg7Is4G5gF7gLsy+24GGodSgcHqeQW2n0UwM0uUcoUwB2iNiI0R0QHcDFyUXSEiNkXEw0BfI84sBO6MiD1wMGi+BHxmUCUfop5XYPvWUzOzRCmBMBnYkpluo8AYySW4FLgpM30FsCIitvW1kaQlkloktbS3tw/isIV5TAQzs0Mdlk5lSScCZwGr0umTgEuAf+1v24hYFhHNEdHc1NQ0bGXymAhmZocqJRC2AlMz01PSeQPxLuCOiOj5c/wc4DSgVdImoF5S6wD3OSQHx0TwradmZgBUlbDOamCmpBkkQXAp8O4BHmcR8Hc9ExHxE2BSz7SkXRFx2gD3OSRjqiupqapwH4KZWarfK4SI6CRp718FbABujYh1kq6SdCGApNmS2kiaga6VtK5ne0nTSa4w7h3+4g9ezxtPfdupmVmilCsEImIlsLLXvCsz31eTNCUV2nYT/XRCR8S4Usox3Br9gjszs4Ny+6QyQEO9x0QwM+uR60Bo9JgIZmYH5TwQ3GRkZtYj14Fw7Nga/ri7g+7uKHdRzMzKLteBMKmhjs7u4PndvkowM8t1IEwcXwfAMzv3lbkkZmbll+tAmJQGwtMvOBDMzPIdCA1pIPgKwcws34Fw/LhaKuQmIzMzyHkgVFaIpmNq3WRkZkbOAwGSfgQ3GZmZORCYOL7OTUZmZjgQmNRQ5yYjMzMcCEwcX8fOfZ3s7egqd1HMzMoq94Fw8FkENxuZWc45EBr8cJqZGZQYCJLmS3pMUqukpQWWz5W0RlKnpIWZ+RdIWpv57JN0cbrse+k+H5V0vaTq4atW6fz6CjOzRL+BIKkSuAZYAMwCFkma1Wu1zcBiYHl2ZkTcExFnR8TZwDxgD3BXuvh7wBnAWcAY4PLBV2Pw/LSymVmilCE05wCtEbERQNLNwEXA+p4V0mEykdTdx34WAndGxJ50m4NDckr6LUWG4Bxp42qrGFdb5SYjM8u9UpqMJgNbMtNt9DNGchGXAjf1npk2Fb0H+GmhjSQtkdQiqaW9vX0Qh+3fxPG1bjIys9w7LJ3Kkk4kaRpaVWDxN4BfRMR/Fdo2IpZFRHNENDc1NY1I+SY1+GllM7NSAmErMDUzPSWdNxDvAu6IiEMGMJb0j0AT8NcD3N+wmji+jmfcZGRmOVdKIKwGZkqaIamGpOlnxQCPs4hezUWSLgfeBiyKiL76HkbcpPF1PPvifg+laWa51m8gREQncAVJc88G4NaIWCfpKkkXAkiaLakNuAS4VtK6nu0lTSe5wri3167/LzARuC+9JfXKYajPoPQMpfnc7v3lKoKZWdmVcpdRzx1BK3vNuzLzfTVF7hJK70B6WSd0RJR07MPh4LMIL+znhGPqylwaM7PyyP2TyuDXV5iZgQMB8MNpZmbgQACSoTQrK+Q7jcws1xwIpENpjqv1FYKZ5ZoDITWxwSOnmVm+ORBSk8bX+n1GZpZrDoTUpPF+fYWZ5ZsDIXXihDG8uK+TnfsO9L+ymdko5EBIzTh+LAAb23eXuSRmZuXhQEid2tQTCLvKXBIzs/JwIKSmHTuWygr5CsHMcsuBkKqpqmDasfVsfM5XCGaWTw6EjFOOH+srBDPLLQdCxilNY3niud10eVwEM8shB0LGKU3j2N/ZzVM79pa7KGZmh50DIeOU9NbTx32nkZnlUEmBIGm+pMcktUpaWmD5XElrJHVKWpiZf0E6GlrPZ5+ki9NlMyTdn+7zlnR4zrI6pWkc4GcRzCyf+g0ESZXANcACYBawSNKsXqttBhYDy7MzI+KeiDg7Is4G5gF7gLvSxV8AvhoRpwHbgQ8MoR7D4vhxNRxTV+U7jcwsl0q5QpgDtEbExojoAG4GLsquEBGbIuJhoLuP/SwE7oyIPZJEEhC3pcu+C1w84NIPM0mc0jTOVwhmlkulBMJkYEtmuo0CYySX4FLgpvT7ccCOiOgc4j6H3alNvvXUzPLpsHQqSzoROAtYNYhtl0hqkdTS3t4+/IXr5dSmcTy9cx+793f2v7KZ2ShSSiBsBaZmpqek8wbiXcAdEdHzKtHngQmSqvrbZ0Qsi4jmiGhuamoa4GEHrudOoyee81WCmeVLKYGwGpiZ3hVUQ9L0s2KAx1nES81FREQA95D0KwC8F/jRAPc5InruNPKtp2aWN/0GQtrOfwVJc88G4NaIWCfpKkkXAkiaLakNuAS4VtK6nu0lTSe5wri3167/FvhrSa0kfQrXDb06Q3fycfVI8Lj7EcwsZ6r6XwUiYiWwste8KzPfV5M0+xTadhMFOowjYiPJHUxHlLrqSqY0jvFrsM0sd/ykcgGnHO9bT80sfxwIBfS85K7bL7kzsxxxIBQw84Rj2Hugi7btfsmdmeWHA6GAV09tAODBLdvLXBIzs8PHgVDAKyYeQ31NJWuedCCYWX44EAqoqqzgVVMaWLN5R7mLYmZ22DgQijh3WiMbtu1kb0dXuYtiZnZYOBCKOHdaI53dwcNtvkows3xwIBRxzrQJAG42MrPccCAUcdy4WqYfV8+Dm92xbGb54EDow7nTGlmzeQfJu/jMzEY3B0Ifzjm5ked27fcDamaWCw6EPpx7sB/BzUZmNvo5EPrgB9TMLE8cCH3wA2pmlicOhH74ATUzy4uSAkHSfEmPSWqVtLTA8rmS1kjqlLSw17Jpku6StEHS+nQENSS9Od1mraRfSjptOCo03GbPOJbO7uA3Tzxf7qKYmY2ofgNBUiVwDbAAmAUskjSr12qbgcXA8gK7uBH4UkS8kmSEtGfT+d8ELouIs9Pt/mEwFRhprz/1OMbVVrHq0afLXRQzsxFVyhXCHKA1IjZGRAdwM3BRdoWI2BQRDwPd2flpcFRFxN3persiYk/PZsD49HsD8NTgqzFyaqsqmXfGCdy1/hk6u7r738DM7ChVSiBMBrZkptsoMEZyEacDOyTdLulBSV9KrzgALgdWSmoD3gN8vtRCH27zz5zEH3d3sHqT7zYys9FrpDuVq4A3Ap8CZgOnkDQtAXwSeHtETAFuAL5SaAeSlkhqkdTS3t4+wsUt7E2vaKKuuoKfPrqtLMc3MzscSgmErcDUzPSUdF4p2oC1aXNTJ/BD4FxJTcCrI+L+dL1bgNcX2kFELIuI5ohobmpqKvGww6u+porzT29i1bpnPM6ymY1apQTCamCmpBmSaoBLgRUl7n81MCENAIB5wHpgO9Ag6fR0/luBDaUX+/Cbf+Yknt65j4f8OmwzG6X6DYT0L/srgFUkP9q3RsQ6SVdJuhBA0uy0L+AS4FpJ69Jtu0iai34u6RFAwLfSfX4Q+IGkh0j6ED49/NUbPvPOmEh1pfip7zYys1FKR9ObPJubm6OlpaVsx3/v9b9l0/O7+c9PvQlJZSuHmdlASHogIpr7W89PKg/AgjMn8eTze3hk6wvlLoqZ2bBzIAzAgrNOZFxtFd/+ryfKXRQzs2HnQBiAhjHVvPu10/jxw0+x+fk9/W9gZnYUcSAM0Af+dAZVFRUs+6/Hy10UM7Nh5UAYoInj6/iLcyfz/ZY22l/cX+7imJkNGwfCICyZewodXd1859fuSzCz0cOBMAinNI1jwZmTuPG+J3lx34FyF8fMbFg4EAbpw+efxov7OvmXu35f7qKYmQ0LB8IgnTWlgcWvn853fr2JXz/+XLmLY2Y2ZA6EIfjb+Wcw4/ixfPr7D7Nrf2e5i2NmNiQOhCEYU1PJly95Ndte2MvVP1lf7uKYmQ2JA2GIXnNyI0vmnspNv93Cvz90RA76ZmZWEgfCMPjkW2cye3ojn7hlrQfRMbOjlgNhGNRWVXLD++bw6ikNXLH8Qe5a51dkm9nRx4EwTMbVVvGd98/hTyY38NHla7jtgTaOpleLm5k5EIbR+Lpqbnz/HM6Z1sinvv8QH12+hu27O8pdLDOzkpQUCJLmS3pMUqukpQWWz5W0RlKnpIW9lk2TdJekDZLWS5qezpekqyX9Pl32seGoULk1jKnmpg+ex2fmv4K71z/D2772C360ditdHovZzI5w/QaCpErgGmABMAtYJGlWr9U2A4uB5QV2cSPwpYh4JTAHeDadvxiYCpyRLrt5EOU/IlVWiI+86TTu+MgbOHZsDR+/eS1v/cq93NqyhY7O7nIXz8ysoFKuEOYArRGxMSI6SH64L8quEBGbIuJh4JBfuzQ4qiLi7nS9XRHRM5DAh4GrIqI7XfYso8yZkxv4ycfeyDcuO5e66ko+c9vDzL76Z3zq+w9xz++eZX9nV7mLaGZ2UFUJ60wGtmSm24DXlrj/04Edkm4HZgA/A5ZGRBdwKvCXkt4JtAMfi4g/lFzyo0RlhXj7WSey4MxJ/OIPz/GjB7ey6tGnue2BNmqqKnjV5AZec3IjZ01p4LQTxjH9uLHUVVeWu9hmlkOlBMJQ9/9G4BySZqVbSJqKrgNqgX0R0SzpL4Dr03UPIWkJsARg2rRpI1zckSOJ809v4vzTm9jf2cWvWp/jvsef54Ent3PDrzbR0dWdrgcnNYxhUkMdkxrqOOGYWhrra2isr6ahvoZxtZWMralibG0VddWV1FVXUFddSXVlBTWVFVRXisoKIanMNTazo00pgbCVpK2/x5R0XinagLURsRFA0g+B80gCoQ24PV3vDuCGQjuIiGXAMoDm5uZR0TNbW1XJvDMmMu+MiQDsO9DFxvbdtLbvovXZXbT9cQ/bXtjHhqd2cu+L+wf1nqSqClFVKSolKiqSkKiQqFASTgIqJCQQ6TwlgQQgRDZTesdLNnCKRs8IZFI5Y84ha+V0/XtnM+24+hE9RimBsBqYKWkGSRBcCry7xP2vBiZIaoqIdmAe0JIu+yFwAfAEcD6Q2/dI11VXMuuk8cw6aXzB5R2d3ezY28HOvQfYtb+LXfs62d3Ryb4DXew/0M2+zi46Ors50BUc6Oqms6ubzu6gszvoSj/dEURAd7z0vWc6IJ1O8jaZfil7e6dw9vGKYgk9Es9glPWvgVHxp8iRLQhUIPIHOn8493UklammauSfEug3ECKiU9IVwCqgErg+ItZJugpoiYgVkmaT/JXfCPy5pM9FxJ9ERJekTwE/V/Ln1QPAt9Jdfx74nqRPAruAy4e/eqNDTVUFJxxTxwnH1JW7KGY2iuloepq2ubk5Wlpa+l/RzMwOkvRARDT3t56fVDYzM8CBYGZmKQeCmZkBDgQzM0s5EMzMDHAgmJlZyoFgZmbAUfYcgqR24MlBbn488NwwFudokcd657HOkM96u86lOTkimvpb6agKhKGQ1FLKgxmjTR7rncc6Qz7r7ToPLzcZmZkZ4EAwM7NUngJhWbkLUCZ5rHce6wz5rLfrPIxy04dgZmZ9y9MVgpmZ9cGBYGZmQE4CQdJ8SY9JapW0tNzlGQmSpkq6R9J6SeskfTydf6ykuyX9If1vY7nLOtwkVUp6UNKP0+kZku5Pz/ctkmrKXcbhJmmCpNsk/U7SBkmvG+3nWtIn0/+3H5V0k6S60XiuJV0v6VlJj2bmFTy3Snw9rf/Dks4dyrFHfSBIqgSuARYAs4BFkmaVt1QjohP4m4iYRTJu9UfTei4Ffh4RM4Gfp9OjzceBDZnpLwBfjYjTgO3AB8pSqpH1v4GfRsQZwKtJ6j9qz7WkycDHgOaIOJNk9MZLGZ3n+jvA/F7zip3bBcDM9LME+OZQDjzqAwGYA7RGxMaI6ABuBi4qc5mGXURsi4g16fcXSX4gJpPU9bvpat8FLi5PCUeGpCnAO4Bvp9MiGbv7tnSV0VjnBmAucB1ARHRExA5G+bkmGfJ3jKQqoB7Yxig81xHxC+CPvWYXO7cXATdG4jckY9ifONhj5yEQJgNbMtNt6bxRS9J04BzZ7tA1AAACBklEQVTgfmBiRGxLFz0NTCxTsUbK14DPAN3p9HHAjojoTKdH4/meAbQDN6RNZd+WNJZRfK4jYivwZWAzSRC8QDJG+2g/1z2Kndth/X3LQyDkiqRxwA+AT0TEzuyySO4xHjX3GUv6M+DZiHig3GU5zKqAc4FvRsQ5wG56NQ+NwnPdSPLX8AzgJGAsL29WyYWRPLd5CIStwNTM9JR03qgjqZokDL4XEbens5/puYRM//tsuco3At4AXChpE0lT4DyStvUJabMCjM7z3Qa0RcT96fRtJAExms/1W4AnIqI9Ig4At5Oc/9F+rnsUO7fD+vuWh0BYDcxM70aoIemIWlHmMg27tO38OmBDRHwls2gF8N70+3uBHx3uso2UiPi7iJgSEdNJzut/RMRlwD3AwnS1UVVngIh4Gtgi6RXprDcD6xnF55qkqeg8SfXp/+s9dR7V5zqj2LldAfxVerfRecALmaalgYuIUf8B3g78Hngc+Ptyl2eE6vinJJeRDwNr08/bSdrUfw78AfgZcGy5yzpC9X8T8OP0+ynAb4FW4PtAbbnLNwL1PRtoSc/3D4HG0X6ugc8BvwMeBf4fUDsazzVwE0k/yQGSq8EPFDu3gEjuonwceITkLqxBH9uvrjAzMyAfTUZmZlYCB4KZmQEOBDMzSzkQzMwMcCCYmVnKgWBmZoADwczMUv8f1EJBHjSIRf8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEICAYAAABbOlNNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xe8HHW9//HXh4QEIwJpYgTSaBKKlBhFroAUCSAQMT9aqEoRBBEQDAS4XiBIKIYmQkSkhS4Xg4ihy/VS5GACoSahBBJaDIQWSEjO5/fHd85lz8nu2dmzszOzu+/n4zGP7M7OznxmDsxn51vN3REREWmzQtYBiIhIvigxiIhIO0oMIiLSjhKDiIi0o8QgIiLtKDGIiEg7SgySW2bWzcw+MrOBNdj3oWb2UNL7jXnsuWa2bYzt1jGzku3JzWyYmT1lZh+a2VFmdqWZnZJosNKUumcdgDQOM/uo4G0vYDGwLHp/hLtPrmR/7r4MWDmh8BrRL4F73P3Ejh+Y2Q7Ale4+OPWopO4pMUhi3P3/buJm9ipwqLvfV2p7M+vu7kvTiK1BDQIezDoIaTwqSpLUmNlZZnazmd1oZh8C+5vZlmb2mJktNLM3zexiM1sx2r67mbmZDY7eXx99fndUfPKomQ0p2P8wM7vPzN41sxfM7IcFn/U3s7+Y2Qdm9hgwhBLainDM7OCo2OddMzvMzL5pZjOiWC8q2H4FMzvdzOaY2TtmdrWZrVLw+cHRZ/82s7EdjrWCmZ1iZi9Fn99kZr1jXMuHge8Al0fFbUOj6/MrM1sVuBMYGH32kZl9Obr+N0bbfWhmz5jZ5gX7XNPM/tvM5pvZK2b204LPvmVm/4qu39tmdl60vpeZ3WBmC6Lr8k8z61cufsk3JQZJ2w+AG4BVgZuBpcCxQD9gK2AkcEQn398POA3oA7wGnAlgZisD9wLXAl8GxgCTzGz96Hu/Az4EvgIcDvwoRqzDgbWB/YGLgbHAdsBGhKS2VbTdodE220bb9wYuiuLaGLg0insN4KtRDG2OA3YFtgbWBD6KjtUpd98aeBT4ibuv7O4vF3z2PrAb8Fr02cru/k708SjgOmA14O62Y5nZCsBfgCeiOHcETjSz7aPvXQKc5+6rAOsAt0XrDyEUG64J9AWOAj4tF7/kmxKDpO0f7n6nu7e6+yfu/oS7P+7uS6Ob2yRgm06+f5u7t7j7Z8BkYNNo/R7ATHe/NtrXk8AdwOjoCWQUcJq7L3L3pwk3x3LOdPfF7v5XYAlwvbvPd/e5wD+AzaLtxgDnu/sr7v4hcAqwX3Sz/X/AHe7+v+6+OPrMCo7xE+AUd5/n7p8C/wX8v+i7tfB3d58a1d9cx+fXb0tgFXc/292XuPts4A/APtHnnwHrmllfd//Q3R8vWN8PWMfdl0V/m8K6JqlDSgySttcL35jZ18zsLjN7y8w+AM4g3GhKeavg9SI+r5weBGwVFWcsNLOFwN7AAGB1oFuHY88pF6i7v13w9hOg4/u2Y3+1w/7mAD2A/tFn/3fc6Kb5bsG2A4E7C2KeEa3/crn4uqjj9fti9HoQoeip8PqdxOdPN4cAw4AXo+KiXaL1VwP3AbeY2TwzO8fMVHdZ55QYJG0dm19eATxD+MW5CnA67X9Rx/U6cL+7r1awrOzuRxNu6K3AWgXbJ9kE9g3CjbVw30uA+cCbhceNirz6FGw7F9ixQ9wruXvhDbwrKh02+XVgVoc4vuTuuwG4+4vuvg8hYV0A/MnMVoqeLn7l7hsA/0EoKhxTZeySMSUGydqXgPeBj81sAzqvX+jMFGBDM9vPzFaMlhFmtn5U7HQH8F9m9gUz2wg4IJnwAbgRON7MBpvZl4DxwI3u3grcCuwRVbL3BM6i/U37cuBsi/pqRJXEuycQ09tAvyieOB4FlpjZCWa2koU+JBub2RZRXAeYWb/onN6PzqHVzLYzs42ioq8PCEVLrQnELxlSYpCsnQAcRKgYvoJQIV2xqMJ1J0Il8JuEIpNfAz2jTY4kVAq/TSg7/2NVUbf3e0Lc/wO8TDiXY6O4no5e3wLMi+IqfBr4DfA34H4LLbUeAb5RbUDu/gzwJ+DVqGio06KpqNnwLsAI4FXg34S/R1vrql2A56MYzwf2dvclhKKy2wlJ4VlCsdIN1cYv2TJN1CMiIoX0xCAiIu0oMYiISDtKDCIi0o4Sg4iItFOXHVH69evngwcPzjoMEZG68uSTT/7b3fuX264uE8PgwYNpaWnJOgwRkbpiZmV7/IOKkkREpAMlBhERaUeJQURE2lFiEBGRdpQYRESkHSUGERFpR4lBRETaUWIQEcmhyZNh8GBYYYXw7+TJ6R27Lju4iYg0ssmT4fDDYdGi8H7OnPAeYEwK8+PpiUFEJGfGjfs8KbRZtCisT4MSg4hITLUo3im2z9deK75tqfVJU1GSiEgMtSjeKbXPPn1gwYLltx84sGvHqZSeGEREYqhF8U6pfQL06tV+fa9eMH58149VCSUGEZEYqi3eqaTI6N13YdIkGDQIzMK/kyalU/EMYO6ezpESNHz4cNew2yKSpsGDQ1FPR4MGwauvdv7djkVGEJ4AvvCF4kVGcfbZFWb2pLsPL7ednhhERGIYP77rxTt5LTIqRYlBRCSGMWO6XryT1yKjUtQqSUSkE5Mnh1/8r70WWgWNH1/5jXvgwOLFUAMHhn1lnQg60hODiEgJbXUDc+aA++fNSSvtv1BNMVQWEkkMZnaVmb1jZs+U+NzM7GIzm21mT5vZ5gWfHWRms6LloCTiERGpVLFWQ0k1Ua2mGCoLibRKMrOtgY+Aa919oyKf7wIcA+wCfBO4yN2/aWZ9gBZgOODAk8AW7v5eZ8dTqyQRSVKpVkMdk0IbM2htTSe2JKXaKsndHwbe7WSTPQhJw939MWA1MxsA7ATc6+7vRsngXmBkEjGJiMRV6smgW7fi26fVAzkradUxrAG8XvB+brSu1PrlmNnhZtZiZi3z58+vWaAi0nxKtRpatqy+6gaSUjeVz+4+yd2Hu/vw/v37Zx2OiDSQUk8AbXUBXa0byHJOhWqklRjmAWsVvF8zWldqvYhIajprNTRmTOiF3Noa/q0kKVTaoikviSStxDAFODBqnfQt4H13fxOYCnzPzHqbWW/ge9E6EZGKdfXGWotWQ5W2aEqqaWwSkmqVdCOwLdAPeBv4T2BFAHe/3MwMuJRQsbwIOMTdW6Lv/gg4JdrVeHf/Y7njqVWSiHRUrGWRWbjJDhrUtY5p1VhhhXDsjkq1aKpmLKa44rZK0iB6ItIQSt1Y2/TqlW7fgUpv9JUmkq7QIHoi0lTKDX+d5tSYUHlv51IV4Fk0jVViEJGGEOcGWs3cCZWqtN4iT8NmKDGISEModmPtKE7ySLISuJIWTXkaNkOJQUQaQuGNFcLNtVC1cyd0LIaqRdPSrjaNTZoSg4jUnVI35bYbqztcd12ycycUrs9T09JaUKskEakrpQa8S6rYJU5rojSaltaCWiWJSEOqdijswqeNfv3CUvjkEacSOM5TRT1TYhCRulLNTbljEdCCBWEpLA6C8pXAeWpaWgtKDCJSV6q5KRd72ijU9uRRrhI4T01La0GJQUTqSjU35ThPFXG2yVPT0lpQYhCRulLNTTnOU0Xc4qDCp4rx48OTRtajoiZFiUFE6k5X2/uX6wTXleKgRmy6qsQgIjUVpyNYWvMQdHza6Ns3LNUUB1XbSiqX3L3uli222MJFJP+uv969Vy/38Fs6LL16hfWVbFPuGIMGuZuFf+N+Lylm7WNvW8xKfyermIEWj3GPzfwm35VFiUEk39pufMVumBA+a1Nqu8JtOjtONUklCZXGn2XMcRODej6LSKKK9UzuqHCOgWrmIchDD+RKe2JnGbN6PotIJsr1FYD2LX+q6ZeQhx7IlbaSykPM5SgxiEiiyt3gOrb8qaZfQl56IFfSSiovMXdGiUFEEtXZDa7Yr+lq+iXksQdyuRZWeYx5OXEqIsotwEjgRWA2MLbI5xOB6dEyE1hY8Nmygs+mxDmeKp9FSsu6lU7alatZn2/HWOKce8O3SgK6AS8BQ4EewFPAsE62Pwa4quD9R5UeU4lBpLg8tNJpiyPpG1+eEkAp1bSwSkPcxJBEUdIIYLa7v+zuS4CbgD062X5f4MYEjisiHaQx+1ic7yY1E1nbsczggAPy37u4HiqW40giMawBvF7wfm60bjlmNggYAjxQsHolM2sxs8fMbFSpg5jZ4dF2LfPnz08gbJHGU+vZx9Ic/qHwWLB8k9Y89i6uh4rlONKufN4HuM3dlxWsG+ShXe1+wIVmtnaxL7r7JHcf7u7D+/fvn0asInUnzo2pmqeKNIZ/aDvu/vuXb/Ya95d4WkNu1EXFchxxyps6W4AtgakF708GTi6x7TTg253s62pgdLljqo5BpLg4dQxxhnAotZ9SPZk7G/6h2vg7W/LYOzrPdSGkWPncHXiZUETUVvm8YZHtvga8SjTPdLSuN9Azet0PmEUnFddtixKDSGnlbkxxKkhLbdOtW9dv0HF0NoxGxyXuzT3vFcJpipsYqi5KcvelwNHAVOB54BZ3f9bMzjCz3Qs23Qe4KQquzQZAi5k9BTwInOPuz1Ubk0gzq2b2sbYil2JDNgAsW1bbopJyRUNm4d9K+jo0SoVwquJkj7wtemIQqU6xp4o4xTht29aqqKTcwHtdOZaeGD6HBtETkUp09qQAnQ8Ml5RKB6TLap/1SoPoiUhFOitaqcWcxsVaCtViLuVGn5+5FvTEICJAOsNBtzV5nTMn3KQLbz/N+is+TXpiEJGK1LoNfj12WGtWSgwiAtS+yCXOPA1qKdTBwoUwcSIsXZrqYbunejQRybUxY2pXlBPnpl9vQ0fUzPz5cOGFcOml8MEHsMkmsP32qR1eTwwiCUpr6IWuqHVs5fZf7qZfl0NHJO3NN+EXvwgX8Ne/hp12gmnTUk0KgPoxiCQlL0NeZxFbnP0X26ZteI68DR2Rujlz3H/6U/eePd1XWMF9zBj3Z59N/DCkNSRGFosSg+RRnjtS1Tq2uPvP8zhCmZg1y/3HP3bv3t19xRXdDz3Uffbsmh0ubmJQc1WRhKywwvItbSBU5La2ph9PoVrHludzz6XnnoOzz4Ybb4QePeDQQ+HEE2teyaLmqiIpy+NY/G3l/qV+/1UbW63333CmTYPRo2HDDeGOO+D44+GVV+CSS3J1sZQYRBKSt7H4O/Yb6Kja2Gq9/4by6KOw666w+eZw331w2mnhwp13HnzlK1lHt7w45U15W1THIHmVpzL0WgxIl+b+615rq/sDD7hvt124KH37uo8f775wYWYhkeKczyISSWqu41IqaXJaqt+AWfvYutqMNe7+m447/O1v8J3vwHbbhfqE888PF+WUU2DVVbOOsCwlBpGcKHeDrnS+5Th1HtXM4ZzHOpVMtbaGeoNvfAN23jlkzksvDXUIJ5wAK6+cdYTxxXmsyNuioiRpNHH6AVTa5LQW+6x0/01h6VL3G29032ijcBHWXtv9D39wX7w468iWg/oxiNSPODfozuZqLlW3Ua7OI878z53JU51K6pYscf/jH93XWy9ctGHDwgX47LOsIyspbmJQPwaRHIjTD6DUsNh9+8Inn3RtIpo0htpuOJ9+CldfDeecEy7eZpvBqafCqFHhD5lj6scgUkfilNeXag4Ly49aGncI67w1sc21jz8OI52uvTYceSQMGAB33QVPPgl77pn7pFCJRM7EzEaa2YtmNtvMxhb5/GAzm29m06Pl0ILPDjKzWdFyUBLxiNSbODfoUsNiv/tu8X3GGc1Us5vF8MEHYUC7wYNDh7T11w99ER55BHbZJVy4RhOnvKmzBegGvAQMBXoATwHDOmxzMHBpke/2AV6O/u0dve5d7piqY5BG1NXy+jyP0VTXFixwP/1099VWCxd05Ej3f/wj66iqQor9GEYAs939ZXdfAtwE7BHzuzsB97r7u+7+HnAvMDKBmETqTlf7QKg4KGFvvw2//GV4fDrjDPjud6GlBe6+G7baKuvoUpFEYlgDeL3g/dxoXUc/NLOnzew2M1urwu9iZoebWYuZtcyfPz+BsEUag4qDEjJ3Lhx7bCgyOv982G03ePppuP122GKLrKNLVVq1JXcCg919E8JTwTWV7sDdJ7n7cHcf3r9//8QDFMmTOL2RC7cZNy48IeShx3XdeeUVOOIIGDoULrsM9t0Xnn8ebrgBNt446+gykcTUnvOAtQrerxmt+z/uvqDg7ZXAuQXf3bbDdx9KICaRutXWG7mtpVFbb2RoP4xFuW3SjKcuvfBCqFSePBm6dYPDDoOTTgqPXM0uTkVEZwshubwMDOHzyucNO2wzoOD1D4DH/PPK51cIFc+9o9d9yh1Tlc/SyOJUJqdZ4dxwldvTp7vvtVeo5e/Vy/34493nzcs6qlQQs/K56icGd19qZkcDUwktlK5y92fN7IwoiCnAz8xsd2Ap8C6hlRLu/q6ZnQk8Ee3uDHcv0fhOpDmUamZauD7ONmnGUxf++U846yy480740pdg7Fg47jhQ0fRy1PNZJGfi9EZOs8dy3feOfvjhUAFzzz3Quzf8/OdwzDHhdZNRz2eROhWn+WmaTVTrsjmsO9x7L2yzTVimT4cJE0KGO/30pkwKFYlT3pS3RXUM0ujidHZLcwC7uhksr7XVfcoU9xEjQkXIGmu4X3ih+8cfZx1ZLqBB9ETyY/Lk0KT0tdfC+Efjx9d5i568WbYs9DcYPx6eegqGDIGTT4YDD4SePbOOLjdUlCSSsba2/2ZwwAFdmwxHyli6FK67DjbaCPbaK4x8eu21MHNmaH6qpNAlSgwiHSTRmatwZjRYfkjtRYtg//0bsLNYWhYvDt2711svPBX06AE33wzPPhuycPckumg1LyUGyY089K6tZqrLQuPGLT8UdjF6eqjQJ5/AxReHoa+POAL69YMpU0Ll8l57hY5qUjUlBsmFpG7I1Sp2Qy82t0G5JFZJG/+4cyc0tQ8/hHPPDRf72GPD8BX33AOPPx7GNGrEoa8zpMpnyYW8tJWPM5NaxyEiYPkZ00qdTymF+5cCCxfCJZfAhReGiSe+972QRbfeOuvI6pIqn6Wu5KV3bZyZ1OI8VRRr+9/Zj9pSx21a8+eHCzpoUOh3sNVW4elg6lQlhRQoMUgulLshlyq6SbpeIk5nrjhJrNhQ2NddB9dfX4edxdL0xhthlrTBg8MAdyNHhvqDKVNgxIiso2secTo75G1RB7fGc/31YTyzwkHaevUK60t9duSRpb9TbSyddeaqdlC5uukslqZXXgl/0B493Lt1cz/wQPfnn886qoZDzA5umd/ku7IoMaQnD71rS92Iu3UrfYNOKu5i++ksidVCQyeSmTPdDznEvXt39xVXdD/sMPeXXso6qoalxCBVS/sGWIpZ8QTQ2ZJE3OWeYtK4Weflb5C4GTPc993XfYUV3Fdayf1nP3N//fWso2p4SgxStbyMw1/pE0NnTxJJHDfJ8691sVXutLS4/+AH4SRWXtn9pJPc33or66iaRtzEoMpnKSkvLYVKVQgffnjx9cuWFd9PpXHX+vzj9N3Iy9+gao88ArvsAsOHw4MPwmmnhXbIEybA6qtnHZ10oMQgJcVpupmGUpPdX3ZZ8fWlZmasNO6459/VllFxmr3m5W/QJe7wwAOw3XahuekTT8DZZ4eEcMYZ0Ldv1hFKKXEeK/K2qCgpHfVavp1U3HH2U82xStWdmCV/LqlqbXW/6y73LbcMAQ8Y4D5xovtHH2UdWdNDdQyShHptEVOLVkl9+4alcJ/V1AHE/W7d/A2WLXP/05/cN9vs8xO57DL3Tz7JOjKJKDFI3cvTDbHUL/dSraIKf/VXus/c3vhL+eyzEPSwYeEk1lnH/aqr3JcsyToy6SBuYkikjsHMRprZi2Y228zGFvn8eDN7zsyeNrP7zWxQwWfLzGx6tExJIh6pf3kZVK9NqfqAUoN5xqkDKFV3UjcT+CxZAlddBRtsEMYQN4MbboAXXoBDDoEVV8w6QumiqgfRM7NuwExgR2Au8ASwr7s/V7DNd4HH3X2RmR0JbOvue0effeTuK1dyTA2i1/jyMqhem1KD60FoCdXZgHoN59NPQ0KYMCE0j9p885A5R40KF0pyK81B9EYAs939ZXdfAtwE7FG4gbs/6O5t/+s8BqyZwHEl56oZxyjJZppJjKdU6gmgsCVUXf7qr8THH8NvfhOmzfzpT2GNNeCvf4WWFthzTyWFRhKnvKmzBRgNXFnw/gDg0k62vxQ4teD9UqCFkDBGdfK9w6PtWgYOHJhswZtUrFz5f7Xl551VzFZS9xA3jlqfT11buNB9/Hj3fv3CiW+3nfsDD4TWR1JXSKvyuZLEAOwfJYCeBevWiP4dCrwKrF3umKp8zlacm2QSA80lMXBenDiSSh4NZ/5891NPdV911XBRdt3V/ZFHso5KqpBmYtgSmFrw/mTg5CLb7QA8D3y5k31dDYwud0wlhmzFudnGaaNfTrEbcaUJJ04cDTfsRLXefNP9F79w/+IXw4XYc0/3f/0r66gkAWkmhu7Ay8AQoAfwFLBhh202A14C1u2wvnfb0wPQD5gFDCt3TCWGbGV5s+3s2F1NJEkksYbw2mvuRx/t3rNnGNxuzBj3Z57JOipJUNzEUHVtkbsvBY4GpkZPBLe4+7NmdoaZ7R5tdh6wMnBrh2apGwAtZvYU8CBwjhe0ZpJ8ijNMQ5wJb5I8dp8+xZu37rJL+TjqetiJJLz0Ehx2GKy9Nlx+eWh6+sILYVahDTfMOjrJQpzskbdFTwzZyrJMvtSx+/Yt/WSgiuUSnnvOff/9w9NBz56hAufVV7OOSmoI9XyWWsqyIrbYsastDmqqiuXp091Hjw4n26uX+wknuL/xRtZRSQriJoaqO7hlQR3cpKO8dYjLpccfD2Vod94Jq6wCxxwDP/859OuXdWSSkjQ7uIlUJIkOZx3Vqk6j7rnD3/8OO+4I3/oW/O//hiGv58yBs85SUpCilBhqrBY3wXpWbAykAw4IvYaruT5xxx1qmr+HO0ydCltvDdtuCzNmwHnnhQt+2mmw2mpZRyh5Fqe8KW9LvdQx5L1SM62hqQuVaj6axvXJ+98jEcuWud9xh/vw4eEE11zT/ZJL3BctyjoyyQFU+Zy9PHecquVkNp3ts1QlcRrXJ89/j6otXep+003uG28cTmroUPdJk9wXL846MsmRuIlBlc81VGpETjNobU0/nkJJVdaW2k+pfcbZvlbXJ89/jy777LMw1PXZZ8PMmfC1r4WRTvfZB7p3zzo6yRlVPudAnjtOJTV6aZztC7cpVkncUa2uT57/HhVbvBiuuALWWw8OPhi+8AW49VZ49tnQQU1JQaqgxFBDabeUKVWxWmx9ZzfJSipo49xUC7cprCSG8Gu9UC2vT0O0XFq0CC68EIYOhZ/8BFZfHaZMgWnTYPRoDX0tyYhT3pS3pV7qGNzT6zhV6Wikla4vFXeldQzFvp9mx7K67cj2/vvuv/61e//+4cJuvbX7vfdq6GupCKp8bi6lKla7dSu+vtRQEV2poK2kVZJUaMEC91/9yr137/CH2Gkn94cfzjoqqVNxE4MqnxtEZ1NPFlOqwrUhK2hLmDw51NO+9loo7ho/Pkczr73zDkycCL/9LXz4IeyxRwj2G9/IOjKpY6p8bjKlyvornay+oSpoO1Gso93hh+egw9u8eXDccaFyZ8IE2HlneOopuOMOJQVJjRJDgyhWsWoGy5ZVVsHbEBW0MYwbF+pxCy1aFNZn4tVX4cgjQ6XyJZfA3nvD88/DzTfDJptkFJQ0KyWGBlGstU9bkZD758mh3GT1cYeWqHdJNdet2syZcMghsM46cNVV8KMfwaxZ8Mc/wvrrpxyMSKA6hgakkUbLy/wazZgRHsNuuQVWWimUY514IqyxRgoHl2alOoYmlptfwzmWWZHZE0/AqFGheOiuu+CXvwyZ6MILlRQkN5QY6lC5DmjNUoFcjdSLzP7xDxg5EkaMCMNg/+d/hkeWX/8avvzlGh1UpIvitGkttwAjgReB2cDYIp/3BG6OPn8cGFzw2cnR+heBneIcr5H7MSQxDWUeRxGt245l1Whtdb/vPvdttgl/hP79Qye199/POjJpUqTVwQ3oBrwEDAV6AE8BwzpscxRwefR6H+Dm6PWwaPuewJBoP93KHbNRE0OcG3rcDmi1mm+5K/vMY6KqqdZW9zvvdP/Wt8LJfvWr7hMnun/8cdaRSZNLMzFsCUwteH8ycHKHbaYCW0avuwP/BqzjtoXbdbY0amKIc9Ovdm7jrqrm5t7Qw10XWrbM/bbb3DfdNJzg4MHuv/ud+yefZB2ZiLvHTwxJ1DGsAbxe8H5utK7oNu6+FHgf6Bvzuw2vrc6g1HDUr732+TZeohFZresPqmn33/CV4UuXhj/QRhuFgewWLQrNTWfODAPdrbRS1hGKVKRuKp/N7HAzazGzlvnz52cdTmIKe+CW0qdP59uk0Zqmmpt7w1aGL1kCV14Z+hu0DXV9003w3HNhKOwVV8w6QpEuSSIxzAPWKni/ZrSu6DZm1h1YFVgQ87sAuPskdx/u7sP79++fQNj5UOyXeKG2JpWltkmrA1o1N/eG6039ySdw6aWhU9phh4XMfccdMH166LFcahwSkTqRRGJ4AljXzIaYWQ9C5fKUDttMAQ6KXo8GHojKu6YA+5hZTzMbAqwL/DOBmOpGZ7+42276775b/HOz0AQ+jV7J1dzcG6Y39Ucfwfnnw5AhcMwxISvefTf8859hkDvNhSCNIk5FRLkF2AWYSWhVNC5adwawe/R6JeBWQrPUfwJDC747Lvrei8DOcY7XSJXPcSpm06i8jdPiqCmbnLq7v/ee+5lnuvfpEy789tu7P/SQ5kKQuoPmY6gP1fZLSOJm3XTNSeOaP9993Dj3VVcNF+X733d/9NGsoxLpMiWGOtLVX+tJ3dCbpjlpXG+84X7CCeFimrmPHu0+bVrWUYlULW5i0CB6OVZuIpmkBoJrpsl5OvXaa3DuuaGl0WefwX77wSmnwAYbZB2ZSCLiDqLXPY1gpHJtzVjbWiO1TSQDnyeHpPoHDBxYPMHUfXPSuGbPDmMWXXttyIYHHxwGt1sglv2gAAAPjklEQVR77awjE8mEmlHkVJwOZUn1D2i45qRxPfdc6H+w/vohEx9xREgSkyYpKUhTU2LIqThPA0nd0BumOWlc06aFHsobbhj6Hxx/fCh7u/TSJnpMEilNRUk5Fad4p+3GncSE9mPGNHAiaPPoo+EC3XUXrLIKnHoq/Pzn0Ldv1pGJ5IqeGHIq7tPAmDHhx25ra3qd3eqKOzz0EOywA3z72/DYY3DmmSHrnnmmkoJIEUoMCSk3eU6lmq54J2nu8Le/wXe+A9/9Ljz7LFxwQciep54Kq62WdYQiuaWipATEaUHUFU1RvJO01laYMgXOOguefBLWWgt++1v40Y80yqlITHpiSEA1Q1JLQpYtgxtvhK9/HX7wA1i4MPRHmD0bjjpKSUGkAkoMFSpWZFRP8w0kXeSVuc8+C3MfbLBB6JC2bFk4qRdegB//GHr0yDpCkbqjoqQY2nogz5kTyvvbegm3FRn16QMLFiz/vby1fKxVkVcmPv00JIQJE8KJbLYZ/OlPMGqURjkVqZL+Dyqj40Q6HYeOaLvJ1kMHsYYo8vr4Y7jwwtAB7aijYMAA+MtfQn3CnnsqKYgkQP8XlVFuIh0I8yXUQwuieiryWs4HH8A554S5EI47DtZbD+6/Hx55BHbdNVx4EUmEipLKiDt1ZT20IKrLMZEWLICLLw7LwoWw884hW2+1VdaRiTQsPTGUUe6mmccio1Lqakykt98OA9kNHgxnnBH6IrS0wF//qqQgUmNKDCW0td5pq3Au1PY+r0VGpdRFp7m5c+HYY8PFP/98+P73YcYMuP122GKLrKMTaQpKDEUUq3AuTAbXXRfWlRqCIk6T0KyajeZ2CI2XXw6jmw4dCpddBvvuC88/H/ombLRR1tGJNBXVMRRRrMLZPd4EOHGahDZUs9FqvfBCmAth8mTo1g0OPRROOilkSxHJRFUzuJlZH+BmYDDwKrCXu7/XYZtNgd8BqwDLgPHufnP02dXANsD70eYHu/v0cset9Qxu1cxoFmdWtaRmXqtrTz8dKjduvTX0Sj7iCDjxRPjqV7OOTKRhxZ3BrdqipLHA/e6+LnB/9L6jRcCB7r4hMBK40MwKRzA70d03jZaySSEN1UyAE6dJaF03G63WE0/AHnuEoSvuvhvGjg1ZcuJEJQWRnKg2MewBXBO9vgYY1XEDd5/p7rOi128A7wD9qzxuTVXTeidOUklq5rW68vDDsNNOMGIE/M//hJZGc+bA2WdD/1z/5yDSdKpNDKu7+5vR67eA1Tvb2MxGAD2AlwpWjzezp81sopn1rDKeRFTTeidOUqmrZqPVcId77oGtt4ZttoHp00MntTlz4LTToHfvrCMUkWLcvdMFuA94psiyB7Cww7bvdbKfAcCLwLc6rDOgJ+GJ4/ROvn840AK0DBw40PPg+uvdBw1yNwv/Xn995+vjfLchtLa6//nP7t/4hju4r7GG+0UXuX/8cdaRiTQ1oMXL3PM9/F9bfqOSXw43+gFecOMvsd0qwL+A0Z3sa1vgL3GOu8UWW9TkolXi+uvde/UKV7Bt6dWrwW7wlVq61P2WW9w32SRckMGD3a+4wv3TT7OOTEQ8fmKotihpCnBQ9Pog4M8dNzCzHsB/A9e6+20dPhsQ/WuE+olnqoynYoX9Cfr1C0ucvgUNMSBdUpYuDZ07NtoI9toLFi+Gq6+GmTNDO9yeuSghFJGYqk0M5wA7mtksYIfoPWY23MyujLbZC9gaONjMpkfLptFnk81sBjAD6AecVWU8FSnsyOYehuVZsCC8butbUCo5dKVlUcPNhbB4Mfz+97D++nDggbDiinDzzWEazYMOCu9FpO5U1Y8hK0n1YyjVn6BQqb4FlfZF6NipDUKFc+6GpIhj0aIwO9p554UhLEaMCPMof//7GuVUJMfS6sdQ1+L0Gyi1TaUtixqi6OnDD+Hcc8PQ18ceG4avmDoVHnsMdttNSUGkQTRlYmgr0onzsFSqb0GlTVrrulPbe++FfgeDBoURT7/+dfj738Pyve8pIYg0mKYbK6lYkU4p5foWVDIHQ13OhTB/fuiRfOml4Wlht91CkdGIEVlHJiI11HRPDJ3NyNa3b1hqMSR1XXVqe+MNOP748Fh1zjkwciRMmwZTpigpiDSBpntiKFV0Ywb//nftjtuWYMaNCzEMHBiSQq4qnufMCXUIf/hDaIK6335w8smwwQZZRyYiKWq6xJBlkU5up/+cNSs8GVx7bciQhxwS6hKGDs06MhHJQNMVJdVVkU6tPftsyFRf+xrccAMcdVSYMOeKK5QURJpY0yWGupjestaefBL23DP0VJ4yBU44IXS+uOgiWHPNrKMTkYw1XVES5LhIp9YeeQTOOivMg7DqqnD66fCzn4UadxGRSFMmhqbiDg8+GBLCgw+GwaDOPjsUG626atbRiUgOKTE0KvfwZHDWWfDoozBgAFxwQZhC84tfzDo6EckxJYZG09oKd9wREsK0aaG51WWXhZZGK62UdXQiUgearvK5YS1dGloWbbwx/PCHoafyVVeFpqhHHqmkICKxKTHUuyVLQgLYYINQo24Wxv144YXwlNCjR9YRikidUVFSvfr005AQJkwIXak33xxuvx322CNM+CAi0kVKDPXm449DB7TzzoO33oJvfxsuvzyMZ6RRTkUkAUoM9eL99+G3v4Xf/CZMM7fddqFOYdttlRBEJFFKDHm3YEHokXzxxSE57LJLGInv29/OOjIRaVBKDHn11lvh6eCyy0Lx0Q9/CKecEuoSRERqqKpaSjPrY2b3mtms6N/eJbZbZmbTo2VKwfohZva4mc02s5vNTE1oXn89DFMxZEjokDZqFDzzDNx2m5KCiKSi2uYrY4H73X1d4P7ofTGfuPum0bJ7wfoJwER3Xwd4D/hxlfHUr5degsMOg7XXht/9LsyF8OKLcP31sOGGWUcnIk2k2sSwB3BN9PoaYFTcL5qZAdsBt3Xl+w3j+efhgANgvfXguuvCvKOzZ4fJctZZJ+voRKQJVZsYVnf3N6PXbwGrl9huJTNrMbPHzKzt5t8XWOjuS6P3c4E1qoynfkyfDnvtFZ4Gbr8djjsOXnklzK88aFDW0YlIEytb+Wxm9wFfKfLRuMI37u5m5iV2M8jd55nZUOABM5sBvF9JoGZ2OHA4wMA0plurlccfD7MC3XknrLJKmDrzuOPCqKciIjlQNjG4+w6lPjOzt81sgLu/aWYDgHdK7GNe9O/LZvYQsBnwJ2A1M+sePTWsCczrJI5JwCSA4cOHl0pA+fXww2Fgu3vvhT594Iwz4JhjYLXVso5MRKSdaouSpgAHRa8PAv7ccQMz621mPaPX/YCtgOfc3YEHgdGdfb+uucM998DWW8M228DTT4cey3PmwGmnKSmISC5VmxjOAXY0s1nADtF7zGy4mV0ZbbMB0GJmTxESwTnu/lz02S+B481sNqHO4Q9VxpMPra1hysxvfhN22inUHVx8cfj3F7+AlVfOOkIRkZKq6uDm7guA7YusbwEOjV4/Amxc4vsvAyOqiSFXli0L/Q3Gj4cZM2DoUPj97+HAAzXKqYjUDQ3DmYTPPoNrroFhw2CffcL7664L/RAOPVRJQUTqiobEqMbixXD11XDOOfDqq/D1r8Mtt8Cee0K3bllHJyLSJUoMXbFoUSgiOvdceOMNGDECLrkEdt1VI52KSN1TYqjEhx+GQe0uuADmzw8tja65BrbfXglBRBqGEkMc770XWhVddFF4PXJkGPr6P/4j68hERBKnxNCZd96BiRPDBDkffhimzTz1VBg+POvIRERqRomhmHnzQke0SZPC3Mp77x3mQti4aKtbEZGGosRQ6NVXYcIEuOqq0Cdh//3DWEbrr591ZCIiqVFiAJg5E84+O8x90K0bHHwwjB0bJssREWkyzZ0YZswIvZRvuQV69oSjjw5DVqy5ZtaRiYhkpjkTgzuMGQM33hjGLTrppDD09eqlppMQEWkezZkYzMLsaL/6VRj6uk+frCMSEcmN5kwMEOZDEBGR5WgQPRERaUeJQURE2lFiEBGRdpQYRESkHSUGERFpR4lBRETaUWIQEZF2lBhERKQdc/esY6iYmc0H5mQdRxX6Af/OOogUNdP5NtO5gs633gxy9/7lNqrLxFDvzKzF3Ztmtp9mOt9mOlfQ+TYqFSWJiEg7SgwiItKOEkM2JmUdQMqa6Xyb6VxB59uQVMcgIiLt6IlBRETaUWIQEZF2lBhSYGZ9zOxeM5sV/du7k21XMbO5ZnZpmjEmKc75mtmmZvaomT1rZk+b2d5ZxNpVZjbSzF40s9lmNrbI5z3N7Obo88fNbHD6USYnxvkeb2bPRX/L+81sUBZxJqHcuRZs90MzczNruOarSgzpGAvc7+7rAvdH70s5E3g4lahqJ875LgIOdPcNgZHAhWa2WooxdpmZdQN+C+wMDAP2NbNhHTb7MfCeu68DTAQmpBtlcmKe7zRguLtvAtwGnJtulMmIea6Y2ZeAY4HH040wHUoM6dgDuCZ6fQ0wqthGZrYFsDpwT0px1UrZ83X3me4+K3r9BvAOULZHZk6MAGa7+8vuvgS4iXDOhQqvwW3A9mZmKcaYpLLn6+4Puvui6O1jwJopx5iUOH9bCD/gJgCfphlcWpQY0rG6u78ZvX6LcPNvx8xWAC4AfpFmYDVS9nwLmdkIoAfwUq0DS8gawOsF7+dG64pu4+5LgfeBvqlEl7w451vox8DdNY2odsqeq5ltDqzl7nelGViaumcdQKMws/uArxT5aFzhG3d3MyvWRvgo4K/uPrceflgmcL5t+xkAXAcc5O6tyUYpaTOz/YHhwDZZx1IL0Q+43wAHZxxKTSkxJMTddyj1mZm9bWYD3P3N6Eb4TpHNtgS+Y2ZHASsDPczsI3fvrD4iMwmcL2a2CnAXMM7dH6tRqLUwD1ir4P2a0bpi28w1s+7AqsCCdMJLXJzzxcx2IPww2MbdF6cUW9LKneuXgI2Ah6IfcF8BppjZ7u7eklqUNaaipHRMAQ6KXh8E/LnjBu4+xt0HuvtgQnHStXlNCjGUPV8z6wH8N+E8b0sxtiQ8AaxrZkOi89iHcM6FCq/BaOABr9/epGXP18w2A64Adnf3oj8E6kSn5+ru77t7P3cfHP2/+hjhnBsmKYASQ1rOAXY0s1nADtF7zGy4mV2ZaWS1Eed89wK2Bg42s+nRsmk24VYmqjM4GpgKPA/c4u7PmtkZZrZ7tNkfgL5mNhs4ns5bouVazPM9j/Cke2v0t+yYKOtCzHNteBoSQ0RE2tETg4iItKPEICIi7SgxiIhIO0oMIiLSjhKDiIi0o8QgIiLtKDGIiEg7/x/WE8ZG2yr1JgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# now we can perform gradient descent to improve this model\n",
    "model = L.Linear(in_size=1, out_size=1)\n",
    "losses = []\n",
    "\n",
    "learning_rate = 1\n",
    "\n",
    "for i in range(100):\n",
    "    model.cleargrads()\n",
    "    loss = F.mean(F.square(model(data_x[:, None]) - data_y[:, None]))\n",
    "    losses.append(float(loss.data))\n",
    "    ## *** YOUR CODE HERE TO PERFORM GRADIENT DESCENT ***\n",
    "    ## Hint: you could access gradients with model.W.grad, model.b.grad\n",
    "    ## Hint2: you could write data into a parameter with model.W.data[:] = some_numpy_array\n",
    "    ## Hint3: if your model doesn't learn, remember to try different learning rates\n",
    "    \n",
    "    # SOLUTION\n",
    "    loss.backward()\n",
    "    \n",
    "    model.W.data -= learning_rate * model.W.grad\n",
    "    model.b.data -= learning_rate * model.b.grad\n",
    "    \n",
    "    # END OF SOLUTION\n",
    "    \n",
    "    if i % 25 == 0:\n",
    "        print(\"Itr\", i, \"loss:\", loss)\n",
    "        \n",
    "plt.plot(np.array(losses))\n",
    "plt.title(\"Learning curve\")\n",
    "plt.figure()\n",
    "plt.plot(data_x, model(data_x[:, None])[:,0].data, c='r')\n",
    "plt.scatter(data_x, data_y, c='b')\n",
    "_ = plt.title(\"Trained model fitness\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train your first deep model\n",
    "\n",
    "Now we have learned the basics of Chainer. We can use it to train a deep model to classify MNIST digits. We will train a model on the MNIST dataset because the dataset is small.\n",
    "\n",
    "First we load the data and see what the images look like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train[i][0] is the ith image that's flattened, and has shape: (784,)\n",
      "train[i][1] is the ith image's label, such as: 3\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAEICAYAAACQ6CLfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAEQJJREFUeJzt3X2wXHV9x/H3Jw8kJDyYGAwZCCRAWkl1jHoljEHEUgEzQ8HBZsIUmioaZ4Rap+iUoaNQR6cMihZBKVEiARTrFGkiZBDMFFOkRi40kGDkOQHSPAABEqLk8ds/9oS5hLu/vdk9u2dzf5/XzJ2793z3nPO9m/vJ2T2/s/tTRGBm+RlSdQNmVg2H3yxTDr9Zphx+s0w5/GaZcvjNMuXwZ0zSvZI+3el1rTs4/IOApNWS/qLqPuqRNFvSY5JelbRR0gJJh1TdV+4cfuuEXwMzIuJQ4BhgGPC1alsyh38QkzRG0h2SXpD0cnH7yL3udqyk30raLGmhpLF91j9R0v2SXpH0sKRTmukjIp6LiBf7LNoFHNfMtqw8Dv/gNgT4IXA0cBTwR+Dave7zN8CngAnATuA7AJKOAO6kdoQeC3wRuE3SYXvvRNJRxX8QR9VrRNJJkl4FtgDnAP/a2q9mrXL4B7GIeCkibouIP0TEFuDrwIf3utvNEbEyIrYCXwZmSRoKnAcsjojFEbE7Iu4BeoGZ/ezn2Yh4W0Q8m+jlvuJp/5HAN4DVpfyS1jSHfxCTNErS9ZLWSNoMLAXeVoR7j+f63F4DDAfGUXu28FfFEf0VSa8AJ1F7htC0iFgL3AX8pJXtWOuGVd2AtdXFwJ8C0yNivaRpwP8C6nOfiX1uHwXsAF6k9p/CzRHxmTb0NQw4tg3btX3gI//gMVzSyD5fw4CDqb3Of6U4kXdZP+udJ2mqpFHAV4H/iIhdwC3AmZJOlzS02OYp/ZwwbEjSX+85HyDpaGovP5Y0+XtaSRz+wWMxtaDv+bqc2km1A6kdyX9D7en23m4GbgTWAyOBz0PtDD1wFnAp8AK1ZwJfop+/meKE32uJE35TgfslbaU27PcY0I5nFLYP5A/zMMuTj/xmmXL4zTLl8JtlyuE3y1RHx/kP0IgYyehO7tIsK6+zle2xTY3v2WL4JZ0BXA0MBX4QEVek7j+S0UzXqa3s0swSlsXAL59o+ml/cYnod4GPURvHPVfS1Ga3Z2ad1cpr/hOAJyPi6YjYTu1a7bPKacvM2q2V8B/Bm98U8nyx7E0kzZXUK6l3B9ta2J2ZlantZ/sjYl5E9EREz3BGtHt3ZjZArYR/LW9+R9iRxTIz2w+0Ev4HgCmSJks6AJgNLCqnLTNrt6aH+iJip6SLgF9QG+qbHxGPltaZmbVVS+P8EbGY2ltJzWw/48t7zTLl8JtlyuE3y5TDb5Yph98sUw6/WaYcfrNMOfxmmXL4zTLl8JtlyuE3y5TDb5Yph98sUw6/WaYcfrNMOfxmmXL4zTLl8JtlyuE3y5TDb5Yph98sUw6/WaYcfrNMOfxmmXL4zTLl8JtlyuE3y5TDb5Yph98sUy3N0ttNho4Zk6w/d8Hxyfqw19Pbf2Xa9rq14QfVrwHcN+O6ZP1TT30iWX98/WHJejvt3Hhgsj554c5kfdiSB8tsx0rUUvglrQa2ALuAnRHRU0ZTZtZ+ZRz5PxIRL5awHTPrIL/mN8tUq+EP4G5JD0qa298dJM2V1CupdwfbWtydmZWl1af9J0XEWknvAO6R9PuIWNr3DhExD5gHcIjGRov7M7OStHTkj4i1xfeNwO3ACWU0ZWbt13T4JY2WdPCe28BpwMqyGjOz9mrlaf944HZJe7bz44i4q5SumrDqX6Yk60+eeW2HOulPeqx84ZQ706unf7VK7TxnV7L+nZffWbc2787Tkused/PLyfrulb9P1i2t6fBHxNPAe0rsxcw6yEN9Zply+M0y5fCbZcrhN8uUw2+WqUHzlt6vfeS2yva9fHv6ba1X/d/pHerkrZY9MylZnz55dbI+5aCNyfpXxq1I1v9hzBP1a+fVrwHMWPG5ZP1QX1XSEh/5zTLl8JtlyuE3y5TDb5Yph98sUw6/WaYcfrNMDZpx/ltmpd8ees27Dk3Wx6x8tel9D9nyx2R959Orm952q44j/bbYlxqs/8rbxyfrP//NmmT9zFGbG+yhvpdmpj9P/dBbmt604SO/WbYcfrNMOfxmmXL4zTLl8JtlyuE3y5TDb5apQTPOv/vhVcn6oQ83WL+VfbewbrdbN7v+R28DnDnql01v++Xd6esjJs4f2vS2rTEf+c0y5fCbZcrhN8uUw2+WKYffLFMOv1mmHH6zTA2acX7r35CRI5P1J+anx/Hv/9A3GuwhPf14yuzz/y5ZH37vg01v2xpreOSXNF/SRkkr+ywbK+keSU8U38e0t00zK9tAnvbfCJyx17JLgCURMQVYUvxsZvuRhuGPiKXApr0WnwUsKG4vAM4uuS8za7NmX/OPj4h1xe31QN0PepM0F5gLMJJRTe7OzMrW8tn+iAggEvV5EdETET3DGdHq7sysJM2Gf4OkCQDF9/RUrmbWdZoN/yJgTnF7DrCwnHbMrFMavuaXdCtwCjBO0vPAZcAVwE8lXQCsAWa1s0lL23rO9Lq1l2b/IbnuYx+c32Dr6XH812Jbsj7j2ovr1iY+kP6QhcH8OQndoGH4I+LcOqVTS+7FzDrIl/eaZcrhN8uUw2+WKYffLFMOv1mm/Jbe/cCO03qS9buvvqZubYTa+0+8O+pe3AnAQc/VH7CLnTvLbsf2gY/8Zply+M0y5fCbZcrhN8uUw2+WKYffLFMOv1mmPM6/H3jmE0rW2z2Wn3LIkPRHg//6yu/VrV36xfcl171tyYnJ+jG3v56s69fLk/Xc+chvlimH3yxTDr9Zphx+s0w5/GaZcvjNMuXwm2VK0eD92GU6RGNjuvyhv/tq28wPJOujvrS2bu3ySekpFd5/wNCmeuoGO9mVrL/zzs/VrU39+vr0ttc811RPVVsWS9gcm9IXhhR85DfLlMNvlimH3yxTDr9Zphx+s0w5/GaZcvjNMuVx/kFu6PFTkvXthx+crG+dcECy/tJfpqcAf/RDP6xbG8KAhqPb4pPPnpKsb5ixNb2B3elrDKpS6ji/pPmSNkpa2WfZ5ZLWSlpefM1spWEz67yBPO2/ETijn+Xfjohpxdfictsys3ZrGP6IWAps6kAvZtZBrZzwu0jSI8XLgjH17iRprqReSb072NbC7sysTM2G/zrgWGAasA64qt4dI2JeRPRERM9wRjS5OzMrW1Phj4gNEbErInYD3wdOKLctM2u3psIvaUKfHz8OrKx3XzPrTg3H+SXdCpwCjAM2AJcVP08DAlgNfDYi1jXamcf587Pxog/Wrf35J3+TXPfKw3vLbmfAjl9wYbI++dL/6VAn+2ZfxvkbzvYQEef2s/iGfe7KzLqKL+81y5TDb5Yph98sUw6/WaYcfrNMeYpua6t3XHt/3dqj16ffLvzp//5wsv6Dib9qqqcBmZx+q/Jg4CO/WaYcfrNMOfxmmXL4zTLl8JtlyuE3y5TDb5Ypj/NbZWLH9mT93hXvSW+gjeP8empU27bdLXzkN8uUw2+WKYffLFMOv1mmHH6zTDn8Zply+M0y5XH+Dhh2zKRk/bELD0/WD308/UnM467vzo+RbkTD0n9+06c+1bZ9/zHS1xgcvqw7p+Auk4/8Zply+M0y5fCbZcrhN8uUw2+WKYffLFMOv1mmGo7zS5oI3ASMpzYl97yIuFrSWODfgUnUpumeFREvt6/V7jVs8tHJ+skLH03WF439WbJ+5rTTk/VuHpEeNumourXfXZK+vuHJSf9Wdjtv+O7L707WR/78t23bd7cYyJF/J3BxREwFTgQulDQVuARYEhFTgCXFz2a2n2gY/ohYFxEPFbe3AKuAI4CzgAXF3RYAZ7erSTMr3z695pc0CXgvsAwYHxHritJ6ai8LzGw/MeDwSzoIuA34QkRs7luLiKB2PqC/9eZK6pXUu4NtLTVrZuUZUPglDacW/B9FxJ6zUxskTSjqE4CN/a0bEfMioicieoYzooyezawEDcMvScANwKqI+Faf0iJgTnF7DrCw/PbMrF0G8pbeGcD5wApJy4tllwJXAD+VdAGwBpjVnha738Zr0s9ovjj2sZa2v2Pqkcn6sIder1vbvWVLS/secvDByfrj//xnyfrd53yzbm3SsNY+Hnuo0seuZ3a8Vrd255c/klz3QAb/UF/D8EfEfUC9N5SfWm47ZtYpvsLPLFMOv1mmHH6zTDn8Zply+M0y5fCbZcof3V2C15eOS9/hva1t/64f35Csf/XF+m9PfWrrYS3t+9jRLyTrd4z7XoMttG+q69Q4PsD5F19ctzb6P5eV3c5+x0d+s0w5/GaZcvjNMuXwm2XK4TfLlMNvlimH3yxTHucvwZGLNyXrHzjp3GT9gfff2tL+vzJuRf1ig0sQqtRomux33/H5ZH3S7buT9dG/8Fh+io/8Zply+M0y5fCbZcrhN8uUw2+WKYffLFMOv1mmPM5fgt0rf5+sj5+dfk/7B+ZcmKy/dvIfknU9VX/7J3/0keS6jfzq6eNaWv+gpfV7G7sqPX3bn9w7+D87v0o+8ptlyuE3y5TDb5Yph98sUw6/WaYcfrNMOfxmmVJEpO8gTQRuAsYDAcyLiKslXQ58Btjzwe6XRsTi1LYO0diYLs/qbdYuy2IJm2OTBnLfgVzksxO4OCIeknQw8KCke4ratyPim802ambVaRj+iFgHrCtub5G0Cjii3Y2ZWXvt02t+SZOoTT615/ORLpL0iKT5ksbUWWeupF5JvTtIX85pZp0z4PBLOgi4DfhCRGwGrgOOBaZRe2ZwVX/rRcS8iOiJiJ7hjCihZTMrw4DCL2k4teD/KCJ+BhARGyJiV0TsBr4PnNC+Ns2sbA3DL0nADcCqiPhWn+UT+tzt48DK8tszs3YZyNn+GcD5wApJy4tllwLnSppGbfhvNfDZtnRoZm0xkLP99wH9jRsmx/TNrLv5Cj+zTDn8Zply+M0y5fCbZcrhN8uUw2+WKYffLFMOv1mmHH6zTDn8Zply+M0y5fCbZcrhN8uUw2+WqYYf3V3qzqQXgDV9Fo0DXuxYA/umW3vr1r7AvTWrzN6OjojDBnLHjob/LTuXeiOip7IGErq1t27tC9xbs6rqzU/7zTLl8Jtlqurwz6t4/ynd2lu39gXurVmV9Fbpa34zq07VR34zq4jDb5apSsIv6QxJj0l6UtIlVfRQj6TVklZIWi6pt+Je5kvaKGlln2VjJd0j6Ynie79zJFbU2+WS1haP3XJJMyvqbaKk/5L0O0mPSvr7Ynmlj12ir0oet46/5pc0FHgc+CjwPPAAcG5E/K6jjdQhaTXQExGVXxAi6WTgNeCmiHhXsexKYFNEXFH8xzkmIv6xS3q7HHit6mnbi9mkJvSdVh44G/hbKnzsEn3NooLHrYoj/wnAkxHxdERsB34CnFVBH10vIpYCm/ZafBawoLi9gNofT8fV6a0rRMS6iHiouL0F2DOtfKWPXaKvSlQR/iOA5/r8/DwVPgD9COBuSQ9Kmlt1M/0YHxHritvrgfFVNtOPhtO2d9Je08p3zWPXzHT3ZfMJv7c6KSLeB3wMuLB4etuVovaarZvGagc0bXun9DOt/BuqfOyane6+bFWEfy0wsc/PRxbLukJErC2+bwRup/umHt+wZ4bk4vvGivt5QzdN297ftPJ0wWPXTdPdVxH+B4ApkiZLOgCYDSyqoI+3kDS6OBGDpNHAaXTf1OOLgDnF7TnAwgp7eZNumba93rTyVPzYdd109xHR8S9gJrUz/k8B/1RFD3X6OgZ4uPh6tOregFupPQ3cQe3cyAXA24ElwBPAL4GxXdTbzcAK4BFqQZtQUW8nUXtK/wiwvPiaWfVjl+irksfNl/eaZcon/Mwy5fCbZcrhN8uUw2+WKYffLFMOv1mmHH6zTP0/An4alFp2edUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAEICAYAAACQ6CLfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAADuFJREFUeJzt3X+s1fV9x/HnC0RQKpUrjlGQ2iLtQpoUmxu1qW1xbp3abtgZiWxVmjmpqWYzcUuNy6LrtoQ0bZ3bOrfrdKLr7LpRA9lIV0rWmnab9WqpYukK2kuFXsQfdaBWhMt7f5wvzRXv+d7LOd9zvof7fj2Sk/s93/f3e75vTnjd74/POferiMDM8plSdwNmVg+H3ywph98sKYffLCmH3ywph98sKYc/MUnfkPS73V7XeoPDPwlIGpL0K3X30Yykv5X00qjHAUn76+4ruxPqbsAmv4i4Frj2yHNJ9wCHa2vIAO/5JzVJsyX9m6RnJf20mF5w1GKLJH1H0j5J6yX1jVr/PEn/JelFSd+TtKyCnmYClwFr230ta4/DP7lNAf4BeCuwEPgZ8NdHLXMV8DvAPOAQ8JcAkuYD/w78GdAH/AGwTtLpR29E0sLiF8TCCfR0GfAs8GAr/yCrjsM/iUXE8xGxLiJeiYj9wJ8DHzxqsfsiYmtEvAz8MbBC0lTgY8DGiNgYEYcjYhMwCFwyxnZ+HBGnRsSPJ9DWKuDe8JdKaufwT2KSTpb0d5J2StpHY297ahHuI54eNb0TmAbMoXG0cHmxR39R0ovA+TSOEFrtZyGwDLi31dew6viC3+R2I/BO4NyI2CNpKfBdQKOWOWPU9ELgIPAcjV8K90XENRX2cyXw7Yh4qsLXtBZ5zz95TJM0Y9TjBOAUGuf5LxYX8m4ZY72PSVoi6WTg08C/RsQI8I/Ar0v6NUlTi9dcNsYFw2NxFXBPG+tbhRz+yWMjjaAfedwK/AVwEo09+f8AXx1jvftoBHIPMAP4PYCIeBpYDtxM4wLd08AfMsb/meKC30tlF/wkvRdYAPxLK/84q5583cUsJ+/5zZJy+M2ScvjNknL4zZLq6jj/iZoeM5jZzU2apfIqL/NaHND4S7YZfkkXAbcDU4G/j4g1ZcvPYCbn6sJ2NmlmJR6KzRNetuXD/uIjol8ALgaWACslLWn19cysu9o55z8H2BERT0XEa8CXaHwoxMyOA+2Efz6v/1LIrmLe60haLWlQ0uBBDrSxOTOrUsev9kfEQET0R0T/NKZ3enNmNkHthH83r/9G2IJinpkdB9oJ/8PAYklvk3QicAWwoZq2zKzTWh7qi4hDkq4H/oPGUN/dEfFEZZ2ZWUe1Nc4fERtpfJXUzI4z/nivWVIOv1lSDr9ZUg6/WVIOv1lSDr9ZUg6/WVIOv1lSDr9ZUg6/WVIOv1lSDr9ZUg6/WVIOv1lSDr9ZUg6/WVIOv1lSDr9ZUg6/WVIOv1lSDr9ZUg6/WVIOv1lSDr9ZUg6/WVIOv1lSDr9ZUg6/WVIOv1lSDr9ZUm3dolvSELAfGAEORUR/FU2ZWee1Ff7CBRHxXAWvY2Zd5MN+s6TaDX8AX5P0iKTVYy0gabWkQUmDBznQ5ubMrCrtHvafHxG7Jf0CsEnSDyLiwdELRMQAMAAwS33R5vbMrCJt7fkjYnfxcy/wAHBOFU2ZWee1HH5JMyWdcmQa+BCwtarGzKyz2jnsnws8IOnI6/xTRHy1kq7MrONaDn9EPAW8u8JezKyLPNRnlpTDb5aUw2+WlMNvlpTDb5ZUFV/sseNZY6i2qamLziyt/+i355XWP/Dh7zatrTztodJ1P/ORy0rrI9u2l9atnPf8Zkk5/GZJOfxmSTn8Zkk5/GZJOfxmSTn8Zkl5nH8SmPqORU1rQ5fPLV33/cubj8MD/M38dS31NBHDI6+U1rW/vG7t8Z7fLCmH3ywph98sKYffLCmH3ywph98sKYffLCmP8/eAw+cvLa2/8Kny8e6vL72naW3WlBml6657eXZpffGma0rrOuFwaf2HF9zVtPZb264sXfekXT8qrVt7vOc3S8rhN0vK4TdLyuE3S8rhN0vK4TdLyuE3S8rj/BV45TfPLa3fsOb+0vr7T/p2af20KSeV1n/pm59sWnvL/SeWrjvzmz8orS/e90hp/fAHzy6tc0Hz0u5t5X9r4Cw8zt9J4+75Jd0taa+kraPm9UnaJGl78bP8kyJm1nMmcth/D3DRUfNuAjZHxGJgc/HczI4j44Y/Ih4EXjhq9nJgbTG9Fri04r7MrMNaPeefGxHDxfQeoOnJm6TVwGqAGZzc4ubMrGptX+2PiACipD4QEf0R0T+N6e1uzswq0mr4n5E0D6D4ube6lsysG1oN/wZgVTG9ClhfTTtm1i3jnvNLuh9YBsyRtAu4BVgDfFnS1cBOYEUnm+x1r8wp/x36V0O/XFr/9Cvl4/gnrj+1tP72td9pXjw8UrpuebWzpr6qGrdu44Y/IlY2KV1YcS9m1kX+eK9ZUg6/WVIOv1lSDr9ZUg6/WVL+Sm8F5gz8d/kCA+XlX6yula6b/id7Wl73rNueLK3XOQyZgff8Zkk5/GZJOfxmSTn8Zkk5/GZJOfxmSTn8Zkl5nN/acl6f/7z28cp7frOkHH6zpBx+s6QcfrOkHH6zpBx+s6QcfrOkPM5vHXXz3vc0rR1+/uhbQFo3ec9vlpTDb5aUw2+WlMNvlpTDb5aUw2+WlMNvlpTH+a3U1HcsKq1fN/u+0vrFj1/VtPbmQzta6smqMe6eX9LdkvZK2jpq3q2SdkvaUjwu6WybZla1iRz23wNcNMb82yJiafHYWG1bZtZp44Y/Ih4E/DlMs0mmnQt+10t6rDgtmN1sIUmrJQ1KGjzIgTY2Z2ZVajX8dwCLgKXAMPC5ZgtGxEBE9EdE/zSmt7g5M6taS+GPiGciYiQiDgN3AudU25aZdVpL4Zc0b9TTjwJbmy1rZr1p3HF+SfcDy4A5knYBtwDLJC0FAhgCPtHBHq1GQ5fPLa3PmjKjtD79jr4q27EKjRv+iFg5xuy7OtCLmXWRP95rlpTDb5aUw2+WlMNvlpTDb5aUv9JrpWac+3xp/RAjpfWZO37atFa+pnWa9/xmSTn8Zkk5/GZJOfxmSTn8Zkk5/GZJOfxmSXmc30q96/Th0vqa595dWh/Ztr3KdqxC3vObJeXwmyXl8Jsl5fCbJeXwmyXl8Jsl5fCbJeXwmyXl8Jsl5fCbJeXwmyXl8Jsl5fCbJeXwmyXl8JslNZFbdJ8B3AvMpXFL7oGIuF1SH/DPwJk0btO9IiKa/5F260lT55xWWv/sgg2l9U8OLR9nC88dY0fWLRPZ8x8CboyIJcB5wHWSlgA3AZsjYjGwuXhuZseJccMfEcMR8WgxvR/YBswHlgNri8XWApd2qkkzq94xnfNLOhM4G3gImBsRR/7G0x4apwVmdpyYcPglvQlYB9wQEftG1yIiaFwPGGu91ZIGJQ0e5EBbzZpZdSYUfknTaAT/ixHxlWL2M5LmFfV5wN6x1o2IgYjoj4j+aUyvomczq8C44Zck4C5gW0R8flRpA7CqmF4FrK++PTPrlIn86e73AVcCj0vaUsy7GVgDfFnS1cBOYEVnWrROGr7inaX106acVFp/+s7FpfVTPdTXs8YNf0R8C1CT8oXVtmNm3eJP+Jkl5fCbJeXwmyXl8Jsl5fCbJeXwmyXlW3Qn9+bf+Elb68/a+WpFnVi3ec9vlpTDb5aUw2+WlMNvlpTDb5aUw2+WlMNvlpTH+a3Uk4d+Vlqf9pP/K62PVNmMVcp7frOkHH6zpBx+s6QcfrOkHH6zpBx+s6QcfrOkPM6f3BULHi6tbznwltL6yPanqmzHush7frOkHH6zpBx+s6QcfrOkHH6zpBx+s6QcfrOkxh3nl3QGcC8wFwhgICJul3QrcA3wbLHozRGxsVONWmuG/vS9pfVrT72jtH7WNz5eWl/ElmNtyXrERD7kcwi4MSIelXQK8IikTUXttoj4bOfaM7NOGTf8ETEMDBfT+yVtA+Z3ujEz66xjOueXdCZwNvBQMet6SY9JulvS7CbrrJY0KGnwIAfaatbMqjPh8Et6E7AOuCEi9gF3AIuApTSODD431noRMRAR/RHRP43pFbRsZlWYUPglTaMR/C9GxFcAIuKZiBiJiMPAncA5nWvTzKo2bvglCbgL2BYRnx81f96oxT4KbK2+PTPrlIlc7X8fcCXwuKQj4zo3AyslLaUx/DcEfKIjHVpbDvYdbmv9uQ/4VG2ymsjV/m8BGqPkMX2z45g/4WeWlMNvlpTDb5aUw2+WlMNvlpTDb5aUIqJrG5ulvjhXF3Zte2bZPBSb2RcvjDU0/wbe85sl5fCbJeXwmyXl8Jsl5fCbJeXwmyXl8Jsl1dVxfknPAjtHzZoDPNe1Bo5Nr/bWq32Be2tVlb29NSJOn8iCXQ3/GzYuDUZEf20NlOjV3nq1L3BvraqrNx/2myXl8JslVXf4B2refple7a1X+wL31qpaeqv1nN/M6lP3nt/MauLwmyVVS/glXSTpfyXtkHRTHT00I2lI0uOStkgarLmXuyXtlbR11Lw+SZskbS9+jnmPxJp6u1XS7uK92yLpkpp6O0PSf0r6vqQnJP1+Mb/W966kr1ret66f80uaCvwQ+FVgF/AwsDIivt/VRpqQNAT0R0TtHwiR9AHgJeDeiHhXMe8zwAsRsab4xTk7Ij7VI73dCrxU923bi7tJzRt9W3ngUuDj1PjelfS1ghretzr2/OcAOyLiqYh4DfgSsLyGPnpeRDwIvHDU7OXA2mJ6LY3/PF3XpLeeEBHDEfFoMb0fOHJb+Vrfu5K+alFH+OcDT496vosa34AxBPA1SY9IWl13M2OYGxHDxfQeYG6dzYxh3Nu2d9NRt5XvmfeuldvdV80X/N7o/Ih4D3AxcF1xeNuTonHO1ktjtRO6bXu3jHFb+Z+r871r9Xb3Vasj/LuBM0Y9X1DM6wkRsbv4uRd4gN679fgzR+6QXPzcW3M/P9dLt20f67by9MB710u3u68j/A8DiyW9TdKJwBXAhhr6eANJM4sLMUiaCXyI3rv1+AZgVTG9ClhfYy+v0yu3bW92W3lqfu967nb3EdH1B3AJjSv+TwJ/VEcPTfp6O/C94vFE3b0B99M4DDxI49rI1cBpwGZgO/B1oK+HersPeBx4jEbQ5tXU2/k0DukfA7YUj0vqfu9K+qrlffPHe82S8gU/s6QcfrOkHH6zpBx+s6QcfrOkHH6zpBx+s6T+Hx9jbUzs60hCAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train, test = chainer.datasets.get_mnist()\n",
    "\n",
    "# use train[data_point_index] to access data\n",
    "print(\"train[i][0] is the ith image that's flattened, and has shape:\", train[12][0].shape)\n",
    "print(\"train[i][1] is the ith image's label, such as:\", train[12][1])\n",
    "# here we visualize two of them\n",
    "plt.imshow(train[12][0].reshape([28, 28,]))\n",
    "plt.title(\"Label: %s\" % train[12][1])\n",
    "plt.figure()\n",
    "plt.imshow(train[42][0].reshape([28, 28,]))\n",
    "_ = plt.title(\"Label: %s\" % train[42][1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we will provide some boilerplate code and train a linear classifier as an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.0625; loss: 2.36098861694\n",
      "accuracy: 0.8525; loss: 0.637746690512\n",
      "accuracy: 0.8775; loss: 0.468824329376\n",
      "accuracy: 0.9; loss: 0.385648724437\n",
      "accuracy: 0.89; loss: 0.405417266488\n",
      "accuracy: 0.9175; loss: 0.321664657295\n",
      "accuracy: 0.8825; loss: 0.388487036824\n",
      "accuracy: 0.905; loss: 0.345183737874\n",
      "accuracy: 0.9125; loss: 0.314195497632\n",
      "accuracy: 0.91; loss: 0.281066377759\n",
      "accuracy: 0.93; loss: 0.295625793934\n",
      "accuracy: 0.9225; loss: 0.249506878257\n",
      "accuracy: 0.9275; loss: 0.289294960797\n",
      "accuracy: 0.895; loss: 0.403458530307\n",
      "accuracy: 0.9275; loss: 0.255096197426\n",
      "accuracy: 0.9125; loss: 0.27183150351\n",
      "accuracy: 0.9275; loss: 0.267053060532\n",
      "accuracy: 0.92; loss: 0.298493586183\n",
      "accuracy: 0.9325; loss: 0.273343095183\n",
      "accuracy: 0.9275; loss: 0.264639821351\n",
      "accuracy: 0.9125; loss: 0.307061653137\n",
      "accuracy: 0.915; loss: 0.24378559649\n",
      "accuracy: 0.93; loss: 0.326000562906\n",
      "accuracy: 0.93; loss: 0.25375210911\n",
      "accuracy: 0.9125; loss: 0.290319340825\n",
      "accuracy: 0.9125; loss: 0.308484270573\n"
     ]
    }
   ],
   "source": [
    "def run(model, batchsize=16, num_epochs=2):\n",
    "    \n",
    "    optimizer = chainer.optimizers.Adam() # we will use chainer's Adam implementation instead of writing our own gradient based optimization\n",
    "    optimizer.setup(model)\n",
    "    \n",
    "    stats = defaultdict(lambda: deque(maxlen=25))\n",
    "    for epoch in range(num_epochs):\n",
    "        train_iter = chainer.iterators.SerialIterator(train, batchsize, repeat=False, shuffle=True)\n",
    "        test_iter = chainer.iterators.SerialIterator(test, batchsize, repeat=False, shuffle=False)\n",
    "        \n",
    "        for itr, batch in enumerate(train_iter):\n",
    "            xs = np.concatenate([datum[0][None, :] for datum in batch])\n",
    "            ys = np.array([datum[1] for datum in batch])\n",
    "\n",
    "            logits = model(xs)\n",
    "\n",
    "            loss = F.softmax_cross_entropy(logits, ys)\n",
    "            model.cleargrads()\n",
    "            loss.backward()\n",
    "            optimizer.update()\n",
    "            \n",
    "            # calculate stats\n",
    "            stats[\"loss\"].append(float(loss.data))\n",
    "            stats[\"accuracy\"].append(float((logits.data.argmax(1) == ys).sum() / batchsize))\n",
    "            if itr % 300 == 0:\n",
    "                print(\"; \".join(\"%s: %s\" % (k, np.mean(vs)) for k, vs in stats.items()))\n",
    "\n",
    "# try a simple linear model\n",
    "run(L.Linear(None, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we will try to improve performance by training an MLP instead. A partial implementation is provided for you to fill in:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.0625; loss: 2.33138298988\n",
      "accuracy: 0.8875; loss: 0.334832838476\n",
      "accuracy: 0.9375; loss: 0.236049288511\n",
      "accuracy: 0.955; loss: 0.134355854988\n",
      "accuracy: 0.93; loss: 0.226021283865\n",
      "accuracy: 0.9575; loss: 0.15454631418\n",
      "accuracy: 0.935; loss: 0.21348623395\n",
      "accuracy: 0.96; loss: 0.124762963057\n",
      "accuracy: 0.9575; loss: 0.157841695249\n",
      "accuracy: 0.955; loss: 0.16892031312\n",
      "accuracy: 0.955; loss: 0.165444886684\n",
      "accuracy: 0.9775; loss: 0.0818471336365\n",
      "accuracy: 0.9675; loss: 0.105744742751\n",
      "accuracy: 0.965; loss: 0.120035322011\n",
      "accuracy: 0.9775; loss: 0.0847537881136\n",
      "accuracy: 0.985; loss: 0.0492774367332\n",
      "accuracy: 0.9675; loss: 0.07063713938\n",
      "accuracy: 0.955; loss: 0.117004701197\n",
      "accuracy: 0.99; loss: 0.0455859348178\n",
      "accuracy: 0.96; loss: 0.135210334361\n",
      "accuracy: 0.97; loss: 0.0754836189747\n",
      "accuracy: 0.975; loss: 0.0557348746061\n",
      "accuracy: 0.98; loss: 0.118801686764\n",
      "accuracy: 0.975; loss: 0.0848817780614\n",
      "accuracy: 0.9775; loss: 0.0548784548044\n",
      "accuracy: 0.97; loss: 0.0848140627146\n"
     ]
    }
   ],
   "source": [
    "class MLP(chainer.Chain):\n",
    "\n",
    "    def __init__(self, n_units, n_out):\n",
    "        super(MLP, self).__init__()\n",
    "        with self.init_scope():\n",
    "            # the size of the inputs to each layer will be inferred\n",
    "            self.l1 = L.Linear(None, n_units)  # n_in -> n_units\n",
    "            self.l2 = L.Linear(None, n_units)  # n_units -> n_units\n",
    "            self.l3 = L.Linear(None, n_out)  # n_units -> n_out\n",
    "\n",
    "    def __call__(self, x):\n",
    "        # *** YOUR CODE HERE TO BUILD AN MLP W/ self.l1, self.l2, self.l3 ***\n",
    "        # \n",
    "        # Hint: you should make use of non-linearities / activation functions\n",
    "        # https://docs.chainer.org/en/stable/reference/functions.html#activation-functions\n",
    "        \n",
    "        # SOLUTION\n",
    "        \n",
    "        a1 = self.l1(x)\n",
    "        h1 = F.relu(a1)\n",
    "        a2 = self.l2(h1)\n",
    "        h2 = F.relu(a2)\n",
    "        a3 = self.l3(h2)\n",
    "        \n",
    "        return a3\n",
    "        # END OF SOLUTION\n",
    "        \n",
    "run(MLP(200, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next you should try to implement logging test loss and see if the model is overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train: accuracy: 0.1875; loss: 2.22465896606\n",
      "Test: accuracy: 0.235; loss: 2.26699201584\n",
      "---\n",
      "Train: accuracy: 0.9; loss: 0.284228492379\n",
      "Test: accuracy: 0.84; loss: 0.512485772669\n",
      "---\n",
      "Train: accuracy: 0.9275; loss: 0.257063900828\n",
      "Test: accuracy: 0.8525; loss: 0.465323387384\n",
      "---\n",
      "Train: accuracy: 0.9275; loss: 0.211005628705\n",
      "Test: accuracy: 0.9; loss: 0.308968510628\n",
      "---\n",
      "Train: accuracy: 0.9425; loss: 0.194557437897\n",
      "Test: accuracy: 0.9; loss: 0.312252407074\n",
      "---\n",
      "Train: accuracy: 0.9325; loss: 0.230096690059\n",
      "Test: accuracy: 0.9225; loss: 0.237260146737\n",
      "---\n",
      "Train: accuracy: 0.9325; loss: 0.167544877529\n",
      "Test: accuracy: 0.9325; loss: 0.237918063998\n",
      "---\n",
      "Train: accuracy: 0.9675; loss: 0.0974924322963\n",
      "Test: accuracy: 0.9275; loss: 0.191263231933\n",
      "---\n",
      "Train: accuracy: 0.9625; loss: 0.119327769876\n",
      "Test: accuracy: 0.9375; loss: 0.199909154177\n",
      "---\n",
      "Train: accuracy: 0.9675; loss: 0.103237079382\n",
      "Test: accuracy: 0.9425; loss: 0.183929723501\n",
      "---\n",
      "Train: accuracy: 0.9575; loss: 0.129617249668\n",
      "Test: accuracy: 0.9375; loss: 0.193765764534\n",
      "---\n",
      "Train: accuracy: 0.9675; loss: 0.0987892493606\n",
      "Test: accuracy: 0.9325; loss: 0.20472435236\n",
      "---\n",
      "Train: accuracy: 0.9675; loss: 0.123416660726\n",
      "Test: accuracy: 0.945; loss: 0.168874151111\n",
      "---\n",
      "Train: accuracy: 0.9625; loss: 0.152844758928\n",
      "Test: accuracy: 0.935; loss: 0.190234075785\n",
      "---\n",
      "Train: accuracy: 0.9625; loss: 0.135103052258\n",
      "Test: accuracy: 0.94; loss: 0.182848321199\n",
      "---\n",
      "Train: accuracy: 0.9775; loss: 0.0902308392525\n",
      "Test: accuracy: 0.9425; loss: 0.179452741146\n",
      "---\n",
      "Train: accuracy: 0.9625; loss: 0.119786954522\n",
      "Test: accuracy: 0.9275; loss: 0.185162221193\n",
      "---\n",
      "Train: accuracy: 0.9825; loss: 0.0669767707586\n",
      "Test: accuracy: 0.935; loss: 0.169698693156\n",
      "---\n",
      "Train: accuracy: 0.965; loss: 0.0940151873231\n",
      "Test: accuracy: 0.9525; loss: 0.150243092477\n",
      "---\n",
      "Train: accuracy: 0.9625; loss: 0.12486405462\n",
      "Test: accuracy: 0.9425; loss: 0.194539924562\n",
      "---\n",
      "Train: accuracy: 0.9675; loss: 0.109370170832\n",
      "Test: accuracy: 0.9375; loss: 0.18019688189\n",
      "---\n",
      "Train: accuracy: 0.9775; loss: 0.0936650168896\n",
      "Test: accuracy: 0.955; loss: 0.158670171201\n",
      "---\n",
      "Train: accuracy: 0.975; loss: 0.0834726336598\n",
      "Test: accuracy: 0.9475; loss: 0.187857807577\n",
      "---\n",
      "Train: accuracy: 0.9725; loss: 0.117689839602\n",
      "Test: accuracy: 0.955; loss: 0.134179378152\n",
      "---\n",
      "Train: accuracy: 0.9875; loss: 0.0546641278267\n",
      "Test: accuracy: 0.9425; loss: 0.212662078142\n",
      "---\n",
      "Train: accuracy: 0.97; loss: 0.0904663267732\n",
      "Test: accuracy: 0.9575; loss: 0.173362127542\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "def better_run(model, batchsize=16, num_epochs=2):\n",
    "\n",
    "    optimizer = chainer.optimizers.Adam() # we will use chainer's Adam implementation instead of writing our own gradient based optimization\n",
    "    optimizer.setup(model)\n",
    "    \n",
    "    stats = defaultdict(lambda: deque(maxlen=25))\n",
    "    test_stats = defaultdict(lambda: deque(maxlen=25))\n",
    "    for epoch in range(num_epochs):\n",
    "        train_iter = chainer.iterators.SerialIterator(train, batchsize, repeat=False, shuffle=True)\n",
    "        for itr, batch in enumerate(train_iter):\n",
    "            xs = np.concatenate([datum[0][None, :] for datum in batch])\n",
    "            ys = np.array([datum[1] for datum in batch])\n",
    "\n",
    "            logits = model(xs)\n",
    "\n",
    "            loss = F.softmax_cross_entropy(logits, ys)\n",
    "            model.cleargrads()\n",
    "            loss.backward()\n",
    "            optimizer.update()\n",
    "            \n",
    "            # calculate stats\n",
    "            stats[\"loss\"].append(float(loss.data))\n",
    "            stats[\"accuracy\"].append(float((logits.data.argmax(1) == ys).sum() / batchsize))\n",
    "            if itr % 300 == 0:\n",
    "                test_iter = chainer.iterators.SerialIterator(test, batchsize, repeat=False, shuffle=False)\n",
    "                # *** YOUR CODE implement logging of stats on test set ***\n",
    "                \n",
    "                # SOLUTION\n",
    "                \n",
    "                for test_itr, test_batch in enumerate(test_iter):\n",
    "                    test_xs = np.concatenate([datum[0][None, :] for datum in test_batch])\n",
    "                    test_ys = np.array([datum[1] for datum in test_batch])\n",
    "                    \n",
    "                    test_logits = model(test_xs)\n",
    "                    \n",
    "                    test_loss = F.softmax_cross_entropy(test_logits, test_ys)\n",
    "                    \n",
    "                    test_stats[\"loss\"].append(float(test_loss.data))\n",
    "                    test_stats[\"accuracy\"].append(float((test_logits.data.argmax(1) == test_ys).sum() / batchsize))\n",
    "                 \n",
    "                # END OF SOLUTION\n",
    "                \n",
    "                print(\"Train: \" + \"; \".join(\"%s: %s\" % (k, np.mean(vs)) for k, vs in stats.items()))\n",
    "                print(\"Test: \" + \"; \".join(\"%s: %s\" % (test_k, np.mean(test_vs)) for test_k, test_vs in test_stats.items()))\n",
    "                print(\"---\")\n",
    "\n",
    "better_run(MLP(200, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try different variants!\n",
    "\n",
    "- Does using a ConvNet improve performance (reduce overfitting?)\n",
    "- Try changing the learning rate and observe the effect\n",
    "- Does the model train if you give it correlated gradients? (consecutively sample many batches of \"1\", then many batches of \"2\", ... etc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}