Machine_Learning_Coding_Problem.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Machine Learning Coding Problem.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyPaTCqctG+We3IDY+N0dB2a",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/phonism/notes/blob/master/Machine_Learning_Coding_Problem.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Kmeans\n",
        "K-means clustering is an unsupervised machine learning algorithm that seeks to segment a dataset into groups based on the similarity of datapoints.\n",
        "\n",
        "The kMeans algorithm finds those k points (called centroids) that minimize the sum of squared errors.\n",
        "\n",
        "Thus, the Kmeans algorithm consists of the following steps:\n",
        "+ We initialize k centroids randomly.\n",
        "+ Then we calcuate the distance from each point to each centroid.\n",
        "+ Assign each point to their nearest centroid.\n",
        "+ Centroids are shifted to be the average value of the points belonging to it.If the centroids did not move, the algorithm is finished, else repeat.\n"
      ],
      "metadata": {
        "id": "x95aLfDxqUAs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "from numpy.random import uniform\n",
        "import random\n",
        "import seaborn as sns\n",
        "from sklearn.datasets import make_blobs\n",
        "import matplotlib.pyplot as plt\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "\n",
        "centers = 5\n",
        "X_train, true_labels = make_blobs(n_samples=100, centers=centers, random_state=42)\n",
        "X_train = StandardScaler().fit_transform(X_train)\n",
        "sns.scatterplot(\n",
        "    x=[X[0] for X in X_train],\n",
        "    y=[X[1] for X in X_train],\n",
        "    hue=true_labels,\n",
        "    palette=\"deep\",\n",
        "    legend=None)\n",
        "plt.show()\n",
        "\n",
        "def euclidean(point, centroids):\n",
        "    return np.sqrt(np.sum((point - centroids) ** 2, axis=1))\n",
        "\n",
        "class KMeans(object):\n",
        "    def __init__(self, n_clusters=5):\n",
        "        self.n_clusters = n_clusters\n",
        "\n",
        "    def fit(self, x):\n",
        "        min_val = np.min(x, axis=0)\n",
        "        max_val = np.min(x, axis=0)\n",
        "        self.centroids = np.array([random.choice(x) for _ in range(self.n_clusters)])\n",
        "        iteration = 0\n",
        "        prev_centroids = None\n",
        "        while np.not_equal(self.centroids, prev_centroids).any() and iteration < 300:\n",
        "            sorted_points = [[] for _ in range(self.n_clusters)]\n",
        "            for p in x:\n",
        "                dists = euclidean(p, self.centroids)\n",
        "                sorted_points[np.argmin(dists)].append(p)\n",
        "            new_centroids = []\n",
        "            for cluster in sorted_points:\n",
        "                if len(cluster) > 0:\n",
        "                    new_centroids.append(np.mean(cluster, axis=0))\n",
        "                else:\n",
        "                    new_centroids.append(np.array([0, 0]))\n",
        "            prev_centroids = self.centroids\n",
        "            self.centroids = np.array(new_centroids)\n",
        "\n",
        "kmeans = KMeans(n_clusters=centers)\n",
        "\n",
        "x = np.random.rand(1000, 2)\n",
        "kmeans.fit(X_train)\n",
        "\n",
        "sns.scatterplot(\n",
        "    x=[X[0] for X in X_train],\n",
        "    y=[X[1] for X in X_train],\n",
        "    hue=true_labels,\n",
        "    palette=\"deep\",\n",
        "    legend=None)\n",
        "\n",
        "plt.plot(\n",
        "    [x for x, _ in kmeans.centroids],\n",
        "    [y for _, y in kmeans.centroids],\n",
        "    '+',\n",
        "    markersize=10)\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "lEADrCj9qSJK",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 515
        },
        "outputId": "c1514a47-2226-4ecd-ea5a-1eb60bf56996"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD5CAYAAAAk7Y4VAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3ic1ZX48e+dXiSNujRqlmy5dyOMAQPGdNM7BAIkJIRkyWaTbdnNLxuWbLIpm2zYkA0QwiYhoZfQQ+/B2DIY9yIb2ZKs3qWRNO3+/hh5rPGMmjUq1pzP8/Awc+fqfY8Gcead+957rtJaI4QQYvozTHYAQgghJoYkfCGESBCS8IUQIkFIwhdCiAQhCV8IIRKEJHwhhEgQpngcRCn1IHAR0KC1XhTj9TXAs8Bn/U1Pa63vGu64mZmZuri4OB4hCiFEQti0aVOT1jor1mtxSfjA74B7gD8M0ec9rfVFozlocXEx5eXlY4lLCCESilLqwGCvxWVIR2v9LtASj2MJIYQYHxM5hn+yUupTpdTLSqmFg3VSSt2mlCpXSpU3NjZOYHhCCDG9TVTC/xiYobVeCvwS+PNgHbXW92uty7TWZVlZMYehhBBCHIMJSfha6w6tdVf/45cAs1IqcyLOLYQQImRCEr5SKlcppfofr+w/b/NEnFsIIURIvKZlPgKsATKVUtXA9wAzgNb6XuAq4KtKKT/QA1ynpUynmECeXh/7atqpa+omw2VjVn4qrmTrZIclxISKS8LXWl8/zOv3EJq2KcSECwSCvLL+AA8+vz3cds7KIm69ZCFOu2USIxNiYslKWzHt1TZ184eXdka0vbbhIFX1XZMUkRCTQxK+mPZ6+vz4A8Go9u5e3yREI8TkkYQvpr2cdAczcpMj2pw2E3mZzkmKSIjJIQlfTHspSVb+4cYTKJufjcGgmFOUxp1fPhl3ZtJkhybEhIpXLR0hprRit4t/vulEOrq9OG1mnHbzZIckxISThC8Shs1iwmaRP3mRuOSvXySczu4+DtZ30ecNUJCdRHa6Y7JDEmJCSMIXCaW5vYf7nt7Kh9tqAUhNsnLnl1cxqyB1kiMTYvzJTVuRUHYfaA0ne4C2rj4ee203fT7/JEYlxMSQhC8SyqGm6MVWOw+04umVhC+mP0n4IqEUu11RbSsX5JDskBILYvqThC8SypyiVK45ew4GgwJg7ow0LjujFJNR/lcQ05/ctBUJRQchM9XOtefMQQHuDCfJTrm6F4lBEr44rvn8QWoaOmlu7yUz1U5+dtKQV+ub9zbyv09+GtH2rc+t4MwTCsc7VCEmnSR8cdzyB4K8vamKe578lIwUGxefNpNXNxzAaDBw8qJc5hSlYTwq+b+1qTrqOG+VV0nCFwlBEv4k87a00rl3L7119TgKC0iaPRtzstR4GUpTew/7q9tp7+6jrbOPTJeNi1bP5Pcv7iAQDO2r8+w7Ffzgq6eyaFbkTpoleSnsqmymxxsg2N+3JC/6Ru6x8PoDHGrootcXwJ3hxJUkG6yIqUUS/iTydXWx/8Hf0fze++G2gquvpPDaqzGYpdZLLI2tHn78UDm7D7QCYDQovnzZIvYcbA0ne4Cghpc/rIxI+F0eL3OKUun1FpLksNDe2ccHWw6x5oSCMcfV6fHyzFsVPPXWXoIaCrOT+KebTqTYnTLmYwsRLzI1YRL1VFVHJHuA6qeeoae2bpIimvoqqtvDyR4gENS8+EElhTnJUX17+/wc3kkzGNS8sv4AP/zdRl54/zMefXU3B+s7+OFXT43LFf7eqlaeeDOU7AGqGrp49FVZ0CWmFkn4gL+7m96GBgK9vRN63kBvX3RjMEiwL0a7AEJX0kdraPVQkBWd8C84pQSlQtMv65q7efiVXRGvb9/fQltXfN7rQ43dUW2f7GmgyyObrIipI+GHdDp372H/bx6ka98+UpcuofiWm3AWz5iQc9vz8zC7XPja28NtzlkzseXmTMj5j0exruTXLC8gJ93Gd794Es+9t59gMMjla0pZNDMj3MfnD+L1R+961esNxCWunBgF2BaUpI+oDHNbZy+tHX2kJFnIcNnjEo8QscQl4SulHgQuAhq01otivK6Au4F1gAe4RWv9cTzOPRY9dfVs//f/INAdujpr+2Qzu5uaWPyDuzC74nMjbyi27CwWfO//cfCRx+jctZu0shUUXHE55uTopCZCSvNdfPumE7n/z1tp6+pjzYoCrlw7G3emk5J8WDEvC63BbDJG/Fy6y8oJ87LZtKsh3Oa0mSjMPnKD3OcPUFXfSWNrD5mpDopykjCbI48zmNmFqZyzsojXNhwEIC3Zyo3nzx+2HPPOyhZ+/vAm6po9pCZb+bvrlrNibnb4m4kQ8aQOj3GO6SBKnQ50AX8YJOGvA75OKOGfBNyttT5puOOWlZXp8vLyMcc3mNZPNrPjzu9HtS/56Y9InjP7mI/r6+qme/9++hoasWZl4pxZMmQSD/T1EejuxpSSgsGU8F+6RqSloxevL0CGyxaV3I+mtea1DQdpaPFQ3+Lh072NzHCncMN585hXnA5AIBDkjfIq7nliM1qDUvC1K5dy9sqiEa/C9fT6qKrvotfrJy8ziay0oa/Wm9t7+NYv3qWl48hQosVk4O6/X0NBtnzoi2OjlNqktS6L9VpcsovW+l2lVPEQXS4l9GGggfVKqVSllFtrXTvEz4w7kyP6a7gymTDaj/1rdcDr5dCzz1H9+JPhtvwrLqPwumswWmNP0zNarYO+JmJLT7GNuG9tUzf3/3krfd4AhTnJlM3Poa7Zw6HGbvyBIHOL0qht7ubXT23h8PWP1nDfM1uYX5LOjNyRzbRx2MzMnZE24ria2noikj2A1x+kvsUjCV+Mi4m6aZsPVA14Xt3fFkUpdZtSqlwpVd7Y2DiuQdkLC8g575yItqIbrsOe5z7mY/YeOkT1k09HtNU88yw9NTXHfMxE4w8E2fFZM394cQePvrqbiqrW4X9oCJ5eP339Y/VV9Z28tuEgW/c1caipi3/99Qds3ddMR7cXfyByjN8f0LTH6aZuLMkOC1ZL5LcTpUI1+oUYD1Nu/EBrfT9wP4SGdMbzXCaHgxk3fI6Mk1fhbW7BlpONc9ZMlHFk47ax+D09EDzq5qDWBDw9Y4w2cez4rJnv3vvX8BTHJ97Yw4/uWE1uhpPObi+uJOuo9qTNSrNTmJ1EVcOR0sjW/rF5reHx13dzxzXLcNpMdA8ok2y3mshKHb+bqO5MJ1+7cgm/ePST8DeLG8+fR0G2LLwT42OiEn4NMHDtekF/26Qzu1JIW74sbsez5eRgyczA29R85BxpqTLzZoT8gSB/fmcfA9ZQ4fUHeX/zISpq2tiyt4k5RWl89YrFlBaObPjElWTlH24s496nP2VnZSu5GQ4uWj2TJ9/cC0BXj4/0FBvfvmUlP/vTJto6+0hNsvKtz63AnTl+yVcpxWnL8pnhTqGhpYf0FCszclOwyr67YpzE5aYtQP8Y/guD3LS9ELiDIzdt/0drvXK4Y473TVuA3oYG/J1dWNLTsKSNfPx1KF379lP95+ewZWViSU8jZdEikiZoqufxzucP8P/u+5Ad+5sj2s9cUcDOAy3UNXsAyEq189O/PW1U0xi7e3xU1nbwxsaDvLu5JjzM883rl7O2rAgIjau3dfbhSraQlSp73Yrjz7jftFVKPQKsATKVUtXA9wAzgNb6XuAlQsm+gtC0zC/E47zHKuD1YjAaaSnfRMUvfxVK+JmZzP2Hb5Iyf96Yj2/NysJZVED1408R9PvJOuM0iq6/DltOdhyin97MJiOXnDYzKuGX5Lt46+Mjhc8a23poaOkZVcJ32kM3VX2BIA2tPXT1eLn09FmUzT/y7Ssz1U7mOA7jCDGZ4jVL5/phXtfA38TjXGPhqa6m4c23af90C2knlhHs68PfGRrX9TY1sfunP2Ppf/0YS3r6mM7TuWsXB//4SPh541vvYHO7Kbr26jEdN1Esn5PFt28q45m3K7BaTFx82kweenlnRB+jQeG0j/7P12Q0sGx2FgtK0gkGNDZr9DGa23rYc7CVhrYeinKTmV2YSpJdauaL41/CDBZ6W1rY+Z8/obc6dOugq2IfrqVLKLz+WqqfeArt9+NtbqGvqXnMCb992/aotqZ33sN90TrMTueYjj0cHQjg7ejA5HAct1M9HTYzpy7Np2xBLkpBMKDZfaCVg3Wd4T43rZtPXtaxj69bTMaYf/3tXX3c88Rmygcs0PrCRQu49IxSjIaRL4bq9HixmoxYLMc+AUCIeEuYhO+prgkn+8PaP91C2gkrKL75RmpffgVfWzumlLHPf7YXhGacGp1OCAYJ9PTgLJ05bAL2tnfQU10NWmMvyMeSmjqq8/bUHKLmuedp+XA9zlmzKPrcdSTPLj3m32OyHZ5JgwmuWlvKirnZNLf3kJ3uoCTPNS7bEh6o64hI9gB//MsuTlrkJn8EHzBNbT28tamKVz86gDvTyXVnz2V+SbqsnBVTQsIk/MGmWga6u6l64ilm3HA9ttwc7Lm5Yz6Xa8kSSm77Er2HalFGA+bUVFKXL0UHg3iqqlFGA7acnIiYeurq2PuLX9K5M1Tgy1FSzNx//BaO/JjLFaL4PR723Xs/7Vu2AtD28Sd07t7D0v/68ZjWFUwVTruFxaWZw3fs19ntRSlIGuXm5L190bV1fP4gPt/wNXcCQc3z7+/n6bcqAKhr9rBtXzM/+9vTKckf/1IdQgxn2iV8rTW+1lYMFgumpCNXZPaCApIXzKdzx5Gx4PRVK2nfvgOCQZTFQvqqULUHz8EqPFVVGKxWnCXFWDMyjj7NkLzNzVT+9v/QgVCSMNptpMyfx75f30/jW2+jjEbyr7ycvAsvCNfsaS3/OJzsATyfVdL07vsUXX/tiM7Z19AYTvaHBbq76ampmRYJf6Q6PV4+3FrLE2/swWwycMN58zhhXk7MsfpYCrKToubjLyhJx+cPEgzq8ObnsbS09/DCe/sj2nz+IJV1HZLwxZQwrcoj9zY2cfBPj7D5m/9IxX2/oW3bDoL+0P+4FlcKc77xdWZ+5Uukn7yKgquvxJSUTEf/eLs5JRmDyUTHzl18+vf/xO6f/Iyd3/8hO3/4Y3obGoY6bYSg38+hZ58PJ3uAQE8vLR9toO2TT0BrtN9P9WNP0D7gw6dj+46oY7Vt/jTiOEMxmM2oGJumGI7Tcfxj9fGuBn75+Gbqmj1U1Xfxoz+Us+OzlhH/fF5WEnd95RQWz8ok2WHm1CV5LJ+TzT//6j0qqtuG/FmT0YAjxoIw6wgLsAkx3qZNwtfBIPWvvErn7j24L1pHoLOL+ldeiUikttwc3OsuIO+idRx67gUaXn8DAEtWJslzZhPo7eXgw48S9B6pud5dsY/OXbtHFYd3QLnjw7ytrRhtkdP92j7ZHH6cGmPxV/pJK0e86teWmxP1bSB1+TIcM4pG9PPTgT8Q4KUP9ke1f7Dl0KiOU5CdRF6mg9VL86lr7uZPr+zC59dsqRi61Edaio0vXrQwoi03w8lMuboXU8S0GdLxtrbR+Pa7ZJ+9loN/fDjc3rJ+A4t/9AOSZs0MtyXPncPCu75H1569mJKSSJ47B3teHt62ttBN06P0NTVHtQ3GaLGQd9E69hz1IeFavJjGt9+NaHMWF4cfpy5fSuZpp9L03geh5yuWk3HKySM+rzIayTnvXJwzS+iurMSWk0PynNlYJqDM81ShlIH0VDsQWXtnNIXWQscJ7ay1rybyg9toGP766OTFbv7DdQrb9zeT4bKxaGYmuRnjOzNLiJGaNgnfYDGTfsoqGt+JTKpBr5fOXbsjEr7BbCZl3lxS5s2N6GtOSSFzzekcevrZiPak0lmjiiV1+TJm/93XqX7qGQxWK4XXXI01OwuzKwVfewcAjhlFpC5bGv4ZW1YWpX/zNfKvuAy0xuZ2x6zmORRzkpO05cviWirieGI0KC5ZPZP1W+vChdDsVhOrFo3uRrzdaubac+byw99tCLdZzUaWjOCmsc1qYunsLJbOzhpd8EJMgLiVVhgPoy2t0L59BxX3/C+9hyKrLs/8ypdwr7tgRMfobWqm7sWXqPnzcxjtNopv+jyZp5+GyTH61Zf+7m4wGDD1l1vuqavHc/AgBpMJx4yiUd8MFsPTWrOvup1dlS0YjYr5xekUH8Oetb19fnZWtvD2x9W4kiysmJtNfasHHYTcDAfzitOH3dxEiMkwVGmFaZXwgz4fDW+/y757/jfcZrBYWPzjH5A0c+YQPxkqt9C5cxf1b7yF0Wolc/Up2NxubNlypZbIAoEg7396iF88+jH+gMZsMnD9uXMpLXCxfK4UxBNTz7jX0pkqDGYzmaecjMWVQv3rb2LNzCDrzDXDJnuAjm3b2fHv/xF+3vDGmyz64fdHnfADfX1ovx/TOK+oFROjurErnOwhNM3y8df38PkL5rNwVmZoxa4Ia2roov5QB8GgJjcvhaxc2chlKplWCR/A5HSQvvJE0leeOOKfCfr91Dz7fESbDgRo/nB91Dj/YHQgQPv2HVQ98RS+1lbyLrqQ9FUnYUlNnJum01FrR2842R/W6w0Q1BpZOxupvraDh369Hk93aJabxWripq+uIq9wdCvGxfiZdgn/mB29aQnAKIa7Oiv2sf17d4WPs+/X9xEMBslbd368IhSTIMNlx2wy4PMf+ftw2EzMzE8ddi/dRLNra1042QN4+/x8/NFBSfhTyLSZhz8WBpOJvEsvPqrREF55OxKdu3ZHfGiknVhGsKeH3f/9P1Q/9QyeqqohflpMVXlZSfz9DSdg6y+C5rSZ+PrVy5hbJEnsaM0DdhQ7rLGuk2AgxsWUmBRyhd/PtWghC773Xepe/gsGu43c884lZe6cEf+80XZkrrc1Kwt7npsDf/gjAE1A7Ysvs+gHd2F3j2yKoK+zk87de+jauw97vpvkefPkBvIkMBoUpyx2U+JeQ1tXH2kpNtwyrz6mBUvdbPskcpHbipOKMIxDkTtxbCTh9zPabKStWEbaimObw54yfx6m5GT8nZ1knLKK+ldfi3jd29yMp7JyRAlfBwLUvfwKB/90pKZ+yqKFzP3Hb426gqYYO6UUeVlJYyrHnAhmzMrg/CsW8c5fdhMMalafVcqsebLpz1QiCT9OHEWFLPrBXbRv3Yo5NY26l1+J6jPSr7a99fVUPfZERFvHtu14DhyUhC+mLLvDwspTi5m3KBe0JkV2DptyJOHHkXNGEc4ZRaEyyJWVVD/xVPg1o9OJc4R1bYI+P9rvj24fUONHiKkqxTW6UhZi4kjCHwfKYCB33QVYMzOpf+NNnMXF5J5/Lo7Cgqi+vs4uPAcO4O/qwp6Xh72wAFt2NqllJ9BWvincz5SchL0g+ueFEGKkEibhB/r66K6spLe2HktaKs6SYswpKeN2Pmt6Grnnn0v22WtRRmPMHY+87e1U/vZ34fo/ymxmwXf/ldSlS5h56xeoy8+j6YO/klRaSuHVV474hq8QQsQSl4SvlDofuBswAg9orX901Ou3AD8FDu8xeI/W+oF4nHsktNY0vvMu+351b7gt57xzKL758+O+ItZgGvwt7t7/WUSxN+3zse/e37D4Rz/Anuem+JabyL/yCox2G0aLbKIthBibMc+XUkoZgV8BFwALgOuVUgtidH1Ma72s/58JS/YAvXV1fPbA/0W01b/yGp6Dkzs33hejbn7voUMEPB4gNDRkcaVIshdCxEU8JsiuBCq01vu11l7gUeDSOBw3bgKeHoJ9fVHtvs7ohSITyRZj/9zU5cuwpMlMHCFE/MUj4ecDAy+Vq/vbjnalUmqLUupJpVThYAdTSt2mlCpXSpU3Ng69w9BIWbOysB81Q8ZgsUz6Xq/OmSXM/sYdGPvr3ifNnUPxF26OWMQlxPGurdXDrm11bPukhvrajskOJ6FN1E3b54FHtNZ9SqmvAL8H1sbqqLW+H7gfQuWR43Fyc0oyc7/5DfY/8CAd27ZjK8in9PbbcBTE+lyaOEaLhey1Z5KycCGBnh4sWZmYpcqmmEZamrp59Lcbaeovu2AyG/j87asoLE6f5MgSUzwSfg0w8Iq9gCM3ZwHQWg/cI/AB4CdxOO+oOEuKmf+db+Nrb8focGJxjd8MndGy5chqRDE9HdzfEk72AH5fkHdf28s1N5dhtkjxuYkWjyGdjcBspVSJUsoCXAc8N7CDUmrg2MklwM44nHfUTA4Hdrd7SiV7Iaaz9raeqLbmhi58vsAkRCPGfIWvtfYrpe4AXiE0LfNBrfV2pdRdQLnW+jngb5VSlwB+oAW4ZaznFUJMfYXFaVFtS08sxOE89plnLU3dNDd0YbaYyHYn4XBaxxJiQplWWxwKIaaWvj4/2zcf4o0XdtLX5+eEk2dw8pqZpKY5jul4NQdb+eN9H9HXGyo9MndRDhdcsYgUl9TtOSxhtjgUQky8poYu2lo82OxmsnKTsVqPpBWr1cSKk4oonZdNwB8gJdWO0WhAa017Ww8KRUqqLeZK9KN5+/y88eKucLIH2L2tnqUnFkrCHyFJ+EKIY1a5r4lHHtiIzxsak191Rgmnnz0bpRRGswFT/65gAwuqdXX28clHB3n/jQqUgtPPmc3SEwtxJg09NNPb66O2OnqxYkdr9H0CEZskfCHEMenu6uP5x7eEkz1AV6eXTzZUsXlDFa40O6eeVUpRSXrEFfz+3Y289fLu8PPXX9iFK83BwmV5Q57P4bQwd1EuW8qrI9ozc2Sj9JGShC+EOCY9Hh+tTZ7w85RUG06nhdeeD03Ca6zvYv/eJm7921NxF4RWj2ut2bwxtE4zLcPBkhMKCAaD9Pb4CAY1WgcxGmNP1zSZjKxeW0prczdVn7ViNBk487w55BW6xvk3nT4k4QshIrS39VBf005vr5+snGRy8lIwGKLH2J3JVvKLUqk52AbA7Pk5bP0kYgkOwYCmtqY9nPCVUuTkpdDe2sOSsgLee30vSinWnDeHvzyzjdrqdhYtz2fe4lxcadHj8pk5SVx/60raWjyYLUbSMpwxYxOxScIXQoS1t/bw5B82hZO4waD43JdXMnNO9H7KdruZdVcu5smHNtHa5CEQCGCxmPDgDf/snIU5pKba6e3xYbObaWnupnReFnaHhXdf20MwoDl5TQkfvfcZXR2helc1B9uor+1g3ZWLwvcABrLZzeTmy1X9sZDdhYUQYbXVbeFkDxAMal59bgc9nti7rbkLXHzhjlO59RurWb22lLMunAeAxWrknIsX0Nbi4Y/3b+Dh32zg4P5mHv7NRzz+u00kJVsJBkJTws0WYzjZH/bpxiramj1R5xNjI1f4QogwT7cv4rkyKLLdybQ0eUhOCcTcpzYp2UpScmiGTbLLxudvX0VXZx+vPLsdT1fog6L6QCtP/fET5i3OZeP7lbS3ejCZDPj9QRTRQzJKKRjBVE0xOnKFL4QIy8o9MuNFGRTnXDSf5oYufnv3+9z/3++xa2stgUBw0J83W0yUzM4kKcUaTvaHdbb3YrOZAfi0vJozzpuDxWrE0+0lPTOyaOCqM0pIyzi2xVlicHKFL4QIc+encNVNJ/CXZ7ZRODOdrR9XU1sdKmns6fLyxO838eVvnjbsGLrNZgYFDFjIrxThG6wdbb1seK+Syz+3HLvTzNITCzi4v5lD1R3MmZ9N8ewMjEa5Ho03eUeFEGEms5EFS93c9q3TOPmMmeFkf5jWoVo2w8nMdrJ67ayIttPPnUNapgOb3YzRZGDpiQXkFbpwpTqoOdBG5b4WikrSyC1wkZQse0KMB7nCF0JESUqxEdQaZ5KF7qOGZkZS+MxsMXHKmlmUzMmio60HV5qD3LwUbHYzM2ZloIMaV6odvz/Ai09uZcum0HTOPdvrych28vmvrIp5v0CMjSR8IURMKS47F161mCd+v4nDNRaXn1RITt7IyovbHBZKSjOj2l0DEnlrsyec7A9rbuimsb5LEv44kIQvhBjUnAU5fPmbp9HS1I0jyUKOOwW749hLG8d01Fg/yASd8SIJXwgxKIPRQG6+a9wWOqVlOFhWVsDmjUfq42TmJpEl9XHGhSR8IcSkMVtMrDl/LgXFaezaVs+MmenMW5xLsmvq3LT1dzQT6OvGlJSB0R6953TQ14cymVFq6s+BkYQvhJhUKal2VqyawYpVMyY7lAg64Ke7YhNNL91L0NOBxT2brAtvx5pTDICvvZHuHR/Qtf19rPlzSFlxbvi1qUoSvhBiSF2dfXR39mF3mofdaKSpoYsdmw9xYH8L85e4mb0gO+Im7fHE23iQhqf+C3RooZm3di+NL96L+/rvosxmWt99jK4tb4Veq/8Mz+715N38n5jTciKO4+9qA4MBk2Py99KWhC9EAtNa09fjw2w1xVzoVFXZwjMPb6at2UNSipVLr1vGzDmZMXeo6mjr4dEHN9LSGJqn/9neJpZVFbLuikWYzLFLHh+tpambil0NHKpqY9bcbIpLM0hOmZzhHV9rXTjZH+at3UugqwWUomvL2xGvBbrb8TZVhRO+v7ud7u3v0/bhMyijmbQ1n8M5uwyDdfI+AKf+oJMQYlw0N3bzxou7eODuD3jxiS3UH4pcZNXR1sMTv98ULmLW1dHH478rH3ThVUNdZzjZH7Z5YxUtzcMv1ALo7OjlyYc+5i/PbGdLeQ3P/OkTPnijAr8/MPwPjwOjI/pGtdHpQlkdoAxgiP4QUwPaeio20fzagwS6WvG3N9D47C/ord41rjEPJy4JXyl1vlJqt1KqQin17RivW5VSj/W//pFSqjge5xVCHJu+Xh8vP7ONv761j5ambjZvrObhBzbQNmC7wI623qgqlj5vYNAqlrGu+hXELI4WS2NdJ3VHbWG48YPKiE1WJpIlewbJy84+0qAMZF7wFcwpGZjTcnCtuiSivzmzCHNWEQBBn5eOTX+JOmb3no3jGvNwxjyko5QyAr8CzgGqgY1Kqee01jsGdLsVaNValyqlrgN+DFw71nMLIY5Na7OH/bsbI9o623tpbugktX/jEbvTjMlswO8bMKyhwDHI3rNZOUlkZDtpbjhyRb9sZSHJqTY+29vExx8dxGBQrDipiILitKghpGBQH31ItI7dPhGM9iRSytZhL16MMlsxubKwZBUCoSt518qLsGYV4flsC1b3TD/RMIMAACAASURBVOwzl2FOyQi9bjRiTM2B2n0RxzS5oheiTaR4jOGvBCq01vsBlFKPApcCAxP+pcCd/Y+fBO5RSimt9eT8lxQiwRmNBpRBoY9KpgO3F0zPcLLuikU89/iW8MKotefPJSsnKeYxU1LtXHtLGTu31nFwfzPzl7gpnZfNoYNt/PG+jwAoKE6j+kArn26sxpVmZ+6inPAc/6ycZFypNtrbesPHXLDUTVrm5FTN7KncSt3j/4n2hb7luE6+nNRTrsBocxDo7aZ7519pffcxjEmpmJLTMZiOLEhTBiOpKy+kZ89GdCBUctqUkY81fw7t5S+jjCas+XOwZk/szKR4JPx8oGrA82rgpMH6aK39Sql2IANoOvpgSqnbgNsAioqK4hCeEOJoaZkOTjqthPXv7A+3FZdmkJV7JJkrg2LR8nxy8lJoa+kh2WUjOzd5yBuwmTnJnJaTDMwGQjeFG/fs5sYLTWiTjVajm5eePnItuP7d/dxyxynkuFNwpdm5/ssnsXnDQQ7sa2HhMjfzl+ZhsUz83BJ/dxuNL/xvONkDtH/4DI7SFdiLFtBbtZPmVx4AINjTSVtjFUZHCq6VF4b7W/PnknfLD+mr3Y8ymjCmZFD3yPch4AdAWR3k3XgX1tySCfu9ptwsHa31/cD9AGVlZfINQIhxYDIZOeXMWRQWp1FV2UKOO4UZszJwHjVcYzIbcRekkpGdRF+vf8SzbQ7rq9mD4e2foH19WOafyYa9kf9L9/X6qa5sJccdmrKYnZvMORcvIOAPjvpc8RTs6cLf3hDVHuhqBaBn/+ao1zo/fYOkZWdhtIRmFSmlsObOxJo7Ex0M0PDs3eFkD6D7PHTv2XDcJfwaoHDA84L+tlh9qpVSJsAFNMfh3EKIY5SUbGX+Ejfzl7iH7Fdd2cqbL++isa6ThcvzWLm6JGrDkliCPi+t7z955CpZmfD7o6/hAv7IqY9KqXFL9loH8TUfwt/RjCkpFXNGHspojupndKZizinGV18Z0W5KzQbAnB79npkzCzEYB0mpWuNvj055gc6W0f8SYxCPWTobgdlKqRKllAW4DnjuqD7PATf3P74KeFPG74WY+prqO3novvVUVjTT3eVlw3uVvPHiTnze4adKal8vvqYjNXIC+z9k1QmRi4+MRgMFxWlxj3swnj3l1Pz2H6l75C6qH/gHOre8jfb7ovoZ7UlkX/g1TK7QnHplspC57nYsWaExd3vJMkyu7HB/ZbHjOuli1CAJXxlNpJSdH9XunHdyPH6tERvzFX7/mPwdwCuAEXhQa71dKXUXUK61fg74LfCQUqoCaCH0oSCEmOIa67uikvvOrXWceb6HzGEKnBnsySQtOYO2954AINjbRU7bO1x2zeVs2lBLssvGytNKcBeMT2G2o/la62h4/pdof399fx2k6eX7sebNjlkSweqeRd4tP8Tf3ojB5sScnhuul2PJzMd945146yvRAT+W7CIsmYVRxxjIMXMZmetup+2Dp0MLsU6/FlvB3Hj/mkOKyxi+1vol4KWj2v5twONe4Op4nEsIMXEs1uihFYvFiNE0/JCLUorkJWsJdLXTufl1DBYbqbMXUbDIzcITZqAMhvCWhxMh4OlA9x01p18HQ+Py/QlfBwP01VTQvXcjymjCMfsErHmlMdcYmFNzMKfmRLXH4m0+hK+lBnN6Hrk3/jtGqwOjPfZsp/E05W7aCiGmjmx3CoUlaVR91hpuW7tuHqnpIysPYE7NJvO8W0k9+VKU0YQpZfLmoRuTUjHYkwn2dB5pNJgw9c+dB+it2kXtn+4Ml1Ro+/AZ8j7/fWz5c475vL2H9lL78F3hDxtbyRKyLvobSfhCiKklOcXG5Tcsp+ZAG+2tPeTmu8gvcsW84h2MMpowp+WOY5QjY3Zlk33ZN2l45ucEe7tQZitZF/4N5ox8IDSFtL38pcj6OQE/3Ts+OOaEH/T20vr2IxHfLHo/20LfoQrMk/DhJwlfCDGk1DQHqWmTs/gp3hwzl5J/60/xd7ZgdKZgTnMf+fDSQYJ9PVE/Ezh6GGgUgt4e+uo/i2r3t0VP+ZwIUjxNCJFQzKnZ2AvnYUnPi/imogxGXCeui+qftOj0Yz6X0Z6Mc96qqHZLzuTU/pcrfCFEQvO1NaD9XkwpWdiLFpJz1T/Ttv5ZlMlM6qrLsOUf+0waZTSRuvJi/G319Oz/FGWykHbGdVjdpXH8DUZOEr4QIiEEfd7+rQhDV/XBvh46t79Hy5sPofs8OOaeRPraz+OcuxL7rGUAEfVxdMBPoKcLg9WBwTzyjdzNGXlkX/EP+NubUCYz5rScSdsOURK+EGJa87XW07Xjfbp3rcc+YyHJS8/CklVIX90+ml++L9zPs/sjjM40Ms/7YkSiB/A219C+4QU8uzdgySsl/bRrsOSW4O9oAgyYh6mCabQ6MGZPfm0wSfhCiGkr0NdD02sP0rO3HABv3X6692zEfeNdeBsORvXv3vE+aauvxJScfuQYvd00vXwfvQe2A9Czt5yOpDSM9mTaN7wABiNpp19L8uI1GB1DL0abbJLwhRDTlr+1LpzsB7Z5Gw5gHJDUD7NkFqIskVsq+tsbwskeQlUujfYk2v76dLit5fXfYXJlkRTjBu1o9NVX0rN/M4HudhylK7Dmz8Fgjr3/wLGQWTpCiOnLYIAYO24F+7qxumdjLVwQblMmC2lrb8BojZyCqowWMBy5Nra5S+kZ8AFwmGf3+jGF6m04SO0f/42WNx+i/aPnqP3TnfTs/3RMxzyaXOELIaYtc1ouSYvPoGvr2+E2S24J3tr9OGYuJ+eKb+GtryTo7cWSVRCzHo45PZe01VfR+u6jAPi7WrHkltBXsyeinzE5g0CfJ+oDY6R6q3cR7I3c/7f13cewzViI0TZ8ddKRkIQvhJi2DGYrjtllmFyZeBurQit+dZCeA9tIPe0aTPYkTElDV+tUBiMpZedjzSulr74Sc1ouRqeLnopN4QRtTErDYLbhaziIsXDeMcUa9PXFaOuNXPk7RpLwhRDTmjk9j6ZXfoPRnkzPZ1vQ3h5yrv42RuvI6gFBaAGVY9ZyHLOWA9DXeJCUsnWgQpu0B/0+2j54CnNWwTHHaSuYBwYjBI9UJ01ddRlGe/xuBEvCF0JMa9acGbiv+y6efR8T9HTgmF2GdQyLqQBMzlS6d3+Er3HgTB+FOfXYawZZ3TNxf+57tK1/lkBXC64TL8RResKY4jyaJHwhxLRnzS2J61aCRkcK2RffQf2ff4G/5RAGq4OM87+CZQxX+MpgxD5jIdb8OaCDcZ2dc5gkfCHEtBHwdNBzcAeeik1Ys2dgn7UcS381zHizumeRd9N/EOhoxmB3jrg2/nAMpugtF+NFEr4QYlrQOkjHplfCs2m6AFP5y7hvuBOzK2tczmlyujA5J2bHrniQefhCiGnB39YQsRgKjiyyEiGS8IUQ04IOBtEBf4z24TdcTxSS8IUQ04LZlU3yinMi2gz2ZCxZk1+0bKoY0xi+UiodeAwoBiqBa7TWrTH6BYCt/U8Paq0vGct5hRDiaMpkIvXkK7Ck59O59W2s7lJSVpyLJd09bufUAT9Bv/eYV9dONKW1PvYfVuonQIvW+kdKqW8DaVrrf47Rr0trPeode8vKynR5efnwHYUQYoBQ7XvTuNad7z1UQftHz+NtqCR58RqcC1eP283h0VBKbdJal8V6bayzdC4F1vQ//j3wNhCV8IUQYiKNZoOSY+FtqqH24X8Pb07e8tYf8bU3kHnuF1HG8ZtWOVZj/fjL0VrX9j+uAwabiGpTSpUrpdYrpS4b6oBKqdv6+5Y3NjaOMTwhhBi7oK+XnoM7aN/0Ct271uPraAon+8M6P3kdX3vTJEU4MsNe4SulXgdirRf+zsAnWmutlBpsfGiG1rpGKTUTeFMptVVrvS9WR631/cD9EBrSGS4+IYQYb907P6Tx+XvCz60F80heehadn74RblMmC8pgnIzwRmzYhK+1Pnuw15RS9Uopt9a6VinlBhoGOUZN/7/3K6XeBpYDMRO+EEJMJf72Jppf+11EW1/1LhxzToxoSzvjOkxTYAx/KGMdw38OuBn4Uf+/nz26g1IqDfBorfuUUpnAqcBPxnheIYSYEEG/N6pOPYApKY3MC27H11yNrXgJtoK54Q3Sp6qxJvwfAY8rpW4FDgDXACilyoDbtdZfAuYD9ymlgoTuGfxIa71jjOcVQogJYUrJwDlvFd27Pgy3KaMZa04JlimwMflojCnha62bgbNitJcDX+p//Fdg8VjOI4QQk8VgtpJ25g0Y7El0bX8fc0Y+GWfdhDkrenesqU6KpwkhxDAs6W4yz/sSaauvRllscdtycKJJwhdCiBFQRhOmlIzJDmNMpJaOEEIkCEn4QgiRICThCyFEgpCEL4QQCUISvhBCJAhJ+EIIkSAk4QshRIKQhC+EEAlCEr4QQiQISfhCCJEgJOELIUSCkIQvhBAJQhK+EEIkCEn4QgiRICThCyFEgpCEL4QQCUISvhBCJIgxJXyl1NVKqe1KqWD/xuWD9TtfKbVbKVWhlPr2WM4phBDi2Iz1Cn8bcAXw7mAdlFJG4FfABcAC4Hql1IIxnlcIIcQojWlPW631TgCl1FDdVgIVWuv9/X0fBS4Fdozl3EIIIUZnIsbw84GqAc+r+9tiUkrdppQqV0qVNzY2jntwQgiRKIa9wldKvQ7kxnjpO1rrZ+MdkNb6fuB+gLKyMh3v4wshRKIaNuFrrc8e4zlqgMIBzwv624QQQkygiRjS2QjMVkqVKKUswHXAcxNwXiGEEAOMdVrm5UqpauBk4EWl1Cv97XlKqZcAtNZ+4A7gFWAn8LjWevvYwhZCCDFaY52l8wzwTIz2Q8C6Ac9fAl4ay7mEEEKMjay0FUKIBCEJXwghEoQkfCGESBCS8IUQIkFIwhdCiAQhCV8IIRKEJHwhhEgQkvCFECJBSMIXQogEIQlfCCEShCR8IYRIEGOqpSOmrmAwSG1XPe29XaTbU8lNzprskIQQk0wS/jTkDwb44MBG7i//E76gH7vZxjdP+TLLcmUrYSESmQzpTEOHOur49caH8AX9APT4evnlhw/S2N08yZEJISaTXOFPQ82eVoI6GNHW6e2mrbeDLGdGRHtTdwsH2qrxBQMUutzkp8TazVIIMR1Iwp+G0h2pGJQhIuknWZyk2lIi+tV2NvCT935NTWcdAHazjX9b8w1mpRdPZLhCiAkiQzrTUH5yLreV3YDRYATAarJyx0m3RF3db2/YHU72EBr6eX7X6/gD/gmNVwgxMeQKfxoyGU2cUXwSszOKae/tJMORRm5S9CydQ50NUW2VbdV4Az5MRvnTEGK6kSv8acpoMFLoymNRzlzcydkopaL6LMyaE9V2RvEqHBb7RIQohJhgkvAT2NysWdy87CpsJisGZeCcWaexesaJkx2WEGKcjOl7u1LqauBOYD6wUmtdPki/SqATCAB+rXXZWM4r4iPJ4mDdnLWcWLCMQDBAliNdhnKEmMbG+n/3NuAK4L4R9D1Ta900xvOJOFNKkX3UzVwhxPQ0poSvtd4JxBwfFkIIMbVM1Bi+Bl5VSm1SSt02VEel1G1KqXKlVHljY+MEhSeEENPfsFf4SqnXgVjLL7+jtX52hOdZrbWuUUplA68ppXZprd+N1VFrfT9wP0BZWZke4fGFEEIMY9iEr7U+e6wn0VrX9P+7QSn1DLASiJnwhRBCjI9xH9JRSjmVUsmHHwPnErrZKyaA1++lx9c72WEIIaaAsU7LvBz4JZAFvKiU2qy1Pk8plQc8oLVeB+QAz/Tf2DUBD2ut/zLGuMUw/AE/2xv38PSOv9Dt9XDx3LM5IW8xSVbnZIcmhJgkSuupO0xeVlamy8tjTu0Xw9jZWMGdb/4czZH/vnecdDOnF6+axKiEEONNKbVpsLVOsspmmvq0bkdEsgd4YfcbnJi/DLvZNurj1XU1sqNhD7WdDczPms3czBKcFvm2IMTxRBL+NOUwRdfDcZodGA2jv23T7Gnlp+/fS1X7IQCe3fUqNy+7inVz1soaDCGOI1JL5zjT4mllb/Nn1HcNvUZhSe487KYjV/IKxeULzsditIz6nAfaqsPJ/rBHtz1PY3fLqI8lhJg8coV/HNnVWMF///UBWnvbsZtt3H7i51mZvzRc976tt4MeXw+pNhfFaYX8+9q/Z2v9Tjy+Xpbkzmf2MW5s4otRH98b8BLQUjdfiOOJJPzjRIunjf/+MJTsIbRZyf98+Ft+fO6/UuBys6VuJ78pf5hGTwsLs+fwheXXUJxWQHFawZjPXejKw262RUzvXFN8MpkOqcEjxPFEhnSOE6097bT2tEe0BXSQRk8L+5oP8OP3f02jJzTEsr1hD/du/CPdXk+4r9fvY0fDXp7f9RrvVn5EXefIy1bkpeTw3TO+wUkFy8lNyuKaRRdz5YILMEtlTSGOK/J/7HEi2ebEaXbQ7TuSxBWKTEcaOxr2EggGIvpXtFTS7GnFaXEA8HHtVn7+19+EX3cnZfOdM75OdlLmiM5fmlHMN1Z9kd6Al6T+Yw4nEAywt7mSj6o/xqAMnFSwnNL0YgzHcONYCDF28n/ecSLbmclXV34+PF6vUNy49HLsJju9/uiVtE6zIzz9sr23kz9sfiri9dquBva3HhxVDCajacTJHmB3036+99bPeHHPmzy/+3W+9+bP2Nvy2ajOKYSIH7nCj5PK1iq2N+zBG/CxKGcus9JmxP1KtixvCT85919p7G4h1ZZCQYobb9BHk6eNsrwllB/aEu57w9LLwpuW+4M+Or3dUcfr9ffFNb6jvbbvXQYu7AvoIO9UfsTczFnjel4hRGyS8ONgf8tBvvfWz+nrT6DGbQa+u+bvWJA9O67nMRgMFLryKHTlhdssmFlZsJTXK97n2kWX4Av6SLEmsSRnfriPURlZU7yKVyreGdBmoMiVH9f4jtbn98ZoG98PGSHE4CThx0F5zZaIRBbQQV7Y8wZzMmdi6h+CGU+LsueSZHFS3VGL3WQjPyWXuq5GdjZVkOPMorG7GY3mnFmnU17zKemOVC6ee3ZcZvAM5dzS0yO+dUBodo8QYnJIwo+DDm9ndFtfJ1oHgbEl/PbeDva1HKS1p42cpCxmphXhsESuojUajMxKn8Gs9Bl4vD38acszvLbvPQAKUtwUufL5a1U5WY50luctor23o7/MwlIMxvG7jTM/q5Rvn/Y1XtzzJgYMXDh3LfNkOEeISSMJPw5OKljOqxWR5f3XzT4Ts9Ec0dbY3UJF82e09LRTnJbPrPRibCbroMft9np4aPPTvHvgo3DbDUsu56K5Z4Vv3h6tuqM2nOwhNJ1zuXth6PyeFt7c/wEAF8w5E5NhfP/zW01WVuQtZmnOArRiQr7tCCEGJwk/DuZmzOKfVn+Vp3a8hDfg47J557I0d35En1ZPG3d/+AB7mo/MUrn9xBtZO/PU8HOtNY2eFgLBAFmOdKo7aiOSPcBj256nLG8J+a5Ym5BB14C59wDdPg8mg4nCFDdVHbUApNtTOavk1Amrg2M0SqIXYiqQhB8HFpOZsvwlLMqeS1AHo4ZcACrbqiOSPcBDm59mcc58spzpdHs9vFO5nke3Poc34OPM4pNZPWNl1HH8QT+9gcFvfLqTs7GZrBEzcMprtvAPq2+nobuJYDBIgcsdnsEjhEgckvDjyGYefHgm1hRIj68Hf9AHwJ7m/fzukyfCr73x2QekOVwUpuRR1XGkcNkMVz7ZzsEXS7mTs/nX0+/ggU2PcrC9hkXZc7l5+VW4k7NxJ2cfy68lhJgmJOFPkAKXG7PRjC/gC7edUlRGhj0dCG1YcrT3D2zk707+Ev/38ePsbz3AMvdCrl10CclWJ1pr9rceZEvdToI6yJLc+cxKn4FBGZiXVcqdZ36Tbl8PLmsStmOofy+EmH4k4U+QQlce3z3jb3l4y5+p6ahjdfFKLig9E4spdGM3Nykr6meKUvMpdOXxL2f8DR5vD8nWJKymUHnjipZK7nzz5/iCoYqVT25/ke+t/SbzMksBSLI6R72dYVAHqWytorKtGqvJysy0IvlWIMQ0Igl/As3LKuVfTr+DXn8fLmtyxErcBVlzKHTlhevO2002Lpt3HmajCbPRhMMceV/g/QMbw8keQnP/X6t4L5zwj8XOxgr+4+27CeggAJn2dP7fmq+TlxL7BrEQ4vgiCX+C2c22mFsM5iZn8S+n38GBtmp8AR+Frjzyh0i0XTFKJXT0dRHUQQzKQLfXg8VoGXFFy15/H09seyGc7AGaelrY2VghCV+IaWJMCV8p9VPgYsAL7AO+oLVui9HvfOBuQquQHtBa/2gs552uMh1pZDrSRtT39OKTeO/Ahoi280pPp9nTynuVH/FO5UcUpeZx6bzzKM0oHvZ4voCfJk9rVHt7b8eI4hFCTH1jXWb5GrBIa70E2AP8y9EdlFJG4FfABcAC4Hql1IIxnjfhzcucxT+tvp3S9GJmphXxrVO+zLzMUp7e/jKPbnue2q4GPqrezPffvpvq/vn3Q0m2Ojmv9PTo82Qd+xCREGJqGdMVvtb61QFP1wNXxei2EqjQWu8HUEo9ClwK7BjLuROd1WRlWe5C5mXOxmQwYjNbqets4K3KDyP69fh7qW6vpSDFPewxTy06kb6Al5f2vIXTbOeGpZdTml4yaP+GriYq26rxB/1RRd2EEFNPPMfwvwg8FqM9H6ga8LwaOGmwgyilbgNuAygqKopjeNNHIBhgZ2MFz+16lc6+bi6cexbL3AsxGoxYjOaoOf9Hl3gYTLojlSsXrGNtyamYjCZSrEmD9j3UWc8P3/klDd3NANhMVr675hvMzhj8A0IIMbmGHdJRSr2ulNoW459LB/T5DuAH/jTWgLTW92uty7TWZVlZ0VMVRWhK5vffuZvNdTvY13qA/1n/IB8f2kqmI51rF18c0bcgxc2MUZRBVkqR7kgdMtkDbK3bFU72ELrp+/yu1/HH2PBcCDE1DHuFr7U+e6jXlVK3ABcBZ+mBu10cUQMUDnhe0N8mjtHW+l0c/VY/t+s1Tsxfwprik8lLzmVHwx7cydksyJpNpjM97jHUdTVEtVV31OIN+jDJXrdCTEljnaVzPvBPwBlaa88g3TYCs5VSJYQS/XXA58Zy3kRnNVqi2mxmK0oZcJqtLHcvDFfIHC9Lcubz4p43I9rWlpwStV5ACDF1jHWWzj1AMvCaUmqzUupeAKVUnlLqJQCttR+4A3gF2Ak8rrXePsbzJrTFOfOiyipfuWDdkKWW421O5ixuPeE6nGYHJoOJS+aew8lFJ0zY+YUQo6dij8JMDWVlZbq8vHyyw5iSPmut4pPabXR5PZTlLWZ2RsmIb87GU1N3K0EdINORHvc9fIUQo6eU2qS1Lov1mgy2HqdK0gopSSscvuM4y3SObKGYEGLyySWZEEIkCEn4QgiRICThCyFEgpCEL4QQCUISvhBCJAhJ+EIIkSCm9Dx8pVQjcCBOh8sEmuJ0rIl2PMcOEv9kOp5jB4n/WMzQWscsRDalE348KaXKB1uMMNUdz7GDxD+ZjufYQeKPNxnSEUKIBCEJXwghEkQiJfz7JzuAMTieYweJfzIdz7GDxB9XCTOGL4QQiS6RrvCFECKhScIXQogEMW0TvlLqaqXUdqVUUCk16LQopVSlUmpr/wYuU6L4/ihiP18ptVspVaGU+vZExjgUpVS6Uuo1pdTe/n/HrKGslAr0v++blVLPTXScMeIZ8v1USlmVUo/1v/6RUqp44qOMbQSx36KUahzwfn9pMuKMRSn1oFKqQSm1bZDXlVLqf/p/ty1KqRUTHeNQRhD/GqVU+4D3/t8mOsYwrfW0/AeYD8wF3gbKhuhXCWROdryjjR0wAvuAmYAF+BRYMNmx98f2E+Db/Y+/Dfx4kH5dkx3raN5P4GvAvf2PrwMem+y4RxH7LcA9kx3rIPGfDqwAtg3y+jrgZUABq4CPJjvmUca/BnhhsuPUWk/fK3yt9U6t9e7JjuNYjDD2lUCF1nq/1toLPApcOv7RjcilwO/7H/8euGwSYxmpkbyfA3+vJ4GzlFJqAmMczFT+WxiW1vpdoGWILpcCf9Ah64FUpZR7YqIb3gjinzKmbcIfBQ28qpTapJS6bbKDGYV8oGrA8+r+tqkgR2td2/+4DsgZpJ9NKVWulFqvlJrsD4WRvJ/hPjq0V3M7kDEh0Q1tpH8LV/YPiTyplJr87dJGbir/rY/UyUqpT5VSLyulFk5WEMf1FodKqdeB3BgvfUdr/ewID7Naa12jlMomtBn7rv5P7HEVp9gnzVDxD3yitdZKqcHm/s7of+9nAm8qpbZqrffFO1YBwPPAI1rrPqXUVwh9U1k7yTElio8J/a13KaXWAX8GZk9GIMd1wtdanx2HY9T0/7tBKfUMoa/H457w4xB7DTDwKq2gv21CDBW/UqpeKeXWWtf2f/VuGOQYh9/7/Uqpt4HlhMaiJ8NI3s/DfaqVUibABTRPTHhDGjZ2rfXAOB8gdJ/leDGpf+tjpbXuGPD4JaXU/yqlMrXWE14ULqGHdJRSTqVU8uHHwLlAzDvtU9BGYLZSqkQpZSF0E3HSZ7r0ew64uf/xzUDUNxalVJpSytr/OBM4FdgxYRFGG8n7OfD3ugp4U/fflZtkw8Z+1Jj3JcDOCYxvrJ4DbuqfrbMKaB8wZDjlKaVyD9/rUUqtJJR3J+dCYbLvGo/XP8DlhMb6+oB64JX+9jzgpf7HMwnNaPgU2E5oOOW4iL3/+TpgD6Gr4ikRe39cGcAbwF7gdSC9v70MeKD/8SnA1v73fitw6xSIO+r9BO4CLul/bAOeACqADcDMyY55FLH/Z//f+KfAW8C8yY55QOyPALWAr//v/lbgduD2/tcV8Kv+320rQ8y6m6Lx3zHgvV8PnDJZsUppBSGESBAJPaQjhBCJRBK+EEIkCEn4QgiRICThCyFEgpCEL4QQCUISvhBC65ETOQAAAA5JREFUJAhJ+EIIkSD+P8wH+ELjo49dAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD5CAYAAAAk7Y4VAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3hc1Zn48e+ZXtT7qFmyJblXhLGppoPpLUAgQEJCSJYsSza7y242CyGbLEl+SZYN2RBCWBIIvQTTAsZ0gosMxr0jW5LVuzSSpp3fHyOPNZ5Rs0bF0vt5Hj+eOffMva/G8jtnzj1Faa0RQggx+RnGOwAhhBBjQxK+EEJMEZLwhRBiipCEL4QQU4QkfCGEmCIk4QshxBRhisVJlFKPAhcDdVrreVGOrwBeBr7oLXpRa33fYOdNS0vTBQUFsQhRCCGmhI0bNzZordOjHYtJwgceAx4E/jRAnQ+11hcP56QFBQWUlZWNJC4hhJhSlFIH+jsWky4drfUHQFMsziWEEGJ0jGUf/nKl1OdKqTeUUnP7q6SUuk0pVaaUKquvrx/D8IQQYnIbq4T/KTBNa70Q+DXwl/4qaq0f1lqXaq1L09OjdkMJIYQ4BmOS8LXWbVrrjt7HrwNmpVTaWFxbCCFE0JgkfKVUllJK9T5e2nvdxrG4thBCiKBYDct8ClgBpCmlKoF7ADOA1voh4GrgW0opH9AFXKdlmU4xhtzdXvZVtVLT0Elqoo0ZOUkkxlvHOywhxlRMEr7W+vpBjj9IcNimEGPO7w/w5toDPPrKtlDZuUvzufXSuTjtlnGMTIixJTNtxaRX3dDJn17fEVa2ev1BKmo7xikiIcaHJHwx6XX1+PD5AxHlnd3ecYhGiPEjCV9MepkpDqZlxYeVOW0mstOc4xSREONDEr6Y9BLirHzvxhMonZ2BwaAoyU/m3m8sx5UWN96hCTGmYrWWjhATWoErkX+56UTaOj04bWacdvN4hyTEmJOEL6YMm8WEzSK/8mLqkt9+MeW0d/ZwsLaDHo+f3Iw4MlIc4x2SEGNCEr6YUhpbu/jdi1v4ZGs1AElxVu79xjJm5CaNc2RCjD65aSumlF0HmkPJHqClo4dnVu+ix+sbx6iEGBuS8MWUcqghcrLVjgPNuLsl4YvJTxK+mFIKXIkRZUvnZBLvkCUWxOQnCV9MKSX5SXzpnBIMBgXAzGnJXH5GESaj/FcQk5/ctBVTig5AWpKda88tQQGuVCfxTmndi6lBEr44rnl9Aarq2mls7SYtyU5ORtyArfVNe+r53+c/Dyv77peXcOYJeaMdqhDjThK+OG75/AHe21jBg89/TmqCjUtOm85b6w9gNBhYPi+LkvxkjEcl/3c3Vkac592yCkn4YkqQhD/OPE3NtO/ZQ3dNLY68XOKKizHHyxovA2lo7WJ/ZSutnT20tPeQlmjj4lOn88fXtuMPBPfVefn9vfz4W6cwb0b4TpqF2QnsLG+ky+Mn0Fu3MDvyRu6x8Pj8HKrroNvrx5XqJDFONlgRE4sk/HHk7ehg/6OP0fjhR6Gy3GuuIu/aazCYZa2XaOqb3fz08TJ2HWgGwGhQfOPyeew+2BxK9gABDW98Uh6W8DvcHkryk+j25BHnsNDa3sPHmw+x4oTcEcfV7vbw0rt7eeHdPQQ05GXE8c83nUiBK2HE5xYiVmRowjjqqqgMS/YAlS+8RFd1zThFNPHtrWwNJXsAf0Dz2sfl5GXGR9Tt7vFxeCfNQEDz5toD/OSxDbz60Rc8/dYuDta28ZNvnRKTFv6eimaeeyeY7AEq6jp4+i2Z0CUmFkn4gK+zk+66Ovzd3WN6XX93T2RhIECgJ0q5AIIt6aPVNbvJTY9M+BeeXIhSweGXNY2dPPnmzrDj2/Y30dIRm/f6UH1nRNlnu+vocMsmK2LimPJdOu27drP/94/SsW8fSQsXUHDLTTgLpo3Jte052ZgTE/G2tobKnDOmY8vKHJPrH4+iteRXLM4lM8XGD752Eqs+3E8gEOCKFUXMm54aquP1BfD4Ine96vb4+dXq3dx1bsmI4sqMsgDbnMKUIS3D3NLeTXNbDwlxFlIT7SOKQ4iBxCThK6UeBS4G6rTW86IcV8ADwErADdyitf40Ftceia6aWrb98D/xdwZbZy2fbWJXQwPzf3wf5sTY3MgbiC0jnTn3/DsHn3qG9p27SC5dQu6VV2COj0xqIqgoJ5G7bzqRh/+yhZaOHlYsyeWqs4pxpTkpzIEls9LRGswmY9jrUhKtnDArg40760JlTpuJvIw4Hnh8I3edW4LX56eitp365i7SkhzkZ8ZhNhuPDiGq4rwkzl2az+r1BwFIjrdy4wWzB12OeUd5E798ciM1jW6S4q38w3WLWTIzI/TNRIhYilUL/zHgQeBP/Ry/ECju/XMS8Nvev8dVd3V1KNkf1lVRSXdt3YgSvrejk879++mpq8eanoZzemG/STxuxnRm/tN38Xd2YkpIwGCa8l+6BmQ2GzllYTazC1PweP2kJtrCkrvJGJmgtdZ8sqWGotwk4h0WPt9TzzRXAjecP4vs9OCIKL8/wLsbK3nwuU1oDUrBt69ayDlL84c0Czcp3sbXL5vH+csK6Pb4yE6LIz154NZ6Y2sX9/9xA01twa7ElvYefvJ/63ngH1eQmyEf+iL2YpJdtNYfKKUKBqhyGfAnHbyDtlYplaSUcmmtqwd4zagzOSK/hiuTCaP92L9W+z0eDr28ispnnw+V5Vx5OXnXfQmjNfowPaPV2u8xEV1Kgm3IdasbOnn4L1vo8fjJy4yndHYmNY1uDtV3hjY3r6rv4LcvbKb3Hi9aw+9e2szswhSmZQ1tpI3DZmbmtOQhx9XQ0hVK9od5fAFqm9yS8MWoGKubtjlARZ/nlb1lEZRStymlypRSZfX19aMalD0vl8zzzw0ry7/hOuzZrmM+Z/ehQ1Q+/2JYWdVLL9NVVXXM55xqfP4A279o5E+vbefpt3axt6J58BcNwN3to8fjB6Citp3V6w+yZV8Dhxo6+LfffgxAW6cnlPyPxKFpjdFN3WjiHRaslvBvJEoF1+gXYjRMuP4DrfXDwMMApaWlepDqI2JyOJh2w5dJXb4MT2MTtswMnDOmo6J0CwyVz90FgaNuDmqN3901wminju1fNPKDh/4WGuL43Jrd3H/HqWSlOmnv9JAYZx3WnrTpyXbyMuKoqOug1mCgrreLZsv7+6G3C+3i368D85H/Dhn+AAVmA+lJo3cT1ZXm5NtXLeC/n/4s9M3ixgtmkZshE+/E6BirhF8F9J27nttbNu7MiQkkL14Us/PZMjOxpKXiaWg8co3kJBl5M0Q+f4C/vL+PPnOo8PgCfLTpEHurWti8p4GS/GS+deV8ivKG1n2SGGflezeW8tCLn0N5MwuTgzNzn39nDy3tPWwxm9h+73nsrmjhF3/eSEt7D0lxVr775SW40kYv+SqlOG1RDtNcCdQ1dZGSYGVaVgJW2XdXjJKx+s1aBdyhlHqa4M3a1vHuvz+su64OX3sHlpRkLMlD73/tjzU1hdn/djeVf1mFLT0NS0oyCfPmYU1LG/zFAq01nVE2I2lu66auyQ3A7oPN/OSxDfz8708b8jDG6TmJ3PP15ZRXt7Fmw0Eef2NHqJsHgv3vi4rT+dU/nEFLew+J8RbSk0Z/r1uzyciMnCRm5MgWi2L0xWpY5lPACiBNKVUJ3AOYAbTWDwGvExySuZfgsMyvxuK6x8rv8WAwGmkq28jeX/8mmPDT0pj5vbtImD1rxOe3pqfjzM+l8tkXCPh8pJ9xGvnXX4ctMyMG0U9uZpORS0+bzvb9jWHlhTmJvPvpkYXP6lu6qGvqGta4dac9eFPV6w9Q19xFR5eHy06fwVef2xyqk5ZkJ20Uu3GEGE+xGqVz/SDHNfB3sbjWSLgrK6l75z1aP99M8omlBHp68LUHt7zzNDSw6+e/YOH/+ymWlJQRXad9504OPvFU6Hn9u+9jc7nIv/aaEZ13qlhcks7dN5Xy0nt7sVpMXHLadB5/Y0dYHaNB4bQP/9fXZDSwqDidOYUpBPwam9UEfRI+QGNLF7sPNlPX0kV+VjzFeUnE2WXNfHH8mzKdhZ6mJnb818/orgzeOujYu4/EhQvIu/5aKp97Ae3z4WlsoqehccQJv3Xrtoiyhvc/xHXxSsxO54jOPRjt9+Npa8PkcBy3Qz0dNjOnLMyhdE4WSkHAr9l1oJmDNe2hOjetnB0aQ38sLCZj6Lf/zrOLQ+WtHT08+NwmyvpM0PrqxXO47IwijIahT4Zqd3uwmoxYLMc+AECIWJsyCd9dWRVK9oe1fr6Z5BOWUHDzjVS/8SbellZMCcMf/3z01Hx7bnDEqdHphEAAf1cXzqLpgyZgT2sbXZWVoDX23BwsScPr1+2qOkTVqldo+mQtzhkzyP/ydcQXFw3755korIdnuZrg6rOKWDIzg8bWLjJSHBRmJ8ZsW8K+/3YHatrCkj3AE3/dyUnzXOQM4QOmoaWLdzdW8Na6A7jSnFx3zkxmF6bIzFkxIUyZhN/fUEt/ZycVz73AtBuux5aViT0ra9jnfmDNnrCkkbhgAYW3fZ3uQ9UoowFzUhJJixeiAwHcFZUoowFbZmZYTF01Nez571/TviO4wJejsICZ//RdHDlRpytE8Lnd7HvoYVo3bwGg5dPPaN+1m4X/76cjmlcwUTjtFuYXDf3Gd3unB6Ugbpibk3f3+CPKvL4AXm9k+dH8Ac0rH+3nxXf3AlDT6GbrvkZ+8fenU5gz+kt1CDGYSZfwtdZ4m5sxWCyY4o60yOy5ucTPmU379iN9wSnLltK6bTsEAiiLhZRlwdUe3AcrcFdUYLBacRYWYE1NPfoyA/I0NlL+h/9D+4NJwmi3kTB7Fvt++zD1776HMhrJueoKsi+6MLSEQ3PZp6FkD+D+opyGDz4i//prh3TNnrr6ULI/zN/ZSVdV1aRI+EPV7vbwyZZqnluzG7PJwA3nz+KEWZnBvvohyM2Iw2kzhY0UmlOYgtcXIBDQoc3Po2lq7eLVD/eHlXl9Acpr2iThiwlhUi2P3F3fwME/P8Wmu/6Jvb/7PS1btxPwBf/jWhITKLnzO0z/5tdJWb6M3GuuwhQXT1tvf7s5IR6DyUTbjp18/o//zK6f/YIdP/oJO37yU7rr6ga6bJiAz8ehl18JJXsAf1c3TevW0/LZZ6A12uej8pnnaO3z4dO2bXvEuVo2fR52noEYzGZUlE1TDMdpP/6x+nRnHb9+dhM1jW4qaju4/09lbP+iacivz06P475vnsz8GWnEO8ycsiCbxSUZ/MtvPmRvZcuArzUZDTiiTAizDnEBNiFG26RJ+DoQoPbNt2jftRvXxSvxt3dQ++abYYnUlpWJa+WFZF+8kkOrXqXu7TUAWNLTiC8pxt/dzcEnnybgObLmeufefbTv3DWsODx9ljs+zNPcjNEWPtyv5bNNocdJUSZ/pZy0dMizfm1ZmRHfBpIWL8IxLX9Ir58MfH4/r3+8P6L8482HhnWe3Iw4stMcnLowh5rGTv785k68Ps3mvQMv9ZGcYONrF88NK8tKdTJdWvdigpg0XTqe5hbq3/uAjHPO4uATT4bKm9auZ/79PyZuxvRQWfzMEubedw8du/dgiosjfmYJ9uxsPC0twZumR+npM2v2V6t388CaPRF1Cu5+7cgTdQpXpjm4quGzUFHi/PnUv/dB2GucBQWhx0mLF5J22ik0fBhc2yVpyWJST14+5J9fGY1knn8ezumFdJaXY8vMJL6kGMsYLPM8UShlICXJDoSvvTOchdaC5wnurLWvKvyD22gYvH20fL6L/0w8mW37G0lNtDFvehpZqaM7MkuIoZo0Cd9gMZNy8jLq3w9PqgGPh/adu8ISvsFsJmHWTBJmzQyra05IIG3F6Rx68eWw8riiGaHHd51bErFZRsHdr1F+/0Wh596ODpo3OKl8oR6D1Urel67BmpGOOTEBb2sbAI5p+SQtWhh6jS09naK/+zY5V14OWmNzuaKu5jkQc5yT5MWLYrpUxPHEaFBceup01m6pCS2EZreaWDZveDfi7VYz1547k588tj5UZjUbWTCEm8Y2q4mFxeksLE4fXvBCjIFJk/DN8fGknrSU5g1lkQeHOCJOGQy4LroIhaLqL6sw2m0U3PQV4oqGN7TRHBdHxpkrSFl6IhgMmHqXW57/s/txHzyIwWTCMS0/4maw0W4jbvr0aKcUQzSrIIWff+c0dpY3YTQqZhekUHAMe9YuLknnvtuW896nlSTGWVgyM4O9VS3sqWghK9XBrIKUQTc3EWKimVS/sfElxeRceQX7HvzfUJnBYiH+qJZ8NH6Ph/YdO6ld8y5Gq5W59/4Am8uFLePYW2qmoyZZ2bMyscsiaqNKKUVRXhJFeSNbm8ZmNbF4ZgYLitL46PND/PCRtfj8GrPJwPXnzURrzeKZ8m8pji+TKuEbzGbSTl6OJTGB2rffwZqWSvqZK4bUam7buo3tP/zP0PO6Ne8w7yc/GnbC9/f0oH2+iGQvjk+V9R3899Of4vMHl+/0+gI8+/ZuvnLhbObOSAvO2BUhDXUd1B5qIxDQZGUnkJ4lG7lMJJMq4QOYnA5Slp4Y7E4ZooDPR9XLr4SVab+fxk/WRvTzR3Pn2cVov5/WbdupeO4FvM3NZF98ESnLTsKSNHVumk5GzW3doWR/WLfHT0DrofYUThm11W08/tu1uDuDo9wsVhM3fWsZ2SP8tiViZ9Il/GN29KYlQGhXikHcdW4Jbbt2s+2e+0Ln2ffb3xEIBMheeUEsoxRjLDXRjtlkwOs78vvhsJmYnpMUsVH6VLdzS00o2QN4enx8uu6gJPwJZNKMwx8Jg8lE9mWXHFVoCM28HYr2nbvCPjSSTywl0NXFrl/9D5UvvIS7omKAV4uJKjs9jn+84QRsvYugOW0mvnPNImbmSxI7WmNdR0RZfU07AX+UxpQYF9LC75U4by5z7vkBNW/8FYPdRtb555Ews2TwF/Yy2o6M9bamp2PPdnHgT08A0ABUv/YG8358H3bX0IYIetvbad+1m449+7DnuIifNWtEN5DFsTEaFCfPd1HoWkFLRw/JCTZcMq4+qjkLXWz9LHyS25KT8jHEaJE7MXKS8HsZbTaSlywiecmxjWFPmD0LU3w8vvZ2Uk9eRu1bq8OOexobcZeXDynha7+fmjfe5OCfj6ypnzBvLjP/6bvDXkFTjJxSiuz0uBEtxzwVTJuRygVXzuP9v+4iENCcenYRM2bJpj8TiST8GHHk5zHvx/fRumUL5qRkat54M6LOUL/adtfWUvHMc2FlbVu34T5wUBK+mLDsDgtLTylg1rws0JoE2TlswpGEH0POafk4p+UHl0EuL6fyuRdCx4xOJ84hrmsT8PrQvsh9Xfuu8SPERJWQOLylLMTYkYQ/CpTBQNbKC7GmpVG75h2cBQVkXXAejrzciLre9g7cBw7g6+jAnp2NPS8XW0YGSaUn0FK2MVTPFB+HPTfy9UIIMVRTJuH7e3roLC+nu7oWS3ISzsICzAkJo3Y9a0oyWRecR8Y5Z6GMxqg7HnlaWyn/w2Oh9X+U2cycH/wbSQsXMP3Wr1KTk03Dx38jrqiIvGuuGvINXyGEiCYmCV8pdQHwAGAEHtFa33/U8VuAnwOH9xh8UGv9SCyuPRRaa+rf/4B9v3koVJZ5/rkU3PyVUZ8RazD1/xZ37v8ibLE37fWy76HfM//+H2PPdlFwy03kXHUlRrsNo0U20RZCjMyIx0sppYzAb4ALgTnA9UqpOVGqPqO1XtT7Z8ySPUB3TQ1fPPJ/YWW1b67GfXB8x8Z7o6yb333oEH63Gwh2DVkSEyTZCyFiIhYDZJcCe7XW+7XWHuBp4LIYnDdm/O4uAj09EeXe9siJImPJFmX/3KTFi7Aky0gcIUTsxSLh5wB9m8qVvWVHu0optVkp9bxSKq+/kymlblNKlSmlyurrB95haKis6enYjxohY7BYxn2vV+f0QorvvANj77r3cTNLKPjqzWGTuIQ43rU0u9m5tYatn1VRW9023uFMaWN10/YV4CmtdY9S6pvAH4GzolXUWj8MPAxQWlo6tMVsBmFOiGfmXXey/5FHadu6DVtuDkW334YjN9rn0tgxWixknHUmCXPn4u/qwpKehllW2RSTSFNDJ0//YQMNvcsumMwGvnL7MvIKUsY5sqkpFgm/CujbYs/lyM1ZALTWjX2ePgL8LAbXHRZnYQGzv3833tZWjA4nlsTRG6EzXLZMmY0oJqeD+5tCyR7A5w3wweo9fOnmUswWWXxurMWiS2cDUKyUKlRKWYDrgFV9Kyil+vadXArsiMF1h83kcGB3uSZUshdiMmtt6Yooa6zrwOv1j0M0YsQtfK21Tyl1B/AmwWGZj2qttyml7gPKtNargL9XSl0K+IAm4JaRXlcIMfHlFSRHlC08MQ+H89hHnjU1dNJY14HZYiLDFYfDaR1JiFOK0kNc8308lJaW6rKyKHvUCiGOCz09PrZtOsSaV3fQ0+PjhOXTWL5iOknJjmM6X9XBZp743Tp6uoNLj8ycl8mFV84jIVHW7TlMKbVRa10a7diUmWkrhBgdDXUdtDS5sdnNpGfFY7UeSStWq4klJ+VTNCsDv89PQpIdo9GA1prWli4UioQkW9SZ6Efz9PhY89rOULIH2LW1loUn5knCHyJJ+EKIY1a+r4GnHtmA1xPsk192RiGnn1OMUgqj2YCpd1ewvguqdbT38Nm6g3y0Zi9KwennFrPwxDyccQN3zXR3e6mujJys2NYceZ9ARCcJXwhxTDo7enjl2c2hZA/Q0e7hs/UVbFpfQWKynVPOLiK/MCWsBb9/Vz3vvrEr9PztV3eSmOxg7qLsAa/ncFqYOS+LzWWVYeVpmbJR+lBJwhdCHJMut5fmBnfoeUKSDafTwupXgoPw6ms72L+ngVv//hRcucHZ41prNm0IztNMTnWw4IRcAoEA3V1eAgGN1gGMxujDNU0mI6eeVURzYycVXzRjNBk48/wSsvMSR/knnTwk4QshwrS2dFFb1Up3t4/0zHgysxMwGCL72J3xVnLyk6g62AJA8exMtnwWNgWHgF9TXdUaSvhKKTKzE2ht7mJBaS4fvr0HpRQrzi/hry9tpbqylXmLc5g1P4vE5Mh++bTMOK6/dSktTW7MFiPJqc6osYnoJOELIUJam7t4/k8bQ0ncYFB8+RtLmV4SuZ+y3W5m5VXzef7xjTQ3uPH7/VgsJtx4Qq8tmZtJUpKd7i4vNruZpsZOimalY3dY+GD1bgJ+zfIVhaz78As62oLrXVUdbKG2uo2VV80L3QPoy2Y3k5UjrfpjIbsLCyFCqitbQskeIBDQvLVqO13u6LutuXIT+eodp3Drnady6llFnH3RLAAsViPnXjKHliY3Tzy8nid/v56D+xt58vfrePaxjcTFWwn4g0PCzRZjKNkf9vmGCloa3RHXEyMjLXwhRIi70xv2XBkUGa54mhrcxCf4o+5TGxdvJS4+OMImPtHGV25fRkd7D2++vA13h4dPDcCBZl544jNmzc9iw0fltDa7MZkM+HwBFJFdMkopGMJQTTE80sIXQoSkZx0Z8aIMinMvnk1jXQd/eOAjHv7Vh+zcUo3fH+j39WaLicLiNOISrLg7gt8KNhmDibu9tRubzQzA52WVnHF+CRarEXenh5S08EUDl51RSHLqsU3OEv2TFr4QIsSVk8DVN53AX1/aSt70FLZ8Wkl1ZXBJY3eHh+f+uJFv3HXaoH3oNpsZFNBnIr9ShG6wtrV0s/7Dcq748mLsTjMLT8zl4P5GDlW2UTI7g4LiVIxGaY/GmryjQogQk9nInIUubvvuaSw/Y3oo2R+mdXAtm8GkZTg59awZYWWnn1dCcpoDm92M0WRg4Ym5ZOclkpjkoOpAC+X7msgvTCYrN5G4eNkTYjRIC18IESEuwUZAa5xxFjo7wm/YDmXhM7PFxMkrZlBYks6jj6zjpm8vJys7AZvdzLQZqeiAJjHJjs/n57Xnt7B5Y3A45+5ttaRmOPnKN5dFvV8gRkYSvhAiqoREOxddPZ/n/riRw2ssLj4pj8zs/pcX/9Xq3TywZk9E+Yrfrw17fufZxdx1bgnNje5Qsj+ssa6T+toOSfijQBK+EKJfJXMy+cZdp9HU0IkjzkKmKwG7o/8W/l3nlnDXuSVhZQV3v0b5/Rf1f5Gj+vpBBuiMFkn4Qoh+GYwGsnISR22iU3Kqg0WluWzacGR9nLSsONJlfZxRIQlfCDFuzBYTKy6YSW5BMju31jJtegqz5mcRnzhxbtr62hrx93RiikvFaI/cczrg7UGZzCg18cfASMIXQoyrhCQ7S5ZNY8myaeMdShjt99G5dyMNrz9EwN2GxVVM+kW3Y80sAMDbWk/n9o/p2PYR1pwSEpacFzo2UUnCF0IMqKO9h872HuxO86AbjTTUdbB90yEO7G9i9gIXxXMyuPPs4jGKNLY89Qepe+H/gQ5ONPNU76H+tYdwXf8DlNlM8wfP0LH53eCx2i9w71pL9s3/hTk5M+w8vo4WMBgwOcZ/L21J+EJMYVprerq8mK2mqBOdKsqbeOnJTbQ0uolLsHLZdYuYXpIWdYeqtpYunn50A031wXH6X+xpYFFFHt+5ct6Q42lq6GTvzjoOVbQwY2YGBUWpxCeMT/eOt7kmlOwP81Tvwd/RBErRsfm9sGP+zlY8DRWhhO/rbKVz20e0fPISymgmecWXcRaXYrCO3+ijid/pJIQYFY31nax5bSePPPAxrz23mdpD4ZOs2lq6eO6PG0OLmHW09fDsY2X9Tryqq2kPJfvDNm2ooKlx8IlaAO1t3Tz/+Kf89aVtbC6r4qU/f8bHa/bi8/kHf/EoMDoib1QbnYkoqwOUAQyRK3mqPmVdezfSuPpR/B3N+FrrqH/5v+mu3DmqMQ8mJglfKXWBUmqXUmqvUuruKMetSqlneo+vU0oVxOK6Qohj09Pt5Y2XtvK3d/fR1NDJpg2VPPnIelr6bBfY1tIdsYql1+PvdxXLaK1+BVEXR4umvqadmqO2MNzwcXnYJitjyZIxjfhF5xwpUAbSLvwm5oRUzMmZJC67NKy+OS0fc3o+AAGvh7aNf404Z+fuDaMa82BG3KWjlAfoRbIAACAASURBVDICvwHOBSqBDUqpVVrr7X2q3Qo0a62LlFLXAT8Frh3ptYUQx6a50c3+XfVhZe2t3TTWtZPUu/GI3WnGZDbg8/bp1lDg6Gfv2fTMOFIznDTWHWnRL1qaR3ySjS/2NPDpuoMYDIolJ+WTW5Ac0YUUCOijT4nW0cvHgtEeR0LpSuwF81FmK6bEdCzpeUCwJZ+49GKs6fm4v9iM1TUd+/RFmBNSg8eNRoxJmVC9L+ycpsS0Mf85wq4fg3MsBfZqrfcDKKWeBi4D+ib8y4B7ex8/DzyolFJa6/H5lxRiijMaDSiDQh+VTPtuL5iS6mTllfNY9ezm0MSosy6YSXpmXNRzJiTZufaWUnZsqeHg/kZmL3BRNCuDQwdbeOJ36wDILUim8kAzn2+oJDHZzsx5maEx/umZ8SQm2Wht6Q6dc85CF8lp47NqZlf5Fmqe/S+0N/gtJ3H5FSSdfCVGmwN/dyedO/5G8wfPYIxLwhSfgsF0ZEKaMhhJWnoRXbs3oP3BJadNqTlYc0poLXsDZTRhzSnBmjG2I5NikfBzgIo+zyuBk/qro7X2KaVagVSg4eiTKaVuA24DyM/Pj0F4QoijJac5OOm0Qta+vz9UVlCUSnrWkWSuDIp5i3PIzE6gpamL+EQbGVnxmMzR95yF4Ibip2XGA8GROVpr6nfv4saLTGiTjWaji9dfPNIWXPvBfm6542QyXQkkJtu5/hsnsWn9QQ7sa2LuIhezF2ZjsYz92BJfZwv1r/5vKNkDtH7yEo6iJdjz59BdsYPGNx8BINDVTkt9BUZHAolLj8wotubMJPuWn9BTvR9lNGFMSKXmqR+B3weAsjrIvvE+rFmFY/ZzTbhROlrrh4GHAUpLS+UbgBCjwGQycvKZM8grSKaivIlMVwLTZqTiPKq7xmQ24spNIjUjjp5u34DJPpqeqt0Y3vsZ2tuDZfaZrN8T/l+6p9tHZXkzma7gkMWMrHjOvWQOfl9g2NeKpUBXB77Wuohyf0czAF37N0Uca/98DXGLzsZoCY4qUkphzZqONWs6OuCn7uUHQskeQPe46dy9/rhL+FVAXp/nub1l0epUKqVMQCLQGINrCyGOUVy8ldkLXMxe4BqwXmV5M++8sZP6mnbmLs5m6amFERuWRBPwemj+6PkjrWRlwueLbMP5feFDH5VSo5bstQ7gbTyEr60RU1wS5tRslNEcUc/oTMKcWYC3tjys3JSUAYA5JfI9M6flYTD2k1K1xtcamfL87U3D/yFGIBajdDYAxUqpQqWUBbgOWHVUnVXAzb2Prwbekf57ISa+htp2Hv/dWsr3NtLZ4WH9h+WseW0HXs/gQyW1txtvw5E1cvz7P2HZCeGTj4xGA7kFyTGPuz/u3WVU/eGfqHnqPiof+R7tm99D+7wR9Yz2ODIu+jamxOCYemWykLbydizpwT53e+EiTIkZofrKYifxpEtQ/SR8ZTSRUHpBRLlz1vJY/FhDNuIWfm+f/B3Am4AReFRrvU0pdR9QprVeBfwBeFwptRdoIvihIISY4OprOyKS+44tNZx5gZu0QRY4M9jjiVtwBi0fPgdAoLuDzJb3ufxLV7BxfTXxiTaWnlaIK3d0FmY7mre5hrpXfo329a7vrwM0vPEw1uziqEsiWF0zyL7lJ/ha6zHYnJhTskLr5VjScnDdeC+e2nK034clIx9LWl7EOfpyTF9E2srbafn4xeBErNOvxZY7M9Y/5oBi0oevtX4deP2osv/o87gbuCYW1xJCjB2LNbJrxWIxYjQN3uWilCJ+wVn4O1pp3/Q2BouNpOJ55M5zMfeEaSiDIbTl4Vjwu9vQPUeN6deBYL98b8LXAT89VXvp3LMBZTThKD4Ba3ZR1DkG5qRMzEmZEeXReBoP4W2qwpySTdaNP8RodWC0Rx/tNJom3E1bIcTEkeFKIK8wmYovmkNlZ62cRVLK0JYHMCdlkHb+rSQtvwxlNGFKGL9x6Ma4JAz2eAJd7UcKDSZMvWPnAbordlL953tDSyq0fPIS2V/5EbacI2v8/2r17og1/wfSfWgP1U/eF/qwsRUuIP3iv5OEL4SYWOITbFxxw2KqDrTQ2txFVk4iOfmJUVu8/VFGE+bkrFGMcmjMiRlkXH4XdS/9kkB3B8psJf2iv8OcmgMEh5C2lr0evn6O30fn9o/DEv4Da/YMOeEHPN00v/dU2DeL7i8203NoL+Zx+PCThC+EGFBSsoOk5PGZ/BRrjukLybn15/jamzA6EzAnu458eOkAgZ6uiNf4j+4GGoaAp4ue2i8iyn0tkUM+x4IsniaEmFLMSRnY82ZhSckO+6aiDEYST1wZUT9u3unHfC2jPR7nrGUR5ZbM8Vn7X1r4QogpzdtSh/Z5MCWkY8+fS+bV/0LL2pdRJjNJyy7HlnPsI2mU0UTS0kvwtdTStf9zlMlC8hnXYXUVxfAnGEY8E3k4fGlpqS4rKxvvMIQQk0DA6+ndijDYqg/0dNG+7UOa3nkc3ePGMfMkUs76CpYUF4HeoZsPvFvOA2v2DHruO88uHrBf39/jxtfagDKZMSdnjup2iEqpjVrr0qjHJOELISYzb3MtHds/onPnWuzT5hK/8Gws6Xl0HdhK9RP3hNWNX3IBaed/LWxdewBPYxWt61/FvWs9Z9Rcya47Z2HJKsTX1gAYMI/zKph9DZTwpUtHCDFp+Xu6aFj9KF17gg1HT81+OndvwHXjfXjqDkbU79z+EcmnXoUpPuXIObo7aXjjd3Qf2BYqa/tsNUZ7PK3rXwWDkeTTryV+/gqMjoEno403SfhCiEnL11wTSvZ9yzx1BzD2SeqHWdLyUJbwLRV9rXVhyR6CSy+0/O3F0POmtx/DlJhOXJQbtMPRU1tO1/5N+DtbcRQtwZpTgsEcff+BYyGjdIQQk5fBAFF23Ar0dGJ1FWPNmxMqUyYLyWfdgNEaPgRVGS1gCG8bdx31AQDg3rV2RKF66g5S/cR/0PTO47SuW0X1n++la//nIzrn0aSFL4SYtMzJWcTNP4OOLe+FyixZhXiq9+OYvpjMK7+Lp7acgKcbS3pu1PVwzClZJJ96Nc0fPA3AV5P2YUrOoqdqd1g9Y3wq/h53xAfGUHVX7iTQHb7/b/MHz2CbNhejbfDVSYdCEr4QYtIymK04iksxJabhqa8IzvjVAboObCXptC9hssdhiht4tU5lMJJQegHW7CJ6asv5x+QsjM5EuvZuDCVoY1wyBrMNb91BjHmzjinWgLcnSll3+MzfEZKEL4SY1Mwp2TS8+XuM9ni6vtiM9nSRec3dGK1DWw8IghOoHDMW45ixGICe+oMklK4EFdykPeDz0vLxC5jTc485TlvuLDAYIXBkddKkZZdjtMfuRrAkfCHEpGbNnIbruh/g3vcpAXcbjuJSrCOYTAVgcibRuWsd3vq+I30U5qRjXzPI6pqO68v30LL2ZfwdTSSeeBGOohNGFOfRJOELISY9a1ZhTLcSNDoSyLjkDmr/8t/4mg5hsDpIveCbWEbQwlcGI/Zpc7HmlIAOxHR0zmGS8IUQk4bf3UbXwe24927EmjEN+4zFWHpXw4w1q2sG2Tf9J/62Rgx255DXxh+MwRS55WKsSMIXQkwKWgdo2/hmaDRNB2AqewPXDfdiTkwflWuanImYnGOzY1csyDh8IcSk4GupC5sMBUcmWYkgSfhCiElBBwJovy9K+eAbrk8VkvCFEJOCOTGD+CXnhpUZ7PFY0vPHKaKJZ0R9+EqpFOAZoAAoB76ktW6OUs8PbOl9elBrfelIriuEEEdTJhNJy6/EkpJD+5b3sLqKSFhyHpYU16hdU/t9BHyeY55dO9ZGtDyyUupnQJPW+n6l1N1Astb6X6LU69BaD3vHXlkeWQhxLIJr35tGdd357kN7aV33Cp66cuLnr8A599RRuzk8HKO5PPJlwIrex38E3gMiEr4QQowlg9kyquf3NFRR/eQPQ5uTN737BN7WOtLO+xrKOHrDKkdqpB9/mVrr6t7HNUB/A1FtSqkypdRapdTlA51QKXVbb92y+vr6EYYnhBAjF/B203VwO60b36Rz51q8bQ2hZH9Y+2dv421tGKcIh2bQFr5S6m0g2nzh7/d9orXWSqn++oemaa2rlFLTgXeUUlu01vuiVdRaPww8DMEuncHiE0KI0da54xPqX3kw9NyaO4v4hWfT/vmaUJkyWSJ2yppoBk34Wutz+jumlKpVSrm01tVKKRdQ1885qnr/3q+Ueg9YDERN+EIIMZH4WhtoXP1YWFlP5U4cJSeGlSWfcR2mCdCHP5CR9uGvAm4G7u/9++WjKyilkgG31rpHKZUGnAL8bITXFUKIMRHweSLWqQcwxSWTduHteBsrsRUswJY7M7RB+kQ10oR/P/CsUupW4ADwJQClVClwu9b668Bs4HdKqQDBewb3a623j/C6QggxJkwJqThnLaNz5yehMmU0Y80sxJJxfI3xH1HC11o3AmdHKS8Dvt77+G/A/JFcRwghxovBbCX5zBsw2OPo2PYR5tQcUs++CXN65O5YE50sniaEEIOwpLhIO//rJJ96Dcpii9mWg2NNEr4QQgyBMpowJaSOdxgjImvpCCHEFCEJXwghpghJ+EIIMUVIwhdCiClCEr4QQkwRkvCFEGKKkIQvhBBThCR8IYSYIiThCyHEFCEJXwghpghJ+EIIMUVIwhdCiClCEr4QQkwRkvCFEGKKkIQvhBBThCR8IYSYIiThCyHEFDGihK+UukYptU0pFejduLy/ehcopXYppfYqpe4eyTWFEEIcm5G28LcCVwIf9FdBKWUEfgNcCMwBrldKzRnhdYUQQgzTiPa01VrvAFBKDVRtKbBXa72/t+7TwGXA9pFcWwghxPCMRR9+DlDR53llb1lUSqnblFJlSqmy+vr6UQ9OCCGmikFb+Eqpt4GsKIe+r7V+OdYBaa0fBh4GKC0t1bE+vxBCTFWDJnyt9TkjvEYVkNfneW5vmRBCiDE0Fl06G4BipVShUsoCXAesGoPrCiGE6GOkwzKvUEpVAsuB15RSb/aWZyulXgfQWvuAO4A3gR3As1rrbSMLWwghxHCNdJTOS8BLUcoPASv7PH8deH0k1xJCCDEyMtNWCCGmCEn4QggxRUjCF0KIKUISvhBCTBGS8IUQYoqQhC+EEFOEJHwhhJgiJOELIcQUIQlfCCGmCEn4QggxRUjCF0KIKWJEa+mIiSsQCFDdUUtrdwcp9iSy4tPHOyQhxDiThD8J+QJ+Pj6wgYfL/ow34MNutnHXyd9gUZZsJSzEVCZdOpPQobYafrvhcbwBHwBd3m5+/cmj1Hc2jnNkQojxJC38SajR3UxAB8LK2j2dtHS3ke5MDStv6GziQEsl3oCfvEQXOQnRdrMUQkwGkvAnoRRHEgZlCEv6cRYnSbaEsHrV7XX87MPfUtVeA4DdbOM/VtzJjJSCsQxXCDFGpEtnEsqJz+K20hswGowAWE1W7jjplojW/ba6XaFkD8Gun1d2vo3P7xvTeIUQY0Na+JOQyWjijIKTKE4toLW7nVRHMllxkaN0DrXXRZSVt1Ti8XsxGeVXQ4jJRlr4k5TRYCQvMZt5mTNxxWeglIqoMze9JKLsjIJlOCz2sQhRCDHGJOFPYTPTZ3DzoquxmawYlIFzZ5zGqdNOHO+whBCjZETf25VS1wD3ArOBpVrrsn7qlQPtgB/waa1LR3JdERtxFgcrS87ixNxF+AN+0h0p0pUjxCQ20v/dW4Ergd8Noe6ZWuuGEV5PxJhSioyjbuYKISanESV8rfUOIGr/sBBCiIllrPrwNfCWUmqjUuq2gSoqpW5TSpUppcrq6+vHKDwhhJj8Bm3hK6XeBqJNv/y+1vrlIV7nVK11lVIqA1itlNqptf4gWkWt9cPAwwClpaV6iOcXQggxiEETvtb6nJFeRGtd1ft3nVLqJWApEDXhCyGEGB2j3qWjlHIqpeIPPwbOI3izV4wBj89Dl7d7vMMQQkwAIx2WeQXwayAdeE0ptUlrfb5SKht4RGu9EsgEXuq9sWsCntRa/3WEcYtB+Pw+ttXv5sXtf6XT4+aSmedwQvZ84qzO8Q5NCDFOlNYTt5u8tLRUl5VFHdovBrGjfi/3vvNLNEf+fe846WZOL1g2jlEJIUabUmpjf3OdZJbNJPV5zfawZA/w6q41nJizCLvZNuzz1XTUs71uN9XtdcxOL2ZmWiFOi3xbEOJ4Igl/knKYItfDcZodGA3Dv23T6G7m5x89REXrIQBe3vkWNy+6mpUlZ8kcDCGOI7KWznGmyd3MnsYvqO0YeI7CgqxZ2E1HWvIKxRVzLsBitAz7mgdaKkPJ/rCnt75CfWfTsM8lhBg/0sI/juys38uv/vYIzd2t2M02bj/xKyzNWRha976lu40ubxdJtkQKkvP44Vn/yJbaHbi93SzImk3xMW5s4o2yPr7H78GvZd18IY4nkvCPE03uFn71STDZQ3Czkv/55A/89Lx/IzfRxeaaHfy+7Enq3U3MzSjhq4u/REFyLgXJuSO+dl5iNnazLWx454qC5aQ5ZA0eIY4n0qVznGjuaqW5qzWszK8D1Lub2Nd4gJ9+9Fvq3cEulm11u3lowxN0etyhuh6fl+11e3hl52o+KF9HTfvQl63ITsjkB2fcyUm5i8mKS+dL8y7hqjkXYpaVNYU4rsj/2ONEvM2J0+yg03skiSsUaY5kttftwR/wh9Xf21TOz9/czr2XBEdnfVq9hV/+7feh4664DL5/xnfIiEsb0vWLUgu4c9nX6PZ7iLM4hvQaf8DPnsZy1lV+ikEZOCl3MUUpBRiO4caxEGLk5H/ecSLDmca3ln4l1F+vUNy48ArsJjvdvsiZtE6zg8c+rgWgtbudP216Iex4dUcd+5sPDisGk9E05GQPsKthP/e8+wte2/0Or+x6m3ve+QV7mr4Y1jWFELEjLfwYKW+uYFvdbjx+L/MyZzIjeVrMW7Kl2Qv42Xn/Rn1nE0m2BHITXHgCXhrcLZRmL6Ds0OZQ3RsWXs6769sA8AW8tHs6I87X7euJaXxHW73vA/pO7PPrAO+Xr2Nm2oxRva4QIjpJ+DGwv+kg97z7S3p6E6hxq4EfrPgH5mQUx/Q6BoOBvMRs8hKzQ2UWzCzNXcjbez/i2nmX4g14SbDGsSBzNrAuGI8ysqJgGW/ufT/0OqMykJ+YE9P4jtbj80QpG90PGSFE/yThx0BZ1eawRObXAV7dvYaStOmYertgRtO8jJnEWZxUtlVjN9nISciipnec/s76fdR3NqLRnDvjdMqqPifFkcQlM8+JyQiegZxXdHrYtw4Iju4RQowPSfgx0OZpjyzraUfrADCyhN/a3ca+poM0d7WQGZfO9OR8HJbwWbRGg5FVG3t4YE1jb0lV6NgFv9jZ+ygBgFNmlzI9pa53mYWFGIyjdxtndnoRd5/2bV7b/Q4GDFw08yxmSXeOEONGEn4MnJS7mLf2hi/vv7L4TMxGc1hZfWcTexu/oKmrlYLkHGakFGAzWfs9b6fHzeObXuSDA+tCZTcsuIKLZ54dunl72F3nlnDXuSXsbtjPv6/5OQDrP7uYM5e+w1nTT+aVXW8D4AU2HoILS87EZBjdf36rycqS7PkszJyDVozJtx0hRP8k4cfAzNQZ/POp3+KF7a/j8Xu5fNZ5LMyaHVan2d3CA588wu7GI6NUbj/xRs6afkroudaaencT/oCfdEcKlW3VYcke4Jmtr1CavYCcxGibkEFHn7H3AJ1eNyaDibwEFxVt1QCk2JM4u/CUMVsHx2iURC/ERCAJPwYsJjOlOQuYlzGTgA5EdLkAlLdUhiV7gMc3vcj8zNmkO1Po9Lh5v3wtT29Zhcfv5cyC5Zw6bWnEeXwBH93+/m98uuIzsJmsYSNwyqo2871Tb6eus4FAIEBuoot0p8ySFWKqkYQfQzZz/90z0YZAur1d+AJeAHY37uexz54LHVvzxcckOxLJS8imou3IwmXTEnPIcPY/WcoVn8G/nX4Hj2x8mvUEb+jevPhqXPEZuOIzjuGnEkJMFpLwx0huoguz0YzX7w2VnZxfSqo9BQhuWHK0jw5s4B+Wf53/+/RZ9jcfYJFrLtfOu5R4qxOtNfubD7K5ZgcBHWBB1mxmpEzDoAzMSi/i3jPvIsm/h++deja2Y1j/Xggx+UjCHyN5idn84Iy/58nNf6GqrYZTC5ZyYdGZWEzBG7tZcekRr8lPyiEvMZt/PePvcHu6iLfGYTUFlzfe21TOve/8Em8guGLl89te456z7mJWWhEAcVYn/37RomHFGNABypsrKG+pxGqyMj05X74VCDGJSMIfQ7PSi/jX0++g29dDojU+bCbunPQS8hKzQ+vO2002Lp91PmajCbPRhMMcfl/gowMbQskegmP/V+/9MJTwj8WO+r3853sP4NcBANLsKfz7iu+QnRD9BrEQ4vgiCX+M2c22qFsMZsWn86+n38GBlkq8fi95idnkDJBoO6IsldDW00FABzAoA50eNxajZcgrWnb7enhu66uhZA/Q0NXEjvq9kvCFmCRGlPCVUj8HLgE8wD7gq1rrlij1LgAeIDgL6RGt9f0jue5kleZIJs2RPKS6pxecxIcH1oeVnV90Oo3uZj4sX8f75evIT8rmslnnU5RaMOj5vH4fDe7miPLW7rYhxSOEmPhGOs1yNTBPa70A2A3869EVlFJG4DfAhcAc4Hql1JwRXnfKm5U2g38+9XaKUgqYnpzPd0/+BrPSinhx2xs8vfUVqjvqWFe5iR+99wCVvePvBxJvdXJ+0emR10k/9i4iIcTEMqIWvtb6rT5P1wJXR6m2FNirtd4PoJR6GrgM2D6Sa091VpOVRVlzmZVWjMlgxGa2UtNex7vln4TV6/J1U9laTW6Ca9BznpJ/Ij1+D6/vfhen2c4NC6+gKKWw3/p1HQ2Ut1TiC/giFnUTQkw8sezD/xrwTJTyHKCiz/NK4KT+TqKUug24DSA/Pz+G4U0e/oCfHfV7WbXzLdp7Orlo5tkscs3FaDBiMZojxvwfvcRDf1IcSVw1ZyVnFZ6CyWgiwRrXb91D7bX85P1fU9cZXL/HZrLygxV3Upza/weEEGJ8Ddqlo5R6Wym1Ncqfy/rU+T7gA/480oC01g9rrUu11qXp6ZFDFUVwSOaP3n+ATTXb2dd8gP9Z+yifHtpCmiOFa+dfElY3N8HFtGEsg6yUIsWRNGCyB9hSszOU7CF40/eVnW/ji7LhuRBiYhi0ha+1Pmeg40qpW4CLgbN1390ujqgC8vo8z6Xvco5i2LbU7uTot3rVztWcmLOAFQXLyY7PYnvdblzxGcxJLybNmRLzGGo66iLKKtuq8QS8mGSvWyEmpJGO0rkA+GfgDK21u59qG4BipVQhwUR/HfDlkVx3qrMaLRFlNrMVpQw4zVYWu+ay2DV3VGNYkDmb13a/E1Z2VuHJEfMFhBATx0hH6TwIxAOrlVKblFIPASilspVSrwNorX3AHcCbwA7gWa31thFed0qbnzkrYlnlq+asHHCp5VgrSZvBrSdch9PswGQwcenMc1mef8KYXV8IMXwqei/MxFBaWqrLysrGO4wJ6YvmCj6r3kqHx01p9nyKUwuHfHM2lho6mwloP2mOlJjv4SuEGD6l1EatdWm0Y9LZepwqTM6jMDlv8IqjLM05tIliQojxJ00yIYSYIiThCyHEFCEJXwghpghJ+EIIMUVIwhdCiClCEr4QQkwRE3ocvlKqHjgQo9OlAQ0xOtdYO55jB4l/PB3PsYPEfyymaa2jLkQ2oRN+LCmlyvqbjDDRHc+xg8Q/no7n2EHijzXp0hFCiClCEr4QQkwRUynhPzzeAYzA8Rw7SPzj6XiOHST+mJoyffhCCDHVTaUWvhBCTGmS8IUQYoqYtAlfKXWNUmqbUiqglOp3WJRSqlwptaV3A5cJsfj+MGK/QCm1Sym1Vyl191jGOBClVIpSarVSak/v31HXUFZK+Xvf901KqVVjHWeUeAZ8P5VSVqXUM73H1ymlCsY+yuiGEPstSqn6Pu/318cjzmiUUo8qpeqUUlv7Oa6UUv/T+7NtVkotGesYBzKE+FcopVr7vPf/MdYxhmitJ+UfYDYwE3gPKB2gXjmQNt7xDjd2wAjsA6YDFuBzYM54x94b28+Au3sf3w38tJ96HeMd63DeT+DbwEO9j68DnhnvuIcR+y3Ag+Mdaz/xnw4sAbb2c3wl8AaggGXAuvGOeZjxrwBeHe84tdaTt4Wvtd6htd413nEciyHGvhTYq7Xer7X2AE8Dl41+dENyGfDH3sd/BC4fx1iGaijvZ9+f63ngbKWUGsMY+zORfxcGpbX+AGgaoMplwJ900FogSSnlGpvoBjeE+CeMSZvwh0EDbymlNiqlbhvvYIYhB6jo87yyt2wiyNRaV/c+rgEy+6lnU0qVKaXWKqXG+0NhKO9nqI4O7tXcCqSOSXQDG+rvwlW9XSLPK6XGf7u0oZvIv+tDtVwp9blS6g2l1NzxCuK43uJQKfU2kBXl0Pe11i8P8TSnaq2rlFIZBDdj39n7iT2qYhT7uBko/r5PtNZaKdXf2N9pve/9dOAdpdQWrfW+WMcqAHgFeEpr3aOU+ibBbypnjXNMU8WnBH/XO5RSK4G/AMXjEchxnfC11ufE4BxVvX/XKaVeIvj1eNQTfgxirwL6ttJye8vGxEDxK6VqlVIurXV171fvun7Ocfi936+Ueg9YTLAvejwM5f08XKdSKWUCEoHGsQlvQIPGrrXuG+cjBO+zHC/G9Xd9pLTWbX0ev66U+l+lVJrWeswXhZvSXTpKKadSKv7wY+A8IOqd9gloA1CslCpUSlkI3kQc95EuvVYBN/c+vhmI+MailEpWSll7H6cBpwDbxyzCSEN5P/v+XFcD7+jeu3LjbNDYj+rzvhTYMYbxjdQq4Kbe0TrLgNY+XYYTnlIq6/C9HqXUUoJ5d3waCuN913i0/gBXEOzr6wFqgTd7sntrSAAAAMJJREFUy7OB13sfTyc4ouFzYBvB7pTjIvbe5yuB3QRbxRMi9t64UoE1wB7gbSClt7wUeKT38cnAlt73fgtw6wSIO+L9BO4DLu19bAOeA/YC64Hp4x3zMGL/r97f8c+Bd4FZ4x1zn9ifAqoBb+/v/a3A7cDtvccV8Jven20LA4y6m6Dx39HnvV/7/9u5YwIAYBiGYcE1/sAGo4clBL185Oi2d3Wr1woAEelJB6BE8AEiBB8gQvABIgQfIELwASIEHyDiA7VS73Y/HcngAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## K-nearest Neighbors (KNN)\n",
        "The k-nearest neighbors (KNN) algorithm is a simple, easy-to-implement supervised machine learning algorithm that can be used to solve both classification and regression problems.\n",
        "\n",
        "The KNN algorithm assumes that similar things exist in close proximity. In other words, similar things are near to each other.\n",
        "\n",
        "There are other ways of calculating distance, and one way might be preferable depending on the problem we are solving. However, the straight-line distance (also called the Euclidean distance) is a popular and familiar choice.\n",
        "\n",
        "+ Initialize K to your chosen number of neighbors\n",
        "+ For each example in the data\n",
        "    + Calculate the distance between the query example and the current example from the data.\n",
        "    + Add the distance and the index of the example to an ordered collection\n",
        "+ Sort the ordered collection of distances\n",
        " and indices from smallest to largest (in ascending order) by the distances\n",
        "+ Pick the first K entries from the sorted collection\n",
        "+ Get the labels of the selected K entries\n",
        "+ If regression, return the mean of the K labels\n",
        "+ If classification, return the mode of the K labels"
      ],
      "metadata": {
        "id": "UrH-rhL7hL6i"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "\n",
        "class KNN(object):\n",
        "    def __init__(self):\n",
        "        pass\n",
        "\n",
        "    def euclidean(self, q, x):\n",
        "        # l2\n",
        "        return np.sqrt(np.sum((q - x) ** 2, axis=1))\n",
        "\n",
        "    def manhattan(self, q, x):\n",
        "         |x1 - x2| + |y1 - y2|.\n",
        "    \n",
        "    def mode(self, x):\n",
        "        vals, counts = np.unique(x, return_counts=True)\n",
        "        return vals[np.argmax(counts)]\n",
        "\n",
        "    def fit(self, x, y, query, k):\n",
        "        dist = self.euclidean(q, x)\n",
        "        dist_with_index = sorted([(dist[idx], idx) for idx in range(len(dist))])\n",
        "        return self.mode([y[idx] for val, idx in dist_with_index[:k]])\n",
        "\n",
        "knn = KNN()\n",
        "\n",
        "n = 100\n",
        "x = np.random.rand(n, 2)\n",
        "y = np.random.randint(2, size=n)\n",
        "q = np.random.rand(1, 2)\n",
        "\n",
        "print(knn.fit(x, y, q, 9))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AR6Wnc109T-C",
        "outputId": "56211c58-dbf5-4860-c234-b5ebf1a3ffea"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Linear Regression**\n",
        "### Introduction\n",
        "Linear Regression is a supervised machine learning algorithm where the predicted output is continuous and has a constant slope. It’s used to predict values within a continuous range, (e.g. sales, price) rather than trying to classify them into categories (e.g. cat, dog).\n",
        "\n",
        "The main type of linear regression is \n",
        "$$f(x)=w_{1}x1 + w_{2}x2 + w_{3}x3 + b$$\n",
        "which $w$ represents the weights, our model will try to learn. The variables $x1, x2,x3$ represent the input data's attributes. The variable $y$ represents our prediction.\n",
        "\n",
        "### **Making Predictions**\n",
        "Let's say our predict function outputs an estimate of ctr(click-through rate)  given our current weights and some attributes $x1,x2,x3$. \n",
        "```python\n",
        "def forward(self, x):\n",
        "    y_pred = np.matmul(x, self.w) + self.b\n",
        "    return y_pred\n",
        "```\n",
        "### **Cost Function**\n",
        "Now we need a cost function to audit how our model is performing. We use MSE function. \n",
        "$$MSE=\\frac{1}{2n}\\sum_{i=1}^{n}(y_i - (w_{1}x1_{i} + w_{2}x2_{i}+w_{3}x3_{i} + b))^2$$\n",
        "Our model will try to identify weight values that most reduce our cost function.\n",
        "```python\n",
        "def cost_function(self, x, y_true):\n",
        "    n = len(y_true)\n",
        "    y_pred = self.forward(x)\n",
        "    sq_error = (y_pred - y_true) ** 2\n",
        "    return 1.0 / (2 * n) * sq_error.sum()\n",
        "```\n",
        "\n",
        "### **Gradient Descent**\n",
        "To minimize MSE we use Gradient Descent to calculate the gradient of our cost function. Gradient descent consists of looking at the error that our weight currently gives us, using the derivative of the cost function to find the gradient (The slope of the cost function using our current weight), and then changing our weight to move in the direction opposite of the gradient. We need to move in the opposite direction of the gradient since the gradient points up the slope instead of down it, so we move in the opposite direction to try to decrease our error.\n",
        "\n",
        "Again using the Chain rule we can compute the gradient vector of partial derivatives describing the slope of the cost function for each weight.\n",
        "$$f^{'}(w_{1}) = -x1(y - (w_{1}x1 + w_{2}x2 + w_{3}x3 + b)$$\n",
        "$$f^{'}(w_{2}) = -x2(y - (w_{1}x1 + w_{2}x2 + w_{3}x3 + b)$$\n",
        "$$f^{'}(w_{3}) = -x3(y - (w_{1}x1 + w_{2}x2 + w_{3}x3 + b)$$\n",
        "```python\n",
        "def compute_gradients(self, x, y_true):\n",
        "    y_pred = self.forward(x)\n",
        "    difference = y_pred - y_true\n",
        "    gradient_b = np.mean(difference)\n",
        "    gradients_w = np.matmul(x.transpose(), difference)\n",
        "    gradients_w = np.array([np.mean(grad) for grad in gradients_w])\n",
        "```\n",
        "\n",
        "### **Train**\n",
        "Training a model is the process of iteratively improving your prediction equation by looping through the dataset multiple times, each time updating the weight and bias values in the direction indicated by the slope of the cost function (gradient). Training is complete when we reach an acceptable error threshold, or when subsequent training iterations fail to reduce our cost.\n",
        "```python\n",
        "def __init__(self):\n",
        "    self.w = np.random.rand(feature_num)\n",
        "    self.b = np.random.rand(1)\n",
        "\n",
        "def update_weightes(self, x, y_true, learning_rate):\n",
        "    gradients_w, gradient_b = self.compute_gradients(x, y_true)\n",
        "    self.w = self.w - learning_rate * gradients_w\n",
        "    self.b = self.b - learning_rate * gradient_b\n",
        "\n",
        "def train(self, x, y):\n",
        "    epoch_num = 1000\n",
        "    learning_rate = 0.01\n",
        "    for epoch in range(epoch_num):\n",
        "        model.update_weightes(x, y, learning_rate)\n",
        "```\n",
        "\n",
        "### **Normalize**\n",
        "As the number of features grows, calculating gradient takes longer to compute, and the gradient may be very huge and we may meet some overflow err. We can speed this up by “normalizing” our input data to ensure all values are within the same range. This is especially important for datasets with high standard deviations or differences in the ranges of the attributes. Our goal now will be to normalize our features so they are all in the range -1 to 1.\n",
        "```python\n",
        "def normalize(self, x):\n",
        "    for feature in x.T:\n",
        "        fmean = np.mean(feature)\n",
        "        frange = np.amax(feature) - np.amin(feature)\n",
        "        feature -= fmean\n",
        "        feature /= frange\n",
        "    return x\n",
        "```"
      ],
      "metadata": {
        "id": "cixrSV0Gtzqr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "\n",
        "feature_num = 3\n",
        "\n",
        "class LinearRegression(object):\n",
        "    def __init__(self):\n",
        "        self.w = np.random.rand(feature_num)\n",
        "        self.b = np.random.rand(1)\n",
        "\n",
        "    def forward(self, x):\n",
        "        y_pred = np.matmul(x, self.w) + self.b\n",
        "        return y_pred\n",
        "    \n",
        "    def normalize(self, x):\n",
        "        for feature in x.T:\n",
        "            fmean = np.mean(feature)\n",
        "            frange = np.amax(feature) - np.amin(feature)\n",
        "            feature -= fmean\n",
        "            feature /= frange\n",
        "        return x\n",
        "\n",
        "    def cost_function(self, x, y_true):\n",
        "        n = len(y_true)\n",
        "        y_pred = self.forward(x)\n",
        "        sq_error = (y_pred - y_true) ** 2\n",
        "        return 1.0 / (2 * n) * sq_error.sum()\n",
        "    \n",
        "    def compute_gradients(self, x, y_true):\n",
        "        y_pred = self.forward(x)\n",
        "        difference = y_pred - y_true\n",
        "        gradient_b = np.mean(difference)\n",
        "        gradients_w = np.matmul(x.transpose(), difference)\n",
        "        gradients_w = np.array([np.mean(grad) for grad in gradients_w])\n",
        "\n",
        "        return gradients_w, gradient_b\n",
        "    \n",
        "    def update_weightes(self, x, y_true, learning_rate):\n",
        "        gradients_w, gradient_b = self.compute_gradients(x, y_true)\n",
        "        self.w = self.w - learning_rate * gradients_w\n",
        "        self.b = self.b - learning_rate * gradient_b\n",
        "    \n",
        "    def train(self, x, y):\n",
        "        epoch_num = 1000\n",
        "        learning_rate = 0.01\n",
        "        for epoch in range(epoch_num):\n",
        "            model.update_weightes(x, y, learning_rate)\n",
        "\n",
        "model = LinearRegression()\n",
        "\n",
        "N = 1000\n",
        "# y = 1 * x1 - 2 * x2 + 3 * x3 + 12\n",
        "def gen_data(x):\n",
        "    y = []\n",
        "    for i in range(N):\n",
        "        y.append(x[i][0] - 2 * x[i][1] + 3 * x[i][2] + 12)\n",
        "    return np.array(y)\n",
        "\n",
        "x = np.random.rand(N, 3)\n",
        "y = gen_data(x)\n",
        "\n",
        "x = model.normalize(x)\n",
        "model.train(x, y)\n",
        "print(model.w, model.b)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HYPydWTvsuW9",
        "outputId": "7665ba68-e7bd-4da6-a310-128822b9fbfb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[ 0.99828682 -1.99113219  2.99342082] [-268782.10094641]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Logistic Regression\n",
        "### Introduction\n",
        "Logistic regression is a classification algorithm used to assign observations to a discrete set of classes. Unlike linear regression which outputs continuous number values, logistic regression transforms its output using the logistic sigmoid function to return a probability value which can then be mapped to two or more discrete classes.\n",
        " \n",
        "### **Sigmoid Function**\n",
        "In order to map predicted values to probabilities, we use the sigmoid function. The function maps any real value into another value between 0 and 1. In machine learning, we use sigmoid to map predictions to probabilities.\n",
        "$$Sigmoid(x) = \\frac{1}{1 + e^{-x}}$$\n",
        "```python\n",
        "def sigmoid(x):\n",
        "    return 1.0 / (1 + np.exp(-x))\n",
        "```\n",
        "\n",
        "### **Cost Function**\n",
        "Instead of Mean Squared Error, we use a cost function called Cross-Entropy, also known as Log Loss. Cross-entropy loss can be divided into two separate cost functions: one for $y=1$ and one for $y=0$.\n",
        "$$L = \\frac{1}{m}(-y^{T}log(y\\_pred) - (1-y)^{T}log(1-y\\_pred))$$"
      ],
      "metadata": {
        "id": "0lpiTbEENz-s"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "\n",
        "def sigmoid(x):\n",
        "    return 1.0 / (1 + np.exp(-x))\n",
        "\n",
        "class LogisticRegression(object):\n",
        "    def __init__(self):\n",
        "        self.w = np.random.rand(feature_num)\n",
        "        self.b = np.random.rand(1)\n",
        "    \n",
        "    def forward(self, x):\n",
        "        y_pred = np.matmul(x, self.w) + self.b\n",
        "        y_pred = sigmoid(y_pred)\n",
        "        return y_pred\n",
        "    \n",
        "    def normalize(self, x):\n",
        "        for feature in x.T:\n",
        "            fmean = np.mean(feature)\n",
        "            frange = np.amax(feature) - np.amin(feature)\n",
        "            feature -= fmean\n",
        "            feature /= frange\n",
        "        return x\n",
        "\n",
        "    def cross_entropy(self, x, y_true):\n",
        "        m = len(y_true)\n",
        "        y_pred = self.forward(x)\n",
        "        class1_cost = -y_true * np.log(y_pred)\n",
        "        class2_cost = (1 - y_true) * np.log(1 - y_pred)\n",
        "        cost = class1_cost - class2_cost\n",
        "        cost = cost.sum() / m\n",
        "        return cost\n",
        "    \n",
        "    def compute_gradients(self, x, y_true):\n",
        "        y_pred = self.forward(x)\n",
        "        difference = y_pred - y_true\n",
        "        gradient_b = np.mean(difference)\n",
        "        gradients_w = np.matmul(x.transpose(), difference)\n",
        "        gradients_w = np.array([np.mean(grad) for grad in gradients_w])\n",
        "\n",
        "        return gradients_w, gradient_b\n",
        "    \n",
        "    def update_weightes(self, x, y_true, learning_rate):\n",
        "        gradients_w, gradient_b = self.compute_gradients(x, y_true)\n",
        "        self.w = self.w - learning_rate * gradients_w\n",
        "        self.b = self.b - learning_rate * gradient_b\n",
        "    \n",
        "    def train(self, x, y):\n",
        "        epoch_num = 1000\n",
        "        learning_rate = 0.01\n",
        "        for epoch in range(epoch_num):\n",
        "            model.update_weightes(x, y, learning_rate)\n",
        "    \n",
        "    def predict(self, x, y):\n",
        "        print(y)\n",
        "        print(self.forward(x))\n",
        "\n",
        "model = LogisticRegression()\n",
        "\n",
        "N = 1000\n",
        "# y = 1 * x1 - 2 * x2 + 3 * x3 - 1\n",
        "def gen_data(x):\n",
        "    y = []\n",
        "    for i in range(N):\n",
        "        y_true = x[i][0] - 2 * x[i][1] + 3 * x[i][2] - 1\n",
        "        if y_true > 0:\n",
        "            y.append(1)\n",
        "        else:\n",
        "            y.append(0)\n",
        "    return np.array(y)\n",
        "\n",
        "x = np.random.rand(N, 3)\n",
        "y = gen_data(x)\n",
        "\n",
        "x = model.normalize(x)\n",
        "model.train(x, y)\n",
        "print(model.w, model.b)\n",
        "model.predict(x[:10], y[:10])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HPeDIxp3tJEJ",
        "outputId": "14e94ec5-f017-433c-a96b-4bbed7d0ec7f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[ 10.44756978 -20.78139095  31.64716938] [0.53287193]\n",
            "[0 0 1 0 0 1 1 1 1 1]\n",
            "[5.48030739e-01 1.92401937e-05 1.00000000e+00 8.33032784e-02\n",
            " 3.62622457e-10 9.99999998e-01 9.99941493e-01 9.99862098e-01\n",
            " 9.99999097e-01 9.99970778e-01]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Decision trees\n",
        "Tree-based methods are simple and useful for interpretation since the underlying mechanisms are considered quite similar to human decision-making.\n",
        "\n",
        "The methods involve stratifying or segmenting the predictor space into a number of simpler regions. When making a prediction, we simply use the mean or mode of the region the new observation belongs to as a response value.\n",
        "\n",
        "Decision trees can be used for both regression and classification tasks.\n",
        "\n",
        "+ Dividing the predictor space into several distinct, non-overlapping regions\n",
        "    + In order to split the predictor space into distinct regions, we use binary recursive splitting, which grows our decision tree until we reach a stopping criterion. Since we need a reasonable way to decide which splits are useful and which are not, we also need a metric for evaluation purposes.\n",
        "    + We can leverage the concept of entropy to calculate the information gain, resulting from a possible split. \n",
        "$$H(X)=-\\sum_{i=1}^{n}P(x_i)logP(x_i)$$\n",
        "+ Predicting the most-common class label for the region any new observation belongs to\n",
        "\n",
        "### training\n",
        "The main algorithm can be basically divided into three steps:\n",
        "\n",
        "+ Initialization of parameters (e.g. maximum depth, minimum samples per split) and creation of a helper class\n",
        "+ Building the decision tree, involving binary recursive splitting, evaluating each possible split at the current stage, and continuing to grow the tree until a stopping criterion is satisfied\n",
        "+ Making a prediction, which can be described as traversing the tree recursively and returning the most-common class label as a response value\n"
      ],
      "metadata": {
        "id": "iCwKvN48DKvq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "\n",
        "class Node(object):\n",
        "    def __init__(self, feature=None, threshold=None, left=None, right=None, value=None):\n",
        "        self.feature = feature\n",
        "        self.threshold = threshold\n",
        "        self.left = left\n",
        "        self.right = right\n",
        "        self.value = value\n",
        "    \n",
        "    def is_leaf(self):\n",
        "        return self.value is not None\n",
        "\n",
        "class DecisionTree(object):\n",
        "    def __init__(self, max_depth=10, min_samples_split=2, impurity_func=\"entory\"):\n",
        "        self.max_depth = max_depth\n",
        "        self.min_samples_split = min_samples_split\n",
        "        self.imurity_func = impurity_func\n",
        "        self.root = None\n",
        "\n",
        "    def _is_finished(self, depth):\n",
        "        if (depth >= self.max_depth\n",
        "            or self.n_class_labels == 1\n",
        "            or self.n_samples < self.min_samples_split):\n",
        "            return True\n",
        "        return False\n",
        "    \n",
        "    def entropy(self, y):\n",
        "        proportions = np.bincount(y) / len(y)\n",
        "        ans = -np.sum([p * np.log2(p) for p in proportions if p > 0])\n",
        "        return ans\n",
        "    \n",
        "    def gini(self, y):\n",
        "        proportions = np.bincount(y) / len(y)\n",
        "        ans = 1 - np.sum([p ** 2 for p in proportions])\n",
        "        return ans\n",
        "    \n",
        "    def clac_impurity(self, y, func=\"entropy\"):\n",
        "        if func == \"entropy\":\n",
        "            return self.entropy(y)\n",
        "        else:\n",
        "            return self.gini(y)\n",
        "    \n",
        "    def create_split(self, x, thresh):\n",
        "        left_idx = np.argwhere(x < thresh).flatten()\n",
        "        right_idx = np.argwhere(x > thresh).flatten()\n",
        "        return left_idx, right_idx\n",
        "    \n",
        "    def information_gain(self, x, y, thresh):\n",
        "        # Therefore, we are interested in comparing those cuts that generate less impurity. \n",
        "        # For this, Information Gain is used. \n",
        "        # This metric indicates the improvement when making different partitions \n",
        "        # and is usually used with entropy\n",
        "        parent_loss = self.clac_impurity(y, self.imurity_func)\n",
        "        left_idx, right_idx = self.create_split(x, thresh)\n",
        "        n, n_left, n_right = len(y), len(left_idx), len(right_idx)\n",
        "        if n_left == 0 or n_right == 0:\n",
        "            return 0\n",
        "\n",
        "        child_loss = (n_left / n) * self.clac_impurity(y[left_idx], self.imurity_func) \\\n",
        "            + (n_right / n) * self.clac_impurity(y[right_idx], self.imurity_func)\n",
        "        return parent_loss - child_loss\n",
        "    \n",
        "    def best_split(self, x, y, features):\n",
        "        # we will stick with the split that generates the highest Information Gain.\n",
        "        split = {\"score\": -1, \"feat\": None, \"thresh\": None}\n",
        "\n",
        "        # the cost function of a decision tree seeks to find those cuts that minimize impurity\n",
        "        for feature in features:\n",
        "            X_feat = x[:, feature]\n",
        "            thresholds = np.unique(X_feat)\n",
        "            for thresh in thresholds:\n",
        "                score = self.information_gain(X_feat, y, thresh)\n",
        "                if score > split[\"score\"]:\n",
        "                    split[\"score\"] = score\n",
        "                    split[\"feat\"] = feature\n",
        "                    split[\"thresh\"] = thresh\n",
        "        return split[\"feat\"], split[\"thresh\"]\n",
        "    \n",
        "    def build_tree(self, X, y, depth=0):\n",
        "        self.n_samples, self.n_features = X.shape\n",
        "        self.n_class_labels = len(np.unique(y))\n",
        "\n",
        "        # stopping criteria\n",
        "        if self._is_finished(depth):\n",
        "            most_common_Label = np.argmax(np.bincount(y))\n",
        "            return Node(value=most_common_Label)\n",
        "\n",
        "        rand_feats = np.random.choice(self.n_features, self.n_features, replace=False)\n",
        "        best_feat, best_thresh = self.best_split(X, y, rand_feats)\n",
        "\n",
        "        left_idx, right_idx = self.create_split(X[:, best_feat], best_thresh)\n",
        "        left_child = self.build_tree(X[left_idx, :], y[left_idx], depth + 1)\n",
        "        right_child = self.build_tree(X[right_idx, :], y[right_idx], depth + 1)\n",
        "        return Node(best_feat, best_thresh, left_child, right_child)\n",
        "    \n",
        "    def traverse_tree(self, x, node):\n",
        "        if node.is_leaf():\n",
        "            return node.value\n",
        "        \n",
        "        if x[node.feature] <= node.threshold:\n",
        "            return self.traverse_tree(x, node.left)\n",
        "        return self.traverse_tree(x, node.right)\n",
        "\n",
        "    def fit(self, X, y):\n",
        "        self.root = self.build_tree(X, y)\n",
        "    \n",
        "    def predict(self, X):\n",
        "        predictions = [self.traverse_tree(x, self.root) for x in X]\n",
        "        return np.array(predictions)\n",
        "\n",
        "from sklearn import datasets\n",
        "from sklearn.model_selection import train_test_split\n",
        "    \n",
        "data = datasets.load_breast_cancer()\n",
        "X, y = data.data, data.target\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(\n",
        "    X, y, test_size=0.2, random_state=1\n",
        ")\n",
        "\n",
        "def accuracy(y_true, y_pred):\n",
        "    ans = np.sum(y_true == y_pred) / len(y_true)\n",
        "    return ans\n",
        "\n",
        "tree = DecisionTree(max_depth=10, min_samples_split=3, impurity_func=\"gini\")\n",
        "tree.fit(X_train, y_train)\n",
        "y_pred = tree.predict(X_test)\n",
        "acc = accuracy(y_test, y_pred)\n",
        "\n",
        "print(\"gini Accuracy:\", acc)\n",
        "\n",
        "tree = DecisionTree(max_depth=10, min_samples_split=3, impurity_func=\"entropy\")\n",
        "tree.fit(X_train, y_train)\n",
        "y_pred = tree.predict(X_test)\n",
        "acc = accuracy(y_test, y_pred)\n",
        "\n",
        "print(\"entropy Accuracy:\", acc)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oykLH7O0t4I3",
        "outputId": "f5cfc8b6-f510-44fa-c486-37f92c1c9212"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "gini Accuracy: 0.9385964912280702\n",
            "entropy Accuracy: 0.9385964912280702\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Other Tree Model based on Decision Tree\n",
        "\n",
        "### Some Basic Concept\n",
        "+ **Bias** is the simplifying assumptions made by the model to make the target function easier to approximate.\n",
        "+ **Variance** is the amount that the estimate of the target function will change given different training data.\n",
        "\n",
        "### Random Forest\n",
        "Random forest is an ensemble learning method, which tries to build a multitude of decision tree (CART) models to predict the target variable. It builds many decision trees with random sample and decision variables. If used for regression purposes, it outputs the mean of all the decision trees. In case of a classification problem, the mode of the outputs of the decision trees is the final output.\n",
        "\n",
        "### Bagging\n",
        "Bagging refers to Bootstrap Aggregation, which is the technique used to reduce the variance of decision tree models. In the Bagging algorithm, several subsets of the data are randomly chosen with replacement and are fed into separate decision tree models, creating an ensemble of decision tree models. The output is the average of the predictions made by all the decision trees, making it more robust and reducing the variance.\n",
        "\n",
        "### Boosting\n",
        "The idea behind boosting is to create a ‘powerful’ committee of many weak classifiers. Here, samples are collected from the training data without replacement and are fed into the tree models. The point to note here is that this sampling is not parallel, it is done sequentially; and in each step, more weightage is given to the wrongly classified labels. The final output is the weighted majority vote of all the models trained in this ensemble learning technique. The main aim of the boosting algorithm is to increase the predictive accuracy of the weak classifiers.\n",
        "\n",
        "### XGboost\n",
        "XGBoost is an optimized Gradient Boosting Machine Learning library..."
      ],
      "metadata": {
        "id": "yaDT5Ge-osY2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Glossary\n",
        "[https://ml-cheatsheet.readthedocs.io/en/latest/glossary.html](https://ml-cheatsheet.readthedocs.io/en/latest/glossary.html)\n",
        "\n",
        "+ **Normal Distribution:**\n",
        "    + The Normal Distribution is probably the most important and commonly-encountered probability distribution.\n",
        "    + In a normal distribution, 68% of data is within one σ from μ and 95% of data is within two σ.\n",
        "    + It is “convenient” to operate on normal distributions. Many intractable problems can now be solved analytically. And many operations on normal distributions return a normal distribution.\n",
        "    + One of the convenience is that when we multiply two normal distributions, its result is just another normal distribution scaled by a factor s\n",
        "    + its summation is also normally distributed.\n"
      ],
      "metadata": {
        "id": "OHGmEmUFYZyV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Layers\n",
        "## Dropout\n",
        "A dropout layer takes the output of the previous layer’s activations and randomly sets a certain fraction (dropout rate) of the activatons to 0, cancelling or ‘dropping’ them out.\n",
        "\n",
        "Dropout is only used during training; At test time, no activations are dropped, but scaled down by a factor of dropout rate. This is to account for more units being active during test time than training time.\n",
        "\n",
        "+ A layer in a neural net outputs a tensor (matrix) A of shape (batch_size, num_features).\n",
        "+ The dropout rate of the layer is set to 0.5 (50%).\n",
        "+ A random 50% of the values in A will be set to 0.\n",
        "+ These will then be multiplied with the weight matrix to form the inputs to the next layer.\n",
        "\n",
        "\n",
        "```python\n",
        "layer_output *= np.random.randint(0, high=2, size=layer_output.shape) # dropping out values\n",
        "\n",
        "# scaling up by dropout rate during TRAINING time, so no scaling needs to be done at test time\n",
        "layer_output /= 0.5\n",
        "```"
      ],
      "metadata": {
        "id": "BwDeRNYQiFa7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        ""
      ],
      "metadata": {
        "id": "bEgnQbRjiF41"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        ""
      ],
      "metadata": {
        "id": "AWUGnUwmiF_o"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        ""
      ],
      "metadata": {
        "id": "jTHvF3MGiGBG"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        ""
      ],
      "metadata": {
        "id": "hPJj6qMkiGCR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        ""
      ],
      "metadata": {
        "id": "G_k8hflCiGEG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "FRGLjekTsxLz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "zjF-jURWmoLt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        ""
      ],
      "metadata": {
        "id": "fVa5gq8NtxP-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Gradient Desent\n",
        "\n",
        "## [Adam](https://machinelearningmastery.com/adam-optimization-from-scratch)\n",
        "> **TODO**: 先实现一个basic dnn，可以使用手写数字的例子，然后把所有的basic tech都用上。all from scratch\n",
        "\n",
        "The Adaptive Movement Estimation algorithm, or Adam for short, is an extension to gradient descent and a natural successor to techniques like AdaGrad and RMSProp that automatically adapts a learning rate for each input variable for the objective function and further smooths the search process by using an exponentially decreasing moving average of the gradient to make updates to variables.\n",
        "+ Gradient descent is an optimization algorithm that uses the gradient of the objective function to navigate the search space.\n",
        "+ Gradient descent can be updated to use an automatically adaptive step size for each input variable using a decaying average of partial derivatives, called Adam.\n",
        "+ How to implement the Adam optimization algorithm from scratch and apply it to an objective function and evaluate the results.\n",
        "\n"
      ],
      "metadata": {
        "id": "EwbcysqlmonH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "\n",
        "class DeepNeuralNetwork(object):\n",
        "    def __init__(self):\n",
        "        self.input_layer_size = 128\n",
        "        self.hidden_layer_size = 256\n",
        "        self.output_layer_size = 2\n",
        "        self.init_weights()\n",
        "        self.init_bias()\n",
        "    \n",
        "    def init_weights(self):\n",
        "        self.w1 = np.random.randn(self.input_layer_size, self.hidden_layer_size) * \\\n",
        "                np.sqrt(2.0 / self.input_layer_size)\n",
        "        self.w2 = np.random.randn(self.hidden_layer_size, self.output_layer_size) * \\\n",
        "                np.sqrt(2.0 / self.hidden_layer_size)\n",
        "    \n",
        "    def init_bias(self):\n",
        "        self.b1 = np.full((1, self.hidden_layer_size), 0.1)\n",
        "        self.b2 = np.full((1, self.output_layer_size), 0.1)\n",
        "\n",
        "    def relu(self, x):\n",
        "        return np.maximum(0, x)\n",
        "\n",
        "    def relu_prime(self, x):\n",
        "        x[x < 0] = 0\n",
        "        x[x > 0] = 1\n",
        "        return x\n",
        "    \n",
        "    def sigmoid(self, x):\n",
        "        return 1 / (1 + np.exp(-x))\n",
        "\n",
        "    def forward(self, x):\n",
        "         # Hidden layer\n",
        "        x = np.dot(x, self.w1) + self.b1\n",
        "        x = self.relu(x)\n",
        "\n",
        "        # Output layer\n",
        "        x = np.dot(x, self.w2) + self.b2\n",
        "        x = self.relu(x)\n",
        "        return x\n",
        "    \n",
        "    def backprop(self, x, y):\n",
        "        \"\"\"\n",
        "        反向传播的本质是利用chain rule求解整个function对w的导数\n",
        "        \"\"\"\n",
        "        y_pred = self.forward(x)\n",
        "        d = (y_pred - y) * self.relu_prime(y_pred)\n",
        "        print(loss)\n",
        "\n",
        "dnn = DeepNeuralNetwork()\n",
        "x = np.random.randn(dnn.input_layer_size)\n",
        "y = np.array([1, 0])\n",
        "print(dnn.forward(x))\n",
        "dnn.backprop(x, y)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 344
        },
        "id": "OzDmU5vn5vIo",
        "outputId": "fdf19807-b3f8-429d-ca6c-9ea1c28484c1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[0.75108166 0.        ]]\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-13-bafe5ad66832>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     50\u001b[0m \u001b[0my\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     51\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdnn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 52\u001b[0;31m \u001b[0mdnn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackprop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m<ipython-input-13-bafe5ad66832>\u001b[0m in \u001b[0;36mbackprop\u001b[0;34m(self, x, y)\u001b[0m\n\u001b[1;32m     44\u001b[0m         \u001b[0my_pred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     45\u001b[0m         \u001b[0md\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0my_pred\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrelu_prime\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my_pred\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 46\u001b[0;31m         \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mloss\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     47\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     48\u001b[0m \u001b[0mdnn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mDeepNeuralNetwork\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mNameError\u001b[0m: name 'loss' is not defined"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "aaZHbRemqPRH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# TF-IDF (Term Frequency-Inverse Document Frequency)\n",
        "IDF =Log[(Number of documents) / (Number of documents containing the word)]\n",
        "\n",
        "TF = (Number of repetitions of word in a document) / (# of words in a document)\n",
        "\n",
        "TF-IDF = TF * IDF\n",
        "\n",
        "we can get the TF-IDF matrix for each word in each document. And words TF-IDF vector represents a document. We could use cosine of the vectors to represent the similarity between docments."
      ],
      "metadata": {
        "id": "BaHaH-eeIyyM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "ZL0Vujx8788n"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Language Model\n",
        "Language modeling is the task of predicting what word comes next.\n",
        "\n",
        "## N-gram language model\n",
        "## Word2vec\n",
        "Skip-gram is used to predict the context word for a given target word. On the other hand, the cbow model predicts the target word according to its context. The context is represented as a bag of the words contained in a fixed size window around the target word.\n",
        "\n",
        "## Attention"
      ],
      "metadata": {
        "id": "oRqjOCz5KQb7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Normalization\n",
        "## Batch Normalization\n",
        "Batch normalization is a technique for training very deep neural networks that standardizes the inputs to a layer for each mini-batch. This has the effect of stabilizing the learning process and dramatically reducing the number of training epochs required to train deep networks. \n",
        "+ Probably Use Before the Activation\n",
        "+ Use Large Learning Rates\n",
        "+ Less Sensitive to Weight Initialization\n",
        "+ Don’t Use With Dropout\n",
        "\n",
        "how do we use batch norm in testing?\n",
        "+ use train's statistics\n",
        "\n",
        "## Layer Normalization\n",
        "Layer Normalization directly estimates the normalization statistics from the summed inputs to the neurons within a hidden layer so the normalization does not introduce any new dependencies between training cases.\n"
      ],
      "metadata": {
        "id": "xc4xID0rlOUJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Transformer\n",
        "## Self-attention\n",
        "$$q^{i} = w^{q}a^{i}$$\n",
        "$$k^{i} = w^{k}a^{i}$$\n",
        "$$v^{i} = w^{v}a^{i}$$\n",
        "$$\\sum_{i}softmax(\\alpha_{1,i}=q^{1}*k^{i})v^{i}$$\n",
        "Every input must have three representations called key, query, and value. To obtain these representations, every input is multiplied with a set of weights for keys, a set of weights for querys, and a set of weights for values.\n",
        "\n",
        "To obtain attention scores, we start with taking a scaled dot product between each input's query with all keys, including itself. Then take the softmax across these attention scores.\n",
        "\n",
        "The softmaxed attention scores for each input is multiplied by its value representations then we get the weighted values. \n",
        "\n",
        "For each input take all the weighted values and sum them element-wise.\n",
        "\n",
        "The dimension of query and key must always be the same because of the dot product score function. However, the dimension of value may be different from query and key. The resulting output will consequently follow the dimension of value.\n",
        "\n",
        "## Multi-Head self-attention\n",
        "with multi-headed attention we have not only one, but multiple sets of Query/Key/Value weight matrices (the Transformer uses eight attention heads, so we end up with eight sets for each encoder/decoder). Each of these sets is randomly initialized. Then, after training, each set is used to project the input embeddings (or vectors from lower encoders/decoders) into a different representation subspace.\n",
        "\n",
        "## position encoding\n",
        "To address this, the transformer adds a vector to each input embedding. These vectors follow a specific pattern that the model learns, which helps it determine the position of each word, or the distance between different words in the sequence. The intuition here is that adding these values to the embeddings provides meaningful distances between the embedding vectors once they’re projected into Q/K/V vectors and during dot-product attention.\n",
        "\n",
        "## The Residuals\n",
        "each sub-layer (self-attention, ffnn) in each encoder has a residual connection around it, and is followed by a layer-normalization step.\n",
        "![image.png](https://jalammar.github.io/images/t/transformer_resideual_layer_norm_2.png)\n",
        "\n",
        "## Feed Forward Neural Netword\n",
        "\n",
        "## The Decoder Side\n",
        "The encoder start by processing the input sequence. The output of the top encoder is then transformed into a set of attention vectors K and V. These are to be used by each decoder in its “encoder-decoder attention” layer which helps the decoder focus on appropriate places in the input sequence.\n",
        "\n",
        "The following steps repeat the process until a special symbol is reached indicating the transformer decoder has completed its output. The output of each step is fed to the bottom decoder in the next time step, and the decoders bubble up their decoding results just like the encoders did. And just like we did with the encoder inputs, we embed and add positional encoding to those decoder inputs to indicate the position of each word.\n",
        "\n",
        "In the decoder, the self-attention layer is only allowed to attend to earlier positions in the output sequence. This is done by masking future positions (setting them to -inf) before the softmax step in the self-attention calculation.\n",
        "\n",
        "The \"Encoder-Decoder Attention\" layer works just like multiheaded self-attention, except it creates its Queries matrix from the layer below it, and takes the Keys and Values matrix from the output of the encoder stack.\n",
        "\n",
        "![image2.png](https://jalammar.github.io/images/t/transformer_resideual_layer_norm_3.png)\n",
        "\n",
        "## The Final Linear and Softmax Layer\n",
        "The decoder stack outputs a vector of floats. How do we turn that into a word? That’s the job of the final Linear layer which is followed by a Softmax Layer.\n",
        "\n",
        "The Linear layer is a simple fully connected neural network that projects the vector produced by the stack of decoders, into a much, much larger vector called a logits vector.\n",
        "\n",
        "The softmax layer then turns those scores into probabilities (all positive, all add up to 1.0). The cell with the highest probability is chosen, and the word associated with it is produced as the output for this time step.\n",
        "\n",
        "## The Loss Function\n",
        "cross-entropy\n",
        "\n",
        "\n",
        "## Follow Up 1: What are masks in transfomer?:\n",
        "there are two marks\n",
        "+ padding mask: for the indefinite length\n",
        "+ sequence mask: self-attention will use the other words' context it will see the output in some case. so in decoder the self-attention layer is only allowed to attend to earlier positions in the output sequence. This is done by masking future positions (setting them to -inf) before the softmax step in the self-attention calculation.\n",
        "\n",
        "## Follow Up 2: Why transformer use multi-head?\n",
        "Multi-headed attention was introduced due to the observation that different words relate to each other in different ways. For a given word, the other words in the sentence could act as moderating or negating the meaning, but they could also express relations like inheritance (is a kind of), possession (belongs to), etc. \n",
        "\n",
        "This means that separate sections of the Embedding can learn different aspects of the meanings of each word, as it relates to other words in the sequence.\n",
        "\n",
        "## Follow Up 3: Why we need scaled before softmax in self-attention?\n",
        "Beacuse the dot product between key and query will be huge. The gradient of the softmax will be small, so we divide the scores by 8 (the square root of the dimension of the key vectors used in the paper – 64). why dk. If the q and k are independent random variable with mean 0, variance 1. The dot production of q and k has mean 0 and variance dk.\n",
        "\n",
        "## Follow up 4: What is the role of feed forward layer in Transformer?\n",
        "$$max(0, XW_{1} + b_{1})W_{2} + b_{2}$$\n",
        "The feed-forward layer is weights that is trained during training and the exact same matrix is applied to each respective token position.\n",
        "\n",
        "Since it is applied without any communcation with or inference by other token positions it is a highly parallelizable part of the model.\n",
        "\n",
        "The role and purpose is to process the output from one attention layer in a way to better fit the input for the next attention layer.\n",
        "\n",
        "### Follow up 5: Why transformer use layer-norm?\n",
        "Layer Normalization directly estimates the normalization statistics from the summed inputs to the neurons within a hidden layer so the normalization does not introduce any new dependencies between training cases.\n",
        "\n",
        "if using batchnorm, it would be uncertain what would be the appropriate normalization constant (the total number of elements to divide by during normalization) to use. Different batches would have different normalization constants which leads to instability during the course of training"
      ],
      "metadata": {
        "id": "ScgXQwG7Paku"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn.functional as F\n",
        "\n",
        "x = [\n",
        "  [1, 0, 1, 0], # Input 1\n",
        "  [0, 2, 0, 2], # Input 2\n",
        "  [1, 1, 1, 1]  # Input 3\n",
        " ]\n",
        "x = torch.tensor(x, dtype=torch.float32)\n",
        "\n",
        "w_key = [\n",
        "  [0, 0, 1],\n",
        "  [1, 1, 0],\n",
        "  [0, 1, 0],\n",
        "  [1, 1, 0]\n",
        "]\n",
        "w_query = [\n",
        "  [1, 0, 1],\n",
        "  [1, 0, 0],\n",
        "  [0, 0, 1],\n",
        "  [0, 1, 1]\n",
        "]\n",
        "w_value = [\n",
        "  [0, 2, 0],\n",
        "  [0, 3, 0],\n",
        "  [1, 0, 3],\n",
        "  [1, 1, 0]\n",
        "]\n",
        "w_key = torch.tensor(w_key, dtype=torch.float32)\n",
        "w_query = torch.tensor(w_query, dtype=torch.float32)\n",
        "w_value = torch.tensor(w_value, dtype=torch.float32)\n",
        "\n",
        "keys = x @ w_key\n",
        "querys = x @ w_query\n",
        "values = x @ w_value\n",
        "\n",
        "attn_scores = querys @ keys.T\n",
        "\n",
        "attn_scores_softmax = F.softmax(attn_scores, dim=-1)\n",
        "# For readability, approximate the above as follows\n",
        "attn_scores_softmax = [\n",
        "  [0.0, 0.5, 0.5],\n",
        "  [0.0, 1.0, 0.0],\n",
        "  [0.0, 0.9, 0.1]\n",
        "]\n",
        "attn_scores_softmax = torch.tensor(attn_scores_softmax)\n",
        "weighted_values = values[:,None] * attn_scores_softmax.T[:,:,None]\n",
        "outputs = weighted_values.sum(dim=0)\n",
        "print(outputs)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aub8DDQ8yRr1",
        "outputId": "6acd1b2b-b101-433b-c1b8-d6f6b9925cb0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[2.0000, 7.0000, 1.5000],\n",
            "        [2.0000, 8.0000, 0.0000],\n",
            "        [2.0000, 7.8000, 0.3000]])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# BERT\n",
        "Bidirectional Encoder Representations from Transformers: BERT\n",
        "+ BERT has deep bidirectional representations meaning the model learns information from left to right and from right to left. The bidirectional models are very powerful compared to either a left-to-right model or the shallow concatenation of a left-to-right and a right-to-left model. As opposed to directional models, which read the text input sequentially (left-to-right or right-to-left), the Transformer encoder reads the entire sequence of words at once. Therefore it is considered bidirectional, though it would be more accurate to say that it’s non-directional. This characteristic allows the model to learn the context of a word based on all of its surroundings (left and right of the word).\n",
        "+ BERT uses the Masked Language Model (MLM) to use the left and the right context during pre-training to create a deep bidirectional Transformers.\n",
        "\n",
        "## Masked Language Model\n",
        "Before feeding word sequences into BERT, 15% of the words in each sequence are replaced with a [MASK] token. The model then attempts to predict the original value of the masked words, based on the context provided by the other, non-masked, words in the sequence. In technical terms, the prediction of the output words requires:\n",
        "+ Adding a classification layer on top of the encoder output.\n",
        "+ Multiplying the output vectors by the embedding matrix, transforming them into the vocabulary dimension.\n",
        "+ Calculating the probability of each word in the vocabulary with softmax.\n",
        "\n",
        "## Next Sentence Prediction\n",
        "In the BERT training process, the model receives pairs of sentences as input and learns to predict if the second sentence in the pair is the subsequent sentence in the original document. During training, 50% of the inputs are a pair in which the second sentence is the subsequent sentence in the original document, while in the other 50% a random sentence from the corpus is chosen as the second sentence. The assumption is that the random sentence will be disconnected from the first sentence.\n",
        "+ A [CLS] token is inserted at the beginning of the first sentence and a [SEP] token is inserted at the end of each sentence.\n",
        "+ A sentence embedding indicating Sentence A or Sentence B is added to each token. Sentence embeddings are similar in concept to token embeddings with a vocabulary of 2.\n",
        "+ A positional embedding is added to each token to indicate its position in the sequence. The concept and implementation of positional embedding are presented in the Transformer paper.\n",
        "\n",
        "To predict if the second sentence is indeed connected to the first, the following steps are performed:\n",
        "\n",
        "+ The entire input sequence goes through the Transformer model.\n",
        "+ The output of the [CLS] token is transformed into a 2×1 shaped vector, using a simple classification layer (learned matrices of weights and biases).\n",
        "+ Calculating the probability of IsNextSequence with softmax.\n",
        "\n",
        "## How to use bert\n",
        "BERT can be used for a wide variety of language tasks, while only adding a small layer to the core model:\n",
        "\n",
        "+ Classification tasks such as sentiment analysis are done similarly to Next Sentence classification, by adding a classification layer on top of the Transformer output for the [CLS] token.\n",
        "+ In Question Answering tasks (e.g. SQuAD v1.1), the software receives a question regarding a text sequence and is required to mark the answer in the sequence. Using BERT, a Q&A model can be trained by learning two extra vectors that mark the beginning and the end of the answer.\n",
        "+ In Named Entity Recognition (NER), the software receives a text sequence and is required to mark the various types of entities (Person, Organization, Date, etc) that appear in the text. Using BERT, a NER model can be trained by feeding the output vector of each token into a classification layer that predicts the NER label.\n",
        "\n",
        "## Follow Up 1: There are three embeddings in BERT, what is it, why it can add?\n",
        "It have \n",
        "+ token embedding: all words and two sepcial token [CLS] [SEP], [CLS] token is inserted at the beginning of the first sentence and a [SEP] token is inserted at the end of each sentence.\n",
        "+ sentence embedding: A sentence embedding indicating Sentence A or Sentence B is added to each token.\n",
        "+ position embedding. A positional embedding is added to each token to indicate its position in the sequence.\n",
        "We add these three embeddings. Why? If we treat our input as one-hot representation, the sum of the embeddings is actually equal to the concat of these one-hot representations's embedding. And I think the idea is like the bag of the word model.\n",
        "\n",
        "## Follow Up 2: What the bert's learning rate warm-up and why?\n",
        "It means that if you specify your learning rate to be say 2e-5, then during training the learning rate will be linearly increased from approximately 0 to 2e-5 within the first 10,000 steps. After a few epoches, our learning rate will be decay.\n",
        "\n",
        "Because the parameters of the model are extremely large, at the beginning of the training process. All parameters are random, and if the learning rate is very large, the model can easily overfit the samples.\n",
        "\n",
        "## Follow Up 3: the loss function of the BERT?\n",
        "When training the BERT model, Masked LM and Next Sentence Prediction are trained together, with the goal of minimizing the combined loss function of the two strategies.\n",
        "\n",
        "## Follow Up 4: How to calculate the parameter amount of bert\n",
        "+ input embedding: 23 million\n",
        "    + token: 30000 * 768\n",
        "    + position: 512 * 768\n",
        "    + sentence: 2 * 768\n",
        "+ single encoder: 7 million \n",
        "    + multi-head self-attetion: 768 * 768 * 3 2 million\n",
        "    + add & norm 768 * 2\n",
        "    + feed forward neural network: 768 * 3072 + 3072 * 768 5 million\n",
        "    + add & norm 768 * 2\n",
        "+ all: input embedding + 12 * single encoder = 107 million\n",
        "\n",
        "## Follow Up 5: How pre-trained BERT model generates word embeddings for out of vocabulary words?\n",
        "Subwords are used for representing both the input text and the output tokens. When an unseen word is presented to BERT, it will be sliced into multiple subwords, even reaching character subwords if needed. That is how it deals with unseen words.\n",
        "\n",
        "## Follow Up 6: What are the limitations of BERT?\n",
        "+ Bert uses the [mask] token during training, but not in prediction, which may affect BERT's performance.\n",
        "+ Only use 15% tokens per sentence. So the convergence time is greater than the left-to-right language model.\n",
        "+ The bert mask only mask single word even sometime mask a subword. Lack of modeling the semantics of entities and phrases. This is what ERNIE do.\n",
        "\n",
        "## Follow Up 7: Bert family?\n",
        "+ Albert: Cross-layer Parameter Sharing\n",
        "+ Roberta: Dynamic Masking, remove NSP task, larger batch size\n",
        "\n",
        "## Follow Up 8: What is ERNIE?\n",
        "The architecture of ERNIE is the same as BERT, except that it has a task embedding. And ERNIE has different pretrain tasks, such as bert only mask single word. ernie mask words, phrases, and entities. ERNIE also uses dialogue and search click log for next sentence prediction task. Finally ERNIE use a continuous, multi-task learning approach(simple as training the model in saveral stages, randomly selecting tasks for training in each stage, and the number of the tasks is gradually increased)\n",
        "\n",
        "## Follow Up 9: What is ERNIE-ANN\n",
        "ERNIE-ANN aims to learn the sentence representations for recall. The training paradigm is different from the point-wise and pair-wise. It's call matrix-wise in Baidu. May be also called in-batch negative sample. The main idea is that I don't generate the random samples by my own, I use the other samples within a batch to be the negative samples. This method will trained very fast. Based on this method, I have made a lot of improvements about the hard negtive sample mining, both online and offline.\n",
        "+ offline method: the most hardest sample in the batch is still not so hard, so we need the offline mining. We first train a base model. Then build a ann index offline, and generate each sample's similar but not so similar sample as the hard sample. And we add these negative sample in the traning set.\n",
        "+ online method: The online method is quite straightforward, I use the hardest sample within a batch instead of all the other samples. The hardest sample is the sample that is most similar sample to the sample within a batch. We could also add some randomly noise on the sample embedding to generate the hard sample.\n",
        "In real production, we combine both approaches, the offline hard negative mining contributes the most. \n",
        "\n",
        "## Follow Up 10: what is ELMo?\n",
        "The way ELMo works is that it uses bidirectional LSTM to make sense of the context. Since it considers words from both directions, it can assign different word embedding to words that are spelled similarly but have different meanings. This enables ELMo to capture contextual information from the sequences but since ELMo uses LTSM it does not have long-term dependency compared to transformers.\n",
        "\n",
        "## Follow Up 11: what is GPT?\n",
        "GPT is use decoders from the transformer instead of encoders in a unidirectional approach. The pretrain task is left-to-right language model task. Predict next token.\n",
        "\n",
        "## Follow Up 12: Can BERT be used for sentence generating tasks?\n",
        "Sentence generation requires sampling from a language model, which gives the probability distribution of the next word given previous contexts. But BERT can't do this due to its bidirectional nature.\n",
        "\n",
        "## Follow Up 13: why we need position emebedding?\n",
        "We all know that in languages the order of the words and their position in a sentence really matters. The meaning of the entire sentence can change if the words are re-ordered. When implementing NLP solutions, the recurrent neural networks have an inbuilt mechanism that deals with the order of sequences. The transformer model, however, does not use recurrence or convolution and treats each data point as independent of the other. Hence, positional information is added to the model explicitly to retain the information regarding the order of words in a sentence. Positional encoding is the scheme through which the knowledge of order of objects in a sequence is maintained.\n"
      ],
      "metadata": {
        "id": "j2sI925oG2G7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import math\n",
        "import re\n",
        "from random import *\n",
        "import numpy as np\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "\n",
        "class Embedding(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(Embedding, self).__init__()\n",
        "        self.tok_embed = nn.Embedding(vocab_size, d_model)  # token embedding\n",
        "        self.pos_embed = nn.Embedding(maxlen, d_model)  # position embedding\n",
        "        self.seg_embed = nn.Embedding(n_segments, d_model)  # segment(token type) embedding\n",
        "        self.norm = nn.LayerNorm(d_model)\n",
        "\n",
        "    def forward(self, x, seg):\n",
        "        seq_len = x.size(1)\n",
        "        pos = torch.arange(seq_len, dtype=torch.long)\n",
        "        pos = pos.unsqueeze(0).expand_as(x)  # (seq_len,) -> (batch_size, seq_len)\n",
        "        embedding = self.tok_embed(x) + self.pos_embed(pos) + self.seg_embed(seg)\n",
        "        return self.norm(embedding)\n",
        "\n",
        "class ScaledDotProductAttention(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(ScaledDotProductAttention, self).__init__()\n",
        "\n",
        "    def forward(self, Q, K, V, attn_mask):\n",
        "        scores = torch.matmul(Q, K.transpose(-1, -2)) / np.sqrt(d_k) # scores : [batch_size x n_heads x len_q(=len_k) x len_k(=len_q)]\n",
        "        scores.masked_fill_(attn_mask, -1e9) # Fills elements of self tensor with value where mask is one.\n",
        "        attn = nn.Softmax(dim=-1)(scores)\n",
        "        context = torch.matmul(attn, V)\n",
        "        return context, attn \n",
        "\n",
        "class MultiHeadAttention(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(MultiHeadAttention, self).__init__()\n",
        "        self.W_Q = nn.Linear(d_model, d_k * n_heads)\n",
        "        self.W_K = nn.Linear(d_model, d_k * n_heads)\n",
        "        self.W_V = nn.Linear(d_model, d_v * n_heads)\n",
        "    def forward(self, Q, K, V, attn_mask):\n",
        "        # q: [batch_size x len_q x d_model], k: [batch_size x len_k x d_model], v: [batch_size x len_k x d_model]\n",
        "        residual, batch_size = Q, Q.size(0)\n",
        "        # (B, S, D) -proj-> (B, S, D) -split-> (B, S, H, W) -trans-> (B, H, S, W)\n",
        "        q_s = self.W_Q(Q).view(batch_size, -1, n_heads, d_k).transpose(1,2)  # q_s: [batch_size x n_heads x len_q x d_k]\n",
        "        k_s = self.W_K(K).view(batch_size, -1, n_heads, d_k).transpose(1,2)  # k_s: [batch_size x n_heads x len_k x d_k]\n",
        "        v_s = self.W_V(V).view(batch_size, -1, n_heads, d_v).transpose(1,2)  # v_s: [batch_size x n_heads x len_k x d_v]\n",
        "\n",
        "        attn_mask = attn_mask.unsqueeze(1).repeat(1, n_heads, 1, 1) # attn_mask : [batch_size x n_heads x len_q x len_k]\n",
        "\n",
        "        # context: [batch_size x n_heads x len_q x d_v], attn: [batch_size x n_heads x len_q(=len_k) x len_k(=len_q)]\n",
        "        context, attn = ScaledDotProductAttention()(q_s, k_s, v_s, attn_mask)\n",
        "        context = context.transpose(1, 2).contiguous().view(batch_size, -1, n_heads * d_v) # context: [batch_size x len_q x n_heads * d_v]\n",
        "        output = nn.Linear(n_heads * d_v, d_model)(context)\n",
        "        return nn.LayerNorm(d_model)(output + residual), attn # output: [batch_size x len_q x d_model]\n",
        "\n",
        "class PoswiseFeedForwardNet(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(PoswiseFeedForwardNet, self).__init__()\n",
        "        self.fc1 = nn.Linear(d_model, d_ff)\n",
        "        self.fc2 = nn.Linear(d_ff, d_model)\n",
        "\n",
        "    def forward(self, x):\n",
        "        # (batch_size, len_seq, d_model) -> (batch_size, len_seq, d_ff) -> (batch_size, len_seq, d_model)\n",
        "        return self.fc2(gelu(self.fc1(x)))\n",
        "\n",
        "class EncoderLayer(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(EncoderLayer, self).__init__()\n",
        "        self.enc_self_attn = MultiHeadAttention()\n",
        "        self.pos_ffn = PoswiseFeedForwardNet()\n",
        "\n",
        "    def forward(self, enc_inputs, enc_self_attn_mask):\n",
        "        enc_outputs, attn = self.enc_self_attn(enc_inputs, enc_inputs, enc_inputs, enc_self_attn_mask) # enc_inputs to same Q,K,V\n",
        "        enc_outputs = self.pos_ffn(enc_outputs) # enc_outputs: [batch_size x len_q x d_model]\n",
        "        return enc_outputs, attn\n",
        "\n",
        "class BERT(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(BERT, self).__init__()\n",
        "        self.embedding = Embedding()\n",
        "        self.layers = nn.ModuleList([EncoderLayer() for _ in range(n_layers)])\n",
        "        self.fc = nn.Linear(d_model, d_model)\n",
        "        self.activ1 = nn.Tanh()\n",
        "        self.linear = nn.Linear(d_model, d_model)\n",
        "        self.activ2 = gelu\n",
        "        self.norm = nn.LayerNorm(d_model)\n",
        "        self.classifier = nn.Linear(d_model, 2)\n",
        "        # decoder is shared with embedding layer\n",
        "        embed_weight = self.embedding.tok_embed.weight\n",
        "        n_vocab, n_dim = embed_weight.size()\n",
        "        self.decoder = nn.Linear(n_dim, n_vocab, bias=False)\n",
        "        self.decoder.weight = embed_weight\n",
        "        self.decoder_bias = nn.Parameter(torch.zeros(n_vocab))\n",
        "\n",
        "    def forward(self, input_ids, segment_ids, masked_pos):\n",
        "        output = self.embedding(input_ids, segment_ids)\n",
        "        enc_self_attn_mask = get_attn_pad_mask(input_ids, input_ids)\n",
        "        for layer in self.layers:\n",
        "            output, enc_self_attn = layer(output, enc_self_attn_mask)\n",
        "        # output : [batch_size, len, d_model], attn : [batch_size, n_heads, d_mode, d_model]\n",
        "        # it will be decided by first token(CLS)\n",
        "        h_pooled = self.activ1(self.fc(output[:, 0])) # [batch_size, d_model]\n",
        "        logits_clsf = self.classifier(h_pooled) # [batch_size, 2]\n",
        "\n",
        "        masked_pos = masked_pos[:, :, None].expand(-1, -1, output.size(-1)) # [batch_size, max_pred, d_model]\n",
        "        # get masked position from final output of transformer.\n",
        "        h_masked = torch.gather(output, 1, masked_pos) # masking position [batch_size, max_pred, d_model]\n",
        "        h_masked = self.norm(self.activ2(self.linear(h_masked)))\n",
        "        logits_lm = self.decoder(h_masked) + self.decoder_bias # [batch_size, max_pred, n_vocab]\n",
        "\n",
        "        return logits_lm, logits_clsf"
      ],
      "metadata": {
        "id": "7xDC7B6IybZy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "3VqAayYGkBgD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Some other stuff\n",
        "## Give a set of ground truths and 2 models, how do you be confident that one model is better than another?\n",
        "+ Better performance: The primary objective of model comparison and selection is definitely better performance of the machine learning software/solution. The objective is to narrow down on the best algorithms that suit both the data and the business requirements.\n",
        "+ Longer lifetime: High performance can be short-lived. For example, one ctr model's goal is only consider CTR, it will rank some clickbait result, and ther other model's goal consider CTR and duration or retention rate. In CTR metric, second model may be not better. But in long run, it will better than the first one.\n",
        "+ Speedy production: Low parameter, fast predict, easy to push the model into production environment.\n",
        "+ Easier retraining: the model can be retrained or incrementally trained is also an important concern for real business problems."
      ],
      "metadata": {
        "id": "iEe30sJrqkm7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "QZhqA61HxgjI"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}