13. Reinforcement learning.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "13. Reinforcement learning.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.2"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/DataScienceUB/DeepLearningMaster2019/blob/master/13.%20Reinforcement%20learning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FmhtoqY3qXMK",
        "colab_type": "text"
      },
      "source": [
        "#  Reinforcement Learning\n",
        "\n",
        "> *Reinforcement learning is an area of machine learning inspired by behaviorist psychology, concerned with how software agents ought to take **actions in an environment** so as to maximize some notion of **cumulative reward**. (Source: Wikipedia)*\n",
        "\n",
        "Reinforcement learning provides the capacity for us not only to teach an artificial agent how to act, but to allow it to learn through it’s own interactions with an environment. \n",
        "\n",
        "By combining the complex representations that deep neural networks can learn with the goal-driven learning of an RL agent, computers have accomplished some amazing feats, like beating humans at over a dozen Atari games, and defeating the Go world champion.\n",
        "\n",
        "RL algorithms must enable the agent to learn the correct pairings itself through the use of *observations*, *rewards*, and *actions*.\n",
        "\n",
        "Typical aspects of a task that make it an RL problem are the following:\n",
        "1. Different actions yield different rewards. For example, when looking for treasure in a maze, going left may lead to the treasure, whereas going right may lead to a pit of snakes.\n",
        "2. Rewards are delayed over time. This just means that even if going left in the above example is the right things to do, we may not know it till later in the maze.\n",
        "3. Reward for an action is conditional on the state of the environment. Continuing the maze example, going left may be ideal at a certain fork in the path, but not at others.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YCTsY2EjqXMK",
        "colab_type": "text"
      },
      "source": [
        "## Multi-armed bandit\n",
        "\n",
        "The simplest reinforcement learning problem is the multi-armed bandit. Essentially, there are $n$-many slot machines, each with a different fixed payout probability. \n",
        "\n",
        "The goal is to discover the machine with the best payout, and maximize the returned reward by always choosing it. \n",
        "\n",
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/slot.jpg?raw=1\" alt=\"\" style=\"width: 400px;\"/>\n",
        "\n",
        "This question has been the subject of active research since the 1950s, and many variations have been studied.\n",
        "\n",
        "#### A-B Testing\n",
        "\n",
        "Traditional A-B testing can be thought of as a special case of the multi-armed bandit problem, in which we choose to pursue a strategy of pure exploration in the initial testing phase, followed by a period of pure exploitation in which we choose the most valuable “arm” 100% of the time.\n",
        "\n",
        "If the exploitation phase can be assumed to be much longer than the exploration phase, this approach is usually reasonable, as the wasted resources during the exploration are insignificant relative to the total rewards. However, in cases where the cost of the exploration phase is non-negligible, or in cases in which arm values are changing dynamically on short enough timescales that it becomes impractical to repeatedly perform new A-B tests, alternative approaches are needed.\n",
        "\n",
        "The n-armed bandit is a nice starting place because we don’t have to worry about aspects #2 and #3. All we need to focus on is learning which rewards we get for each of the possible actions, and ensuring we chose the optimal ones."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VbkAevR3qXML",
        "colab_type": "text"
      },
      "source": [
        "### Epsilon-greedy\n",
        "\n",
        "The most straightforward algorithm for continuously balancing exploration with exploitation is called “epsilon-greedy”. \n",
        "\n",
        "Here, we pull a randomly chosen arm a fraction $\\epsilon$ of the time. The other $1-\\epsilon$ of the time, we pull the arm which we estimate to be the most profitable. As each arm is pulled and rewards are received, our estimates of arm values are updated. \n",
        "\n",
        "This method can be thought of a a continuous testing setup, where we devote a fraction $\\epsilon$ of our resources to testing.\n",
        "\n",
        "The following python code implements a simple 10-Armed Bandit using the epsilon-greedy algorithm. \n",
        "\n",
        "The payout rate of the arms are normally distributed with $\\mu = 0$ and $\\sigma = 1$. Gaussian noise is also added to the rewards, also with $\\mu = 0$ and $\\sigma = 1$. (See [Sutton and Barto](http://people.inf.elte.hu/lorincz/Files/RL_2006/SuttonBook.pdf) book, section 2.1)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ue1k_ifjqXMM",
        "colab_type": "code",
        "outputId": "249981cb-6308-4edd-e513-e472d3337952",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 104
        }
      },
      "source": [
        "import numpy as np \n",
        "import seaborn \n",
        "from tqdm import tqdm\n",
        "\n",
        "class Bandit: \n",
        "    def __init__(self): \n",
        "        self.arm_values = np.random.normal(0,1,10) #arm values\n",
        "        self.K = np.zeros(10)                      #action counter\n",
        "        self.est_values = np.zeros(10)             #estimated values\n",
        "\n",
        "    def get_reward(self,action):                   #get noisy reward from machine \"action\"\n",
        "        noise = np.random.normal(0,1) \n",
        "        reward = self.arm_values[action] + noise \n",
        "        return reward \n",
        "\n",
        "    def choose_eps_greedy(self,epsilon):           #exploitation or exploration decision\n",
        "        rand_num = np.random.random() \n",
        "        if epsilon>rand_num: \n",
        "            return np.random.randint(10)           #exploration\n",
        "        else: \n",
        "            return np.argmax(self.est_values)      #exploitation\n",
        "\n",
        "    def update_est(self,action,reward):            #estimation of the value of an action\n",
        "        self.K[action] += 1 \n",
        "        alpha = 1./self.K[action] \n",
        "        self.est_values[action] += alpha * (reward - self.est_values[action]) \n",
        "        \n",
        "a= Bandit()\n",
        "print(\"The arm vales of the 10 machines are: \\n\",a.arm_values)\n",
        "print(\"The reward if we play first machine is: \\n\",a.get_reward(0))\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The arm vales of the 10 machines are: \n",
            " [ 1.1796864   0.40726757  0.31826565  0.75944305  0.44926722  1.04328852\n",
            "  0.31688177 -0.20897799 -1.0835095   0.67392407]\n",
            "The reward if we play first machine is: \n",
            " 2.366974080810383\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pDkI0LSEqXMO",
        "colab_type": "text"
      },
      "source": [
        "How are we estimating the value of an action?\n",
        "\n",
        "If by the $t$-th time step action $a$ has been chosen $K_a$ times prior to $t$, yielding rewards \n",
        "$R_1, R_2, . . . , R_{Ka}$, then its value is estimated to be:\n",
        "\n",
        "$$\n",
        "Q_t(a) = \\frac{R_1 + R_2 + . . .+ R_{Ka}}{K_a}\n",
        "$$\n",
        "\n",
        "A problem with this straightforward implementation is that its memory and computational requirements grow over time without\n",
        "bound (we have to maintain, for each action $a$, a record of all the rewards that have followed the\n",
        "selection of that action), but we can derive an incremental formula for computing averages with small, constant computation\n",
        "required to process each new reward.\n",
        "\n",
        "\\begin{eqnarray*}\n",
        "Q_{k+1} & = & \\frac{1}{k+1} \\sum_{i=1}^{k+1} R_i \\\\\n",
        "  & = & \\frac{1}{k+1} \\left( R_{k+1} + \\sum_{i=1}^k R_i\n",
        "\\right) \\\\\n",
        "  & = & \\frac{1}{k+1} \\left( R_{k+1} + kQ_k + Q_k - Q_k\n",
        "\\right) \\\\\n",
        "  & = & \\frac{1}{k+1} \\left(R_{k+1} + (k+1) Q_k - Q_k\n",
        "\\right) \\\\\n",
        "  & = & Q_k + \\frac{1}{k+1} \\left( R_{k+1} - Q_k \\right)\n",
        "\\end{eqnarray*}\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q-EfFQJtqXMO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def experiment(bandit,Npulls,epsilon):\n",
        "    history = [] \n",
        "    for i in range(Npulls): \n",
        "        action = bandit.choose_eps_greedy(epsilon)\n",
        "        R = bandit.get_reward(action) \n",
        "        bandit.update_est(action,R) \n",
        "        history.append(R) \n",
        "    return np.array(history)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IObBsHLcqXMQ",
        "colab_type": "text"
      },
      "source": [
        "Let's make three different experiments: $\\epsilon = 0$, $\\epsilon = 0.1$ and $\\epsilon = 0.01$. Data will be averages over 2000 tasks."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Le9GmMEqqXMQ",
        "colab_type": "code",
        "outputId": "2c8e5e45-dd45-4915-9e28-1e046a4ec4b2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 231
        }
      },
      "source": [
        "Nexp = 2000\n",
        "Npulls = 5000 \n",
        "avg_outcome_eps0p0 = np.zeros(Npulls) \n",
        "avg_outcome_eps0p01 = np.zeros(Npulls) \n",
        "avg_outcome_eps0p1 = np.zeros(Npulls) \n",
        "\n",
        "for i in tqdm(range(Nexp)): \n",
        "    bandit = Bandit() \n",
        "    avg_outcome_eps0p0 += experiment(bandit,Npulls,0.0) \n",
        "    bandit = Bandit() \n",
        "    avg_outcome_eps0p01 += experiment(bandit,Npulls,0.01) \n",
        "    bandit = Bandit() \n",
        "    avg_outcome_eps0p1 += experiment(bandit,Npulls,0.1) \n",
        "\n",
        "avg_outcome_eps0p0 /= np.float(Nexp) \n",
        "avg_outcome_eps0p01 /= np.float(Nexp) \n",
        "avg_outcome_eps0p1 /= np.float(Nexp) \n",
        "\n",
        "# plot results \n",
        "import matplotlib.pyplot as plt \n",
        "\n",
        "plt.plot(avg_outcome_eps0p0,label=\"eps = 0.0\", alpha=0.5) \n",
        "plt.plot(avg_outcome_eps0p01,label=\"eps = 0.01\", alpha=0.5) \n",
        "plt.plot(avg_outcome_eps0p1,label=\"eps = 0.1\", alpha=0.5) \n",
        "plt.ylim(0,2.2) \n",
        "plt.legend() \n",
        "plt.gcf().set_size_inches((8,3))\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 2000/2000 [03:35<00:00,  9.35it/s]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAecAAADFCAYAAABuHjrdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3XeYZFd94P3vqdTVOecw3ZOzRqMZ\nzSiPJIQCQQKEhBBYQiAZyaxt7H3XeN/nwfba+2J2vcbWghBIwggLGEAIkEAB5RnFCdLknDvnXLnu\nef84Fburw8x0T/fU/D7P009XuOHcU1X3d8+5JyitNUIIIYSYPWwznQAhhBBCJJPgLIQQQswyEpyF\nEEKIWUaCsxBCCDHLSHAWQgghZhkJzkIIIcQsI8FZCCGEmGUkOAshhBCzjARnIYQQYpZxzNSOS0pK\ndH19/UztXgghhDjntm/f3qW1Lp1ouRkLzvX19Wzbtm2mdi+EEEKcc0qpk5NZTqq1hRBCiFlGgrMQ\nQggxy0hwFkIIIWaZGbvnnEowGKSpqQmfzzfTSUlrbrebmpoanE7nTCdFCCFECrMqODc1NZGbm0t9\nfT1KqZlOTlrSWtPd3U1TUxMNDQ0znRwhhBApzKpqbZ/PR3FxsQTmaaSUori4WGonhBBiFptVwRmQ\nwHwOSB4LIcTsNuuCsxBCCHGhk+A8yxw/fpx169Yxf/587rzzTgKBQMrlvvWtbzF//nwWLVrESy+9\ndI5TKYQQYjpJcJ5l/uZv/oavf/3rHDlyhMLCQp544olRy+zbt4+NGzeyd+9eXnzxRR566CHC4fAM\npFYIIcR0mLC1tlKqFvgJUA5o4Ida638fsYwC/h24BfAA92qtPzibhL1xsIPOQf/ZbGKU0twMNiwq\nG3eZp556iocffphAIMC6det45JFHsNvt5OTkcP/99/PHP/6RiooKNm7cSGlpKQ8//DCPPvooDoeD\npUuXsnHjxjNOn9aa1157jZ/97GcA3HPPPfz93/89Dz74YNJyv/vd7/jc5z5HRkYGDQ0NzJ8/ny1b\ntnDZZZed8b6FEELMHpMpOYeAv9ZaLwXWA3+mlFo6YpmbgQWRvweA709pKs+R/fv384tf/IK3336b\nHTt2YLfb+elPfwrA8PAwa9asYe/evVxzzTX8wz/8AwD//M//zIcffsiuXbt49NFHR23z4MGDrFq1\nKuVfX19f0rLd3d0UFBTgcJhrppqaGpqbm0dts7m5mdra2tjzsZYTQghxfpqw5Ky1bgVaI48HlVL7\ngWpgX8JitwI/0Vpr4D2lVIFSqjKy7hmZqIQ7HV599VW2b9/O2rVrAfB6vZSVmXTYbDbuvPNOAL7w\nhS/w6U9/GoCVK1dy9913c9ttt3HbbbeN2uaiRYvYsWPHOToCIYQQ6eC0BiFRStUDFwPvj3irGmhM\neN4UeS0pOCulHsCUrKmrqzu9lJ4DWmvuuecevvWtb024bLQ70h/+8Ac2bdrEc889x//8n/+T3bt3\nx0q+YErO0aA+0htvvEFBQUHseXFxMX19fYRCIRwOB01NTVRXV49ar7q6msbGeHaPtZwQQojz06Qb\nhCmlcoBfA3+ptR44k51prX+otV6jtV5TWjrhdJbn3PXXX8/TTz9NR0cHAD09PZw8aWb3siyLp59+\nGoCf/exnXHnllViWRWNjI9deey3f/va36e/vZ2hoKGmb0ZJzqr/EwAwm4F977bWx/Tz55JPceuut\no9L5yU9+ko0bN+L3+zl+/DiHDx/m0ksvnfL8EEIIMTMmFZyVUk5MYP6p1vqZFIs0A7UJz2sir51X\nli5dyj/90z/x0Y9+lJUrV3LDDTfQ2moK/9nZ2WzZsoXly5fz2muv8c1vfpNwOMwXvvAFVqxYwcUX\nX8yf//mfjwq4p+vb3/42//qv/8r8+fPp7u7my1/+MgDPPvss3/zmNwFYtmwZd9xxB0uXLuWmm27i\ne9/7Hna7/ewOXgghxKyhzG3icRYw9bdPAj1a678cY5mPAV/DtNZeBzystR63KLdmzRq9bdu2pNf2\n79/PkiVLJp/6cygnJ2dUqfh8NpvzWggh0pVSarvWes1Ey03mnvMVwBeB3UqpaMum/w7UAWitHwWe\nxwTmI5iuVF86k0QLIYQQYnKttd8Cxh2MOdJK+8+mKlGzUTqVmoUQQsxuMkKYEEIIMctIcBZCCCFm\nGQnOQgghxCwjwVkIIYSYZSQ4zzJnO2XkfffdR1lZGcuXLz9XSRZCCDHFJDjPMmc7ZeS9997Liy++\neK6TLYQQYgqd1tja59ThV2CofWq3mVMOCz4y7iLn+5SRV199NSdOnDjjNAghhJh5UnJOIFNGCiGE\nmA1mb8l5ghLudJApI4UQQswGszc4zwCZMlIIIcRsINXaCWTKSCGEELOBBOcE6TBl5F133cVll13G\nwYMHqampSdnaWwghxOw24ZSR00WmjJxZszmvhRAiXU12ykgpOQshhBCzjATnSUqnUrMQQojZTYKz\nEEIIMctIcBZCCCFmGQnOQgghxCwjwVkIIYSYZSQ4zzKTmTKyu7uba6+9lpycHL72ta/NQCqFEEJM\nJwnOs8xkpox0u9384z/+I//yL/8yAykUQggx3Wbt2NpvNb9Fl7drSrdZklnCldVXjrvM+TBlZHZ2\nNldeeSVHjhw5430JIYSYvaTknOB8mTJSCCFEepu1JeeJSrjTQaaMFEIIMRvM2uA8E86XKSOFEEKk\nN6nWTnC+TBkphBAivUlwTnC+TBkJUF9fz1/91V/x4x//mJqaGvbt23dW+xVCCDF7yJSRkyRTRgoh\nzikrDI1bIOgxf0s+kfx+0Adtu6BmLURus82Igy9C607Y8Dczl4ZUtAZfP2SmKDD5h+CDn8DKOyG7\n+Jwma8qmjFRK/Ugp1aGU2jPG+xuUUv1KqR2Rv2+mWk4IIc6ZoA/CwZlOhQkQAI1bwdt7euu2fAjH\n3jABui3F6ffQi3DkVeg7ddbJTElrCHrN43Awnn7/IIT85v3eEyad2oqvFxo9cJLZRgh8A2eWFm8f\ndB+NPw8MQ19j8rb9IwpPjVvgve/DUOfo7R17wwTurY9D12HzmmXFvzPhEGz7D7OfGTKZBmE/Br4L\n/GScZTZrrT8+JSmapdKp1CzEpA11gtMNGblTt83hbvD1QfG8hNe6IDBkTvKuHMgpm3g7R183geGq\nvxr93lvfMf83fGPsUqXW0HkQShdNT8nT2wvvPQq1a01wbt4G5csgI88E1PorwZll8jdRzzFwF0x8\nceGLdMVMDIxRnh7ILIwf11CnKUHaneZ55yHY82u46HOwMzI2w7V/a9KVVWz23bYLTrwNV/y5uRDo\nPATX/Dd457uQXQJzN8Dup5P327oLDvwBVt4RKfm/D54uWHqrCYiD7WY/g+3QugOaP4Dq1VC9xhzH\nnqdh9Z+AK9tsr3ELONxmmwD1V4AjE468Yp5v+AZYIdg0YkCmdX8KR1+L5FM/5JRC+14ommfyu213\nPO92P22OZ7grvs0jL8NgG7z9sEnvDJgwOGutNyml6qc/KbH9xVpCi+kxU7cyxCxnhQEFtoQKta2P\nm/9jnaACw9B9BCpWmkDQ32xOxpUXjb2fLT8cvc0tjyUvE32vfR/s+x3MvQZyypMD+qn3UqTHY4JK\n1Bv/DItvMenpawS7C3LLTWD+8D9NegHWP2guDjLywJ1nXguHoO+kCVQ9R6HhGsjISd6ff8gEksFW\nyE64oGjeZi4ewATmaF6deDu+TPve5G2tf9AE0J2/MM+jASpRXyN8+BQs+TgMmPYweHogu9SkIb82\nfmECkF8NC2+CrQkjDW74hqmGhnhgBnh9RC+V6AVZ+z4TmMHsA0wgGxmYtz4BQ6YxLbt+mfxe9JgA\nNv1vk7dRzR+Yv6i3H4a8KihbamoGEiXmH8D+Z036Rnr/B/HHe5+BtV+Bfc+a78/KO0YvP5ww4FX3\nEeg5PnqZc2xS95wjwfn3WuvlKd7bAPwaaAJagP+qtd47crnIsg8ADwDU1dVdEm0JHXX8+HFyc3Mp\nLi6WAD1NtNZ0d3czODhIQ0PDTCcnfQR9MNCcHDzOxlCHCSRdh03JwmY3r2ttXiuenxxEwQQmKwju\nfFO1qFS8pKS1KS2ceg/mX2/e2/kLWHMfuLLMifj1b0HhHCiaa072hfXw5v8y62cVw7oH4vvy9ppg\nfuglU9pSNihbEg84V/21ec0euf4f6oTDf4SFN8YD8cIbzfq55aYklWjRzVCxIr7/qLVfiV8wRM3d\nYPJ+/vWw7Uepq1XnXgPH3pwg0yPWf9Uc28gLhqh515m8Of5mclXrdKtZC01bp2XTHitEprLLeTeV\n8WpfzsBk7zlPRXDOAyyt9ZBS6hbg37XWCybaZqoGYcFgkKamJnw+34RpEmfO7XZTU1OD0+mc6aSk\nj50bzdX26i/SFOhnS99Bbp13K/aQH2wOcLjMcu37TAkwsRGKfxDe/R6suhsKas1rI0sxyz4FOmyu\n/sEEotpLY283DTbx7Gvf4M7seRTP/ygc32TeWHQzFNTByXfiVXmpXPPfRgfCkRZ/zATt3b86vXtx\niVWn08DSmjAap5LOJ2di2Ary5MB+1rrLWesun+nkzD7X/M3oC+GzMNngfNaDkGitBxIeP6+UekQp\nVaK1Pu2BsZ1Op5TmzpFfHfoV1dnVXF59+bTvyxP00O3tpjavdtr3dda8vaaKbd51p3e17Ok2/z/4\nT14fOMBg9SqGgkPkv/OIed2RASs+a6powZTOMgtNkNv/XKSa9SnTInf/c6O3v/c3yc+PvAod+802\n7S6OvvW/wQrTGhqmOBqYAQ6+AJgakycHD7DeXcFiV2HSprTWDLz2j7htDjKUfexjjN73O03Wjp+z\nN9DNMlcxtikogXitEBaabJu5uPyj5xTHgv08VLDyrLedSlCHcSo7LaFhyu2Z2M/RRUBHyEOpPROl\nFAEdxjXeZzOB48EBXhg+wadz5vHM0FGuyaphmasIgCHL3Ns+ERpkLSY4e6wQLmXDMeJYtdZomJLP\ncbYIaYsweuzvvg4zE72Oz3qPSqkKFakLUUpdGtlm99luV0yepa0J7yO3DbfxyI5HYpOJdHo62dGZ\nPKyoJ+jBE/RMer+7O3czHBxdgjrad5RHdjxCIGyqF39z5Dc8d+y5MdP42qnXeKf5HZ47+hzb27fz\nyI5H6PWlaNka9MGWx+jvOUq/vx9fyMfBnoM8suORUenWWrO9fXvsda014b6TSS03fSGfufcWjNTU\nDHebBjyNW0xjkm0/Mo1mYhnUg27+gLdPvkrX6/9oqmQjwY+Ek5gCs43BtthrVtDHq29/i+5wZF/v\nPWpKx28/DL3m9k5zaIgXt30XrTU7/V1s9SVX9baFhnnXF98mAy3w/vdhyw8iJxA4HOxjf6BnVNaF\n0XisIG94m0a9tzPQxU8HD/JEf8q7UWitaQwNcTDQywe+DnxWKOVyhwJ9eK0QWmuGrXhjpj2BbjZ7\nW9gVGH29PmAF2Opr57dDx3jV00hQhwlEjmUs/zGwjycH9seeHwv2A6YEfTjQF/uetYWG2envYm+g\nh/d8bXzg64it0xwaiqXRYwU5GRxgf6AHrTX+yP67wz4e6dvFY/172eHv5LdDR5Pzf4RjwX7ei7wf\n1BbBEQ21TgQHYmlNxWOFCEXWORUc5OmhI+wN9NAR8vB4/14e6dvFc8Op74V2hb3jbvtNr7m3/syQ\nqYZP/G5Ff5WJ4fbHA/t4fvjEqO1s8bfzaP9ugjocy6/T0RwaiuXvWCytk/IurK2U3zmtNc2hIcLa\noiPkwa/DSet1hb0MWKNvcQS1RWMo3sD3maGjse9+SFv0hn1s9rawwx85V1jjp3e6TFhyVkr9HNgA\nlCilmoC/A5wAWutHgduBB5VSIcALfE5Li6PTprWm29dNSWbJaa1naYtHdz7KqrJVXF41din4WN8x\nABoHG1Puwx/28+O9PwbgoVUPAdDt7aZtuI15BfMIR35Qg4FBnDYngXCAzc2bOdh7kNsX3h5Li03Z\n2NZmblf0+nrJz8in329OGru7dlOYUUhNbk3Sva0DPQdijxsHTfeItiN/pNBVyPHyBbxw/AU+v+B2\nsrY8zmZPEwf7dkFhA9gcFGSZY+n195LV30IoMx+7M5vfn3iBRk8777e+z7qsGlwFdWx++9sA3P+J\n/6A/OMQvd/2I6/q6TEmy9lKe3b8Rjw7xudyFJrgCnv5mHLXrcX1gOit4rSA7B/azE5jnaeHGrDqC\nzdtpD3upcZjGQtFj++nmv+eO3AWU2DNpCQ9zMNBLR9hLsd1NtnJwRWZV0mfwuyHzGYWweNvbAsDq\njFLsysawFYydWNdnlMf2EQx6eW/wROwk1BoapjU0zEJnAXZlQ2vN3kBPLG1W5KcZ1GGOBgdY5Czg\nHW9rLA0DVgAHiqxIqfRYsJ8Xh5PbhpwMDXJr9lwe7d/NIlchCsWwDtIYHATgEncZ230dfDFvMQFt\nMazNifUdbytLXUVJJcDnho/TH/bHnh8PDhDQYeY68zkW7Ofm7HoanHl0h30MWAHK7JmxZYPaSqrK\n3h3o5m1vC+GsWuY582L5lSjL5uQ1T7wLzqdy5vGbhOV8mWHe9bbyxbzFNCecwI9GAl9POH7L7ZG+\nXSileDB/BUAsn9ZllPNY/x5cys4NWbWU27NoDXt4IRLsbsqeQ6k9k1ybK7atw4E+XvaYLlEPFayk\nzzJ50mP52OxriS3XGBykI+Sh2O5GofDrMJk2B78cPBxbd5e/i86wlzUZZXzo7yTT5sBjJbf81mh+\nP3wcG4o5TtMAbmRZuCnh+KMXN87IZ/dY9EIuC5a4itju68CmFBdnlMbW6bcCPDN0hJuy5hDUFiV2\nd+w7/pX8ZQS0RU7ke/aOt5Ud/k7uzlvMJm8zjcFBrsisYomrkJc8p2gMDnKxu4y1GWVs8bVT7sgi\nqK2kzxIg2+bknrwlDFiBWJ48uOav+f62/8OqjFIuz6zksX7TLa3Ckc2nsufSFTbdxd71tdER8iR9\n7gudhWTZXcyEWTUIyfkoGA6i0bjO4gPs9/dzauAUm5s3s6F2A0uKzOAgOzp3sLBwIf3+fgoyCshy\nZgFwauAUboebXl8vxZnF/PLgL7EpG59e8Gk8QQ/1+fWEI1d7u7t2U5Nbw96uveztTl0yuqzqMt5t\neTf2/MGLHsTSFo/vfjwWlAFcdlesNJzogZUP0O5p53dHfseN9Tfy0omXxj3e/Ix8bm64mbxQiFPv\n/CsvZthNFa8rK7bMipb9XJVZxSOF+RDyU9KyO/YjSlK+DKww1wwPcyLYz8ngAEV2tzmJurIg5DP9\nFwvqYv1BV7vL2OPvJqDD1DvzuCazGgvNfw7ELxLmOvOZ48zldU+8pFnjyEk6YQHckbuA54aP441c\n2X82dwEve07RlxBwvpK/jCcG9o0qZeTaXNyVuzBWdfhI3y5GUkrx+dxF/DQhbQBr3eX4dJjd/rHv\nHt2Xt5S+yAky0UMFK2P7WuQq5GBgdC3FTdlzRgXl03VFZlXsImOk5RnFXJJRllQCHsutOXNjJ/VE\ntc5crnBXsnHQtCSOfu7r3RWxEuyEMgtMH9qoogboOc6nc+axO9DNYctn2gf0J88Od1VmFZsjx3ZN\nVg0LnQWxk/5IhXY3veHR7Wg+kTOXjpCHE6EBXMoeu7i5t3g1T+a40U1b4q2aC+tNn2KAkgWmVsZv\nlk/8XubYXAylKC0mqbsMTkV+71lFZtutu8AKUeXI4bqsGp6KfN8cykZBzTq6Qp54C+8E2TYnOTYn\n7bllJn3uPJYs/jTlrft4o91c4FJQa277BDymzYKvN3ZclY5sWkMJtW/51aZB4vHNgKk+typWRHoF\njNP4LiPX5EflRRR0HGDACmApBVWrQdlNl66RHG5zfqi+BJyRi76mraYPdyS/19ZtYO0lfzp+fp6m\nKW0QNh3O9+A8EBggy5HFY7sfQ2vNQ6sewhP08NT+p/jkvE9SkV0xap03Gt9gMDDITQ034VAO/GE/\nJwZO8Nqp15KWq86p5vKqy/nVoV8lve6wOQiNUaWY6PaFt/P0oacnXG4sldmVtA63TrzgSCE/dB4w\nrXZTXax4e8GeEQ/CQx2mn2mUwwX5daaFcUfkpF250pw40tQX8xYT0hY/jwSYc+Hu3EX8dPDgxAtO\nl7r1qbtB1a2DUylOoqejYjkLe5o5FOiFvGrTijurON4mIJEry5yYPd3m+6q16UZlheBkJHhVrTIn\n/qAXmraZQGN3mUCRV2VO4p6ueFej0kUmAHXsM12cqi42fbEBcivN7YehDnMbpGI5uHKh66DpylN5\nUbwbV1QkSFG1Clp2mHUyC01awwHTT7f7iLnAyK82FxHZpRD2mwE/MgtN4MkpM+s7MkxDQm+fWX9k\nf3JtmeO3LDMqGdrkn2XByYRuTAV1Jh+yi03acytHt9HwDZi8KZprbh31HDP5Y7ObQVWig5qULozn\nX/0VJm9CAbNvmzPeGCvaIyK7xHwmkQsKiueZHgoas+zxzWaZsoQREAMe079Z2eIDw1hhc4wj8xzM\nMtrC7crmvuX3jX7/LEhwniLNQ8387sjvKMks4Y5Fpn9ctCp5bv5cjvWbK/orqq8gy5HFyydfZl7B\nPG6svxGA4eAwgXCAbGc2j+9+fMz9jFToLkx933UmhYPQ32hOSMpmnoeD8WDbc8ycHArnmB9v9MRS\nudL8eKLPcyvMSaNj4lKTmAWKGiC/Jv75jaf+CnOR1jTit126yFyIZReb/qt9p2L32gGYc5lp1a61\nKQENtprSUslC0x3LnmEu/KLfnaDX7MeVZU7ynYfMgB41l8QH5YhWeVth05I+r9IMSBEtgTVcNfZx\naMukw52f+n0rZNIb1bHfBKnSxWbAi0TePnPBmarfMphjtsLxbmeJBlpMCS+raOy0phL0mj7RJfOT\n2kOcMa3hxFvx5+Pl3WRYlim1urLMtruPmF4MqQLlaW87ZI55ihruPXjRg1PaxeyctdZOdx+0m87x\n0YZUEB/EIxqYAd5ujl9VRhtEnY0ZCcyhQHLfVMuCzv3m/q6/H7oi1aOuHHPV3XfKnFQqVphBGqIj\nGvWeTO5qM7LkO9iW1FhKJChbDKj4hUu0NJRKXpU5eY/kzjdByGY3n110pKjWnSag5ZSZz9mREQ+Q\nBbVmudZdJiBmFpiuV0GvKYEC1KwxQdfuNJ+1UjDnCvMdGe42I14pFa8iBBOU7U6z7cRSWkbCSbh6\ndTzQKWVO0KlO0oklIWdmfD9ZJZDZYUpoMPqkbLObIAWRAKlMf/DxKNvYgRmSAzMQa1aV6iSeamzn\npH2p1IEZzGd8JpyZpkQ61TILTbX62bLZ4hf1Sk3NNmPbntqwFtIhnOrcdzuV4DyOo31HaRuOB5EX\njr9A+3A7tnTpTxkOmhNX9Hii92XKlpjgUFBrquY8I1r/do6oEk3Vf3b4tHvSnd9yyuKjI01G1cUQ\nHDYDimhtAlRiySqxZFI013xWVjg+CEVuhanOyyqO578z04xgVdhgGrTl15rSYlTNGhOcE4OnM9NU\nrUaHkEzcb8VKk8ZowHFmxt8Ph0wLIqVMSXi8fEklur+S+WOXKCfLZjcXiJOVmCdTxV1gLlAS8zad\nKGWqwxOrmS8QVqrhUc8BCc4Rr5x8hbbhNq6quYo/HPtDyvu7x/tnfki3mK4j5qRUNIl+4f7B+P00\nMN2JBltN6SqryDSqSgym0VJbX+PobaWboobIvbEJev/lVrI6GObSjHLTx/PiL/DIwZ+Z+3hgSoiu\nbFOjEC3pRgPZyPuGYIJoRo4Z8tEKxUfyGovdaf4K55hqzuxIi/vMAnOvsnWnKTGULk7edyJlGx08\nsk0V7JXVV/JW81vJ7zlc8cFTRqVnEqeOzMKx33O4Tak7XfrL5laaCyVHxkynZPqk87GNI8M+M8d9\nYV0CjSEYDnKo9xADgQH+cMwMtDCZhlczarAV+kf0WY3OEtN7It7oAUxAaN1p7qMNtJp7d75In0hP\njwke5/D+7z15E09VWe3IYbW7jM/lLsQ9mWqqkWMeR2UVmarh/Jrk14saYM7lkF9DacM18UBStsQM\nk1iy0Fz8lC42j/OrWe+uMIE5uwQKarlm3seZ7yqIl/zya0wpt2J5PEjGtrnGNGJpuCo5cCYOsTkO\nZ6TLCQV18Wrp2LHnmpJ06aIJt5PKldVXsrJ0cgN4rC5fPbmNzrmCuQs/Pm73Pmw2lhSPU+pOIT9j\ndFVzliNrzPe/tPxL1OfVn9Y+xrK8ZNQAiXFKnXHwynNN7j7rsuJl475fm3t6g/ysKFnB8pLlzC+Y\nP+l16vLqTmsfUZ9f/PkzWm8y1lasndRy2c6zrKE5xy7YkvNgYJDh4DAV2RU8tnuMMXRnktaAHn3/\nLBSA4YQp0KItE52ZJtBG7/X2NcbvDUadeBulFNPRBDDfnsFV7iq2+NvpCI09kMk9eUvItjl5IH85\nP4x0Pal35nEiaAaaW+wqosfy8cnshlgjjC/O/SRtbTt4LtqdxmaDOVewuuxi2LnRDC5RuQp6jvH5\nvCX8zOo298WDHhZVrOb6uuvZ1bmLt068zI29neyvXMqyuqt44fgL1OfVc8vcW3ik/5RpKJRZaIKy\n023GfI6KTp2XXQKXfAmAZSXLWPbR73CDsvP9PfHv0B2r/pSSzBLebHzTdF/LTt13/ZaGW6jKqcJl\nd6Vso3Db/Nv47ZHfAtCQ38CcvDmcGjzFwR5zW8Fld3FT/U08e/RZKFnAXYvv4oP2DyjLKqMgo4Ci\nzCI2NW2idbjVDLiS4NKKSznYe5B+fz+OyMXPqrJV7OhIHpjm3mX38kHHB8zNn0uhu5BMRyZLipbw\nzOFn+MS8T/DLg2aCg2J3MbV5tfH1bTY+2nATNmVjYeHCWB/6By96kE5vZ6w3wYbaDRS6C9natpVP\nzPsEzxx+ZlQ+KKXQWnPX4rsoyChgV9cuXDYXx/qPke3MZkPthlj+3bnoTnZ07KDD08FgYNCkt3gJ\nJwZOcFnVZSwqXITT5uSx3Y9Rm1vL9XXXk+nIZHPzZvZ0me/jZVWXsaJkBQ6bg+ahZrq93dTl1lHg\nLuDyqsvxhXzkuMzFYHS/pVmlLCxcyEWlF8UajEbzbyAwwEsnXuLuJXfzwvEXYn35wYwpoLVmV9cu\njvcfp324nbAOc3XN1Wxq2kS7X0rBAAAgAElEQVRdXh2nBkwXwGtqr2Fu/lyeOxYfQe6S8ktYU74G\nX9hHtjN71PdoRckKijKLKM8qj31WK0pWcFVNcs3K3N65dPu62d6+Pfba1TVXo1C82RQfk/za2mt5\ncu+TSekPhoMpz6ELCheQ68qlLLMsKTCuLl+N2+6mIb+B3V272dW5i6+s+Aqtw62xwlGiSysuZVXZ\nKn6464exffb6evn5gZ+TYc9gbcVaanNryXPl8fzx5+nwxG8vZdgz8If9rKtcx8VlF8c+l4dWPcSx\n/mO8ePzFUfv7xNxPsLV9K/kZ+VxUOs4ELtPsgm2t/f0d30ejmZM3h5MDZ9efczLq8uqYkzuHzc2R\nKk4rHGlRmFytV5JZYhqfHd/MUlcR+yqXUJ1TxdrCxbh7T9J2/DUKbRkcCvax199NjSOHXssfG+0o\n1u8wIUCWObJiAfOLeYtj/Xnvy1tKc3iYLb72WD/Mz+TMZ3+gh32BHqocOdya3cCQDib1Ac63Z3B3\n7iIOBfroCnu51F2eNMxfU2TUnj+kGGHo/tobcS7/FLz7CI8MH4aQj6/mLefnQ4dYW7mehVkVphFM\n4mw0G74Bg2088uF3TcvcogZURg5fXflVTrZs4/nmN8CVTVVOFbfNv81kb+Q+0WTbBwwEBnhq31OA\nOXlEGwIC3DDnBsI6zGLPkCmdppg+MWyF+cGuH3B51eWsKluV9J4v5OPXh39NRXYF19ZeS5+/j25v\nNwsK441g3ml5Z1RgfGjVQ7QMtdDj62F+wXzcDnOf1hvykumIV08f6DnAYGBwzBJEl7eLXx78JS67\nC6fNyXBwmIdWPcRrp17jQM8BNtRuYGnxUixt0TzUTG1uLUErSDAcjPWtH4sn6GFf9z4uLrsYu81O\n42Ajzx19jvtX3h8v7RMPYtEBbnZ27qR9uJ2P1n80aXvR5YrcRSwtXkptbi25rlz6/H3jDtDTONhI\ny1AL6yrXjZve8YxM49ms4wl6aBlqYX5hcqk0aAXZ2rqV4dAwV1VfFftMowLhACErRKYjkx2dO1hc\ntJh93fsYDg5zdc3Vo/Z72/zbqMqJNxr7+YGfJzUmTQzEkz2+xoFGvGEvCwsXxo7lQM8BjvQd4Y5F\nd7CjYwc9vh6qcqpYXGRqiB7f/XhsDISHVj1E0ApiV/bY7y96wVKRXcGnF3x6zH37w378YT8n+0+S\nn5HP74/9nnuW3ZN04fHQqofwhXz8aM+PyM/I5+4ld8fW11rT5e3CbrPjsDnIsGcQDAdjF1OHeg8R\ntsIsKTa1d9HfxmVVl6FQWNqafO3QGZKuVBM429bUkzG3YG5sZK6vXvRVbMqGL+RjW+tWArs2sn7u\nTfzY32haPWfkQWYB96+8H4Wi7dW/o9qezZ5AN/PLV5HZl1yF3Roa5jdDR7krdyGFdvMDPxEciI0E\nFdBhfjywH6eyc3/+Mt7ztZGrnCzLKCaV931tlNozmevM51VPIwcDvVyXVRsbh3mTt5k9/m6uyKxi\nuasoeXzh/JrkKvbSReiOA+wMdFFlz2a7v4PrsmoJN1xNVkPkBNOxnyfb3mI4HOChvsjw7COnJezY\nb6rgl30KgKHAEEPBIZ45/AxZjizuXX6vyYuhVooyi3AoB/bo7E1nwB/2s719O+sq1jEcGub5Y8/T\n4+vhgZUPxEqX0y36vbx32b0TBsbT0eXtothdTNAK4g15yc/Ip324nV8f/jVfXPpFcl1TOF9zCnu7\n91LiLqE8e/yJFc4kQE6V5qFm8lx5054XU2GifGobbuOZw89wc8PNNOSbdikHew5S5C6iNKs05Tpn\n4/3W99nevp35BfNHXXAlpqnQXXjG93CP9R/DruzMyZsDwJ6uPdTl1U36tsBsIcF5DL6Qj1dOvRKr\nKjpby0uWs7drLzpSWXx51eW80/IOYKrwOjwdNA01cUn5JfGV/EPwzv8Fm52N5XX07P+dGampfAEP\nDU7djFxBbebnPd3ZerrCXv4wfILP1t9C1lA7BDxYWjNgBSiI/rAu/y/mPnb1ajOva3+TaTxUttTM\nH9u+x3TBic6rqpQpASemzwpiaYuMTf9qquCv/q+TSt/29u2xatbp5Al66PB0UJ9fP637STSTwWk2\niA71muresohrG26jw9MxbluBYDiIcxLtGaaCpS36/H0UuU+zP/YFSIJzCi1DLbxy8hWGgkMTLlub\nWxu7N5Sfkc9nFnwGt8PNUGCIn+wz4yzfVH9T7MT9VvNbLC5aTFlWGQOBAXKcOWNXqUanA1Q2wis+\ng975CxyRcZBnZD7VZZ+K9K/FzH7UccAEyugPe/fTpsvPZQ9FRuLRydXxnh4zJeGim+PzDkdZlll2\nvOPq2G8GIDjdgRbS0IUenIVIdxKcR9jTtYdNTZsmXpB4Iw2NxhfykenITAqae7r20JDfcHqt/ywL\nug6Zvqmb/uV0kz91KleawSLefcS0Vq5enfIeqpgZ3d5uNPq0J0ARQpwfZISwESYbmKOUUihUyvt+\no7pUeHpM/9agF3qPm/GhDzxnqngzC01Xpon60U6VK/4C2nZB+XJTdT7S4o+Z/yPv74pZoTgzdZsA\nIcSF5YIIzs1DYwx/OMKn5n9qUlXeMeEgNH9g5v5NEhlpq/mDUatMmYwcc+86Uc0aMyRe3frkfs7R\nYReFEEKcF9I6OPvDfp7Y/cSEy92/4n5QJHX9SKltt5lhZvWfmOdHX4fm7eOvM1Vq10Lj1vjzy74G\nJzabwf6zS+HQS9BwTfx9peDq/weOvwFzrjSjgI035ZoQQohZI62D84B/YFLLTbpF4/7fxx8Pd09/\nYL74C2bIze6jZizm+qvhwO9NlbVS0BDv98i6B0avb3fA/I+Yx86qMx9EXwghxDmV1sFZT2IsLFeq\neYcT+QfNhBCJpVZPD2z54dkl7qq/hkMvQPu+5NcTW0lHVUc6xTtcsHzsDvxCCCHSQ1oH5wM9B1K+\nvrZiLQ35DeS78s3MOpYF7bshr8bMNxsOwtv/ZmbeSSXad/d0JE66DibQLr3V/FmWmf1HWk0LIYQg\njYNzMByMjZWbKGX/0WNvmn66APOuNfeSp1K0ZXT5MvjwqXif4iibTQKzEEKImLQNzmNOZhEKmDGt\nE6e86z4cfzyVgbliOcy9Nv68oNZ0ZTrD2YOEEEJcGNI2OKeyoHABbP4/Zg7cS75k+h4PtJj5jafC\nopvB7jJ9mzNyUpeGKyc3NZ8QQogL1wUTnJeXLOfqysvh2Dbw9sFb35maDV90J+z8hXlctWr8ZYUQ\nQohJSP/g3LSNWysup7qvEw7/72nYgYKcMjhHsxYJIYRIf2kZUZoGm8wIWZEhNasHO2Fw7KrrkGUR\nsjRuxxjTDV71V2bAj8atMOcyqF0PVtD0cy6sh7Vfnp4DmQYtfV4q891nNcFG95Cf7AwHbufo/Ooe\n8hO2NGV57hRrCnHmtp/spTjbRX3JaYxpP006BnyU5GSggeFAiDz39Mz+dKJrmMoCNxljnZvGEbY0\nCrDZpn4yHa01Wk/PtoVxenMJngeCVpCtbVvN+NKNW7g+q3bCdXY39fPhqT4z6AfgD4X5PrfTs/xL\nsO5PwZEB0UnR7S5wus395Lkbxp9taZY50jHIL7Y2sqc5eXCWnuEA/Z5gynUsS3Oye5i9Lf10DvoB\n+Mm7J/np+6mn3BzvvXOtzxPgg1Pxiect6/QmeQmFLbyB8Gmvdy4c7xpm2D9GV78pZll6zDwIhS32\ntvTT2OM5o23vae6nrX9y06RuOtTJbz6c3FC8U2X7yd5Rx9Y15Oen75/i7aNdvH6ggyc2H+f9Y910\nD/nH3M6gL8iupj56hgOT3veAL8hvPmzmpb3t9HuCDJ3m5/3wq4d5Zpry69mdLfz7q/GGtDsa+/jO\ny4cIha1JrT/kD7GrqY+WPi9d4+RbY4+HAd/oc1Nznzf2vQmFLT441YsvGI5tKxi22NPcz8iJnfo8\nAXY29k0qjTMt7UrOT+17Cu9AM/hMAFrkmnjO3315V9CRs4SLsipRdRs4Gi7Fd9JiR7fiutIifMEw\nHVnLqav1Qc3apHUtS9M64KO6IHPSaRzwBXli83E+sqSc94938/l1dWS5HIQtzabDnZTmZLC8Op8h\nf4hQ2KKlz8fiitykq9Q9zf3ML8vB7bTT2u8lGNKU5WXQHCkZD/pClI8ovfZ7zZe812NOEL5gmKZe\nD8/tbAXg6zcsHJXW5/e0crg9PoZ3dJkB7+gfTGKw8AXDfP+No3xsZSULy8fuJjbsDxHWOlby+OW2\nRpp7vVTku7lxWQWeQIhhf5hFFaO3obWmqddLTWHmqJoArTX/8fYJAJZX5dPc5+W3HzbzmdU11BXH\nJzNp7fdiaTjQOsCVC0rYfKiL5j4v91xez/997QgAS6vyuHFZBaGwhT9kkZ1hfjaWpQlZGodN0THo\npyI/dW1BvzdIKGxRnJOBZWkCYStlrcNYfMEwdpvCabfF9vvbyEn35hUVLK4YPdm8ZWmUgud3t1Ff\nksWyqnza+n30egIsqUxefsAXZMuxHnY396f8Djzx1nH8oTBfu25B0uta61geQervj9aaXk+QouzU\ng/28vK99zHUB3j7SxZziLGoKR09Ak6hjwEdephO3086jbx5lQVkO1y8pH7WcNxCme9gf214obOEL\nWeRkxE+F/lCYTYe6KMp2semQqXG7/ZKa2PfMGwgDsO1EL7lus947R7t552g3X76qIfZd9ofCeANh\nbDbFE5uPA+CwKb523XxeP9jBzsZ+/vIjC1J+d493DZPlMtvu9wb50dvHx82nxONz2BXvHjUT7TT2\neFJORTvoC9I+4Gd+Wc6obQz7Q3gCYUpzM8bcz7HO4djjYNji9QMdACYv7cllvn5PkA9O9ZLhsHGs\na5jPX1rHY5uOJS0z1nE9vb2JDKeNm5dXUpqbEfucfrnVTOd79/o63j3azbHOYd48aD6rL6yfw4t7\nWukaCpDlsjO3NCdpe4O+EEsq83A5bLQPmABfkpNByLLOqIZiuqRdcPaGvNC+97TW6cg2/Y5/8OYx\noJiF5dnAIDsb+7lucTnP7mihuc/L1QtXs9rm4P1j3XgDYXY09lFdmElzr5f1c4tZP7eIp7c30dTr\nBeC+KxrwhcL85sNmLqopYE19IQ6b4q3DXQC8sr89tt+v37CQxzYfi/3wMxw2fr+rNZbGIX+ISxuK\neHlfe+ykcbhjkI+tqGLjFvNFnVuanfSjeXDDPNxOO0c7h9h2oof8TGdsW73DAd4+2pUUeN881El9\ncRZzirPp9wR5eX/7qFLD0c748u8c6SI7w0F+phNfKMwLu9ti72090WO2ebCT3uEAlzYUse1kL28d\n7uLKBSWU57o51jVkaiyI/zibI3nX1u/jyXdOxLY3pzgLS2va+n24nXY08R9ortvBnWtreXzzcT66\nrByFSrrafmFPayxfjnUNUVOYyaA/RNjSsbwDyM5wsLu5H4DvvHwo9vq+lgEun1fM45ET7H1XNOAN\nhnl+dyv93iDZGXaG/WFuv6SGjkE/80tzyMt0oJTieNdwLJBG0zroC7GuoYgVNfnkup0caBsg2+Wg\nJCeD4UAIp91GhsPG4fYhllfn8f03jpLrdrB6TiFLK/PYuCVeM/HC7jb2tQzgtNtYXp2P22lj2B+K\nXXABHGof5HD7EMe7TB5kOGwc6Rhib8sAVy8sYdOhrtiyWmuOdg7xwak+6ouzeftI/L2NW06xpr6Q\neaU5+EMWW473JH03Nh3qZEF5DjalYheGu5v7eXV/Bzcuq+BUjwdPIMQnLqqKXWhE/eDNoxRkOblz\nbR2WpTnV46GuKIstx3vYcryHK+bHp9D0h8J0DwUIW5riHBdZLkestqY4x4U3EGZXUz+LK/P48FQv\nVy8s5bX9HbHjB/iL6xdgsyme29XCiS4P9189F7tSPPpm6vHnn97eRHGOi/Vzi9l8OJ4ng77k0uwT\nm49z0/IKinNcvLKvg/YBH5mu+Ak/ZGl2NvWzs9F8z376/im+sH4O3kAYl8NGMFITkfiZ2EcE1uY+\nLxV5buyRi3V/KIzDZuPZnc2c6Bpdg/FvrxzmoWvn4bTZkkq7ifnwzAdNnOz2UJzjonvIXLzfvb6O\nkuwMDnUMEghZLKvKJxi2YvsF8ARCvHEwfsvwJ++eID/TSceAn8+uqaG6IDN2YRE1Mg1gquDtNkXH\noA9/0KK2KH4x5g9a/PbDZoqyXdxzeT1HOuLnoJ++N7qW7qn3TsYev7q/g/I8N+0DPuZGvrcA33v9\nCPUlWaPy65YVlbxxsIO8TCcLynJYUz9zc8yn3XzOj7z/bWjZwXAgxO2586nPNCWuIx1D9HqClOVm\n0NrvY+Hqqymeu5rtB46xaaBizO2trMlnV1N/7PnIAJjo/qvnJl0Ruhw2AqHJVfNMRkluBl2DY1cB\npXLVgpKkk8lkuJ12fMFwyvdW1RWw49TpVwvlZTpTlrZnk2jgnK3bSzdjfZfG+41NlYeunUdbv49n\nPji31eSpXFxXELtInUhtUVbSBXNdURanJnFL4eqFJaysKeC7CTUdAKtqC2jt98VKkOOx2xRhS5Ph\ntOEPmvNafqYzViOXyrWLy2Kl6ol85aqG2AXwqroCllTk8fMtycH39ktqeHp706S2F1WU7Tqt2wkj\n173n8vozWncsk53PecLgrJT6EfBxoENrvTzF+wr4d+AWwAPcq7WecK7EaQvOv7kLgMY+L5+05o96\n//3a+9DYzEAkQgghxDgmupVwuiYbnCcToX4M3DTO+zcDCyJ/DwDfn0wCp4VvgLDWNPZ5U7497CpB\nK4cEZiGEELPahPectdablFL14yxyK/ATbYrg7ymlCpRSlVrr1nHWmRbBE5tpSdHyc3/pLQQc2Xgd\n+ec6SUIIIcRpm4oGYdVAY8Lzpshro4KzUuoBTOmaurq6Kdh1sqG2nUnP23KW0ZNVz4C7esr3JYQQ\nIv2lau1+LpzT+l2t9Q+11mu01mtKS0unfPseW3J3jZa8i6Y1MOdlOqkunHwXqpFGdmmZyMiuOjev\nqOD2S2rOeP+TtaQyl69eMy/23OWY3q/NX35kwcQLpVBfMn53mwXlo7uNzCZzS2d+cI2JpPrOrqmf\nuLviVPrM6hpW1oyuBYv2RphuBVmnv5+bllfgtJ/+CX5JZS4ra/JZW1/E0qrTO1+crq/fsHDMbo9L\nKuOv/9m187lyQUnK5c6FhnEGoVlRPXHt6On+zoLhmWk0PRUl52YgcaSPmshr59xrzaYLVZ52sV5X\nse8MtrGuoYg19UV87/V4q8ZMl50vrp/DD0f0zfvylQ1orXnjUCfF2S5e3Z/cKrEy302my84NS8t5\n54jpdxjtqrOgPIeblldQmpsR60sJ5otvU+Cw27AsTVjrpP6tu5r7Odk9zDULSynIMhcjX79hIY09\nHmoKM/nltkYubShO6r4D8MXL5vCf757k3svrOdXjoTQ3A08gTCBk8er+dkIpBpnIz3RitymuW1yO\ny2Hj3svr+fE7J7hyfglhrXnzYCc1hZnUFWXxztFuPrKkHI2mc9DPribTD7so28WW4z3kZTqZU5TF\ndYvL0MDhjkEyHPZYOh/cMI/vv2G6sSilsCmFlaKx4tr6Ii6fV0zQsghbOtL9zbRKX16dz4A3mDQI\nyicuquK5nS0AXDGvJKnr2LKqPPa2DFCWl8Htl9QQtjTBkKZ90EdxtoufvHsyad9leRl8/tI69rcO\n8tLetth3wxsIx7prRI18vra+iFy3g0yXHUekK8rvdrQkbX/93GLy3E52JAyScO/l9bT2+2jq9bC3\nxfTd/+yaGva1DLCuoZhdzX0c6xzmTy6bQ8jSvHu0m7Z+H819Xr5+w0JOdg/zzAfNrJ5TyLKqPDIc\nNiyLWPeWv7h+AYO+EBlOWyz/czIcWFrjCYTJcNq4dVV1rNvaTcsruGZhKYO+IAO+UKyf7MqaAl7a\n28YnL6qifSDeCvrBDfPY2zKQ9B3/y48soGsokNTlJVFid57E55fNK2b93GIA6oqzyHI5eO9YN+sa\nilg3t5juyOAgUUXZLj6+spJD7UO8d6w7aR+JvQdWVOezsjafI+1DFGS5cDttsc9mbX0Riypy0Wgy\nnXYCIYtctxN/KMxrBzrIcztZVVtA97Cf53a2xlov372ujpf2ttEVOY7CLBd/cnl9rL9zUbaL65eU\nUZKTwdHOIbqHAmw/aQbMuWxeMe8e7Y6cIyqT0r2uoYjtJ3uTepA8uGEegbDFUGRsg82HO5lflhPr\nyx3tFhj9/UYtqczlSMdQUvCpL8niUPtg7PnSqjz2tQxQludmf6t53eWwsba+KNYlFOATF1WiNfx+\nVytZLjt3ratjX8tArL81wA1Ly1k+InhG03bF/JJYt70/v34BgZDFo28eZVlVHjcsLeffXol3v7rt\n4moaezyEEvr7O+2KB66ex8G2QXY391NTmEl1YSbvH4t398tw2rj/qrk47TZ+ta0x1uU11+3ghqXl\nnOj20OcJJPUUuHxe8YyNMzWprlSRe86/H6O19seAr2Faa68DHtZaXzrRNqejtfb/95PPMhgOcoNV\nTyYOKj/+//KbXfGTQnmem6JsF/tbzUkuVVeXjy4rZ1lVPmFLo7XGMaI/pi8Y5sl3TrC0Ko+rFowu\n/fd74gMG3LG2dsLBSSxLs7dlgLxMBy6Hjcr8My+JJ3rynROELc1HlpRTmO0kd5zhBQ+0DcT6KH98\nZSXFORljDhrhDYRxO20opWjs8VCWlzGq435Tr4dfbWvi9ktqkvorptI56GfAF2ReaU7sh/r1Gxbi\nC4YJWxqbUrgcNn6w6Si5bidfXD8naf3ooCVfvWZerD/p6wc6KM5xcbxrmI+tqGTzkS4cNhX7vCxL\nc7x7mLkl2bT2+8YcztQfChMKa7Jcdga8IfInKDEN+IKc6vZQX5JNltPOUCDExi2nGPaHU7b4PNY5\nhN2mYjUi0XyMXoT1Dge4dnEZYAZ6eOq9k1y/uDxpEJVUJjO0YtjSBMcZDMUbCPPjd07wqYurqch3\ns3HLKdoGfPzlRybXcrWp14PWJH3+iZ8vmD6yQ/4QWS4H2S47rZH2IrluBy19Pp7f3cqc4iw+vXrs\n2qETXcPUFmVhtym01mw/2UswrHE5FJfMifdRDVuahyP9a91OOzcvr6Aox0WOy5Eyn7TWDPrPflhO\nrTXdwwFKcsYe0CMV890n5fcyFLbY3zrI4spcej0BynLHHyr3SMcguW4n5XluGns8VOS7CVsal93G\ni3vbONg2yH+5bj4Ouw2tNY09Xpr6PPiDFlfML2HriR7WNRTFBpyJfn4dgz4ae7wsrcwj02WnscfD\n09ubuKg2n+sWl6O1ZtvJXmwKCrJczCsdXXPV7w2S4bDhdtoJWxpvMBwbaMQfCuO02bDZzMAvTrtK\nOh9rrXlpbzvLqvJi37O9Lf38cW87q2oLuHZxWWyQqCMdQ1w5vyTWR3tfywAv7W3j/qvnJg1AE92v\n1pzWQEGnYyq7Uv0c2ACUAO3A3wFOAK31o5GuVN/FtOj2AF/SWk8Ydac8OIeD/O1PPgvAkoL7uOPm\nj+J2u9l2oofNh7u4cVlFrFqotd/LsD9ETWEWB9oGefNgJ3etq6Wxx8vFtQVnPV5s15CfA62DXDG/\neEbuVcwGobA16sJmIt95+RBup50HN8ybeOEIbyBMa783aRSg2cQTMKMtne7JebaZivtuW0/0UJjl\nSjkqVSq+YBin3ZY06MXZCIQslGLUACgXslDYYtgfnvDCM7qs3abG/B5ordnfOsiC8pwZy+NwpPZo\nTX3htAXXszVlwXm6THVw1odf4b+/9V0A7rvlP1kQuXdiWZpDHYMsKs+9YAPl+WLIH8JhU7P2RyWE\nEGdrssE5bYbvDA+bewthmysWmMFU7aUae1jMPiOrl4QQ4kKVNvU7bb3mPnJYpb5XKoQQQpwv0iY4\n72o1Le/89tnfHUUIIYQYT9oE51anaWRSUjN6PG0hhBDifJI2wXkobO4522xy31IIIcT5LW2CswuN\nVjby3QUznRQhhBDirKRNcHZrsLBzRdUVM50UIYQQ4qykTXAuyACtbNQUSLcpIYQQ57e0Cc4ZDguH\n3YFCBhoRQghxfkub4BwKB0EpbCptDkkIIcQFKm0i2YeBY6BshKzQxAsLIYQQs1jaBOegDqGUIqQl\nOAshhDi/pU1wRmtAyT1nIYQQ5720Cc41Kh+UjaqcqplOihBCCHFW0iY4u7DjsjmkQZgQQojzXtpE\nsrAOYVcydKcQQojzX9oEZ8sKY1P2mU6GEEIIcdbSJzhjYbdJcBZCCHH+S5vgHLZCUnIWQgiRFtIm\nOFtaSs5CCCHSQxoF5xB25ZzpZAghhBBnLT2Cs2VhWSFsdtdMp0QIIYQ4a+kRnNGE0dilj7MQQog0\nkB7RTGvCWDhs0s9ZCCHE+S89gjOasNbSWlsIIURaSIvgrLWp1nbY0uJwhBBCXODSIppZWqPR2KXk\nLIQQIg2kRXCOlpylQZgQQoh0MKloppS6SSl1UCl1RCn1jRTv36uU6lRK7Yj8fWXqkzoOrbHQOKTk\nLIQQIg1M2LxZKWUHvgfcADQBW5VSz2qt941Y9Bda669NQxonZGkLC2kQJoQQIj1MpuR8KXBEa31M\nax0ANgK3Tm+yTo+lNVqBDTXTSRFCCCHO2mSCczXQmPC8KfLaSJ9RSu1SSj2tlKpNtSGl1ANKqW1K\nqW2dnZ1nkNzULCsMgE3uOQshhEgDUxXNngPqtdYrgZeBJ1MtpLX+odZ6jdZ6TWlp6RTt2lRrA6Ck\n5CyEEOL8N5ng3AwkloRrIq/FaK27tdb+yNPHgUumJnlCCCHEhWcywXkrsEAp1aCUcgGfA55NXEAp\nVZnw9JPA/qlL4sR0pOSspFpbCCFEGpiwtbbWOqSU+hrwEmAHfqS13quU+h/ANq31s8CfK6U+CYSA\nHuDeaUzzKJalAanVFkIIkR4mNVOE1vp54PkRr30z4fHfAn87tUmbPAvTIEylx5gqQgghLnDpEc20\nKTlLa20hhBDpIC2iWX/WwhYAAAcoSURBVKRWW6q1hRBCpIW0CM7RBmHIICRCCCHSQFoEZ8uS1tpC\nCCHSR1pEM4201hZCCJE+0iI4W5EGYdJaWwghRDpIq2gmJWchhBDpIC2Cs47ec06PwxFCCHGBS4to\nprUZhESKzkIIIdJBegTnyH+bBGchhBBpID2Cc2ziCwnOQgghzn9pEZxj/ZxlEBIhhBBpIC2Cs9bR\nim0JzkIIIc5/6RGcY4OQSHAWQghx/kuL4GxFgrNNSs5CCCHSQFoE51iDMJsEZyGEEOe/tAjO0QZh\naXI4QgghLnDpEc2i8zlLtbYQQog0kB7BOdogTKq1hRBCpIG0CM46NiuVBGchhBDnv7QIztEpI2X4\nTiGEEOkgPYIzZuIL6ecshBAiHaRFcI7NfCHV2kIIIdJAWgRnHavWTovDEUIIcYFLi2hmxWalSovD\nEUIIcYFLi2gWn/hCCCGEOP+lSXA2JWdprS2EECIdpEdwRobvFEIIkT4mFc2UUjcppQ4qpY4opb6R\n4v0MpdQvIu+/r5Sqn+qEjidaqy1dqYQQQqSDCYOzUsoOfA+4GVgK3KWUWjpisS8DvVrr+cB3gG9P\ndULHY1fgsClcdik5CyGEOP9NJppdChzRWh/TWgeAjcCtI5a5FXgy8vhp4Hp1DouxxTkuKvPcFGa7\nztUuhRBCiGnjmMQy1UBjwvMmYN1Yy2itQ0qpfqAY6EpcSCn1APBA5OmQUurgmSR6DCV/xsauiRcT\nEyhhxOcmTpvk4dmTPDx7kodnbzrycM5kFppMcJ4yWusfAj+cjm0rpbZprddMx7YvJJKPZ0/y8OxJ\nHp49ycOzN5N5OJlq7WagNuF5TeS1lMsopRxAPtA9FQkUQgghLjSTCc5bgQVKqQallAv4HPDsiGWe\nBe6JPL4deE3LyCBCCCHEGZmwWjtyD/lrwEuAHfiR1nqvUup/ANu01s8CTwD/qZQ6AvRgAvi5Ni3V\n5RcgycezJ3l49iQPz57k4dmbsTxUUsAVQgghZhfpGCyEEELMMhKchRBCiFkmLYLzRMOLXsiUUj9S\nSnUopfYkvFaklHpZKXU48r8w8rpSSj0cycddSqnVCevcE1n+sFLqnlT7SldKqVql1OtKqX1Kqb1K\nqb+IvC75OElKKbdSaotSamckD/8h8npDZMjfI5EhgF2R18ccElgp9beR1w8qpW6cmSOaOUopu1Lq\nQ6XU7yPPJQ9Pk1LqhFJqt1Jqh1JqW+S12fV71lqf13+YRmpHgbmAC9gJLJ3pdM2WP+BqYDWwJ+G1\n/wV8I/L4G8C3I49vAV4AFLAeeD/yehFwLPK/MPK4cKaP7RzmYSWwOvI4FziEGcpW8nHyeaiAnMhj\nJ/B+JG9+CXwu8vqjwIORxw8Bj0Yefw74ReTx0shvPANoiPz27TN9fOc4L/8K+Bnw+8hzycPTz8MT\nQMmI12bV7zkdSs6TGV70gqW13oRpQZ8ocbjVJ4HbEl7/iTbeAwqUUpXAjcDLWuserXUv8DJw0/Sn\nfnbQWrfq/7+9e2eNIgrDOP5/i6Cighc0hSl0QbASBRHFFCFgwAvaCoKifgErQQJ+BNHCTktREBXt\nvKYXolEj8bKiTYhuFS+NqLwW512dJRvcDSFzsvv8YNiZ2WWZeeDMycw5edf9aax/AyZIVfGUY4si\ni++x2ROLA4Okkr8wM8NmJYEPA9fd/Ye7fwCqpGtAVzCzPuAAcDm2DWU4X7Jqz53QOTcrL7qhpGNZ\nLHrdfSrWPwG9sT5blso4xKPB7aQ7P+XYhngcOwbUSBey98C0u/+KjxTzaCgJDNRLAnd1hsAF4Az8\n/Z3ctSjDuXDgvpmNWiorDZm15wUt3yn5cXc3M/0/XQvMbAVwEzjt7l+t8NsuyvH/3P03sM3MVgG3\ngS0lH9KiYmYHgZq7j5rZQNnHs8j1u/ukma0HHpjZ6+KbObTnTrhzbqW8qDT6HI9liNda7J8ty67P\n2Mx6SB3zVXe/FbuV4xy4+zQwAuwmPSKs3yQU85itJHA3Z7gHOGRmH0nDd4PARZRh29x9Ml5rpD8U\nd5JZe+6EzrmV8qLSqFhu9Thwp7D/WMxO3AV8icc894AhM1sdMxiHYl9XiHG6K8CEu58vvKUcW2Rm\n6+KOGTNbBuwljd2PkEr+wswMm5UEvgsciZnIm4DNwJOFOYtyuftZd+9z942k69xjdz+KMmyLmS03\ns5X1dVI7HCe39lz2rLn5WEiz6d6SxrCGyz6enBbgGjAF/CSNiZwijTs9At4BD4E18VkDLkWOL4Ed\nhe85SZo4UgVOlH1eC5xhP2mM6gUwFst+5dhWhluBZ5HhOHAu9ldIHUMVuAEsif1LY7sa71cK3zUc\n2b4B9pV9biXlOcC/2drKsL3sKqTZ6s+BV/U+I7f2rPKdIiIimemEx9oiIiIdRZ2ziIhIZtQ5i4iI\nZEads4iISGbUOYuIiGRGnbOIiEhm1DmLiIhk5g+kZ+n9SFP7xwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 576x216 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "47bk4a_-qXMS",
        "colab_type": "text"
      },
      "source": [
        "Although ε-greedy action selection is an effective and popular means of balancing exploration and exploitation in reinforcement learning, one drawback is that when it explores **it chooses equally among all actions**. \n",
        "\n",
        "This means that it is as likely to choose the worst-appearing action as it is to choose the next-to-best action. In tasks where the worst actions are very bad, this may\n",
        "be unsatisfactory. The obvious solution is to vary the action probabilities as a graded function of estimated value.\n",
        "\n",
        "The greedy action will be given the highest selection probability, but all the others will be ranked and weighted according to\n",
        "their value estimates. To this end we can use a softmax action selection rule (with a temperature parameter $\\tau$):\n",
        "\n",
        "$$\n",
        "\\frac{ e^{Q_t(a)/ \\tau}}{\\sum_{i=1}^n e^{Q_t(i)/ \\tau}}\n",
        "$$\n",
        "\n",
        "For high temperatures ( $\\tau \\to \\infty $ ), all actions have nearly the same probability. For a low temperature ( $ \\tau \\to 0^{+} $), the probability of the action with the highest expected reward tends to 1."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dNhWHWZSqXMT",
        "colab_type": "text"
      },
      "source": [
        "### Contextual Bandit\n",
        "\n",
        "In the bandit problem described above, it is assumed that **nothing is known about each arm other than what we have learned from prior pulls**.\n",
        "\n",
        "We can relax this assumption and assume that for each arm there is a d-dimensional “context” vector. For example, if each arm represents a digital ad, the features in these vectors may correspond to things like banner size, web browser type, font color, etc. We can now model the value of each arm using these context vectors as well as past rewards in order to inform our choice of which arm to pull. This scenario is known as the contextual bandit problem."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h-ayQwNIqXMT",
        "colab_type": "text"
      },
      "source": [
        "## Q-learning\n",
        "\n",
        "Unlike other methods, which attempt to learn functions which directly map an observation to an action, Q-Learning attempts **to learn the value of being in a given state, and taking a specific action there**. \n",
        "\n",
        "The agent is in a state $s$ and has to choose one action $a$, upon which it receives a reward $r$ and come to a new state $s’$. The way the agent chooses actions is called **policy**.\n",
        "\n",
        "Let’s define a function $Q(s, a)$ such that for given state $s$ and action $a$ it returns an estimate of a total reward we would achieve starting at this state, taking the action and then following some policy. Under certain conditions, there certainly exist policies that are optimal, meaning that they always select an action which is the best in the context. Let’s call the $Q$ function for these optimal policies $Q^*$.\n",
        "\n",
        "If we knew the true $Q^*$ function, the solution would be straightforward. We would just apply a greedy policy to it. That means that in each state $s$, we would just choose an action $a$ that maximizes the function $Q^*$, $argmax_a Q^*(s, a)$. Knowing this, our problem reduces to find a good estimate of the $Q^*$ function and apply the greedy policy to it.\n",
        "\n",
        "Let’s write a formula for this function in a symbolic way. It is a sum of rewards we achieve after each action, but we will discount every member with γ:\n",
        "\n",
        " $$ Q^*(s, a) = r_0 + \\gamma r_1 + \\gamma^2 r_2 + \\gamma^3 r_3 + ... $$\n",
        "\n",
        "$\\gamma$ is called a discount factor and when set it to $\\gamma < 1$ , it makes sure that the sum in the formula is finite. Value of each member exponentially diminish as they are more and more in the future and become zero in the limit. The $\\gamma$ therefore controls how much the function $Q$ in state $s$ depends on the future and so it can be thought of as how much ahead the agent sees. \n",
        "\n",
        "Typically we set it to a value close, but lesser to one. The actions are chosen according to the greedy policy, maximizing the $Q^*$ function.\n",
        "\n",
        "When we look again at the formula, we see that we can write it in a recursive form:\n",
        "\n",
        " $$Q^*(s, a) = r_0 + \\gamma (r_1 + \\gamma r_2 + \\gamma^2 r_3 + ...) = r_0 + \\gamma max_a Q^*(s', a)$$\n",
        " \n",
        "We just derived a so called **Bellman equation**.\n",
        " \n",
        "One of the possible strategies to solve the Bellman equation is by applying the **Q-learning** algorithm:\n",
        "\n",
        "```\n",
        "For each state-action pair (s, a), initialize the table entry Q(s,a) to zero\n",
        "Observe the current state s\n",
        "Do forever:\n",
        "- Select an action a from s and execute it \n",
        "- Receive immediate reward r\n",
        "- Observe the new state s'\n",
        "- Update the table entry for Q\n",
        "- s=s'\n",
        "```\n",
        "#### Action selection\n",
        "\n",
        "We could apply different strategies for action selection:\n",
        "+ **Random approach**. Only in circumstances where a random policy is optimal would this approach be ideal.\n",
        "+ **$\\epsilon$- greedy approach**:  A simple combination of the greedy and random approaches yields one of the most used exploration strategies. At the start of the training process the $\\epsilon$ value is often initialized to a large probability, to encourage exploration in the face of knowing little about the environment. The value is then annealed down to a small constant (often 0.1), as the agent is assumed to learn most of what it needs about the environment. Despite the prevalence of usage that it enjoys, this method is far from optimal, since it takes into account only whether actions are most rewarding or not.\n",
        "+ **Boltzmann Approach**. Instead of always taking the optimal action, or taking a random action, this approach involves choosing an action with weighted probabilities. To accomplish this we use a softmax over the networks estimates of value for each action.  In practice we utilize an additional temperature parameter ($\\tau$) which is annealed over time. While this measure can be a useful proxy, it is not exactly what would best aid exploration. What we really want to understand is the agent’s uncertainty about the value of different actions.\n",
        "+ **Bayesian Approaches**. What if an agent could exploit its own uncertainty about its actions? This is exactly the ability that a class of neural network models referred to as Bayesian Neural Networks (BNNs) provide. Unlike traditional neural network which act deterministically, BNNs act probabilistically. This means that instead of having a single set of fixed weights, a BNN maintains a probability distribution over possible weights. In a reinforcement learning setting, the distribution over weight values allows us to obtain distributions over actions as well. The variance of this distribution provides us an estimate of the agent’s uncertainty about each action. In order to get true uncertainty estimates, multiple samples are required, thus increasing computational complexity.\n",
        "\n",
        "\n",
        "#### Table updating\n",
        "\n",
        "The table entry for $Q$ is updated by using this formula:\n",
        "\n",
        "$$\n",
        "Q(s,a) = Q(s,a) + \\alpha [r + \\gamma max_{a'} Q(s',a') - Q(s,a) ]\n",
        "$$\n",
        "\n",
        "where \n",
        "\n",
        "+ $0<\\alpha<1$ is the learning rate. Setting it to 0 means that the Q-values are never updated, hence nothing is learned. Setting a high value such as 0.9 means that learning can occur quickly.\n",
        "+ $0<\\gamma<1$ is the discount factor. This models the fact that future rewards are worth less than immediate rewards. \n",
        "\n",
        "See http://www.scholarpedia.org/article/Temporal_difference_learning for a short description."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2MWg2N2hqXMU",
        "colab_type": "text"
      },
      "source": [
        "### Solving the FrozenLake problem\n",
        "\n",
        "We are going to to solve the FrozenLake environment from the OpenAI gym. \n",
        "\n",
        "OpenAI gym (https://gym.openai.com/) provides an easy way for people to experiment with their learning agents in an array of provided toy games. The FrozenLake environment consists of a 4x4 grid of blocks, each one either being the start block, the goal block, a safe frozen block, or a dangerous hole. \n",
        "\n",
        "> **FrozenLake-v0**\n",
        "\n",
        "> The agent controls the movement of a character in a grid world. Some tiles of the grid are walkable, and others lead to the agent falling into the water. Additionally, the movement direction of the agent is uncertain and only partially depends on the chosen direction. The agent is rewarded for finding a walkable path to a goal tile.\n",
        "\n",
        "> The surface is described using a grid like the following:\n",
        "\n",
        ">``SFFF       (S: starting point, safe)``\n",
        "\n",
        ">``FHFH       (F: frozen surface, safe)``\n",
        "\n",
        ">``FFFH       (H: hole, fall to your doom)``\n",
        "\n",
        ">``HFFG       (G: goal, where the frisbee is located)``\n",
        "\n",
        "> The episode ends when you reach the goal or fall in a hole. You receive a reward of 1 if you reach the goal, and zero otherwise.\n",
        "\n",
        "The objective is to have an agent learn to navigate from the start to the goal without moving onto a hole. At any given time the agent can choose to move either ``up, down, left``, or ``right``. \n",
        "\n",
        "The catch is that there is a wind which occasionally blows the agent onto a space they didn’t choose. As such, perfect performance every time is impossible, but learning to avoid the holes and reach the goal are certainly still doable. The reward at every step is 0, except for entering the goal, which provides a reward of 1. Thus, we will need an algorithm that learns long-term expected rewards. This is exactly what Q-Learning is designed to provide."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KicUducwqXMU",
        "colab_type": "text"
      },
      "source": [
        "In it’s simplest implementation, Q-Learning is a table of values for every state (row) and action (column) possible in the environment. Within each cell of the table, we learn a value for how good it is to take a given action within a given state. \n",
        "\n",
        "In the case of the FrozenLake environment, we have 16 possible states (one for each block), and 4 possible actions (the four directions of movement), giving us a $16 \\times 4$ table of Q-values. We start by initializing the table to be uniform (all zeros), and then as we observe the rewards we obtain for various actions, we update the table accordingly."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qqh9n7XyqXMV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import gym\n",
        "import numpy as np\n",
        "env = gym.make('FrozenLake-v0')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3F-wv_ViqXMY",
        "colab_type": "code",
        "outputId": "1ac26626-c745-43e4-a587-4321e829def5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 332
        }
      },
      "source": [
        "#Initialize table with all zeros\n",
        "Q = np.zeros([env.observation_space.n,env.action_space.n])\n",
        "\n",
        "# Set learning parameters\n",
        "lr = .9\n",
        "gamma = 0.95\n",
        "num_episodes = 10000\n",
        "\n",
        "#create lists to contain total rewards and steps per episode\n",
        "rList = []\n",
        "\n",
        "for i in range(num_episodes):\n",
        "    #Reset environment and get first new observation\n",
        "    s = env.reset()\n",
        "    rAll = 0\n",
        "    d = False\n",
        "    j = 0\n",
        "    \n",
        "    #The Q-Table learning algorithm\n",
        "    while j < 999999:\n",
        "        j+=1\n",
        "        #Choose an action by greedily (with noise) picking from Q table\n",
        "        a = np.argmax(Q[s,:] + np.random.randn(1,env.action_space.n)*(1./(i+1)))\n",
        "        #Get new state and reward from environment\n",
        "        s1,r,d,_ = env.step(a)\n",
        "        #Update Q-Table with new knowledge\n",
        "        Q[s,a] = Q[s,a] + lr*(r + gamma*np.max(Q[s1,:]) - Q[s,a])\n",
        "        rAll += r\n",
        "        s = s1\n",
        "        if d == True:\n",
        "            break\n",
        "    rList.append(rAll)\n",
        "    \n",
        "print(\"Score over time: \" +  str(sum(rList[-100:])/100))\n",
        "print(\"Final Q-Table Values\")\n",
        "print(Q)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Score over time: 0.8\n",
            "Final Q-Table Values\n",
            "[[1.70216414e-01 9.91289589e-04 8.07651609e-04 9.94384986e-04]\n",
            " [3.11849273e-05 2.41157068e-04 2.23941046e-04 3.10609046e-01]\n",
            " [2.52090813e-01 1.00321309e-04 1.47766959e-04 6.91033227e-04]\n",
            " [6.06716105e-04 6.52081250e-04 7.59932127e-05 6.45606522e-04]\n",
            " [6.93535446e-02 1.41301294e-04 6.20534292e-04 2.38752473e-07]\n",
            " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
            " [1.77946039e-07 8.62823977e-08 1.47888298e-01 4.44273898e-07]\n",
            " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
            " [1.79765630e-05 1.62084109e-04 5.82167070e-06 5.55430222e-02]\n",
            " [2.78345654e-05 6.73250716e-01 6.70158380e-05 1.85569486e-05]\n",
            " [9.11405139e-01 8.56105056e-06 1.65664270e-06 2.37234780e-05]\n",
            " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
            " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
            " [0.00000000e+00 0.00000000e+00 5.98543694e-01 0.00000000e+00]\n",
            " [0.00000000e+00 9.57241931e-01 0.00000000e+00 5.06933574e-03]\n",
            " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qowXSM2NqXMb",
        "colab_type": "text"
      },
      "source": [
        "FrozenLake-v0 is considered \"solved\" when the agent obtains an average reward of at least 0.78 over 100 consecutive episodes."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hclfnwZyqXMf",
        "colab_type": "text"
      },
      "source": [
        "## Q-Learning with Neural Networks\n",
        "\n",
        "Now, you may be thinking: tables are great, but they don’t really scale, do they? While it is easy to have a 16x4 table for a simple grid world, the number of possible states in any modern game or real-world environment is nearly infinitely larger. For most interesting problems, tables simply don’t work. \n",
        "\n",
        "We instead need some way to take a description of our state, and produce $Q$-values for actions without a table: that is where neural networks come in. By acting as a function approximator, we can take any number of possible states that can be represented as a vector and learn to map them to $Q$-values."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rstH67xgqXMg",
        "colab_type": "text"
      },
      "source": [
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/pong.jpg?raw=1\" alt=\"\" style=\"width: 300px;\"/>\n",
        "\n",
        "\n",
        "In the case of the FrozenLake example, we will be using a one-layer network which takes the state encoded in a one-hot vector (1x16), and produces a vector of 4 $Q$-values, one for each action. \n",
        "\n",
        "Such a simple network acts kind of like a glorified table, with the network weights serving as the old cells. The key difference is that we can easily expand the Tensorflow network with added layers, activation functions, and different input types, whereas all that is impossible with a regular table. \n",
        "\n",
        "The method of updating is a little different as well. \n",
        "\n",
        "Instead of directly updating our table, with a network we will be using backpropagation and a loss function. Our loss function will be sum-of-squares loss, where the difference between the current predicted $Q$-values, and the “target” value is computed and the gradients passed through the network. \n",
        "\n",
        "In this case, our $Q_{target}$ for the chosen action is the equivalent to the $Q$-value computed in equation above ($\n",
        "Q(s,a) + \\alpha [r + \\gamma max_{a'} Q(s',a') - Q(s,a) ]\n",
        "$).\n",
        "\n",
        "$$\n",
        "Loss = \\sum (Q_{target} - Q_{predicted})^2\n",
        "$$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VNdiYcYyqXMg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import gym\n",
        "import numpy as np\n",
        "import random\n",
        "import tensorflow as tf\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "\n",
        "env = gym.make('FrozenLake-v0')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jac8lo1jqXMi",
        "colab_type": "code",
        "outputId": "823c0c0b-78df-4630-bd7c-b6418d989877",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 89
        }
      },
      "source": [
        "tf.reset_default_graph()\n",
        "\n",
        "#These lines establish the feed-forward part of the network used to choose actions\n",
        "inputs1 = tf.placeholder(shape=[1,16],dtype=tf.float32)\n",
        "W = tf.Variable(tf.random_uniform([16,4],0,0.01))\n",
        "Qout = tf.matmul(inputs1,W)\n",
        "predict = tf.argmax(Qout,1)\n",
        "\n",
        "#Below we obtain the loss by taking the sum of squares difference between the target and prediction Q values.\n",
        "nextQ = tf.placeholder(shape=[1,4],dtype=tf.float32)\n",
        "loss = tf.reduce_sum(tf.square(nextQ - Qout))\n",
        "trainer = tf.train.GradientDescentOptimizer(learning_rate=0.1)\n",
        "updateModel = trainer.minimize(loss)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Colocations handled automatically by placer.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eeuHz50PqXMk",
        "colab_type": "code",
        "outputId": "052f4230-24a0-4e2c-bd47-d8a134ab74ef",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "init = tf.global_variables_initializer()\n",
        "\n",
        "# Set learning parameters\n",
        "y = .99\n",
        "e = 0.1\n",
        "num_episodes = 2000\n",
        "\n",
        "#create lists to contain total rewards and steps per episode\n",
        "jList = []\n",
        "rList = []\n",
        "with tf.Session() as sess:\n",
        "    sess.run(init)\n",
        "    for i in range(num_episodes):\n",
        "        #Reset environment and get first new observation\n",
        "        s = env.reset()\n",
        "        rAll = 0\n",
        "        d = False\n",
        "        j = 0\n",
        "        #The Q-Network\n",
        "        while j < 99:\n",
        "            j+=1\n",
        "            #Choose an action by greedily (with e chance of random action) from the Q-network\n",
        "            a,allQ = sess.run([predict,Qout],\n",
        "                              feed_dict={inputs1:np.identity(16)[s:s+1]})\n",
        "            if np.random.rand(1) < e:\n",
        "                a[0] = env.action_space.sample()\n",
        "            #Get new state and reward from environment\n",
        "            s1,r,d,_ = env.step(a[0])\n",
        "            #Obtain the Q' values by feeding the new state through our network\n",
        "            Q1 = sess.run(Qout,\n",
        "                          feed_dict={inputs1:np.identity(16)[s1:s1+1]})\n",
        "            #Obtain maxQ' and set our target value for chosen action.\n",
        "            maxQ1 = np.max(Q1)\n",
        "            targetQ = allQ\n",
        "            targetQ[0,a[0]] = r + y*maxQ1\n",
        "            #Train our network using target and predicted Q values\n",
        "            _,W1 = sess.run([updateModel,W],\n",
        "                            feed_dict={inputs1:np.identity(16)[s:s+1],\n",
        "                                       nextQ:targetQ})\n",
        "            rAll += r\n",
        "            s = s1\n",
        "            if d == True:\n",
        "                #Reduce chance of random action as we train the model.\n",
        "                e = 1./((i/50) + 10)\n",
        "                break\n",
        "        jList.append(j)\n",
        "        rList.append(rAll)\n",
        "print(\"Percent of succesful episodes: \" + str(sum(rList[-100:])/100) + \"%\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Percent of succesful episodes: 0.66%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sh7im78ZqXMm",
        "colab_type": "text"
      },
      "source": [
        "We can see that the network beings to consistly reach the goal around the 400 episode mark."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nwkXMdciqXMm",
        "colab_type": "code",
        "outputId": "bfcd61ff-2259-46d1-8269-7ced2e96c415",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 286
        }
      },
      "source": [
        "plt.plot(rList)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f16941a1b00>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAGkdJREFUeJzt3X10XPWd3/H315IlWU+2nmxkS7Zs\nY4gNhmDEQ0IS6AKJ7WxsmqRZOOSEbDjrpl16sifpA2lamrJ72pC09Gy2dFmyySHhZENYNrt1E6cm\nzRKyy8YsMo82YCyMjS2MJeRn/CDL/vaPuZLHoxnNSJ65M/rN53WOjmZ+9zf3fnVn9NHV7zd3rrk7\nIiISlmnFLkBERPJP4S4iEiCFu4hIgBTuIiIBUriLiARI4S4iEiCFu4hIgBTuIiIBUriLiASoslgb\nbm1t9a6urmJtXkRkStq8efO77t6WrV/Rwr2rq4uenp5ibV5EZEoys1259NOwjIhIgBTuIiIBUriL\niARI4S4iEiCFu4hIgLKGu5l9z8z6zWxLhuVmZt82s14ze8nMVuS/TBERmYhcjtwfBlaOs3wVsCT6\nWgf86fmXJSIi5yPr+9zd/ddm1jVOl7XADzxxvb5NZjbLzNrdfW+easzZ4NGT/OOb+1m1vB2A/sMn\neHHPIW5eNidt/4PHhni6d5CPX9Y+Ztnw6TP85Pk+Pr2ig2nT7JxlW/oO8cyb+/ng4haWtjeOtrs7\nf/VcHzctnc2f/92btNRX0VRbxS1XzOP+J7bxyKZdrFrezr+8YTHHhk7z+OY9XHdhK//nxbfpaJrB\nF69fTP/hkzzz5iD/d8s7vPbOEfoOHueTV8zjso6Z/Nmvd7B6eTsv7D5IzfRpzG+uY2l7A8/tOsD2\n/qPsO3yCxhnT+cRlc7l52Ry+/w872fzWAdzh9BlnQUstZ9xZMb+JP/nbXgA+cflc3hp8j7aGam5c\nOofHenbz+jtH6GyuZVFbHYvb6jl0/BT7Dp9gYWs92945zMHjp6ivruSqrmbc4YaL2/jmxtc4dPwU\nW/oOs6itjgoztvcfBaCqYhrDZ87Q1lBNXVUlQ6fP8L4LGvn73gFqpldwydxGnu4dBOCTK+bx5Gv9\ndDbX8tKeQyyfNxOA5roqnnp9gIaaSiqnGZd3zmJ2QzW1VZW8ffA4LfVV/M3zb3P81GkAPri4hYpp\nxrGh07zcd4ih4TMAzJ1ZQ1tjDcvaG3hx9yHePnSc4dPOVV1N/Or1AVZdegGdzbU83rOHzuZa3js5\nPPpz1EyfxoLmOrb3H6Eu+vmPnDjF4NEhTpw6zaK2eua31NJQXckjm3bR2VTLrv3v0dlUy/b+ozRU\nV/Lhi1q5ZO5MmuuqOD50mnt/+gpXdTXx7M4DAFy5oIktfYc4OXyGBS217Bo8RsU0459cPJum2uls\n3nWAd4+e5PCJYWbOmE5LfRV7D57g+KnTzJwxndqqCvYeOsHt18xn69uHOTY0zGUds9h76DgHj53i\nso5Z/OaNd6mtquTE8Gn2HjxBS30VC1vrOHz8FL39R2mur6KuqpKm2ioGjp5k5ozpXHxBAz9/eS9m\nxtGTw1ROM66/qI3jp05TV1XJ9v4jzKiqZPu+I8xvrmX/e0Nc1dXM4ROn2LzrADcuncNzuw4wd1YN\n7rD30An2vzfE6TPO3Fk1rFjQxE+e66N7QRMdTTPYd/gkv9mReE1UTjPM4LoLW/nVtoHR37fpFUZr\nfTWzaqs4dfoMg0dPcuDYKZa1N/LK3sOjj+1qraO3/ygNNZWJ52TwPS6+oIHn3jrIotY6OptrAXjq\n9QGumD+LrX2HWTKnntqqCq5e2MyeA8d55e3DzG6s5vjQaZ576yANNZUsaKllS99hlrY3sritjl2D\nx3i57xAXzq6nt/8oC1pq6Wqp472Tw7y1/xj9R04yp7Ga1cvbueX987i8c1YOqTZ5lss1VKNw/6m7\nX5pm2U+Bb7j730f3fwn8O3cfc4aSma0jcXTP/Pnzr9y1K6f34ufslgee5oXdB3n+P95MU10VN3zr\nSXYOHmPHf1k9JqABbv/zTTzdO8jTd/8W82bNOGfZg0+9wTd+/hr3fWo5v3PV/HOWdd39s9HbO7/x\n8dHbv3x1H3d+f+yJWX986/v50qMvjN5vqK7kyMnhMf0+/8EuHv6HnTn/vCIyNf3RLZfy2WsXTOqx\nZrbZ3buz9Yt1QtXdH3L3bnfvbmvLevbshO05cAyA4TOJP1g7BxP3bWyuA9B34DjA6FFdsv3vDQFw\n8NipnLd/5MTYwE5e12i/NMEOMHD0ZM7bEpGpa7LBPhH5CPc+oDPpfkfUJiIiRZKPcF8PfC5618y1\nwKFijLeLiMhZWSdUzexHwA1Aq5ntAf4TMB3A3R8ENgCrgV7gGPC7hSpWRERyk8u7ZW7LstyB389b\nRQXgnnncXUQkRDpDVUQkQAp3EZEAKdxFRAKkcBcRCVBZhHv2c3BFRMJSFuEuIlJuFO4iIgFSuIuI\nBEjhLiISoLII91w+1lhEJCRlEe4iIuVG4S4iEiCFu4hIgBTuIiIBKotwL/Z0quZzRSRuZRHuIiLl\nRuEeA10oRETipnAXEQmQwl1EJEBlEe7FntAs9vZFpPyURbiLiJQbhXsMNKEqInFTuIuIBEjhLiIS\noLIIdy/yOaqaUBWRuJVFuIuIlBuFeww0oSoicVO4i4gESOEuIhKgsgj3Yk9oFnv7IlJ+cgp3M1tp\nZtvMrNfM7k6zfL6ZPWlmz5vZS2a2Ov+liohIrrKGu5lVAA8Aq4BlwG1mtiyl238AHnP3K4Bbgf+V\n70KnMk2oikjccjlyvxrodfcd7j4EPAqsTenjQGN0eybwdv5KFBGRiarMoc88YHfS/T3ANSl9vg48\nYWb/CqgDbspLdSIiMin5mlC9DXjY3TuA1cAjZjZm3Wa2zsx6zKxnYGAgT5sufZpQFZG45RLufUBn\n0v2OqC3ZncBjAO7+G6AGaE1dkbs/5O7d7t7d1tY2uYpFRCSrXML9WWCJmS00syoSE6brU/q8BdwI\nYGZLSYR7+RyaZ6EJVRGJW9Zwd/dh4C5gI/AqiXfFbDWze81sTdTtK8DvmdmLwI+Az7trMEJEpFhy\nmVDF3TcAG1La7km6/QpwXX5LExGRydIZqmWwfREpP2UR7iIi5UbhLiISIIV7DPRuGRGJm8JdRCRA\nZRHuuoaqiJSboMM97uEQDb+ISKkIOtzjPmLWEbqIlIqgw71U6IheROKmcBcRCVBZhHuxh0uKvX0R\nKT9Bh7smVEWkXAUd7ppQFZFyFXS4lwod0YtI3BTuIiIBCircR4ZFUs9IzTRaMtKe7roiI235GGnJ\nebhGwzoikidBhXupUmaLSNyCCveJjm2PdLc0Dxxpy8dwec7r0Ni8iORJUOFeqjShKiJxU7iLiAQo\nqHAfnbj01Pb0o96aUBWRUAUV7qVKmS0icQsq3DWhKiKSEFS4lypNqIpI3BTuIiIBCircz56hmtKe\nqf/o4zShKiJhCSrcS5UyW0TiFlS4a0JVRCQhqHAXEZEEhXsM9G4ZEYlbUOE+OqE65gzVDP1Hl2tC\nVUTCklO4m9lKM9tmZr1mdneGPp8xs1fMbKuZ/UV+y5zalNkiErfKbB3MrAJ4ALgZ2AM8a2br3f2V\npD5LgK8C17n7ATObXaiCx691gv1HH6cJVREJSy5H7lcDve6+w92HgEeBtSl9fg94wN0PALh7f37L\nFBGRicgl3OcBu5Pu74nakl0EXGRmT5vZJjNbmW5FZrbOzHrMrGdgYGByFU9BmlAVkbjla0K1ElgC\n3ADcBnzHzGaldnL3h9y9292729ra8rRpERFJlUu49wGdSfc7orZke4D17n7K3d8EXicR9kWReoHs\nYs9o5vxuGRGRPMkl3J8FlpjZQjOrAm4F1qf0+RsSR+2YWSuJYZodeaxzUuIeDtHwi4iUiqzh7u7D\nwF3ARuBV4DF332pm95rZmqjbRmDQzF4BngT+jbsPFqroXMV9xKwjdBEpFVnfCgng7huADSlt9yTd\nduDL0Zek0BG9iMQtqDNURUQkIchwH/PxA0WeUdVwjYjELchwH6EJVREpV0GHuyZURaRcBR3upUJH\n9CISN4W7iEiAggz3MRfI1hmqIlJmggz3EZpQFZFyFXS4a0JVRMpV0OFeKnRELyJxU7iLiAQoyHBP\nveB1ttGSdBfIzicN14hI3IIM9xGaUBWRchV0uGtCVUTKVdDhLiJSrhTuMdBwjYjELchwH/ORv1nG\nSwo9mqLhGhGJW5DhPkITqiJSroIOd02oiki5CjrcRUTKlcI9BhquEZG4lUW4Zz9DtcDb13CNiMQs\n6HDXhKqIlKugw11EpFwFHe56t4yIlKugw71UaLhGROIWZLiPPUM16yMKVUqO2xcRya8gw32EJlRF\npFwFHe4iIuUq6HDXhKqIlKucwt3MVprZNjPrNbO7x+n3KTNzM+vOX4kiIjJRWcPdzCqAB4BVwDLg\nNjNblqZfA/Al4Jl8FzlRTuo1VLN85G+hz1At7OpFRMbI5cj9aqDX3Xe4+xDwKLA2Tb8/BO4DTuSx\nvvOiCVURKVe5hPs8YHfS/T1R2ygzWwF0uvvP8libiIhM0nlPqJrZNOB+4Cs59F1nZj1m1jMwMHC+\nm85KE6oiUq5yCfc+oDPpfkfUNqIBuBT4lZntBK4F1qebVHX3h9y9292729raJl+1iIiMK5dwfxZY\nYmYLzawKuBVYP7LQ3Q+5e6u7d7l7F7AJWOPuPQWpOAdjjqCzHFEX/hqqOqQXkXhlDXd3HwbuAjYC\nrwKPuftWM7vXzNYUusDzoQlVESlXlbl0cvcNwIaUtnsy9L3h/MsSEZHzoTNUp/D2REQyCTrcRUTK\nVZDhPsH5VB1xi0hwggz3EZpQFZFyFXS4i4iUq6DDXROqIlKugg53EZFyFVS4jxw5p54RmumI2ke/\nj+0wso58HIznfESvI38RyZOgwr1UZfs8eRGRfAsq3FPfrZLt3SvjLbbowRN5A0ym7Vmua9G7bUQk\nT4IK91SaUBWRchV0uIuIlKugwn10QjW1PcOY9+iEaprFeZ1QzXUtOvIXkTwJKtxLlYZrRCRuQYV7\nyU6o5roSTaiKSJ4EFe4iIpIQdLjr3TIiUq6CCvezZ6imbx/Tf5zlOkNVRKayoMK9VCmzRSRuQYV7\nyU6o5ryCCWxMRGQcQYW7iIgkBB3umlAVkXIVVLifDdeUj/zN1H/0e4E/8jfvHUVExhdUuJcqHdGL\nSNyCCveSnVDVGaoiErOgwl1ERBKCDndNqIpIuQoq3DOfoVrkj/zVGaoiErOgwr1U6RqqIhK3oMK9\nZCdUdQ1VEYlZUOEuIiIJOYW7ma00s21m1mtmd6dZ/mUze8XMXjKzX5rZgvyXOnGaUBWRcpU13M2s\nAngAWAUsA24zs2Up3Z4Hut39MuBx4Jv5LjQXGa+hmuUjf9OvS9dQFZGpK5cj96uBXnff4e5DwKPA\n2uQO7v6kux+L7m4COvJb5tSmI3oRiVsu4T4P2J10f0/UlsmdwM/TLTCzdWbWY2Y9AwMDuVeZo5Kd\nUNUZqiISs7xOqJrZZ4Fu4Fvplrv7Q+7e7e7dbW1t+dy0iIgkqcyhTx/QmXS/I2o7h5ndBHwNuN7d\nT+anvPOjCVURKVe5HLk/Cywxs4VmVgXcCqxP7mBmVwB/Bqxx9/78l5mbTGeoZuyf8rhz16UzVEVk\n6soa7u4+DNwFbAReBR5z961mdq+ZrYm6fQuoB/7SzF4ws/UZViciIjHIZVgGd98AbEhpuyfp9k15\nrmtS0k2ojnfUHNeEau4rOM/Hi4hEdIaqiEiAFO4iIgEKKtzPnqHq597PcoZqvq6het7vltGEqojk\nSVDhXqoyfZ68iEihBBXupXuGqj7yV0TiFVS4i4hIgsJdRCRAQYV76gRq6gTrmP4pjzt3XfmbUM15\nzF1D8yKSJ0GFe6nSfKqIxC2ocC/VCdXcV3CejxcRiQQV7iIikqBwFxEJUFDhnnFCNesZqunWlccJ\n1ZxXMIGNiYiMI6hwL1WaUBWRuAUV7ppQFRFJCCrcRUQkQeEuIhKgoMNdF8gWkXIVZLinftxAtswt\n9EfyZvr4AxGRQgky3Eec9wRnnranI3oRiVvQ4S4iUq4U7iIiAQo63DWhKiLlKshwTw3ZbBOmhc5k\nZb6IxC3IcB9RKhOqOqQXkbgFHe4iIuVK4S4iEqCgw10TqiJSroIO9xHZz1At7vZFRPIt6HAvlQlV\nHdGLSNxyCnczW2lm28ys18zuTrO82sx+HC1/xsy68l2oiIjkLmu4m1kF8ACwClgG3GZmy1K63Qkc\ncPcLgf8B3JfvQkVEJHe5HLlfDfS6+w53HwIeBdam9FkLfD+6/Thwo1ncgyIiIjKiMoc+84DdSff3\nANdk6uPuw2Z2CGgB3s1Hkckee3Y33/m7HWmXvXt0CIB/8cPN1FRWjI51f+HhZ6mqGPt3bOfgMQC+\n9OjzzJhecc6y7f1HAfhvT2zj8c17MtZz8/1Pjd7e/95Q2j4/fGZXxscn+9lLe3PqJyLxaK2v5t2j\nJ/O6znRZVAixTqia2Toz6zGznoGBgUmtY1btdJbMqU/7dc3CZgCWz5vJkjn1XLmgCYBL5jam7X/d\nhS0AXNYxc8yyj10yB4Cbls4Zs+yCxhoAOppmnLv9RYntf3hJ6zk1f2Bxyzn3l8+byYKWWoDR74lt\nzWZpe2Pan7tm+rlPVW1VBbVVFWn7AnQ2z8i4rLW+OuOyyZg3K/O2MrloTj2NNZUsnzeTf379ooz9\nLp7TkLa9q6WWf7/6fRkf11iTy3FLeh+Mnq/KacaS2fWj7dMrjI8um0P7zJpxHz/yupqMm5bOprmu\nakz7b1/Wzn9ec8mY9ua6KuZG9cxpPPu8Lm6rO6dfQ00lv9PdybdvuwJIBMzHl7ePLr952Zwx6/70\nlR201ldz54cW8odrx277t943mw9dmHitXzK3kduvmU9D0n5/8LMr+MJ1CwFY95FFVFcmXsNzx9l/\ny9obuXRe+t+Bz147f0zbBxa1sPKSC+he0ETdOL8PqS6Ze3YbVZXT+MrNF43ebqqdzurlF3DHBxZw\n/2cu5/Zr5nPDxW188frFrPvI2dfq3Jk1NFRXcnnHTC7rmMmiaJ/PbqimqnJaxjdYfPfz3TnXeT4s\n2+eumNkHgK+7+8ei+18FcPf/mtRnY9TnN2ZWCbwDtPk4K+/u7vaenp48/AgiIuXDzDa7e9a/ELkc\nuT8LLDGzhWZWBdwKrE/psx64I7r9aeBvxwt2EREprKz/u0Zj6HcBG4EK4HvuvtXM7gV63H098F3g\nETPrBfaT+AMgIiJFktPApLtvADaktN2TdPsE8M/yW5qIiExW0GeoioiUK4W7iEiAFO4iIgFSuIuI\nBEjhLiISoKwnMRVsw2YDQG7n5Y/VSgE+2iAPVNfElGpdULq1qa6JCbGuBe7elq1T0cL9fJhZTy5n\naMVNdU1MqdYFpVub6pqYcq5LwzIiIgFSuIuIBGiqhvtDxS4gA9U1MaVaF5RubaprYsq2rik55i4i\nIuObqkfuIiIyjikX7tku1l3gbXea2ZNm9oqZbTWzL0XtXzezPjN7IfpanfSYr0a1bjOzjxWwtp1m\n9nK0/Z6ordnMfmFm26PvTVG7mdm3o7peMrMVBarp4qR98oKZHTazPyjG/jKz75lZv5ltSWqb8P4x\nszui/tvN7I5028pDXd8ys9eibf+1mc2K2rvM7HjSfnsw6TFXRs9/b1T7eV3mMkNdE37e8v37mqGu\nHyfVtNPMXoja49xfmbKheK8xd58yXyQ+cvgNYBFQBbwILItx++3Aiuh2A/A6iYuGfx3412n6L4tq\nrAYWRrVXFKi2nUBrSts3gbuj23cD90W3VwM/Bwy4FngmpufuHWBBMfYX8BFgBbBlsvsHaAZ2RN+b\nottNBajro0BldPu+pLq6kvulrOcfo1otqn1VAeqa0PNWiN/XdHWlLP/vwD1F2F+ZsqFor7GpduSe\ny8W6C8bd97r7c9HtI8CrJK4fm8la4FF3P+nubwK9JH6GuCRfuPz7wC1J7T/whE3ALDNrT7eCPLoR\neMPdxztxrWD7y91/TeJaA6nbm8j++RjwC3ff7+4HgF8AK/Ndl7s/4e7D0d1NQMd464hqa3T3TZ5I\niB8k/Sx5q2scmZ63vP++jldXdPT9GeBH462jQPsrUzYU7TU21cI93cW6xwvXgjGzLuAK4Jmo6a7o\n36vvjfzrRbz1OvCEmW02s3VR2xx3H7nq9jvAyIUyi7Efb+XcX7pi7y+Y+P4pxn77AokjvBELzex5\nM3vKzD4ctc2Laomjrok8b3Hvrw8D+9x9e1Jb7PsrJRuK9hqbauFeEsysHvgr4A/c/TDwp8Bi4P3A\nXhL/GsbtQ+6+AlgF/L6ZfSR5YXSEUpS3Rlni8oxrgL+Mmkphf52jmPsnEzP7GjAM/DBq2gvMd/cr\ngC8Df2Fm6a8mXRgl97yluI1zDyBi319psmFU3K+xqRbufUBn0v2OqC02ZjadxJP3Q3f/CYC773P3\n0+5+BvgOZ4cSYqvX3fui7/3AX0c17BsZbom+98ddV2QV8Jy774tqLPr+ikx0/8RWn5l9Hvht4PYo\nFIiGPQaj25tJjGdfFNWQPHRTkLom8bzFub8qgU8CP06qN9b9lS4bKOJrbKqFey4X6y6YaEzvu8Cr\n7n5/UnvyePU/BUZm8tcDt5pZtZktBJaQmMjJd111ZtYwcpvEhNwWzr1w+R3A/06q63PRjP21wKGk\nfx0L4ZwjqmLvryQT3T8bgY+aWVM0JPHRqC2vzGwl8G+BNe5+LKm9zcwqotuLSOyfHVFth83s2ug1\n+rmknyWfdU30eYvz9/Um4DV3Hx1uiXN/ZcoGivkaO58Z4mJ8kZhlfp3EX+GvxbztD5H4t+ol4IXo\nazXwCPBy1L4eaE96zNeiWrdxnjPy49S1iMQ7EV4Eto7sF6AF+CWwHfh/QHPUbsADUV0vA90F3Gd1\nwCAwM6kt9v1F4o/LXuAUiXHMOyezf0iMgfdGX79boLp6SYy7jrzGHoz6fip6fl8AngM+kbSebhJh\n+wbwP4lOUMxzXRN+3vL9+5qurqj9YeCLKX3j3F+ZsqForzGdoSoiEqCpNiwjIiI5ULiLiARI4S4i\nEiCFu4hIgBTuIiIBUriLiARI4S4iEiCFu4hIgP4/dGfqIoMcAZsAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yi8BStR5qXMs",
        "colab_type": "text"
      },
      "source": [
        "It also begins to progress through the environment for longer than chance around the 600 mark as well."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x01U6BLjqXMt",
        "colab_type": "code",
        "outputId": "7fbd134d-bf15-4f11-af82-1065eabc1033",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 286
        }
      },
      "source": [
        "plt.plot(jList)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f16939115c0>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJztnXecVsXVx39nO70uXVgUUFCUpoIi\nimJBEzVq1LwmGjUhJmqsMVgTE5OoMRrUqDE2YoxiQdFgoStNdOkdliIsLOyysLvAsn3eP557n73P\n89ze78P5+vHDs7fMnDt37pkzZ87MkBACDMMwTPqSEbQADMMwjLewomcYhklzWNEzDMOkOazoGYZh\n0hxW9AzDMGkOK3qGYZg0hxU9wzBMmsOKnmEYJs1hRc8wDJPmZAUtAAB07txZFBQUBC0GwzBMpFi6\ndOk+IUS+0XWhUPQFBQUoLCwMWgyGYZhIQUTfmbmOXTcMwzBpDit6hmGYNIcVPcMwTJpjqOiJ6DUi\nKiWiNYpjHYloJhFtlv7tIB0nInqWiIqIaBURDfNSeIZhGMYYMxb9GwAuSjo2EcBsIUR/ALOlvwFg\nPID+0v8TALzojpgMwzCMXQwVvRDiKwD7kw5fBmCy9HsygMsVx/8tYnwNoD0RdXdLWIZhGMY6dn30\nXYUQJdLvPQC6Sr97AtipuK5YOpYCEU0gokIiKiwrK7MpBsMwDGOE48FYEduL0PJ+hEKIl4UQI4QQ\nI/LzDeP9mQCoa2jCu4U7UVPfiPeXFkNt28k/TV+HRUX7NNNYt7sKS787AACYu6EUuyuO6OZZUnkE\nczbsxQdLi3GkrhGV1fX4ZOVuZw+SRPGBaszbWAoAqK5rwAcaz7b0u/1YX1Klm9a32/dj456Dutc0\nNQm8W7gTDY1NKec27KnCk59vwNMzNgIAlu04gD9NX4cv1u7Bgs2J5bpzfzW+2lSGhUX7sG3fYdW8\ntM7N3ViK4gPVunKqUVR6EF9vLY8/Q11D6jPsrarBrHV7NdNYXVyJVcUVqG1oxHuFO+NlvXnvQSzZ\nWq6b/+HaBny4vBgAUHawFp+v2QMA2Fp2CLe/vRyNTQJvLt4ef7Z50nPuqazBPe+uxP7Dddh3qBb3\nvLsSpVU18TIEmt9Lvcp7Kak8gtnrtZ9Jyc79zfVpT2WN6n1Pz9iIhz5abVj/vcLuhKm9RNRdCFEi\nuWZKpeO7AByjuK6XdIyJIH+bsRH//GorXpy3Bdv2HUaHltk4b2DX+Pmd+6vxr/nb8K/527D98UtU\n07j42fkAgO2PX4Ib3/gW7VpkY+XvLtDM8/vPLcS+Q7UAgKU7DqCk4gjmbizDST3boW/nVq481wXP\nfIXqukZsf/wSPPThGkxdvgvHdWmNIce0T7juyhcXx2XX4ocvGV/zzrc78cCHq1FZXY+fjzk24dxF\nf58f//2Ls4/DVS8uQpMA/jV/W0q6Z/91LpoU7ZFante9skT13I2vf4s2uVlY/eiFmnKqMe7prwAA\nk64dgvveX4XdFUdw57gBCddc8cIi7Ko4olkG339+AQDg1rHH4R9zt6B1bhbGD+6O85/5SvM5ZB6Z\nthYfLCtG746t8PBHa7CupAprH70Q5/7tSwBAQaeWeG5OEcZuKMXrN56Gn77+LVpkZwIAjtQ3Ynv5\nYdQ2NGLNrirM21iKA9V1aBKxPKcu34X73l+FsoO1uHVsv4R8L3t+IUoP1urK1lxGX6K2oQnbH78E\nV7ywELsraxLuO1TbgGfnFAEApq8qwfJHtOu/V9i16D8GcIP0+wYA0xTHr5eib0YCqFS4eJiIUVR6\nCABQWlUDADhY05Bwvrah0XKalUfqdc/LSj6Wby12V9TYzkuL6rrmtDZI1nhWBrmWfjIHqusAAPul\nf7VoFCJBkSejd84MB2sbjC/SQH5v5YdSn2GXSSu17GDs3VbV6NcBJXululdd14Cd+2NWe6Oi91V+\nOCZPUdmh+LEj9Y04Uh97x8UHqvFdeXX8WmUZVkjvQ+2ZSg/WphzTolbRy9ldWZNyvrGxOdMD1eaf\n3U0MLXoiehvAOQA6E1ExgN8BeBzAu0R0M4DvAFwtXf4pgIsBFAGoBnCjBzIzPiErhtZ5WThc556i\nDROHpGdsmZMZsCSMEWrtnIrHLXRQCGYrGSp6IcSPNE6dp3KtAHCrU6GYkCB9RBmkbu1G4SMzQlgf\nXooMQgjMXLcX4xTutigj+/YTa2PsWElFDVYVV6TcQ/Cup2aW4CXgmbGMCcKg0L2SwS1FsGZXpSvp\nuMn7S4sx4c2leOubHUGL4hlyvWhoErj0+YXBCqMBaRhKfsKKntFGqp9hsnpLD9YYRsJYwa1nMxp7\nCALZv72nMphIj3Rjza5K7D+sP84SVljRM9pIOlDL6vXTUJHzGvPkXIyfNF//Yhs4Vfdh6PUcLSgt\nZD/L/XvPLcAPXrDeawjenmdFzzjAz49MzqumPjXm2Ql++nDD1BhUHqnHpr368f9+0WQynEh1MNag\niSbSuNEmcgSPFULguWFFz+gQggoaho/EL/x81KtfWowLpDj2oHlt4TZL10etSoRhQJgVPWOIXT92\nTb21kMzSg6kxyH5hZG3vKK927Ic3arT8tLA3epCX2uxiM8jzNdKVMIxxsaJntDHw0Rtx5zsrLF1/\n2p9mp4oQ/DcCABjz17n4/nMLNM+78THLM3EZ8xjVj+Bt6XDAip7xjK+36a9jEjV27Lfun03GT+s1\nLI2kW6g9j9uPuKviCKrr7M8gDius6BltHJpDTpcVIPLeR+9nt/qz1SUY9/SX+GLtHt/y9BO/GhYr\ndcLqiotnPj4H17/6jVWRQg8resYVhBApa55ozag1n6aj263m5nkO26WIDaPVLt0ibAPZ8vusrmsM\nVTx6cjkVSqutuoVaPa5vbIrPc/ADVvSMNga6T3n6jUXbcebjc7B2d/MM0UwPFwpzC7ciItLNTeIl\nj36yDsP+ONOVtNzw0Xvx7owGpu+fuhqn/3m25YAFu7CiZ1xhydbYJmQ7FHHGTi36sFmkUSNsjY/W\n+zQrZxDRK3UNTZ7Mep4hue9qXZ4XogUrekYbA0VrpIczIlS7wqYUo4jXRag+GKufq4Cxda1nUPzy\nP0txyqMzTEgXbiL0KTJhRu2Dy3Ko6cOufN1cI98LzPSIauobVXdY8hNHPTcX6khjk9CMtJm9oVT1\nuBHKuhuGasyKntHGgo9ejQi46B0x7A/NfuYwfMx2OOHhz3H5P8K56qMWVpSomSr4xqLtGPTIF05E\nCj2s6ENOY5NQ3WvUT8z4RtUGNZ0OxiotPa+tewGgobHJ9LorjU0iVJuxNDYJNCbJbrbM1u52ZzVQ\nuzNj/Xi3Rzus6EPO959bgH4PfhZM5g4tcqeDsX7T78HPMOHNpaauvcLGKoYyXii2MU/OxaBHPnc/\nYRdx+tzqO0yxGjeD3c3BGZ9Y5+La63bRXKZY8VvN6ndT0fvVZsxav9fUdSuLw7XRiNq+rVFpZ6Mi\npxXC1vywRZ8GPPzRGhRMnO5Z+lquG6PKnJXp3hfseffeqbVpIQE/FJsQAk/N2OR9Rso8HZ63k4/d\nNF/6cgsem77eDXEMUasbfjcErOjTgDe//i5oEVSt/qi5bvyCvQ2JBFEeby4O/pvxE1b0jGn09La6\n68ZDYZijD7U4ehONRBANiWHsvk9yyLCiZ0xj5oNRNgZRWAIhXQlGudm7L4iO39E2iMuKnnEV5fej\n5bopmDg9PgU8LPg5vT6KHq3FW5wvOf3+0mLb9/7k1W9U31EU1LX6Foj+woqe8Qy9wdgwjCsEhd/G\npBsLt322xruG2Wp5KK1xM5Z5GHZ4ChpW9BFn9BNzfM1vwr8L8ZNXl2ieV1qrTgdjZ67biw0+Lenr\nJ2t3V6Jg4nRsLfNnExIzik7tvVrd2tBNhfrGwm0omDgdC4r2uZamn+iVRMHE6ThY4+/mJhxHH3GK\nD6TGT3sFETBjXXOcuZExFaWoGy8m82ghl6HZmH07WH0c5XuV+d+qEneEMUCtmvz+k3Upx+zuMBWG\nzbmDhi16xjR6yvCLtXtTruHBWH2OsvFATRyVg5mom4BdN2F4z6zoGdsYGexRsuiDwOSyOo6xbdGG\nQUOpEIRUVqN0TF/u0yfCip4xxIqiSAyv9EAYj3Cq077eWo6Rf56Nw7Xmfa9PfL7BWaY6JAxY+qQa\nk8vw0U/W4t73VvqStxvc9Ma3muc8a/N8arXYR88YorkEgkElPZpcN//8cisAYMOeKgzv0zFgaVzC\nYY/s9YXbXc1GPUzRPU05x+ba81EgQjYXExSyQrf63bvpukn+oMc+NQ879zdvW/jivC347furXMvP\nLiH1diRg2g3h08NMX1WCM/4yG+c+NQ/lh2ot3evXzFg/gx68wJGiJ6K7iGgtEa0horeJKI+I+hLR\nEiIqIqIpRJTjlrBMMMj62uoH46WPftu+w/jvNzvifz/x+QZMKdxpO72gB+zcRvk0YY86OVjbgN2V\nNdi677Bq9I9MkLNZ31pibd5HQn3SEzvsPnoi6gng1wBGCCFOApAJ4FoATwB4RgjRD8ABADe7ISgT\nHHYUxZ+mr8PHK3fH/374ozW+y8DEsN2IhXQwPWGHKZ90f32jRxn5JL9T100WgBZElAWgJYASAOcC\neF86PxnA5Q7zYALGjqL41/xtCX87nQkbFYs7ClJGwb1kFr/qRVPEC822ohdC7ALwFIAdiCn4SgBL\nAVQIIeTQg2IAPZ0KyQSDbM/Z9dEzwaGnl1YWVzhPxPnlvuHGWjMNTcbbeSYuzWAxA49x4rrpAOAy\nAH0B9ADQCsBFFu6fQESFRFRYVlZmVwzGQ0JWVz3FrQ8zbB84kOr2unlyYUCSOEN9K0F/8k7ej9cK\nur2OsPvoAYwDsE0IUSaEqAcwFcCZANpLrhwA6AVgl9rNQoiXhRAjhBAj8vPzHYjBeI3dwdijETMD\nhn70jJQrRT4/tyj++/6pq7H/cJ25RHQEPVhTj3veXYmKapNpuUkA9bDBKx+9TziJo98BYCQRtQRw\nBMB5AAoBzAVwFYB3ANwAYJpTIRkmnSB4r6se+HC16vG3FZFKhug0Wq8u2IYPlhVjb1VN8+UBaGCj\nHMmlVtWMRW/LEAr7YKwQYglig67LAKyW0noZwG8B3E1ERQA6AXjVBTmZAEj20XvFm4u3Y8Fm/1cp\nXLbjgOV7jCx2U4tsuaR8HvpoNapq6l1Jyy4ZPk6KU13UzKDAhRCqL6Wksib1oA4Nfq1X4RGOZsYK\nIX4H4HdJh7cCOM1Jukw48crl8PC0td4kbMAVLyxC744tLd2z/3AdOrXO9Ugia/zn6x1omZOFBy4e\n6E0GOi9c9vtnHSWzn01Z9D7IYReeGesjjU0Cj3+2Afsszv4DgC83lWHaCtXhDs8wqrhRCXn0EzO9\nHzd1Y5OXlqaJh1FOigtmDMefTB0NxoZgwhSvdeMjX20qw0tfbsH2fYfx0k+GW7r3hte+AQBcNsT/\naFVrg7HRtPBci7oxoXjIRS990E1tEBa9FQPDLTeZmTh6W+GVYffRM9aRrYL6RuOY3DBg9Imoz1YN\nWvXEfO8fLTfX+5H1wNyN5ha00puib5potoWqBL1wnRkfvRs9z+BrtTNY0TOGaH1MYXXdXPHCItw5\nZYWpa+Vne3rmJlPX3z9VPZqlOUFTybhG0CGvfg7GquHX4wddzk5hRc9okly3zfWC08hctYEZfZBO\n45fK/d/d0IXhVaj2B2N174zAhCmG8Q2vFEAQyzp4uXyzX8iP4JYP3AqJi5rpPz8RBdJ4hK23y4qe\n0ST5Ezb3wYSrghvhthJQSy9ZF4bZoG8IePzIarthGBnm0gv2bN17HoxljgbMfohRXlDNS4vS7bTf\nLSw2vighf6H6O92I+pOxomcMkSu5Fz56s/HJdnSI2cgbNzEVXhniVsvKnrdA9BWgV1hxL/kBK3om\nUBo9/AjunLICRaUHPUtfDTOPE2I9H2rZ1DAqbyJyaZDYzGBs8ApdC1b0AeBVdQiD5RA2auqDn7Pg\nxEd/pK7RVVmccriuAZ+uLlE951bUjdEMcKHxWz29cH8TfjUOrOgZ04T8mwkFbq+Z/vuP9dcB8luR\nTV22C796axnW7a6S8nc3/fmby3DHO+bmQPiJ8jEXbylXvybE3wcr+gCIWO/YU4L+ONx2VZhRvFYm\nGe2uPOJEHM84XGfNl2+WyiPWVuMMwmI/aGLF0LDpfFb0jCHpvJVgEA2Nm8UYJoUSdKOthlsD38pn\ns5pmGIqFFX0aYeVD277vMLaWHfItv6MFIWLr5vgZvx0kQYhnpWyFEK5b/a421BxHz3jJOU/Nw7l/\n+zJoMQLH7V7K29/swI2vf4upy7QHFMOsu5XWqhk5gx7s9Ct7q9kEXS7JsKJnTBOyumuadburUFmt\n7ld1+5l2VcR86rsrtH3raegBixHR+mEGpeKurlePhArz98GKPgBCXB808E5iPz6Oi5+dj2teXux9\nRlCs3a97jYtr3XhYfmakDLouG4UnejE57ddvL7d0fRgaAFb0aYTX9elAdZ3HOXjHhj3qE6fcj7px\nOT2fVamyOEKgnxwTW4/ejXRMXBPiEmNFHwBR7bo/+sm6oEU46jDcWCNEysUvWRImTIXn8RMI2QZT\nrOiDIKR1UxNPF+UKuDS8erawKiAjlD0cPYOEzPiomNDAip5xlXSMtXcbK0VkvFWeI1H00/Yuadcw\ntdaNX5E5IS4wVvQB4JUu3Heo1qOUzRPmyq5G2H30YUWOQtHqkTWZXJU0KljteSasXhmCJpMVfRpx\n+p9ne5Kul9U0aMXomevGpVIzSsdt8c22e2rRLMqy/HjlbncEUsGqEnWjMdeqJ6uLKxWy2EmXFzVL\nW5y82nIfrXZ2w9gn6AbML7Se0+msa6f5W73GKodqG3CotgHbyw+7n7gHsKKPGMMfm4UajQkbbmPH\nPxy1xsEPeb3MI90aFKvP49fjd2qdm/D3Sb/7Aif97otEWUL8MljRB4DT7z5sa5QrCeuHCgA19Y22\nym5vVU3C3/UW91UN8fefguUFuxLcKP5TZXG1S7sc06GFtRusLJrvA6zoA8Dpe3/wo9WuyGGWMFsq\nVhj71DwMfORzy/clj33c9Ma3hvfoz4w1n7dx0afHu7GL1kQ4t4l6KbOijyCfrt6je76uocn0Xqx6\nGCkktfN+um6sZlVSWWN8kQnmb96neS7qCsEtPB3AD6CUzc2MtZGujXvswIo+ALzWhQMe+gzX/NPc\n2i5VOpso+BHD7aS3EGqlqnguJ42f0TO63dmyKqtS6aZLz08Ny+GVIaudrOgDwI8qUPjdAVPXaa3q\neDTjRGGp6UlH+i9c+iKOUXsQsTF5YxxE9+jdyuvRpyFRi0iRUauLK3dW+CuDjwrvT9PXe5xDNFav\n1M3XIP+Qtk+28ep5/vKp13UthiNFT0Ttieh9ItpAROuJaBQRdSSimUS0Wfq3g1vCRp2o9Wz1GqYl\n29Q3SLZKGIvktYXbHKcRxucyg+kJU9K/bgeXmHJ5BFC4Xrmlpi7X3qDGTZxa9JMAfC6EOAHAKQDW\nA5gIYLYQoj+A2dLfTID86q2lKJg43fJ9enXb7xnuUekNxdf6EsC0FbtQMHE6SqvcGQT2BYsFPXPd\nXvVk3JDFBXbpbABjBVN6XhlqGrKW3raiJ6J2AMYAeBUAhBB1QogKAJcBmCxdNhnA5U6FZJxhFKVj\nh6aw1WQXcbJZhfLW9wqLAQCb9tqfJRq2QT0Zr6Si0DQRiYTzLZjHiUXfF0AZgNeJaDkRvUJErQB0\nFUKUSNfsAdDVqZCMefQs94KJ0/GHpDXlz3pyLqpq6lEwcTreLdypep+aTtfS86/Ot+b2MNteCAE8\nPWOjpbTt0tgkMPQPM1xLL1lZu9k78aoheGX+Vjw3p0jzvOuDzha43eIOT26g9WzKd6l8F5e/sNDw\nXj9xouizAAwD8KIQYiiAw0hy04iYY0v1MYloAhEVElFhWVmZAzGih1vfuR2/oZr/edeBWPc2WUkb\nKSQ1JfPN9v2WZTLLszqKx20OOIxGEhCa5efmMsVuI8s2adZm/es8MrzNNFxLtnlXx7TQkkvr/XxX\nXu2hNNZxouiLARQLIZZIf7+PmOLfS0TdAUD6t1TtZiHEy0KIEUKIEfn5+Q7EiB4haOATkCur/PH+\n5dP1GP7HmbpKxrXBqbAVhkNk14OyeJLdEaUH3VuYzrOGwKiRT7P3ZoTW8z48bY3hNWHAtqIXQuwB\nsJOIjpcOnQdgHYCPAdwgHbsBwDRHEqYRYV37PNla+edXW1F+WH9/WCH89adGbTBWiRP3Soh1BwOg\nIiLzULIc3n87gLeIKAfAVgA3ItZ4vEtENwP4DsDVDvNIG8La4jdb9OraVM169zvq5jfvr/I3Qxdx\n473/8X/q+/W6vh69VAWM2tUvN6W6WxNmydrMP7zfiLFgQ/84U/3eEDTXjhS9EGIFgBEqp85zki5j\nDtc/8uS/dbYFFdJ/TjGbxvqSKnvpBzyhyC1eXeA8tj8KhLXnFryqdgbPjPWR0FbiJB998nG9e5hE\n/H7FXr0HJyGmQHji6N0i6vWdFb2PuF1Z3BoQla1qa8vnRrzme4wQzcrS0VI3Ppezo3EXofozLQiD\n+8UJrOiZZos+6SOXFf/BmobUe1Sud5K3Hczk7vsH6nK3zXD1yogroKjgZXv74fJi7xKXYEXvI65H\n3bicjhX5moQ7PnonsIpzH7fqqN1kwtpRdNYr0z9f7cOOcazofaKhsQk3Ty40vG5r2SFc+vwCVBps\nkVZT34grXljkimyye2BvVQ0ueXa+yXtcyTrtaF7sS7jipzYsZ8989NbvcWOBs7CPY1nhBy8sNLW/\nsx9hyqzofUK5uJLea500ezNWFVdizgb1xaJkVhVXYvWuSpeki7G3qhZrd5uLbnG7N+EVvs8s9WpZ\nx6OE8BoQ1gVbvqMCa0x8o340bk7j6BmTNKkMVD03ezM6tMrBj0f2Sbl+6rJd2OxgMSwr2Pm2wvBB\nhtH4W77D53X6XU6Pkv71myCWNzCDl/U9gxV9+qC22uPfZm4CAFVFP3/zPv29SV2seXaSCto/H5Mh\n3ITVDeEVYWj8vcLus2ku9qWAXTdpRJPfU0ktoS7bzv3aa3k3NQlXdmLyOnwwqFIXaO7Frd9jb7KX\nnE6UKD1YgzveWW7KN81IsEWfPrit551OaFGipWuNNm1YtMWdXabSlRKp/NTCU8NKfAkEm/Xryc83\nYtqK3ejWNs9FqYLHbg/WjB2T4UPXjxW9T7i9UYerrhsb97jZ0KQjQghsLnVhjMVnf8i0FbuRl51p\n614BkbbuGy875H58SazofaIxxK6bID9O76NuwlvubuD28y3aUo5FW8rRqVWOo3TSzQ5wUsxG78iP\nsmIfvU+YqShzN5Ri2ord3guThB1lke4K1ClubTxhVMofBVBfjkbsum6q6xrw6CfqK4/KcHhlGmHG\ndXPjG9/6IEkqdjobYdDzYTYaZ2hsmm2VoMo53SzyoHhl/jYsKNKOngP88dGzRe8Tbvvo1dYDt4sd\na8Wtp/l8jf2Ny19ZsA1by/T94CFojxzxXflhvPzVFt1rjMrAL8LQ+HuGzWera2xyVw6bsKL3CbcV\n/Qvz9D9+SwT4gT700RrjizSoa2jCVS8tdlGa8FFV04A/f7pB95orX3RnKYxE2KRXYvsTMXGjH4EN\nrOh9IiQNuypBuW7qGpwXyuFaZ6GL6aDOKgzWRfKT+JLXAcvhNnbHpMwYeH6UFSt6n3DboncTe64b\n588zedF2x2k4JbxvxTxeVC2ni5qlG3aD5sx89+yjTyNCregDsuiPuDR7UjkLM9nyCnGxpzVc7DGW\nmVj7iMMr04kQ1/wwN0JGCAAvKsYrvt1+wNL96eZicAvb5SJVJT+sVD/x8hth100aEV1VGn6USwyk\n+v31S367S/HuTGJvymj5jKjhpS3Eg7FpRJiNZnuum/A8kHK8IM0MycDgckzEU4ueXTfpiRACU5d5\nv0/kBpOrJgYZR+82rJ8YL2DXDWOZLzeV4e53V3qez0V/925bwBAZ9InreSd9NWGSM91J57L2dFEz\ndt2kJ2Fb3+zT1fZnp4aB3QnbNCZ+NHqbtzCMWbzcT8KPHaZY0TP4wIYbKQw7TMl8vra5oUo2jibN\n3uyzNEw6wj56xhRhUoxuENZuOvvo3cGP7e2ixLZ9hz1Lm7cSZEJLaPR8kiC8IUqwhKZeuIyXYbhs\n0UeUVcUVqKpJXH+kpLImIGm8Yd/B2qBFUIX1vDtwOfoHD8ZGECEELn1+IW547ZuE4/e9vyogibzB\nrfXW3Yb1U3CE1Z0Xdji8MsIsN7HGBeM+bIm6g91iDNNEuqjArpsIwvU8aFjTM9EiEqtXElEmES0n\nov9Jf/cloiVEVEREU4jI2S7DacDO/byeil+wRe8OdvzGJZVHcKA6PGvjR4WouG7uALBe8fcTAJ4R\nQvQDcADAzS7kERnUDPqznpzruxxHK6zng+Oal792dYvLo4awu26IqBeASwC8Iv1NAM4F8L50yWQA\nlzvJg3EfL2f5BQ2HVzJRIwpx9H8HcB8AeW3YTgAqhBDyurHFAHqq3UhEE4iokIgKy8rSxwqIwmDU\nP7/aGrQInsFqnokaoV4CgYi+B6BUCLHUzv1CiJeFECOEECPy8/PtihE6wq/mgaXfWducwwv2Harz\nJF026N2h9GB6zfsIM2GPoz8TwKVEtB3AO4i5bCYBaE9EWdI1vQDsciQh4zphUIbrS8wtoWwVnrrv\nDvWNUTBZ0oNQh1cKIe4XQvQSQhQAuBbAHCHEdQDmArhKuuwGANMcSxkhlJ6byup6NDQm73jkL/Uq\n+Xtdr8oPmZs168aesXVJzxeGRoxhrBBq140OvwVwNxEVIeazf9WDPCLBKX+YgYenrQlUhjveWZ5y\nzOu43eGPzTJ1nXKvV4Y5evFe02cZX2KMEGIegHnS760ATnMj3ahR39iE2oZEK/XjFbsN76upb0Re\ndqYnMqmtNZ/OVm+6bUrNpD+hdt0wqVzwzFcY/PsZlu874eHPPZBGm3RWhmn8aEyaEpUJU4yE2prV\nYYzrbmgKdtyAYZhmIrEEAqNP+NQ88MXacK48yTBHI+y6YRiGSXOiMDOWcYmCidODFiEtmPLtzqBF\nYBhLsEXPMBZhRc8wqbCiZ9J7kDOIAAAcsUlEQVSKpgisNcQwfsOK3mvCOBqbxrCeZ5hUWNF7zMGa\nBuOLGNdIXhKBYRhW9AzDMGkPK3qGYZg0hxU9wzBMmsOKnmEYJs1hRc8wDJPmsKJnGIYJED9CglnR\nMwzDpDms6BmGYQKE17phGIZJc9h1wzAMwziGFT3DMEyaw4qeYRgmzWFFzzAMk+awomcYhklzWNEz\nDMMEyKAebT3PgxU9wzBMgGRm8ObgDMMwjENY0TMMw6Q5rOgZhmHSHFb0Dlj63QF8snJ30GIwzFHH\nT0b2CVqESMGK3gFXvrgIt7+9PGgxGOaow4fxS03GHp8fXOY2YUXPMEzkID+WfNSgc+tc/OLsYwPL\n3w6s6BmGiRwB6nkQAYQABbCBbUVPRMcQ0VwiWkdEa4noDul4RyKaSUSbpX87uCcuwzBMsAgRrOvI\nDk4s+gYA9wghBgEYCeBWIhoEYCKA2UKI/gBmS38zDMOkDRlBdilsYFvRCyFKhBDLpN8HAawH0BPA\nZQAmS5dNBnC5UyHt8vmaEkyatRkNjU2a11Qeqce0FbsM05q9fi92VRxxUzyGYWwSpOuEKHoWfZYb\niRBRAYChAJYA6CqEKJFO7QHQVeOeCQAmAEDv3r3dECMBIQRu+c8yAECLnAxMGHOc6nX3vLsCs9aX\n4sQe7dCvS2vN9G6eXIj2LbOx4pELXJeVYRhrBG5QBy6ANRwPxhJRawAfALhTCFGlPCeEEABUN8oS\nQrwshBghhBiRn+8sXKn8UC3mbiiFEALTVuxCfWMTmhS5zlpfiu37Dqveu7uiBgBQU99omE9Fdb0j\nORnGb1b/ng0TL4iaRe9I0RNRNmJK/i0hxFTp8F4i6i6d7w6g1JmIxlz/2je48Y1v8fHK3bjjnRV4\nbk4RmhQbMX6zbT/OeWqe12IwTOgIMgzRS4J+qqMp6oYAvApgvRDiacWpjwHcIP2+AcA0++Lps6q4\nAq/M34pNew8CAPYdqgMAlFbVoLHJhx13GSbkREsdRYeoWfROfPRnAvgJgNVEtEI69gCAxwG8S0Q3\nA/gOwNXORNRm0ZZyPP7ZBtVzfuyszjBhJ00N+sCfK2rqxbaiF0IsgLbBcJ7ddK2Q3KoKhXZvYk3P\nmKTwoXH4Yu0ePPjhmqBFcZ2ohQGaJWiXVF2DdiRfGIn0zFg9P1kjK/rIkZ0ZzMdLSN8eYJrq+UC5\n8MRuqNMJ2Q4j0Vb0OpVYaLyH9SVVqDySGj1T29CIFTsrbMnR0NiEwu37bd3LNDNx/ECs/8NFvudL\nRKHtis+552xH9zux6K8c1stR3l5zxnGdfM9z218uxnkDu7JF7yfJ3bcmE66b8ZPm40cvf51y/N73\nVuHyfyxE+aHalHPCwNz764yNuOqlxWZEZnQQQqBFTqbv+VIsc9/zNYPTbeacKPowDzgGJZqsc0b3\n6xyQBPaItqJP+lsOtCHSd92sK6lKObZNirNXu0uZVENjE9burkw4P2PtXjPiMibxW8FkZ2UEatEP\n7tlO85wZRX1Ctzaa55wUpTLvTY+NR6dWOQ5ScxmdB+vdsWXKsR7t8nDTmX0Tjv3n5tNtZz9uUFdM\nu/VMzfMbH/O/Z6pHpBV96mBs879uDsYqU3ryi4245NkFKCo9GD+2TWMyFmMN+ZX5sVmykpzMjEAN\n+pws7c/QjEHetkW2o/vN3JuTlYGMkJn4Ws+WpSLn5UN7IjOpmJ2OCeVma7+33Cz/e6Z6RFrR67pu\nVFxoyWveCCGwpexQwjG1+Hul62bFjpgfX47ZZ9xDSE2q35Ei2Zlk6J7zEr2nJSL0VyzNcVrfjinX\n6OlfJ9EpybfqpXTvBQNMp9tfZ6kRK1iZtHTvBcenHAuns84bIq7oE/+WP1YidYs+OeZ+8qLtqE0a\nVHnqi40p9x1NFSJI5FemZpF5SdChenrZt8nLSrAc2+alRkR71UYll4uenFnJ5rIOLXOdL7Glp+TP\nPzF1eS213khIh2U8IeKKPtmij/1bWlWLIypr13y0InF/11XFlSnXLN5aHv+9p7IGtQ2NqhWiuq7B\nhsTe8psLU62WKCEXs9+uG2XeYaNtXjaUHVE/lZOVt2BFLiOPycKJ5xqmodfo3HfhCarHkzvrwuFb\nj9IyCNFW9El/y5Vt9oZS/HxyYcr1+5IiatQsOdl109gkMPIvs3H3lJUJFWJ/dcxlc9Mbqelr0cYF\nC8YMQ49p70s+XiG/vwtP7BZY3kFg1KMwcit5JbqVjs7A7toDwsnkZev7r1vnOPtetAyFlF5+WFt3\nD4i2ok923Sje3FaDAdLK6npV32aDpOgbJCf/jHV7EpSAWgy+EX4NYjlxQYwbqLqadAq/ufB4DOvt\nXoMyoGtrXD0iFq8tv78/XzEYfTqlRk54ifyKrxreCzPvGoNjOrbwLW+jt9agMEX9nPFtxWI95/gu\nqsdbqCj1lgYhtGRCKxGav/9J1w7B0ofGGd7jt5538ztxSqQVffKgnZV1zE75wwxVi0UesJUrRXJl\nt/Od+eWK6No21/a9uTqRH0pa5mSifUv3wuza5mVjdP/YMtX9u8SswuzMDFw8uLtreejRrW0egGar\nuW1eNvp3bYM8H6MmjNrnotJD+hd45qP3Jl3DfE1eN6JPbGB6YPe2pupkcs/IaZvZoWVitFO7pOin\nzq3tf49uE2lFn+q6sfbmaupTQ3MaGmNpHK6N+eCJnFeIzAzC/PvGmr7+eyfbU3LH5tuLZvjtReo+\nTTUam0S8nJ+++hTVa8w2GgCQlUm49JQemHnXGJw/qLlXcc/5AxzPCtVjzj1nY8Fvx2LG3WMSjsvK\nLche/Xu3jMLSh8apWqmq8zwcSjv1V2dgyQOpy1NpKdwPfjnKdNrqjQUlhJSe1T9x8lEGEVY8cj7+\n+zPtOHci4PZz+2HW3WdjQFdzbqPkUnJabl3a5mHuvefgznH94zIpybbwHXhNeCSxQbJFb1Uhf7xy\nd8oxuZs8/LFZ8TycVogRfTrgGJVJHFrIVqZftM7NNG1GxeYoxH530LCiurczL3+2FK3RP+ljzcrM\nsN1wmeHY/Nbo1aEl2ubFrLCCTq0AAMebVBpWOV0lLFJGzltmQNc26NQ6F51MWoQn9tCecGWG/Na5\n6KpS57RcgVrvXY1Rx6ovUyDX8ZY5mTgu6T0TAe1b5uAMg9mnGRkU3xVOllRuNNRE98Lt1bdzq3iU\nWLI+OqVX6nsJysqPtKJPVk5uLGTWoBKAX6ti+VvhgYsHWro+zAtRNQlFs6chp5W3YCWU0k7YpVo4\nohrjBnXFJ7eNxg/l8QKbdamHSiM36+6z8dpPT1W9/pPbRmPMgMQd1vQeU02sX53TvE3mUBt+4WwL\noZGAtXkOz//fMN3zUyak9g7MpJ/sUs3IIMy6ewz++ZPhAIDCB8dhwW8Te9EpUTcGr3j6r0eb6ok3\nu3ljyI3Nz0Yfi39dPyLhWq1esNdEWtEnV4cX521xnGZDUm04Ut+IoX+cGf/bjgKw+iEFEddtNsdG\n0ey60brHShFZsUZ72xigtdKTGtyrXbzsT+llbyAtV2XwsV+X1milEXk1uFe7lHLUG9NRs0pb5mah\nZ/vY4PFpBdo9By20ZuYmV8MhUlSX0WCqErW1i47v1jqedmuTDbEZ+nVpg5ZSxE6n1rno1SHx3Sd/\nu0YW/ok92iXUH62ZtHIqct3p0S72LjIyCCf1bJtwbavcYGbMRlrRezGD0khJRWX548/vPMv8xYpy\nlBWGFsrHd6NBkv2bZqitb8LMu8bgnQkjDa916v768xWDddcykfn4tsRr9Ge5Nv+edfcYzL33HNXr\nrNTr/90+Gq1zs+JpX3d6H7x3i7EP/ednNa/7oqXAkq3mv18zFJ/cNtrUwOe8e8/B4vvV4+HvGjcg\nIeVkBax8/k9/rV6P7VQ9tdnyavzxshNTegMAsGii+jYbcYteRabkMszKSFS5t5/bz5xQDom0ovfK\n8C2YOF3znJ0NwoNoHI7pYM36lf3hFw/Wj2Hv1jYv7lPVWuTKii/UyozK47u1Qf+ubVSt9OQIiJ4d\nYg2W3TqSl52JU0zMSzg5yfLXy0/Zs+vSNg99O7dSvc6KzCdJC6Ip7znVhFXft3OzX1yrx5ksR4uc\nTAzu1c7UonMFnVuhe7tUo6FtXpbhO1fme1wXjTIyFgEA4j58IHXwVauW9mjfIqU3AAD5bdT9681L\nd8T+1qv/yb21Lhppug0reh9osrh/rdZjTUmyZHOzMvDCdeo+UKLm8vmlwoerxa/P7Yf/3Hw6hvfR\nVhJZGYQrhvXEAxcPxH9/dnpcyVhBbWVBM7x/yyhMunYIgFivY8qEkRg3sDl2u74xsYzdqhpaFuWs\nu8eo+m/1rHF54BfQl08vDaM21E7gQI6keBdOPBcz72qOQtKSIiszA1N/dYblfP53+2jMvuechGNq\nrlBlvtlJFrDVYZr3fjEq3jNL+QxNFtXX95+Hz+7Q7iHLjyAvZKbclCT5VWZlEj5S9BT9ctNGWtFH\nZZu0ZL+/EVpRK33zE62bk3u104w3J1B88FIv4iN2bezjHd2/M+p1ds4ZMyAfRLHQOL2IiORvlyi2\nZgsAnKm4z8qM4REFHdFGoShPP7ZTwgzL2obEJS/kFR2VlqsdBvVoq3q8Xxf1nkX/rtr5De/TbP0r\nP3CjxcOUVp/WOIXsIjDbmVJanfKEvp7tWyREP+lN9BvWu4O5jBSc1LNd3CrWU3DK7zpZhotOsjZr\nukOrnHjPTH7k1lK901r1M7kMu7XLw8Du6vUAaG4v5MlhyuCN5ACCNnnZ8bEOwL85NpFW9FEheUXM\nyTedpnrdzLtiUQPXjypQnVGo9Pf992en45Xr1SM5ZDKTwr6Sv61+KqsI6il6s1WyjcoA26y7z8YH\nvzwjwSKbebd7cfLJFv1x+a3wxo2n4okrB7uWh8wnt43WPPfXq07BJRrzIJ65Zkj8t7IckhVL8sf/\nv18353f/+BNw//jUeQ9m4v9fuX4EOrfOkfI0bhG8VEFy2gKpMmu1Aa//9NT4pDo73Xn5mR+8ZCBe\nvWEEhvdpbqzm3zc2PhvasvKV0s2TBp5rFEZHp9a5eOG6YZh19xj8+6bTUsbAMtmiNyYqFn3yBKKz\nB+QnTA6S6d+1DS48sRsyMgjXnHpMynmlZXxGv85o11J/HXJ54Ecup+QYcbUlb5VWs1qaZuiSNBBK\nALq2zcPwPh0SeivdLMTbq8vTLFDHpPGCLm3ycM7xXeJRGG4yWCU+WqZVbhbG9Ffv7bTMyYq7SZSN\ndvIYTrK126VNXvy+nKwM/OLsVFdcXHHqKPBxg7rGFaUZw1/LJ+0KcsOkIoiWtT/2hGZXnZ0vX+7F\n5GZl4LykJT+O6dgyPp/B6pIlnaVyOkH6vvKTYuUvHtwd/bq0SQmjBWI9Dj/wZ7Utjwi7nn/9xlNR\n39AU7+JPmTAyPkj49NWn4H+rSnD/1NWq904cfwJGFHTAbf9dHj9GAGbcNQbbysxtdCJbJvJKm8kh\nfmof2biBXTDp2iGYNGuzynpB2gXeJi8Lk64dgm37qrFtX+KUfeWHe8vZx2F7eTXuc2GlTTlaZMyA\nfIzp3xmPTV8PILb2yfdO7uE4fbN89ZuxGPPXuZbuUdZdN/YflctYfqXz7xuLksoaAMAXd46J99Tk\nfPXGjV68bhheW7gNN57ZN16mTph/31jsqjiieT45EkUPJ2EN8iNrGYhyQ2DVoL/u9D5o1yIb3z+5\nB845Pl9VoSfz5s2nYe3uqoRxJi+JtqIP+TKhY5MWejpdMUuwTV42Ljyxm6aiz8vOxPdO7pGo6InQ\nvV0L1WiGZGIWfax8ZNfRcfmtsGJnBRqbhKZ/nIhw2ZCemLlub4qi12tYzzyuM849IWYlPTNzU8I5\n5QBsVmYGnvqhO5NGZB/9+YO6xmb3Slw2pKcr6ZsleQE0vXopD5Yqy1JtKQ6rJOd4TMeWcQPjeMVW\ng/IMWD1Lcvzg7hjv4lpDSllSEcjOMv8dy9FVyVFWZpAVuVY9lsMvrbpTMjMoXufMlttZ/fNxVn/j\nBsEtIq3oQ7azWQLJM+LUsDr5Suv6KRNGpsQ2Eyhu0Q/t3QFPXnUyvn9yD9w2tj9eXbAVPx9zLJ6d\nvRmAehz1E1eejHOO74J731upSFMb5bdx69h+6NE+D/27tsHcDaX48cg+Jp/QGvLCY7X1jbjutN64\na8pKgzu8IdnVICvzs/p3xvWjChLPqSyW58pGKzquECWPXX4SzurfGZcN6YmtZYcTGgE/UT5xjkG4\n5b9vOi3eUFw/qgAtczJx1fBU16YRctEoLfqPbj0zrtgb4w1BiBWLTSKt6MP8Pk4tsB6VYITWN3y6\nxnoi3du3QOnBWmQQcPWI2IfRu1NLPHrZSQCaBzDV4qhb5WbhquG9EhW9TnkrLcScrAxcc2pvAPai\nM8zSRVqtMzck+5kmx0R3b5enOhYDJJZl8viCHZqT09f0rXKzcMWw2DIPd51vfvs/LzGaOa50hWRm\nULxuWaW9FGWjnJ2qjICR3VlBbHzjNZFW9FbDFv3g79cMQZMQri7lK2OlA0AUi7L4clNZyuCojBzv\na3aJBj2XxEOXWFvPxw1uHt0XuVkZ+NFp5j78+8efgL8kbSfpFs/+aGhCFIcWaq/wvIFdcMWwnpi6\nbJdjOdyem/f2z0eqLmPglPiYgrC+RIhdHrxkII7v1ibFpSojW/Q+ieMrkX6khsbwKfpu7fLiFpMR\nVi0HM5Nh5DXphYhFTVw1XFuW+gaLil5HXC+iW4zIzszAjWf2NT279ubRfY0vssmlp/SIh87JYwdq\n69vIs4mVRUlEePrqISnXJtzXOtFwyMtOfOZOrWLv3W1rdNRxnRKsXrdQhle2beFP3WmZk4XrRxVo\numaMBmujTKQt+jBipYq0b5mDP/9gMB74UH1ANhkz1tr7t5yBRVv2aS5UpUSOxMgxORgW9vr/6a/P\nwvby1Iik1396Ktq2yLa03IKSqb86AxXVdWhsMjdl/Xsn90BJZQ2uH5U6NvHBL8/A4i3lqrJ8fNuZ\n2CNFyiTz7i9GYdGWffHZl5/dMQbLvjsQP/+P64bhszUlni7trMWzPxoKIFZOn64qwTgNd5UaQgDX\njDgGD364BgDwzDXBrO4INLtuWNGHjLD40lpkZ8Y3I7e6rs3/nd7btKI387THdGyJazqac2XISqNF\ntrlqEITVboVBPdqqzmRVxl/bweo4Q2YG4RaVWHdAPwLl5F7tcbJGByz5vfbt3CphrZz8NrkpA79+\ncekpsVDWYb07mC4rpS5VNno/GGquN+wFTYJ99KHEzgu55/wB+HpbORYWlbsiw0UndsPjVw7GkD/E\nljK2M/j40o+Ha4aLvfTj4fhsTQkOVNdr+trt8qcfnISTerbFyGO1l0h4++cjUXmkDhv3HMJPzyxw\nNf8geOqHp+CEgCJN0o0nrzxZd/JY1Ghkiz6cGIVlqTGwe1vcfl5/XPfK164o+5ekjQ5kjHa4V0Nv\n/Y6LTupmeX0Ps3RqnYvbztVfJnjUcZ0kOTwRwXf0xiwYa1ytMnvbLPISH2HSqfIM9iyNZZujjCeK\nnoguAjAJQCaAV4QQj3uRz9gTuuDsAfn4clMZ+nVpjaLSQ+jcOgcdW+VACGDksZ3QoWU2np1ThLZ5\nWcjMoHg3/umrh+CHLy3Gjv3V8fRuP7cfnptThFHHdsLireUY1rs9lu2oSMm3fcts/OUHgxPcNI9f\nMVh3Qat05bHLT8JgG6tYmuWT20Zjxc4DxhcykeKFHw/He4U70V9ab+mDX44y3gTdZ5nSCbK7ZZpm\ngkSZADYBOB9AMYBvAfxICLFO654RI0aIwsJCV+VgGIZJd4hoqRDCcHamF+GVpwEoEkJsFULUAXgH\nwGUe5MMwDMOYwAtF3xPATsXfxdIxhmEYJgACmzBFRBOIqJCICsvKyoISg2EYJu3xQtHvAqAcju8l\nHUtACPGyEGKEEGJEfr5/q7gxDMMcbXih6L8F0J+I+hJRDoBrAXzsQT4MwzCMCVwPrxRCNBDRbQC+\nQCy88jUhxFq382EYhmHM4UkcvRDiUwCfepE2wzAMY41Ir17JMAzDGOP6hClbQhCVAfjO5u2dAexz\nURy3YLmsEVa5gPDKxnJZIx3l6iOEMIxmCYWidwIRFZqZGeY3LJc1wioXEF7ZWC5rHM1yseuGYRgm\nzWFFzzAMk+akg6J/OWgBNGC5rBFWuYDwysZyWeOolSvyPnqGYRhGn3Sw6BmGYRgdIq3oiegiItpI\nREVENNHnvI8horlEtI6I1hLRHdLx3xPRLiJaIf1/seKe+yVZNxLRhR7Ktp2IVkv5F0rHOhLRTCLa\nLP3bQTpORPSsJNcqIhrmkUzHK8pkBRFVEdGdQZQXEb1GRKVEtEZxzHL5ENEN0vWbiegGj+T6KxFt\nkPL+kIjaS8cLiOiIotxeUtwzXHr/RZLsjrZM0pDL8ntz+3vVkGuKQqbtRLRCOu5neWnphuDqmBAi\nkv8jtrzCFgDHAsgBsBLAIB/z7w5gmPS7DWKbrQwC8HsA96pcP0iSMRdAX0n2TI9k2w6gc9KxJwFM\nlH5PBPCE9PtiAJ8htvf4SABLfHp3ewD0CaK8AIwBMAzAGrvlA6AjgK3Svx2k3x08kOsCAFnS7ycU\nchUor0tK5xtJVpJkH++BXJbemxffq5pcSef/BuCRAMpLSzcEVseibNEHusGJEKJECLFM+n0QwHro\nr7t/GYB3hBC1QohtAIoQewa/uAzAZOn3ZACXK47/W8T4GkB7IurusSznAdgihNCbJOdZeQkhvgKw\nXyU/K+VzIYCZQoj9QogDAGYCuMhtuYQQM4QQDdKfXyO2GqwmkmxthRBfi5i2+LfiWVyTSwet9+b6\n96onl2SVXw3gbb00PCovLd0QWB2LsqIPzQYnRFQAYCiAJdKh26Qu2Gty9wz+yisAzCCipUQ0QTrW\nVQhRIv3eA6BrAHLJXIvEDzDo8gKsl08Q5XYTYpafTF8iWk5EXxLRWdKxnpIsfshl5b35XV5nAdgr\nhNisOOZ7eSXphsDqWJQVfSggotYAPgBwpxCiCsCLAI4DMARACWLdR78ZLYQYBmA8gFuJaIzypGS5\nBBJuRbGlqy8F8J50KAzllUCQ5aMFET0IoAHAW9KhEgC9hRBDAdwN4L9E1NZHkUL33pL4ERKNCd/L\nS0U3xPG7jkVZ0Zva4MRLiCgbsRf5lhBiKgAIIfYKIRqFEE0A/oVmd4Nv8gohdkn/lgL4UJJhr+yS\nkf4t9VsuifEAlgkh9koyBl5eElbLxzf5iOinAL4H4DpJQUByjZRLv5ci5v8eIMmgdO94IpeN9+Zn\neWUBuALAFIW8vpaXmm5AgHUsyoo+0A1OJB/gqwDWCyGeVhxX+rd/AECOCPgYwLVElEtEfQH0R2wQ\nyG25WhFRG/k3YoN5a6T85VH7GwBMU8h1vTTyPxJApaJ76QUJllbQ5aXAavl8AeACIuoguS0ukI65\nChFdBOA+AJcKIaoVx/OJKFP6fSxi5bNVkq2KiEZKdfR6xbO4KZfV9+bn9zoOwAYhRNwl42d5aekG\nBFnHnIwuB/0/YqPVmxBrnR/0Oe/RiHW9VgFYIf1/MYA3AayWjn8MoLvingclWTfC4ci+jlzHIhbR\nsBLAWrlcAHQCMBvAZgCzAHSUjhOAf0hyrQYwwsMyawWgHEA7xTHfywuxhqYEQD1ifs+b7ZQPYj7z\nIun/Gz2SqwgxP61cx16Srr1Ser8rACwD8H1FOiMQU7xbADwPaWKky3JZfm9uf69qcknH3wBwS9K1\nfpaXlm4IrI7xzFiGYZg0J8quG4ZhGMYErOgZhmHSHFb0DMMwaQ4reoZhmDSHFT3DMEyaw4qeYRgm\nzWFFzzAMk+awomcYhklz/h/zaIlVfIjVJAAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oYI8q4jXqXMw",
        "colab_type": "text"
      },
      "source": [
        "## Deep Q-networks\n",
        "\n",
        "While our ordinary Q-network was able to barely perform as well as the Q-Table in a simple game environment, Deep $Q$-Networks are much more capable. In order to transform an ordinary Q-Network into a DQN we will be making the following improvements:\n",
        "+ Going from a single-layer network to a multi-layer convolutional network.\n",
        "+ Implementing Experience Replay, which will allow our network to train itself using stored memories from it’s experience.\n",
        "+ Utilizing a second “target” network, which we will use to compute target $Q$-values during our updates.\n",
        "\n",
        "<img src=\"https://github.com/DataScienceUB/DeepLearningMaster2019/blob/master/images/deepq1.png?raw=1\" alt=\"\" style=\"width: 800px;\"/>\n",
        "\n",
        "\n",
        "See https://jaromiru.com/2016/09/27/lets-make-a-dqn-theory/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I75sIlTpqXMx",
        "colab_type": "text"
      },
      "source": [
        "### Convolutional Layers\n",
        "\n",
        "Since our agent is going to be learning to play video games, it has to be able to make sense of the game’s screen output in a way that is at least similar to how humans or other intelligent animals are able to. Instead of considering each pixel independently, convolutional layers allow us to consider regions of an image, and maintain spatial relationships between the objects on the screen as we send information up to higher levels of the network.\n",
        "\n",
        "### Experience Replay\n",
        "\n",
        "The second major addition to make DQNs work is Experience Replay. \n",
        "\n",
        "The problem with online learning is that the *samples arrive in order* they are experienced and as such are highly correlated. Because of this, our network will most likely overfit and fail to generalize properly.\n",
        "\n",
        "The key idea of **experience replay** is that we store these transitions in our memory and during each learning step, sample a random batch and perform a gradient descend on it. \n",
        "\n",
        "The Experience Replay buffer stores a fixed number of recent memories, and as new ones come in, old ones are removed. When the time comes to train, we simply draw a uniform batch of random memories from the buffer, and train our network with them. \n",
        "\n",
        "### Separate Target Network\n",
        "\n",
        "This second network is used to generate the $Q$-target values that will be used to compute the loss for every action during training. \n",
        "\n",
        "The issue is that at every step of training, the $Q$-network’s values shift, and if we are using a constantly shifting set of values to adjust our network values, then the value estimations can easily spiral out of control. The network can become destabilized by falling into feedback loops between the target and estimated $Q$-values. In order to mitigate that risk, the target network’s weights are fixed, and only periodically or slowly updated to the primary $Q$-networks values. In this way training can proceed in a more stable manner.\n",
        "\n",
        "Instead of updating the target network periodically and all at once, we will be updating it frequently, but slowly."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dngNSL5rqXMx",
        "colab_type": "text"
      },
      "source": [
        "While the DQN we have described above could learn ATARI games with enough training, getting the network to perform well on those games takes at least a day of training on a powerful machine."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qeRPcATCqXNE",
        "colab_type": "text"
      },
      "source": [
        "## Other examples\n",
        "\n",
        "https://www.youtube.com/watch?v=gn4nRCC9TwQ\n",
        "\n",
        "https://www.youtube.com/watch?v=wBrwN4dS-DA\n",
        "\n",
        "https://www.youtube.com/watch?v=n7370dzCZ1o\n",
        "\n",
        "https://www.youtube.com/watch?v=fv-oFPAqSZ4\n",
        "\n",
        "https://www.youtube.com/watch?time_continue=33&v=tXlM99xPQC8"
      ]
    }
  ]
}