Policy_Value_New.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Policy_Value_New.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Chris-Fourie/rl_at_ammi/blob/master/Policy_Value_New.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZO_beNp3sWOZ",
        "colab_type": "text"
      },
      "source": [
        "## Policy Evaluation\n",
        "$$\n",
        "\\def\\E{\\mathbb{E}}\n",
        "\\def\\given{\\mid}\n",
        "\\def\\states{\\mathcal{S}}\n",
        "\\def\\argmax{\\text{argmax}}\n",
        "$$\n",
        "The first step to improving a policy is to evaluate the state-value function $v_\\pi$ for an arbitraty policy (see Sutton & Barto 4.1 for details). The state-value according to a policy is computed as:\n",
        "\n",
        "$$\n",
        "\\begin{aligned}\n",
        "v_\\pi (s) &= \\E_\\pi \\left[ G_t \\given S_t = s\\right]\\\\\n",
        "&= \\E_\\pi \\left[ R_{t+1} + \\gamma G_{t+1} \\given S_t = s\\right]\\\\\n",
        "&= \\E_\\pi \\left[ R_{t+1} + \\gamma v_\\pi ( S_{t+1}) \\mid S_t = s \\right]\\\\\n",
        "&= \\sum_a \\pi(a|s) \\sum_{s^\\prime, r} p(s^\\prime, r \\mid s,a) \\left[r + \\gamma v_\\pi (s^\\prime)\\right]\n",
        "\\end{aligned}\n",
        "$$\n",
        "\n",
        "The value of $v_\\pi$ can be updated iteratively for all $s$:\n",
        "\n",
        "$$\n",
        "\\begin{aligned}\n",
        "v_{k+1}(s) &= \\E_{\\pi} \\left[ R_{t+1} + \\gamma v_k (S_{t_1} \\given S_t = s)\\right] \\\\\n",
        "&= \\sum_a \\pi(a|s) \\sum_{s^\\prime, r} p(s^\\prime, r \\mid s,a) \\left[r + \\gamma v_\\pi (s^\\prime)\\right]\n",
        "\\end{aligned}\n",
        "$$\n",
        "\n",
        "For any policy $\\pi$, your task to to implement a policy evaluation function based on the following pseudocode:\n",
        "\n",
        "\n",
        "![policy-evaluation](https://i.ibb.co/j4zZ9Xw/policy-evaluation.png)\n",
        "\n",
        "(source: S&B Section 4.1, page 75)\n",
        "\n",
        "Recall $p$ represents the transition probabilities, which we will get from a simple GridWorld below, $\\gamma$ is the discount factor, whereas $\\theta$ is a  small threshold which indicates when to stop updating our value function. A simple GridWorld looks something like the following:\n",
        "\n",
        "![gridworld](https://i.ibb.co/Lk166s4/gridworld.png)\n",
        "\n",
        "(source: S&B Section 4.1, page 76)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kcyiLAxzsWOe",
        "colab_type": "text"
      },
      "source": [
        "### Exercise 1\n",
        "Complete the `policy_eval` function above so that the following evaluates without error:\n",
        "```python\n",
        "v = policy_eval(random_policy, env)\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IlHK529IyDIG",
        "colab_type": "code",
        "outputId": "65076158-92ff-4b84-e78b-5fa3cc2464d1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 103
        }
      },
      "source": [
        "!git clone https://github.com/dennybritz/reinforcement-learning"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'reinforcement-learning'...\n",
            "remote: Enumerating objects: 1213, done.\u001b[K\n",
            "remote: Total 1213 (delta 0), reused 0 (delta 0), pack-reused 1213\u001b[K\n",
            "Receiving objects: 100% (1213/1213), 5.34 MiB | 23.57 MiB/s, done.\n",
            "Resolving deltas: 100% (785/785), done.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l1hb2MCusWOh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import sys\n",
        "if \"/content/reinforcement-learning\" not in sys.path:\n",
        "  sys.path.append(\"/content/reinforcement-learning\") \n",
        "from lib.envs.gridworld import GridworldEnv"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l-uVxEF5sWOv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "env = GridworldEnv()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aJpUFdu-sWO5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Implemented during class\n",
        "def policy_eval(policy, env, discount_factor=1.0, theta=0.00001):\n",
        "    \"\"\"\n",
        "    Evaluate a policy given an environment and a full description of the environment's dynamics.\n",
        "    \n",
        "    Args:\n",
        "        policy: [S, A] shaped matrix representing the policy.\n",
        "        env: OpenAI env. env.P represents the transition probabilities of the environment.\n",
        "            env.P[s][a] is a list of transition tuples (prob, next_state, reward, done).\n",
        "            env.nS is a number of states in the environment. \n",
        "            env.nA is a number of actions in the environment.\n",
        "        theta: We stop evaluation once our value function change is less than theta for all states.\n",
        "        discount_factor: Gamma discount factor.\n",
        "    \n",
        "    Returns:\n",
        "        Vector of length env.nS representing the value function.\n",
        "    \"\"\"\n",
        "    # Start with a random (all 0) value function\n",
        "    V = np.zeros(env.nS)\n",
        "    while True:\n",
        "      \n",
        "        delta = 0\n",
        "        # For each state, perform a \"full backup\"\n",
        "        for s in range(env.nS):\n",
        "            v = 0\n",
        "            \n",
        "            # Look at the possible next actions\n",
        "            for a, action_prob in enumerate(policy[s]):\n",
        "              \n",
        "                # For each action, look at the possible next states...\n",
        "                for  prob, next_state, reward, done in env.P[s][a]:\n",
        "                  \n",
        "                    # Calculate the expected value. Ref: Sutton book eq. 4.6.\n",
        "                    v += action_prob * prob * (reward + discount_factor * V[next_state])\n",
        "                    \n",
        "            # How much our value function changed (across any states)\n",
        "            delta = max(delta, np.abs(v - V[s]))\n",
        "            V[s] = v\n",
        "              \n",
        "        # Stop evaluating once our value function change is below a threshold\n",
        "        if delta < theta:\n",
        "            break\n",
        "    return np.array(V)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nu4YWgt6sWPB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "random_policy = np.ones([env.nS, env.nA]) / env.nA\n",
        "v = policy_eval(random_policy, env)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eahtKZidsWPK",
        "colab_type": "text"
      },
      "source": [
        "Run the cell below to verify your $v_\\pi$ has converged to the proper value."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FVB6jrAzsWPO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "expected_v = np.array([0, -14, -20, -22, -14, -18, -20, -20, -20, -20, -18, -14, -22, -20, -14, 0])\n",
        "result = np.testing.assert_array_almost_equal(v, expected_v, decimal=2)\n",
        "print(result)\n",
        "print(\"If `result` prints None, then your algorithm was successfully implemented\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lR3Erk1GsWPX",
        "colab_type": "text"
      },
      "source": [
        "# Policy Iteration\n",
        "\n",
        "Now that we can evaluate the state value function for any given policy, we can use the values to improve our policy. \n",
        "\n",
        "First consider the state-action value function (q-function), which consists of selecting $a$ in $s$ and then behaving according to $\\pi$:\n",
        "\n",
        "$$\n",
        "\\begin{aligned}\n",
        "q_\\pi(s,a) &= \\E \\left[ R_{t+1} + \\gamma v_\\pi (S_{s+1}) \\given S_t = s, A_t = a \\right] \\\\\n",
        "&= \\sum_{s^\\prime, r} p(s^\\prime, r \\given, s, a) \\left[ r + \\gamma v_\\pi (s^\\prime) \\right]\n",
        "\\end{aligned}\n",
        "$$\n",
        "\n",
        "Let $\\pi^\\prime$ be a policy such that, for all $s \\in \\states$:\n",
        "$$\n",
        "q(s, \\pi^\\prime (s)) \\geq v_\\pi (s)\n",
        "$$\n",
        "\n",
        "Then the following holds for all $s \\in \\states$:\n",
        "$$\n",
        "v_\\pi^\\prime (s) \\geq v_\\pi (s)\n",
        "$$\n",
        "(See S&B page 78 for proof)\n",
        "\n",
        "Consider a simple policy improvement which consists of $\\pi$ selecting an action according to the maximum state-action value:\n",
        "\n",
        "$$\n",
        "\\begin{aligned}\n",
        "\\pi^\\prime (s) &= \\argmax_{a} q_\\pi (s,a) \\\\\n",
        "&= \\argmax_a \\E \\left[ R_{t+1} + \\gamma v_\\pi (S_{t+1}) \\given S_t = s, A_t = a \\right] \\\\\n",
        "&= \\argmax_a \\sum_{s^\\prime, r} p(s^\\prime, s \\given s, a) [r + \\gamma v_\\pi (s^\\prime)]\n",
        "\\end{aligned}\n",
        "$$\n",
        "\n",
        "Alternating between policy evaluation and policy improvement is known as **policy iteration**. Implement the algorithm based on the following pseudocode:\n",
        "\n",
        "![gridworld.png](https://i.ibb.co/GHpV8kV/policy-iteration.png)\n",
        "\n",
        "(source: S&B page 80)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R1B_VsImsWPZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "pp = pprint.PrettyPrinter(indent=2)\n",
        "env = GridworldEnv()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XGyLfXKOsWPl",
        "colab_type": "text"
      },
      "source": [
        "### Exercise 2\n",
        "Complete the `policy_improvement` function, based on the above pseudocode, so that the following runs without error:\n",
        "```\n",
        "policy, v = policy_improvement(env)\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_JuXGYh_sWPq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def policy_improvement(env, policy_eval_fn=policy_eval, discount_factor=1.0):\n",
        "    \"\"\"\n",
        "    Policy Improvement Algorithm. Iteratively evaluates and improves a policy\n",
        "    until an optimal policy is found.\n",
        "    \n",
        "    Args:\n",
        "        env: The OpenAI envrionment.\n",
        "        policy_eval_fn: Policy Evaluation function that takes 3 arguments:\n",
        "            policy, env, discount_factor.\n",
        "        discount_factor: gamma discount factor.\n",
        "        \n",
        "    Returns:\n",
        "        A tuple (policy, V). \n",
        "        policy is the optimal policy, a matrix of shape [S, A] where each state s\n",
        "        contains a valid probability distribution over actions.\n",
        "        V is the value function for the optimal policy.\n",
        "        \n",
        "    \"\"\"\n",
        "    \n",
        "    def one_step_lookahead(state, V):\n",
        "        \"\"\"\n",
        "        Helper function to calculate the value for all action in a given state.\n",
        "        \n",
        "        Args:\n",
        "            state: The state to consider (int)\n",
        "            V: The value to use as an estimator, Vector of length env.nS\n",
        "        \n",
        "        Returns:\n",
        "            A vector of length env.nA containing the expected value of each action.\n",
        "        \"\"\"\n",
        "        A = np.zeros(env.nA)\n",
        "        # Fill in the blank here. You should loop actions, then calculate the\n",
        "        # expected value by looping next state values and rewards\n",
        "        for a in range(env.nA):\n",
        "            # Fill in the blank here. You should loop actions, then calculate the\n",
        "            # expected value by looping next state values and rewards\n",
        "        return A\n",
        "    # Start with a random policy\n",
        "    policy = np.ones([env.nS, env.nA]) / env.nA\n",
        "    \n",
        "    while True:\n",
        "        # Implement the main algorithm here, check the pseudocode as useful reference\n",
        "        break\n",
        "    \n",
        "    return policy, np.zeros(env.nS)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hFFqHVzxsWP2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "policy, v = policy_improvement(env)\n",
        "print(\"Policy Probability Distribution:\")\n",
        "print(policy)\n",
        "print(\"\")\n",
        "\n",
        "print(\"Reshaped Grid Policy (0=up, 1=right, 2=down, 3=left):\")\n",
        "print(np.reshape(np.argmax(policy, axis=1), env.shape))\n",
        "print(\"\")\n",
        "\n",
        "print(\"Value Function:\")\n",
        "print(v)\n",
        "print(\"\")\n",
        "\n",
        "print(\"Reshaped Grid Value Function:\")\n",
        "print(v.reshape(env.shape))\n",
        "print(\"\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eAD93o78sWQB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Test the value function\n",
        "expected_v = np.array([ 0, -1, -2, -3, -1, -2, -3, -2, -2, -3, -2, -1, -3, -2, -1,  0])\n",
        "np.testing.assert_array_almost_equal(v, expected_v, decimal=2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I0rpOroesWQJ",
        "colab_type": "text"
      },
      "source": [
        "## Value Iteration\n",
        "The isue with policy iteration is that every time we improve our policy, we must do a full sweep through all states again to evaluate the new policy! This is extremely inefficient. Value iteration combines both evaluation and improvement in the same step. Implement value iteration based on the following pseudocode:\n",
        "\n",
        "\n",
        "![gridworld.png](https://i.ibb.co/SVTmBFQ/value-iteration.png)\n",
        "\n",
        "(source: S&B Section 4.4, page 83)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LtMKX6-QsWQL",
        "colab_type": "text"
      },
      "source": [
        "### Exercise 3\n",
        "Complete the `value_iteration` function below, based on the above pseudocode, so that the following evaluates without error\n",
        "```\n",
        "policy, v = value_iteration(env)\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v5yBKZKOsWQN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def value_iteration(env, theta=0.0001, discount_factor=1.0):\n",
        "    \"\"\"\n",
        "    Value Iteration Algorithm.\n",
        "    \n",
        "    Args:\n",
        "        env: OpenAI env. env.P represents the transition probabilities of the environment.\n",
        "            env.P[s][a] is a list of transition tuples (prob, next_state, reward, done).\n",
        "            env.nS is a number of states in the environment. \n",
        "            env.nA is a number of actions in the environment.\n",
        "        theta: We stop evaluation once our value function change is less than theta for all states.\n",
        "        discount_factor: Gamma discount factor.\n",
        "        \n",
        "    Returns:\n",
        "        A tuple (policy, V) of the optimal policy and the optimal value function.        \n",
        "    \"\"\"\n",
        "    \n",
        "\n",
        "    V = np.zeros(env.nS)\n",
        "    policy = np.zeros([env.nS, env.nA])\n",
        "    \n",
        "    # Implement!\n",
        "    return policy, V"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fapSyrHAsWQW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "policy, v = value_iteration(env)\n",
        "\n",
        "print(\"Policy Probability Distribution:\")\n",
        "print(policy)\n",
        "print(\"\")\n",
        "\n",
        "print(\"Reshaped Grid Policy (0=up, 1=right, 2=down, 3=left):\")\n",
        "print(np.reshape(np.argmax(policy, axis=1), env.shape))\n",
        "print(\"\")\n",
        "\n",
        "print(\"Value Function:\")\n",
        "print(v)\n",
        "print(\"\")\n",
        "\n",
        "print(\"Reshaped Grid Value Function:\")\n",
        "print(v.reshape(env.shape))\n",
        "print(\"\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LXCsMRo-sWQf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Test the value function\n",
        "expected_v = np.array([ 0, -1, -2, -3, -1, -2, -3, -2, -2, -3, -2, -1, -3, -2, -1,  0])\n",
        "np.testing.assert_array_almost_equal(v, expected_v, decimal=2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "buewi_IUsWQm",
        "colab_type": "text"
      },
      "source": [
        "## Exercise 4\n",
        "Now that you have implemented Policy Iteration and Value Iteration, plot the average running time for both algorithms by varying the discount rate $\\gamma$ between 0 and 1. What do you observe?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAUy7uTfsWQp",
        "colab_type": "text"
      },
      "source": [
        "## Alternative implementation\n",
        "The algorithms you've seen in this notebook can alternatively be solved using matrix notation and in closed form in some cases. See [this](https://drive.google.com/file/d/1UR20JtQRjFyrvCseusVuPBmQIpB3XFAH/view?usp=sharing) notebook."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jD7MwFnEsWQs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}